Heap and Garbage Collection
We have had a small number of Out of Memory problems in Shibboleth. In order to detect actual memory problems, someone responsible for Shibboleth administration needs to understand the normal Shibboleth memory use patterns. These show up in the Tomcat logs/gc.log file.
...
I believe that there is another memory use that gets bumped up during the 30 minute period that is not showing up in the GC log because it is a type of memory not managed by GC. That would explain why GC is being triggered by much lower starting numbers (1800M) where before it was triggered at 2750M. Interestingly, after 30 minutes we have a jump back from starting value 1605M in line 1 to the more normal 2759M in line 2, and then after then next full cycle the ending number has returned to 890M. That means that something happened to Java memory around 22:35, but whatever it is it doesn't show up as heap or GC.I propose that we increase Java memory in general as well as the heap size. It seems clear that we need more than a 3G heap to process InCommon.
Linux VM
Shibboleth runs on a 6G VM
The Java+Tomcat+Shibboleth process uses about 3.6G of active data resident in real memory, but the process address space is 6.25G of virtual memory (the extra space being memory that the process has reserved by system calls but is not really using and based on observation will never actually use.
The Linux system seems to keep 0.6G free and 1.4G used as data cache, or a combined 2G of memory not allocated to programs and available for assignment.
Java has a 3G heap. The 3.6G process memory use suggests that in addition to the 3G heap (which is fully used because Java fills it up before starting GC), there is about 0.6G of code, classes, connections, and other objects used by Java, Tomcat, Shib, and their libraries.
In newer versions of Java there is an additional type of Java memory pool used for NIO (new I/O). It is used for "direct" I/O buffers. Each buffer in this area is connected to an object on the heap, and when the object on the heap is freed by GC the corresponding direct memory buffer is also freed. However, while Tomcat and other Java Web Servers use NIO for modern high performance web I/O to browsers, Shibboleth does nothing this exotic. So there is no need to change this parameter.
Increasing the heap delays GC until the extra space is full, so it increases real memory use. Minimally if we want to bump the heap size from its current 3G to 4G and we do not want to take any chances, we would also bump the VM by 1G. However, memory is cheap and it would be useful to see how all the numbers we are seeing change if we also increase system memory.
So the proposal for now is to bump the VM from 6G to 8G and the heap from 3G to 4G and then watch the numbers. We expect that each routine (young) GC will run a bit longer but GC will happen less frequently. The users will not see a difference.