Gen 2 Heap GC

For the past two weeks I had been trouble-shooting a resource leak problem.  The data processing job, which works fine for a day worth of data, would bog down after processing 4 or 5 days of data.  Typical symptoms are 100% CPU usage with max RAM consumption.  Connections to the process’s WCF service would time out too.

After some experimentations, I ruled out resource leaks and concluded that the problem is memory usage.  The process builds a humongous dictionary and only releases all objects at the end after 40++ minutes.  This means almost all the objects end up in generation 2 heap at the end, and gen 2 garbage collection is expensive.  My observations from perf counters (GC % time is 99%), connection timeouts (GC freezes user threads), and memory dump analysis (no thread was doing actual work) supported the theory.

The solution, unfortunately, is to retool how the process uses memory.  I also threw in induced garbage collection in between steps to prevent generation 2 heap build-up.

Advertisement