Excess garbage in Metadata

[carl-mastrangelo]

Metadata today is a Hashmap of ArrayLists, and each arraylist has an object array. This creates a lot of garbage since Metadata objects are short lived.

Some ideas on how to improve the situation:

• Recycle the objects
• Store headers as a flat array of an initial size, and swap to using a full map if slow. (and avoid Strings when possible)

[buchgr]

This creates a lot of garbage since Metadata objects are short lived.

Short lived objects are a good thing, as they are "free" to be cleaned up by the GC. GC times are usually a function of the surviving objects (that need to be copied & traversed).

That being said, I don't know what you see in your profiler, but just wanted to point that out :).

[carl-mastrangelo]

@buchgr I ran a benchmark last night, after I noticed ~30% time spent in GC. I raise NewSize and MaxNewSize to 40G (thats gigabytes) and saw ~200kqps performance improvement. It takes about 1 second per 10G to start a GC (minor), which moved GCs running from once every ~50msec to once ever 4 seconds.

I don't think anyone in production will be using such a large new gen, so we need to allocate less garbage.

[pgrosu]

@carl-mastrangelo So why not just use an Object Pool pattern to store and over-write the object data within them, and thus keep a relatively average-amount collection of objects so that you don't have to overwork the GC?

[buchgr]

I understand that. I would argue that's not a good way to look at it, though. Obviously, with more heap, GC will run less often. So you increased the young gen size ~80x and the time between GCs increases ~80x. What I am saying is that short-lived objects, that is objects that die between GCs (i.e. in less than 50ms) are not what determine your GC duration. That is, it's likely not what that 30% is spent on (you need to inspect the GC logs to know for sure). What's more likely happening here is that by massively increasing your heap size and thus times between GC (from 50 millis to 4secs), you are also making what would else be long-lived objects (and thus be expensive to collect) to short-lived objects. My guess would be that this what's driving the throughput improvement.

So following that logic, what we want is to allocate fewer bytes (not fewer objects), thus increase the time between GCs, thus increase the chance that an object dies young (between GCs) and thus lower GC duration.

I agree that less garbage is good, though.

[ejona86]

@carl-mastrangelo, can this be closed now, as fixed by #2274?

[carl-mastrangelo]

No, the object canonicalization is still covered by this.

Also deadlines still create lots of trash. We don't see it in profiles often because Deadlines are usually not set, but they still cost.