SOLR-17547: Collect ZK Metrics via Curator

[HoustonPutman]

https://issues.apache.org/jira/browse/SOLR-17547

So there are two ways we can do this. AdvancedTracerDriver and CuratorListener. The CuratorListener gives us much better information on what is being done, and we can easily do what needs to be done. All metrics we have currently can be collected given the information passed to the CuratorListener interface. Unfortunately, the CuratorListener interface only works for Async actions, such as the curator Async API and Watch events. all other events need to go through the AdvancedTracerDriver to be collected.

The AdvancedTracerDriver gives us most of the information we need. There are a few issues:

• The matching of events is done based on Strings hardcoded in Curator. These could change at any time. (Minor annoyance)
• The "Get Children" command trace does not give the information needed for the number of Children fetched.
• The "Multi Transaction" command trace:
  • The trace does not give the number of commands that are in the transaction, which is something we track currently.
  • The MultiTransaction trace is not an AdvancedTrace, it's a normal Trace for some reason. AdvancedTracerDrivers cannot read regular traces, so we can't even see these. We need to update Curator to use an AdvancedTrace for this command.

So this is close to good to go, but we need some Curator fixes before we move forward.