Inefficient usage of AWS DescribeDBInstances and DescribeDBClusters API

[jlisthood]

Currently, the collector calls FindRdsInstance once per full snapshot if CollectSystemInformation is configured. This leads to at least one AWS API call per database every collection period (multiple if using Aurora).

When the collector is configured to collect on hundreds of DBs in an AWS account, the O(N) AWS API calls all happening one after another can lead to throttling:

Rds/System: Encountered error when looking for instance: Throttling: Rate exceeded
	status code: 400, request id: <REDACTED>

This leads to missed system state snapshots and affects other tools trying to call the same APIs within the account.

AWS's DescribeDBInstances and DescribeDBClusters APIs actually allow describing up to 100 instances or clusters at a time in one API call by the use of filters see here. What are our thoughts about modifying the collector logic to handle multiple system snapshots at once to use fewer AWS API calls?

[lfittl]

What are our thoughts about modifying the collector logic to handle multiple system snapshots at once to use fewer AWS API calls?

I think overall this would make sense (to reduce rate limiting issues), but the challenge is that we'd have to rework that part of the collector, since currently each server is processed independently (in its own Go routine).

Would have to do more thinking on how this could be implemented, since I think we'd want it, if we don't introduce a huge maintenance burden with it.