Shape lookup data uses a lot of RAM #3511

We currently validate very read request by using the request parameters to define a shape, then retrieving the shape's handle by using the shape as a key to lookup the handle in an ETS table. We do this for every request, even if the params include a handle.

This means that our api requires shape information to be held in fast in-memory storage.

However this shape data is massive. The shape metadata table and associated shape -> handle lookup table require hundreds of MB of RAM on a server with thousands of shapes.

We need to reduce this RAM usage and move to some form of on-disk storage of the shape information.