cmg: Make Gateway fully async

[phoevos]

The Gateway is currently implemented as a FastAPI server, but most of its external interactions are synchronous. Even though suboptimal, this approach worked while the Gateway was simply a thin layer taking care of quick admin operations like registering tasks for execution. With the introduction of the serverless functionality, however, the Gateway can quickly become the bottleneck under load, while multiple requests wait for models to be deployed (a synchronous operation by design, meaning that requests don't return until a container has been created to handle them). Now, we should look into making the Gateway fully async, focusing on the following interactions:

• Postgres

  • Database stays the same; changes required on the client side to support the asyncio integration
  • We have experience implementing this switch for the CogStack Catalogue, where we used asyncpg and aiosqlite to support both production and development workflows

• MinIO

  • Given that MinIO has been phasing out its commitment to the open-source project (having recently announced a feature freeze, after deciding to distribute it as source-only), we should eventually look into replacing the object store, with proposed alternatives being Garage, Seaweed, and Ceph. Even though, this is not a priority, moving to a vendor-neutral, S3-compatible client can prove beneficial down the line. At the same time, however, the MinIO Python client doesn't provide async support, which makes our decision easier
  • With the above in mind, on the Catalogue side we opted for using aiobotocore, an async-compatible port of the core library behind boto3, the official AWS client for managing AWS resources

• RabbitMQ

  • We might have a couple of alternatives here. Currently, we're using pika with BlockingConnection to manage our queue. We should explore transitioning to the AsyncioConnection adapter, looking into different libraries (e.g. aio-pika) if that's not enough
  • We don't have experience with adopting async for our queue operations yet, so there is no tested path forward

• CMS healthcheck and /info calls

  • This change should be relatively straightforward by switching to an async HTTP client with httpx which we already use for the auto deployment feature