Analysis: Support for multiple Storage backends

[altinnadmin]

Description

Currently we're using CosmosDB as backend for metadata JSON-documents and Azure Blob storage for storing data and attachments.

There are four main problems with this:

1. Does not work on Linux, making it hard to test locally and provide a complete local test environments for apps. This also means we need to use time on providing a "mock" of Platform that apps running locally can use. Sadly it seems like MS is not listening as this has been requested for 4 years. Update: seems like it's now in preview.
2. This is a dependency on Azure, making it harder than needed to move the Storage component on-prem or to other clouds.
3. There is a risk that using only Cosmos and BLOB as backends can "bleed into" the designs of our external Storage APIs.
4. Not open source

We took these dependencies with open eyes, since there are a lot of other document databases and BLOB/S3-like object storage services out there, and since using managed PaaS-services gives us big Ops-benefits since we're a small team.

And since these dependencies are taken only by our Storage microservice, abstracting these dependencies away, it should be easy to change backend without changing anything else (apps, the rest of platform, etc).

In scope

What's in scope of this analysis?

• Find good alternatives to Cosmos and Blob that also can run locally in Linux-containers on a dev machine.

Out of scope

What's out of scope for this analysis?

• Create new bindings to other cloud PaaS services.

Constraints

• Alternatives must be able to run on Linux in containers
• Alternatives must be open source with liberal licensing
• Alternatives must be able to scale

Analysis

• Refactor repository services in Platform Storage to make them easier to test.  #95 seems very relevant to this work
• LocalTest is missing an implementation for "get many instances" app-localtest#30 is an example of a kind of problem this will solve
• Controllers in Storage are using the CosmosDB library altinn-studio#5996 is a required change

Conclusion

Short summary of the proposed solution.

Tasks

• Is this issue labeled with a correct area label?
• QA has been done

[SandGrainOne]

@altinnadmin Authorization also interface directly with CosmosDB with a separate implementation.

[TheTechArch]

@lorang92 techforum

[SandGrainOne]

There are now options for running a CosmosDB emulator on linux. This is also true for the new Storage emulator, Azurite.

[altinnadmin]

There are now options for running a CosmosDB emulator on linux

That's nice, but does not solve the other more serious problems related to Cosmos. It's breaking 2 of our architure principles.
Also, an emulator is not the same as beeing able to run the real thing locally.

[SandGrainOne]

There are no plans of adding support for multiple different storage backends as of now. Though there are work being done to replace CosmosDB with PostgreSQL. That alone will make us more independent of Azure.

#193