NubesGen new architecture

[cmaneu]

NubesGen re-architecture proposal

We've shipped V0.1 and our first public technical preview with what is our "MVP" (minimum viable product). To support our future roadmap, substantial architectural choices needs to be done. This document is here to detail these changes and to seek feedback from the first users and partners before starting implementation.

Goals

• Flexibility:
  • Support more services:
  • Support paid/partner offerings:
  • Support internal-only features/configurations:
  • Support sovereign & on-premises regions:
• Composition:
  • Different UIs
  • Different outputs: Bicep, Pulumi, etc...
  • Different CI/CD systems:
• **A more "open" product:
  • Allowing external contributions more easily
  • Allowing public and private extensions

Bird views

A model

• Services: A service represent one (or more) resources deployed. Each service is a "deployment module" (a Terraform module, a bicep template, ...) that can be composed together. Services can output values
• Service type: we can have public and private service types
  • Public ones includes: hosting, database, storage, messaging, ...
  • You can extend Nubes to use private options - like a custom VM that you manage and that offer a "PaaS-like" experience for your users
  • There are "reserved" service types, like "compute", "database", "gitops", etc...

Service package

Each service is contained as a "package", which is a folder with the following files:

• a Service YAML file
• The template files (as a Mustache template)

A File format

NubesGen is an API, but we need to facilitate accessing this API:

• For users without Infrastructure as code knowledge, they may want that NubesGen takes care of everything for them.
• For supporting deployment of custom workflows with non-default values: let's say you want to help people deploy a famous CMS (Content Management System) with Nubes. All the default options are great but you need to add one environment variable, and the default database environment variable name needs to be changed.

Warning: Do NOT reinvent the wheel
The purpose of NubesGen is not to become yet another Infrastructure as Code dialect.
The sole purpose of this file format is to let users access the opinionated defaults and choices made within Nubes in an easy way. If you just want to spinup a new storage, then you should probably use another tool (AZ Cli, ARM, Bicep...)

projectName: my-blazor-quiz
region: francecentral
resources:
- name: frontend
  type: webapp
  subtype: docker
  size: free
  docker: 
    registry: github
      username: cmaneu
      password: ${pipeline.secrets.github_registry_accesstoken} 
    image: cmaneu/my-blazor-quiz
  envVars: 
  - key: MONGO_CS
    value: ${quiz-db.connection_string}
- name: quiz-db
  type: cosmosdb-mongodb
  size: serverless
gitops: true
monitoring: true
traits: 
# Traits allows to change configuration cross-resources
  - type: security
    level: medium

An API

NubesGen is only an API. As such, it should expose:

• An API definition metadata: This will help our experiences to dynamically generate the UI.
• Accept data in several ways: simple GET parameters, a POST query, a JSON file, a YAML file, etc...

A generation pipeline

The generation pipeline is a multi-step pipeline, allowing to generate a model prior to generating the files. This allows for several scenarios, including:

• Generating a (visual) graph for the generated components,
• Computing the cost estimation of deployed resources,
• Doing some validation

This pipeline is composed of several steps, and developers can extend it:

• Parameters parsing and validation: This phase parses the input against the service definitions loaded in the current app context, and validates them.
• Model assembling: As services can request other services, we're assembling all these parameters into a graph of objects to generate.
• Cross-concerns modifiers: We'll implement some cross-concerns modifiers, like security (enable Azure Defender), management (add a Tag, a resource lock, ...), monitoring (connect to an Azure Monitor workspace). This stage will go through all services and modify as needed the parameters
• Template generation: This stage will take individual templates and generate them
• Packaging & publishing: the API supports several ways of exporting, and may support more in the future. This stage takes the generated templates and send them to the destination (tar.gz and zip for now).

The last two stages can be parallelized.

[imod]

This sounds great! The first thing coming to my mind is the scoping: Is this only meant to target azure or do you think this could attract a broader audience e.g. AWS, GCP? I know this would require a lot more skills in terms of the final artefacts to be generated, but beside that, everything else sounds like it would be reusable.
If so, then some of the tools will support more targets then other (e.g. Bicep only for Azure, but Terraform and Pulumi all of them).
Even if this is not the goal from the start, I think keeping this in mind while designing the underlying architecture would be beneficial.

[cmaneu]

Our idea for now is to focus on Azure. The main reason behind it is we want to push best practices and opinionated options in the config we generate. We can do this because we're experts in it.
Personally, I'm not opposed to the fact we can support other targets in the future.
We can try to keep it in mind while designing. If a tradeoff should be made at some point, I'll incline towards supporting only Azure for now.

[agoncal]

@cmaneu What is specific to Azure in NubesGen ATM ? Is it just the Terraform templates ? Or are there any specific Azure setup in the NubesGen engine itself ?

[cmaneu]

@agoncal it's mainly in the Terraform and GitHub actions templates. The engine - despite having hardcoded values - is not really cloud-specific

[sinedied]

Regarding the File format, I'm not sure I see the link between what is explained here and the API. To me it sounds more like a serialized model of what options one user chose for its project. The goal here doesn't seem clear to me: is it to have a different interface to access the generator, being able to replay/change project settings, update to a newer version? Why YAML vs JSON or something else?

[cmaneu]

@sinedied The file format is here to support two scenarios:

• Packaged settings for common solutions: think about having a "one click" to get to Nubes website. This can be used in several places like: any tutorial, open-source projects documentation (one-click deploy strapi on Azure), Azure Architecture Center
• Iterative infrastructure updates: Let's say you've used Nubes to create a Web App and a Postgresql DB. Now you want to add a Redis Cache and a Function. We want to provide a simple way for users to "load" their existing configuration.

This file format may also be used in other features that I can't disclose it publicly (linked to non-announced features on other products & services).

So, there is no direct link between the NubesGen API and this file format. To me, it was interesting to include this in this discussion, but the separation of concerns lacks clarity.

About the file format, I don't have strong opinions. We must agree on one, and YAML seems to be the simplest to edit.

[mikhailshilkov]

Hi there, thank you for kicking this off! A Pulumi developer here, we'd love to spread our love here :)

One complication is that Pulumi is multi-language (four currently, maybe more in the future). We have internal code generators that can take an intermediate format and produce examples in all four languages. Is there an extensibility point where we could plug a generator like this, instead of writing N copies of templates? Our gen is in Go, so it would have to be a custom binary or HTTP extension.

[cmaneu]

Hi @mikhailshilkov,
That's a great remark. My idea for now is to have a Java interface that you should implement to add the generator to NubesGen. So we can probably figure out something there.

I'll loop in @jdubois about Go-Java interop. We can go the HTTP way, but that would imply to run a Go server with NubesGen which may complicate our deployment architecture.

[damienaicheh]

Hi there, concerning the different UIs to support, I think it will be interresting to have only one model managed by the API.
With that we will be able to generate a VS Code extension (to easily use Nubes Gen in all projects), and the Website form. With that we will create one ecosystem of tooling based on only one data model, which will be easier to maintain.

[sinedied]

I fully agree on that!
One idea would be to agree on a model format like JSON, that could be used to generate the matching models in any other UIs. I'm thinking of JSON as it would probably have to be compatible with different languages as we already use Java, C# and TypeScript in the various UIs, but what really matters here I think is having one single source of truth (the generator/API models) to generate models that we can use elsewhere.

[cmaneu]

Can't agree more, it's in my document :)

An API definition metadata: This will help our experiences to dynamically generate the UI.

My idea was to thinker something with OpenAPI. However I'm not sure it's technically feasible. If not, we can default to a JSON file.

[agoncal]

Hi. Concerning "a more 'open' product allowing external contributions more easily" it would be nice to have some Azure credits while developing and testing a new feature. I've been working on deploying a Quarkus app with NubesGen and it has already cost me a few dollars. It's only a small amount and it does matter much for now, but I can see that as a stopper for some developers or in the long run.

[cmaneu]

Hi @agoncal,
I do agree with you. Believe it or not, that's something that is quite complicated for us to do!
Would you like to share more about what kind of components you're deploying? I want to understand the limitation of our free tiers in this context.

[agoncal]

@cmaneu I'm deploying the Quarkus JHipster sample app with NubesGen (Java app + DB). Because I'm struggling a bit (can't view the logs) I'm doing several tests and redeploying again and again. ATM I have been charged $3. Nothing major... but can get higher if I go on further. And BTW I ran out the free tier months ago. I'm sure I could create a new account to get another free tier, but that's not the idea.

But I do understand that this is a complex topic to solve.