Store and distribute proto files in a registry

[adamtagscherer]

Hey!

This discussion is about how we could distribute some of our API objects for other projects and adopters easily.

Problem statement

We have the following objects extension, locator, signature, skill and agent_record that is present in multiple repositories because they are needed to develop features. The problem is that there is a schema drift between these objects because there is not a single source of truth from where everyone would pull them. In this discussion I'd like to fix this issue.

Requirements

• It needs to be a single source of truth so there will be no redefinition of the objects and the schema drifts will be mitigated.
• Somehow distribute the proto files so that other projects can build their services from them. The biggest problem that needs to be solved here is that the proto file has to be present on your filesystem physically otherwise the protoc compilation won't work.
• Generate language bindings at least for Go and Python. Other important languages can be JavaScript, TypeScript and Rust for now.

Possibilities for distribution:

• Gather our proto files in a GitHub repo and include it as a submodule to other repos.
• Manually copy paste the proto files to other repos.
• Use a registry.

Possible solutions

In this section I'd like to bring up some possible solutions for this. The main idea is that these objects should be stored in OASF repository because they belong here.

Generate with a chain of software

I created a PR with it that I've closed as of now: #168. In this PR's description I explained why I think this is a bad solution.
TLDR: JSON Schema 07 is for validating other objects against it, not for generating other formats from it. We can't express everything in a JSON schema file that we need in a proto file. The semantics is lost in the transformation chain. Very error prone, and a lot of extra unnecessary complexity.
Not to talk about the fact that it doesn't solve the distribution of the proto files and language bindings.

Write our own generator

It can be a good idea that we write a generator that creates proto files from our JSON files that we store on the server. Although it would need a lot of work, it might be able to serve out needs but again it doesn't solve the distribution problem. In the future if we'll have 100 API objects that are changing frequently it might worth it to write a generator, but for now it's overkill.

Manage the proto files manually and distribute them

There is a drawback to this approach is that we need to manually manage the parity between the JSON files and the proto files. I think for now it's not a problem because they don't change that frequently and there is 5 of them.
The pros for this solution is that we can use proto files as our starting schema and generate everything that we need, as usually people do. In the directory repository we already use bufbuild as a tool that generates the messages, services and language bindings for the proto files.

buf has a registry called BSR that is built in their tooling that we can use to distribute our proto files. We upload the proto files there and it will generate the language bindings automatically, that we can download with the language's respective package manager. The proto files can be used in another project's proto builds if buf CLI is used.

I investigated some other registry options but none of them was as good as BSR is for us.

Conclusion

I think we should start using BSR as a proto file registry and start distributing our proto files and lanuage bindings with it. We need an org where we can create the repos. @sriaradhyula any chance you can help us provision an agntcy org that we can start using?