Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How about a monorepo 😄 #1044

Open
derberg opened this issue Oct 4, 2023 · 37 comments · Fixed by #1213
Open

How about a monorepo 😄 #1044

derberg opened this issue Oct 4, 2023 · 37 comments · Fixed by #1213
Assignees
Labels
bounty AsyncAPI Bounty program related label enhancement New feature or request

Comments

@derberg
Copy link
Member

derberg commented Oct 4, 2023

Current state

This granularity is technically possible so templated development is flexible.
But the fact that we make it possible for others, to easily create private, independent and use case specific templates doesn't mean we should have this level of granularity in asyncapi GH org

Proposal - discussion

How about we turn generator into a monorepo?

  • generator-react-sdk - not a separate npm package, just natively part of generator
  • generator-filters and generator-hooks also natively a part of generator (we still want to enable people to provide theirs independently)
  • yes, templates as part of generator repo, but in case of templates, I guess we would need to publish separate packages 🤔 although I'm not 100% sure we need to

Why

  • having to use generator-react-sdk as separate package is kinda confusing and in general my impression is that motivation for having it as a separate package was that then it can be written in TS
  • over last 2 year we had zero improvements in hooks and filters, they are hidden small repos and template developers basically work on their own filters and hooks in their template repos.
  • it would not be obligatory to move @asyncapi/*-template to the monorepo. But I think many would, as having it in our repo, with multiple maintainers/champions (yeah, we would have champion like setup like in modelina) would make it easier and quicker to reuse solutions from other templates, filters, components and other stuff. Under one repo it would be easier to focus or reusability and good standards (like always using modelina for models generation)

We would still be flexible, independent templates would be possible, hooks and filters too.

Having all under one umbrella would make generator a more vibrant project, more actively developed and visible.

wdyt?

any disadvantages that I do not see?

@derberg derberg added the enhancement New feature or request label Oct 4, 2023
@ayushnau
Copy link
Contributor

@derberg i saw we need to implement the monorepo for this i would like to work on this. Please let know how should i go about this.

@derberg
Copy link
Member Author

derberg commented Oct 25, 2023

@ayushnau for now it is proposal and a discussion

@ayushnau
Copy link
Contributor

ok @derberg

@derberg
Copy link
Member Author

derberg commented Nov 30, 2023

@jonaslagoni @magicmatatjahu wdyt

@jonaslagoni
Copy link
Member

Sounds good to me 🤙

@Aryann15
Copy link

Hey! would like to work on this issue, can i get it assigned? @derberg
Thanks!

@derberg
Copy link
Member Author

derberg commented Mar 11, 2024

@ayushnau @Aryann15 are you folks still interested with helping here? did you have experience with introducing monorepo in the past?

first step would be project refactor and migration of first package from https://github.com/asyncapi/generator-filters

if you are interested working on it, please first share your plan before opening a PR

@ayushnau
Copy link
Contributor

Hi I would love to work on this issue.

@derberg
Copy link
Member Author

derberg commented Mar 14, 2024

added to bounty:

  • generator will be now a monorepo,
  • main readme should still be focused on generator and there will just need to be a section where we explain that there are other projects included - short sentence intro and link to dedicated readme
  • we need short development guide that will explain the setup of the repo
  • filters are the first things we move
  • filters repo will be archived
  • we still release @asyncapi/generator and we still release @asyncapi/filters, just from one repo. Release can have the same number, we do not need separate versioning

any other questions?

@AyushNautiyalDeveloper
Copy link

AyushNautiyalDeveloper commented Mar 14, 2024

Questions:

  1. Is turborepo final for migrating generator repos to it.
  2. So Generally we put all the functions and miscellaneous stuff like common scripts in. packages. (package containing all the shared stuff of all the apps.) which can be used so will we take some of shared stuff of the repos and put that into packages.
  3. And do we want to put all the ci/cd flows in their own repos or will unite them into one.
  4. We want to maintain git history of each app ?

@AyushNautiyalDeveloper
Copy link

@derberg

@asyncapi-bot asyncapi-bot added the bounty AsyncAPI Bounty program related label label Mar 18, 2024
@aeworxet
Copy link

Bounty Issue's service comment

Text labels: bounty/2024-Q2, bounty/advanced, bounty/coding
First assignment to third-party contributors: 2024-03-22 00:00:00 UTC+12:00
End Of Life: 2024-08-31 23:59:59 UTC-12:00

@asyncapi/bounty_team

@derberg
Copy link
Member Author

derberg commented Mar 20, 2024

Is turborepo final for migrating generator repos to it.

tbh I don't have any preference on what tool we choose. If we do npm workspaces, lerna or turborepo. Afaik you have experience with monorepo, so if you think turborepo is good investment, lets do it.

So Generally we put all the functions and miscellaneous stuff like common scripts in. packages. (package containing all the shared stuff of all the apps.) which can be used so will we take some of shared stuff of the repos and put that into packages.

we still need to release @asyncapi/generator and we will still have to release @asyncapi/filters. If there will be some common code then definitely we should have shared commons.

Filters technically are a feature only for generator templates that run on Nunjucks engine. So template developer uses it in the way that they add filters package to dependencies and then adds module name to filters in the config. So in theory, your job is a simple as move filters in, make sure they are released and that is it. But would be super nice if you can come up with an idea how to make sure @asyncapi/filters are included in asyncapi/generator by default, so nobody have to enable them by configuration.

And do we want to put all the ci/cd flows in their own repos or will unite them into one.

afaik workflows from https://github.com/asyncapi/generator-filters can be dropped as they are all anyway pulled from .github repo (so are the same that also in generator) repo. You just need to make sure that whatever workflows we have here, for release or bump - will still work

We want to maintain git history of each app

Not a strong requirement - just lemme know what options we have.
The plan is that basically this https://github.com/asyncapi/generator-filters will be archived and moved to https://github.com/asyncapi-archived-repos so history will still be there. But if there is some cool git magic available so we can preserve history of filters in generator, lets see. I admit I'm not such a git expert


Please also work closely with people from asyncapi/parser-js#963 as might be that best would be to have the same solution in both

@derberg
Copy link
Member Author

derberg commented Mar 20, 2024

@AyushNautiyalDeveloper so, still interested?

@ayushnau
Copy link
Contributor

yeah Sure @derberg once you answer those question I will add short development guide. as required.

@derberg
Copy link
Member Author

derberg commented Mar 20, 2024

I think we have twins here 😄

@derberg
Copy link
Member Author

derberg commented Mar 20, 2024

btw, my answers are above

@derberg
Copy link
Member Author

derberg commented Mar 21, 2024

@ayushnau ok, please go ahead, the issue is yours

@ayushnau
Copy link
Contributor

ok thanks will start working on this then.

@aeworxet
Copy link

Bounty Issue's Timeline

Complexity Level Assignment date (by GitHub) Start date (by BP rules) End date (by BP rules) Draft PR submission Final PR submission Final PR merge
Advanced 2024-03-21 2024-04-01 2024-05-24 2024-04-19 2024-05-10 2024-05-24
Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.
Keep in mind the responsibility for violations of the Timeline.

@ayushnau
Copy link
Contributor

ayushnau commented Mar 24, 2024

Comparison of Repository Management Tools

  1. So I have Decided to go with Turborepo. Here is a comparision for different options we had. though Nx was also good.
Feature TurboRepo Lerna Yarn Workspaces Nx
Performance High performance with caching and parallelization Decent performance, may slow down with large repos Decent performance, can suffer with large repos High performance with advanced caching and builders
Dependency Management Integrated dependency resolution Basic dependency management Integrated with Yarn, follows Yarn's resolution Integrated dependency graph for efficient builds
Versioning Integrated version management Incremental versioning with fixed or independent Integrated with Yarn, follows SemVer Semantic versioning with versioning strategies
Language Support Primarily JavaScript and TypeScript Primarily JavaScript and TypeScript JavaScript, TypeScript, and more JavaScript, TypeScript, and more
Community Support Growing community support Large and established community Strong community support Strong community support
Ease of Setup Straightforward setup and configuration Moderate setup with configuration options Integrated with Yarn, relatively easy to set up Straightforward setup and integration
Integration with CI/CD Supports various CI/CD pipelines Compatible with common CI/CD tools Compatible with common CI/CD tools Integrated CI/CD support and workflows

Adding a single repo will look like this

  1. Install and do basic setup of TurboRepo. Note: We will be using npm as a package manager as we have used npm in every package.

  2. Add the @asyncapi/filter code to the apps directory of the turborepo.

  3. Add the configuration for the @asyncapi/filter to the Turbo.json file in the root directory.

  4. Add all the required scripts in the package.json file for the @asyncapi/filter. (To run apps package.json scripts from root)

Question:

  1. What should be the name of the repo? It can't be "Generator" as we are going to use it for its own repo.

@ayushnau
Copy link
Contributor

ayushnau commented Mar 24, 2024

https://github.com/ayushnau/generator

Added the @asyncapi/generator-filters repo

And For git history It is possible to maintain the history of every repo merged into one. As i have done with this current repo. @derberg

@derberg
Copy link
Member Author

derberg commented Mar 26, 2024

can you specify what is the goal of https://github.com/ayushnau/generator ? just for testing?
The changes that you introduce should be done in a typical way through a fork

What should be the name of the repo? It can't be "Generator" as we are going to use it for its own repo.

we keep the name for now, is that a blocker? all additional stuff we will move to generator are considered plugins/extensions - apps is probably a good name too

So I have Decided to go with Turborepo. Here is a comparision for different options we had

thanks for great comparizon. Turborepo indeed looks interesting. Also notices even nodejs started using it https://github.com/nodejs/nodejs.org

@ayushnau
Copy link
Contributor

  1. The purpose of that repository is solely for testing and demonstration purposes.Other apps will also be included inside the apps directory (including the generator itself.)
  2. No, it is not a blocker, just asking for clarity.
  3. 👍

@derberg
Copy link
Member Author

derberg commented Mar 27, 2024

awesome, feel free to DM me whenever you need help or there is something new I should look at - I'm a bit under water with notifications, so DM is working very well for me atm

@ayushnau
Copy link
Contributor

#1155

@derberg
Copy link
Member Author

derberg commented Apr 11, 2024

@ayushnau I noticed you closed the PR, all good? what's the plan?

@ayushnau
Copy link
Contributor

@derberg I am getting an issue with running test command.
image
This is coming due to the test trying to access the nims from node_modules. but turborepo have installed it in base node_modules. where all the internal and shared dependencies packages are installed.

Solution:
I think what we can do is just change the directory

 '^nimma/legacy$': '<rootDir>/node_modules/nimma/dist/legacy/cjs/index.js',
    '^nimma/(.*)': '<rootDir>/node_modules/nimma/dist/cjs/$1', 

to correct directory.

@derberg
Copy link
Member Author

derberg commented Apr 11, 2024

yup, go ahead 😀

@aeworxet
Copy link

@ayushnau, please provide an update to the PR.

@ayushnau
Copy link
Contributor

ayushnau commented May 1, 2024

@aeworxet adding

@aeworxet
Copy link

@ayushnau, please provide an update to the PR.

@derberg
Copy link
Member Author

derberg commented Jun 19, 2024

fyi, the implementation of the issue is delayed a bit as I was out for a couple of weeks for conference and holidays - so could not review, and there was noone else who could

@derberg
Copy link
Member Author

derberg commented Jun 20, 2024

not yet completed, we need new release pipeline and move into project one more app

@ayushnau
Copy link
Contributor

ayushnau commented Aug 1, 2024

@derberg can we mark this issue completed

@derberg
Copy link
Member Author

derberg commented Aug 1, 2024

@aeworxet bounty scope completed. I'm not closing the issue as there is still a need to move hooks into generator - but that was not part of bounty scope

@aeworxet
Copy link

aeworxet commented Aug 1, 2024

After a delay that was beyond the control of the Bounty Program Participant
#1044 (comment)

Bounty Issue Completed 🎉

@ayushnau, please go to the AsyncAPI page on Open Collective and submit an invoice for USD 400.00 with the expense title Bounty generator#1044, tag bounty, and full URL of this Bounty Issue in the description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bounty AsyncAPI Bounty program related label enhancement New feature or request
Projects
Status: Completed
7 participants