This covers codestream-server development configured as a mono-repo (all services running in one sandbox) using the devtools framework. There are other ways to do this, such as the open development instructions in the main README, however you won't have access to the features and conveniences applicable to the CodeStream network. It's worth noting that our CI/CD pipelines do not use the mono-repo configuration; they build and install component sandboxes separately (api, broadcaster, ...).
These quick starts get your mongo and codestream server sandbboxes up and running in several configurations. It assumes you have no customizations of your configuration files (or haven't installed them yet) and already have the dev_tools framework installed. They also assume you're ok using all the default settings and naming conventions.
A few things to know:
-
We use a mono-repo configuration for development where each of the component services (api, mailin, mailout, ...) run inside a single sandbox.
-
Development CodeSteam configurations are prepared and distributed by ops. This includes all keys, tokens and secrets you'll need to develop integrations. The out-of-the-box default configuration name is
codestream-cloud-custom-1. -
For sandboxes & playgrounds to co-exist on the same system in development, you'll need to use different run-time environments. All you need to know for now is that you'll see 3 of them.
localis used for a single, standalone backend service.local1is for one half of a dual-environment backend service.local2is for the other half of a dual-environment backend service.
This installs mongo, csbe (codestream backend) and csfe (codestream frontend) sandboxes and corresponding playgrounds. The csfe sandbox is beyond the scope of this document and not really needed here, but you get it for free with this uber-easy installer.
dt-dev-setup --codestream
If that worked you have installed three new sandboxes and playgrounds. The csbe sandbox is configured to use the local run-time environment. Continue with the Quick Starts section for loading playgrounds and starting services.
In this configuration, you're creating two new playgrounds. As a prerequisite, you need to complete the uber-fast stand-alone backend installation above. This installs two additional sandboxes & playgrounds; csbe-local1 and csbe-local2.
dt-dev-setup --codestream-dual
The two additional playgrounds can co-exist on the same system with the stand-alone playground, but each requires its own shell. Continue with the Quick Starts section for loading playgrounds and starting services.
Now that you've got your server configurations, secrets and sandboxes installed, let's put them to use.
Note that these instructions have you run your backend services in the background. You probably won't do that during development but the aim of this document is just to get you started.
Load your csbe playground (which consists of your mongo and csbe sandboxes) and start the services.
dlp csbe # dlp is an alias for dt-load-playground
mdb-service start # start mongo
csbe-service start # start the component services (api, mailout, mailin, ...)
Properties to be aware of:
| property | desc |
|---|---|
| CSSVC_ENV | The run-time environments local* are reserved for development. The CodeStream development platform knows about this environment and is able to provide additional resources based on it. For example, interactive callbacks from Slack can be routed to your sandbox via the CodeStream proxies. |
| CSSVC_CFG_FILE | When you load your sandbox, this variable will be defined to point to the appropriate codestream services configuration file. |
| api URL | https://localhost.codestream.us:12079 for development on your personal computerhttps://<my-server>.codestream.us:12079 for development on your own VM |
| mongo URL | mongodb://localhost/codestream for development on your personal computermongodb://<my-server>.codestream.us/codestream for development on your own VM |
This is pretty much identical to the previous section except your playgrounds are csbe-local1 and csbe-local2 respectively.
In a new shell:
dlp csbe-local1
In a different new shell:
dlp csbe-local2
The commands are the same in all sandboxes. The different properties are:
| property | desc |
|---|---|
| CSSVC_ENV=local | The run-time environments local* are reserved for development. The CodeStream development platform knows about this environment and is able to provide effective resources based on it. |
| CSSVC_CFG_FILE | When you load your sandbox, this variable will be defined to point to the appropriate codestream services configuration file. |
| api URL (local1) | https://localhost.codestream.us:13079 for development on your personal computerhttps://<my-server>.codestream.us:13079 for development on your own VM |
| api URL (local2) | https://localhost.codestream.us:13080 for development on your personal computerhttps://<my-server>.codestream.us:13080 for development on your own VM |
| mongo URL | mongodb://localhost/codestream-local1 for csbe-local1mongodb://localhost/codestream-local1 for csbe-local2 |
Sandboxes have well-defined interfaces. Once loaded, you'll find many tasks are similar in how to approach them.
To control the mongo service, use the mdb-service command.
mdb-service { start | stop | status }
To control the csbe services, use the csbe-service command (though you probably won't do this much).
csbe-service { start | stop | status }
The csbe and csbe-localN sandboxes are based on a mono-repo (codestream-server). Inside them, the individual component sandboxes are still there, though they don't appear as their own sandbox entity (kind of confusing for a newbie - don't worry about it).
Just think of csbe and a superset of a number of other sandboxes. For example, you have commands and services for these components. In our CI/CD pipelines and in production environments, these exist as their own sandboxes but in the csbe sandbox, you can still run these commands.
- api -
cs_api-* - mailout -
cs_outbound-email-* - mailin -
cs_mailin-*
In addition to the <prefix>-service command, all sandboxes support these
standard commands as well (plus many others).
<prefix>-help- quick list of sandbox commands<prefix>-vars- show environment variables in the sandbox
Some integrations require callbacks from 3rd party vendor sites (eg. Slack). We have a public-facing proxy service which has a mechanism for proxying pre-determined requests from these 3rd party services to your primary development VM. You will need this, for example, if you're doing development work on the Slack integration using their interactive components feature. The mechanism uses DNS so each developer can designate one of their development VMs at a time to be a primary, which would receive these proxy requets.
Use the dt-dev-set-primary-vm script to set your primary. You can change it at
any time. This does not work for development on your local computer; only on
network VMs.
The sandbox configuration dictates how the shell environment is set each time you load your codestream-server sandbox. The CodeStream configuration is loaded by CodeStream services upon launch and controls how the services behave.
codestream-services incorporates a number of services to provide all the functionality needed for the clients. Which services to run and their configurations differ based on cloud v. on-prem, and those can be further broken down into a number of installation or product types. Accordingly, the mono-repo offers two configurations; default.sh for cloud development, and the aptly named onprem-development.sh.
The two sandbox configurations use these services:
| Sandbox Config | Services |
|---|---|
| default.sh (cloud) | mongodb, aws sqs, api, pubnub, mailin, mailout |
| local1.sh (dual-env cloud) | mongodb, aws sqs, api, pubnub, mailin, mailout |
| local2.sh (dual-env cloud) | mongodb, aws sqs, api, pubnub, mailin, mailout |
| onprem-development.sh | mongodb, rabbitmq, api, broadcaster, mailout, admin |
Note that MongoDB, RabbitMQ & AWS SQS are outside the scope of a codestream-server sandbox. We do have a mongo sandbox that can be used together with the codestream-server sandbox in a playground. AWS SQS requires an AWS IAM access key or the codestream-server sandbox to run on a managed EC2 development instance.
The CodeStream services (api, outbound email, inbound email, etc...) share a common configuration file. The config file uses a versioned schema to adapt to changes over time in the code base.
A detailed explanation of the CodeStream server configuration file management (not its content) is here but the next section covers how to customize it to your liking.
Your codestream configuration files are located in ~/.codestream/config/.
-
The
dt-dev-setupinstallation script will set you up with a configuration called codestream-cloud-custom-1 that's got an overrides file, custom-overides-for-custom-1.json, which contains an empty json object. -
Each time you fetch updated secrets using the
dt-dev-update-secretscommand, a hook will create your custom config (codestream-cloud-custom-1) by taking the distributed config codestream-cloud and updating it with your custom-overides-for-custom-1.json data (python obj.update() or JS obj.assign()). -
Add any overrides data you want in your config to custom-overrides-for-custom-1.json file. There's an example file for reference in that directory as well (*.example).
Happy coding!.