Skip to content

shore-core.md

Latest commit

 

History

History
154 lines (93 loc) · 12.9 KB

shore-core.md

File metadata and controls

154 lines (93 loc) · 12.9 KB

Shore Core Architecture

Preface

This document set's out to illustrate the shore-core component of the shore framework.

Shore Core

shore-core is a term that describes the combination of a Renderer & Backend.

There is no one single entity in the system which describes shore-core.

Currently the established boundary is pkg/commands pkg/backend, pkg/renderer may be regarded as shore-core excluding the wrapping cobra commands.

pkg/commands specifically holds implementations that can (should?) be decoupled and reused by other systems.

The term is required to make sure boundaries aren't breached during development.

Specifically, shore-cli, pkg/renderer/jsonnet, pkg/backend/spinnaker should not be regarded as core, nor should be part of the main "core".

Context

As was described earlier - shore-core is an conceptual layer with no single entity.

It's the intention to define a clear shore-core in the future.

To deliver on the listed requirements shore-core is designed to be extensible.

The Core should:

  1. Provide interfaces (Render Interface, Backend Interface, RemoteTesting Interface) for concrete implementations.
  2. Govern which "plugin" is used and when (I.E. |> shore render should only use a Renderer, |> shore save Should use both Renderer & Backend)

To achieve the extensibility requirement, some of the core requirements are delegated to either Render or Backend.

Examples of delegation:

  1. Renderer implements the library loading if the feature is missing (I.E. Jsonnet, HCL)
  2. Backend implements remote-testing.

TODO: After the cli experience has been validated, core will get more concrete structs & interfaces.

Plugin System

Interfaces and boundaries have been setup to allow extending shore-core with custom functionality.

Due to the complexity of plugin systems, the design & implementation will be left to a later date. By initially focusing on the Jsonnet renderer and Spinnaker backend, the plugin system is not needed immediately. When expansion beyond Jsonnet and Spinnaker is required, we will we begin to design the plugin system.

High Level Diagram

Diagram Of Shore Core

Shore Core Orchestrates:

  • Renderer
  • Backend

Renderer

A Renderer is an abstraction layer which wraps a templating language in order to process the project inputs.

A templating laguage engine takes in code, processes the data and outputs the rendered format (reference implementation uses Jsonnet and outputs JSON) for further processing by another system.

Based on the Renderer implementation, the templating laguage can be swapped to support other templating languages (I.E. HCL, Jsonnet, CUElang, Dhall).

Currently only Jsonnet is supported.

For a concrete implementation shore-cli will coordinate between the project & a renderer to pass in additional files that aren't supported by default. Mainly render.yaml and shared libraries if they are not supported natively by the templating language.

Shared libraries are snippets of code hosted in Github that can be pulled to the local machine. The code snippets are used to encapsulate logic in a different location that isn't the main code base.

Renderer Interface

Render()

Backend

The backend is an abstraction layer of a generic pipeline backend such as Spinnaker, Tekton, ArgoCD, etc..

A backend should implement the high level concepts of working with pipelines. These concepts are encapsulated by the interface definition.

The Pipeline in a context of a backend is an automated stored procedure that could be invoked that will carry out process on a remote server (in the Cloud, on someone else's computer!).

Examples of pipelines are: Spinnaker Pipelines, Jenkins Jobs, Tekton Jobs, AWS StepFunctions.

Currently only Spinnaker is listed as a supported backend.

Backend Interface

SavePipeline()
ExecutePipeline()
WaitForPipelineToFinish()
TestPipeline()

Implementation specific details

Golang

Golang - Context

The core is developed using the golang language.

Golang - Pros

  1. Self contained binary/executable
  2. Cross-compiling supported Out Of The Box.
  3. Configuration languages are either easily integrated or already built using Golang.
  4. Interoperability via CGO extension.
  5. Builtin Plugin system.
  6. Easy learning curve for developers already familiar with different languages.
  7. Many libraries that make life easier (e.g. cobra, viper, spin-cli, jsonnet).
  8. DevOps friendly (prime intended user of the solution)
  9. Real Decoupled Interfaces!

Golang - Cons

  1. Possibly Large binaries.
  2. Tight Coupling to Golang "mentality".
  3. Doesn't have the "batteries included" that "Python/Ruby" and other scripting languages come with.

Other Considered Solutions

  1. Python
  2. Bash|Shell|sh

Python Pros:

  1. Readily available on most developer centric machines/hosts.
  2. Easy onboarding.
  3. Large variety of libraries and abstractions.
  4. Mature, stable & documented testing frameworks (pytest, hypothesis)

Python Cons:

  1. Requires Python as a dependency on machines without a real need (I.E. alpine).
  2. Makes future integrations harder to implement. Python-C-Bindings are hard to develop, maintain.
  3. Requires maintaining more API interfaces to other languages (rendering engines) that do not support Python Out Of The Box.

Bash Pros:

  1. Readily available on most (all!) hosts on the planet.
  2. No need to setup a dev environment.
  3. Integrations are as easy as setting up a new executable (no apparent language barriers)

Bash|Shell|sh Cons:

  1. Harder to maintain when code-bases get larger.
  2. May cause installation overhead.
  3. Requires testing many different shell flavors (zsh, bash, shell, etc...)
  4. Doesn't support Windows OOTB.