Skip to content

Commit c3bed8d

Browse files
committed
Move network here
1 parent 70e9e6f commit c3bed8d

27 files changed

+893
-0
lines changed

network/README.md

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
# Network
2+
3+
The micro network is an **open network for micro services**.
4+
5+
## Overview
6+
7+
The power of collaborative development has mostly been restricted to trusted environments within organisations.
8+
These platforms unlock incredible productivity and compounding value with every new service added. They provide
9+
an always-on runtime and known developer workflow for engineers to collaborate on. This has largely been
10+
difficult to achieve outside of organisations.
11+
12+
The micro network looks to solve this problem using a shared global network for micro services.
13+
14+
## Contents
15+
16+
- [Architecture](#architecture)
17+
* [Features](#features)
18+
* [Services](#services)
19+
* [Identity](#identity)
20+
* [Evolution](#evolution)
21+
* [Runtime](#runtime)
22+
* [Governance](#governance)
23+
* [Network](#network)
24+
* [Collaboration](#collaboration)
25+
* [Value Exchange](#value-exchange)
26+
- [Economics](#economics)
27+
- [Glossary](#glossary-of-terms)
28+
29+
## Architecture
30+
31+
The network is a globally distributed network based on existing open source technology developed by micro and others.
32+
It provides the core requirements for collaborative service development at scale. This includes a common
33+
communication protocol, service discovery system, secure governance, service runtime and resource exchange model.
34+
35+
### High Level
36+
37+
At the high level we're using the open source micro toolkit as the foundation for an overlay network on the internet and a platform for microservices.
38+
Each user runs a local node which provides access to the network, a runtime for running services, a local http api gateway and web browser.
39+
40+
![Architecture](images/arch.jpg)
41+
42+
In future this design will include automatic data synchronization and backup for long term storage.
43+
44+
### Features
45+
46+
The network features a p2p compute runtime with a high bandwidth backbone. The network itself provides discovery, governance and a runtime.
47+
The design is such that whatever resource you put in, you get back distributed across the network. It's global and shared by default. You can
48+
buy more resource using tokens but you'll also get tokens for any requests made to your services.
49+
50+
![Architecture](images/arch2.jpg)
51+
52+
The network is evolutionary and our goal is to reach 1B nodes with over 10x more services running on the network itself.
53+
54+
### Services
55+
56+
The network provides a micro service runtime. Each service can be used as a building block for the next.
57+
This layering allows the creation of unique services on a foundation of hundreds of others. This is the microservice
58+
architecture pattern used by Google, Netflix, Uber and others which we're bringing to the masses.
59+
60+
![Architecture](images/arch3.png)
61+
62+
Developers contribute services to the network and build on each others work.
63+
64+
### Identity
65+
66+
Most networks being bootstrapped require some form of identity system. Many of these start with base level asymmetric cryptography
67+
using public/private key pairs. While we'll look to use these we believe this system is not enough. Anyone can participate in the
68+
network and anyone can generate a public/private key pair. Bad actors will likely cycle these and DDOS the network.
69+
70+
What we propose is a two layer identity system, with layer 1 as the public/private key pair (potentially using SPIFFE) and layer 2
71+
as a reputation based scoring system which over time solidifies your reputation and trusted identity in the network. The longer
72+
you exist in the network and participate the more likely it is we can identify you as a trusted party. New participants will be
73+
provided some base level access but must either pay to participate or stay on the network long enough to get further access.
74+
75+
The identity system will likely be as follows
76+
77+
- Layer 0 - the node connected to the network
78+
- Layer 1 - the app providing a service to the network
79+
- Layer 2 - the reputation scoring based on node + service
80+
81+
### Evolution
82+
83+
No system in nature is static and so we should not expect any technical infrastructure or software to remain static. We believe
84+
evolutionary software is far superior to anything else and this in principle needs to be promoted by the underlying runtime
85+
on which it runs. The micro network will be an evolutionary system which requires all services run on it to also behave in such
86+
a manner.
87+
88+
Software will be downloaded from its public source form. It will then be built and run, all within the same location. Updates to
89+
the source will result in periodic generation of new versions of a running service. Previous instantiations will remain for
90+
their lifetime or until they die for any unexpectected reason. This is evolution and we believe evolution is the most powerful
91+
method for software delivery.
92+
93+
### Runtime
94+
95+
While our primary focus is to provide a globally distributed network for micro services this can't be fully realised without a
96+
method of deploying software. It's important that in the context of creating evolutionary systems we provide a runtime which
97+
supports this method of development. The micro runtime is a process management system tightly coupled to the network itself
98+
with the ability to run applications from source code.
99+
100+
Developers no longer need be concerned with packaging and shipping their code. If it exists in a public repository it can be
101+
deployed to the micro network. We will primarily look to support Git via GitHub as the source of truth. This allows anyone
102+
to run a copy of whatever already exists in the network but it also creates a level of trust around completely **Open Source Services**.
103+
104+
Beyond this, our expectation is to standardise on wasm (web assembly) which lets us provide a universal runtime for any
105+
language which supports wasm compilation. We'll likely provide rudimentary language detection early on to auto-compile wasm
106+
binaries. Eliminating another intermediate step for developers to ship software.
107+
108+
### Governance
109+
110+
While we can provide an evolutionary network and a runtime which supports such a system, it is very unlikely that such a system
111+
can be sustained standalone. Platforms are managed by operators and every system in nature has some form of governance. The
112+
micro network will be no different.
113+
114+
Having learned from years of managing platforms it is abundantly clear that we do not want human involvement to manage something
115+
on the order of 1B nodes. With likely 10x more services. Even if the task of management is distributed to node owners (which
116+
defeats the purpose of our goals) it would be impossible to perform such coordination at scale. Because of this the system
117+
we develop will have to be self governing.
118+
119+
The micro network will include self governance driven through automation. No humans will be involved in commanding the placement
120+
of services or have the ability to shut them down. The governance mechanism for all services and the network itself will be in code,
121+
it will be a service running on the network itself. It will also be an open source service anyone can see the code for and
122+
contribute to.
123+
124+
Like any other service, this system will be evolutionary.
125+
126+
### Network
127+
128+
In the case of micro, the "network" is an overloaded term both referring to the underlying system made up of nodes or peers which
129+
provide compute and also the overlaying communication system used for services to discover and communicate with each other.
130+
131+
In this case we'll speak to the service layer network. The micro network provided to developers consists of four pieces;
132+
registry, transport, broker and client/server. These components can be found in [Go Micro](https://github.com/micro/go-micro)
133+
for more info.
134+
135+
The registry provides a service discovery mechanism for applications in the network. Think of this as DNS but in a way that's
136+
self describing and built for transient systems. Our assumption is that developers should not have to leave the network
137+
to understand what a service is built for and that any number of prototypes with high percentage failure rates will exist on the
138+
network itself. Because of this we need a system that describes services in their entirety and expiration of dead nodes to
139+
prevent various failure modes.
140+
141+
The transport provides a common synchronous communication system. Ideally this is abstracted away so the developer does not
142+
have to worry about what the protocol is. In our mind most frameworks and libraries provide rich http support with prominent
143+
ecosystems. We want to enable this while providing a highly efficient binary protocol beneath the covers.
144+
145+
The broker is an asynchronous communication system. It allows for publishing messages to an unknown number of interested parties.
146+
In some cases there may be no subscribers in which case we need to account for future interest and potential short term storage
147+
and access. It is critical that event based programming be a core tenant of the network. This likely also promotes a functional
148+
programming model which will be native to the network.
149+
150+
The client/server abstracts away the registry, transport and broker so that from a development standpoint all that remains is a
151+
focus on the domain itself. Ideally inter-service communication should be thought more as high level domain driven design and
152+
later on flow based programming.
153+
154+
The network encapsulates low level details of distributed system communication and allows the developer to focus on whats important.
155+
156+
### Collaboration
157+
158+
It is not enough to provide an environment for running applications. Many platforms already exist for this function. Individuals
159+
can use cloud hosting to do this. Networks are being built to run decentralized applications (dApps). This is all good and well
160+
but at the base level is just compute, access to resource.
161+
162+
What's more useful is a focus on **collaboration**. Organizations create a single cloud hosting account as a trusted environment
163+
for their engineers and teams to build software together which then becomes a unified product offering. They do not have to worry
164+
about whether or not they can trust other teams or engineers using it. They do not have to worry about services disappearing or
165+
even building everything from the ground up. Teams working independently produce services which can be accessed by others teams. This
166+
becomes the basis for new applications, the foundation which they no longer have to rebuild. The velocity of their development
167+
increases with every new service added. New products can be built at rapid pace, and the compounding value of this in the long
168+
term has been demonstrated by Amazon, Facebook, Google, Uber, and many more.
169+
170+
Collaboration is the key. This must be our focus and is our focus. Because we focus on microservice development. We provide
171+
the building blocks in a network and push forward the idea of collaborative software development so that we as individuals
172+
can collectively build towards what does not exist outside of organizations. This is an opportunity to build something at
173+
the scale of Google and others in a coordinated effort without any central intermediary or the need to work at a company.
174+
175+
Collaboration will be our central focus, not compute, not storage, nor singular applications, but instead multi application
176+
services built collectively by the network.
177+
178+
### Value Exchange
179+
180+
What do we mean by value exchange? It is not enough to provide an environment for collaborative development. We need some way of
181+
rewarding those who are participants in the network and provide resources or services to the network itself.
182+
183+
Value exchange is basically the idea of exchanging some *thing* in return for some *value*. In our case we will exchange **resources**
184+
for the use of the network **runtime** or **services**. Resources can either be compute offered to the network via the runtime,
185+
the ability to actually run services, or some form of monetary payment. Resources will be given a quantified value in the network
186+
and abstracted away into a native exchange method using an api **token**.
187+
188+
Api tokens will behave as a utility token, they will have a lifetime and it will be possible to generate them if a user participates
189+
in the network or makes monetary payment to acquire them. It will be possible to refresh expired tokens.
190+
191+
Our expectation is that the token becomes the native exchange method for resource in the micro network but the value of a token
192+
will be self adjusting based on the size of the network. The value of the network is based on the number of participants, resources
193+
and services, not the token itself.
194+
195+
## Economics
196+
197+
Building such an architecture is good in theory but in practice requires network effects to drive its use, growth and ability to thrive.
198+
It is not enough to create an open network and ask developers to run peers in the network or contribute, there has to be some economic
199+
incentive to be part of it.
200+
201+
This is where [Token Economics](https://en.wikipedia.org/wiki/Token_economy) plays a role. Those contributing resources to the scale
202+
of the network and/or services on the network should be rewarded accordingly. Our goal is to build a native tokenized transaction
203+
system into the network to provide this resource usage model via exchange of tokens for services.
204+
205+
Here's a good explanation of [Token Economics](https://www.investinblockchain.com/token-economics/).
206+
207+
## Glossary of Terms
208+
209+
- **Network** - is the entire system for communication and coordination
210+
- **Backbone** - is the core public high bandwidth main network itself
211+
- **Super Node** - is a host which runs the *backbone* of the network
212+
- **Peer Node** - is a host which runs *services* in the network
213+
- **Service** - is a process which runs in the network and provides some utility
214+
- **Token** - is an economic incentive used in exchange for the use of a service
215+
216+
## Release
217+
218+
Coming soon
219+
220+
## Contribute
221+
222+
Join the [slack](https://micro.mu/slack/) or [reddit](https://www.reddit.com/r/micronet/) community to discuss

network/config/kubernetes/README.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Kubernetes Deployment
2+
3+
## Dependencies
4+
5+
- Kubectl
6+
- Kubectx
7+
- Helm
8+
- Consul
9+
- NATS
10+
11+
## Steps
12+
13+
1. Spin up managed k8s on DO, GCP, AWS
14+
15+
2. Setup local env
16+
- Install Kubectl https://kubernetes.io/docs/tasks/tools/install-kubectl/
17+
- Install Kubectx https://github.com/ahmetb/kubectx
18+
- Install Helm https://github.com/helm/helm
19+
* See https://github.com/helm/helm/blob/master/docs/rbac.md
20+
21+
3. Install Consul
22+
- https://www.consul.io/docs/platform/k8s/run.html
23+
- kubectl port-forward consul-consul-server-0 8500:8500
24+
25+
4. Install Micro
26+
- kubectl apply -f ../kubernetes
27+
- Create external load balancers https://www.digitalocean.com/docs/kubernetes/how-to/add-load-balancers/
28+
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
namespace: default
5+
name: greeting-api
6+
spec:
7+
replicas: 1
8+
selector:
9+
matchLabels:
10+
name: greeting-api
11+
template:
12+
metadata:
13+
labels:
14+
name: greeting-api
15+
spec:
16+
containers:
17+
- name: discovery
18+
command: [
19+
"/greeting-api",
20+
"--server_address=0.0.0.0:9090",
21+
"--broker_address=0.0.0.0:10001"
22+
]
23+
env:
24+
- name: MICRO_REGISTRY
25+
value: "consul"
26+
- name: MICRO_REGISTRY_ADDRESS
27+
value: "consul-consul-server"
28+
image: microhq/greeting-api
29+
imagePullPolicy: Always
30+
ports:
31+
- containerPort: 9090
32+
name: api-port
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
namespace: default
5+
name: greeting
6+
spec:
7+
replicas: 1
8+
selector:
9+
matchLabels:
10+
name: greeting-srv
11+
template:
12+
metadata:
13+
labels:
14+
name: greeting-srv
15+
spec:
16+
containers:
17+
- name: greeting
18+
command: [
19+
"/greeting-srv",
20+
"--server_address=0.0.0.0:8080",
21+
"--broker_address=0.0.0.0:10001"
22+
]
23+
env:
24+
- name: MICRO_REGISTRY
25+
value: "consul"
26+
- name: MICRO_REGISTRY_ADDRESS
27+
value: "consul-consul-server"
28+
image: microhq/greeting-srv
29+
imagePullPolicy: Always
30+
ports:
31+
- containerPort: 8080
32+
name: greeting-port
33+
- name: health
34+
command: [
35+
"/health",
36+
"--health_address=0.0.0.0:8081",
37+
"--server_name=greeting",
38+
"--server_address=0.0.0.0:8080"
39+
]
40+
image: microhq/health
41+
imagePullPolicy: Always
42+
livenessProbe:
43+
httpGet:
44+
path: /health
45+
port: 8081
46+
initialDelaySeconds: 3
47+
periodSeconds: 3
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: greeting
5+
namespace: default
6+
labels:
7+
app: greeting
8+
spec:
9+
ports:
10+
- port: 8080
11+
protocol: TCP
12+
selector:
13+
name: greeting-srv
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
namespace: default
5+
name: greeting-web
6+
spec:
7+
replicas: 1
8+
selector:
9+
matchLabels:
10+
name: greeting-web
11+
template:
12+
metadata:
13+
labels:
14+
name: greeting-web
15+
spec:
16+
containers:
17+
- name: greeting-web
18+
command: [
19+
"/greeting-web"
20+
]
21+
env:
22+
- name: MICRO_REGISTRY
23+
value: "consul"
24+
- name: MICRO_REGISTRY_ADDRESS
25+
value: "consul-consul-server"
26+
image: microhq/greeting-web
27+
imagePullPolicy: Always
28+
ports:
29+
- containerPort: 9090
30+
name: web-port

0 commit comments

Comments
 (0)