Skip to content

workflow.md

Latest commit

 

History

History
216 lines (157 loc) · 11.5 KB

workflow.md

File metadata and controls

216 lines (157 loc) · 11.5 KB

Coin update workflow

This document attempts to describe the process for updating or deleting an existing coin configuration or adding a new coin. The information required has not changed from the old process but the mechanism by which it is consumed has been changed. In an effort to increase efficiency and reduce the liklihood of mistakes we have automated as much of the process as possible.

Adding or updating a coin definition in the blockchain-configuration-files repository is only the first part of the process. The configuration files created here are consumed by different processes, including:

  • Docker image creation automation
  • EXR proxy deployment automation
  • BlockDX configuration

The entire "automation" workflow is spread over several repositories. This documentation is a preliminary WIP and feedback is welcome to help improve it.

Ultimate authorities

The sources of truth for BlockDX coin configurations are the following files within this repository:

Path purpose
manifest-latest.json master ultimate authority
autobuild/configs/*.base.j2 ultimate authority
autobuild/templates/*j2 auxiliary templates
wallet-confs/*.conf public facing config fragments
xbridge-confs/*.conf public facing config fragments

Ultimate authority files are created manually or by automation workflows while public facing config fragments are generated by automation workflows. In the past the public facing config fragments were manually generated which allowed mistakes to creep in. The automation workflows in this and other repositories attempt to speed up consumption processes and reduce errors.

Eventually automation may be available to generate most of the ultimate authority information too given only a coin name and Github URL, but for now they must be manually created.

These sources of truth are manipulated/consumed by five automation scripts in this blockchain-configuration-files repository.

Automation scripts

There is one primary script intended for "daily" use in the autobuild directory, and three auxilliary scripts located in the autobuild/utils sub-directory intended for "special" use tasks. There is one additional script in the tools directory which can be used periodically to discover coins which need checking and possibly updating.

tools/check-versions.py

Purpose: to identify coins for which there are newer versions available than we currently support.

Pre-reqs:

  • manifest-latest.json
  • Github personal access token

Execution:

  • customise the script with your Github personal access token
  • within your local repo/tools directory:
  mark@x230:~/src/blockdx-configs/tools$ python3 check-versions.py
  ABS Version inconsistency. Old: v1.0.0. New: v1.0.0b
  AEX Version inconsistency. Old: v2.2. New: v2.2.1
  AUS Version inconsistency. Old: v0.17.4.1. New: v1.17.1
  ...
  XZC Version inconsistency. Old: v0.14.0.2. New: v0.14.9.2

Output: A list of coins for which there is a newer version than we currently support. Warnings are also printed for any coin for which we couldn't find releases or tags information on the expected Github URL.

autobuild/app.py

Purpose: builds individual coin wallet.conf and xbridge.conf fragments from the new format .j2 master sources-of-truth. With no parameters it builds wallet and XBridge configs for ALL coins listed in the manifest. Optionally, users can specify a single coin or list of coins they wish to generate configs for using the --coin <coin1>,<coin2>,... option.

Pre-reqs:

  • manifest-latest.json
  • autobuild/configs/*.base.j2
  • autobuild/templates/*.j2

Execution: within your local repo/autobuild directory, for example:

mark@x230:~/src/blockdx-configs$ cd autobuild
mark@x230:~/src/blockdx-configs/autobuild$ python3 app.py
mark@x230:~/src/blockdx-configs/autobuild$ python3 app.py --coin BTC,LTC

Output: generated wallet and xbridge configuration fragments in

  • wallet-confs/*.conf
  • xbridge-confs/*.conf

autobuild/utils/cleanup-manifest.py

Purpose: consolidate duplicate coin sections in manifest-latest.json into a single entry with multiple versions, and remove deprecated {wallet,xbridge}-confs files. This is mainly intended to be a one-time step in preparation for running create-j2-confs.py but it may be useful to run it at other times.

Pre-reqs:

  • manifest-latest.json

Execution: within your local repo/autobuild/utils directory, for example:

mark@x230:~/src/blockdx-configs$ cd autobuild/utils
mark@x230:~/src/blockdx-configs/autobuild/utils$ python3 cleanup-manifest.py

Output: updated manifest-latest.json in the current directory. This allows you to easily compare the output with the input and "approve/reject/amend" any changes.

Warning: check the output very carefully before copying it back to the top level directory and committing any changes.

autobuild/utils/create-j2-confs.py

Purpose: create j2 skeletons from manifest-latest.json and wallet/xbridge config fragments. This is mainly intended to be a one-time operation to convert the old style two-files per coin templates into a single-file per coin template to be used going forwards. It can be used on a more constrained input set by specifying

Pre-reqs:

  • manifest-latest.json
  • wallet-confs/*.conf
  • xbridge-confs/*.conf

Execution: within your local repo/autobuild/utils directory, for example:

mark@x230:~/src/blockdx-configs$ cd autobuild/utils
mark@x230:~/src/blockdx-configs/autobuild/utils$ python3 create-j2-confs.py
ABET,ABS,AEX,AGM,APR,ATB,AUS,BAD,BCD,BCH,BCZ,BIT,BITG,BLAST,BLOCK,BSD,BTC,BTDX,BTG,BTX,BZX,CARE,CDZC,CHC,CHI,CHN,CIV,CNMC,COLX,CRAVE,D,DASH,DGB,DIVI,DMD,DOGE,DOGEC,DSR,DVT,DYN,ECA,EMC,EMC2,ENT,FAIR,FGC,FJC,FLO,GALI,GBX,GEEK,GIN,GLC,GMCN,GXX,HASH,HATCH,HLM,HTML,INN,IOP,IXC,JEW,JIYOX,KLKS,KREDS,KYDC,KZC,LBC,LTC,LUX,LYNX,MAC,MLM,MNP,MONA,MRX,MUE,N8V,NIX,NMC,NOR,NORT,NYEX,NYX,ODIN,OHMC,OPCX,ORE,PAC,PART,PHL,PHR,PIVX,POLIS,PURA,QBIC,QTUM,RAP,REEX,RPD,RVN,SCC,SCN,SCRIBE,SEND,SEQ,SIB,SPK,STAK,SUB1X,SWIFT,SYS,TRB,TRC,UFO,UNO,VIA,VITAE,VIVO,VSX,VTC,WAGE,WGR,XC,XMCC,XMY,XN,XP,XVG,XZC,ZNZ
mark@x230:~/src/blockdx-configs/autobuild/utils$ python3 create-j2-confs.py --coins SYS,MUE
SYS,MUE

Output:

  • autobuild/configs/*.base.j2

It addition to writing the j2 template files the script writes on the console the list of coins which were successfully processed.

autobuild/utils/manifest-management.py

Purpose: interactive manifest management

Pre-reqs:

  • manifest-latest.json

Execution: within your local repo/autobuild/utils directory, for example:

mark@x230:~/src/blockdx-configs$ cd autobuild/utils
mark@x230:~/src/blockdx-configs/autobuild/utils$ python3 manifest-management.py

Output:

  • manifest-latest.json

Warning: this script is still WIP and not ready for regular use.

Workflow

Update coin configs

Audience: everyone

The new process to update a coin, eg: on a version upgrade is as follows:

  1. fork the https://github.com/blocknetdx/blockchain-configuration-files repo
  2. clone the repo to your local machine
  3. checkout a branch from master with a suitable name, eg: bump-thing-v4.2.2
  4. update information in the manifest-latest.json
  5. update information in autobuild/configs/thing.base.j2
  6. run app.py as described above to update the {wallet,xbridge}-confs/thing.conf
  7. commit the changes to
    • manifest-latest.json
    • autobuild/configs/thing.base.j2
    • wallet-confs/thing.conf
    • xbridge-confs/thing.conf
  8. push the branch back to your repo

Other parts of the workflow depend on who is performing them and take place in other repos but are documented here for simplicity in the order they are expected to happen.

Build a new Docker image

Audience: Not directly applicable to 3rd party developers who are simply looking to list a coin because the workflow includes building a Docker image in the blocknetdx Docker hub repo. 3rd party developers could choose to branch the https://github.com/blocknetdx/dockerimages repo and adapt it to push the image to their own Docker hub repo, or reach out to Blocknet team members for assistance through Discord #block-dx-listing or a Github issue.

  • Go to https://github.com/blocknetdx/dockerimages/actions and choose the BUILD IMAGE FROM TEMPLATE workflow.
  • Click the Run workflow dropdown.
  • Choose the appropriate branch (most likely master) in the Use workflow from dropdown.
  • Enter the URL of your branch in the Full path of a branch where the config can be found. field, eg: https://raw.githubusercontent.com/walkjivefly/blockchain-configuration-files/bump-BLOCK-v4.3.3
  • Enter the full name of the coin you wish to create a Docker image for in the The full name of a wallet. Example - bitcoin, dash, blocknet. field
  • Enter the version you wish to build in Tag or version of an image, please check manifest-latest.json. By default it takes the newest version.
  • Click the green Run workflow button.

If successful, the workflow takes around 30 minutes to build and push a new image to Docker hub. The image will have a -staging tag at this point signifying it has not been tested and released for public use.

Build exrproxy-env deployment package

Audience: Not applicable to 3rd party developers who are simply looking to list a coin.

  • Follow the Automated Docker-Based EXR Service Node Setup guide to deploy one testnet Service Node and two testnet Trading Nodes on three different machines.
  • Configure each of these three nodes to support the coin to be tested and any counter-party coin you want to use to trade against the coin to be tested.
  • When you arrive at the step where ./builder.py is called, be sure to pass it the (raw form) path to the branch you pushed to your Github Repo in step 8 above, like this:
./builder.py --branchpath 'https://raw.githubusercontent.com/walkjivefly/blockchain-configuration-files/bump-thing-v4.2.2'

Testing (CLI)

Audience: Not directly applicable to 3rd party developers, although they could choose to follow a similar process if they don't want to jump through the hoops required (at this time) to test with BlockDX.

  • Bring up the Docker containers on all three machines mentioned in the previous section and let them sync if necessary.
  • Fund, encrypt and unlock the wallets as required.
  • Restart the servicenode as a servicenode.
  • On one trading machine attempt an XBridge trade between your new coin and any counter-party using dxmakeorder.
  • Check the order is visible on the other trading machine and attempt to take it using dxtakeorder.
  • Confirm several orders complete successfully, check that interrupted trades are correctly rolled back and all funds end up where they should. Record the trade ids for inclusion in the pull request.

Testing (BlockDX)

Instructions coming soon (TM).

Comment: this is where blocknetdx/block-dx#339 would be really useful right now!

Upon successful testing

Audience: everyone