Development on this project has ceased. Effort has shifted to developing VizHub. For more context and a writeup, see Introducing VizHub Alpha

This codebase has been Open Sourced so that the things figured out here can be visible to developers, in order to port the functionality into VizHub.

This repository contains the Web application once deployed at https://datavis.tech.

Old README content

To understand the overall direction of the product, see Product Vision in the Wiki.

Technology Stack

• Semantic UI CSS framework with user interface components.
• React User interface component system.
• Semantic-UI-React React components for Semantic UI.
• Nextjs Server-side rendering and routing.
• ShareDB Real-time collaboration infrastructure.
• Express Node.js library used for the server.
• Passport Node.js library for authentication.
• Redis Used for storing sessions.
• MongoDB Used for persistent storage (via ShareDB).
• Docker

Getting Started

This section is for getting started with Datavis.tech on your development environment.

First set up SSH keys, then clone the repository with the following command.

git clone [email protected]:curran/datavis-tech.git
cd datavis-tech

The repository uses Git submodules for static content like images and built CSS. This is so the large files don't bloat the main repository over time. Note that our Semantic UI build is located under /static.

Run the following commands to initialize the Git submodule under /static.

git submodule init
git submodule update

Use NPM to install dependencies. See also Install Node.js.

npm install

Start Redis and MongoDB locally using the default ports. See also Install MongoDB and Install Redis.

Start the server locally for development with:

npm start

Now access the app at http://localhost:3000/

Launch the app via Docker

This is the recommended way to run the system during development.

1. Install Docker (instructions for ubuntu)

Note: If using the lastest "artful" Ubuntu 17.10, the command generated by the instructions will not work for adding the PPA. The version needs to be changed to "zesty", the release before "artful", for whith there are supporting Docker packages that work. Try this one instead:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu zesty stable"

2. Install docker-compose (how to install)
3. Launch: sudo docker-compose up

Now you can expect automatic live reloading of client side code changes without restarting or refreshing.

If you change server-side code, you'll need to restart the server for changes to be seen.

Note: Use of sudo is required with a basic Docker installation. To remove the necessity of sudo (for example when setting up a production deploy), you can follow these instructions: Post-installation steps for Linux - Manage Docker as a non-root user. This is highly recommended.

Note: if dependencies in package.json change, you need to rebuild the image before launching again (will re-run npm install for the image):

docker-compose build --force-rm --no-cache app

To rebuild the image without clearing cache:

docker-compose build app

If you're just getting started with the system, please watch this video to learn how to run and use it locally: Datavis Tech Manual Testing.

Debug

Server code

In order to debug server code launch app via:

yarn debug

Inside docker:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up

It is possible to integrate node inspector with IDE:

1. VS code -- create launch.json witht the next content:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "attach",
            "name": "datavis",
            "protocol": "inspector",
            "address": "localhost",
            "port": "56745",
            "remoteRoot": "/usr/src/app/",
            "localRoot": "${workspaceRoot}/"
        }
    ]
}

Note: yarn debug should be launched before.

Testing and Linting

We use unit tests with Jest in this project. To run the tests, run

npm test
or
npm test -- --watch

via docker-compose:
docker-compose run --no-deps --rm app npm test -- --watch

We also use ESLint with JavaScript Standard Style. To lint the code, run

npm run lint

Testing and linting both run in our Continuous Integration setup, and must both pass before any Merge Request is merged.

Contributing

Please review our Product Development Process.

All work should be based on tracked issues in the current sprint, prioritized by the To Do Board.

If you want to work on something that doesn't have an issue yet, please create a new issue for it.

Work that closes an issue should be submitted as a Merge Request. To work on an issue:

• Assign yourself to that issue, so others know you're working on it.
• Create a new branch for your commits on this issue.
• Please prefix your branch name with one of these types and a /
• For example test/edit-page, or feature/reference-autocomplete
• See also git: Basic Branching and Merging.
• After your first commit to the branch, create a "Merge Request"
• Put "WIP" (short for "Work in Progress") in the merge request title if the issue is not resolved yet.
• For the last commit that closes the issue, e.g. issue #42, put the text "Closes #42" in the commit message.
• This will cause the issue to be closed automatically when the branch gets merged.
• Lastly, run npm lint and correct any code style errors there coming from your new code.
• When the issue is resolved and the code is ready for review, remove "WIP" from the merge request title.
• To direct a reviewer's attention to the merge request, add a comment with a "@" mention (which triggers an email notification to be sent)
• For example "@curran This is ready for review.".

Production

This section contains detailed information on how to launch the application in production on Amazon Web Services.

Launching the Virtual Machine

In Amazon EC2, click "launch instance"

• Select "Ubuntu Server"
• Select "t2.small" -> "Next: Configure Instance Details"
• Check "Enable CloudWatch detailed monitoring" -> "Next: Add Storage"
• Use defaults -> "Next: Add Tags"
• Click on "Click to add a Name tag", choose a name -> "Next: Configure Security Group"
• Click "Add rule", select "HTTP" -> "Review and Launch" -> "Launch Instance"

Assigning the Elastic IP

We map datavis.tech to an Amazon Elastic IP, which can be moved between virtual machines.

To move the IP from one instance to another:

• Go to "Elastic IPs" on the left
• Select the Elastic IP 52.2.27.108
• Click "Actions" -> "Dissociate Address" -> "Dissociate Address"
• Click "Actions" -> "Associate Address"
• Select the instance -> "Associate"

After doing this, you can SSH into the server using the following command (replace secret.pem with your key file name):

ssh -i secret.pem [email protected]

To prepare the maching for installations:

sudo apt-get update

Set up SSH Keys

Set up SSH keys for this machine in GitLab, which you'll need in order to clone the repository.

When logged into the virtual machine, generate a new key pair with the following command (use your email):

ssh-keygen -t rsa -C "[email protected]" -b 4096

• Press enter to accept the default "Enter file in which to save..."
• Enter a passphrase
• Copy the text output by cat ~/.ssh/id_rsa.pub
• Go to https://gitlab.com/curran/datavis-tech
• Click on your avatar in upper right -> "Settings" -> "SSH Keys"
• Paste the text in the "Key" box -> click "Add Key"

Clone the Repository

git clone [email protected]:curran/datavis-tech.git
cd datavis-tech

Set up SSL Keys (needed for HTTPS & WSS protocols)

After SSHing into the remote machine, at the home directory:

mkdir ssl

In the local directory with the SSL key, for example called datavis.tech:

scp -i secret.pem ./datavis.tech [email protected]:ssl/datavis.tech

In the directory with the SSL certificates, first combine them then upload:

cat e23afde09cde93de.crt gd_bundle-g2-g1.crt > dvt_combined.crt
scp -i secret.pem ./dvt_combined.crt [email protected]:ssl/dvt_combined.crt

Configure NGINX

These instructions draw from Node.js + Nginx - What now? and Nginx: Configuring HTTPS servers.

After SSHing into the machine:

sudo apt-get install nginx -y

Paste the following configuration into /etc/nginx/sites-available/default:

upstream app_datavistech {
  server 127.0.0.1:3000;
  keepalive 8;
}

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;
  return 301 https://$host$request_uri;
}

server {
  server_name datavis.tech;
  access_log /var/log/nginx/datavistech.log;

  ssl_certificate /home/ubuntu/ssl/dvt_combined.crt;
  ssl_certificate_key /home/ubuntu/ssl/datavis.tech;
  listen 443 ssl;

  location / {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;

    proxy_pass http://app_datavistech/;
    proxy_redirect off;
  }
}

sudo vim /etc/nginx/sites-available/default
sudo /etc/init.d/nginx restart
tail /var/log/nginx/error.log

Install Node.js

Check the latest version of NVM at github.com/creationix/nvm

wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.33.2/install.sh | bash
source ~/.bashrc
nvm install node

Install Redis

Install Redis, from Digital Ocean: How to Install and Use Redis.

cd ~
sudo apt-get install build-essential -y
wget http://download.redis.io/releases/redis-stable.tar.gz
tar xzf redis-stable.tar.gz
cd redis-stable
make
sudo make install
cd utils
sudo ./install_server.sh

Choose all the default settings.

Install MongoDB

MongoDB stores the persistent ShareDB documents for this app. Steps from Install MongoDB Community Edition on Ubuntu.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6
echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list
sudo apt-get update
sudo apt-get install mongodb-org -y
sudo service mongod start

Deploy with PM2

npm install pm2 -g
pm2 start process.yml --env production

Generate a startup script:

pm2 startup

(follow the directions output there - run the command it generates as sudo)

Update Deployment

git pull
git submodule update
npm install
npm run build
pm2 restart all

The above commands are also in deploy.sh, which you can run with sh ./deploy.sh.

Using Elastic Block Store

We use Amazon Elastic Block Store for permanent storage of the MongoDB database content.

To set up, in AWS Web UI, create volume, attach to VM (use default of dev/sdf).

See also http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

lsblk
sudo file -s /dev/xvdf # should say "data"
sudo mkfs -t ext4 /dev/xvdf # Only run this if previous command said "data"
sudo mkdir /data
sudo mount /dev/xvdf /data
sudo cp /etc/fstab /etc/fstab.orig
sudo vim /etc/fstab
# paste this: /dev/xvdf       /data   ext4    defaults,nofail        0       2
sudo mount -a

sudo mkdir /data/mongodb
sudo chmod go+w /data/mongodb

sudo service mongod stop
sudo vim /etc/mongod.conf
# default dbPath is:
#   dbPath: /var/lib/mongodb
# change it to:
#   dbPath: /data/mongodb
sudo service mongod start
tail -f /var/log/mongodb/mongod.log