Getting Started

Overview

Estuary visualizes the story an artifact takes in the Red Hat build to release pipeline, with a focus on the automation of container rebuilds due to CVEs. This repository contains the API and scrapers for the Estuary front end.

Development

Prerequests

The versions listed below are the one which were tested and work. Other versions can work as well.

• Install tox version 3.23.0
• Install docker version 20.10.5
• Install docker-compose version 1.28.5

or

• Invoke the command: make dependencies in a python3 virtualenv

To setup a local development environment:

• Invoke the command: make up

Or

• Spin Gitpod environment

Dependency Management

To manage dependencies, this project uses pip-tools so that the production dependencies are pinned and the hashes of the dependencies are verified during installation.

The unpinned dependencies are recorded in setup.py, and to generate the requirements.txt file, run make pin_dependencies. This is only necessary when modifying the requirements.in files. To upgrade a package, use the -P argument of the pip-compile command.

When installing the dependencies in a production environment, run pip install --require-hashes -r requirements.txt. Alternatively, you may use pip-sync requirements.txt, which will make sure your virtualenv only has the packages listed in requirements.txt.

To ensure the pinned dependencies are not vulnerable, this project uses safety, which runs on every pull-request or can be run manually by make safety.

Run the Unit Tests

Since the unit tests require a running Neo4j instance, the tests are run in Docker containers using Docker Compose. The commands required to run the unit tests are abstracted in scripts/run-tests.sh. This script will create the Docker image required to run the tests based on docker/Dockerfile-tests, create a container with Neo4j, create another container to run the tests based on the built Docker image, run the tests, and then delete the two created containers.

To install Docker and Docker Compose on Fedora, run:

$ sudo dnf install docker docker-compose

To start Docker, run:

$ sudo systemctl start docker

To run the tests, run:

$ sudo scripts/run-tests.sh

To run just a single test, you can run:

sudo scripts/run-tests.sh pytest-3 -vvv tests/test_file::test_name

Run the Infra Tests

• Invoke the command: make infra

Run the Functional Tests

• Invoke the command: make functional

Run the Static Analysis Tests

• Invoke the command: make static_analysis

Run the Safety Tests

• Invoke the command: make safety

Code Styling

The codebase conforms to the style enforced by flake8 with the following exceptions:

• The maximum line length allowed is 100 characters instead of 80 characters

In addition to flake8, docstrings are also enforced by the plugin flake8-docstrings with the following exemptions:

• D100: Missing docstring in public module
• D104: Missing docstring in public package

The format of the docstrings should be in the Sphynx style such as:

Get a resource from Neo4j.

:param str resource: a resource name that maps to a neomodel class
:param str uid: the value of the UniqueIdProperty to query with
:return: a Flask JSON response
:rtype: flask.Response
:raises NotFound: if the item is not found
:raises ValidationError: if an invalid resource was requested

The codebase type checking is implemented with Mypy.

Code Documentation

To document new files, please check here.

Authorization

If authentication is enabled, Estuary can authorize users based on their employee type and a user whitelist configured through the membership of an LDAP group.

Employee Type

You may set the list of valid employee types with the configuration item EMPLOYEE_TYPES. These employee types map to the employeeType LDAP attribute of the user that is added to the OpenID Connect token received by Estuary.

Configuring the Whitelist

To configure a whitelist of users, they must be part of an LDAP group configured with Estuary. The following configuration items are required:

• LDAP_URI - the URI to the LDAP server to connect to in the format of ldaps://server.domain.local.
• LDAP_EXCEPTIONS_GROUP_DN - the distinguished name to the LDAP group acting as the whitelist.

The following configuration items are optional:

• LDAP_CA_CERTIFICATE - the path to the CA certificate that signed the certificate used by the LDAP server. This only applies if you are using LDAPS. This defaults to /etc/pki/tls/certs/ca-bundle.crt.
• LDAP_GROUP_MEMBERSHIP_ATTRIBUTE - the LDAP attribute that represents a user in the group. This defaults to uniqueMember.