Introduction

This contains the configuration for the Cloud Foundry Buildpacks team Concourse deployment.

Pipelines

• dependency-builds: build binaries for Cloud Foundry buildpacks
• buildpacks: test and release all of the buildpacks
• brats: run BRATS against the master branch of buildpacks
• buildpack-verification: generate static site for buildpack verification
• buildpacks-ci: testing tasks for correct usage, rebuild CI docker images
• cf-release: deployment of latest buildpacks to cf-release develop

Concourse State

Jobs and tasks in the buildpacks-ci repository store state in public-buildpacks-ci-robots. See repository README for details.

Commands and recipes

Updating all the Pipelines

./bin/update-pipelines

Debugging the build

fly intercept -j $JOB_NAME -t task -n $TASK_NAME

Clearing the git resources

fly intercept -c $RESOURCE_NAME rm -rf /tmp/git-resource-repo-cache

To build a new version of a binary

1. Check out the binary-builds branch

2. Edit the YAML file appropriate for the build (e.g. ruby-builds.yml)

3. Find the version number and package SHA256 of the new binary. For many binaries, the project website provides the SHA256 along with the release (for example, jruby.org/download provides the SHA256 along with each JRuby release). For others (such as Godep), you download the .tar.gz file and run shasum -a 256 <tar_file> to obtain the SHA256.

4. Add any number of versions and their checksums to the array, e.g.

   ruby:
   - version: 2.2.2
     sha256: 5ffc0f317e429e6b29d4a98ac521c3ce65481bfd22a8cf845fa02a7b113d9b44

5. git commit -am 'Build ruby 2.2.2' && git push

Build should automatically kick off at https://buildpacks.ci.cf-app.com/pipelines/binary-builder and silently upload a binary to the buildpacks.cloudfoundry.org bucket under dependencies/, e.g. https://buildpacks.cloudfoundry.org.s3.amazonaws.com/dependencies/ruby/ruby-2.2.2-linux-x64.tgz

Note that the array is a stack, which will be emptied as the build succeeds in packaging successive versions.

Making Changes to Build Scripts

When you want to change how a binary gets built, there are two places you may need to make changes. All binaries are built by the dependency-builds pipeline, and you may need to change the task that builds them. For many binaries, the dependency-builds pipeline runs recipes from the binary-builder repo; for those binaries, you will usually need to change the recipe rather than the concourse task.

For the list of currently supported binaries, check out our dependency-builds pipeline.

The concourse task that orchestrates the building is buildpacks-ci/tasks/build-binary-new/builder.rb; many of the recipes are in binary-builder.

To test these changes locally, you can execute the concourse task for it, but point to local changes. For instance:

$ cd buildpacks-ci
$ STACK=cflinuxfs2 fly -t buildpacks e -c tasks/build-binary-new/build.yml -j dependency-builds/build-r-3.4.X -i buildpacks-ci=.

For binaries that use recipes in binary-builder, you can also test in Docker. For instance:

$ docker run -w /binary-builder -v `pwd`:/binary-builder -it cloudfoundry/cflinuxfs2:ruby-2.2.4 ./bin/binary-builder --name=ruby --version=2.2.3 --md5=150a5efc5f5d8a8011f30aa2594a7654
$ ls
ruby-2.2.3-linux-x64.tgz

Buildpack Repositories Guide

buildpacks-ci pipelines and tasks refer to many other repositories. These repos are where the buildpack team and others develop buildpacks and related artifacts.

Officially-supported Buildpacks

Each officially-supported buildpack has a develop and a master branch.

Active development happens on develop. Despite our best efforts, develop will sometimes be unstable and is not production-ready.

Our release branch is master. This is stable and only updated with new buildpack releases.

• binary-buildpack
• go-buildpack
• nodejs-buildpack
• php-buildpack
• python-buildpack
• ruby-buildpack
• dotnet-core-buildpack
• staticfile-buildpack

Tooling for Development and Runtime

• buildpack-packager Builds cached and uncached buildpacks
• machete Buildpack integration testing framework.
• compile-extensions Suite of utility scripts used in buildpacks at runtime
• libbuildpack Library used for writing buildpacks in Golang
• binary-builder Builds binaries against specified rootfs
• brats Buildpack Runtime Acceptance Test Suite, a collection of smoke tests

BOSH Releases

BOSH releases are used in the assembly of cf-release.

• go-buildpack-release
• ruby-buildpack-release
• python-buildpack-release
• php-buildpack-release
• nodejs-buildpack-release
• staticfile-buildpack-release
• binary-buildpack-release
• java-offline-buildpack-release
• java-buildpack-release
• dotnet-core-buildpack-release

Experimental or unsupported

Buildpacks

These buildpacks are possible candidates for promotion, or experimental architecture explorations.

• hwc-buildpack
• hwc-buildpack-release

Tools

• concourse-filter Redacts credentials from Concourse logs
• new_version_resource Concourse resource to track dependency versions by scraping webpages

Private Repos

Some repositories are private for historical or security reasons. We list them for completeness.

• deployments-buildpacks See repository README.
• buildpacks-ci-robots See repository README.