Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor the GitHub Workflows in this package #329

Closed
jcfreeman2 opened this issue Dec 1, 2023 · 1 comment
Closed

Refactor the GitHub Workflows in this package #329

jcfreeman2 opened this issue Dec 1, 2023 · 1 comment
Assignees
@jcfreeman2
Labels
maintenance

Comments

@jcfreeman2
Copy link
Collaborator

There's an awful lot of code in our Workflows which is basically repeated. For one thing, we have six different files for six different ways of cutting a release (nightly + candidate + frozen) * (sl7 + alma9), and they have lots of lots of stuff in common. It would be good to try to put a lot of this common code into scripts, since (1) it would just be good coding practice to support the DRY principle, and (2) scripts would allow us to recreate these Workflows on local hosts where we can poke around if something goes wrong, rather than being walled off by a GitHub Action.
@jcfreeman2 jcfreeman2 self-assigned this Dec 1, 2023
jcfreeman2 added a commit that referenced this issue Dec 4, 2023
@jcfreeman2
JCF: Issue #329: a first pass at refactoring nightly-spack-alma9.yml
f4eb260 
Some of the changes include:

-Putting environment variables used across the Workflow's jobs into a source-able file called wf-setup-tools.sh

-Adding functions also used across jobs, such as
--"get_spack": get the actual Spack software
--"daqify_spack_environment": adjust repos.yaml, etc., to set up for DAQ software installation
--"tar_and_stage_release": tar up files and make an uploadable tarfile

-Giving the environment variables more descriptive names. The changes are:
-- DAQ_RELEASE ->       RELEASE_TAG
-- DAQ_RELEASE_DIR ->   DAQ_RELEASE_REPO
-- SPACK_RELEASE_DIR -> RELEASE_DIR
-- SPACK_RELEASE ->     SPACK_AREA
-- BASE_DAQ_RELEASE ->       BASE_RELEASE_TAG
-- SPACK_BASE_RELEASE_DIR -> BASE_RELEASE_DIR
-- SPACK_BASE_RELEASE ->     BASE_SPACK_AREA
jcfreeman2 added a commit that referenced this issue Dec 5, 2023
@jcfreeman2
JCF: Issue #329: some low hanging fruit re: Workflow reorganization -…
ef0c515 
… drop the obsolete/unused feature branch workflow, and put the SL7 nightly workflow filename on equal footing with the Alma 9 nightly workflow
jcfreeman2 added a commit that referenced this issue Dec 6, 2023
@jcfreeman2
JCF: Issue #329: use the build-stack.sh script in the nightly SL7 build
7e3b89f
jcfreeman2 added a commit that referenced this issue Dec 6, 2023
@jcfreeman2
JCF: Issue #329: regularize the name of nightly-spack-sl7.yml
901cd0f
jcfreeman2 added a commit that referenced this issue Dec 11, 2023
@jcfreeman2
JCF: Issue #329: drop obsolete-and-now-confusing build-nightly-releas…
62fd848 
…e.sh, rename build-stack.sh to build-release.sh
jcfreeman2 added a commit that referenced this issue Dec 11, 2023
@jcfreeman2
JCF: Issue #329: remove accidental, premature change to build-candida…
2190795 
…te-release.yml
jcfreeman2 added a commit that referenced this issue Dec 11, 2023
@jcfreeman2
JCF: Issue #329: drop obsolete-and-now-confusing build-nightly-releas…
5a5abe2 
…e.sh, rename build-stack.sh to build-release.sh
jcfreeman2 added a commit that referenced this issue Dec 11, 2023
@jcfreeman2
JCF: Issue #329: remove accidental, premature change to build-candida…
e1e01e6 
…te-release.yml
jcfreeman2 added a commit that referenced this issue Dec 13, 2023
@jcfreeman2
JCF: Issue #329: generalize build-release.sh to handle not only night…
328bcea 
…ly but also candidate and frozen releases; update the nightly and candidate workflows
jcfreeman2 added a commit that referenced this issue Dec 15, 2023
@jcfreeman2
JCF: Issue #329: get the frozen build to take advantage of the new re…
48d2e69 
…factoring tools and add additional error checking/reporting to build-release.sh
jcfreeman2 added a commit that referenced this issue Dec 15, 2023
@jcfreeman2
JCF: Issue #329: finishing touches for merge into develop
a21b6d1
@jcfreeman2
Copy link
Collaborator Author

Closing this Issue now that the refactor has been merged. A couple of things:

  • Most of the refactor involved moving common functionality into .github/workflows/wf-setup-tools.sh and scripts/spack/build-release.sh. The latter, in particular, can be used at the command line to manually recreate a GitHub Action
  • This refactor meant that the five release build Workflow files we have (nightly Alma9/SL7, frozen Alma9/SL7, and candidate Alma9 - yes, we don't have a candidate SL7 on the develop branch) reduced from 2327 to 1264 lines, or just over half the original length
  • The refactor also included dropping the obsolete feature branch workflow dunedaq-feature-branch-cpp-ci.yml and regularizing the name of the nightly SL7 build (nightly-spack.yml -> nightly-spack-sl7.yml)

There absolutely are other improvements to be made. But the changes which got merged in are a major step in the right direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jcfreeman2 jcfreeman2

Labels
maintenance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jcfreeman2