[DevOps]: Scope automated metadata ingestion at other sites (prioritize Chrysalis first)

[tomvothecoder]

Objective

Extend SimBoard metadata ingestion beyond Perlmutter/NERSC, starting with Chrysalis, using a shared scheduler-agnostic Python entrypoint plus thin site-specific wrappers.

Current branch: feature/154-ingestion-sites

The branch currently adds:

• backend/app/scripts/ingestion/hpc_archive_ingestor.py: shared entrypoint that delegates to the existing NERSC archive ingestor for now.
• backend/app/scripts/ingestion/sites/chrysalis.sh: thin Chrysalis Jenkins wrapper.
• README updates documenting the shared ingestor and Chrysalis defaults.
• A test that verifies the generic entrypoint delegates through the existing ingestor path.

Task

Finish turning this branch into a deployable ingestion path for Chrysalis first, then use the same pattern for other sites once access is available.

• Review the current branch implementation and keep ingestion logic in Python, not shell wrappers.
• Validate the Chrysalis archive path and Jenkins runtime assumptions.
• Confirm how SIMBOARD_API_BASE_URL and SIMBOARD_API_TOKEN should be stored and injected in the Chrysalis Jenkins job.
• Run the Chrysalis wrapper in dry-run mode and verify archive access, network egress to SimBoard, and candidate counts.
• Enable non-dry-run ingestion only after the dry-run output is validated.
• Apply for or confirm accounts/access for Frontier, Aurora, and Compy.
• After Chrysalis works, add equivalent thin wrappers for the remaining sites as access allows.

Site Priorities

• Chrysalis: priority once available again; Jenkins-based workflow.
• Frontier: apply for account; likely next priority.
• Aurora: apply for account; likely next priority.
• Compy: apply for account.
• Anvil: removed from scope.

References

• PACE overview: https://e3sm.atlassian.net/wiki/spaces/EPG/pages/776437853/Performance+Analytics+for+Computational+Experiments+PACE
• PACE collection/upload reference: https://e3sm.atlassian.net/wiki/spaces/EPG/pages/5477335106/PACE+Collection+and+Upload+Reference
• Existing site script wrappers: https://github.com/E3SM-Project/E3SM_test_scripts/tree/master/jenkins
• Existing PACE archive script: https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/util/pace_archive.sh

[tomvothecoder]

Hey @rljacob, I’m scoping SimBoard metadata ingestion beyond Perlmutter/NERSC.

Your PACE Confluence page lists Chrysalis, Compy, Aurora, Frontier, and Anvil. Should we support all, or prioritize 1-2 first?

My assumption: start with Chrysalis, then reuse same scheduler-agnostic ingestion framework elsewhere.

Current runners from page:

• Jenkins: Chrysalis, Compy, Anvil
• Cron: Frontier
• ALCF GitLab: Aurora

Does priority/order look right, and are these runners still current? Also, I would need to request access to Compy, Frontier, or Aurora so I'm blocked there for now.

[rljacob]

Anvil can be removed. You should apply for accounts on Frontier, Aurora and Compy. The first 2 will probably be fastest. Make Chrysalis the priority once it comes back. Rob From: Tom Vo ***@***.***> Date: Tuesday, April 28, 2026 at 1:21 PM To: E3SM-Project/simboard ***@***.***> Cc: Jacob, Robert L. ***@***.***>; Mention ***@***.***> Subject: Re: [E3SM-Project/simboard] [Enhancement]: Scope automated metadata ingestion at other sites (Issue #154) This Message Is From an External Sender This message came from outside your organization. [https://avatars.githubusercontent.com/u/25624127?s=20&v=4]tomvothecoder left a comment (E3SM-Project/simboard#154)<https://urldefense.us/v3/__https://github.com/E3SM-Project/simboard/issues/154*issuecomment-4338001402__;Iw!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yvhSHh4lY$> Hey @rljacob<https://urldefense.us/v3/__https://github.com/rljacob__;!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yvg4g8e0o$>, I’m scoping SimBoard metadata ingestion beyond Perlmutter/NERSC. Your PACE Confluence page<https://urldefense.us/v3/__https://e3sm.atlassian.net/wiki/spaces/EPG/pages/5477335106/PACE*Collection*and*Upload*Reference__;KysrKw!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yvF0pnqVo$> lists Chrysalis, Compy, Aurora, Frontier, and Anvil. Should we support all, or prioritize 1-2 first? My assumption: start with Chrysalis, then reuse same scheduler-agnostic ingestion framework elsewhere. Current runners from page: * Jenkins: Chrysalis, Compy, Anvil * Cron: Frontier * ALCF GitLab: Aurora Does priority/order look right, and are these runners still current? Also, I would need to request access to Compy, Frontier, or Aurora so I'm blocked there for now. — Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https://github.com/E3SM-Project/simboard/issues/154*issuecomment-4338001402__;Iw!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yvhSHh4lY$>, or unsubscribe<https://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAHHG7AW33RWP2RPJSXA2QL4YDZBHAVCNFSM6AAAAACW6BWFDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGMZYGAYDCNBQGI__;!!G_uCfscf7eWS!fdyZLsYJBPlI7ti3lBPdE5YdYox-ZsuIlLmJk3if_IE3AvTlz4v1azpwSPH-W_YdQEwN1KOXdnCJv1yv26dcKnY$>. You are receiving this because you were mentioned.