build: add Quadlet configuration and RPM packaging

[seantronsen]

Pull Request Template

Thank you for your contribution! Please ensure the following before submitting:

Checklist

• My code follows the style guidelines of this project
• I have added/updated comments where needed
• I have added tests that prove my fix is effective or my feature works
• I have run make test (or equivalent) locally and all tests pass
• DCO Sign-off: All commits are signed off (git commit -s) with my real name and email
• REUSE Compliance:
  • Each new/modified source file has SPDX copyright and license headers
  • Any non-commentable files include a <filename>.license sidecar
  • All referenced licenses are present in the LICENSES/ directory

Description

First draft at the idea of creating an RPM quadlet for metadata-service. It's still in the draft stage with TODO markers placed anywhere further design decisions were required.

Should also await the fix for #33 as it would introduce config file changes.

Fixes #31

Important

Must await:

• Feature/dry quadlet rpms github-actions#14
• feat!: update release with new fabrica-based services; remove old services release#50

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

---

For more info, see Contributing Guidelines.

[seantronsen]

metadata-service.yaml restates compiled-in defaults, splitting the real configuration across two places. If we're moving to systemd drop-ins for overrides, that hurts: env resolves above config, but viper discards set-but-empty vars, so a drop-in can override a yaml value and never clear one. Once tokensmith_url is in the yaml, only editing an RPM-owned %config(noreplace) file turns it off. Leaner yaml means fewer values systemd can't reach.

[synackd]

I wouldn't want a systemd override to overrule a config file value if set by a user. I'm for commenting the defaults in the config file and have the override set things as long as the user can override that in the config file.

[seantronsen]

I wouldn't want a systemd override to overrule a config file value if set by a user. I'm for commenting the defaults in the config file and have the override set things as long as the user can override that in the config file.

Noted, but that might also be a problematic approach since it would break the precedence order: compiled-in defaults < config.yaml < env vars < cli opts.

However, I don't exactly like idea of users having to create a "unset vars" drop in conf just to use the config.yaml file either.

[seantronsen]

I wouldn't want a systemd override to overrule a config file value if set by a user. I'm for commenting the defaults in the config file and have the override set things as long as the user can override that in the config file.

Noted, but that might also be a problematic approach since it would break the precedence order: compiled-in defaults < config.yaml < env vars < cli opts.

However, I don't exactly like idea of users having to create a "unset vars" drop in conf just to use the config.yaml file either.

@synackd for now it might be best to comment all the fields out. My biggest concern at the moment is that we'd be having config items split across multiple locations, thus making them more difficult to track.

In addition, the current idea is for meta-packages + CLI scripts to further configure these generic packages for actual site installations. There are a few ways that could go... RPM installs another config file for the specific deployment type or it could install a drop in (more consistent with non-fabrica projects).

It might be worth chatting about this tomorrow.

[synackd]

I wouldn't want a systemd override to overrule a config file value if set by a user. I'm for commenting the defaults in the config file and have the override set things as long as the user can override that in the config file.

Noted, but that might also be a problematic approach since it would break the precedence order: compiled-in defaults < config.yaml < env vars < cli opts.
However, I don't exactly like idea of users having to create a "unset vars" drop in conf just to use the config.yaml file either.

User experience is exactly what I'm trying pointing out here. Ideally, the user shouldn't need to worry about Systemd overrides for items that are found in a service's config file since the latter is the idiomatic way of configuration.

I would caution against the use of environment variables as the default config mechanism since, as you point out, it has a higher precedence than a config file and there are effectively two ways to override them: adding an env var override somewhere (but Systemd overrides are already doing this), or injecting CLI flags (which would also need to happen via Systemd). Both of these raise the likelihood of misconfiguration, so we should carefully consider what we relegate to environment variables.

@synackd for now it might be best to comment all the fields out. My biggest concern at the moment is that we'd be having config items split across multiple locations, thus making them more difficult to track.

In addition, the current idea is for meta-packages + CLI scripts to further configure these generic packages for actual site installations. There are a few ways that could go... RPM installs another config file for the specific deployment type or it could install a drop in (more consistent with non-fabrica projects).

It might be worth chatting about this tomorrow.

We should consider this carefully. I would argue that almost everything should go into the config file so that environment variables can be reserved for temporary overrides and/or troubleshooting. I think repetition is a reasonable cost in this case, but it would be worth discussing. Perhaps the only exception would be the ACME certificate naming.

[seantronsen]

I wouldn't want a systemd override to overrule a config file value if set by a user. I'm for commenting the defaults in the config file and have the override set things as long as the user can override that in the config file.

Noted, but that might also be a problematic approach since it would break the precedence order: compiled-in defaults < config.yaml < env vars < cli opts.
However, I don't exactly like idea of users having to create a "unset vars" drop in conf just to use the config.yaml file either.

User experience is exactly what I'm trying pointing out here. Ideally, the user shouldn't need to worry about Systemd overrides for items that are found in a service's config file since the latter is the idiomatic way of configuration.

I would caution against the use of environment variables as the default config mechanism since, as you point out, it has a higher precedence than a config file and there are effectively two ways to override them: adding an env var override somewhere (but Systemd overrides are already doing this), or injecting CLI flags (which would also need to happen via Systemd). Both of these raise the likelihood of misconfiguration, so we should carefully consider what we relegate to environment variables.

@synackd for now it might be best to comment all the fields out. My biggest concern at the moment is that we'd be having config items split across multiple locations, thus making them more difficult to track.
In addition, the current idea is for meta-packages + CLI scripts to further configure these generic packages for actual site installations. There are a few ways that could go... RPM installs another config file for the specific deployment type or it could install a drop in (more consistent with non-fabrica projects).
It might be worth chatting about this tomorrow.

We should consider this carefully. I would argue that almost everything should go into the config file so that environment variables can be reserved for temporary overrides and/or troubleshooting. I think repetition is a reasonable cost in this case, but it would be worth discussing. Perhaps the only exception would be the ACME certificate naming.

Agreed wholesale on the UX side especially. I'll mark this as a discussion item for the meeting tomorrow morning. Some services do not have dedicated config files which presents one oddity. Another is "how do we want to deploy these config.yaml files?"

The latter will likely require a more involved discussion around:

• moving away from drop-ins and toward configs, what should the sane defaults be? should everything be commented out?
• how to handle alternative deployment strategies in a sane way? (e.g., do we still provide meta-package configuration RPMs that set up the sane defaults for specific deployment strategies (single-node, multi-node, etc.))

[synackd]

Those are good questions we should address.

Some services do not have dedicated config files which presents one oddity. Another is "how do we want to deploy these config.yaml files?"

This would be something we would want to adapt if we went that route (which, in my opinion, we should go for the reasons above). Currently, the config files live in the release repo so we would want to copy them to the relevant repos, make them more complete, and add comments.

• moving away from drop-ins and toward configs, what should the sane defaults be? should everything be commented out?

I think the values from the tutorial are probably sane defaults since most folks are likely coming from there. That would ease the tutorial configuration friction as well.

• how to handle alternative deployment strategies in a sane way? (e.g., do we still provide meta-package configuration RPMs that set up the sane defaults for specific deployment strategies (single-node, multi-node, etc.))

This probably has more diverse opinions and likely would involve long-term planning, and I think that thorough configuration documentation could fill any gaps. For the quadlet deployments, I think the RPM with config files and any needed config override scripts are probably fine. With config files, customizations are preserved via .rpmnew/.rpmsave files which allows users to diff any updates. Starting with the quadlets as a simple start is good, but it's worth thinking through long-term.

[alexlovelltroy]

Going back to basics with the Twelve-Factor-App basics and the default precedence model from viper, I think we should consider this as a test case for an RFD to define configuration preferences. We're already basically using the viper precedence model as: compiled defaults < config file < env vars < CLI flags. Layering the quadlet/rpm file behavior on top makes it a little muddled. Is a %config(noreplace) the right way to preserve local modifications? Can we expect admins to look in multiple places for config information?

I think the split ends up looking like this:

• durable, user-owned service config should live in the .yaml file: port,host, smd_url, etc... which should be as complete as possible with plenty of comments including examples from the tutorial. NO SECRETS
• runtime mechanics should live in the systemd/quadlet drop-in: volume layout, secrets, health checks, etc... with comments to indicate available env vars for override
• live env defaults are problematic in RPMs. 10-defaults.conf should really just be container runtime invariants that aren't already in the yaml file

This explicitly states a standard we've been developing without naming. We should discuss adding an RFD/ADR along the lines of:

1. Compiled defaults: safe local/dev defaults only.
2. Config file: primary durable operator interface.
3. Environment variables: deployment injection, secrets, temporary overrides, container/platform wiring.
4. CLI flags: ad hoc/manual/debug overrides, highest precedence.
5. Systemd drop-ins: unit behavior and site/meta-package overrides, not the default user config surface.
6. RPM packages: install service-owned config as %config(noreplace) and install vendor Quadlets under /usr/share/containers/systemd; admin overrides belong in /etc.
7. Kubernetes: same model maps to ConfigMaps for non-secret config, Secrets for confidential values; env vars are acceptable but ConfigMap/Secret volume files are better when operators expect editable config files.

[synackd]

Initial review of just the config file.

Broadly looks good, but there might be old config options present.

A format proposal I have is to organize the default values and corresponding flag/env vars in the comments.

Here's the config file example I had in mind:

# SPDX-FileCopyrightText: © 2026 OpenCHAMI a Series of LF Projects, LLC
# SPDX-License-Identifier: MIT

# /etc/metadata-service/config.yaml
#
# =============================================================================
# Metadata‑Service - Example Configuration
# =============================================================================
#
# This file serves as a reference that users can copy, uncomment, and edit
# for the options they need.  All keys shown use the service's default
# values. Each option lists the flag (--flag-name) and the environment
# variable (METADATA_SERVICE_...) that can be used to override it.

# -------------------------------------------------------------------------
# Server Configuration
# -------------------------------------------------------------------------

# TCP port to listen on
#
# Default: 8080
# Flag:    --port
# Env var: METADATA_SERVICE_PORT
port: 8080

# Host/IP address to bind to
#
# Default: 0.0.0.0 (i.e. all interfaces)
# Flag:    --host
# Env var: METADATA_SERVICE_HOST
host: "0.0.0.0"

# HTTP request read timeout, in seconds
#
# Default: 15
# Flag:    --read-timeout
# Env var: METADATA_SERVICE_READ_TIMEOUT
#read_timeout: 15

# HTTP request write timeout, in seconds
#
# Default: 15
# Flag:    --write-timeout
# Env var: METADATA_SERVICE_WRITE_TIMEOUT
#write_timeout: 15

# HTTP connection idle timeout, in seconds
#
# Default: 60
# Flag:    --idle-timeout
# Env var: METADATA_SERVICE_IDLE_TIMEOUT
#idle_timeout: 60

# Enable debug‑level logging.  When true, log entries include file and line
# information to aid troubleshooting.
#
# Flag:    --debug
# Env var: METADATA_SERVICE_DEBUG
debug: false

# Enable Prometheus metrics collection and expose the `/metrics` endpoint.
# Flag:    --enable-metrics
# Env var: METADATA_SERVICE_ENABLE_METRICS
enable_metrics: false

# Port on which the Prometheus metrics endpoint listens.
# Separate from the main service port for security isolation.
# Flag:    --metrics-port
# Env var: METADATA_SERVICE_METRICS_PORT
metrics_port: 9090

# -------------------------------------------------------------------------
# Storage Configuration
# -------------------------------------------------------------------------

# Directory used by the file‑based storage backend.
# Stores ClusterDefaults, Group, InstanceInfo, and WireGuardPeer resources.
#
# NOTE: When using the container, this should be the path where a volume/host
# directory is mounted into the container. The default /data is unwritable by
# the non-root container user.
#
# Default: /data
# Flag:    --data-dir
# Env var: METADATA_SERVICE_DATA_DIR
data_dir: "/app/data"

# -------------------------------------------------------------------------
# WireGuard Configuration (Optional)
# -------------------------------------------------------------------------

# Path to the WireGuard state file for persistence.
#
# Example: /data/wireguard/state.yaml
# Flag:    --wireguard-state-file
# Env var: METADATA_SERVICE_WIREGUARD_STATE_FILE
#wireguard_state_file: "/data/wireguard/state.yaml"

# When true, the service rejects all requests that do not originate from the
# configured WireGuard CIDR.  Set to false to allow regular network access.
#
# Flag:    --wireguard-only
# Env var: METADATA_SERVICE_WIREGUARD_ONLY
wireguard_only: false

# -------------------------------------------------------------------------
# TokenSmith Configuration (Optional - for dynamic authentication)
# -------------------------------------------------------------------------

# Base URL of the TokenSmith service.
# Supports both mTLS and bootstrap‑token authentication modes.
#
# Default: ""
# Example: https://tokensmith.example.com
# Flag:    --tokensmith-url
# Env var: METADATA_SERVICE_TOKENSMITH_URL
#          TOKENSMITH_URL
tokensmith_url: "http://tokensmith:8080"

# Bootstrap token used when the service cannot present a client certificate.
# Required only if `tokensmith_url` is set and mTLS is not used.
#
# Flag:    --tokensmith-bootstrap-token
# Env var: METADATA_SERVICE_TOKENSMITH_BOOTSTRAP_TOKEN
#          TOKENSMITH_BOOTSTRAP_TOKEN
#tokensmith_bootstrap_token: ""

# Path to the client certificate (PEM) for mTLS authentication with TokenSmith.
#
# Flag:    --tokensmith-service-identity-cert
# Env var: METADATA_SERVICE_TOKENSMITH_SERVICE_IDENTITY_CERT
#          TOKENSMITH_SERVICE_IDENTITY_CERT
#tokensmith_service_identity_cert: ""

# Path to the client private key (PEM) for mTLS authentication with TokenSmith.
#
# Flag:    --tokensmith-service-identity-key
# Env var: METADATA_SERVICE_TOKENSMITH_SERVICE_IDENTITY_KEY
#          TOKENSMITH_SERVICE_IDENTITY_KEY
#tokensmith_service_identity_key: ""

# Optional CA certificate (PEM) used to verify the TokenSmith server's
# TLS certificate.  If omitted the system's default CA pool is used.
#
# Flag:    --tokensmith-service-identity-ca
# Env var: METADATA_SERVICE_TOKENSMITH_SERVICE_IDENTITY_CA
#          TOKENSMITH_SERVICE_IDENTITY_CA
#tokensmith_service_identity_ca: ""

# Target downstream service name for which TokenSmith should issue a token.
#
# Default: "smd"
# Flag:    --tokensmith-target-service
# Env var: METADATA_SERVICE_TOKENSMITH_TARGET_SERVICE
#          TOKENSMITH_TARGET_SERVICE
tokensmith_target_service: "smd"

# Diagnostic hint: a comma‑separated list of scopes that the bootstrap token
# is allowed to request.  Not enforced by the service.
#
# Example: "metadata:read,groups:read"
# Flag:    --tokensmith-bootstrap-policy-scopes-hint
# Env var: METADATA_SERVICE_TOKENSMITH_BOOTSTRAP_POLICY_SCOPES_HINT
#          TOKENSMITH_BOOTSTRAP_POLICY_SCOPES_HINT
#tokensmith_bootstrap_policy_scopes_hint: ""

# Number of seconds before a token's expiry when the service should attempt a
# refresh.  This avoids edge‑case failures when the token is about to expire.
#
# Flag:    --tokensmith-refresh-skew-sec
# Env var: METADATA_SERVICE_TOKENSMITH_REFRESH_SKEW_SEC
#tokensmith_refresh_skew_sec: 300

# -------------------------------------------------------------------------
# SMD Integration Configuration
# -------------------------------------------------------------------------

# Enable the background worker that synchronizes node/group membership from
# the System Management Database (SMD). Set to false to disable the cache.
#
# Flag:    --smd-sync-enabled
# Env var: METADATA_SERVICE_SMD_SYNC_ENABLED
smd_sync_enabled: true

# Interval, in seconds, at which the SMD cache is refreshed.
#
# Flag:    --smd-sync-interval
# Env var: METADATA_SERVICE_SMD_SYNC_INTERVAL
smd_sync_interval: 60

# -------------------------------------------------------------------------
# Additional (Non‑Config‑File) Flags
# -------------------------------------------------------------------------
# The following flags are persistent (global) and are NOT represented in the
# YAML configuration, but they can still be used on the command line or via
# environment variables:
#
#   --mock-smd            (env: METADATA_SERVICE_MOCK_SMD)
#        Use built‑in mock SMD data instead of contacting a real SMD service.
#
#   --config <path>       (env: METADATA_SERVICE_CONFIG)
#        Path to an alternative configuration file.
#
#   --debug               (already listed above)
#
#   --help, --version, etc. are standard Cobra flags.

# =============================================================================
# End of example configuration.
# =============================================================================

[seantronsen]

Initial review of just the config file.

Broadly looks good, but there might be old config options present.

A format proposal I have is to organize the default values and corresponding flag/env vars in the comments.

Here's the config file example I had in mind:

Your version is superior. The only mod I would like to include from mine is the header which mentions precedence and the locations of the drop down files etc.

I'll make the changes and add you as an author to the relevant commit.

[seantronsen]

@synackd see config file header when you get a moment.

[synackd]

@synackd see config file header when you get a moment.

LGTM 👍

[synackd]

We should probably update the docs so the user knows where to find the example config.

[seantronsen]

We should probably update the docs so the user knows where to find the example config.

I added a small blurb to the deployment page, but the docs overall need a lot of updates and it may be best to handle that in a separate PR.

[synackd]

🪨