Infra: Secrets are a committed manifest of placeholders with no rotation path or external provider wired up

[Emmy123222]

Goal this serves — Protecting the credentials that control donation infrastructure

Why this matters

The Kubernetes secret manifest is committed with empty placeholder values and a comment explaining that real values should come from an external provider or be created out of band. That comment is the entire mechanism. Nothing implements the external path, nothing rotates a credential, and nothing detects a credential that has been in use far too long.

These credentials protect the database holding donation records and the accounts that submit transactions. The current arrangement depends on every operator reading a comment and doing the right thing manually.

Evidence

# k8s/secret.yaml
# the Helm chart with secrets.provider=external (External Secrets Operator).
stringData:
  POSTGRES_PASSWORD: ""
  RESEND_API_KEY: ""
  AWS_SECRET_ACCESS_KEY: ""

The Helm chart references an external provider option, so the intent exists. No External Secrets Operator resources are defined anywhere, no rotation procedure is documented, and nothing checks credential age.

The committed file contains no real credentials, which is correct — the gap is that no supported path exists for supplying them safely.

Why this is hard

Rotation without downtime needs overlap. A database password cannot be changed instantaneously across running pods, so the design must support two valid credentials during a rotation window — that dual-validity period is where most rotation designs fail.

Stellar keys are different. A key controlling an account cannot simply be replaced; rotation means moving authority on-chain, which is a different operation with its own risk.

Provider choice affects everyone. Whichever backing store is chosen must be workable for a solo maintainer and a larger deployment alike.

Local development must stay easy. A secrets architecture that makes running the project locally painful will be bypassed.

Compromise response is part of this. Knowing which credential leaked, what it could reach, and how quickly it can be revoked is as important as rotation.

Suggested approach

Implement the external provider path the chart already anticipates, with a documented local development story that does not require it.

Support rotation with an overlap window so a credential can be replaced without downtime, and document the distinct procedure for Stellar account authority.

Track credential age and surface anything overdue. Write the compromise runbook alongside the mechanism rather than after an incident.

Acceptance criteria

• An external secrets provider path is implemented end to end, not only referenced in a comment.
• Local development works without the external provider, with a documented path.
• Database credential rotation is possible without downtime, using an overlap window, and is rehearsed.
• Rotation of Stellar account authority is documented as a distinct procedure with its own risks stated.
• Credential age is tracked and anything overdue is surfaced.
• No plaintext credential is required in any committed file for any environment.
• A compromise runbook covers identifying scope, revoking and reissuing, per credential type.
• Pods consume secrets in a way that supports rotation without a manual restart, or the restart requirement is documented.

Scope

Roughly 5,000–7,000 lines, including tests.

Relevant files

• k8s/secret.yaml
• helm/greenpay/values.yaml
• k8s/backend.yaml
• docs/runbooks/

[malaysiaonelove]

Hi maintainer, I’m interested in working on this issue. I have relevant experience and have reviewed the requirements. Please assign it to me I’m confident I can deliver a high-quality solution.

[malaysiaonelove]

Hi maintainer, I’m interested in working on this issue. I have relevant experience and have reviewed the requirements. Please assign it to me I’m confident I can deliver a high-quality solution.

[Manuelshub]

Hello maintainer, I'd love to work on this, and my approach would be that I’ll implement the external secrets flow that the Helm chart already anticipates, using the External Secrets Operator to provision credentials into Kubernetes without requiring any plaintext secrets in the repository. I’ll first trace the current secret consumption and Helm configuration, then add the required resources and update the workloads so the external path works end to end.

For rotation, I’ll design the database credential flow around an overlap window so the old and new credentials can coexist during a rollout, allowing rotation without downtime. I’ll test this against a production-like environment and document the exact rotation and recovery procedure. Stellar account authority will be handled separately with a dedicated runbook covering the on-chain process, risks, and recovery considerations.

I’ll also implement credential-age tracking so overdue credentials are surfaced, and write a compromise runbook covering scope identification, revocation, reissuing, and validation for each credential type. Local development will retain a simple non-external path so developers are not forced to depend on the external provider.

Finally, I’ll add tests for secret delivery, rotation, credential-age checks, failure/recovery scenarios, and accidental plaintext exposure. I’ll exercise the complete operational flow, run the relevant CI/release checks, and provide reproducible commands and redacted evidence in the PR.

[grantfox-oss]

🦊 GrantFox — @Manuelshub has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #513)
2. Your PR will be reviewed by the Stellar-Search maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Third Campaign campaign!

@Manuelshub's PR #544 was approved and merged by @Emmy123222.

🏆 @Manuelshub: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (392 total points). Track your full progress on GrantFox.

👏 Great work, @Manuelshub! Keep contributing to Stellar-Search.