research: design a privacy-safe BYOK inference plugin boundary

[Abhash-Chakraborty]

Problem

Find intentionally fails closed for remote inference today. A future BYOK option must not turn into an implicit cloud upload path or leak captions, OCR, embeddings, faces, or originals to undeclared providers.

Scope

Design an opt-in provider/plugin boundary for user-supplied inference credentials. This is a design/security issue first; implementation must wait for the architecture to be approved.

Acceptance criteria

• Define a provider manifest and capability contract (captioning, embeddings, OCR, detection) without provider-specific code in core jobs.
• Define exactly which payloads may leave the instance and require per-provider, per-capability consent.
• Store secrets encrypted or through a secret-file/keychain boundary; never return keys from APIs or logs.
• Keep local inference as the default and fail closed when a remote provider is unavailable.
• Add allowlist, timeout, rate-limit, redaction, audit-log, and key-rotation requirements.
• Document how self-hosted endpoints differ from third-party BYOK providers.
• Include a threat model and a test strategy before any adapter is merged.

Non-goals

No provider SDK, hosted API call, telemetry, or credential UI should land in this issue.

[himanshu051116]

Hi @Abhash-Chakraborty,

I would like to work on this issue under GSSoC 2026.

I have reviewed the issue requirements, CONTRIBUTING.md, AGENTS.md, the existing remote-acceleration design, agent security policy, and the repository's privacy-focused architecture.

Proposed Solution

I will keep this contribution design-focused, as required by the issue, and will not add provider SDKs, external API calls, telemetry, or credential UI.

My implementation plan is:

1. Create a focused BYOK inference plugin-boundary architecture document under docs/plans/not-started/.

2. Define a versioned provider manifest and capability contract for:

   • Captioning
   • Embeddings
   • OCR
   • Object detection

   The contract will prevent provider-specific logic from leaking into core processing jobs.

3. Define an explicit consent and data-egress matrix showing which payloads may leave the local instance for each provider and capability, including:

   • Original images
   • Derived images or thumbnails
   • Captions and OCR text
   • Embeddings
   • Metadata

4. Specify a secure credential boundary covering encrypted storage, secret files and OS keychain-compatible approaches. Keys will never be exposed through APIs, logs, errors, audit entries, or configuration responses.

5. Preserve local inference as the default and document fail-closed behavior for unavailable, misconfigured, or unauthorized remote providers.

6. Define separate trust and security requirements for:

   • User-controlled self-hosted endpoints
   • Third-party BYOK providers

7. Specify security controls for destination allowlisting, request timeouts, rate limiting, payload redaction, audit logging, key rotation and provider revocation.

8. Include a threat model covering malicious providers, SSRF, credential leakage, over-broad consent, replay, accidental data egress and provider unavailability.

9. Add a focused test strategy for manifest validation, consent enforcement, secret redaction, fail-closed behavior, capability isolation and audit events.

10. Reconcile the proposal with the existing remote-acceleration and security documentation without implementing any provider adapter before architecture approval.

I will keep the PR limited to issue #353 and target the repository's canary branch. Please assign this issue to @himanshu051116 if the proposed approach is approved.