Skip to content

feat: add Postgres-backed DPU job queue#444

Open
avikalpg wants to merge 3 commits into
mainfrom
nia/dpu-postgres-queue
Open

feat: add Postgres-backed DPU job queue#444
avikalpg wants to merge 3 commits into
mainfrom
nia/dpu-postgres-queue

Conversation

@avikalpg

@avikalpg avikalpg commented May 24, 2026

Copy link
Copy Markdown
Member

Summary

  • add a Postgres-backed DPU job queue with schema bootstrap, enqueue, claim, ack, and fail helpers
  • expose authenticated DPU job APIs at /api/dpu/jobs/{claim,ack,fail}
  • keep Pub/Sub compatibility behind DPU_QUEUE_TRANSPORT=pubsub|postgres
  • point setup instructions to self-hosted GHCR DPU and remove the Vibinex Cloud setup CTA for now

Paired PRs

Verification

  • npx tsc --noEmit --pretty false
  • npm test -- --runInBand utils/db/tests/dpuJobs.test.ts using the test Supabase direct host override
  • local Next smoke on :3001: seeded a dpu_jobs row, claimed through /api/dpu/jobs/claim, acked through /api/dpu/jobs/ack, verified status=completed attempts=1

Notes

  • The configured Supabase pooler user currently returns tenant/user not found; direct host works with the same test project credentials.
  • next lint still fails on pre-existing utils/data.ts no-prototype-builtins plus repo-wide warnings; not introduced here.

Summary by CodeRabbit

  • New Features

    • Added job queue processing with claim/ack/fail flows and secure job authentication.
    • Added PostgreSQL-backed job transport option for self-hosted deployments.
  • Changes

    • Docker setup updated to use the new image and compute server URL dynamically.
    • Self-hosting is now the primary deployment option; publish path adjusted for Postgres transport.
  • Documentation

    • Hosting/setup docs updated to reflect self-hosting and simplified cloud instructions.
  • Tests

    • New Jest tests covering job queue behaviors and retries.

@vercel

vercel Bot commented May 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
team-monitor-website Ready Ready Preview, Comment Jun 5, 2026 7:22pm
vibinex-server Ready Ready Preview, Comment Jun 5, 2026 7:22pm

@coderabbitai

coderabbitai Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

Too much diff to scan? Review this PR in Change Stack to start with the highest-impact changes.

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 83096211-1f62-4aa1-8daf-33cdf5a139ec

📥 Commits

Reviewing files that changed from the base of the PR and between a0edc6c and 14b0572.

📒 Files selected for processing (4)
  • components/setup/DockerInstructions.tsx
  • components/setup/HostingSelector.tsx
  • utils/db/dpuJobs.ts
  • utils/dpuAuth.ts
🚧 Files skipped from review as they are similar to previous changes (3)
  • components/setup/DockerInstructions.tsx
  • utils/dpuAuth.ts
  • components/setup/HostingSelector.tsx

Walkthrough

Implements a PostgreSQL-backed DPU job queue (schema, operations, tests), adds DPU auth validation, three POST API routes (claim, ack, fail), routes publishMessage to the DB transport when configured, and updates setup UI/Docker instructions to default to self-hosting.

Changes

DPU Job Queue Implementation

Layer / File(s) Summary
Job queue schema, operations, and tests
utils/db/dpuJobs.ts, utils/db/__tests__/dpuJobs.test.ts
DpuJob interface and DpuJobStatus type define the queue model. DPU_JOBS_SCHEMA_SQL creates dpu_jobs with indexing. ensureDpuJobsTable() lazily initializes the schema. enqueueDpuJob() inserts jobs; claimDpuJobs() atomically selects and updates pending/expired jobs to processing with lease and attempts increment. ackDpuJob() marks processing→completed; failDpuJob() marks processing→pending (retry) or →failed (terminal). Tests validate FIFO, state transitions, ack gating, retries, and terminal failures.
DPU authentication validation
utils/dpuAuth.ts
Exports validateDpuAuth() and safeCompare() to check DPU_AUTH_TOKEN, extract bearer or x-dpu-auth-token header, and respond 500 (misconfigured) or 401 (unauthorized) as appropriate.
Job lifecycle API endpoints
pages/api/dpu/jobs/claim.ts, pages/api/dpu/jobs/ack.ts, pages/api/dpu/jobs/fail.ts
Three POST handlers: claimHandler claims jobs by installationId with capped limit and lease, returning claimed jobs; ackHandler acknowledges a job as completed; failHandler marks processing jobs as failed or re-queues them for retry. All enforce method, validate DPU auth and required fields, and map DB outcomes to 400/404/500/200 responses.
Pub/Sub transport conditional routing
utils/pubsub/pubsubClient.ts
publishMessage is now async and, when DPU_QUEUE_TRANSPORT === 'postgres', calls enqueueDpuJob(...) instead of publishing to Google Pub/Sub. createTopicNameInGcloud short-circuits topic creation for the postgres transport.
Setup UI and Docker instructions
components/setup/DockerInstructions.tsx, components/setup/HostingSelector.tsx, pages/docs/setup/hosting.tsx, pages/docs/setup/providerAppInstall.tsx
DockerInstructions uses dpuImage (ghcr.io/vibinex/dpu:latest) and computes SERVER_URL from window.location.origin (with server placeholder). The docker run template now sets DPU_QUEUE_TRANSPORT, SERVER_URL, and DPU_AUTH_TOKEN. HostingSelector removes the "Vibinex Cloud" option; hosting docs remove cloud build content and default provider install flow to "selfhosting".

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • vibinex/vibinex-server#379: Modifies components/setup/DockerInstructions.tsx and related logic for Docker command/env generation.

Suggested reviewers

  • tapishr
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'feat: add Postgres-backed DPU job queue' is fully related to the main change: adding a Postgres-backed DPU job queue system with schema, helpers, API endpoints, and Pub/Sub compatibility switching.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch nia/dpu-postgres-queue

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@avikalpg

avikalpg commented Jun 4, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review this PR again.

@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

@avikalpg Sure, I'll kick off a fresh review of this PR now!

✅ Action performed

Full review finished.


const HostingSelector: React.FC<HostingSelectorProps> = ({ selectedHosting, setSelectedHosting }) => {
const hostingOptions = [
{ value: 'cloud', label: 'Vibinex Cloud' },

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel that you should only leave this commented out, and not totally remove this option. Unless you don't intend to bring it back again.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/setup/DockerInstructions.tsx`:
- Line 21: Update the Docker instructions to use the supported transport value
by replacing the DPU_QUEUE_TRANSPORT environment value from "http" to
"postgres"; this aligns with the runtime switch in utils/pubsub/pubsubClient.ts
which only recognizes 'postgres' (otherwise it falls back to Pub/Sub) so update
the DPU_QUEUE_TRANSPORT entry in DockerInstructions.tsx to "postgres".

In `@utils/db/dpuJobs.ts`:
- Around line 6-16: The DpuJob interface declares id as string but claimDpuJobs
currently returns raw DB rows (numeric BIGSERIAL ids); fix claimDpuJobs by
mapping result.rows to a proper DpuJob shape and convert row.id to string (e.g.,
id: String(row.id)) before returning, mirroring the conversion used in
enqueueDpuJob; ensure the mapping uses the same field names (installation_id,
msg_type, payload, status, attempts, locked_until, created_at, updated_at) so
consumers always receive ids as strings.
- Around line 40-47: The schemaReady promise set in ensureDpuJobsTable can
remain rejected after conn.query(DPU_JOBS_SCHEMA_SQL) fails, causing all future
calls to await the same rejected promise; modify ensureDpuJobsTable so that when
creating schemaReady it attaches rejection handling that clears schemaReady (set
back to null) on error before re-throwing the error (e.g., use
conn.query(...).then(() => undefined).catch(err => { schemaReady = null; throw
err; })), referencing schemaReady, ensureDpuJobsTable, conn.query, and
DPU_JOBS_SCHEMA_SQL in your change.

In `@utils/dpuAuth.ts`:
- Around line 14-16: The current token check in utils/dpuAuth.ts uses direct
string comparison (providedToken !== configuredToken) which is vulnerable to
timing attacks; replace it with a constant-time comparison using Node's
crypto.timingSafeEqual by converting both tokens to Buffers, first verify both
are defined and have the same byte length (otherwise treat as non-match), and
then call crypto.timingSafeEqual to decide to send res.status(401).json({ error:
'Unauthorized' }) and return false (ensure the same error path is used for
length mismatch and failed comparison); update the logic around
providedToken/configuredToken in the authentication function to follow this safe
comparison flow.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 31c05d22-0eba-4b7d-9a2e-c395476d78a0

📥 Commits

Reviewing files that changed from the base of the PR and between 643794a and a0edc6c.

📒 Files selected for processing (11)
  • components/setup/DockerInstructions.tsx
  • components/setup/HostingSelector.tsx
  • pages/api/dpu/jobs/ack.ts
  • pages/api/dpu/jobs/claim.ts
  • pages/api/dpu/jobs/fail.ts
  • pages/docs/setup/hosting.tsx
  • pages/docs/setup/providerAppInstall.tsx
  • utils/db/__tests__/dpuJobs.test.ts
  • utils/db/dpuJobs.ts
  • utils/dpuAuth.ts
  • utils/pubsub/pubsubClient.ts
💤 Files with no reviewable changes (1)
  • components/setup/HostingSelector.tsx

Comment thread components/setup/DockerInstructions.tsx Outdated
Comment thread utils/db/dpuJobs.ts
Comment thread utils/db/dpuJobs.ts
Comment thread utils/dpuAuth.ts Outdated
- HostingSelector: comment out cloud option instead of deleting (per Avikalp)
- DockerInstructions: fix DPU_QUEUE_TRANSPORT from 'http' to 'postgres' (pubsubClient only matches 'postgres')
- dpuJobs: reset schemaReady to null on rejection so it can recover after transient DB errors
- dpuJobs: map claimDpuJobs rows to convert BIGSERIAL id to string, matching DpuJob interface
- dpuAuth: use crypto.timingSafeEqual instead of !== for timing-safe token comparison
@avikalpg

avikalpg commented Jun 5, 2026

Copy link
Copy Markdown
Member Author

All review comments addressed in the latest commit (14b0572):

Avikalp's comment — HostingSelector.tsx:

  • Cloud option is now commented out (not deleted), so it can be re-enabled when ready.

CodeRabbit — 4 issues:

  1. DPU_QUEUE_TRANSPORT=httppostgres (DockerInstructions.tsx): Fixed. The runtime switch in pubsubClient.ts only matches 'postgres'; 'http' was silently falling through to Pub/Sub.

  2. claimDpuJobs id type mismatch (dpuJobs.ts): Fixed — mapping result.rows to convert the BIGSERIAL id to string via .toString(), consistent with how enqueueDpuJob handles it.

  3. schemaReady not cleared on rejection (dpuJobs.ts): Fixed — added rejection handler that resets schemaReady = null before rethrowing, so the next call can retry after a transient DB error.

  4. Timing-safe token comparison (dpuAuth.ts): Fixed — replaced !== with crypto.timingSafeEqual via a safeCompare helper. Also handles the undefined token case cleanly in one branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant