Skip to content

AGENTS.md

Latest commit

 

History

History
113 lines (82 loc) · 4.13 KB

AGENTS.md

File metadata and controls

113 lines (82 loc) · 4.13 KB

Data Fair Processings — Agent Guidelines

Project Overview

@data-fair/processings is a sister service of data-fair. It runs scheduled and on-demand processings (NPM-installable plugins) that ingest, transform and publish datasets into a running data-fair instance. The project is a monorepo with four workspaces:

  • api/ — Express REST API
  • worker/ — Polling worker that runs the processings tasks in child processes
  • ui/ — Vue 3 + Vuetify SPA (built with Vite)
  • shared/ — TypeScript types and helpers shared between api, worker and ui

Dev environment

The dev environment is managed by zellij (terminal multiplexer) and docker compose. Never start, stop, or restart dev processes yourself — the user manages them through zellij panes (npm run dev-zellij).

The dev environment also serves as the test environment: the Playwright test suite hits the running dev API, dev worker and dev UI. There is no separate test server.

Each git worktree gets its own .env (random ports + <branch>.localhost subdomain) so multiple worktrees coexist without port collisions. Use the worktree scripts:

./dev/worktree.sh feat-xyz       # create ../processings_feat-xyz with a fresh .env
./dev/delete-worktree.sh feat-xyz # tear down docker compose and remove the worktree

Checking status

bash dev/status.sh

Shows the health of all services (nginx, dev API, dev UI, dev worker, simple-directory, data-fair upstream, events, openapi-viewer, mongo, elasticsearch) and lists log files with sizes and timestamps.

Log files

All dev processes write to dev/logs/:

  • dev-api.log — API server
  • dev-worker.log — worker process
  • dev-ui.log — UI dev server (Vite)
  • docker-compose.log — all docker compose services

Troubleshooting

  1. Run bash dev/status.sh to identify which services are down
  2. Read the relevant log file in dev/logs/ for error details
  3. Report findings to the user — do not attempt to fix infrastructure issues yourself

Port assignments

Port numbers and the DEV_HOST subdomain live in .env (gitignored, generated by dev/init-env.sh). Do not modify port assignments by hand.

Test users

Test fixtures live in dev/resources/users.json and dev/resources/organizations.json. All test users and orgs are prefixed with test_ (e.g. test_admin1@test.com, test_org1). The cleanup endpoint DELETE /api/v1/test-env only purges documents whose owner.id matches ^test_ so that interactive dev work under non-prefixed accounts (e.g. superadmin, albanm) survives test runs.

Special accounts:

  • test_superadmin@test.com / password superpasswd — superadmin
  • all other test_*@test.com — password passwd

Test-env API

When NODE_ENV=development the API exposes /api/v1/test-env:

  • DELETE / — delete all test_*-owned processings, runs and limits
  • GET /pending-tasks — list runs in triggered/scheduled/running status
  • GET /raw-processing/:id, GET /raw-run/:id — raw mongo docs
  • POST /set-env, POST /set-config — runtime overrides for testing
  • DELETE /plugins — wipe the installed plugins directory

These endpoints are used by the Playwright support helpers in tests/support/axios.ts.

Testing

npm test                          # all tests
npm run test-unit                 # unit tests only
npm run test-api                  # API tests only
npm run test-e2e                  # e2e tests only
npx playwright test path/to/file  # specific file

Tests are organized under tests/features/<topic>/<name>.{unit,api,e2e}.spec.ts. The state-setup project pings /api/v1/test-env/pending-tasks and tails the dev API and worker logs into the test reporter output.

The full test suite is long — when iterating on changes always run only the related test cases. The full suite runs on git push via husky.

Linting & Type Checking

npm run lint             # ESLint (root + ui workspace)
npm run lint-fix         # auto-fix
npm run check-types      # TypeScript

Building

npm run build-types      # generate type definitions for shared schemas
npm -w ui run build      # build the Vue SPA