Skip to content

arch-explore.md

Latest commit

 

History

History
258 lines (179 loc) · 18.3 KB

arch-explore.md

File metadata and controls

258 lines (179 loc) · 18.3 KB
Raw
name Explore Feed
description Public discovery surfaces for projects and journals — the in-app /bulletin_board feed and the public /api/v1/explore JSON API. Covers public_for_explore scopes, cursor pagination, Meilisearch + ActiveRecord fallback, and live-update wiring.
type project

Explore Feed

Two surfaces share the same underlying data and logic:

  1. In-app feed — embedded in /bulletin_board (Inertia). See arch-bulletin-board.md for how the feed is composed into the community hub. Search hits GET /bulletin_board/search (JSON, debounced).
  2. Public API — GET /api/v1/explore/{projects,journals} (JSON). Documented for API consumers in docs/developer/api.mdx; this page covers the internals an agent maintaining the feature needs to know.

Both surfaces are unauthenticated (no API key required). The bulletin board controller uses allow_unauthenticated_access/allow_trial_access/skip_onboarding_redirect and skips the Pundit verify callbacks (skip_after_action :verify_authorized/:verify_policy_scoped, only: %i[index search event events_feed event_ics]); the API controller uses skip_before_action :authenticate_api_key!, only: %i[projects journals] while keeping the rest of the API key-gated.

Public Visibility — public_for_explore Scopes

Every public query starts from one of two scopes — they are the trust boundary for the feed:

# app/models/project.rb
scope :public_for_explore, -> { kept.listed }

# app/models/journal_entry.rb
scope :public_for_explore, -> {
  kept.where(project_id: Project.public_for_explore.select(:id))
}

A journal entry is public iff its parent project is kept and not is_unlisted. Soft-deleting a project or flipping is_unlisted cascades to the journal feed without any explicit cleanup. Don't bypass these scopes when adding new queries — the controllers deliberately skip Pundit because these scopes ARE the policy.

Two Controllers, Shared Logic

BulletinBoardController (in-app) and Api::V1::ExploreController (public API) implement nearly identical pagination + search logic. They share these constants/conventions:

Concept Bulletin board Public API
Sort options active, newest active, newest
Default project sort active active
Journal sort always newest (no choice) always newest (no choice)
Default page limit 5 (EXPLORE_LIMIT) 20 (DEFAULT_LIMIT)
Max limit 50 (EXPLORE_LIMIT_MAX) 50 (MAX_LIMIT)
Search hard cap 1000 (MEILISEARCH_SEARCH_LIMIT) 1000 (MEILISEARCH_SEARCH_LIMIT)
Cursor format (browse) `Base64.urlsafe_encode64("<iso8601 or 'none'> ")`
Cursor format (search) `Base64.urlsafe_encode64("search ")`

The duplication is intentional today — the two controllers serialize differently (the in-app feed renders markdown excerpts and resolves cover images more aggressively for the masonry layout; the API returns simpler payloads with absolute URLs). Don't merge them blindly; if you change pagination or search, update both and verify neither cursor format broke.

Sorting

Projects, sort=active (default)

"Most recently active" — measured by the latest public journal entry's created_at. Projects with zero public journals sort last.

Implemented as a LEFT JOIN on a derived latest_activity subquery:

SELECT projects.*, latest_activity.last_activity_at AS explore_activity_at
FROM projects
LEFT JOIN (
  SELECT project_id, MAX(journal_entries.created_at) AS last_activity_at
  FROM journal_entries
  WHERE … public_for_explore conditions …
  GROUP BY project_id
) latest_activity ON latest_activity.project_id = projects.id
WHERE projects.kept AND NOT is_unlisted
ORDER BY latest_activity.last_activity_at DESC NULLS LAST, projects.id DESC

The explore_activity_at virtual attribute is read off each project row to encode the next cursor.

Projects, sort=newest

ORDER BY projects.created_at DESC, projects.id DESC.

Journals (always newest)

DISTINCT ON (project_id) to dedupe to one entry per project (the latest), then ORDER BY created_at DESC, id DESC. This means a project never appears twice in the journals feed — only its newest public entry. Consumers wanting all entries should fetch by project_id.

Cursor Pagination

Cursors are opaque base64 strings of the form <timestamp>|<id>. The id tiebreak is required because timestamps can collide.

Project cursor (mode-aware)

For sort=newest: iso8601(created_at)|id. Decode raises ArgumentError if the timestamp is missing — the API returns 400 {"error": "Invalid cursor"}.

For sort=active: iso8601(latest_activity_at)|id, OR if a project has no public journals, the literal sentinel string none|id. Encoder substitutes PROJECT_ACTIVITY_NULL_CURSOR_VALUE = "none". Decoder treats "none" as nil and dispatches to a different WHERE clause:

# Has activity timestamp
WHERE last_activity_at < :cursor_at
   OR (last_activity_at = :cursor_at AND projects.id < :cursor_id)
   OR last_activity_at IS NULL

# No activity (cursor crossed into the NULL region)
WHERE last_activity_at IS NULL AND projects.id < :cursor_id

The three-clause OR is the explicit equivalent of NULLS LAST semantics in cursor form: rows with a timestamp come first (ordered DESC), then rows with NULL (ordered by id DESC).

Journal cursor

Single mode: iso8601(created_at)|id. Same id-tiebreak pattern.

Search cursor (offset-based)

Search does paginate, but with a different cursor than browse. When a query is present the project path:

  • Fetches up to [offset + limit + 1, MEILISEARCH_SEARCH_LIMIT].min ranked IDs from Meilisearch.
  • Slices ranked_ids[offset, limit + 1] to get the page (plus the +1 lookahead for has_more).
  • Loads those projects with array_position(...) to preserve relevance order.
  • Encodes the next cursor as encode_project_search_cursor(offset + limit) → Base64("search|<offset>"). Decode raises ArgumentError (→ 400 Invalid cursor) if the prefix isn't "search", the offset is missing, or it's negative.

Journal search does not offset-paginate the search itself — it filters the scope to the Meilisearch-matched IDs, then applies the normal timestamp journal cursor over the DISTINCT ON latest-entry set.

MEILISEARCH_SEARCH_LIMIT is 1000, so there's no way to paginate past the top 1000 project search hits. Acceptable today because relevance falls off quickly.

Search: Meilisearch with ActiveRecord Fallback

Every search path uses Meilisearch first and falls back to the model's search scope (pg_search) on connection failure or API error. The method is search_project_ids in the API controller and search_projects_for_explore in the bulletin board controller (the latter takes a limit:):

def search_project_ids(query, limit:)
  project_ids         = Project.ms_search(query, filter: "is_unlisted = false", sort: ["journal_count:desc", "created_at:desc"], limit: limit).map(&:id)
  journal_project_ids = JournalEntry.ms_search(query, sort: ["created_at:desc"], limit: limit).map(&:project_id).uniq
  (project_ids + (journal_project_ids - project_ids)).uniq  # direct project hits first, then journal-only hits
rescue Meilisearch::ApiError, Meilisearch::CommunicationError, Errno::ECONNREFUSED
  # pg_search fallback — same shape, slower, no relevance scoring; scoped to public_for_explore
  project_matches = Project.public_for_explore.search(query).select(:id).limit(limit).map(&:id)
  journal_matches = JournalEntry.public_for_explore.search(query).select(:project_id).limit(limit).map(&:project_id)
  (project_matches + (journal_matches - project_matches)).uniq
end

Two-tier ranking for project search: direct matches on project name/description rank above projects only matched via their journal content. Inside each tier, Meilisearch's own score order is preserved.

The pg_search fallback is fast even without Meilisearch: GIN expression indexes (index_projects_on_search_tsvector, index_journal_entries_on_search_tsvector, index_users_on_search_tsvector, migration 20260723000000) match the exact tsvector expressions pg_search generates, so fallback queries are index-served. JournalEntry.search is content-only (an associated_against join would defeat the index); journal feed fallbacks that also need project-name matches use JournalEntry.search_including_project, which ORs in entries of projects whose name/description match — mirroring the Meilisearch index's project_name coverage. Admin users/projects search has the same rescue-to-pg_search fallback (losing only typo tolerance). If a searched column list changes in a pg_search_scope, the corresponding index expression must change with it.

The is_unlisted = false Meilisearch filter is required — without it, unlisted projects can be surfaced via journal-content matches even though they're filtered out of the public scope at SQL time. Belt-and-suspenders, but cheap.

Live Updates (ActionCable)

The in-app feed subscribes to a static stream bulletin_explore that fires when public stats might have changed:

# app/models/project.rb (also similar in journal_entry.rb)
after_commit :broadcast_bulletin_explore_update

def broadcast_bulletin_explore_update
  return unless bulletin_explore_stats_changed?           # discarded_at, is_unlisted, or new/destroyed
  return unless bulletin_explore_public_now? || bulletin_explore_public_before_last_save?
  ActionCable.server.broadcast("live_updates:bulletin_explore", { stream: "bulletin_explore", action: "update" })
end

The "now or before" check is critical: a project transitioning out of public visibility (e.g., admin sets is_unlisted: true) must still broadcast so the feed can drop it. Conversely, a private project being edited (with is_unlisted unchanged) doesn't broadcast.

Frontend (pages/bulletin_board/index.tsx) debounces incoming broadcasts at 500ms (EXPLORE_LIVE_REFRESH_DEBOUNCE_MS) — a project soft-delete cascading to its journal entries fires multiple after_commits in quick succession, and the user only needs one refresh.

The public /api/v1/explore/... API is unaffected by ActionCable — clients poll on their own cadence.

Serialization

In-app feed (rich)

serialize_project_for_explore, serialize_journal_for_explore — render markdown excerpts via Nokogiri, resolve cover images via uploaded images or markdown images, build relative hrefs for client navigation. Recording media is NOT exposed on the public explore feed — recordings are restricted to the journal author, project owner, and project collaborators.

Public API (lean)

serialize_project, serialize_journal — plain text excerpts, single cover_image_url field, absolute URLs (#{request.base_url}/...).

Cover image priority (both surfaces)

Project cards:

  1. project.unified_thumbnail (cached zine/poster, see below)
  2. The most recent public journal entry that has an attached image (in-app calls this the "cover entry")
  3. In-app only: the first markdown image extracted from the latest entry
  4. Otherwise nil

Journal cards:

  1. The entry's own attached image
  2. In-app only: the first markdown image extracted from the entry
  3. project.unified_thumbnail (so journal cards still anchor visually to the project even when the entry has no media)
  4. Otherwise nil

unified_thumbnail is a has_one_attached ActiveStorage attachment on Project populated by ComputeProjectUnifiedThumbnailJob — wraps ShipChecks::UnifiedScreenshotFinder.find_url (zine detection in repo, LLM-assisted; skip_nil: true so a "no zine" result isn't cached) + ShipChecks::UnifiedScreenshotProcessor (raster → JPEG bytes; PDF/SVG unsupported since Rails 8.1.3.1 blocks libvips's untrusted loaders). The job accepts perform(project_id, source_url:, force:, allow_representative:): a caller-supplied source_url (preflight) is used directly and skips the finder; otherwise force/allow_representative are forwarded to find_url. Cache key columns: unified_thumbnail_source_url (raw URL the rasterization is from), unified_thumbnail_etag (HTTP ETag for conditional GET), and unified_thumbnail_checked_at (last attempt timestamp). Both controllers preload via .preload(unified_thumbnail_attachment: :blob) on the project scope and { project: { unified_thumbnail_attachment: :blob } } on journal preloads — the bare with_attached_* macro also drags in variant_records and preview_image_attachment (Bullet's "AVOID eager loading" warning), which we don't use.

Refresh contract

Zines are added near ship time, not at project creation, so discovery is triggered only when a zine plausibly exists. The job is enqueued from:

  • ProjectsController#refresh_cover — the owner-triggered "Check for my zine" button on the project page (verified owners only via ProjectPolicy#refresh_cover?; force: true, allow_representative: false; Rack::Attack throttled refresh_cover/user). The frontend polls #cover_status, which reports working/found/none by comparing unified_thumbnail_checked_at to the POST time.
  • ShipPreflightJob — after preflight's checks finish, if HasZinePage passed it reuses the already-built SharedContext to find the zine URL (no second GitHub fetch) and enqueues with that source_url (allow_representative: false).
  • AttachShipUnifiedScreenshotJob after a ship's frozen_screenshot is populated (keeps the representative-image fallback).
  • Project after_commit when repo_link changes (cleared or swapped) — to purge the now-stale cover. It deliberately does NOT scan on repo add/change (a freshly linked repo has no zine yet).
  • RefreshStaleUnifiedThumbnailsJob — recurring hourly, but only projects that already have a unified_thumbnail attached (a cheap etag refresh to catch zine updates), with unified_thumbnail_checked_at IS NULL OR < 24h ago, jittered across a 30-minute window, capped at PER_RUN_LIMIT = 200 per run.

Inside the job, the freshness model is never purge without positive proof:

  • repo_link blank → purge directly.
  • Finder returns a URL → conditional GET with If-None-Match: <stored etag>. 304 bumps checked_at only. 200 re-rasterizes via UnifiedScreenshotProcessor.transcode_to_jpeg and stores the new etag. 404/410 or any error raises TransientError (no purge — retry_on re-enqueues with polynomial backoff, capped at 5 attempts).
  • Finder returns nil but an attachment exists → probe the cached source URL via the same conditional GET. 304/200 keeps or refreshes the attachment. 404 is the only path that purges. Errors raise TransientError. This is the regression guard against the prior false-purge bug where transient finder failures (SharedContext swallows all HTTP errors to nil) would purge a working attachment.

Other safety rails:

  • download_with_etag caps body size at MAX_DOWNLOAD_BYTES = 50.megabytes (via Content-Length and a defensive post-read check) so a giant file can't blow worker memory.
  • Solid Queue limits_concurrency to: 1, key: "unified_thumbnail:<project_id>" serializes per-project runs so concurrent triggers don't race on ActiveStorage.attach.
  • Kill switch: Rails.cache.write("unified_thumbnail:paused", true) makes every run a no-op without a deploy.

Backfill rake task: bin/rake fallout:backfill_unified_thumbnails (envs BATCH_SIZE, DELAY_SECONDS) enqueues every kept project with a repo_link, throttled. Safe to re-run — repeat invocations hit the 304 fast-path.

Markdown image extraction (security)

journal_markdown_image_url rejects //evil.example (protocol-relative) and any data:/javascript: URLs. Only http(s):// and same-origin paths (/, ./, ../) are returned. Without this, a hostile journal entry could inject an <img src> into the public feed that leaks viewer IPs to attacker hosts.

Edge Cases

Risk Handling
Cursor with missing/garbage timestamp Time.iso8601 raises ArgumentError; controller rescues → 400 Invalid cursor
Search returning 0 IDs Both controllers short-circuit and return an empty data: [] payload (no SQL IN () issue)
Project with no public journals in active sort Sorts last via NULLS LAST; cursor uses "none" sentinel to advance through the NULL region without ambiguity
Unlisted project surfaced via journal-content match Project.ms_search filters is_unlisted = false; final SQL uses public_for_explore regardless
Hostile markdown image in public excerpt URL allowlist (http(s) + same-origin only) before passing to <img src>
Cascade deletes flooding the cable Frontend debounces bulletin_explore at 500ms
Pundit accidentally enabled on these controllers Explicit skip_after_action :verify_authorized, only: %i[index search event events_feed event_ics] (and same for :verify_policy_scoped); the public_for_explore scopes are the trust boundary
Drift between in-app and public API The two controllers don't share helpers — when changing pagination/sort/search, audit both

Where to Touch What

Change Controllers / Files
New sort option BulletinBoardController#order_projects_for_explore + Api::V1::ExploreController#order_projects + EXPLORE_SORTS/SORTS constants in both
Change public visibility rules Project#public_for_explore and/or JournalEntry#public_for_explore (cascades everywhere)
Add a new explore category EXPLORE_CATEGORIES (bulletin board) / CATEGORIES (API) + new entries method/action + serializer
Adjust live-update sensitivity Project#bulletin_explore_stats_changed? and the parallel method on JournalEntry
Change cursor format Both controllers' encode_*_cursor / decode_*_cursor — clients hold cursors across requests, so format changes break in-flight pagination

Related