-
Notifications
You must be signed in to change notification settings - Fork 0
Home
FRC Team 190
Scouting Website
Developer Wiki
The 190 Scouting Website is a full-stack web application built by FRC Team 190 (Gompei and the Herd) to collect, store, visualize, and analyze scouting data during FRC competitions. It replaces google sheets with a live, networked tool accessible from any device.
Key capabilities include:
- Aggregating quantitative match data from a Microsoft SQL Server database populated by the team's scouting app
- Displaying per-team statistics with configurable charts (bar, line, pie, scatter, radar) and colorblind-accessible heat maps
- Side-by-side match previews for upcoming qualification matches
- Qualitative scouting entry with a canvas-drawn auto-path tool
- Pit scouting forms with image capture and compression
- Drag-and-drop pick lists with OPR/EPA auto-fill and exportable alliance selections
- Live integration with The Blue Alliance (TBA) API and Statbotics for OPR, EPA, rankings, and match schedules
- Offline resilience via IndexedDB caching on the frontend and JSON file caching on the backend
- Auto-deployment script that polls GitHub every 10 seconds and restarts both servers on new commits
| Layer | Technology |
|---|---|
| Frontend framework | Svelte 5 (SvelteKit-free; uses @mateothegreat/svelte5-router) |
| Frontend build | Vite 7 |
| Charts | ECharts 6 |
| Data grid | AG Grid Community |
| Backend | Express 5 (Node.js) |
| Primary database | Microsoft SQL Server (mssql 12) |
| JSON cache store | Filesystem (backend/data/*.json) |
| Frontend offline cache | Browser IndexedDB (scoutingDB) |
| External data | The Blue Alliance API v3, Statbotics API v3 |
| Session management | express-session |
| Testing (backend) | Jest + Supertest |
| Testing (frontend) | Vitest + @testing-library/svelte |
| Path | Purpose |
|---|---|
| backend/ | Express server, database access, external API client |
| backend/index.js | All Express routes and middleware |
| backend/database.js | SQL Server connection and query logic |
| backend/externalApi.js | TBA / Statbotics fetch wrappers with JSON file caching |
| backend/data/ | JSON cache files (matches, teams, oprs, epas, alliances, etc.) |
| backend/_tests_/ | Jest tests for database, externalApi, and routes |
| frontend/ | Svelte SPA |
| frontend/src/App.svelte | Root component; defines client-side routes and registers the service worker |
| frontend/src/components/ | Shared UI: Navbar, Team, Teamgrid, Eventgrid, teamHoverCard |
| frontend/src/pages/ | Home page and all data page components |
| frontend/src/pages/datapages/ | 11 feature pages (see Section 5) |
| frontend/src/pages/graphcode/ | ECharts wrapper modules (bar, line, pie, scatter, radar) |
| frontend/src/utils/api.js | All frontend API calls; IndexedDB-backed caching for TBA data |
| frontend/src/utils/indexedDB.js | openDB, get/set/clear IndexedDB store helpers |
| frontend/src/utils/pageUtils.js | Shared constants, math helpers, color-mode logic, metric metadata |
| frontend/src/stores/ | Svelte writable stores (eventCode, sidebarState) |
| frontend/public/service_worker.js | Minimal SW; caches the app shell for offline use |
| auto-run.js | Production: polls GitHub every 10 s, restarts servers on new commits |
| auto-run-dev.js | Development variant of auto-run.js |
| run-dev.sh / run-dev.bat | One-command local startup scripts |
| .env | Shared environment variables (see Section 3) |
All configuration is loaded from a single .env file at the repository root. Both backend and frontend read it (frontend via Vite's import.meta.env, backend via dotenv).
| Variable | Description |
|---|---|
| VITE_BACKEND_PORT | Port the Express server listens on (default: 8000) |
| VITE_FRONTEND_PORT | Port Vite dev server listens on (default: 5173) |
| VITE_SERVER_IP | LAN IP of the host machine; used in production CORS origins and API base URL |
| VITE_TESTING | 1 = development (uses localhost); 0 = production (uses VITE_SERVER_IP) |
| VITE_AUTH_KEY | The Blue Alliance API read key (X-TBA-Auth-Key header) |
| SESSION_SECRET | Random string for signing express-session cookies |
| DB_USER | SQL Server login username |
| DB_PASSWORD | SQL Server login password |
The backend connects to SQL Server on the host defined by VITE_SERVER_IP, port 49172. Each FRC event is stored as a separate SQL Server database whose name is the TBA event code (e.g. 2025nhalt1).
index.js is the single Express entry point. It registers all middleware (JSON body parser, CORS, session) and defines every HTTP route. There are no sub-routers.
On startup: when POST /api/postEventCode is called with a new event code, the server immediately triggers externalAPI.populateEventData(eventCode) to warm the JSON caches. A setInterval fires populateEventData every 5 minutes thereafter to keep live data (rankings, match results) fresh.
| Route | Description |
|---|---|
| /api/getEvents | Returns all non-system SQL Server databases as event list |
| /api/getAvailableTeams?eventCode= | Returns distinct team numbers from the Activities table for the event |
| /api/getAllData?eventCode=&lastId= | Returns aggregated match rows; supports incremental fetch via lastId |
| /api/getSingleMetric?eventCode= | Returns data grouped by team then by match number |
| /api/getRatings?eventCode= | Returns driver (Grace) rating data from driverRatings.json |
| /api/getHPRatings?eventCode= | Returns HP (Ananth) rating data from HPRatings.json |
| /api/getQualitativeScouting?eventCode=&localCounts= | Returns only entries newer than what the client already has |
| /api/getPitScouting?eventCode=&localTeams= | Returns pit data excluding teams the client already has; strips robot image |
| /api/getPitScoutingImage?eventCode=&teamNumber= | Returns the base64 robot picture for a specific team |
| Route | Description |
|---|---|
| /api/getMatchAlliances?eventCode= | Full match objects from TBA (qual + elim) |
| /api/getTeams?eventCode= | Team list → {_teams: {num: name}, _teamNumbers: [...]} |
| /api/getEventDetails?eventCode= | Event name, short name, location |
| /api/getTeamStatuses?eventCode= | Qualification ranking per team {teamNum: rank} |
| /api/getOPR?eventCode= | Returns {oprs, dprs, ccwms} objects |
| /api/getAlliances?eventCode= | Alliance selections + {available: bool} |
| /api/getEventEpas?eventCode= | Statbotics EPA array for all teams |
| /api/getElimsHaveStarted?eventCode= | Returns {elimsHaveStarted: bool} based on playoff match results |
| /api/getMatchScores?eventCode=&matchNumber=&driveStation= | Score for the alliance of the given drive station in a qual match |
| Route | Body fields |
|---|---|
| /api/postEventCode | { eventCode } - sets the active event and triggers cache population |
| /api/postRatings | { event, team, rating } - appends a driver rating entry |
| /api/postHPRatings | { event, team, rating } - appends a human player rating entry |
| /api/postPitScouting | { event, team, formData } - stores or overwrites pit data for a team |
| /api/postQualitativeScouting | { event, team, match, formData } - stores qual scouting for a team+match |
| /api/postGompeiMadnessBracket | { bracket } - stores the in-memory Gompei Madness bracket state |
Connects to SQL Server using the mssql package. The connection config is built from environment variables. Every public function calls sql.connect(config) before querying - mssql reuses the connection pool automatically.
This is the most complex function in the codebase. It queries all rows from [eventCode].[dbo].[Activities] and processes them into a merged dual-value format.
Record types in the Activities table:
- Match_Event - a scoring action (e.g. coral placement, climb attempt)
- EndAuto - snapshot row saved when autonomous period ends
- EndMatch - snapshot row saved at the end of the match
Processing logic:
- Two parallel accumulators run simultaneously: one for the full match (all rows) and one for auto-only (rows up to and including each team's EndAuto row)
- Numeric fields are summed across multiple scouter rows for the same team+match key, then averaged by scouter count
- AutoClimb and StartingLocation are overridden from EndAuto rows (not summed)
- Zone time fields (NearBlueZoneTime, FarBlueZoneTime, etc.) are averaged from EndMatch rows
- Match_Event rows are counted (MatchEventCount) and timestamped (MatchEventDetails with match-relative times)
- The final output merges auto and full values: each numeric metric becomes a two-element array [autoValue, fullValue]. Metadata fields (Team, Match, etc.) remain single values
The [autoValue, fullValue] format lets every chart page switch between auto-only and full-match views without re-fetching data.
All TBA and Statbotics calls go through externalApi.js. The module owns two responsibilities:
- populateEventData(eventCode) - fetches all 7 data types in parallel via Promise.allSettled and writes each to backend/data/<filename>.json keyed by event code. Individual failures do not abort the others.
- readFromCache(filename, eventCode, fallback) - reads the JSON file; if the entry is missing, empty, or the file is corrupt, it re-calls populateEventData and tries once more before returning the fallback.
The 7 cached data types and their filenames:
| File | Source / Content |
|---|---|
| matches.json | TBA /event/{code}/matches - full match objects including score breakdowns |
| teams.json | TBA /event/{code}/teams/simple - team numbers and nicknames |
| eventDetails.json | TBA /event/{code} - name, short_name, location |
| teamStatuses.json | TBA /event/{code}/teams/statuses - qual rankings |
| oprs.json | TBA /event/{code}/oprs - OPR, DPR, CCWM per team |
| alliances.json | TBA /event/{code}/alliances - playoff alliance selections |
| epas.json | Statbotics /team_events?event={code} - EPA metrics per team |
App.svelte is the root component. It uses @mateothegreat/svelte5-router to define client-side routes, mounts the persistent Navbar, and registers the service worker on load.
| Route | Component |
|---|---|
| / | Home.svelte |
| /pickLists | pickLists.svelte |
| /singleMetric | singleMetric.svelte |
| /teamView | teamView.svelte |
| /pitScouting | pitScouting.svelte |
| /gracePage | gracePage.svelte |
| /ananthPage | ananthPage.svelte |
| /marchMadness | marchMadness.svelte |
| /matchPreview | matchPreview.svelte |
| /qualPage | qualPage.svelte |
| /qualDataView | qualDataView.svelte |
The Home page is the central control hub. On load it reads the saved event code from localStorage. Its primary action is "Cache All Data", which:
- Clears all IndexedDB stores
- Fetches all quantitative scouting data from /api/getAllData and writes it to IndexedDB
- Merges local and server pit scouting data (localStorage ↔ backend), then syncs both directions
- Merges local and server qualitative scouting data the same way
- Fetches and caches OPR data in localStorage
- Saves a timestamp of the last sync
Home also provides buttons to set the active event (dropdown from /api/getEvents + a manual text input), clear all local data stores, and shows a notification banner for success/error feedback.
All HTTP communication between the frontend and backend passes through api.js. The file is split into two categories of calls:
Direct fetch calls (no caching): getEvents, getAvailableTeams, getAllData, getSingleMetric, qualitative and pit scouting endpoints, rating pages. These are used for data that is already managed locally.
Cached fetch calls (fetchWithCache): fetchTeams, fetchMatchAlliances, fetchEventDetails, fetchTeamStatuses, fetchOPR, fetchAlliances, fetchEventEpas, fetchElimsHaveStarted. These use a common helper that:
- Checks IndexedDB for existing data first
- Uses a short timeout (5 s) if cached data exists; uses 30 s if there is no fallback
- On network failure, returns IndexedDB data if available; otherwise throws
- On success, writes fresh data back to IndexedDB
fetchRobotClimb is a special function that pulls the raw TBA match object and extracts endGameTowerRobot and autoTowerRobot fields for a specific team from the score_breakdown.
A thin wrapper around the browser IndexedDB API. The database is named scoutingDB, version 2. It contains two categories of stores:
- scoutingData - stores quantitative match rows (keyPath: Id) from getAllData
- Nine key-value stores for TBA/Statbotics data: matchAlliances, teams, eventDetails, teamStatuses, OPR, alliances, EPA, elimsStarted, matchScores
Exported functions: openDB (lazy singleton), setIndexedDBStore (bulk row insert or single key-value put), getIndexedDBStore (get all rows or get by key), clearIndexedDBStore, clearAllStores, getLastId (finds max Id for incremental fetches).
pageUtils.js exports constants and helpers used across every data page:
| Export | Purpose |
|---|---|
| METRIC_DISPLAY_NAMES | Map from raw DB column names to human-readable labels |
| EXCLUDED_FIELDS | Set of fields never shown in charts/tables (metadata, zone times) |
| INVERTED_METRICS | Metrics where lower = better (e.g. TimeOfClimb); used to flip color scales |
| BOOLEAN_METRICS | Metrics treated as 0/1 flags |
| CLIMBSTATE_METRIC | Name of the categorical climb state field |
| COLOR_MODES | Object defining color palettes for normal, protanopia, deuteranopia, tritanopia, and 'alex' modes |
| ZONE_TIME_FIELDS | Set of the 6 field-zone timing metrics |
| MATCH_NUMBERS | Array [1..100] for match number dropdowns |
| mean / median / sd / percentile | Basic statistics functions |
| lerpColor | Linear interpolation between two hex colors |
| getEventCode / getColorblindMode | Reads localStorage for current event and color mode |
| loadFromStorage / saveToStorage | sessionStorage+localStorage combined read/write |
| getAnanthRatings / getGraceRatings | Returns arrays of rating image URLs |
| ROW_HEIGHT / HEADER_HEIGHT | Constants for AG Grid row sizing |
| ELIM_LEVEL_ORDER | Sort key for comp_level strings (qm < ef < qf < sf < f) |
The Navbar is a collapsible left sidebar. It uses the isSidebarOpen Svelte store (stores/sidebarState.js) to toggle between collapsed and expanded states. Navigation calls goto() from the router and immediately collapses the sidebar on mobile.
The Navbar polls checkAlliances() every 30 seconds (and on storage changes) to conditionally show the Alliances and March Madness navigation links, which only appear once alliance selection data is available from TBA. It also conditionally shows the Elims tab once playoff matches have scores.
The most feature-rich page. Displays a complete statistical profile for one team at a time, selected from a dropdown populated from IndexedDB.
Features:
- Auto/full toggle: all chart data switches between [0] (auto) and [1] (full match) values from the getAllData array format
- OPR fetch: pulls the team's OPR from the backend and displays it as a header stat
- Multiple chart types: any numeric metric can be charted as bar, line, pie, scatter, or radar; charts are dynamically added and removed
- Color-coded metric table: all numeric metrics are shown in a heat-map table using percentile-based color interpolation; supports all 5 color blind modes
- Avoidance/Defense charts: polar-style ECharts visualizations for zone time data
- Auto path canvas: replays drawn auto paths from qualitative scouting data on a field image
- Pit data section: displays pit scouting answers and robot photo
- Qualitative notes: shows all qualitative match observations per match
- Grace/Ananth ratings: shows the team's driver and human player ratings as image-based icons
- Estimated points: fetches TBA score breakdowns to compute the team's contribution to each match score
Shows one selected metric across all teams simultaneously, sorted by value. An AG Grid table provides the primary view with per-column sparkline-like inline bars. A secondary "Eventgrid" component shows the same data as colored cards with hover details. Supports the same 5 chart types and color-blind modes as Team View. Includes an OPR column fetched from TBA.
The Eventgrid component (components/Eventgrid.svelte) is a reusable team card grid used here and in Pick Lists. Each card shows the team number, nickname, and a color-coded value for the selected metric. Clicking a card shows a teamHoverCard popup.
Side-by-side comparison of all six teams in an upcoming match. The match is selected from a dropdown of all qual matches fetched from TBA. For each team it shows: OPR, current ranking, Grace/Ananth ratings, and a configurable chart of any metric. Supports chart type switching and colorblind modes.
A fully-featured alliance selection and pick list manager. The page has two views: picklists and alliance-selection.
Pick list view features:
- Create/rename/delete named pick lists
- Drag-and-drop team ordering within and between lists
- Mark teams as 'picked' (grayed out) as alliance selection progresses
- Auto-fill by OPR: creates a ranked pick list sorted by descending OPR fetched from TBA
- Auto-fill by EPA: creates a ranked pick list sorted by descending Statbotics EPA
- Export to clipboard in formatted text or JSON
- Import from exported JSON string
- hovering a team number shows the teamHoverCard popup with full stats
Alliance selection view features:
- Supports up to 8 configurable alliances, each with captain + 2 (or 3) picks
- Multiple named "alliance selection" configurations can be saved and switched between
- Auto-populate captains from TBA ranking data
- Drag teams from pick lists onto alliance slots
- Export the full alliance selection to clipboard
A mobile-friendly form for collecting pre-competition robot information. Scout selects a team from a dropdown (populated from TBA), fills in a structured form, and optionally captures or uploads a robot photo.
Image handling: photos are compressed using imageCompression.js (utils/imageCompression.js) before storage. The page stores data locally in localStorage ("retrievePit") and syncs to the backend via POST /api/postPitScouting. Images are stored as compressed base64 strings and fetched separately via /api/getPitScoutingImage to keep list responses small.
A two-phase form for match scouting. The scout enters their name, match number, alliance, and team number, then proceeds through:
- Auto phase: a canvas drawn on top of a field image allows the scout to trace the robot's autonomous path. Paths are stored as arrays of {x, y} coordinate points. Tools include draw, erase, undo, and clear.
- Teleop phase: a set of slider and text questions (e.g. Defense strength, Avoidance behavior). Sliders have labeled tick marks.
On submit, the combined data (auto path + teleop answers) is serialized and sent to POST /api/postQualitativeScouting. Data is also saved locally to localStorage ("retrieveQual") as a backup.
Displays all collected qualitative scouting data as a card grid, one card per team. Each card shows the team's pit scouting answers and one panel per match that was scouted, including the drawn auto path replayed on the field image. Teams can be hidden/shown via a filter dropdown, and cards can be reordered by drag-and-drop.
Two pages (gracePage.svelte, ananthPage.svelte) for collecting subjective driver and human-player ratings. Each team is presented with a horizontal set of image-based rating buttons. Ratings are stored via /api/postRatings and /api/postHPRatings respectively. The pages use custom meme-image rating scales unique to the team's culture.
A bracket-style elimination visualization. It polls TBA for match data and elim status, then renders the playoff bracket showing which alliances advance. The bracket state can be shared with the server via POST /api/postGompeiMadnessBracket. Only visible in the Navbar after eliminations have started.
A large overlay card (80% viewport width × 92% height, centered) that shows comprehensive stats for a hovered team. Props: team number, eventCode, teamAggCache (pre-aggregated match data), globalStats (percentile boundaries for color scaling), cachedOPRs, cachedRobotPics.
Displays: team name, OPR, all numeric metrics in a color-coded table (using the same percentile color logic as Team View), with a metric selector dropdown to pick which stat to highlight.
A scrollable grid of team number buttons. Used in Team View to let the user select which team to inspect. Highlights the selected team and grays out teams without data.
A card grid used in Single Metric. Each card shows a team's number, nickname, and color-coded metric value. Clicking a card opens the teamHoverCard.
A minimal single-team chip component used in the Pick Lists page to represent draggable team items.
The system has three layers of caching, each serving a different purpose:
| Layer | What it caches / When it is used |
|---|---|
| Backend JSON files (backend/data/) | TBA & Statbotics responses, keyed by event code. Populated on event code set and every 5 minutes. Protects against TBA rate limits and outages during competition. |
| Browser IndexedDB (scoutingDB) | TBA/Statbotics data fetched through api.js (match alliances, teams, OPR, EPA, etc.). Enables offline use after initial load. Invalidated by clearAllStores() on the Home page. |
| Browser localStorage | Event code, colorblind mode, pit scouting data, qualitative scouting data, OPR data, last sync timestamp. Persists across page refreshes and browser restarts. |
Cache invalidation strategy: the Home page's "Cache All Data" button is the primary way scouts refresh all data at the start of a day or after connectivity issues. TBA data auto-refreshes on the backend every 5 minutes. The frontend falls back to IndexedDB on any network failure, with no manual intervention required.
The scouting Android app → SQL Server Activities table → getAllData() on the backend → IndexedDB scoutingData store on the frontend → data page components read from IndexedDB directly.
getAllData supports incremental fetch via lastId: the frontend passes the maximum Id it already has, and the backend returns only newer rows. This is used by some pages to poll for live updates without re-downloading everything.
externalApi.populateEventData() → backend/data/*.json → express routes return JSON → api.js fetchWithCache() → IndexedDB stores → page components.
On the frontend, all TBA data goes through fetchWithCache. If the network call fails, the page silently falls back to IndexedDB. If both fail, an error is thrown and the page shows an error state.
Scout fills form → local localStorage save (backup) → POST to backend → backend writes to JSON file (qualitativeScoutingData.json or pitScoutingData.json). On the next "Cache All Data" run, the Home page reconciles local and backend data bidirectionally.
- Node.js 18+
- Access to a Microsoft SQL Server instance populated by the scouting app
- A TBA API read key
- A .env file at the repository root with all required variables (see Section 3)
From the repository root:
node run-dev.sh # Linux/macOS
run-dev.bat # Windows
These scripts install dependencies if needed, then start the backend (node backend/index.js) and frontend (vite dev) concurrently.
Run auto-run.js on the competition server:
node auto-run.js
This script:
- Installs dependencies if node_modules is missing
- Spawns the backend (npm run start) and frontend (npm run dev) as child processes with output piped to the console
- Polls git fetch + git log every 10 seconds to detect new commits on the main branch
- On a new commit: git pull, kill both child processes (via taskkill on Windows), wait 3 seconds, restart
cd backend && npm test # Jest; generates coverage/lcov-report/
cd frontend && npm test # Vitest
Five color modes are supported across all heat-map displays. The active mode is stored in localStorage as "colorblindMode" and read by getColorblindMode() in pageUtils.js.
| Mode key | Description |
|---|---|
| normal | Blue (low) → Yellow (mid) → Red (high) |
| protanopia | Adjusted palette avoiding red-green confusion (red-blind) |
| deuteranopia | Adjusted palette for green-blind viewers |
| tritanopia | Adjusted palette for blue-yellow blind viewers |
| alex | Monochrome percentile bands: white / light gray / dark gray / black based on quartiles |
The colorFromStats() function in pageUtils.js handles the mapping: it computes the value's quartile (p25/p50/p75) and linearly interpolates between the two surrounding band colors in the active mode. INVERTED_METRICS have their color scale flipped so that lower values appear "better" (green/warm).
Every numeric metric from getAllData is returned as a 2-element array: index 0 is the auto-period value, index 1 is the full-match value. Pages that support an auto/full toggle pass the correct index to extractValues() to flatten the data before charting.
Team numbers appear in several formats across data sources: "190", "frc190", "team190", or as integers. The pattern String(raw).replace(/\D/g, "") is used throughout to strip all non-digits before comparison or display.
The getQualitativeScouting endpoint accepts a localCounts map {teamNum: entryCount} and returns only teams/matches where the backend has more entries than the client. This minimizes bandwidth on slow pit WiFi.
All routes that require an event code use the validateEventCode middleware, which reads req.query.eventCode or req.body.event and returns 403 if missing. This prevents empty-string cache entries.
The externalApi module uses isCacheValid() before deciding whether to re-populate. An empty array [] or empty object {} is treated the same as undefined - meaning TBA returned no data (event not yet posted) and the cache should be retried.
The repository has a rich branch history reflecting the project's iterative development. Notable branches seen in the git log:
| Branch | Feature area |
|---|---|
| feature-bluealliance-refactor | Current branch - externalApi caching refactor |
| frontend-feature-teamviewapi | TBA API integration for team view |
| frontend-feature-pickLists | Pick list drag-and-drop system |
| frontend-feature-graphpage | ECharts graph integration |
| frontend-feature-singleMetric | Single metric cross-team view |
| feature-qual-page | Qualitative scouting canvas form |
| feature-database-integration | SQL Server backend integration |
| feature-improve-navbar | Collapsible sidebar redesign |
| feature-backend-indexeddb | IndexedDB offline caching layer |
| polyend-dataloading | Incremental data loading (lastId pattern) |
| backend-development / frontend-development | Long-running integration branches |