Feature/persist findings to db#15
Conversation
|
hey @ionfwsrijan u can review and merge the PR , if all are okay . |
There was a problem hiding this comment.
Pull request overview
This PR adds persistence of scan job metadata and aggregated scanner findings to a SQLite database after /scan and /scan-url complete, aligning with the project’s goal of building a durable dataset for later analysis/training.
Changes:
- Adds
backend/app/db.pywithinit_db()(schema creation) andget_db()connection helper. - Initializes the DB on FastAPI startup and attempts to write
jobs+findingsrows after each scan completes. - Documents the SQLite schema in
backend/README.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
backend/README.md |
Adds documentation for the jobs / findings SQLite schema and expected field meanings. |
backend/app/main.py |
Initializes DB on startup and adds post-scan persistence for /scan and /scan-url. |
backend/app/db.py |
Introduces SQLite initialization and connection helpers using aiosqlite. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
@khushboo-khatoon Still checks fail |
|
Hey @ionfwsrijan I have fixed all the failing cases. Thank you for your help and patience throughout the process ((: |
|
@khushboo-khatoon Failing checks after merging with main |
Closes #2
What this PR does
After each scan completes, findings and job metadata are now persisted to SQLite. Both
/scanand/scan-urlendpoints write to thefindingsandjobstables. Atry/exceptwrapper ensures DB write failures only log a warning and never crash the scan response.Type of change
ML tier (if applicable)
Changes
Backend
/scanendpoint to insert a row intojobsand one row per finding intofindingsafter scan completes/scan-urlendpoint with the same DB write logicidviauuid4()try/except— failure logs a warning, scan response is unaffectedNew dependencies
uuid— built-in Python module, no install neededlogging— built-in Python module, no install neededDatabase / schema changes
findingsandjobstables (created in Issue Design and initialize the SQLite schema for scan findings #1)Testing
Started server locally, uploaded a ZIP — confirmed
jobsandfindingsrows were written topatchpilot.db. Verified withsqlite3 patchpilot.db "SELECT * FROM findings LIMIT 5;".Checklist
console.erroror unhandled Python exceptions introducedrequirements.txt/package.jsonupdated if new dependencies addedAnything reviewers should focus on
The
try/exceptblock around DB writes in both/scanand/scan-url— ensures DB failure never affects API response.