Thank you for Raptor!
I was wondering if you have any code-coverage backed reporting and enforcement planned as part of upcoming updates?
Current pipelines, including Raptor, when tasked to audit binaries and repositories, even when relying on SAST tools which we assume would provide decent coverage, tend to ignore notable chunk of a project or binary. That seems to be partly assumptions and hallucinations by the LLM, partly the default on-demand review of code and partly poor SAST coverage (unless custom queries are tasked to be modeled and generated).
The outcome would be missing potential vulnerabilities, albeit often less critical ones.
As I've been experimenting with my own half-manual pipeline, I consistently get better/more results when I enforce the LLM to actually map the entire codebase, create a coverage map and checklist and go through all of it. Interestingly, enforcing that also improves how CodeQL results are consumed, as it would reveal if standard build and queries for CodeQL are failing or partially failing.
Thank you for Raptor!
I was wondering if you have any code-coverage backed reporting and enforcement planned as part of upcoming updates?
Current pipelines, including Raptor, when tasked to audit binaries and repositories, even when relying on SAST tools which we assume would provide decent coverage, tend to ignore notable chunk of a project or binary. That seems to be partly assumptions and hallucinations by the LLM, partly the default on-demand review of code and partly poor SAST coverage (unless custom queries are tasked to be modeled and generated).
The outcome would be missing potential vulnerabilities, albeit often less critical ones.
As I've been experimenting with my own half-manual pipeline, I consistently get better/more results when I enforce the LLM to actually map the entire codebase, create a coverage map and checklist and go through all of it. Interestingly, enforcing that also improves how CodeQL results are consumed, as it would reveal if standard build and queries for CodeQL are failing or partially failing.