Skip to content

feat: find security variants with ephemeral SAST rules and save them as custom rules - #49

Merged
robertohuertasm-datadog merged 39 commits into
mainfrom
rob/innovation-week-K9CODESEC-7112
Oct 27, 2026
Merged

robertohuertasm-datadog merged 39 commits into
mainfrom
rob/innovation-week-K9CODESEC-7112

Conversation

@robertohuertasm-datadog

@robertohuertasm-datadog robertohuertasm-datadog commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds a workflow that takes an uncommitted security fix, writes a SAST rule for the mistake it fixes, runs that rule locally, and, once the user approves it, saves it as a Datadog custom rule.

  • datadog_ephemeral_sast_scan / ephemeral-scan: runs one caller-supplied rule with datadog-static-analyzer, without publishing it and without API keys.
  • datadog_promote_sast_rule / promote-sast-rule: saves an approved rule, its examples and description through the existing custom-rule API (default ruleset security-variants).
  • dd-codesec-find-security-variants skill: drives the workflow, with a confirmation pause before searching and before saving.

Jira: K9CODESEC-8034 (MCP side), umbrella K9CODESEC-7800. Analyzer side: K9CODESEC-8033.

Production hardening (last commit)

  • The ephemeral scan passes --rules-with-config: the analyzer runs only the supplied rule but keeps the repository config's ignored paths, only paths, file size limit and gitignore setting. Support is detected from --help (development builds report the last release's version); older analyzers get an upgrade error with install commands.
  • paths must stay inside directory; the scan has a 10-minute timeout; findings are kept when the analyzer exits non-zero.
  • Saving refuses to publish over an existing rule unless overwrite / --overwrite is set after the user agrees, and warns if Datadog returns no revision id.
  • Telemetry: findings_count counts violations instead of files, and the new errors map to curated, path-free messages. docs/TELEMETRY.md lists both operations.
  • README, CLAUDE.md and AGENTS.md document both tools, the CLI commands and DD_AUTH_DOMAIN.

Reviewer notes

  • E2E Test 10 uses only public fixtures in testdata/ephemeral/ and no credentials. It checks that only the ephemeral rule reports and that generated/ (ignored by the fixture config, not by .gitignore) stays ignored. It requires datadog-static-analyzer 0.9.10 or later (the first release with --rules-with-config) and fails on older analyzers.
  • testdata/ephemeral/src/handler.go intentionally contains an exec.Command call so the test can prove go-security rules from the config do not run; SAST flags it.
  • A pre-existing issue found while testing, dd-auth keys not refreshing in long MCP sessions, is fixed separately in K9CODESEC-8183.

Testing

  • go test -race ./..., golangci-lint (0 issues), gofmt.
  • ./scripts/test-e2e.sh --ci with a locally built analyzer from the K9CODESEC-8033 branch: 11 passed, 0 failed. With analyzer 0.9.8: the upgrade error is returned as expected.
  • Saving against the real API: create (revision 2787), refusal without overwrite, overwrite (revision 2788, rule_existed: true).

Release order

  • Release datadog-static-analyzer with --rules-with-config (K9CODESEC-8033).
  • Add the minimum analyzer version to the upgrade error and the README, and make E2E Test 10 fail instead of warn when the flag is missing.
  • Release the MCP (the ephemeral scan returns the upgrade error to everyone until the analyzer release is out).
  • Merge the public docs update with the minimum analyzer version.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved publication-confirmation, filesystem-safety, and configuration-selection issues affect safe operation.

4 open issues
What changed in this PR

Adds a security-variant workflow to the local Code Security MCP server, turning an uncommitted fix into a temporary SAST rule that users can approve for publication.

Changes:

  • Adds ephemeral scanning and custom-rule publishing through CLI commands and MCP tools.
  • Ships a guided variant-search skill with confirmation steps.
  • Adds validation, telemetry, documentation, and automated coverage.
File Description
testdata/​ephemeral/​static-analysis.datadog.yml Configures rule-isolation and ignored-path checks.
testdata/​ephemeral/​src/​handler.go Provides scanner fixture cases.
testdata/​ephemeral/​rule.js Defines the fixture rule.
testdata/​ephemeral/​query.scm Matches fixture calls.
testdata/​ephemeral/​generated/​ignored.go Tests ignored-path handling.
skills/​embed_test.go Checks the new shipped skill.
skills/​dd-codesec-find-security-variants/​SKILL.md Defines the variant-search workflow.
skills/​dd-codesec-find-security-variants/​references/​save-rule.md Documents publishing and project configuration.
skills/​dd-codesec-find-security-variants/​references/​rule-contract.md Documents rule authoring.
scripts/​test-e2e.sh Exercises ephemeral scans and tool registration.
README.md Introduces tools, commands, and requirements.
internal/​telemetry/​event.go Categorizes new operation errors.
internal/​telemetry/​event_test.go Tests error categorization.
internal/​promote/​client.go Implements custom-rule publishing.
internal/​promote/​client_test.go Tests publishing and confirmation safeguards.
internal/​ephemeral/​run.go Runs temporary rules with target validation.
internal/​ephemeral/​run_test.go Tests scanner execution and targets.
internal/​ephemeral/​ruleset.go Builds temporary ruleset payloads.
internal/​ephemeral/​ruleset_test.go Tests payload construction and validation.
internal/​constants/​constants.go Adds publishing authentication messages.
internal/​binary/​manager.go Detects required analyzer capabilities.
internal/​binary/​manager_test.go Tests capability detection and upgrade errors.
docs/​TELEMETRY.md Documents new operations and privacy exclusions.
cmd/​datadog-code-security-mcp/​start.go Registers both MCP tools.
cmd/​datadog-code-security-mcp/​setup_test.go Updates installed-skill expectations.
cmd/​datadog-code-security-mcp/​promote.go Adds publishing command and handler.
cmd/​datadog-code-security-mcp/​promote_auth_test.go Tests publishing authentication.
cmd/​datadog-code-security-mcp/​promote_args_test.go Tests publishing argument parsing.
cmd/​datadog-code-security-mcp/​main.go Registers new CLI commands.
cmd/​datadog-code-security-mcp/​ephemeral.go Adds ephemeral command and handler.
cmd/​datadog-code-security-mcp/​ephemeral_test.go Tests arguments and violation counting.
CLAUDE.md Documents new tools and skill.
AGENTS.md Updates skill and testing guidance.
.github/​workflows/​e2e.yml Collects ephemeral scan diagnostics.

Review effort: Balanced

Comment thread internal/ephemeral/run.go Resolved
Comment thread internal/promote/client.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Rule-execution failures can appear as successful scans, and input handling still needs corrections.

No open issues 🎉

4 previously missed issues

In code that hasn't changed since last review

Medium severity Use normalized authentication configuration for client construction

cmd/​datadog-code-security-mcp/​promote.go:265

Use the normalized auth configuration rather than reading DD_SITE again. With environment-provided keys, setAuthCredentials returns without applying the normalized config. Thus DD_SITE=" datadoghq.com " passes MCP startup validation but promotion constructs an invalid host; the CLI path applies the normalized site and succeeds. Construct the client from auth.LoadConfig() so both interfaces use the validated values.

Medium severity Surface rule execution errors when findings are empty

internal/​ephemeral/​run.go:95

A generated visitor that throws on every match can produce [] without a failing process exit. The analyzer filters out results with no violations even when they contain execution errors (see the analyzer's result filtering). Accepting that JSON here reports a broken rule as a successful zero-finding scan. Surface and check rule-execution diagnostics before reporting success, and add a throwing-visitor regression case. Preserve partial findings separately from execution status.

Medium severity Reject explicit path lists that resolve to no targets

internal/​ephemeral/​run.go:175

An explicit scope containing only blank entries, such as paths: [" "] or --path "", returns no targets without an error. analyzerArgs then omits every -u, unexpectedly scanning the entire directory. Reject a supplied path list that resolves to no targets rather than silently widening the scan, and add a regression case for an all-blank list.

Medium severity Count Unicode characters instead of UTF-8 bytes

internal/​promote/​client.go:236

len(short) counts UTF-8 bytes rather than the documented 60 characters. A description containing 31 é characters is therefore rejected before any API call. Count runes instead, and cover a non-ASCII description below the limit in a regression test.

4 resolved since last review

Review effort: Balanced

robertohuertasm-datadog and others added 21 commits October 22, 2026 17:27
Co-authored-by: Cursor <cursoragent@cursor.com>
…rity fix

The agent can discover it from a natural-language question, and the generated rule stays in the system temp directory instead of the repository.

Co-authored-by: Cursor <cursoragent@cursor.com>
…rule API

The tool uploads the exact source after the user confirms. The skill names the IDE refresh only when the session is Cursor or VS Code.

Co-authored-by: Cursor <cursoragent@cursor.com>
The promote call was reading the environment only, so a Datadog user configured with DD_AUTH_DOMAIN could not save a rule.

Co-authored-by: Cursor <cursoragent@cursor.com>
The custom-rule detail page decodes creation_message as base64 and treats a decode error as a missing rule.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ving

A stop after the scanner list waited for a free-text ok, and the flow died.

Co-authored-by: Cursor <cursoragent@cursor.com>
The invariant check and the config offer were still free text, so the flow waited. Refresh is a fallback when the rule is not visible yet.

Co-authored-by: Cursor <cursoragent@cursor.com>
A chat can drop text written before a question card in the same turn, and the model often keeps that text in its reasoning. Pauses that need reading now end the turn with plain markdown and a direct question. A no to the extra search still leads to the report and the save question. The fix covers arrow functions and methods, the save question names the rule, and a saved run ends with the list of places to fix. Save and config steps move to references/.

Co-authored-by: Cursor <cursoragent@cursor.com>
The id sent in the create request was returned as the revision id, but Datadog stores its own. Read data.id from the response, fall back to the rule's last_revision, and return an empty id rather than the request id if both fail. The skill now shows that id.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ick fix

The IDE kept an older revision of a rule that already existed, and the refresh brought the fix back.

Co-authored-by: Cursor <cursoragent@cursor.com>
- Pass --rules-with-config so the analyzer runs only the supplied rule while
  keeping the repository config's ignored paths, only paths, file size and
  gitignore settings. Detect support from --help and ask for an upgrade
  when the analyzer lacks it.
- Keep scan paths inside the scan directory, bound the scan with a timeout
  and keep findings when the analyzer exits non-zero.
- Refuse to overwrite an existing custom rule unless overwrite is set after
  the user agrees, and warn when Datadog returns no revision id.
- Rename the default ruleset to security-variants.
- Count violations, not files, in ephemeral telemetry and classify the new
  errors with curated messages.
- Add a public E2E fixture for the ephemeral scan and document both tools.

Refs: K9CODESEC-8034, K9CODESEC-7800
Co-authored-by: Cursor <cursoragent@cursor.com>
…ectory

Co-authored-by: Cursor <cursoragent@cursor.com>
…ence check

Co-authored-by: Cursor <cursoragent@cursor.com>
… effective config

Co-authored-by: Cursor <cursoragent@cursor.com>
The analyzer calls visit, beforeAll, and afterAll as plain functions in
strict mode, so this is undefined. The guard called them as methods, so a
rule storing state on this passed locally and threw once published.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved correctness and privacy issues remain, and analyzer execution parity and live publication safeguards need final human review.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Mark outside-directory path errors as verbatim

internal/​telemetry/​event.go:162

The outside-directory error includes the supplied path, but this classification leaves verbatim false. decorateMessage therefore treats any exit status followed by digits in that path as a process exit code. For example, paths: ["../exit status 4111111111111111"] produces error.message = "path outside scan directory (exit status 4111111111111111)", exposing filename content despite the path-free telemetry contract. Set verbatim: true for this validation error and add a regression with an exit-status-like string in the rejected path.

1 resolved since last review

Review effort: Balanced

Comment thread internal/ephemeral/guard.go Resolved
The guard ran the rule's top-level code outside any try/catch, so a
throwing initializer stopped the script and the scan looked clean. It now
catches that too, and visit reports the exception once per file.
The ephemeral scan now runs the user's rule unchanged and passes
--debug yes. The analyzer prints a line for each file where the rule
throws, does not compile, or times out, and the scan reports those files
as an error. The previous try/catch wrapper changed how the rule ran and
missed syntax errors and timeouts.

The analyzer output is streamed: only the failure lines and the last
2000 bytes are kept.
… new errors

- promote: check confirmation, slug, and examples before fetching credentials
- promote: explain the empty rule left behind when the revision save fails
- promote: accept only ASCII slugs
- ephemeral: report cancellation and timeout as such; compare resolved paths
- telemetry: curated messages for rule input, invalid JSON, and probe errors
- docs: AGENTS.md tree, analyzer default skips in the variants skill
- CheckExamples reads the analyzer's 'Apply rule' debug lines and reports an
  example it never scanned (wrong extension, generated, or minified) instead
  of counting it as 0
- Example filenames with surrounding spaces are rejected, not trimmed
- ScannerError carries the analyzer log; telemetry matches it by type so the
  caller's rule or query text cannot change the classification
- Tool and CLI descriptions mention --rules-with-config and file paths

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Caller-controlled data can still reach telemetry, overlapping targets can duplicate findings, and a new test assumes a POSIX shell.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Deduplicate overlapping analyzer targets

internal/​ephemeral/​run.go:214

Overlapping targets are forwarded separately: paths: ["src", "src/handlers"], or the same target supplied relatively and absolutely, produces multiple -u arguments covering the same files. Analyzer 0.9.10's get_files walks each target independently and appends files without deduplication, so overlapping files can produce duplicate findings and inflate findings_count. Collapse equal targets and targets covered by an ancestor before invoking the analyzer. Update the test that expects src twice, and cover nested targets in both input orders.

1 resolved since last review

Review effort: Balanced

Comment thread internal/telemetry/event.go Resolved
Comment thread internal/telemetry/event_test.go Resolved
- decorateMessage reads the code from *exec.ExitError, not from error text,
  so a rejected path or rule name cannot put digits into telemetry
- SBOM results keep the original error (not serialized) so SCA failures
  still report their exit code
- Rule and ruleset name errors are typed and classified as invalid arguments
- Tests create real exit codes with the test binary, so they run on Windows
Validation, API, and scanner-output errors now carry typed reasons, so
caller paths, rule names, and API response bodies cannot steer the
telemetry kind. API errors are classified by status; bodies are capped.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Caller-supplied rule execution and remote publication require final human review of analyzer compatibility and safeguards after the outstanding corrections.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Low severity Preserve imports and declarations during rule extraction

skills/​dd-codesec-find-security-variants/​references/​save-rule.md:14

Extracting only the function can remove context required by the generated rule. Step 3 supports import edits and class methods, but a query that captures an import alongside the function cannot match an example without that import. CheckExamples scans the supplied text unchanged, so promotion can reject a valid rule with expected 1, got 0 and send the agent back to rewriting it. Preserve the necessary imports and enclosing declarations in both examples.

2 resolved since last review

Review effort: Balanced

Comment thread internal/ephemeral/ruleset.go Resolved
Comment thread internal/telemetry/event.go Resolved
- Match the analyzer's name for a rule already prefixed with its ruleset
- Reject control characters in rule and ruleset names
- Join failure lines split by a newline in a filename; an unfinished one
  still counts as a failure
- Use the validated DD_SITE for the custom rule client
- Reject MCP paths that are not an array of strings
- Bound the wait for analyzer output with WaitDelay; test cancellation
  during the scan
- Test that a real SBOM exit code reaches telemetry
- E2E: count the failing-rule checks once, guard jq, keep their outputs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Caller-supplied rule execution, diagnostic parsing, and shared-rule publishing require final human integration review alongside the requested corrections.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Deduplicate overlapping analyzer targets

internal/​ephemeral/​run.go:221

Equivalent or overlapping targets are passed to the analyzer more than once. For example, paths: ["src", "./src"] becomes two -u src arguments. Analyzer 0.9.10 walks each target separately without deduplicating its file list, so matching files are scanned twice, duplicate findings are returned, and findings_count is inflated. Remove equivalent targets and descendants already covered by another target before invoking the analyzer. Update TestScanPassesRulesWithConfig, which currently expects duplicate arguments, and cover ancestor/descendant targets in both orders.

Low severity Preserve valid context in rule examples

skills/​dd-codesec-find-security-variants/​references/​save-rule.md:14

Function-only examples can reject a working rule. Step 3 supports class methods, assigned arrow functions, and import edits, but CheckExamples writes the supplied text verbatim. Removing an enclosing class, variable declaration, or relevant import can make a query stop matching or its visitor fail. The pre-save check then blocks publication and tells the agent to rewrite a valid rule. Keep the minimal valid surrounding syntax and context needed by the query, visitor, and fix in both examples, with only the security fix differing.

2 resolved since last review

Review effort: Balanced

- Ephemeral scans pass each target once and skip targets inside another,
  so the analyzer does not report the same finding twice
- save-rule.md keeps the enclosing class, assignment, and imports the rule
  needs in both examples
A validation error leaves an empty rule, and saving the same request again fails the same way. Lead with the server's detail, and reserve the overwrite retry for failures that can succeed on another try.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Analyzer diagnostic parsing and shared custom-rule publication warrant final human validation of the scanner and API integration contracts.

No open issues 🎉

Review effort: Balanced

@whitemerch whitemerch left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing worth noting, the code is clean

@robertohuertasm-datadog
robertohuertasm-datadog merged commit 8b3cc81 into main Oct 27, 2026
14 checks passed
@robertohuertasm-datadog
robertohuertasm-datadog deleted the rob/innovation-week-K9CODESEC-7112 branch October 27, 2026 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants