Skip to content

feat(integrations): add Slack bot runtime tools#4060

Open
farizanjum wants to merge 14 commits into
Tracer-Cloud:mainfrom
farizanjum:feat/slack-bot-runtime-tools
Open

feat(integrations): add Slack bot runtime tools#4060
farizanjum wants to merge 14 commits into
Tracer-Cloud:mainfrom
farizanjum:feat/slack-bot-runtime-tools

Conversation

@farizanjum

@farizanjum farizanjum commented Jul 15, 2026

Copy link
Copy Markdown

Fixes #3230

Summary

  • Add vendor-first Railway tools to inspect the latest successful Slack bot deployment and request a confirmed redeploy.
  • Add a Slack thread replay tool with cursor pagination, bounded output, and Slack token redaction.
  • Register Railway as a CLI-backed integration, add Railway evidence source metadata, and update tool discovery, telemetry, tests, and docs.

Demo

  • Live Slack replay returned 59 messages from a test thread, confirming cursor pagination beyond Slack's first page.
  • Live Railway deployment inspection verified the Railway JSON response shape and safely reported that a test service had no successful deployment.
  • No Railway redeploy was triggered.

Testing

  • uv run python -m pytest tests/integrations/test_registry.py tests/tools/test_slack_bot_runtime_tool.py tests/tools/test_registry_index.py tests/tools/test_telemetry.py -q --tb=short --basetemp=.pytest_tmp -p no:asyncio
  • 62 focused tests passed.
  • uv run python -m ruff check config core gateway integrations platform surfaces tools tests
  • uv run python -m ruff format --check config core gateway integrations platform surfaces tools tests
  • uv run python -m mypy config core gateway integrations platform surfaces tools

Code Understanding and AI Usage

Did you use AI assistance (ChatGPT, Claude, Copilot, etc.) to write any part of this code?

  • No, I wrote all the code myself
  • Yes, I used AI assistance (continue below)

If you used AI assistance:

  • I have reviewed every single line of the AI-generated code
  • I can explain the purpose and logic of each function/component I added
  • I have tested edge cases and understand how the code handles them
  • I have modified the AI output to follow this project's coding standards and conventions

Explain your implementation approach:
I split the issue into three focused tools so read-only inspection and replay do not inherit redeploy approval requirements. Railway operations use explicit project, service, and environment selectors without writing Railway link state into the workspace. The replay tool follows Slack cursors with a bounded result, and both external boundaries return structured errors with credential redaction. The tools live under their respective vendor integrations because each tool directly calls only one vendor.

Checklist before requesting a review

  • I have added proper PR title and linked to the issue
  • I have performed a self-review of my code
  • I can explain the purpose of every function, class, and logic block I added
  • I understand why my changes work and have tested them thoroughly
  • I have considered potential edge cases and how my code handles them
  • If it is a core feature, I have added thorough tests
  • My code follows the project's style guidelines and conventions

Demo

  • Screen recording:
opensre-issue3230-demo.mp4
  • Live Slack replay returned 59 messages from a test thread, confirming cursor pagination beyond Slack's first page.
  • Live Railway inspection reported a successful Git-connected deployment with commit a8faaa54 and message feat: add Railway health demo.
  • Redeploy was exercised through the confirmation guard only; no unconfirmed deployment was triggered.

Testing

  • uv run python -m pytest tests/integrations/test_registry.py tests/integrations/test_vendor_first_layout.py tests/tools/test_slack_bot_runtime_tool.py tests/tools/test_registry_index.py tests/tools/test_telemetry.py -q --tb=short --basetemp=.pytest_tmp -p no:asyncio
  • 68 focused tests passed.
  • uv run python -m ruff check config core gateway integrations platform surfaces tools tests
  • uv run python -m ruff format --check config core gateway integrations platform surfaces tools tests
  • uv run python -m mypy config core gateway integrations platform surfaces tools

Copilot AI review requested due to automatic review settings July 15, 2026 12:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown
Contributor

Greptile code review

This repo uses Greptile for automated review. Before merge, aim for Confidence Score: 5/5 with zero unresolved review threads — see CONTRIBUTING.md.

Run a review — add a PR comment with:

@greptile review

Give it ~5-10 minutes (sometimes longer) for results, then fix feedback and re-trigger until you reach Confidence Score: 5/5.

Optional: automate with the greploop skill.

@farizanjum

Copy link
Copy Markdown
Author

@greptile review

…e-tools

# Conflicts:
#	integrations/slack/tools/__init__.py
@greptile-apps

greptile-apps Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds three new tools (inspect Railway deployment, redeploy Railway service, replay Slack thread) along with the necessary Railway integration plumbing — CLI client, config model, verifier, catalog wiring, and docs. The implementation follows established patterns throughout: subprocess isolation with a timeout, credential redaction before error surfacing, explicit confirm=true guard on the redeploy path, is_available gated on the Railway CLI binary, and cursor-safe pagination for the Slack thread client.

  • Railway tools (inspect_railway_deployment, redeploy_railway_service): CLI-backed, vendor-first layout, max(createdAt) picks the latest successful deployment, and the redeploy tool is correctly marked requires_approval=True with a confirm parameter guard.
  • Slack thread replay (replay_slack_thread_locally): paginates conversations.replies with cycle detection, bounds output to 500 messages, and redacts xox* tokens from message text using the updated pattern (now includes xoxe.xoxp-).
  • Platform / registry updates: Railway is registered in INTEGRATION_SPECS, EffectiveIntegrations, INTEGRATION_TOOL_PACKAGES, load_env_integrations, and the CLI setup handler.

Confidence Score: 5/5

Safe to merge; all new paths are additive and do not touch existing functionality.

The Railway and Slack additions are self-contained, the confirmation guard on redeploy is correctly enforced, credential redaction covers all token sources, and the cursor-pagination cycle guard works correctly. The truncated flag underreports on the last page when the cap is hit mid-page, but this only affects the metadata field — no messages are lost from the response and no downstream state is corrupted.

integrations/slack/thread_client.py — the truncated flag logic

Important Files Changed

Filename Overview
integrations/railway/client.py New Railway CLI wrapper: credential injection, subprocess execution with timeout, JSON parsing, deployment inspection with max-createdAt selection, and structured error redaction. Well-structured with proper guards.
integrations/slack/thread_client.py New Slack thread fetcher with cursor pagination and cycle detection. The truncated flag has a false-negative when the message limit is reached on the final page with no remaining cursor.
integrations/railway/tools/railway_deployment_tool/inspect_tool.py Inspect tool correctly gates on CLI availability, delegates scope resolution, and returns structured results. Read-only, no approval required.
integrations/railway/tools/railway_deployment_tool/redeploy_tool.py Redeploy tool correctly gates on explicit confirm=true before calling CLI, marks requires_approval=True, and returns structured deployment ID on success.
integrations/slack/tools/slack_thread_replay_tool/tool.py Replay tool correctly routes missing-token errors to configuration_or_delivery_error and all other errors to delivery_error. Input validation and availability gating look correct.
integrations/config_models.py Adds RailwayIntegrationConfig with proper normalizers, plus GitLabIntegrationConfig and SentryIntegrationConfig in the same hunk as preparatory consolidations.
platform/notifications/redaction.py Regex extended to cover xoxe.xoxp- token format; existing redact_token and redact_slack_token functions are unchanged.
tests/tools/test_slack_thread_replay_tool.py Comprehensive test coverage for both Railway and Slack replay tools including availability checks, confirmation guards, token classification, and null message handling.

Sequence Diagram

sequenceDiagram
    participant LLM as LLM Planner
    participant IT as InspectRailwayDeploymentTool
    participant RT as RedeployRailwayServiceTool
    participant RC as RailwayClient
    participant CLI as Railway CLI
    participant ST as ReplaySlackThreadLocallyTool
    participant TC as thread_client
    participant SA as Slack API

    LLM->>IT: run(project, service, environment)
    IT->>RC: resolve_scope_object()
    RC-->>IT: RailwayScope
    IT->>RC: inspect_deployment(scope)
    RC->>CLI: railway deployment list --limit 100 --json
    CLI-->>RC: JSON list
    RC->>RC: _deployment_record() max by createdAt
    RC-->>IT: DeploymentInfo
    IT-->>LLM: "{status:ok, deployment:{...}}"

    LLM->>RT: "run(project, service, environment, confirm=true)"
    RT->>RC: resolve_scope_object()
    RC-->>RT: RailwayScope
    RT->>RC: redeploy(scope)
    RC->>CLI: railway redeploy --yes --json
    CLI-->>RC: "{id: new-deployment-id}"
    RC-->>RT: RedeployInfo
    RT-->>LLM: "{status:ok, redeploy:{deployment_id:...}}"

    LLM->>ST: "run(thread_ref=C123/1234567890.123)"
    ST->>TC: parse_thread_ref()
    TC-->>ST: (channel, timestamp)
    ST->>TC: fetch_thread(channel, timestamp)
    loop cursor pagination up to 500 msgs
        TC->>SA: GET conversations.replies
        SA-->>TC: "{messages, response_metadata.next_cursor}"
        TC->>TC: redact_slack_token(message.text)
    end
    TC-->>ST: "{messages, truncated}"
    ST-->>LLM: "{status:ok, thread:{...}}"
Loading

Reviews (5): Last reviewed commit: "style(slack): order replay tool imports" | Re-trigger Greptile

Comment thread integrations/railway/client.py Outdated
@greptile-apps

greptile-apps Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds three new agent tools — inspect_slack_bot_runtime, redeploy_slack_bot, and replay_slack_thread_locally — along with their supporting Railway CLI client, Slack thread client, registry wiring, and tests.

  • Railway tools (inspect_tool.py, redeploy_tool.py, client.py): call the Railway CLI via subprocess to fetch the latest successful deployment or trigger a redeploy; stateless, option-injection-safe, with token redaction on error output and an approval guard on the mutating operation.
  • Slack thread replay tool (thread_client.py, tool.py): pages through conversations.replies with cursor deduplication, a 500-message cap, and per-message xox*- token redaction using the updated regex in redaction.py.
  • Registry/discovery wiring: adds integrations.railway.tools and integrations.slack.tools to INTEGRATION_TOOL_PACKAGES, registers Railway as a skip_classification integration, adds \"railway\" to EvidenceSource, and updates telemetry allowlists and tests.

Confidence Score: 4/5

Safe to merge with minor follow-ups; the core logic is correct and all 62 tests pass.

The Railway and Slack clients handle errors, timeouts, and credential redaction correctly. The redeploy tool has a proper double-confirmation guard. The main concerns are: Railway tools appear available even without the CLI (failing only at call time), the no successful deployment state is labelled as a configuration error rather than its own category, and the truncated flag conflates two different exit conditions in the pagination loop. None of these affect correctness of the happy path or security posture.

integrations/railway/client.py and the two Railway tool files — the missing is_available guard and the error_type classification are both rooted there.

Important Files Changed

Filename Overview
integrations/railway/client.py New Railway CLI wrapper: handles subprocess execution, JSON parsing, error redaction, and deployment record selection; hardcodes SUCCESS as the only accepted deployment status and limits inspection to the last 20 deployments
integrations/railway/tools/slack_bot_runtime_tool/inspect_tool.py BaseTool subclass for read-only Railway deployment inspection; no is_available guard means the tool is always shown as available even without Railway CLI installed
integrations/railway/tools/slack_bot_runtime_tool/redeploy_tool.py BaseTool subclass for Railway redeploy with requires_approval and confirm guard; same missing is_available as inspect tool; error_type for a no successful deployment scenario is misclassified
integrations/slack/thread_client.py New Slack thread fetch client with cursor pagination, 500-message cap, token redaction on all message text fields, and robust error handling
integrations/slack/tools/slack_thread_replay_tool/tool.py BaseTool wrapping thread_client; correctly gates availability on SLACK_BOT_TOKEN presence; clean error mapping
platform/notifications/redaction.py Extends Slack token regex to cover xoxe.xoxp- prefix; backward-compatible change
tools/registry_discovery.py Adds integrations.railway.tools and integrations.slack.tools to INTEGRATION_TOOL_PACKAGES discovery list; straightforward wiring
tests/tools/test_slack_bot_runtime_tool.py Comprehensive 451-line test file covering tool discovery, Railway subprocess stubbing, Slack pagination, token redaction, and confirmation guard; well-structured

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Agent
    participant InspectTool
    participant ReplayTool
    participant RedeployTool
    participant RailwayClient
    participant RailwayCLI
    participant SlackThreadClient
    participant SlackAPI

    Agent->>InspectTool: run(project, service, environment)
    InspectTool->>RailwayClient: resolve_railway_scope()
    InspectTool->>RailwayClient: inspect_deployment(scope)
    RailwayClient->>RailwayCLI: deployment list --project ... --limit 20 --json
    RailwayCLI-->>RailwayClient: JSON deployment list
    RailwayClient->>RailwayClient: _deployment_record() find SUCCESS
    RailwayClient-->>InspectTool: DeploymentInfo
    InspectTool-->>Agent: status and deployment metadata

    Agent->>ReplayTool: run(thread_ref)
    ReplayTool->>SlackThreadClient: parse_thread_ref(thread_ref)
    SlackThreadClient-->>ReplayTool: channel and timestamp
    ReplayTool->>SlackThreadClient: fetch_thread(channel, timestamp)
    loop Cursor pagination max 500 messages
        SlackThreadClient->>SlackAPI: GET conversations.replies
        SlackAPI-->>SlackThreadClient: messages and response_metadata
        SlackThreadClient->>SlackThreadClient: redact_slack_token on message text
    end
    SlackThreadClient-->>ReplayTool: messages count truncated
    ReplayTool-->>Agent: status and thread

    Agent->>RedeployTool: "run(project, service, environment, confirm=true)"
    RedeployTool->>RailwayClient: resolve_railway_scope()
    RedeployTool->>RailwayClient: redeploy(scope)
    RailwayClient->>RailwayCLI: redeploy --project ... --yes --json
    RailwayCLI-->>RailwayClient: deployment id
    RailwayClient-->>RedeployTool: RedeployInfo
    RedeployTool-->>Agent: status and redeploy metadata
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Agent
    participant InspectTool
    participant ReplayTool
    participant RedeployTool
    participant RailwayClient
    participant RailwayCLI
    participant SlackThreadClient
    participant SlackAPI

    Agent->>InspectTool: run(project, service, environment)
    InspectTool->>RailwayClient: resolve_railway_scope()
    InspectTool->>RailwayClient: inspect_deployment(scope)
    RailwayClient->>RailwayCLI: deployment list --project ... --limit 20 --json
    RailwayCLI-->>RailwayClient: JSON deployment list
    RailwayClient->>RailwayClient: _deployment_record() find SUCCESS
    RailwayClient-->>InspectTool: DeploymentInfo
    InspectTool-->>Agent: status and deployment metadata

    Agent->>ReplayTool: run(thread_ref)
    ReplayTool->>SlackThreadClient: parse_thread_ref(thread_ref)
    SlackThreadClient-->>ReplayTool: channel and timestamp
    ReplayTool->>SlackThreadClient: fetch_thread(channel, timestamp)
    loop Cursor pagination max 500 messages
        SlackThreadClient->>SlackAPI: GET conversations.replies
        SlackAPI-->>SlackThreadClient: messages and response_metadata
        SlackThreadClient->>SlackThreadClient: redact_slack_token on message text
    end
    SlackThreadClient-->>ReplayTool: messages count truncated
    ReplayTool-->>Agent: status and thread

    Agent->>RedeployTool: "run(project, service, environment, confirm=true)"
    RedeployTool->>RailwayClient: resolve_railway_scope()
    RedeployTool->>RailwayClient: redeploy(scope)
    RailwayClient->>RailwayCLI: redeploy --project ... --yes --json
    RailwayCLI-->>RailwayClient: deployment id
    RailwayClient-->>RedeployTool: RedeployInfo
    RedeployTool-->>Agent: status and redeploy metadata
Loading

Reviews (2): Last reviewed commit: "feat(integrations): add Slack bot runtim..." | Re-trigger Greptile

Comment thread integrations/railway/tools/slack_bot_runtime_tool/inspect_tool.py Outdated
Comment thread integrations/railway/client.py Outdated
Comment thread integrations/slack/thread_client.py
@farizanjum

Copy link
Copy Markdown
Author

@greptile review

@farizanjum

Copy link
Copy Markdown
Author

@greptile review

…e-tools

# Conflicts:
#	tests/integrations/test_registry.py
@muddlebee

Copy link
Copy Markdown
Collaborator

@farizanjum can you work a running demo for this? a screen recording? this was mentioned in the PR submission guidelines

@farizanjum

Copy link
Copy Markdown
Author

@muddlebee Sorry for missing the demo in the initial submission. I have resolved the current merge conflict and will attach a short, trimmed terminal recording to this PR shortly. It will show the real Railway deployment inspection, Slack thread replay, redeploy confirmation guard, and focused tests.

@farizanjum

Copy link
Copy Markdown
Author

@muddlebee I have now attached the requested demo recording to the PR description. It shows the real Railway deployment inspection with commit metadata, Slack thread replay pagination, the redeploy confirmation guard, and the focused test run. Please take another look when you have a chance.

@muddlebee

Copy link
Copy Markdown
Collaborator

thank you, I will review shortly. looks good overall

@muddlebee muddlebee left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue #3230 acceptance is mostly covered (inspect / replay / confirmed redeploy, vendor-first placement, solid tests). Blocking follow-ups: Slack token resolution should share bot_api helpers, and thread_client should harden null reactions/messages. Also tighten Railway SUCCESS search beyond the newest 20 rows and fail closed when redeploy JSON has no deployment id.

Comment thread integrations/slack/thread_client.py Outdated
"ts": item.get("ts", ""),
"reactions": [
reaction.get("name", "")
for reaction in item.get("reactions", [])

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

item.get("reactions", []) returns None when Slack sends "reactions": null, so this listcomp raises TypeError outside the httpx try/except. Same risk on payload.get("messages", []) above. Use (item.get("reactions") or []) and (payload.get("messages") or []), matching integrations/slack/bot_api.py.

Comment thread integrations/slack/thread_client.py Outdated


def fetch_thread(channel: str, timestamp: str) -> dict[str, Any]:
token = os.getenv("SLACK_BOT_TOKEN", "").strip()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only reads process env. Other Slack bot tools resolve via resolve_bot_token() (env + keyring + integration store). Wizard-configured tokens will fail here while slack_read_messages works. Prefer resolve_bot_token / bot_token_configured from integrations.slack.bot_api, and reuse its conversations.replies helper instead of a parallel httpx client.

outputs = {"thread": "Captured Slack thread messages.", "error": "Failure detail."}

def is_available(self, _sources: dict[str, dict[object, object]]) -> bool:
return bool(os.getenv("SLACK_BOT_TOKEN", "").strip())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate availability with bot_token_configured(sources) like the other Slack bot tools. os.getenv alone hides this tool when the token lives in the keyring or integration store.

Comment thread integrations/railway/client.py Outdated
"--environment",
scope.environment,
"--limit",
"20",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--limit 20 then filtering to SUCCESS can report deployment_unavailable when the only successful deploy is older than the newest 20 failed/removed rows. Raise the limit or page until a SUCCESS is found.

Comment thread integrations/railway/client.py Outdated
@muddlebee

Copy link
Copy Markdown
Collaborator

@farizanjum pls check docs/adding-tools-and-integrations.md I understand this was added as slack specific, but we need to make the railway integration more generic to fit in first for overall integration.

check docs here https://www.opensre.com/docs/integrations-overview

Copilot AI review requested due to automatic review settings July 22, 2026 09:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings July 22, 2026 09:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@farizanjum

Copy link
Copy Markdown
Author

@greptile review

@farizanjum

farizanjum commented Jul 22, 2026

Copy link
Copy Markdown
Author

@muddlebee

The requested Railway-first refactor and upstream main reconciliation are complete and pushed.

  • Replaced the Slack-specific Railway runtime tools with generic Railway deployment tools.
  • Added Railway configuration, setup, verification, discovery, and user-facing documentation.
  • Reused the shared Slack web client helpers and made thread parsing safe for null messages and reactions.
  • Increased Railway deployment inspection to 100 records and select the newest successful deployment.
  • Kept redeploy behind explicit confirmation and require Railway to return a deployment ID.

Verification completed:

  • 91 focused tests passed.
  • Ruff, formatting, mypy, and docs JSON validation passed.
  • Live Railway verification, deployment inspection, and confirmed redeploy were exercised against a disposable project.

Please take another look when you have a chance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inspect, replay, and redeploy Slack bot behavior

3 participants