Mattermost ChatOps Bot - Security Model and Best Practices

This document outlines the security architecture, threat model, and best practices for deploying and operating the Mattermost ChatOps Bot.

1. Security Model Overview
2. Script Allowlist System
3. Permission System
4. Input Validation and Sanitization
5. Subprocess Sandboxing
6. Audit Logging
7. Supply Chain Security
8. Deployment Security
9. Network Security
10. Secret Management
11. Monitoring and Incident Response
12. Security Checklist

The Mattermost ChatOps Bot implements defense-in-depth security with multiple layers:

┌─────────────────────────────────────────────────────────────┐
│ Layer 1: Authentication (Mattermost Bot Token)              │
├─────────────────────────────────────────────────────────────┤
│ Layer 2: Ban Enforcement (Router-level blocking)            │
├─────────────────────────────────────────────────────────────┤
│ Layer 3: Permission Checks (Admin/Operator/User)            │
├─────────────────────────────────────────────────────────────┤
│ Layer 4: Script Allowlist (Pre-approved scripts only)       │
├─────────────────────────────────────────────────────────────┤
│ Layer 5: Argument Validation (Type + regex + sanitization)  │
├─────────────────────────────────────────────────────────────┤
│ Layer 6: Subprocess Sandboxing (rlimits + timeout)          │
├─────────────────────────────────────────────────────────────┤
│ Layer 7: Output Sanitization (Prevent injection)            │
├─────────────────────────────────────────────────────────────┤
│ Layer 8: Audit Logging (Cryptographic signatures)           │
└─────────────────────────────────────────────────────────────┘

1. Default Deny - Nothing is permitted unless explicitly allowed
2. Least Privilege - Users get minimum permissions needed
3. Defense in Depth - Multiple security layers, no single point of failure
4. Fail Secure - Errors result in denial, not bypass
5. Audit Everything - Full audit trail for forensics and compliance

Without allowlist: Arbitrary command execution leads to:

• Remote code execution (RCE)
• Data exfiltration
• Lateral movement
• Privilege escalation
• Denial of service

With allowlist: Only pre-approved scripts can execute.

File: config/script-allowlist.json

{
  "scripts": [
    {
      "name": "deploy",
      "path": "./scripts/deploy.sh",
      "description": "Deploy application to environment",
      "requiredPermission": "operator",
      "timeout": 1800000,
      "arguments": {
        "service": {
          "type": "enum",
          "values": ["api-server", "web-app", "database"],
          "required": true
        },
        "environment": {
          "type": "enum",
          "values": ["staging", "production"],
          "required": true
        }
      },
      "async": true,
      "allowedChannels": ["ops", "deployments"]
    }
  ]
}

✅ DO:

• Review all scripts before adding to allowlist
• Use enum type for arguments with finite values
• Set requiredPermission to most restrictive level needed
• Use allowedChannels to limit blast radius
• Set appropriate timeout to prevent runaway processes
• Use async: true for long-running scripts

❌ DON'T:

• Allow scripts that accept arbitrary commands
• Use string type without regex validation
• Grant user permission for destructive scripts
• Allow scripts in public channels unless necessary
• Set timeout > 1 hour without strong justification

BANNED < USER < OPERATOR < ADMIN

Table: users

CREATE TABLE users (
  mattermost_id TEXT PRIMARY KEY,
  username TEXT NOT NULL,
  permission_level TEXT NOT NULL, -- admin, operator, user, banned
  custom_flags TEXT, -- JSON: ["deploy", "restart"]
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL
);

CRITICAL: Create the first admin user before deploying:

cd /opt/mattermost-bot
sudo -u mattermost-bot bun run scripts/bootstrap-admin.ts <mattermost-user-id> <username>

Security Notes:

• This script can only be run by the system owner (requires filesystem access)
• First admin bypasses permission checks (bootstrapping problem)
• Subsequent admins must be added by existing admin using !adduser

1. Cannot Ban Admins - Safety feature prevents admin lockout
2. Mattermost ID Binding - Permissions tied to Mattermost user ID (not username)
3. Default Deny - Unknown users default to user permission (lowest)
4. Explicit Grants Only - No implicit permission escalation
5. Audit Trail - All permission changes logged to audit_log table

Script Name Validation (src/utils/security.ts):

// Only alphanumeric, dash, underscore
const scriptNameRegex = /^[a-zA-Z0-9_-]+$/;

// Blocks:
// - Path traversal: ../, ../../, absolute paths
// - Shell metacharacters: ; & | ` $ ( ) { } [ ] < >
// - Null bytes: \x00

Argument Validation:

interface ArgumentSpec {
  type: 'string' | 'number' | 'boolean' | 'enum';
  regex?: string;        // Pattern for string validation
  min?: number;          // Min value for numbers
  max?: number;          // Max value for numbers
  length?: number;       // Max length for strings
  values?: string[];     // Enum values
  required?: boolean;    // Argument is mandatory
}

ANSI Escape Codes - Removed to prevent terminal injection:

output = output.replace(/\x1b\[[0-9;]*m/g, '');

Markdown Escaping - Prevents Mattermost formatting injection:

// Escape: _ * ` ~ |
output = output.replace(/([_*`~|])/g, '\\$1');

Length Truncation - Prevents DoS:

if (output.length > 4000) {
  output = output.substring(0, 4000) + '\n... (truncated)';
}

const result = await Bun.spawn([scriptPath, ...args], {
  env: {
    // Minimal environment - blocks inheritance of secrets
    PATH: '/usr/bin:/bin',
    HOME: '/opt/mattermost-bot',
    USER: 'mattermost-bot',
    LANG: 'en_US.UTF-8',
    TZ: 'UTC'
  },
  cwd: '/opt/mattermost-bot/scripts',  // Working directory restriction
  stdin: 'ignore',                      // No stdin (prevents interactive prompts)
  stdout: 'pipe',                       // Capture output
  stderr: 'pipe',                       // Capture errors
  timeout: scriptConfig.timeout * 1000  // Enforce timeout
});

1. No Shell Interpretation - Direct execution (not sh -c)
2. Timeout Enforcement - Scripts killed after timeout (default: 5 minutes)
3. Minimal Environment - Only essential environment variables
4. Working Directory Restriction - Scripts run in scripts/ directory
5. No Interactive Input - stdin disabled

Recommended (requires systemd or manual rlimit setting):

Implementation (systemd service):

[Service]
LimitCPU=1800
LimitAS=1G
LimitNOFILE=256
LimitNPROC=50

File: logs/script-executions.log

Format: JSON Lines (one JSON object per line)

{
  "timestamp": "2024-01-27T10:30:45.123Z",
  "executionId": "uuid-v4",
  "scriptName": "deploy",
  "arguments": {
    "service": "api-server",
    "environment": "production"
  },
  "userId": "mattermost-user-id",
  "channelId": "mattermost-channel-id",
  "success": true,
  "exitCode": 0,
  "durationMs": 45230,
  "error": null
}

Ed25519 Cryptographic Signatures - Each log entry is signed:

{
  "signature": "base64-encoded-ed25519-signature",
  "publicKey": "base64-encoded-public-key",
  "previousHash": "sha256-hash-of-previous-entry"
}

Verification (scripts/verify-audit-log.ts):

bun run scripts/verify-audit-log.ts

Security Properties:

• Tamper-proof - Any modification invalidates signature
• Deletion-resistant - Hash chain detects missing entries
• Non-repudiation - Cryptographic proof of execution

Daily checks:

# Check for failed executions in last 24 hours
jq 'select(.success == false and .timestamp > (now - 86400))' logs/script-executions.log

# Count executions by script
jq -r .scriptName logs/script-executions.log | sort | uniq -c | sort -rn

# Find executions by specific user
jq "select(.userId == \"mattermost-user-id\")" logs/script-executions.log

Retention: Keep audit logs for minimum 90 days for compliance.

File: bunfig.toml

[install]
frozen = true               # Enforce deterministic installs
minimumReleaseAge = 259200  # Block packages < 3 days old (defense against rapid-publish attacks)

[install.scopes]
allowScripts = false        # Disable lifecycle scripts by default

Lock File:

• Use bun.lock (text format, v1.2+) for version control
• NEVER delete lock file
• ALWAYS use --frozen-lockfile in production

Trusted Dependencies (package.json):

{
  "trustedDependencies": []
}

Currently empty - No dependencies require lifecycle scripts.

Before deployment:

# Verify supply chain
bun run scripts/verify-supply-chain.ts

# Checks:
# - Lock file exists
# - Lock file matches package.json (frozen check)
# - bunfig.toml has security settings
# - No vulnerabilities (bun audit)

Software Bill of Materials for vulnerability tracking:

# Install syft (first time only)
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

# Generate SBOM
./scripts/generate-sbom.sh

# Scan for vulnerabilities
grype sbom:sbom.spdx.json

File: /etc/systemd/system/mattermost-bot.service

[Service]
# Security hardening
NoNewPrivileges=true          # Prevent privilege escalation
PrivateTmp=true               # Isolated /tmp
ProtectSystem=strict          # Read-only /usr, /boot, /efi
ProtectHome=true              # No access to /home
ReadWritePaths=/opt/mattermost-bot/data /opt/mattermost-bot/logs

# Network restrictions
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

# Resource limits
LimitCPU=1800
LimitAS=1G
LimitNOFILE=256
LimitNPROC=50

# Bot config (contains token - keep secret!)
chmod 600 /opt/mattermost-bot/config/bot.config.json
chown mattermost-bot:mattermost-bot /opt/mattermost-bot/config/bot.config.json

# Scripts (must be executable)
chmod +x /opt/mattermost-bot/scripts/*.sh

# Data directory (bot writes here)
chmod 700 /opt/mattermost-bot/data
chown mattermost-bot:mattermost-bot /opt/mattermost-bot/data

# Logs directory
chmod 700 /opt/mattermost-bot/logs
chown mattermost-bot:mattermost-bot /opt/mattermost-bot/logs

Recommended: Keep SELinux in enforcing mode

# Check SELinux status
getenforce

# Set appropriate context
semanage fcontext -a -t bin_t "/opt/mattermost-bot/scripts(/.*)?\.sh"
restorecon -Rv /opt/mattermost-bot/scripts

Minimal exposure:

• Bot does NOT listen on any ports (client-only)
• Only outbound connections to Mattermost server
• No inbound connections required

Firewall rules (RHEL 9 firewalld):

# Allow outbound HTTPS to Mattermost (if default-deny policy)
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" destination address="<mattermost-server-ip>" port port="443" protocol="tcp" accept'
firewall-cmd --reload

Mattermost Connection:

• Bot connects via WebSocket over TLS (wss://)
• Bot connects via HTTPS for REST API
• Certificate validation enabled (do not disable)

Configuration:

{
  "mattermost": {
    "url": "https://mattermost.example.com",
    "websocketUrl": "wss://mattermost.example.com/api/v4/websocket"
  }
}

Certificate Trust:

• Uses system CA certificates (/etc/pki/tls/certs/ca-bundle.crt on RHEL)
• For self-signed certificates, add CA to system trust store (do NOT disable validation)

Storage:

• File: config/bot.config.json
• Permissions: 600 (owner read/write only)
• Owner: mattermost-bot user

Rotation:

1. Generate new bot token in Mattermost System Console
2. Update config/bot.config.json
3. Restart bot: systemctl restart mattermost-bot

DO NOT:

• ❌ Commit bot token to git
• ❌ Store token in environment variables (visible in process list)
• ❌ Log token in any logs
• ❌ Share token via Mattermost or email

Generation:

sudo -u mattermost-bot ssh-keygen -t ed25519 -f /opt/mattermost-bot/.ssh/id_ed25519_bot

Permissions:

chmod 700 /opt/mattermost-bot/.ssh
chmod 600 /opt/mattermost-bot/.ssh/id_ed25519_bot
chmod 644 /opt/mattermost-bot/.ssh/id_ed25519_bot.pub

Deployment:

• Add public key to target servers (not the private key!)
• Use ansible_ssh_private_key_file in inventory to specify bot key

Systemd Status:

systemctl status mattermost-bot

Logs:

# Real-time logs
journalctl -u mattermost-bot -f

# Last 100 lines
journalctl -u mattermost-bot -n 100

# Errors only
journalctl -u mattermost-bot -p err

Monitoring Targets:

Prometheus Integration (optional):

• Metrics server: http://localhost:9090/metrics
• Metrics: bot_script_executions_total, bot_script_duration_ms
• See MONITORING.md for full setup

Suspected Compromise:

1. Isolate - Stop bot: systemctl stop mattermost-bot
2. Preserve - Backup logs and database: cp -r /opt/mattermost-bot/{logs,data} /secure/backup/
3. Investigate - Review audit logs for unauthorized activity
4. Remediate - Rotate bot token, review permissions, patch vulnerabilities
5. Restore - Restart bot: systemctl start mattermost-bot

Unauthorized Script Execution:

1. Identify user: Check audit log userId field
2. Review script: Check scriptName and arguments
3. Check permission: sqlite3 data/bot.db "SELECT * FROM users WHERE mattermost_id = '<user-id>'"
4. Ban user if malicious: !ban @user "Unauthorized activity" permanent
5. Remove script from allowlist if compromised

• First admin user created via bootstrap script
• Bot token generated and stored with 600 permissions
• All scripts reviewed and added to allowlist
• Argument validation schemas defined for all scripts
• Supply chain verified: bun run scripts/verify-supply-chain.ts
• SBOM generated: ./scripts/generate-sbom.sh
• No vulnerabilities: grype sbom:sbom.spdx.json
• Systemd hardening enabled (see service file)
• File permissions set correctly (600 for config, 700 for data/logs)
• SELinux in enforcing mode (RHEL 9)
• Firewall configured (outbound HTTPS only)

• Bot connects successfully to Mattermost
• Test !ping command
• Test permission enforcement (operator can run scripts, user cannot)
• Test ban enforcement (banned user cannot use bot)
• Verify audit logging: tail -f logs/script-executions.log
• Verify audit signatures: bun run scripts/verify-audit-log.ts
• Set up monitoring alerts (service down, failed scripts)
• Document bot token rotation procedure
• Schedule quarterly security reviews

• Review audit logs weekly for anomalies
• Rotate bot token every 90 days
• Update dependencies monthly: bun update
• Re-run vulnerability scan monthly: grype sbom:sbom.spdx.json
• Review and update script allowlist as needed
• Test backup and restore procedures quarterly
• Review user permissions quarterly (principle of least privilege)

For security issues:

1. Do NOT post security issues in public channels
2. Contact system administrator directly
3. Include: Description, affected component, reproduction steps, potential impact

For vulnerability reports:

• If vulnerability is in Mattermost ChatOps Bot: Contact repo maintainer
• If vulnerability is in dependency: Report to dependency maintainer, update bot

• Bun Security Scanner API: https://bun.com/docs/install/security-scanner-api
• Bun Audit: https://bun.com/docs/install/audit
• SBOM Format (SPDX): https://spdx.dev/
• Vulnerability Scanner (Grype): https://github.com/anchore/grype
• OWASP Top 10: https://owasp.org/www-project-top-ten/
• CWE Top 25: https://cwe.mitre.org/top25/

Document Version: 1.0.0 Last Updated: 2024-01-27 Status: Production Ready