# utils/tail_job_log.sh [JOB_ID | FILE | DIRECTORY]
#
# Follow (tail -f) the most recent job log file (.log or legacy .out).
# Default output directory: $JOB_WORKSPACE if set, otherwise outputs/ next to the scripts.
#
# Examples:
# utils/tail_job_log.sh # latest log in outputs/
# utils/tail_job_log.sh 12345 # latest log for Slurm job 12345
# utils/tail_job_log.sh outputs/12345.log # tail a specific file
# utils/tail_job_log.sh outputs/12345-JAX-llama2-70b/ # job dir (follows log symlink)
# utils/tail_job_log.sh logs/ # latest log in logs/
# utils/prometheus.sh <command> [options]
#
# Commands:
# view <data-dir> [-p PORT] Start Prometheus UI to browse persisted metrics
# list [<workspace>] List jobs with saved Prometheus data
# install Download the Prometheus binary to /tmp/prometheus
#
# Examples:
# utils/prometheus.sh list # List jobs with metrics data
# utils/prometheus.sh list /shared/maxtext_jobs # Custom workspace
# utils/prometheus.sh view outputs/12345-JAX-llama2-70b/prometheus # Open Prometheus UI on :9190
# utils/prometheus.sh view outputs/12345-JAX-llama2-70b/prometheus -p 9191 # Custom port (if 9190 is occupied)
This starts a read-only Prometheus instance (no scraping) serving the persisted TSDB data. Open http://localhost:9190 (or your chosen -p port) to query and graph all metrics (GPU, host, network, training scalars, and Ray) that were collected during the run. During live jobs, the Prometheus port defaults to 9190 but auto-increments (9191, 9192, ...) if the port is already occupied — check the job log for the actual port.
Standalone utility for sending and receiving Telegram messages via the Bot API. Handles URL-encoding for rich content (Markdown, code blocks, special characters). Credentials are loaded from ~/.tg_config after one-time setup. Supports multiple bot profiles (-b <name>); truly concurrent recv sessions require distinct bot tokens per profile.
utils/telegram_bot.sh send "Hello *world*"
utils/telegram_bot.sh -b alerts send "Job failed"
echo "Deploy complete" | utils/telegram_bot.sh send
utils/telegram_bot.sh recv --timeout 300recv is a one-shot wait: when a message arrives it prints the text and exits 0. Interactive loops must call recv again after handling that message.
See Notifications for full usage, programmable patterns, and options.
Sends Telegram push notifications for Slurm job state changes, hang detection, and periodic log updates. Credentials are loaded from ~/.tg_config after one-time setup.
utils/slurm_job_monitor.sh -j <slurm_job_id>See Notifications: slurm_job_monitor.sh for monitor options, credential resolution, and tips.
Detects available artifacts in a job's output directory and dispatches the appropriate analysis tools (tgs_tagger, TraceLens, IRLens). Saves structured results to analysis.json in the job directory, including a job_status field (completed / failed / cancelled / running / unknown) used for staleness detection. Re-runs are fast: finished jobs with up-to-date analysis are skipped automatically; TraceLens results are cached per profiling window. See Performance for details on each tool and the analysis workflow.
# utils/analyze_job.py [-f] [--skip-tgs] [--skip-irlens] [--skip-tracelens] [PATH ...]
#
# Examples:
# utils/analyze_job.py outputs/12345-run.log # analyze a finished job
# utils/analyze_job.py outputs/12345-JAX-llama2-70b/ # job dir
# utils/analyze_job.py -f outputs/12345-run.log # force re-analysis (bypass staleness check)
# utils/analyze_job.py --skip-tracelens outputs/*.log # skip TraceLens
Web dashboard for visualizing analysis results — TGS charts, TraceLens GPU utilization, IRLens HLO analysis, side-by-side comparison, and file browser. Reads analysis.json files generated by analyze_job.py.
# utils/perf_server.py [--host HOST] [--port PORT] [--outputs-dir DIR]
#
# Examples:
# utils/perf_server.py # localhost only, auto-picks port from 8080
# utils/perf_server.py --port 8080 # localhost only, explicit port
# utils/perf_server.py --outputs-dir /shared/maxtext_jobs # custom outputs dir
#
# Default bind is 127.0.0.1. For remote access prefer SSH tunneling
# (`ssh -L 8080:localhost:8080 user@host`) over --host 0.0.0.0; the
# dashboard has no auth and a wide bind exposes job metadata.
Requires pip install fastapi uvicorn.
Extracts steady-state Tokens/GPU/Second from training logs and renames the log file and job directory with the TGS value. Uses steps 5-14 for the measurement (steps 0-4 are discarded as warmup). Set training steps >= 15 for reliable metrics. Will not rename if the job appears to still be running; use -f to force (log file is renamed immediately, directory rename is deferred until the job completes).
# utils/tag_tgs.sh [-f] [-c] [PATH ...]
#
# Examples:
# utils/tag_tgs.sh # latest log in outputs/
# utils/tag_tgs.sh 12345 # Slurm job ID
# utils/tag_tgs.sh outputs/12345-run.log # specific file
# utils/tag_tgs.sh outputs/12345-JAX-llama2-70b/ # job dir (follows log symlink)
# utils/tag_tgs.sh -f outputs/12345-run.log # force rename/cleanup if job looks active
# utils/tag_tgs.sh -c outputs/*.log # cleanup: prompt to delete failed jobs
utils/release_gpu.shStops all running containers, kills stale GPU processes, and waits for GPU memory to be freed. Also called automatically by preflight (before each job) and on scancel (SIGTERM trap).
See Job Submission: How artifacts work for details on the artifact system.
utils/cleanup_artifacts.sh # dry-run: list all artifacts and their status
utils/cleanup_artifacts.sh -c # remove orphans (confirms one by one)
utils/cleanup_artifacts.sh -c -y # remove orphans (skip confirmation)