hipergator

R interface for SLURM job management on UF HiPerGator

Note: This package is designed for internal use by the Florida Museum of Natural History AI Lab at the University of Florida. It provides opinionated helpers specifically for UF HiPerGator workflows (Duo authentication, module system, storage paths). However, the design patterns and code may serve as inspiration for interfacing with other SLURM-based HPC systems.

Overview

Simple, secure R interface for submitting and managing SLURM jobs on the University of Florida HiPerGator cluster. Provides structured configuration, secure path handling, SSH multiplexing with Duo MFA support, and unified job lifecycle management.

Features

• SSH Multiplexing with Duo MFA - authenticate once, no re-authentication needed
• Structured Configuration - cluster, storage, and defaults in one place
• Resource-First Workflow - using hpg_resources() and hpg_submit()
• Unified Job Objects - persistent IDs, status history, and metadata
• Secure Path Handling - keeps work inside configured base directory
• Transfer Helpers - rsync-based uploads, downloads, and remote directory management
• GPU Support - automatic partition selection and resource validation

Installation

# Install from GitHub
remotes::install_github("flmnh-ai/hipergator")

Quick Start

1. Configure Once

Set up your HiPerGator connection:

library(hipergator)

# Configure cluster and storage
hpg_configure(
  cluster = hpg_cluster_config(
    host = "hpg",  # SSH alias or full hostname
    user = "yourufid"
  ),
  storage = hpg_storage_config(
    base_dir = "/blue/yourlab/yourufid"
  ),
  defaults = hpg_defaults(
    timeout_hours = 12,
    poll_interval = 30
  )
)

# View configuration
hpg_configuration(show = TRUE)

2. Authenticate with Duo

# Establish SSH master connection (handles Duo interactively)
target <- hpg_authenticate()

# Check connection status
hpg_check_connection()

The master connection persists for your session - subsequent operations don't require re-authentication.

3. Submit a Job

# Define resources
resources <- hpg_resources(
  cores = 8,
  memory = "32gb",
  time = "04:00:00",
  partition = "hpg-turin",
  gpu = hpg_gpu(count = 1, type = "l4")
)

# Submit job
job <- hpg_submit(
  resources = resources,
  command = "python scripts/train.py",
  job_name = "training_run",
  working_dir = "projects/my_project"
)

# Monitor with progress
hpg_wait(job)

4. Transfer Files

# Upload local directory
hpg_upload(
  target = target,
  local_path = "data/processed/",
  remote_path = "projects/my_project/data"
)

# Download results
hpg_download(
  target = target,
  remote_path = "projects/my_project/output/",
  local_path = "results/"
)

# Create remote directories
hpg_mkdir(target, c(
  "projects/my_project/data",
  "projects/my_project/output"
))

# Check if path exists
hpg_exists(target, "projects/my_project/data")

Core Workflow

Define Resources

# CPU-only job
resources <- hpg_resources(
  cores = 16,
  memory = "64gb",
  time = "08:00:00",
  partition = "hpg-default"
)

# GPU job with modules and conda
resources <- hpg_resources(
  cores = 56,
  memory = "128gb",
  time = "24:00:00",
  partition = "hpg-b200",
  gpu = hpg_gpu(count = 2, type = "b200"),
  modules = c("conda", "cuda/12.0"),
  conda_env = "/blue/yourlab/envs/pytorch"
)

# Get partition suggestions
hpg_suggest_partitions(resources)

Submit and Monitor

# Submit job
job <- hpg_submit(
  resources = resources,
  command = "python train.py --config config.yaml",
  job_name = "experiment_v3",
  working_dir = "projects/training"
)

# Wait for completion with progress bar
hpg_wait(job, show_progress = TRUE)

# Get detailed job info (includes current status)
info <- hpg_job_info(job)
print(info$status)

# Cancel if needed
hpg_cancel(job)

Handle Errors

# Get error output and exit code
error_info <- hpg_get_error(job)
cat(error_info$stderr)

Configuration

Structured Configuration

hpg_configure(
  cluster = hpg_cluster_config(
    host = "hpg.rc.ufl.edu",
    user = "yourufid"
  ),
  storage = hpg_storage_config(
    base_dir = "/blue/yourlab/yourufid"
  ),
  defaults = hpg_defaults(
    timeout_hours = 12,
    poll_interval = 30,
    partition = "hpg-default"
  )
)

Environment Variables

You can also configure via .Renviron:

# Add to .Renviron (edit with usethis::edit_r_environ())
HIPERGATOR_HOST="hpg"
HIPERGATOR_USER="yourufid"
HIPERGATOR_BASE_DIR="/blue/yourlab/yourufid"

View Configuration

# Show structured configuration
hpg_configuration(show = TRUE)

# Get flattened config for scripting
config <- hpg_config()

# Show base directory
hpg_show_base_dir()

Security Features

Path Validation

All remote paths are validated to prevent directory traversal:

# ✓ Valid - within base_dir
hpg_upload(target, "data/", "projects/my_project/data")

# ✗ Invalid - outside base_dir
hpg_upload(target, "data/", "/orange/otherlab/data")  # Error!

# ✗ Invalid - directory traversal attempt
hpg_upload(target, "data/", "projects/../../etc")      # Error!

Command Safety

Commands are checked for dangerous patterns before submission:

# Warns about risky patterns
hpg_submit(resources, "rm -rf /", ...)  # Warning!

GPU Resources

# Request specific GPU type
resources <- hpg_resources(
  cores = 56,
  memory = "128gb",
  time = "24:00:00",
  partition = "hpg-b200",
  gpu = hpg_gpu(count = 2, type = "b200")  # Options: "l4", "b200"
)

# Submit job
job <- hpg_submit(resources, command, ...)

Advanced Usage

Job Metadata

# Access job details
job$hipergator_id   # Unique job identifier
job$slurm_id        # SLURM job ID
job$status_history  # Status transitions with timestamps
job$resources       # Resource request
job$metadata        # Custom metadata

Batch Processing

# Submit multiple jobs
jobs <- lapply(1:10, function(i) {
  hpg_submit(
    resources = resources,
    command = sprintf("python train.py --fold %d", i),
    job_name = sprintf("fold_%02d", i),
    working_dir = "projects/cv_experiment"
  )
})

# Wait for all
lapply(jobs, hpg_wait)

HiPerGator-Specific Features

Module System

resources <- hpg_resources(
  cores = 8,
  memory = "32gb",
  modules = c("conda", "cuda/12.0", "gcc/11.0")
)

Conda Environments

resources <- hpg_resources(
  cores = 8,
  memory = "32gb",
  conda_env = "/blue/yourlab/envs/pytorch"
)

rsync.rc.ufl.edu Optimization

The package automatically uses rsync.rc.ufl.edu for transfers when the SSH target is hpg.rc.ufl.edu, providing better transfer performance.

Job Lifecycle

# Submit → PENDING → RUNNING → COMPLETED
job <- hpg_submit(resources, command, ...)

# Monitor with hpg_wait() - shows spinner during PENDING, progress during RUNNING
hpg_wait(job, show_progress = TRUE)

# Or check status manually
info <- hpg_job_info(job)
print(info$status)  # "PENDING", "RUNNING", "COMPLETED", or "FAILED"

# Get error details if failed
if (info$status == "FAILED") {
  error_info <- hpg_get_error(job)
  cat(error_info$stderr)
}

Configuration Helpers

• hpg_configure() - Store cluster, storage, and defaults
• hpg_configuration(show = TRUE) - View full configuration
• hpg_config() - Get flattened config for scripting
• hpg_show_base_dir() - Display configured base directory
• hpg_cluster_config() - Define cluster settings
• hpg_storage_config() - Define storage paths
• hpg_defaults() - Set default timeouts and intervals

Authentication Helpers

• hpg_authenticate() - Start SSH master connection with Duo
• hpg_check_connection() - Check if connection is active
• hpg_disconnect() - Close SSH master connection

Resource Helpers

• hpg_resources() - Define CPU/memory/time requirements
• hpg_gpu() - Convenience wrapper for GPU jobs
• hpg_suggest_partitions() - Get partition recommendations

Job Management Helpers

• hpg_submit() - Submit job and return job object
• hpg_wait() - Block until completion with progress
• hpg_cancel() - Cancel running job
• hpg_job_info() - Get detailed job metadata (includes current status)
• hpg_get_error() - Fetch error logs and exit code

File Transfer Helpers

• hpg_upload() - Upload files/directories via rsync
• hpg_download() - Download files/directories via rsync
• hpg_mkdir() - Create remote directories
• hpg_exists() - Check if remote path exists

Testing

# Run test suite
devtools::test()

# Or with testthat directly
testthat::test_dir("tests/testthat", reporter = "summary")

Tests cover:

• Resource validation
• SLURM script rendering
• Path security checks
• Job metadata construction
• Configuration management

Troubleshooting

Connection Issues

# Check SSH config
# Ensure you have an entry for HiPerGator in ~/.ssh/config

# Verify connection
hpg_check_connection()

# Re-authenticate if needed
hpg_disconnect()
hpg_authenticate()

Path Errors

# Check base directory
hpg_show_base_dir()

# Verify path is within base_dir
# All paths must be relative to configured base_dir

Job Failures

# Get error details
error_info <- hpg_get_error(job)
cat(error_info$stderr)
cat("Exit code:", error_info$exit_code)

# Check job info
info <- hpg_job_info(job)
str(info)

Design Philosophy

• Fail Fast - Validate early with clear error messages
• Secure by Default - Path validation prevents directory traversal
• Stateless Functions - Job objects are environment-based for persistence
• User Feedback - Clear progress indicators and status messages via cli
• SSH Multiplexing - Authenticate once, no re-prompts

Use Cases

This package powers HPC training workflows in:

• petrographer - Deep learning model training for petrographic image analysis
• Other lab projects - Any R-driven workflow requiring HiPerGator compute

Adaptability

While designed for UF HiPerGator, the package demonstrates patterns useful for other SLURM systems:

• SSH multiplexing for MFA systems
• Structured configuration management
• Secure path handling
• Job lifecycle abstraction
• Resource validation and partition selection

To adapt for your HPC system, modify:

• Cluster configuration defaults
• Partition names and GPU types
• Module system specifics
• Path validation rules

Contributing

This is research software under active development. Breaking changes may occur between versions.

For issues or feature requests related to UF HiPerGator usage, contact the Florida Museum AI Lab.

Acknowledgments

• University of Florida Research Computing for HiPerGator infrastructure
• cli, fs, glue, processx - Modern R utilities
• ssh multiplexing - OpenSSH ControlMaster feature

---

Happy computing on HiPerGator! 🐊