Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This library is designed to be integrated into robot data collection systems, ET

- 🤖 **Teleoperation Session Management**: Track robot data collection sessions with `TeleopSession`
- 🔄 **Data Conversion Session Management**: Track data conversion pipelines with `ConversionSession`
- ☁️ **Wasabi Upload Management**: Track Wasabi cloud storage uploads with `WasabiUploadSession`
- ☁️ **S3 Upload Management**: Track S3 object storage uploads with `S3UploadSession`
- 🔌 **USB Data Copy Management**: Track USB data copy operations with `USBCopySession`
- 📊 **OpenLineage Integration**: Full OpenLineage specification support (START/COMPLETE/RUNNING/FAIL/ABORT)
- 🏷️ **Custom Run Facets**: Robot metadata (robotId, location, repository info) via `CommonRunFacet`
Expand Down Expand Up @@ -227,13 +227,13 @@ uv run python examples/teleop_session.py

Demonstrates a complete teleoperation session with nominal time tracking.

### Wasabi Upload Session
### S3 Upload Session

```bash
uv run python examples/wasabi_upload_session.py
uv run python examples/s3_upload_session.py
```

Demonstrates a complete Wasabi cloud storage upload session with nominal time tracking.
Demonstrates a complete S3 object storage upload session with nominal time tracking.

### Batch ETL Job

Expand Down
22 changes: 11 additions & 11 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,27 +93,27 @@ ConversionSession.complete(output_datasets) # Emits COMPLETE event
**API Documentation**: See docstrings in [session.py](../src/airoa_lineage/conversion/session.py)
**Usage Example**: [examples/data_conversion.py](../examples/data_conversion.py)

### WasabiUploadSession
### S3UploadSession

**Location**: [src/airoa_lineage/wasabi_upload/session.py](../src/airoa_lineage/wasabi_upload/session.py)
**Location**: [src/airoa_lineage/s3_upload/session.py](../src/airoa_lineage/s3_upload/session.py)

**Purpose**: Manages the lifecycle of Wasabi cloud storage upload operations with OpenLineage tracking.
**Purpose**: Manages the lifecycle of S3 object storage upload operations with OpenLineage tracking.

**Key Responsibilities**:
- Track Wasabi data upload sessions (START → COMPLETE)
- Track S3 data upload sessions (START → COMPLETE)
- Automatic run_id generation and management
- Nominal time support for historical data processing
- State validation (prevents duplicate start/complete calls)

**Lifecycle**:
```
WasabiUploadSession.start() # Emits START event
└─> Data upload to Wasabi...
WasabiUploadSession.complete() # Emits COMPLETE event
S3UploadSession.start() # Emits START event
└─> Data upload to S3...
S3UploadSession.complete() # Emits COMPLETE event
```

**API Documentation**: See docstrings in [session.py](../src/airoa_lineage/wasabi_upload/session.py)
**Usage Example**: [examples/wasabi_upload_session.py](../examples/wasabi_upload_session.py)
**API Documentation**: See docstrings in [session.py](../src/airoa_lineage/s3_upload/session.py)
**Usage Example**: [examples/s3_upload_session.py](../examples/s3_upload_session.py)

### USBCopySession

Expand Down Expand Up @@ -278,7 +278,7 @@ For detailed event schemas, see the [OpenLineage specification](https://openline
- [MarquezClient](../src/airoa_lineage/marquez_client/client.py) - Event emission and REST API
- [TeleopSession](../src/airoa_lineage/teleop/session.py) - Robot data collection lifecycle
- [ConversionSession](../src/airoa_lineage/conversion/session.py) - Data conversion pipeline lifecycle
- [WasabiUploadSession](../src/airoa_lineage/wasabi_upload/session.py) - Wasabi cloud storage upload lifecycle
- [S3UploadSession](../src/airoa_lineage/s3_upload/session.py) - S3 object storage upload lifecycle
- [USBCopySession](../src/airoa_lineage/usb_copy/session.py) - USB data copy lifecycle
- [Timestamp Utilities](../src/airoa_lineage/utils/timestamps.py) - UTC timestamp generation
- [CommonRunFacet](../src/airoa_lineage/facets/common.py) - Robot metadata facet
Expand All @@ -289,7 +289,7 @@ For detailed event schemas, see the [OpenLineage specification](https://openline

- [examples/teleop_session.py](../examples/teleop_session.py) - Complete teleoperation session with nominal time
- [examples/data_conversion.py](../examples/data_conversion.py) - Data conversion pipeline with RUNNING events
- [examples/wasabi_upload_session.py](../examples/wasabi_upload_session.py) - Wasabi upload session with nominal time
- [examples/s3_upload_session.py](../examples/s3_upload_session.py) - S3 upload session with nominal time
- [examples/usb_copy_session.py](../examples/usb_copy_session.py) - USB data copy session with nominal time
- [examples/simple_batch_etl.py](../examples/simple_batch_etl.py) - Batch ETL job with input/output datasets

Expand Down
8 changes: 4 additions & 4 deletions docs/cli/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This guide covers common features and configuration methods for the `airoa-linea
## Available CLI Commands

- **[airoa-lineage-usb-copy](usb-copy.md)** - Track USB data copy sessions
- **[airoa-lineage-wasabi-upload](wasabi-upload.md)** - Track Wasabi upload sessions
- **[airoa-lineage-s3-upload](s3-upload.md)** - Track S3 upload sessions

---

Expand All @@ -23,7 +23,7 @@ Verify installation:

```bash
airoa-lineage-usb-copy --version
airoa-lineage-wasabi-upload --version
airoa-lineage-s3-upload --version
```

---
Expand Down Expand Up @@ -321,13 +321,13 @@ This shows what would be sent to Marquez without actually sending it.
For complete working examples, see the respective documentation pages:

- [USBCopy CLI Examples](usb-copy.md#examples)
- [WasabiUpload CLI Examples](wasabi-upload.md#examples)
- [S3Upload CLI Examples](s3-upload.md#examples)

---

## Getting Help

- CLI help: `airoa-lineage-usb-copy --help` or `airoa-lineage-wasabi-upload --help`
- CLI help: `airoa-lineage-usb-copy --help` or `airoa-lineage-s3-upload --help`
- Command help: `<command> start --help`
- Version: `<command> --version`
- GitHub Issues: https://github.com/AIRoA/airoa-lineage/issues
Expand Down
86 changes: 43 additions & 43 deletions docs/cli/wasabi-upload.md → docs/cli/s3-upload.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# WasabiUpload CLI
# S3Upload CLI

The `airoa-lineage-wasabi-upload` command provides a CLI interface for tracking Wasabi data upload operations using OpenLineage.
The `airoa-lineage-s3-upload` command provides a CLI interface for tracking S3 data upload operations using OpenLineage.

## Commands

### `start` - Start Wasabi Upload Session
### `start` - Start S3 Upload Session

Start a Wasabi upload session and emit a START event to Marquez.
Start a S3 upload session and emit a START event to Marquez.

**Synopsis:**

```bash
airoa-lineage-wasabi-upload start [OPTIONS]
airoa-lineage-s3-upload start [OPTIONS]
```

**Options:**
Expand All @@ -27,7 +27,7 @@ airoa-lineage-wasabi-upload start [OPTIONS]
| `--repository-branch` | string | Yes* | Git branch |
| `--nominal-start-time` | string | No | Nominal start time (ISO 8601 format) |
| `--nominal-end-time` | string | No | Nominal end time (ISO 8601 format) |
| `--job-name` | string | No | Job name (default: `wasabi-data-upload`) |
| `--job-name` | string | No | Job name (default: `s3-data-upload`) |
| `--marquez-url` | string | No | Marquez server URL |
| `--facet-prefix` | string | No | Facet prefix |
| `--dry-run` | flag | No | Show what would be done without executing |
Expand All @@ -45,7 +45,7 @@ airoa-lineage-wasabi-upload start [OPTIONS]

```bash
# Basic usage
RUN_ID=$(airoa-lineage-wasabi-upload start \
RUN_ID=$(airoa-lineage-s3-upload start \
--namespace airoa_examples \
--robot-id hsr001 \
--location weblab \
Expand All @@ -55,34 +55,34 @@ RUN_ID=$(airoa-lineage-wasabi-upload start \
--repository-branch main)

# With nominal times (for batch processing)
RUN_ID=$(airoa-lineage-wasabi-upload start \
RUN_ID=$(airoa-lineage-s3-upload start \
--nominal-start-time "2025-11-26T00:00:00+00:00" \
--nominal-end-time "2025-11-26T05:00:00+00:00")

# With custom job name
RUN_ID=$(airoa-lineage-wasabi-upload start \
--job-name "wasabi-experiment-upload")
RUN_ID=$(airoa-lineage-s3-upload start \
--job-name "s3-experiment-upload")

# Dry-run mode (verify configuration)
airoa-lineage-wasabi-upload start --dry-run
airoa-lineage-s3-upload start --dry-run

# JSON output
RUN_ID=$(airoa-lineage-wasabi-upload start --json | jq -r '.run_id')
RUN_ID=$(airoa-lineage-s3-upload start --json | jq -r '.run_id')

# Verbose mode
airoa-lineage-wasabi-upload start --verbose
airoa-lineage-s3-upload start --verbose
```

---

### `complete` - Complete Wasabi Upload Session
### `complete` - Complete S3 Upload Session

Complete a Wasabi upload session and emit a COMPLETE event to Marquez.
Complete a S3 upload session and emit a COMPLETE event to Marquez.

**Synopsis:**

```bash
airoa-lineage-wasabi-upload complete --run-id RUN_ID [OPTIONS]
airoa-lineage-s3-upload complete --run-id RUN_ID [OPTIONS]
```

**Options:**
Expand Down Expand Up @@ -115,16 +115,16 @@ airoa-lineage-wasabi-upload complete --run-id RUN_ID [OPTIONS]

```bash
# Basic usage
airoa-lineage-wasabi-upload complete --run-id "$RUN_ID"
airoa-lineage-s3-upload complete --run-id "$RUN_ID"

# With namespace override
airoa-lineage-wasabi-upload complete --run-id "$RUN_ID" --namespace airoa_examples
airoa-lineage-s3-upload complete --run-id "$RUN_ID" --namespace airoa_examples

# Specify all arguments explicitly (using same values as start)
airoa-lineage-wasabi-upload complete \
airoa-lineage-s3-upload complete \
--run-id "$RUN_ID" \
--namespace airoa_examples \
--job-name wasabi-data-upload \
--job-name s3-data-upload \
--robot-id hsr001 \
--location weblab \
--repository-hash abc123 \
Expand All @@ -133,22 +133,22 @@ airoa-lineage-wasabi-upload complete \
--repository-branch main

# JSON output
airoa-lineage-wasabi-upload complete --run-id "$RUN_ID" --json
airoa-lineage-s3-upload complete --run-id "$RUN_ID" --json

# Quiet mode
airoa-lineage-wasabi-upload complete --run-id "$RUN_ID" --quiet
airoa-lineage-s3-upload complete --run-id "$RUN_ID" --quiet
```

---

### `cancel` - Cancel Wasabi Upload Session
### `cancel` - Cancel S3 Upload Session

Cancel a Wasabi upload session and emit an ABORT event to Marquez.
Cancel a S3 upload session and emit an ABORT event to Marquez.

**Synopsis:**

```bash
airoa-lineage-wasabi-upload cancel --run-id RUN_ID [OPTIONS]
airoa-lineage-s3-upload cancel --run-id RUN_ID [OPTIONS]
```

**Options:**
Expand Down Expand Up @@ -181,19 +181,19 @@ airoa-lineage-wasabi-upload cancel --run-id RUN_ID [OPTIONS]

```bash
# Basic usage
airoa-lineage-wasabi-upload cancel --run-id "$RUN_ID"
airoa-lineage-s3-upload cancel --run-id "$RUN_ID"

# Override specific CommonRunFacet fields
airoa-lineage-wasabi-upload cancel \
airoa-lineage-s3-upload cancel \
--run-id "$RUN_ID" \
--robot-id hsr002 \
--location lab2

# Specify all arguments explicitly (using same values as start)
airoa-lineage-wasabi-upload cancel \
airoa-lineage-s3-upload cancel \
--run-id "$RUN_ID" \
--namespace airoa_examples \
--job-name wasabi-data-upload \
--job-name s3-data-upload \
--robot-id hsr001 \
--location weblab \
--repository-hash abc123 \
Expand All @@ -202,52 +202,52 @@ airoa-lineage-wasabi-upload cancel \
--repository-branch main

# JSON output
airoa-lineage-wasabi-upload cancel --run-id "$RUN_ID" --json
airoa-lineage-s3-upload cancel --run-id "$RUN_ID" --json

# Quiet mode
airoa-lineage-wasabi-upload cancel --run-id "$RUN_ID" --quiet
airoa-lineage-s3-upload cancel --run-id "$RUN_ID" --quiet
```

---

## Examples

For complete working examples, see [examples/cli/wasabi_upload/](../../examples/cli/wasabi_upload/).
For complete working examples, see [examples/cli/s3_upload/](../../examples/cli/s3_upload/).

### Available Examples

The following executable shell scripts demonstrate different usage patterns:

1. **[basic_workflow.sh](../../examples/cli/wasabi_upload/basic_workflow.sh)** - Simple start → complete workflow
1. **[basic_workflow.sh](../../examples/cli/s3_upload/basic_workflow.sh)** - Simple start → complete workflow
- All configuration via CLI arguments
- Ideal for quick testing and learning

2. **[with_config_file.sh](../../examples/cli/wasabi_upload/with_config_file.sh)** - Configuration file usage
2. **[with_config_file.sh](../../examples/cli/s3_upload/with_config_file.sh)** - Configuration file usage
- Loads settings from `config.example.json`
- Minimal CLI arguments required
- Includes dry-run mode for testing

3. **[with_env_vars.sh](../../examples/cli/wasabi_upload/with_env_vars.sh)** - Environment variables
3. **[with_env_vars.sh](../../examples/cli/s3_upload/with_env_vars.sh)** - Environment variables
- Configuration via `AIROA_*` environment variables
- Dynamically retrieves repository info from git

4. **[error_handling.sh](../../examples/cli/wasabi_upload/error_handling.sh)** - Error handling with trap
4. **[error_handling.sh](../../examples/cli/s3_upload/error_handling.sh)** - Error handling with trap
- Automatic session cancellation on error
- Cleanup function with proper exit codes

5. **[retry_logic.sh](../../examples/cli/wasabi_upload/retry_logic.sh)** - Connection retry logic
5. **[retry_logic.sh](../../examples/cli/s3_upload/retry_logic.sh)** - Connection retry logic
- Automatic retry on connection errors
- Maximum 3 attempts with exponential backoff

See [examples/cli/wasabi_upload/README.md](../../examples/cli/wasabi_upload/README.md) for detailed documentation on each example, including prerequisites, usage instructions, and integration patterns.
See [examples/cli/s3_upload/README.md](../../examples/cli/s3_upload/README.md) for detailed documentation on each example, including prerequisites, usage instructions, and integration patterns.

---

## Quick Start

```bash
# 1. Start a Wasabi upload session
RUN_ID=$(airoa-lineage-wasabi-upload start \
# 1. Start a S3 upload session
RUN_ID=$(airoa-lineage-s3-upload start \
--namespace airoa_examples \
--robot-id hsr001 \
--location weblab \
Expand All @@ -258,14 +258,14 @@ RUN_ID=$(airoa-lineage-wasabi-upload start \
--nominal-start-time "2025-11-26T00:00:00+00:00" \
--nominal-end-time "2025-11-26T05:00:00+00:00")

# 2. Perform your Wasabi upload operation
# 2. Perform your S3 upload operation
# (your upload processing here)

# 3. Complete the session (using same metadata as start)
airoa-lineage-wasabi-upload complete \
airoa-lineage-s3-upload complete \
--run-id "$RUN_ID" \
--namespace airoa_examples \
--job-name wasabi-data-upload \
--job-name s3-data-upload \
--robot-id hsr001 \
--location weblab \
--repository-hash $(git rev-parse HEAD) \
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/usb-copy.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,4 +263,4 @@ airoa-lineage-usb-copy complete \
## Related Documentation

- [CLI Overview](overview.md) - Common configuration, environment variables, troubleshooting
- [WasabiUpload CLI](wasabi-upload.md) - Track Wasabi upload operations
- [S3Upload CLI](s3-upload.md) - Track S3 upload operations
4 changes: 2 additions & 2 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ tests/
│ └── test_session.py
├── conversion/
│ └── test_session.py
├── wasabi_upload/
├── s3_upload/
│ └── test_session.py
├── usb_copy/
│ └── test_session.py
Expand Down Expand Up @@ -516,7 +516,7 @@ airoa-lineage/
│ │ └── test_session.py
│ ├── conversion/
│ │ └── test_session.py
│ ├── wasabi_upload/
│ ├── s3_upload/
│ │ └── test_session.py
│ ├── usb_copy/
│ │ └── test_session.py
Expand Down
4 changes: 2 additions & 2 deletions examples/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ Track USB data copy operations from the command line.
- Error handling with automatic cancellation
- Connection retry logic

### Wasabi Upload Session *(Coming Soon)*
### S3 Upload Session *(Coming Soon)*

Track Wasabi cloud upload operations from the command line.
Track S3 cloud upload operations from the command line.

**Status**: Planned for future release

Expand Down
Loading