Skip to content

CI: use cargo-chef for dependency build caching #97

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ezdac wants to merge 5 commits into
base: develop
Choose a base branch
from
Draft

CI: use cargo-chef for dependency build caching #97

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fcbb986
chore: revert to reusable-workflows@v3.1.0-rc.3
ezdac Jan 7, 2026
ea826c4
fix: don't use BuildKit cache mount for /app/target on ephemeral builder
ezdac Jan 7, 2026
fda9e78
feat(ci): use cargo-chef to cache dependency builds
ezdac Jan 7, 2026
2de1258
chore: force a local rebuild, while keeping deps the same
ezdac Jan 7, 2026
60bb792
chore: try out --features flag for chef commands
ezdac Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/push-container.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ jobs:
&& inputs.concurrency-group
|| format('{0}-{1}', github.workflow, (github.head_ref != '' && github.head_ref || github.ref)) }}
cancel-in-progress: true
uses: celo-org/reusable-workflows/.github/workflows/docker-build.yaml@v3.1.0-rc.4
uses: celo-org/reusable-workflows/.github/workflows/docker-build.yaml@v3.1.0-rc.3
with:
runs-on: '["self-hosted", "org", "8-cpu"]'
workload-id-provider: ${{ inputs.workload-id-provider }}
Expand Down
123 changes: 24 additions & 99 deletions fault-proof/Dockerfile.challenger.celo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,108 +19,33 @@ RUN curl -L https://sp1.succinct.xyz | bash && \
~/.sp1/bin/sp1up && \
~/.sp1/bin/cargo-prove prove --version

RUN cargo install cargo-chef


# Construct the project skeleton dynamically with
# cargo-chef (in order to cache dependency builds later)
FROM base AS planner

COPY . .

RUN cargo chef prepare --bin challenger --recipe-path recipe.json

RUN cat recipe.json

# build dependencies and binary
FROM base AS builder

WORKDIR /app
# import the skeleton recipe
COPY --from=planner /app/recipe.json recipe.json

# only build dependencies with cargo-chef
RUN cargo chef cook --release --features eigenda --bin challenger --recipe-path recipe.json

# now copy the actual source code
COPY . .

# Copy workspace root Cargo files
COPY Cargo.toml Cargo.lock ./

# Copy all Cargo.toml files from workspace members
# This allows Cargo to resolve dependencies without source
COPY fault-proof/Cargo.toml ./fault-proof/
COPY utils/client/Cargo.toml ./utils/client/
COPY utils/host/Cargo.toml ./utils/host/
COPY utils/build/Cargo.toml ./utils/build/
COPY utils/celestia/client/Cargo.toml ./utils/celestia/client/
COPY utils/celestia/host/Cargo.toml ./utils/celestia/host/
COPY utils/eigenda/client/Cargo.toml ./utils/eigenda/client/
COPY utils/eigenda/host/Cargo.toml ./utils/eigenda/host/
COPY utils/proof/Cargo.toml ./utils/proof/
COPY utils/signer/Cargo.toml ./utils/signer/
COPY utils/ethereum/client/Cargo.toml ./utils/ethereum/client/
COPY utils/ethereum/host/Cargo.toml ./utils/ethereum/host/
COPY utils/elfs/Cargo.toml ./utils/elfs/
COPY programs/range/ethereum/Cargo.toml ./programs/range/ethereum/
COPY programs/range/celestia/Cargo.toml ./programs/range/celestia/
COPY programs/range/eigenda/Cargo.toml ./programs/range/eigenda/
COPY programs/range/utils/Cargo.toml ./programs/range/utils/
COPY programs/aggregation/Cargo.toml ./programs/aggregation/
COPY scripts/prove/Cargo.toml ./scripts/prove/
COPY scripts/utils/Cargo.toml ./scripts/utils/
COPY validity/Cargo.toml ./validity/
COPY bindings/Cargo.toml ./bindings/

# Copy elf binaries needed by utils/elfs at compile time (needed before building)
COPY elf/ ./elf/

# Create dummy source files to satisfy Cargo's dependency resolution
# This allows us to download and compile dependencies without the actual source
RUN mkdir -p fault-proof/src fault-proof/bin && \
mkdir -p utils/client/src utils/host/src utils/build/src && \
mkdir -p utils/celestia/client/src utils/celestia/host/src && \
mkdir -p utils/eigenda/client/src utils/eigenda/host/src && \
mkdir -p utils/proof/src utils/signer/src && \
mkdir -p utils/ethereum/client/src utils/ethereum/host/src && \
mkdir -p utils/elfs/src && \
mkdir -p programs/range/ethereum/src programs/range/celestia/src && \
mkdir -p programs/range/eigenda/src programs/range/utils/src && \
mkdir -p programs/aggregation/src && \
mkdir -p scripts/prove/src scripts/utils/src && \
mkdir -p validity/src bindings/src && \
echo "fn main() {}" > fault-proof/bin/proposer.rs && \
echo "fn main() {}" > fault-proof/bin/challenger.rs && \
echo "" > fault-proof/src/lib.rs && \
echo "" > utils/client/src/lib.rs && \
echo "" > utils/host/src/lib.rs && \
echo "" > utils/build/src/lib.rs && \
echo "" > utils/celestia/client/src/lib.rs && \
echo "" > utils/celestia/host/src/lib.rs && \
echo "" > utils/eigenda/client/src/lib.rs && \
echo "" > utils/eigenda/host/src/lib.rs && \
echo "" > utils/proof/src/lib.rs && \
echo "" > utils/signer/src/lib.rs && \
echo "" > utils/ethereum/client/src/lib.rs && \
echo "" > utils/ethereum/host/src/lib.rs && \
echo "" > utils/elfs/src/lib.rs && \
echo "" > programs/range/ethereum/src/lib.rs && \
echo "" > programs/range/celestia/src/lib.rs && \
echo "" > programs/range/eigenda/src/lib.rs && \
echo "" > programs/range/utils/src/lib.rs && \
echo "" > programs/aggregation/src/lib.rs && \
echo "" > scripts/prove/src/lib.rs && \
echo "" > scripts/utils/src/lib.rs && \
echo "" > validity/src/lib.rs && \
echo "" > bindings/src/lib.rs

# Download and compile dependencies (this layer will be cached if Cargo.toml/Cargo.lock don't change)
# Use BuildKit cache mounts for faster Cargo operations
# The || true allows this to fail gracefully since we have dummy source files
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/app/target \
cargo build --release --bin challenger --features eigenda || true

# Now copy the actual source code
COPY fault-proof/src ./fault-proof/src
COPY fault-proof/bin ./fault-proof/bin
COPY utils/ ./utils/
COPY programs/ ./programs/
COPY scripts/ ./scripts/
COPY validity/src ./validity/src
COPY bindings/src ./bindings/src

# Build the actual binary (only rebuilds if source changed)
# Use BuildKit cache mounts for faster builds
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/app/target \
cargo build --release --bin challenger --features eigenda

# Copy binary from cache mount to filesystem so it can be copied to runtime stage
RUN --mount=type=cache,target=/app/target \
cp /app/target/release/challenger /app/challenger
# build
RUN cargo build --release --bin challenger --features eigenda

# Runtime stage (minimal image - only runtime dependencies)
FROM debian:trixie-slim
Expand All @@ -137,7 +62,7 @@ RUN apt-get update && apt-get install -y \
COPY resources/ ./resources/

# Copy the built challenger binary (copied from cache mount to filesystem)
COPY --from=builder /app/challenger /usr/local/bin/
COPY --from=builder /app/target/release/challenger /usr/local/bin/

# Set the command
CMD ["challenger"]
123 changes: 24 additions & 99 deletions fault-proof/Dockerfile.proposer.celo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,108 +19,33 @@ RUN curl -L https://sp1.succinct.xyz | bash && \
~/.sp1/bin/sp1up && \
~/.sp1/bin/cargo-prove prove --version

RUN cargo install cargo-chef


# Construct the project skeleton dynamically with
# cargo-chef (in order to cache dependency builds later)
FROM base AS planner

COPY . .

RUN cargo chef prepare --bin proposer --recipe-path recipe.json

RUN cat recipe.json

# build dependencies and binary
FROM base AS builder

WORKDIR /app
# import the skeleton recipe
COPY --from=planner /app/recipe.json recipe.json

# only build dependencies with cargo-chef
RUN cargo chef cook --release --features eigenda --bin proposer --recipe-path recipe.json

# now copy the actual source code
COPY . .

# Copy workspace root Cargo files
COPY Cargo.toml Cargo.lock ./

# Copy all Cargo.toml files from workspace members
# This allows Cargo to resolve dependencies without source
COPY fault-proof/Cargo.toml ./fault-proof/
COPY utils/client/Cargo.toml ./utils/client/
COPY utils/host/Cargo.toml ./utils/host/
COPY utils/build/Cargo.toml ./utils/build/
COPY utils/celestia/client/Cargo.toml ./utils/celestia/client/
COPY utils/celestia/host/Cargo.toml ./utils/celestia/host/
COPY utils/eigenda/client/Cargo.toml ./utils/eigenda/client/
COPY utils/eigenda/host/Cargo.toml ./utils/eigenda/host/
COPY utils/proof/Cargo.toml ./utils/proof/
COPY utils/signer/Cargo.toml ./utils/signer/
COPY utils/ethereum/client/Cargo.toml ./utils/ethereum/client/
COPY utils/ethereum/host/Cargo.toml ./utils/ethereum/host/
COPY utils/elfs/Cargo.toml ./utils/elfs/
COPY programs/range/ethereum/Cargo.toml ./programs/range/ethereum/
COPY programs/range/celestia/Cargo.toml ./programs/range/celestia/
COPY programs/range/eigenda/Cargo.toml ./programs/range/eigenda/
COPY programs/range/utils/Cargo.toml ./programs/range/utils/
COPY programs/aggregation/Cargo.toml ./programs/aggregation/
COPY scripts/prove/Cargo.toml ./scripts/prove/
COPY scripts/utils/Cargo.toml ./scripts/utils/
COPY validity/Cargo.toml ./validity/
COPY bindings/Cargo.toml ./bindings/

# Copy elf binaries needed by utils/elfs at compile time (needed before building)
COPY elf/ ./elf/

# Create dummy source files to satisfy Cargo's dependency resolution
# This allows us to download and compile dependencies without the actual source
RUN mkdir -p fault-proof/src fault-proof/bin && \
mkdir -p utils/client/src utils/host/src utils/build/src && \
mkdir -p utils/celestia/client/src utils/celestia/host/src && \
mkdir -p utils/eigenda/client/src utils/eigenda/host/src && \
mkdir -p utils/proof/src utils/signer/src && \
mkdir -p utils/ethereum/client/src utils/ethereum/host/src && \
mkdir -p utils/elfs/src && \
mkdir -p programs/range/ethereum/src programs/range/celestia/src && \
mkdir -p programs/range/eigenda/src programs/range/utils/src && \
mkdir -p programs/aggregation/src && \
mkdir -p scripts/prove/src scripts/utils/src && \
mkdir -p validity/src bindings/src && \
echo "fn main() {}" > fault-proof/bin/proposer.rs && \
echo "fn main() {}" > fault-proof/bin/challenger.rs && \
echo "" > fault-proof/src/lib.rs && \
echo "" > utils/client/src/lib.rs && \
echo "" > utils/host/src/lib.rs && \
echo "" > utils/build/src/lib.rs && \
echo "" > utils/celestia/client/src/lib.rs && \
echo "" > utils/celestia/host/src/lib.rs && \
echo "" > utils/eigenda/client/src/lib.rs && \
echo "" > utils/eigenda/host/src/lib.rs && \
echo "" > utils/proof/src/lib.rs && \
echo "" > utils/signer/src/lib.rs && \
echo "" > utils/ethereum/client/src/lib.rs && \
echo "" > utils/ethereum/host/src/lib.rs && \
echo "" > utils/elfs/src/lib.rs && \
echo "" > programs/range/ethereum/src/lib.rs && \
echo "" > programs/range/celestia/src/lib.rs && \
echo "" > programs/range/eigenda/src/lib.rs && \
echo "" > programs/range/utils/src/lib.rs && \
echo "" > programs/aggregation/src/lib.rs && \
echo "" > scripts/prove/src/lib.rs && \
echo "" > scripts/utils/src/lib.rs && \
echo "" > validity/src/lib.rs && \
echo "" > bindings/src/lib.rs

# Download and compile dependencies (this layer will be cached if Cargo.toml/Cargo.lock don't change)
# Use BuildKit cache mounts for faster Cargo operations
# The || true allows this to fail gracefully since we have dummy source files
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/app/target \
cargo build --release --bin proposer --features eigenda || true

# Now copy the actual source code
COPY fault-proof/src ./fault-proof/src
COPY fault-proof/bin ./fault-proof/bin
COPY utils/ ./utils/
COPY programs/ ./programs/
COPY scripts/ ./scripts/
COPY validity/src ./validity/src
COPY bindings/src ./bindings/src

# Build the actual binary (only rebuilds if source changed)
# Use BuildKit cache mounts for faster builds
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/app/target \
cargo build --release --bin proposer --features eigenda

# Copy binary from cache mount to filesystem so it can be copied to runtime stage
RUN --mount=type=cache,target=/app/target \
cp /app/target/release/proposer /app/proposer
# incrementally build the
RUN cargo build --release --bin proposer --features eigenda

# Runtime stage (minimal image - only runtime dependencies)
FROM debian:trixie-slim
Expand All @@ -137,7 +62,7 @@ RUN apt-get update && apt-get install -y \
COPY resources/ ./resources/

# Copy the built proposer binary (copied from cache mount to filesystem)
COPY --from=builder /app/proposer /usr/local/bin/
COPY --from=builder /app/target/release/proposer /usr/local/bin/

# Set the command
CMD ["proposer"]
1 change: 1 addition & 0 deletions fault-proof/src/challenger.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ where
/// cached state, and then handles challenging, resolution, and bond-claiming tasks.
pub async fn run(&mut self) -> Result<()> {
tracing::info!("OP Succinct Lite Challenger running...");
tracing::debug!("... and running...");
if self.config.disable_monitor_only {
if self.config.malicious_challenge_percentage > 0.0 {
tracing::warn!(
Expand Down
1 change: 1 addition & 0 deletions fault-proof/src/proposer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,7 @@ where
/// Runs the proposer indefinitely.
pub async fn run(self: Arc<Self>) -> Result<()> {
tracing::info!("OP Succinct Proposer running...");
tracing::debug!("... and running...");
let mut interval = time::interval(Duration::from_secs(self.config.fetch_interval));

// Spawn a dedicated task for continuous metrics collection
Expand Down
Loading