Skip to content

MIGRATION_GUIDE.md

Latest commit

 

History

History
652 lines (476 loc) · 23.6 KB

MIGRATION_GUIDE.md

File metadata and controls

652 lines (476 loc) · 23.6 KB

Preprod Migration Guide

Guide for deploying or upgrading Midnight contracts/dApps to the Preprod network.

1. Compact Compiler

Required Version

  • Compact version manager (compact): 0.4.0
  • Compiler version: 0.28.0 (installed and invoked via the version manager)

Important: You do not invoke compactc directly. The compact version manager finds and runs the correct compiler version. All compilation uses compact compile (or npm run compact in this project).

Clean Install (recommended)

Remove any existing Compact installations first:

rm -rf ~/.compact
rm -f ~/.local/bin/compact

Install

  1. Install the version manager:
curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/midnightntwrk/compact/releases/download/compact-v0.4.0/compact-installer.sh | sh
  1. Add to PATH:
source $HOME/.local/bin/env
  1. Install the compiler version:
compact update 0.28.0

Verify

compact --version    # expect: compact 0.4.0
compact list         # should show → 0.28.0 as the selected version

Known Issues

Issue Cause Fix
compact: command not found PATH not updated after install Run source $HOME/.local/bin/env
Old version still resolving Stale installations in /usr/local/bin or ~/.compact Remove old installations before installing (see Clean Install)

2. Contract — Pragma Update

What Changed

Compact compiler 0.28.0 uses language version 0.20.0. Contracts targeting older language versions (e.g. >= 0.16 && <= 0.18) will fail to compile.

Required Change

Update the pragma in your .compact file(s):

- pragma language_version >= 0.16 && <= 0.18;
+ pragma language_version >= 0.20;

Compile

compact compile src/counter.compact src/managed/counter

Expected output:

Compiling 1 circuits:

Compiled Artifacts

After a successful compile, src/managed/counter/ will contain:

contract/   — index.js, index.d.ts (TypeScript bindings)
keys/       — increment.prover, increment.verifier (ZK keys)
zkir/       — increment.zkir, increment.bzkir (ZK circuit IR)

Known Issues

Issue Cause Fix
language version 0.20.0 mismatch Pragma range doesn't include 0.20.0 Update pragma to >= 0.20

3. JS Dependencies — midnight-js 3.0.0

What Changed

The midnight-js framework packages were upgraded from 2.0.x to 3.0.0. This brings new types from @midnight-ntwrk/ledger-v7 and a new CompiledContract pattern from @midnight-ntwrk/compact-js.

Package Updates

  "@midnight-ntwrk/compact-runtime": "0.14.0",
  "@midnight-ntwrk/ledger": "^4.0.0",
- "@midnight-ntwrk/midnight-js-contracts": "2.0.2",
- "@midnight-ntwrk/midnight-js-http-client-proof-provider": "2.0.2",
- "@midnight-ntwrk/midnight-js-indexer-public-data-provider": "2.0.2",
- "@midnight-ntwrk/midnight-js-level-private-state-provider": "2.0.2",
- "@midnight-ntwrk/midnight-js-network-id": "2.0.2",
- "@midnight-ntwrk/midnight-js-node-zk-config-provider": "2.0.2",
- "@midnight-ntwrk/midnight-js-types": "2.0.2",
+ "@midnight-ntwrk/midnight-js-contracts": "3.0.0",
+ "@midnight-ntwrk/midnight-js-http-client-proof-provider": "3.0.0",
+ "@midnight-ntwrk/midnight-js-indexer-public-data-provider": "3.0.0",
+ "@midnight-ntwrk/midnight-js-level-private-state-provider": "3.0.0",
+ "@midnight-ntwrk/midnight-js-network-id": "3.0.0",
+ "@midnight-ntwrk/midnight-js-node-zk-config-provider": "3.0.0",
+ "@midnight-ntwrk/midnight-js-types": "3.0.0",

CompiledContract Pattern

midnight-js 3.0.0 uses CompiledContract from @midnight-ntwrk/compact-js instead of passing raw contract objects. The contract must be piped through withVacantWitnesses (if no witnesses) and withCompiledFileAssets to resolve the context type to never:

import { CompiledContract } from '@midnight-ntwrk/compact-js';
import { Counter } from '@midnight-ntwrk/counter-contract';

const counterCompiledContract = CompiledContract.make('counter', Counter.Contract).pipe(
  CompiledContract.withVacantWitnesses,
  CompiledContract.withCompiledFileAssets(contractConfig.zkConfigPath),
);

This counterCompiledContract is then passed to deployContract and findDeployedContract via the compiledContract field.

Circuit Type Changes

Circuit IDs are now branded types (ImpureCircuitId) from @midnight-ntwrk/compact-js, not plain strings:

import type { ImpureCircuitId } from '@midnight-ntwrk/compact-js';

export type CounterCircuits = ImpureCircuitId<Counter.Contract<CounterPrivateState>>;

Contract Test Changes

The Simulator from @midnight-ntwrk/compact-runtime now uses a different constructor pattern:

import { Simulator } from '@midnight-ntwrk/compact-runtime';

// Old:
const sim = new Simulator(witnesses);
const ledgerState = sim.ledger(Counter.initialState(new Uint8Array(32)));

// New:
const sim = Simulator.make(Counter.Contract, witnesses);
const ledgerState = sim.state('counter').data;

4. Wallet SDK — wallet-sdk-facade Migration

What Changed

The monolithic @midnight-ntwrk/wallet + @midnight-ntwrk/wallet-api packages are replaced by the new modular wallet SDK ecosystem. This eliminates the need for @midnight-ntwrk/zswap type bridging and most as any casts, since the new wallet uses ledger-v7 types natively.

Package Changes

- "@midnight-ntwrk/wallet": "5.0.0",
- "@midnight-ntwrk/wallet-api": "5.0.0",
- "@midnight-ntwrk/zswap": "^4.0.0",
+ "@midnight-ntwrk/wallet-sdk-facade": "1.0.0",
+ "@midnight-ntwrk/wallet-sdk-hd": "3.0.0",
+ "@midnight-ntwrk/wallet-sdk-shielded": "1.0.0",
+ "@midnight-ntwrk/wallet-sdk-dust-wallet": "1.0.0",
+ "@midnight-ntwrk/wallet-sdk-unshielded-wallet": "1.0.0",
+ "@midnight-ntwrk/wallet-sdk-address-format": "3.0.0",

Architecture

The new wallet is composed of three sub-wallets orchestrated by WalletFacade:

  • ShieldedWallet — handles ZK-shielded transactions (zswap)
  • UnshieldedWallet — handles transparent transactions
  • DustWallet — handles dust (fee) transactions

Each sub-wallet has its own configuration requirements.

Key Derivation

Replace randomBytes-based seed generation with HD wallet key derivation:

import { HDWallet, Roles, generateRandomSeed } from '@midnight-ntwrk/wallet-sdk-hd';
import { toHex } from '@midnight-ntwrk/midnight-js-utils';
import { Buffer } from 'buffer';

const seed = toHex(Buffer.from(generateRandomSeed()));
const hdWallet = HDWallet.fromSeed(Buffer.from(seed, 'hex'));
if (hdWallet.type !== 'seedOk') throw new Error('Failed to initialize HDWallet from seed');

const derivationResult = hdWallet.hdWallet
  .selectAccount(0)
  .selectRoles([Roles.Zswap, Roles.NightExternal, Roles.Dust])
  .deriveKeysAt(0);

if (derivationResult.type !== 'keysDerived') throw new Error('Failed to derive keys');
hdWallet.hdWallet.clear();
const keys = derivationResult.keys;

Wallet Initialization

Using the keys derived above, initialize the three sub-wallets and the facade:

import { Roles } from '@midnight-ntwrk/wallet-sdk-hd';
import * as ledger from '@midnight-ntwrk/ledger-v7';
import { WalletFacade } from '@midnight-ntwrk/wallet-sdk-facade';
import { ShieldedWallet } from '@midnight-ntwrk/wallet-sdk-shielded';
import { DustWallet } from '@midnight-ntwrk/wallet-sdk-dust-wallet';
import {
  createKeystore, InMemoryTransactionHistoryStorage,
  PublicKey, UnshieldedWallet,
} from '@midnight-ntwrk/wallet-sdk-unshielded-wallet';

// `keys` comes from derivationResult.keys in the Key Derivation step above
const shieldedSecretKeys = ledger.ZswapSecretKeys.fromSeed(keys[Roles.Zswap]);
const dustSecretKey = ledger.DustSecretKey.fromSeed(keys[Roles.Dust]);
const unshieldedKeystore = createKeystore(keys[Roles.NightExternal], getNetworkId());

const shieldedWallet = ShieldedWallet(shieldedConfig).startWithSecretKeys(shieldedSecretKeys);
const unshieldedWallet = UnshieldedWallet(unshieldedConfig).startWithPublicKey(
  PublicKey.fromKeyStore(unshieldedKeystore),
);
const dustWallet = DustWallet(dustConfig).startWithSecretKey(
  dustSecretKey,
  ledger.LedgerParameters.initialParameters().dust,
);

const wallet = new WalletFacade(shieldedWallet, unshieldedWallet, dustWallet);
await wallet.start(shieldedSecretKeys, dustSecretKey);

Sub-Wallet Configuration

Each sub-wallet requires different config fields:

ShieldedWallet:

{
  networkId: getNetworkId(),
  indexerClientConnection: { indexerHttpUrl, indexerWsUrl },
  provingServerUrl: new URL(proofServer),
  relayURL: new URL(node.replace(/^http/, 'ws')),
}

UnshieldedWallet:

{
  networkId: getNetworkId(),
  indexerClientConnection: { indexerHttpUrl, indexerWsUrl },
  txHistoryStorage: new InMemoryTransactionHistoryStorage(),
}

DustWallet:

{
  networkId: getNetworkId(),
  costParameters: { additionalFeeOverhead: 300_000_000_000_000n, feeBlocksMargin: 5 },
  indexerClientConnection: { indexerHttpUrl, indexerWsUrl },
  provingServerUrl: new URL(proofServer),
  relayURL: new URL(node.replace(/^http/, 'ws')),
}

WalletProvider Bridge — signRecipe Bug Workaround

The WalletProvider / MidnightProvider interface used by midnight-js is bridged from the facade. However, wallet.signRecipe() has a bug that causes "Failed to clone intent" errors when signing proven transactions.

The Bug

In @midnight-ntwrk/wallet-sdk-unshielded-wallet v1.0.0, TransactionOps.addSignature() clones intents via Intent.deserialize() with a hardcoded 'pre-proof' type marker. This works for UnprovenTransaction (which has PreProof intents), but fails for UnboundTransaction (which has Proof intents after proveTx()). The WASM deserializer cannot parse proof-format bytes using the pre-proof marker.

The bug is in TransactionOps.ts line ~101:

// BUG: hardcoded 'pre-proof' fails for proven (UnboundTransaction) intents
ledger.Intent.deserialize<ledger.SignatureEnabled, ledger.PreProof, ledger.PreBinding>(
  'signature',
  'pre-proof',  // ← should be 'proof' for UnboundTransaction
  'pre-binding',
  originalIntent.serialize(),
)

Side Effect: DUST Locked on Failure

When this error occurs, DUST coins allocated during balanceUnboundTransaction() become stuck in a "pending" state. The wallet SDK does not release pending coins on transaction failure (documented known issue in wallet SDK 1.0.0). The wallet must be restarted to recover them.

The Workaround

Instead of calling wallet.signRecipe(), implement signing manually with the correct proof marker for each transaction type:

import * as ledger from '@midnight-ntwrk/ledger-v7';

/**
 * Sign all unshielded offers in a transaction's intents, using the correct
 * proof marker for Intent.deserialize.
 */
const signTransactionIntents = (
  tx: { intents?: Map<number, any> },
  signFn: (payload: Uint8Array) => ledger.Signature,
  proofMarker: 'proof' | 'pre-proof',
): void => {
  if (!tx.intents || tx.intents.size === 0) return;

  for (const segment of tx.intents.keys()) {
    const intent = tx.intents.get(segment);
    if (!intent) continue;

    const cloned = ledger.Intent.deserialize(
      'signature',
      proofMarker,   // Use the correct marker for the transaction type
      'pre-binding',
      intent.serialize(),
    );

    const sigData = cloned.signatureData(segment);
    const signature = signFn(sigData);

    if (cloned.fallibleUnshieldedOffer) {
      const sigs = cloned.fallibleUnshieldedOffer.inputs.map(
        (_: any, i: number) => cloned.fallibleUnshieldedOffer!.signatures.at(i) ?? signature,
      );
      cloned.fallibleUnshieldedOffer = cloned.fallibleUnshieldedOffer.addSignatures(sigs);
    }

    if (cloned.guaranteedUnshieldedOffer) {
      const sigs = cloned.guaranteedUnshieldedOffer.inputs.map(
        (_: any, i: number) => cloned.guaranteedUnshieldedOffer!.signatures.at(i) ?? signature,
      );
      cloned.guaranteedUnshieldedOffer = cloned.guaranteedUnshieldedOffer.addSignatures(sigs);
    }

    tx.intents.set(segment, cloned);
  }
};

// In balanceTx:
async balanceTx(tx, ttl?) {
  const recipe = await wallet.balanceUnboundTransaction(tx, keys, { ttl });

  const signFn = (payload: Uint8Array) => unshieldedKeystore.signData(payload);
  signTransactionIntents(recipe.baseTransaction, signFn, 'proof');       // proven tx
  if (recipe.balancingTransaction) {
    signTransactionIntents(recipe.balancingTransaction, signFn, 'pre-proof'); // wallet-created tx
  }

  return wallet.finalizeRecipe(recipe);
}

Key insight: After proveTx(), the base transaction is an UnboundTransaction with Proof intents (use 'proof'). The balancing transaction created by the wallet is an UnprovenTransaction with PreProof intents (use 'pre-proof').

Token Type — Critical Breaking Change

The old nativeToken() from @midnight-ntwrk/ledger (v4) returns a tagged 68-character hex token type (02000000...0000). The new wallet SDK stores balances keyed by raw 64-character hex token types from @midnight-ntwrk/ledger-v7.

If you use the wrong token type for balance lookups, the wallet will appear to have zero balance even when funds are present.

- import { nativeToken } from '@midnight-ntwrk/ledger';
- const balance = state.unshielded.balances[nativeToken()];
+ import { unshieldedToken } from '@midnight-ntwrk/ledger-v7';
+ const balance = state.unshielded.balances[unshieldedToken().raw];

Debugging tip: If your wallet shows Synced: true but zero balance, log state.unshielded.balances and compare the actual key (0000...0000, 64 chars) against what your lookup function returns. If you see 02000...0000 (68 chars), you're still using the old nativeToken().

State Shape Changes

- state.syncProgress?.synced        → state.isSynced
- state.balances[nativeToken()]     → state.unshielded.balances[unshieldedToken().raw]

Lifecycle Changes

- wallet.close()  → wallet.stop()
- Wallet & Resource type → WalletContext (custom type bundling wallet + keys)

Removed APIs

  • saveState / wallet.serialize() — serialization not yet supported in facade 1.0.0
  • WalletBuilder — replaced by direct sub-wallet construction
  • toZswapNetworkId() — no longer needed, facade uses string network IDs natively

Private State Provider — Encryption Required

levelPrivateStateProvider now requires either a walletProvider or privateStoragePasswordProvider for encrypting private state storage. Passing neither throws an error at runtime.

  privateStateProvider: levelPrivateStateProvider<typeof CounterPrivateStateId>({
    privateStateStoreName: contractConfig.privateStateStoreName,
+   walletProvider: walletAndMidnightProvider,
  }),

You must provide exactly one of:

  • walletProvider — uses the wallet's encryption public key (recommended when using wallet-sdk-facade)
  • privateStoragePasswordProvider: () => string — a function returning a custom password (min 16 chars)

Providing both will also throw an error.

Address Formatting

The wallet SDK provides bech32m-encoded addresses for all wallet types via @midnight-ntwrk/wallet-sdk-address-format:

import {
  MidnightBech32m, ShieldedAddress,
  ShieldedCoinPublicKey, ShieldedEncryptionPublicKey,
} from '@midnight-ntwrk/wallet-sdk-address-format';

// Shielded address (mn_shield-addr_<network>1...)
const coinPubKey = ShieldedCoinPublicKey.fromHexString(state.shielded.coinPublicKey.toHexString());
const encPubKey = ShieldedEncryptionPublicKey.fromHexString(state.shielded.encryptionPublicKey.toHexString());
const shieldedAddr = MidnightBech32m.encode(networkId, new ShieldedAddress(coinPubKey, encPubKey)).toString();

// Unshielded address (mn_addr_<network>1...)
const unshieldedAddr = unshieldedKeystore.getBech32Address();

// Dust address (mn_dust_<network>1...)
const dustAddr = state.dust.dustAddress;

Important: UnshieldedKeystore uses getBech32Address() (a method), not .address (a property).

Address format reference:

Type Prefix Example
Shielded mn_shield-addr_<network>1... mn_shield-addr_preprod1q...
Unshielded mn_addr_<network>1... mn_addr_preprod1q...
Dust mn_dust_<network>1... mn_dust_preprod1w...

Dust Registration for Fee Generation

On Preprod/Preview, NIGHT tokens generate DUST over time, but only after UTXOs are explicitly registered for dust generation via an on-chain transaction. This must happen before any contract deployment or interaction.

The registration flow:

  1. Check if dust is already available from a previous session
  2. Filter for unregistered NIGHT coins
  3. Call wallet.registerNightUtxosForDustGeneration() with the coins and signing function
  4. Wait for the wallet to report a non-zero dust balance

DUST balance accrues over time once registered. Initial DUST generation may take a minute or two.

Known Issues

Issue Cause Fix
Failed to clone intent during deploy/call signRecipe uses hardcoded 'pre-proof' marker for all intents, but proven transactions have 'proof' intents Bypass signRecipe and sign manually with correct proof markers — see workaround above
DUST balance drops to 0 after failed transaction Pending coins not released on failure (wallet SDK 1.0.0 known issue) Restart the wallet to recover pending DUST
Wallet shows zero balance despite receiving funds Using nativeToken() (68-char tagged) instead of unshieldedToken().raw (64-char raw) for balance lookup Replace nativeToken() with unshieldedToken().raw from @midnight-ntwrk/ledger-v7
Either privateStoragePasswordProvider or walletProvider must be provided levelPrivateStateProvider now requires encryption config Pass walletProvider or privateStoragePasswordProvider in config
Cannot find package '@midnight-ntwrk/wallet-sdk-address-format' Transitive dep not hoisted Add @midnight-ntwrk/wallet-sdk-address-format as direct dependency
Unknown field "zswapLedgerEvents" at runtime Local standalone indexer too old for new wallet SDK Update docker images to versions compatible with wallet-sdk-facade
Unknown field "dustLedgerEvents" at runtime Same as above Same — update indexer docker images
Unknown type "UnshieldedAddress" at runtime Same as above Same — update indexer docker images

5. Docker Infrastructure — Image Updates

What Changed

The wallet-sdk-facade and midnight-js 3.0.0 require updated docker images with GraphQL v3 schema support. The indexer now exposes subscription fields for zswapLedgerEvents, dustLedgerEvents, and unshieldedTransactions that the new wallet SDK depends on for syncing.

Docker Image Updates (standalone.yml)

- image: "midnightnetwork/proof-server:4.0.0"
+ image: 'midnightntwrk/proof-server:7.0.0'

- image: 'midnightntwrk/indexer-standalone:2.1.1'
+ image: 'midnightntwrk/indexer-standalone:3.0.0'

- image: 'midnightnetwork/midnight-node:0.12.0'
+ image: 'midnightntwrk/midnight-node:0.20.0'

Note: The registry consolidated from midnightnetwork/ to midnightntwrk/ on Docker Hub.

For Preprod/Preview deployments where you only need a local proof server (indexer and node are remote):

# proof-server.yml
services:
  proof-server:
    image: 'midnightntwrk/proof-server:7.0.0'
    command: ['midnight-proof-server -v']
    ports:
      - '6300:6300'
    environment:
      RUST_BACKTRACE: 'full'

Proof Server — CLI Flag Changes

The proof server 7.0.0 no longer accepts the --network flag. Use -v for verbose mode only:

- command: ["midnight-proof-server", "--network", "testnet"]
+ command: ["midnight-proof-server -v"]

New Indexer Environment Variables

The indexer 3.0.0 requires additional config:

environment:
  APP__APPLICATION__NETWORK_ID: 'undeployed'
  APP__INFRA__STORAGE__PASSWORD: 'indexer'
  APP__INFRA__PUB_SUB__PASSWORD: 'indexer'
  APP__INFRA__LEDGER_STATE_STORAGE__PASSWORD: 'indexer'

Healthchecks

Updated healthcheck for proof-server and indexer:

# Proof server
healthcheck:
  test: ['CMD', 'curl', '-f', 'http://localhost:6300/version']

# Indexer
healthcheck:
  test: ['CMD-SHELL', 'cat /var/run/indexer-standalone/running']

GraphQL API Path

The indexer GraphQL endpoint path changed from v1 to v3:

- /api/v1/graphql
+ /api/v3/graphql
- /api/v1/graphql/ws
+ /api/v3/graphql/ws

6. Network Configuration

What Changed

Network configs were updated to use the new GraphQL v3 paths and target the Preview and Preprod networks instead of the old testnet.

Network Endpoints

Local (Standalone):

Service Endpoint
Indexer HTTP http://127.0.0.1:8088/api/v3/graphql
Indexer WS ws://127.0.0.1:8088/api/v3/graphql/ws
Node http://127.0.0.1:9944
Proof Server http://127.0.0.1:6300
NetworkId undeployed

Preview:

Service Endpoint
RPC Node https://rpc.preview.midnight.network
Indexer HTTP https://indexer.preview.midnight.network/api/v3/graphql
Indexer WS wss://indexer.preview.midnight.network/api/v3/graphql/ws
Faucet https://faucet.preview.midnight.network
Proof Server http://localhost:6300 (local)
NetworkId preview

Preprod:

Service Endpoint
RPC Node https://rpc.preprod.midnight.network
Indexer HTTP https://indexer.preprod.midnight.network/api/v3/graphql
Indexer WS wss://indexer.preprod.midnight.network/api/v3/graphql/ws
Faucet https://faucet.preprod.midnight.network
Proof Server http://localhost:6300 (local)
NetworkId preprod

Config Class Changes

- TestnetLocalConfig   (testnet, local indexer at v1)
- TestnetRemoteConfig  (testnet, remote indexer at v1)
+ PreviewConfig        (preview network, v3 paths)
+ PreprodConfig        (preprod network, v3 paths)
  StandaloneConfig     (local docker, updated to v3 paths)

7. Preprod Deployment Checklist

Quick reference for deploying a DApp to Preprod:

  1. Proof server: Run locally via Docker (docker compose -f proof-server.yml up)
  2. Wallet: Create or restore from seed — the app connects to remote Preprod indexer and RPC
  3. Fund wallet: Send tNight to the unshielded address via https://faucet.preprod.midnight.network
  4. Wait for DUST: After funding, NIGHT UTXOs are registered for dust generation automatically. Wait for DUST to accrue before deploying.
  5. Deploy contract: Once DUST balance is non-zero, deploy your contract through the DApp

Common Pitfalls

Pitfall Resolution
Failed to clone intent during deploy Wallet SDK signing bug — bypass signRecipe() with manual signing using correct proof markers (see Section 4)
DUST drops to 0 after failed deploy Known wallet SDK issue — restart wallet to release pending coins
Wallet shows zero balance after faucet Ensure you're using unshieldedToken().raw (not nativeToken()) — see Section 4
Either privateStoragePasswordProvider or walletProvider must be provided Pass walletProvider to levelPrivateStateProvider — see Section 4
Proof server fails with "unexpected argument '--network'" Remove --network flag, use -v only — see Section 5
@midnight-ntwrk/wallet-sdk-address-format not found Add it as a direct dependency in package.json

Migration guide covering: Compact compiler, Contract pragma, JS dependencies (midnight-js 3.0.0), Wallet SDK (wallet-sdk-facade), Docker infrastructure, Network configuration, and Preprod deployment.