Skip to content

feat: add Audio Note Recording feature design and implementation (fixes #9) #111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
elevasyncsolutions-jpg wants to merge 1 commit into
base: main
Choose a base branch
from
+2,070 −0
Open

feat: add Audio Note Recording feature design and implementation (fixes #9) #111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions bounty-9/DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Audio Notes — Design Document

## Overview

Audio Notes brings full voice-recording, playback, transcription, and management capabilities to the warpSpeed Notes experience. Users can quickly capture voice memos, view automatic transcriptions, search across audio content, and manage recordings — all from a React Native mobile surface backed by a serverless API.

## Architecture

```
┌──────────────────────────┐ ┌──────────────────────────┐
│ React Native Client │ │ Cloudflare Workers API │
│ ┌────────────────────┐ │ │ ┌──────────────────┐ │
│ │ AudioRecorder │───┼─────┼──│ POST /upload │ │
│ │ AudioPlayer │ │ │ │ POST /transcribe │ │
│ │ TranscriptionView │ │ │ │ GET /notes │ │
│ └────────────────────┘ │ │ │ DELETE /notes/:id │ │
│ │ │ │ └──────────────────┘ │
│ ▼ │ │ │ │
│ ┌────────────────────┐ │ │ ▼ │
│ │ api.ts (client) │───┼─────┼───► R2 Storage │
│ └────────────────────┘ │ │ ┌──────────────────┐ │
│ │ │ │ audio/*.webm │ │
│ │ │ │ transcriptions/* │ │
│ │ │ └──────────────────┘ │
│ │ │ │ │
│ │ │ ▼ │
│ │ │ D1 (SQLite) │
│ │ │ ┌──────────────────┐ │
│ │ │ │ audio_notes │ │
│ │ │ │ transcripts │ │
│ │ │ └──────────────────┘ │
└──────────────────────────┘ └──────────────────────────┘
```

### Data Flow

1. **Recording** → `AudioRecorder` captures PCM/WAV → streams to `POST /upload` → stored in R2 → record inserted into D1.
2. **Transcription** → `POST /transcribe` triggers Whisper (Workers AI) → result stored in D1 → WebSocket push to client.
3. **Playback** → `AudioPlayer` requests signed URL from `GET /notes/:id/url` → streams audio via `expo-av`.
4. **Management** → `GET /notes` (paginated) and `DELETE /notes/:id` (cascades to R2 + D1).

## Data Model

### `audio_notes` table (D1)

| Column | Type | Description |
|-------------|----------|------------------------------------------|
| `id` | TEXT PK | UUID v7 |
| `title` | TEXT | User-facing title (nullable, auto-set) |
| `duration_ms`| INTEGER | Recording length in ms |
| `file_key` | TEXT | R2 object key (e.g. `audio/<uuid>.webm`) |
| `file_size` | INTEGER | Bytes |
| `mime_type` | TEXT | `audio/webm` or `audio/wav` |
| `created_at`| TEXT | ISO-8601 timestamp |
| `updated_at`| TEXT | ISO-8601 timestamp |
| `archived` | INTEGER | Soft-delete flag (0/1) |

### `transcripts` table (D1)

| Column | Type | Description |
|-------------|----------|------------------------------------------|
| `id` | TEXT PK | UUID v7 |
| `note_id` | TEXT FK | References `audio_notes.id` |
| `text` | TEXT | Full transcript |
| `segments` | TEXT | JSON array of `{start, end, text}` |
| `model` | TEXT | Whisper model variant (`base`, `large`) |
| `confidence`| REAL | 0.0–1.0 |
| `language` | TEXT | Detected language code (e.g. `en`) |
| `created_at`| TEXT | ISO-8601 timestamp |

## Component Tree

```
<AudioNotesScreen>
├── <AudioRecorder /> (record/stop/permissions)
├── <FlatList>
│ └── <AudioNoteCard>
│ ├── <AudioPlayer /> (play/pause/seek/scrub)
│ └── <TranscriptionView /> (collapsible transcript)
└── <FAB> (new recording)
```

## Key Decisions

| Decision | Rationale |
|---------|----------|
| **expo-av** for playback | Mature, cross-platform, supports streaming URLs |
| **expo-audio-record** (or Expo Audio API) | Stable recording API with PCM/WAV output |
| **Cloudflare Workers + R2 + D1** | Warpspeed's stack; edge-close, no cold starts for audio |
| **WebSocket for transcription status** | Whisper takes 1–10s; avoid polling |
| **Chunked upload** | Large recordings (>10 MB) streamed in 64 KB chunks |
| **Soft-delete** | Prevents accidental permanent data loss |
| **Signed URLs** | Avoids public R2 buckets; 1-hour expiry |

## Error States

- **Permission denied** → show grant-permission CTA with `Linking.openSettings()`
- **Upload failure** → retry with exponential backoff (3 attempts); save blob locally for later sync
- **Transcription failure** → show "Transcription failed — tap to retry" inline on the card
- **Network offline** → `NetInfo` detection; recordings saved to local filesystem; queued for upload
- **Storage full** → warn user before recording starts
- **Audio decode error** → fallback to "unsupported format" message

## Security & Privacy

- All audio files stored in private R2 bucket.
- Signed URLs expire in 60 minutes.
- Transcriptions never leave the user's org boundary.
- Recording requires explicit microphone permission (iOS `NSMicrophoneUsageDescription`, Android `RECORD_AUDIO`).
- `archived` flag respected in all list queries — no accidental data leaks.

## Performance Targets

- **Record start latency** < 200 ms from button press
- **Upload 1 min audio** < 5 s (on LTE)
- **Transcription (1 min audio)** < 10 s p50
- **Playback start** < 300 ms (signed URL + buffer)
- **List load (50 items)** < 500 ms (D1 query + signed URLs parallelized)
137 changes: 137 additions & 0 deletions bounty-9/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# Audio Notes — warpSpeed Bounty #9

[![Status](https://img.shields.io/badge/status-reference%20implementation-blue)]()

Full audio-note recording, playback, transcription, and management support for the warpSpeed Notes experience.

**Bounty:** $750
**Type:** Reference Implementation
**Stack:** React Native (Expo) + Cloudflare Workers + R2 + D1 + Workers AI (Whisper)

---

## What's Included

| File | Purpose |
|------|---------|
| `DESIGN.md` | Full architecture, data model, component tree, security/privacy, performance targets |
| `types.ts` | All TypeScript types: `AudioNote`, `RecordingState`, `PlaybackState`, `Transcription`, API DTOs, WebSocket events |
| `components/AudioRecorder.tsx` | Reusable recording component with permissions, timer, error handling, discard flow |
| `components/AudioPlayer.tsx` | Playback component with play/pause, seek ±15s, variable speed, mute, progress bar |
| `components/TranscriptionView.tsx` | Collapsible transcription display with segment list, language/confidence badges, low-confidence warning |
| `api.ts` | Full API client: CRUD notes, chunked upload, transcription (polling + WebSocket), signed URLs |

---

## Quick Start

```bash
# Install dependencies
npm install expo-av expo-file-system

# Copy the files into your project:
# cp -r components/ types.ts api.ts src/audio-notes/

# Initialize the API client (app entry point)
import { initApi } from './api';

initApi({
baseUrl: 'https://api.warpspeed.dev',
token: 'your-jwt-token',
});
```

## Usage

### Record an audio note

```tsx
import { AudioRecorder } from './components/AudioRecorder';
import { useApi } from './api';

function RecordScreen() {
const api = useApi();

return (
<AudioRecorder
onRecordingComplete={async (uri, durationMs, mimeType) => {
await api.createAndUpload(
{ durationMs, mimeType, fileSize: 0 },
uri,
(progress) => console.log('Upload:', progress),
);
}}
/>
);
}
```

### Play back an audio note

```tsx
import { AudioPlayer } from './components/AudioPlayer';

<AudioPlayer
source={playbackUrl}
initialRate={1}
onPlaybackComplete={() => console.log('Done')}
/>
```

### Display transcription

```tsx
import { TranscriptionView } from './components/TranscriptionView';

<TranscriptionView
transcription={transcription}
status={transcription?.status ?? 'pending'}
onRetry={() => api.transcribeNote(noteId)}
onSegmentPress={(start) => player.seek(start * 1000)}
/>
```

---

## API Endpoints (Server-Side stubs needed)

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/notes` | List notes (paginated, ?cursor, ?limit, ?includeArchived) |
| `POST` | `/notes` | Create note + return signed upload URL |
| `GET` | `/notes/:id` | Get single note |
| `DELETE` | `/notes/:id` | Soft-delete (archive) note |
| `PATCH` | `/notes/:id/archive` | Set archived flag |
| `PATCH` | `/notes/:id/restore` | Clear archived flag |
| `POST` | `/notes/:id/transcribe` | Trigger Whisper transcription |
| `GET` | `/notes/:id/transcription` | Get transcription result |
| `GET` | `/notes/:id/playback-url` | Get signed playback URL (1hr expiry) |
| `WS` | `/notes/:id/transcribe/ws` | WebSocket for real-time transcription updates |

### Worker Implementation Guidance

Refer to `DESIGN.md` for the D1 schema and R2 object layout. The server should:

1. Generate pre-signed R2 PUT URLs on `POST /notes`
2. Run Workers AI `@cf/openai/whisper` on `POST /notes/:id/transcribe`
3. Push transcription progress via WebSocket (Durable Objects or `ws` channel)
4. Generate pre-signed R2 GET URLs for `GET /notes/:id/playback-url`
5. List with cursor-based pagination from D1

---

## Testing

```bash
# Unit tests (add Jest + Testing Library)
npm test

# Lint
npx eslint components/ types.ts api.ts
```

---

## License

MIT — part of the warpSpeed Bounty Program.
Loading