Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,16 @@ about: Tell us about something that went wrong
title: ''
labels: bug
assignees: ''

---

Thanks for reporting a bug!

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
Expand Down
5 changes: 4 additions & 1 deletion .github/ISSUE_TEMPLATE/documentation-issue.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,22 @@ about: Report an issue found in the Documentation
title: ''
labels: bug, documentation
assignees: ''

---

### Where is the Issue?

Please link to the file(s) and line number(s) that has the issue(s)

### What is wrong with it?

Spelling/typo out of date, doesn't work, broken link, something missing, etc.

### Optional: Suggested change

...

### [PM/Eng] Acceptance Criteria

- [ ] list of clear directives
- [ ] that would enable someone
- [ ] to begin work
7 changes: 5 additions & 2 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,20 @@ about: Request additional Documentation, updates, features
title: ''
labels: enhancement
assignees: ''

---

Thanks for requesting an improvement!

### Context
Why is this important?

Why is this important?

### Outline or additional Details

Description of the feature/update/docs needed

### [PM/Eng]: Acceptance criteria

- [ ] list of clear directives
- [ ] that would enable someone
- [ ] to begin work
8 changes: 4 additions & 4 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ A short description of what you have done to implement/fix the above mentioned f

### Change summary

* A detailed list of bulleted
* changes that go into detail about
* the specifics of the changes
* to the codebase
- A detailed list of bulleted
- changes that go into detail about
- the specifics of the changes
- to the codebase

### Steps to Verify

Expand Down
4 changes: 1 addition & 3 deletions .spellcheckerrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,5 @@
"!CLAUDE.md",
"**/.github/**/*.md"
],
"dictionaries": [
".spellcheckerdict.txt"
]
"dictionaries": [".spellcheckerdict.txt"]
}
25 changes: 25 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,18 @@ This is the **Frequency Developer Gateway** - a NestJS monorepo containing micro
### Architecture

**Monorepo Structure:**

- **Apps** (9 services): Independent microservices with API endpoints and background workers
- **Libs** (16 libraries): Shared functionality and utilities

**Core Services:**

- **Account** (`account-api` + `account-worker`): User account management, authentication via Sign In With Frequency
- **Content Publishing** (`content-publishing-api` + `content-publishing-worker`): Content creation, file upload, IPFS storage, blockchain publishing
- **Content Watcher** (`content-watcher`): Blockchain monitoring for content changes

**Key Libraries:**

- **blockchain**: Polkadot API integration, Frequency chain interactions
- **queue**: BullMQ job processing with Redis
- **cache**: Redis caching layer
Expand All @@ -27,6 +30,7 @@ This is the **Frequency Developer Gateway** - a NestJS monorepo containing micro
## Development Commands

### Building

```bash
# Build all services
npm run build
Expand All @@ -39,6 +43,7 @@ npm run build:libs
```

### Running Services

```bash
# Start individual services in development mode
npm run start:account-api:dev
Expand All @@ -51,6 +56,7 @@ npm run start:content-publishing-worker:dev
```

### Testing

```bash
# Run all tests
npm test
Expand All @@ -75,6 +81,7 @@ jest path/to/test.spec.ts
```

### Linting and Formatting

```bash
# Lint all code
npm run lint
Expand All @@ -95,6 +102,7 @@ npm run spellcheck
```

### OpenAPI Documentation Generation

```bash
# Generate all OpenAPI specs
npm run generate:openapi
Expand All @@ -108,6 +116,7 @@ npm run generate:swagger-ui
```

### Load Testing

```bash
# Run k6 load tests
npm run test:k6:account
Expand All @@ -120,7 +129,9 @@ SCENARIO=light k6 run apps/content-publishing-api/k6-test/batch-announcement-loa
## Architecture Patterns

### Import Path Aliases

Uses `#` prefix for internal module resolution:

- `#account-api/*` → `apps/account-api/src/*`
- `#content-publishing-lib/*` → `libs/content-publishing-lib/src/*`
- `#blockchain` → `libs/blockchain/src`
Expand All @@ -129,33 +140,39 @@ Uses `#` prefix for internal module resolution:
- `#validation` → `libs/types/src/validation`

### Queue Architecture

Each service uses BullMQ with Redis for background job processing:

**Content Publishing Queues:**

- `REQUEST_QUEUE_NAME`: Incoming announcement requests
- `ASSET_QUEUE_NAME`: File upload processing
- `BATCH_QUEUE_NAME`: Batching announcements for efficiency
- `PUBLISH_QUEUE_NAME`: Publishing to blockchain

**Queue Processing Flow:**

1. API receives request → enqueues job
2. Worker processes job → transforms data
3. Publishes to blockchain via batch transactions
4. Monitors transaction status

### Service Communication

- **API Layer**: REST endpoints with OpenAPI/Swagger documentation
- **Worker Layer**: Background job processors using BullMQ
- **Blockchain Layer**: Polkadot API integration for Frequency chain
- **Storage Layer**: IPFS for content, Redis for caching, file system for temporary storage

### Configuration Management

- Environment-specific `.env` files for each service
- Template files in `env-files/` directory
- Joi validation schemas for configuration
- Service-specific config modules in each app

### Database/Storage Architecture

- **No traditional database** - leverages blockchain as source of truth
- **Redis**: Caching, job queues, temporary data storage
- **IPFS**: Persistent content storage
Expand All @@ -164,39 +181,46 @@ Each service uses BullMQ with Redis for background job processing:
## Testing Architecture

### Test Structure

- **Unit tests**: `.spec.ts` files alongside source code
- **E2E tests**: `.e2e-spec.ts` files in each app's test directory
- **Load tests**: k6 scripts in each app's `k6-test/` directory
- **Mocks**: Shared mocks in `__mocks__/` directory

### Module Path Mapping

Jest configuration maps `#` aliases to actual paths, matching TypeScript configuration.

## Key Development Practices

### Error Handling

- Use NestJS HTTP exceptions (`BadRequestException`, `InternalServerErrorException`)
- Blockchain errors are wrapped and handled gracefully
- Queue job failures use exponential backoff retry strategies

### Logging

- Uses Pino logger with structured JSON output
- Set `PRETTY=true` for human-readable development logs
- Log levels: `error`, `warn`, `info`, `debug`, `trace`

### Capacity Management

- Services check Frequency blockchain capacity before operations
- Automatic pausing/resuming of queues based on capacity
- Graceful handling of capacity exhaustion

### Transaction Monitoring

- All blockchain transactions are monitored for finality
- Status tracking via Redis with transaction hash keys
- Automatic cleanup of completed/failed transactions

## File Organization Patterns

### Apps Structure

```
apps/[service-name]/
├── src/
Expand All @@ -211,6 +235,7 @@ apps/[service-name]/
```

### Libs Structure

```
libs/[lib-name]/
├── src/
Expand Down
9 changes: 4 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ bridging the gap between Web2 and Web3 development.
- [🔍 Architecture Map](#-arch-maps)
- [🔍 Gateway Microservices](#gateway-microservices)
- [💻 Getting Started](#getting-started)
- [🚀 Quick Start Guide](#quick-start-guide)
- [💻 Getting Started with Microservices](#microservices-start-guide)
- [🚀 Quick Start Guide](#quick-start-guide)
- [💻 Getting Started with Microservices](#microservices-start-guide)
- [🛫 Deployment](#deployment)
- [📝 Logging](#logging)
- [📊 Metrics](#metrics)
Expand Down Expand Up @@ -125,7 +125,7 @@ Gateway consists of four independent microservices, each designed to handle spec
Frequency blockchain. Below is a detailed overview of each service:

| Service | Description | API Documentation | README |
|----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|---------------------------------------------------------|
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------- |
| Account Service | Manages user accounts and authentication on the Frequency blockchain using [Sign In With Frequency](https://github.com/ProjectLibertyLabs/siwf). It handles tasks such as account creation and key management. | [API Docs](https://projectlibertylabs.github.io/gateway/account) | [README](./developer-docs/account/README.md) |
| Content Publishing Service | Facilitates the creation and publication of content on the Frequency blockchain. It manages tasks such as posting messages, attachments, replies, and reactions. | [API Docs](https://projectlibertylabs.github.io/gateway/content-publishing/) | [README](./developer-docs/content-publishing/README.md) |
| Content Watcher Service | Monitors and retrieves content updates from the Frequency blockchain. It allows applications to efficiently track new content as it's published. | [API Docs](https://projectlibertylabs.github.io/gateway/content-watcher/) | [README](./developer-docs/content-watcher/README.md) |
Expand Down Expand Up @@ -274,8 +274,7 @@ cd apps/content-publishing-api/k6-test
SCENARIO=heavy k6 run batch-announcement-load.k6.js
```

Detailed information, configuration options, and best practices, are included in our *
*[k6 testing documentation](./docs/k6/k6-documentation.md)**.
Detailed information, configuration options, and best practices, are included in our \* \*[k6 testing documentation](./docs/k6/k6-documentation.md)\*\*.

<!-- CONTRIBUTING -->

Expand Down
26 changes: 8 additions & 18 deletions apps/account-api/src/controllers/v1/ics.controller.v1.ts
Original file line number Diff line number Diff line change
@@ -1,25 +1,15 @@
import { EnqueueService } from '#account-lib/services/enqueue-request.service';
import { AddNewPublicKeyAgreementRequestDto, IcsPublishAllRequestDto, UpsertPagePayloadDto } from '#types/dtos/account';
import { AccountIdDto } from '#types/dtos/common';
import {
Body,
Controller,
HttpCode,
HttpException,
HttpStatus,
Param,
Post,
} from '@nestjs/common';
import { Body, Controller, HttpCode, HttpException, HttpStatus, Param, Post } from '@nestjs/common';
import { ApiTags } from '@nestjs/swagger';
import { ApiPromise } from '@polkadot/api';
import { SubmittableExtrinsic } from '@polkadot/api/types';
import { ISubmittableResult } from '@polkadot/types/types';
import { HexString } from '@polkadot/util/types';
import { InjectPinoLogger, PinoLogger } from 'nestjs-pino';
import { BlockchainRpcQueryService } from '#blockchain/blockchain-rpc-query.service';
import {
chainSignature,
} from '#utils/common/signature.util';
import { chainSignature } from '#utils/common/signature.util';

@Controller({ version: '1', path: 'ics' })
@ApiTags('v1/ics')
Expand Down Expand Up @@ -51,12 +41,12 @@ export class IcsControllerV1 {
payload: AddNewPublicKeyAgreementRequestDto,
api: ApiPromise,
): SubmittableExtrinsic<'promise', ISubmittableResult> {
const encodedPayload = this.blockchainService.createItemizedSignaturePayloadV2Type(payload.payload);
return api.tx.statefulStorage.applyItemActionsWithSignatureV2(
accountId,
chainSignature({ algo: 'Sr25519', encodedValue: payload.proof}), // TODO: determine signature algo
encodedPayload,
);
const encodedPayload = this.blockchainService.createItemizedSignaturePayloadV2Type(payload.payload);
return api.tx.statefulStorage.applyItemActionsWithSignatureV2(
accountId,
chainSignature({ algo: 'Sr25519', encodedValue: payload.proof }), // TODO: determine signature algo
encodedPayload,
);
}

buildUpsertPageExtrinsic(
Expand Down
2 changes: 1 addition & 1 deletion apps/account-api/src/services/siwfV2.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ export class SiwfV2Service {
return null;
}

const api: ApiPromise = await this.blockchainService.getApi() as ApiPromise;
const api: ApiPromise = (await this.blockchainService.getApi()) as ApiPromise;
const { pallet, extrinsic: extrinsicName } = payload.endpoint;

switch (`${pallet}.${extrinsicName}`) {
Expand Down
Loading
Loading