Sora Universe

A collaborative web game where users can create game objects using OpenAI's Sora2 video model and place them on an infinite grid canvas.

English | 简体中文

Features

• 🎮 Infinite Grid Canvas: Drag and navigate an unlimited game world
• 🎨 AI-Powered Object Creation: Generate game objects using text prompts or reference images
• 🖼️ Animated Objects: All objects are converted to GIFs with transparent backgrounds
• 🔄 Real-time Collaboration: All users share the same world via WebSocket
• 📦 Object Library: Browse, search, and reuse created objects
• ✏️ Flexible Placement: Resize objects from 1x1 to 10x10 grid cells
• 🎯 Zelda-Inspired UI: Retro game aesthetic

Architecture

Frontend

• React 18 + TypeScript
• Konva.js for canvas rendering
• Zustand for state management
• Socket.IO for real-time updates
• TailwindCSS for styling

Backend

• Node.js + Express + TypeScript
• MongoDB for data persistence
• Socket.IO for real-time communication
• Bull queue for async video processing
• FFmpeg for video-to-GIF conversion with chroma key
• OpenAI Sora2 API for video generation

Prerequisites

• Node.js 18+
• MongoDB
• Redis
• FFmpeg
• OpenAI API key with Sora2 access

Installation

1. Clone the repository

git clone <your-repo-url>
cd sora-universe

2. Configure API keys

Edit config.yaml in the root directory:

openai:
  apiKey: "your-openai-api-key-here"
  baseURL: "https://api.openai.com/v1"

server:
  port: 3001
  mongoUri: "mongodb://localhost:27017/sora-universe"

redis:
  host: "localhost"
  port: 6379

sora:
  model: "sora-2"
  size: "720x720"
  seconds: 4
  promptTemplate: "Generate a game object for a video game. The object should be: {prompt}. IMPORTANT: Place the object on a solid green screen background (#00FF00). The object should be centered and loop seamlessly. Keep the camera static."

3. Install dependencies

Server

cd server
npm install

Client

cd client
npm install

4. Start MongoDB and Redis

# MongoDB
mongod

# Redis
redis-server

5. Run the application

Start server (in server directory)

npm run dev

Start client (in client directory)

npm run dev

The client will be available at http://localhost:5173 The server API will be at http://localhost:3001

Usage

Creating Objects

1. Click on an empty grid cell
2. The object library drawer opens on the right
3. Click the + button to create a new object
4. Enter a prompt (e.g., "A magical glowing sword")
5. Optionally upload a reference image
6. Click Create
7. A placeholder appears on the grid while Sora2 generates the video
8. Once complete, the animated GIF replaces the placeholder

Placing Objects

1. Click an empty grid cell
2. In the opened drawer, click any existing object
3. The object is immediately placed at that position

Managing Objects

• Select: Click on a placed object to select it
• Copy: Select an object, then click "Copy" in the bottom control panel
• Delete: Select an object, then click "Delete"
• Search: Use the search bar in the drawer to find objects by name

Navigation

• Pan: Click and drag the canvas to move around
• Zoom: (Future feature - mouse wheel zoom)

Technical Details

Video Processing Pipeline

1. User submits prompt → Creates GameObject in DB (status: pending)
2. Job added to Bull queue
3. Worker calls Sora2 API with green screen prompt template
4. Polls Sora2 API until video generation completes
5. Downloads MP4 video
6. FFmpeg processes video:
   • Applies chroma key filter to remove green background
   • Converts to GIF at 15 fps
   • Scales to 192x192px (default 3x3 grid size)
   • Sets infinite loop
7. Saves GIF to /public/gifs/
8. Updates GameObject (status: completed, gifUrl: /gifs/{id}.gif)
9. Broadcasts update via WebSocket to all clients

Grid System

• Grid cell size: 64x64 pixels
• Object default size: 3x3 cells (192x192px)
• Object size range: 1x1 to 10x10 cells
• Canvas: Infinite (virtualized grid rendering)

Sora2 Integration

• Model: sora-2 (fast) or sora-2-pro (high quality)
• Resolution: 720x720 (1:1 aspect ratio)
• Duration: 4 seconds
• Background: Green screen (#00FF00) for chroma keying

Project Structure

sora-universe/
├── client/                 # React frontend
│   ├── src/
│   │   ├── components/     # React components
│   │   │   ├── GameCanvas.tsx
│   │   │   ├── ObjectDrawer.tsx
│   │   │   ├── CreateObjectModal.tsx
│   │   │   └── ObjectControls.tsx
│   │   ├── store/          # Zustand state
│   │   ├── types/          # TypeScript types
│   │   └── App.tsx
│   └── package.json
│
├── server/                 # Node.js backend
│   ├── src/
│   │   ├── routes/         # API routes
│   │   ├── services/       # Sora2, FFmpeg services
│   │   ├── models/         # MongoDB models
│   │   ├── workers/        # Bull queue workers
│   │   ├── socket/         # WebSocket handlers
│   │   └── index.ts
│   └── package.json
│
└── config.yaml             # Configuration file

API Endpoints

Objects

• POST /api/objects - Create new game object
• GET /api/objects - Get all objects (paginated, searchable)
• GET /api/objects/:id - Get single object

Placed Objects

• POST /api/placed-objects - Place object on grid
• GET /api/placed-objects - Get all placed objects
• PATCH /api/placed-objects/:id - Update placed object (move/resize)
• DELETE /api/placed-objects/:id - Delete placed object
• POST /api/placed-objects/:id/copy - Copy placed object

WebSocket Events

• object:updated - GameObject status changed
• object:progress - Generation progress update

Troubleshooting

FFmpeg not found

Install FFmpeg:

# macOS
brew install ffmpeg

# Ubuntu
sudo apt install ffmpeg

MongoDB connection error

Ensure MongoDB is running:

mongod --dbpath /path/to/data

Redis connection error

Ensure Redis is running:

redis-server

Sora2 API errors

• Check your OpenAI API key has Sora2 access
• Verify the API key in config.yaml
• Check API rate limits

Future Enhancements

• Object resize handles (drag corners to resize)
• Mouse wheel zoom
• Layer management (bring to front/back)
• User accounts and private worlds
• Object animations and interactions
• Multiplayer cursors
• Export world as image/video

License

MIT

Credits

Built with OpenAI's Sora2 API