Image, Video & Audio Generation

Image, video, and audio generation are working Examples built around asynchronous Replicate predictions. Each flow creates a provider task, returns a task ID immediately, and lets the browser poll a query endpoint until an output URL is available.

Shared architecture

Generator UI → POST /api/ai/<media> → provider factory → Replicate prediction
             ← taskId + normalized status

Generator UI → POST /api/ai/<media>/query { taskId } (poll)
             ← pending | processing | succeeded | failed | canceled + output URLs

Each src/lib/ai/{image,video,audio} directory contains types.ts, index.ts (factory), and replicate-provider.ts. replicate-utils.ts normalizes provider statuses and recursively extracts HTTP URLs from varied output shapes. Factories are initialized lazily so REPLICATE_API_TOKEN is read during a request.

The localized marketing route files under {-$locale}/_marketing/ai mount these components. UI labels are currently English.

Setup

# .env (local only)
REPLICATE_API_TOKEN=r8_your-token

pnpm wrangler secret put REPLICATE_API_TOKEN

Without a token, the factory throws a clear “provider is not configured” error. Never expose the token as VITE_*.

Input mappings

Provider adapters translate product-level requests into model-specific inputs:

• Images map 1024x1024, 1792x1024, and 1024x1792 to aspect ratios; optional source image data becomes input_images.
• Video supports text-to-video, image-to-video, and video-to-video. Veo uses reference_images, Sora uses input_reference, and other models use image_input/video_input.
• Audio maps short locale codes to Qwen language names and translates voice or clone mode; clone mode passes reference_audio.

Changing a model is more than editing the selector. Update the UI option, the provider's input mapping, validation/capabilities, output parsing if needed, and cost/timeout policy together. Pin a version when reproducibility matters rather than silently following a mutable model alias.

Polling behavior

Image and audio poll every five seconds and time out after five minutes. Video polls every fifteen seconds and times out after ten minutes. The long generation runs at Replicate; each Worker request only creates or queries a task, which fits Workers better than holding one request open for the entire generation.

Task IDs and results live only in browser state. Refreshing loses the current job, and returned URLs remain provider-hosted rather than being copied to R2.

Add another provider

1. Extend the media ProviderType union.
2. Implement the corresponding ImageProvider, VideoProvider, or AudioProvider interface.
3. Register it in the factory using a request-time secret.
4. Extend endpoint Zod schemas and the client selector deliberately.
5. Normalize statuses, errors, task IDs, and output URLs to the existing response contract.
6. Add create/query tests for provider errors and unusual output shapes.

Production hardening

All six endpoints are public examples. Add authentication, ownership checks on task queries, rate limits, per-user quotas, model allowlists, input URL policy, prompt/body limits, cost caps, and safe error mapping. In particular, image data URLs and arbitrary reference URLs can create oversized requests or SSRF- like provider fetch behavior unless constrained.

Persist task ownership/status in D1 so users can resume and can only query their own tasks. Copy successful outputs to R2 when your product needs durable media; provider URLs can expire. Add cleanup/retention, content moderation, consent rules for voice cloning, copyright policy, and observability.

Verify

• Each create endpoint returns a task ID and normalized status with a valid token.
• Query endpoints move through pending/processing to a terminal status.
• Image aspect ratio and optional source-image mappings match the selected model.
• Each video scene sends the correct provider input field.
• Audio language/clone inputs match Qwen expectations.
• Failed and canceled tasks stop polling with a useful message.
• Durable results are copied to R2 and task ownership is enforced after hardening.

Related pages