Image, Video & Audio Generation
Build task-based media generation flows with Replicate providers.
The template includes Example image, video, and audio generators backed by Replicate. Each feature uses the same shape: create a remote task, keep its ID, poll a query endpoint, normalize provider status, and render returned URLs.
All six media endpoints are public examples. They do not require a session or implement quotas, rate limits, durable task records, or cost controls. Do not expose them unchanged in production.
What it provides
| Media | Page | Create endpoint | Query endpoint |
|---|---|---|---|
| Image | /ai/image | POST /api/ai/image | POST /api/ai/image/query |
| Video | /ai/video | POST /api/ai/video | POST /api/ai/video/query |
| Audio | /ai/audio | POST /api/ai/audio | POST /api/ai/audio/query |
Set the server-only provider token in .env and restart the app:
REPLICATE_API_TOKEN=your-replicate-api-tokenIf it is missing, each factory throws a provider-not-configured error when the
route requests the default replicate provider.
Architecture
The three feature folders under src/lib/ai/ each contain:
types.ts— provider type, create/query requests, normalized responses, and the provider interface.replicate-provider.ts— Replicate prediction creation, lookup, input mapping, status mapping, and output URL extraction.index.ts— a lazy singleton factory that registers Replicate when the token exists and returns a provider by type.
src/lib/ai/index.ts re-exports all three factories and interfaces. The Route
Handlers depend on those interfaces rather than creating a Replicate client in
the UI.
Task lifecycle
- A generator posts the prompt, model, inputs, and options to its create route.
- The route performs basic shape checks and asks the provider factory for
replicate(unlessprovideris supplied). - Replicate returns a prediction ID; the route responds with
taskId, status, and provider task information. - The client polls the matching
/queryroute—every 5 seconds for image/audio and every 15 seconds for video. - Providers map Replicate
starting,processing,succeeded,failed, andcanceledintopending,processing,succeeded,failed, andcanceled. - The query result extracts HTTP(S) output URLs into
imageUrls,videoUrls, oraudioUrls.
Task state only lives in the browser and at Replicate. Refreshing the page loses the local task ID/history, and the template does not copy results into durable storage.
Included models and inputs
Image
ImageGenerator.tsx supports text-to-image and image-guided generation, sizes
1024x1024, 1792x1024, and 1024x1792, and these choices:
black-forest-labs/flux-2-pro(default)bytedance/seedream-3
The provider defaults to black-forest-labs/flux-2-pro when no model is sent,
maps those sizes to 1:1, 16:9, and 9:16, and sends an optional source image
as input_images.
Video
VideoGenerator.tsx offers text-to-video, image-to-video, and video-to-video
tabs with kwaivgi/kling-v2.5-turbo-pro (default), google/veo-3.1, and
openai/sora-2. It limits the prompt in the UI to 2,000 characters.
Reference input keys vary by model: Veo uses reference_images, Sora uses
input_reference, other image models use image_input, and video input uses
video_input.
Audio
AudioGenerator.tsx uses qwen/qwen3-tts for text-to-audio and
audio-to-audio/voice-clone UI modes. It sends language and mode options; the
provider maps them to Qwen's text, normalized language, custom_voice or
voice_clone, and optional reference_audio fields.
Changing a Replicate model
Update the UI option
Change the model list in the matching generator component. This controls what a user can select, but is not a server-side allowlist.
Update provider input mapping
Read the target model's Replicate input schema and change the corresponding
createTask() implementation. Model IDs may share a provider while expecting
completely different keys, enum values, or file formats.
Verify output extraction
Check the real prediction output. Extend extractImageUrls, extractVideoUrls,
or extractAudioUrls if the model returns a different object shape.
Enforce it on the server
Validate model, scene, prompt, files, and options in the create route. The current routes accept client-supplied model/provider values with only basic checks.
Adding another provider
- Extend
ImageProviderType,VideoProviderType, orAudioProviderType. - Implement the feature's provider interface in a new provider module.
- Register it in the matching factory when its server credentials exist.
- Select it with a trusted server-side decision. If the client may request a provider, validate that value against an allowlist.
- Keep normalized statuses and URL response fields stable so the current UI does not need provider-specific branches.
Production hardening
- Authenticate every create/query route and authorize task ownership.
- Use schemas for prompt, model, provider, options, URLs, file type, and size.
- Add per-user rate limits, concurrency limits, quotas, and budget alerts.
- Persist task IDs, owners, inputs, status, failures, and usage.
- Use bounded polling with backoff, timeouts, retry policy, and cancellation.
- Copy completed outputs to durable object storage before provider URLs expire.
- Moderate prompts/uploads and prevent SSRF through untrusted input URLs.
- Hide provider error internals from clients; add structured logs and metrics.