junkpile

20.1 · wgpu Multi-Input Compositor

A standalone Tauri 2 + Rust/wgpu convergence example that combines the input systems developed in examples 14–19 into one native render graph.

Native webcam ─────────────┐
FFmpeg video frames ───────┤
Video-file audio FFT ──────┤
Microphone FFT ────────────┤
MIDI registry ─────────────┤
OSC registry ──────────────┤
Gesture pad ───────────────┘
               ↓
shared Rust snapshots and latest-frame boundaries
               ↓
source composite → HDR feedback → tone-mapped surface
               ↓
Metal / Vulkan / DX12

Run

npm install
npm run dev:metal

Automatic backend selection:

npm run dev

Prerequisites

• Rust toolchain
• Node.js/npm
• FFmpeg and ffprobe on PATH
• Camera and microphone permission when those sources are started

macOS:

brew install ffmpeg
npm run dev:metal

20.1 additions

Full video transport

The video input now includes:

• Timeline scrubber
• Current-time and duration display
• Play, pause, stop
• Previous/next frame stepping
• Looping
• Playback rates from 0.25× through 4×
• Software or FFmpeg hardware-auto decode mode

Every transport action is mirrored to the video-audio decoder so its FFT and audible preview restart from the same source position.

Three FFT sources

The Audio FFT source can be:

Microphone / input
Video-file audio
Microphone + video-file audio

Microphone and video audio use independent analyzers. Mixed mode combines their spectra, waveform data, bands, level, and transient information without requiring both sources to use the same hardware sample rate.

Audible video-audio preview

Video-file audio can be previewed through the system’s default output device. The same decoded samples drive the video FFT at the output callback’s playback clock.

FFmpeg audio decode
       ↓
bounded interleaved sample ring
       ↓
CPAL output callback
       ├── optional audible output
       └── mono FFT analysis chunks

The Preview video-file audio checkbox mutes or unmutes only the video’s soundtrack. Microphone samples are never routed to the output device, including in mixed FFT mode, which avoids a microphone-monitoring feedback loop.

Existing media inputs

• Direct AVFoundation BGRA webcam capture on macOS
• Nokhwa native-camera fallback on non-macOS systems
• FFmpeg/ffprobe decoded BGRA video frames
• Independent latest-frame boundaries so camera/video cannot create unbounded frame queues

Structured control data

• CPAL microphone capture with RustFFT analysis
• FFmpeg video-audio decode with CPAL preview and RustFFT analysis
• MIDI notes, pitch bend, pressure, and learned parameter mappings through midir
• OSC messages and bundles through rosc
• Pointer/touch/pen gestures normalized in the controls WebView and retained in Rust

Native render graph

1. Camera/video source composite
2. Audio-, MIDI-, OSC-, and gesture-driven distortion
3. HDR ping-pong feedback using Rgba16Float
4. Tone mapping and presentation to the native surface

First test

1. Open a video with an audio track.
2. Confirm the video transport and audible soundtrack work.
3. Select Video-file audio as the FFT source and confirm the audio-reactive image changes.
4. Disable Preview video-file audio and confirm the image remains reactive while the soundtrack is muted.
5. Start a microphone and select Microphone + video-file audio.
6. Confirm only the video soundtrack is audible; the microphone remains analysis-only.
7. Add camera, MIDI, OSC, and gesture input as desired.

Notes

• Audio/video synchronization is restart-based: seek, rate, play, pause, stop, and open operations restart both decoder paths from the same source position.
• The video-audio decoder uses FFmpeg atempo filtering so its audible duration follows video playback-rate changes.
• The output callback consumes the audio ring even when preview is muted, keeping video-audio FFT timing tied to playback rather than decode speed.
• This is a convergence example, not a finished VJ application.

This site is open source. Improve this page.