Architecture

How scheng works.

System diagrams for the render pipeline, graph topology, signal flow, and I/O architecture.

// Diagram 01

Full System Architecture

Every component in a running scheng instrument and how they relate to each other.

INPUTS Syphon In NDI Receive Webcam Video File RTMP / RTSP CONTROL MIDI CC OSC ParamStore smooth() targets → values step_frame() each tick GPU CORE — WgpuRuntime Hot-Reload AssetWatcher naga GLSL compile pipeline cache Shader Graph NodeConfig uniforms u_* input_textures[4] iChannel0–3 Render Pipeline Rgba16Float targets MSAA 1x / 4x / 8x bind groups FrameBlock UBO CustomBlock UBO iChannel0–3 RenderPass/node queue.submit() one batch/frame OutputSinks PreviewSink SyphonSink NdiSink FfmpegSink SpoutSink present() called after submit() non-blocking drop-safe about_to_wait() → tick() → execute_frame() → present() OUTPUTS Syphon Out NDI Out RTMP Stream RTSP Stream Record MP4 Preview Window Video / texture inputs Control signals GPU processing Output / sink
// Diagram 02

Frame Render Loop

What happens every single frame, in order. The render loop never blocks — inputs that fall behind drop frames gracefully.

01 POLL INPUTS Webcam.poll() VideoDecoder .upload_frame() Syphon.poll() 02 PARAMS store.step_frame() MIDI targets → smoothed values 03 BUILD CFG NodeConfig per node frag_shader source uniforms HashMap input_textures[4] 04 GPU RENDER graph.compile() RenderPass per node Rgba16Float targets MSAA if active 05 SUBMIT queue.submit() All GPU commands batched as one CommandEncoder 06 PRESENT sink.present() Preview blit Syphon / NDI FFmpeg pipe — non-blocking — inputs and outputs run on dedicated threads — frame drops preferred over stalling —
// Diagram 03

Graph Node Topology

How nodes connect. Any output can feed any input. The graph compiles to a topologically sorted execution plan each frame.

SINGLE NODE ShaderSource generates content PixelsOut triggers sinks EFFECT CHAIN ShaderSource webcam texture ShaderPass solarize effect ShaderPass color grade PixelsOut → preview FEEDBACK / TEMPORAL ShaderSource current frame ShaderPass mixes prev frame PixelsOut → output PreviousFrame last frame texture A/B MIXER ShaderSource Source A ShaderSource Source B Crossfade u_tbar = MIDI CC1 mix(A, B, tbar) PixelsOut → Syphon / NDI ↑ MIDI CC1 controls mix graph.compile() topologically sorts nodes — execution order is always dependency-correct
// Diagram 04

MIDI → Shader Routing

How a MIDI CC message becomes a shader uniform value — with optional smoothing to remove steppy artifacts.

MIDI Controller CC1 = 64 (any device) midir callback msg[0]=0xB0 ch=1 cc=1 val=64 ParamStore set_by_midi_cc(1, 64) targets["u_tbar"]=0.5 step_frame() each tick smooth → values[0.5] NodeConfig uniforms["u_tbar"] = store.get("u_tbar") CustomBlock GPU binding 6 UBO float u_tbar = 0.5 GLSL Shader uniform float u_tbar; mix(a, b, u_tbar) MIDI THREAD RENDER THREAD (each tick) smooth=0.05 → reaches target in ~20 frames — eliminates steppy CC artifacts
// Diagram 05

Use Case Signal Flows

Three complete real-world signal chains, the colorspace pipeline, and MSAA reference.

LIVE VIDEO MIXER Resolume Syphon Out scheng crossfade Resolume Syphon In ↑ MIDI CC1 T-bar BROADCAST STREAM Webcam FaceTime HD scheng color grade YouTube RTMP ingest ↑ MIDI CC brightness / contrast / hue INSTALLATION Video File loops forever scheng generative NDI Out → projectors COLORSPACE CHAIN Input Rgba8Unorm 8-bit source GPU Render Rgba16Float 16-bit half-float FFmpeg Encode YUV420P bt.709 broadcast color range Player / VLC Rec.709 tagged correct colors ✓ Without bt.709 defaults to bt.601 (SD) washed / wrong colors ✗ scheng sets -colorspace bt709 -color_primaries bt709 -color_trc bt709 automatically on all FFmpeg output MSAA ANTI-ALIASING sample_count = 1 off — aliased edges sample_count = 4 4× MSAA — smooth edges sample_count = 8 8× MSAA — highest quality Pipeline is keyed on (shader_hash, sample_count) — switching recompiles once then caches. Organic shaders rarely benefit from MSAA.
// Diagram 06

Distribution & Packaging

How a compiled scheng instrument is packaged and shipped for each platform. Each platform has different bundling, signing, and dependency requirements.

SOURCE cargo build --release target/release/my-instrument · target/release/my-instrument.exe macOS MyInstrument.app Contents/MacOS/ ← binary Contents/Frameworks/ ← Syphon.framework Contents/Frameworks/ ← libndi.dylib Contents/Resources/assets/shaders/ codesign Developer ID cert --options runtime notarytool Apple malware scan stapler staple Distribute as DMG hdiutil create .dmg Windows MyInstrument/ (folder) my-instrument.exe ← binary Processing.NDI.Lib.x64.dll (if NDI used) Spout.dll (if Spout used) assets/shaders/ ffmpeg must be in PATH on target machine signtool (optional) suppresses SmartScreen warning requires code signing certificate Distribute as ZIP Compress-Archive Linux my-instrument.AppDir/ usr/bin/my-instrument ← binary usr/bin/assets/shaders/ AppRun script ← entry point requires Vulkan driver on target GPU NDI: LD_LIBRARY_PATH=/usr/local/lib appimagetool bundles all into single file runs on any modern distro Distribute as AppImage my-instrument-x86_64.AppImage
// Diagram 07

Asset Resolution at Runtime

How scheng resolves shader files in development vs. a distributed bundle. The hybrid approach gives users hot-reload in development and a self-contained binary in production.

asset_dir() checks for Contents/Resources/ next to binary not found found Development ./assets/shaders/main.frag hot-reload active ✓ Distributed Bundle Contents/Resources/assets/shaders/ self-contained ✓ include_str!() fallback compiled into binary — always works include_str!() fallback same embedded default same code path works in both modes
← Back to scheng home