// Diagram 01

Full System Architecture

Every component in a running scheng instrument and how they relate to each other.

INPUTS Syphon In NDI Receive Webcam Video File RTMP / RTSP CONTROL MIDI CC OSC ParamStore smooth() targets → values step_frame() each tick GPU CORE — WgpuRuntime Hot-Reload AssetWatcher naga GLSL compile pipeline cache Shader Graph NodeConfig uniforms u_* input_textures[4] iChannel0–3 Render Pipeline Rgba16Float targets MSAA 1x / 4x / 8x bind groups FrameBlock UBO CustomBlock UBO iChannel0–3 RenderPass/node queue.submit() one batch/frame OutputSinks PreviewSink SyphonSink NdiSink FfmpegSink SpoutSink present() called after submit() non-blocking drop-safe about_to_wait() → tick() → execute_frame() → present() OUTPUTS Syphon Out NDI Out RTMP Stream RTSP Stream Record MP4 Preview Window Video / texture inputs Control signals GPU processing Output / sink
// Diagram 02

Frame Render Loop

What happens every single frame, in order. The render loop never blocks — inputs that fall behind drop frames gracefully.

01 POLL INPUTS Webcam.poll() VideoDecoder .upload_frame() Syphon.poll() 02 PARAMS store.step_frame() MIDI targets → smoothed values 03 BUILD CFG NodeConfig per node frag_shader source uniforms HashMap input_textures[4] 04 GPU RENDER graph.compile() RenderPass per node Rgba16Float targets MSAA if active 05 SUBMIT queue.submit() All GPU commands batched as one CommandEncoder 06 PRESENT sink.present() Preview blit Syphon / NDI FFmpeg pipe — non-blocking — inputs and outputs run on dedicated threads — frame drops preferred over stalling —
// Diagram 03

Graph Node Topology

How nodes connect. Any output can feed any input. The graph compiles to a topologically sorted execution plan each frame.

SINGLE NODE ShaderSource generates content PixelsOut triggers sinks EFFECT CHAIN ShaderSource webcam texture ShaderPass solarize effect ShaderPass color grade PixelsOut → preview FEEDBACK / TEMPORAL ShaderSource current frame ShaderPass mixes prev frame PixelsOut → output PreviousFrame last frame texture A/B MIXER ShaderSource Source A ShaderSource Source B Crossfade u_tbar = MIDI CC1 mix(A, B, tbar) PixelsOut → Syphon / NDI ↑ MIDI CC1 controls mix graph.compile() topologically sorts nodes — execution order is always dependency-correct
// Diagram 04

MIDI → Shader Routing

How a MIDI CC message becomes a shader uniform value — with optional smoothing to remove steppy artifacts.

MIDI Controller CC1 = 64 (any device) midir callback msg[0]=0xB0 ch=1 cc=1 val=64 ParamStore set_by_midi_cc(1, 64) targets["u_tbar"]=0.5 step_frame() each tick smooth → values[0.5] NodeConfig uniforms["u_tbar"] = store.get("u_tbar") CustomBlock GPU binding 6 UBO float u_tbar = 0.5 GLSL Shader uniform float u_tbar; mix(a, b, u_tbar) MIDI THREAD RENDER THREAD (each tick) smooth=0.05 → reaches target in ~20 frames — eliminates steppy CC artifacts
// Diagram 05

Use Case Signal Flows

Three complete real-world signal chains from input to output.

LIVE VIDEO MIXER Resolume Syphon Out scheng crossfade Resolume Syphon In ↑ MIDI T-bar controls mix BROADCAST STREAM Webcam FaceTime HD scheng color grade YouTube RTMP ingest ↑ MIDI CC: brightness / contrast / hue VIDEO INSTALLATION Video File loops forever scheng generative NDI Out → projectors COLORSPACE CHAIN Input Rgba8Unorm 8-bit GPU Render Rgba16Float 16-bit half-float FFmpeg Encode YUV420P bt.709 broadcast color range VLC / OBS Rec.709 tagged correct colors Without bt.709 defaults to bt.601 (SD) washed / shifted colors scheng sets: -colorspace bt709 -color_primaries bt709 -color_trc bt709 automatically MSAA ANTI-ALIASING sample_count = 1 No MSAA — aliased edges sample_count = 4 4x MSAA — smooth edges sample_count = 8 8x MSAA — highest quality Pipeline keyed on (shader_hash, sample_count) — change anytime
← Back to scheng home