SDK Diagrams

Technical Architecture

Implementation-level diagrams for every layer of the scheng SDK — from graph compilation through GPU execution, shader binding, control routing, texture formats, and distribution packaging.

// Diagram 01

Full System Architecture

Every component in a running scheng instrument. Inputs arrive on dedicated threads, control routes through ParamStore, the GPU render graph executes deterministically each frame, and output sinks are called after queue.submit().

Non-blocking I/O

All input sources (webcam, video, NDI, Syphon, RTMP) run on dedicated threads and communicate via bounded channels. The GPU render thread never blocks waiting for input — when a source falls behind, frames drop gracefully using try_send semantics.

Single CommandEncoder per frame

All node render passes are encoded into one CommandEncoder and submitted as a single batch via queue.submit(). Output sinks are only called after submission — never before. This is the most common source of all-black output bugs.

ParamStore is threadsafe

Wrapped in Arc<Mutex<ParamStore>>. MIDI callbacks and OSC threads write targets from their threads. The render thread calls step_frame() and reads smoothed values. The mutex is held briefly — never across a frame boundary.

Engine vs. instrument boundary

Everything inside the GPU Core box is the engine. Everything outside — graph construction, time, MIDI, device management — belongs to the instrument. The engine does not own time, parse protocols, or manage hardware.

// Diagram 02

Layer Model & Crate Boundaries

scheng is organized into five isolated layers. No layer reaches backward across a boundary. This isolation is the primary guarantee of deterministic execution and long-term SDK stability.

ARCHITECTURAL RULE No layer reaches backward. Topology (scheng-graph) never calls the runtime. The runtime never parses MIDI. I/O crates never mutate graph topology. All interaction is explicit and flows downward through the layer stack.

// Diagram 03

Per-Frame Execution Lifecycle

Every frame executes in exactly this sequence. The hot path avoids allocation, shader recompilation, and blocking I/O. GPU commands are batched into a single CommandEncoder and submitted once.

CRITICAL ORDER — submit before present queue.submit() must be called before any sink's present(). Calling present() before submission reads from a render target that hasn't been written yet — all pixels will be black or from the previous frame. execute_frame() enforces this order automatically.

// Diagram 04

Shader Compat Layer

The compat layer processes your raw GLSL 330 source before GPU compilation. It strips standard uniform declarations (injecting them via UBO instead), rewrites iChannel sampler references, and collects custom u_* uniforms for the CustomBlock.

Binding layout

Binding 0–3: iChannel0–3 texture views. Binding 4: iSampler. Binding 5: FrameBlock UBO (uResolution, uTime, uFrame). Binding 6: CustomBlock UBO (all u_* uniforms in declaration order).

Pipeline cache key

Pipelines are cached by (shader_source_hash, sample_count). Changing NodeConfig.frag_shader invalidates the cache for that node. MSAA level changes also recompile. All other NodeConfig changes are free — no GPU work.

// Diagram 05

MIDI → ParamStore → Shader

The complete data path from a physical MIDI controller to a GPU uniform value. Three separate threads, two handoffs, one smoother.

STEP_FRAME() IS MANDATORY set_by_midi_cc() writes to targets. get() reads from values. They are different HashMaps. The smoother only advances values toward targets when step_frame() is called. Without it, MIDI input appears to have no effect.

// Diagram 06

Texture Formats & Precision

Input textures arrive as 8-bit. Internal render targets use 16-bit half-float. FFmpeg output converts back to 8-bit YUV with bt.709 color tagging. Each stage has a specific format for a specific reason.

// From executor.rs — bt.709 FFmpeg flags applied automatically
"-colorspace", "bt709",
"-color_primaries", "bt709",
"-color_trc", "bt709",
"-color_range", "tv"

// Diagram 07

Distribution Bundle Structure

A scheng instrument compiles to a native binary. Each platform requires a different packaging format. The binary, framework/DLL dependencies, and shader assets must all be co-located so the instrument finds them at runtime without environment variables.

// Hybrid asset resolution — works in dev AND distributed bundle
fn asset_dir() -> std::path::PathBuf {
    let exe = std::env::current_exe().unwrap();
    let dir = exe.parent().unwrap();
    // macOS .app: binary is in Contents/MacOS/, assets in Contents/Resources/
    let resources = dir.parent().unwrap().join("Resources");
    if resources.exists() { resources } else { dir.to_path_buf() }
}

// Embedded fallback so the binary always works, even without shader files
const MAIN_FRAG: &str = include_str!("../assets/shaders/main.frag");

cfg.frag_shader = std::fs::read_to_string(asset_dir().join("assets/shaders/main.frag"))
    .ok()
    .or_else(|| Some(MAIN_FRAG.to_string()));

← Back to SDK Reference