Shaders

Shaders are small programs that we send to a GPU to perform computations. Shaders are extensively used in Smelter, in fact all built-in transformations rely on them.

General concepts

The Smelter utilizes two types of shaders: vertex shaders and fragment shaders.

Vertex shaders

Vertex shaders take data from a single vertex as input and manipulate it to create the desired output shape.

In vertex shaders, videos are represented as two triangles aligned in the following configuration:

The rectangle created by these triangles spans the entire clip space ([-1, 1] x [-1, 1]). Each video input receives its own separate rectangle.

plane_id in base_params push_constant represents number of currently rendered planes (textures). If there are no input textures, plane_id is equal to -1 and a single rectangle is passed to the shader. It is only useful for shaders that generate something in the fragment shader.

Since Smelter doesn’t handle complex geometry, and layouts typically manage positioning, resizing, and cropping, we don’t expect the users to create complex vertex shaders very often.

The following vertex shader should cover most needs for transformations that process a single video in the fragment shader:

struct VertexOutput {
  @builtin(position) position: vec4<f32>,
  @location(0) tex_coords: vec2<f32>,
}

@vertex
fn vs_main(input: VertexInput) -> VertexOutput {
  var output: VertexOutput;

  output.position = vec4(input.position, 1.0);
  output.tex_coords = input.tex_coords;

  return output;
}

Fragment shaders

Fragment shaders focus on processing pixel data to apply visual effects. A single shader instance, responsible for calculating the color of a pixel, is started for every pixel of the output texture.

The output of a fragment shader must be a vector with four elements, each being a floating-point number (f32) between 0.0 and 1.0. These four numbers represent the red, green, blue, and alpha (transparency) values of a pixel.

The children of the Smelter’s Shader component are available as textures. You can sample them using the builtin textureSample function. To customize texture sampling, use the sampler_ param.

let color = textureSample(texture, sampler_, texture_coordinates)

For example see this simple fragment shader, which applies the negative effect:

struct VertexOutput {
  @builtin(position) position: vec4<f32>,
  @location(0) tex_coords: vec2<f32>,
}

@fragment
fn fs_main(input: VertexOutput) -> @location(0) vec4<f32> {
  let color = textureSample(textures[0], sampler_, input.tex_coords);
  return vec4(vec3(1.0) - color, 1.0);
}

Synthax

Since Smelter is implemented using wgpu, the shaders have to be written in WGSL (WebGPU Shading Language). Shaders also have to fulfill some custom requirements to be used by the Smelter. We will detail all necessary conditions in the following sections.

Header

Every user-provided shader should include the code below.

struct VertexInput {
  @location(0) position: vec3<f32>,
  @location(1) tex_coords: vec2<f32>,
}

struct BaseShaderParameters {
  plane_id: i32,
  time: f32,
  output_resolution: vec2<u32>,
  texture_count: u32,
}

@group(0) @binding(0) var textures: binding_array<texture_2d<f32>, 16>;
@group(2) @binding(0) var sampler_: sampler;

var<push_constant> base_params: BaseShaderParameters;

Custom parameters

You can define a custom WGSL struct and bind a value of this type:

@group(1) @binding(0) var<uniform> custom_name: CustomStruct;

You need to provide this struct when creating a node using the shader_params field of the shader node struct.

Entry points

Vertex shader

A vertex shader entrypoint must match the following signature:

@vertex
fn vs_main(input: VertexInput) -> A

Where A can be any user-defined struct suitable for a vertex shader output.

Fragment shader

A fragment shader entrypoint must match the following signature:

@fragment
fn fs_main(input: A) -> @location(0) vec4<f32>

Where A is the output type of the vertex shader.