Webvs

Webvs is a WebGL music visualization system for browsers. It has highly programmable visualization preset format and expression language that allows artists to create visual effects by layering one of several built-in rendering components. The library also exposes a flexible API for custom rendering components.

The code is hosted on Github. For bug report and discussions, please use the Github Issues page.

Webvs in runtime at minimum depends on Underscore.js (>= 1.5.0). Webvs also depends on Backbone Events Module. If your application does not already use Backbone then you must include Backbone-Events, which contains a stand alone version of the Backbone Events module. For FPS stat display, include Stats.js. Support for audio playback with HTML5 audio and SoundManager2 is currently supported.

Introduction

Webvs renders visualizations by layering different renderings one after the other. Each rendering is performed by objects called Components or Effects. Each Component type will have a different rendering behaviour which can be configured through the preset definitions. Some components may have sub components, in which case the rendering of all the subcomponents is modified/controlled in some manner by the parent. eg. EffectList, which renders a set of components into a separate buffer which is then blended back into the main render.

A visualization Preset in Webvs is thus just a hierarchy of Components configurations, with an implied EffectList (usually called Root Component or Main) at the root. When a preset is loaded, Webvs instantiates a hierarchy of Components with initial parameters from the Preset JSON.

Component's may accept a variety of different parameters that modify their behaviour. Some components accept code written in an expression language (Webvs internally compiles this to Javascript or GLSL). This allows users tweak component's rendering behaviour in arbitrarily different and complex ways.

Webvs components configurations are event driven. This allows Webvs to respond to changes in component parameters in real-time. Eg: Changing colors, Code etc. This feature can be used to build Live-Edit User Interfaces for presets development.

Basic Usage

A typical usage involves creating an Analyser and a Main object. The Analyser interfaces with your audio source and generates the visualization data, while the Main object serves as the primary interface for controlling the visualization. 

var analyser = new Webvs.WebAudioAnalyser();
var webvs = new Webvs.Main({
    canvas: document.getElementById("canvas"),
    analyser: analyser,
    showStat: true
});
webvs.loadPreset({
    "clearFrame": true,
    "components": [
        {
            "type": "SuperScope",
            "source": "WAVEFORM",
            "code": {
                "perPoint": "x=i*2-1;y=v;"
            },
            "colors": ["#ffffff"]
        }
    ]
});
webvs.start();
analyser.load("music.ogg");
analyser.play();

Examples

Here are some example music visualizations rendered using Webvs.

Webvs.Main

Main object is the primary interface that controls loading of presets, starting stopping animations, etc. It maintains the root Component and the hierarchy of components under it.

loadPreset main.loadPreset(preset)
Loads a preset into this webvs instance. preset is an object hash that contains the preset. The root object should have a components property which will contain an Array for component configurations for all the components. All component configurations should have a type property containing the string name of the Component. Other properties are specific to each component. The resources.uris property in preset is used to register resources with Webvs.ResourceManager and has the same format accepted by the registerUri.

If you want to clear every frame of the preset, then set clearFrame to true in the root object hash. 

webvs.loadPreset({
    "clearFrame": true,
    "components": [
        {
            "type": "SuperScope",
            "source": "WAVEFORM",
            "code": {
                "perPoint": "x=i*2-1;y=v;"
            },
            "colors": ["#ffffff"]
        }
    ]
});

start main.start()
Starts running the animation when ready. The animation may not start playing immediately because preset may use external resources which needs to be loaded asynchronously by the resource manager.

stop main.stop()
Stops the animation.

set main.set(attribute, value)
Sets the current value of a preset property. Currently meta is the only supported attribute. This value can be used to store metadata, such as preset name or author. Eg. main.set("meta", {name: "presetname"})

get main.get(attribute)
Gets the current value of a preset property. Eg. main.get("meta")

resetCanvas main.resetCanvas()
Resets and reinitializes all the components and canvas.

notifyResize main.notifyResize()
This function should be called if the canvas element's width or height attribute has changed. This allows Webvs to update and resize all the buffers.

toJSON main.toJSON()
Returns a JSON representation of the currently loaded preset.

destroy main.destroy()
Frees up all resource.

rsrcMan main.rsrcMan
An instance of Webvs.ResourceManager that manages external file resources.

Webvs.WebAudioAnalyser

WebAudioAnalyser connects webaudio sources to Webvs. This includes audio elements, or AudioNode objects.

constructor new Webvs.WebAudioAnalyser([options])
When creating WebAudioAnalyser objects you can pass in an options hash containing parameters for the analyser. The fftSize field sets the size of the fft samples in the WebAudio Analyser node. Use the threshold and decay fields to configure the beat detection algorithm.

connectToNode analyser.connectToNode(sourceNode)
Connects the analyser to sourceNode which should be an AudioNode in a WebAudio graph.

load analyser.load(sourcem readyFunc)
This is a helper that sets up a WebAudio AudioNode graph and connects the audio analyser to the source. source can be either a string url to audio file or it can be an Audio tag DOM element. readyFunc is a callback that, if provided, will be called when the audio source is ready to play.

Webvs.SMAnalyser

SMAnalyser connects SoundManager2 sounds to Webvs. The following soundManager setup is recommended for use with Webvs. 

soundManager.setup({
    url: "/location/of/soundmanager2/swf",
    flashVersion: 9,
    preferFlash: true,
    useHighPerformance: true,
    useFastPolling: true
});

constructor new Webvs.SMAnalyser([options])
When creating SMAnalyser objects you can pass in an options hash containing parameters for the analyser. The threshold and decay fields to configure the beat detection. algorithm.

createSound analyser.createSound(options)
Creates a SoundManager sound object passing options as arguments and then connects this analyser to the sound.

setSound analyser.setSound(sound)
Connects the analyser to the SoundManager sound object sound to this analyser. Make sure the sound is created with usewaveformdata: true and useeqdata: true options. This allows the analyser to retrieve the waveform and eq data from SoundManager.

Webvs.AnalyserAdapter

Analysers separate the sound source from the Webvs visualization engine. AnalyserAdapter is a generic interface that any analyser has to implement. Several builtin analysers for usual sound sources such as WebAudio or SoundManager2 are already packaged with Webvs, however this interface is useful when you may want to write a custom analyser.

beat beat
boolean value that should be set to true when a beat is detected.

update update
Update is called every frame. Analysers should implement code that analyses sound data in this function.

getWaveForm getWaveForm
This function should return a Float32Array of waveform samples. Each value should be in a -1 to 1 range.

getSpectrum getSpectrum
This function should return a Float32Array of spectrum samples. Each value should be in a 0 to 1 range.

Events

Many object in Webvs generate and or listen to events. Webvs uses the Backbone Events module for this. All backbone event module functions such as on, off, trigger, listenTo etc. are available on these objects. Additionally most objects also have get, set methods for some parameters. These methods fire change event just like Backbone models.

Catalog of Events
Here is the complete list of events in Main, ResourceManager and Components.

Webvs.Component

Component is the base class for all components in Webvs. It provides some basic functionalities and specifies an interface to be implemented by other components. Here is a simple example 

function TriangleComponent(gl, main, parent, opts) {
    TriangleComponent.super.constructor.apply(this, arguments);
}
Webvs.defineClass(TriangleComponent, Webvs.Component, {
    defaultOptions: {
        x: 0, y: 0, color: "#FF0000"
    },

    onChange: {
        color: "updateProgram"
    },

    init: function() {
        this.updateProgram();
    },

    draw: function() {
        this.program.run(this.parent.fm, null,
                         this.opts.x, this.opts.y);
    },

    destroy: function() {
        this.program.destroy()
        this.super.destroy.call(this);
    },

    updateProgram: function() {
        var gl = this.gl;
        if(this.program) {
            this.program.destroy();
        }
        this.program = new PolygonProgram(this.gl, this.opts.color);
    }
});

var PolygonProgram = function(gl, options) {
    var color = _.map(
        Webvs.parseColorNorm(options.color || [1,1,1]),
        function(value) { return Webvs.glslFloatRepr(value); }
    );
    PolygonProgram.super.constructor.call(this, gl, {
        copyOnSwap: true,
        vertexShader: [
            "attribute vec2 a_position;",
            "uniform vec2 u_offset;",
            "void main() {",
            "   gl_PointSize = 1.0;",
            "   setPosition(a_position + u_offset);",
            "}"
        ],
        fragmentShader: [
            "void main() {",
            "   setFragColor(vec4("+color.join(",")+", 1.0));",
            "}"
        ]
    });
};
PolygonProgram = Webvs.defineClass(
    PolygonProgram, Webvs.ShaderProgram, {
    draw: function(x, y) {
        this.setVertexAttribArray(
            "a_position",
            new Float32Array([-0.8,-0.6, 0.46,-0.5, -0.7,0.7])
        );
        this.setUniform("u_offset", "2f", x, y);
        this.gl.drawArrays(mode, 0, points.length/2);
    }
});

constructor new Component(gl, main, parent, options)
All components are expected to follow the same constructor prototype. Components rarely have to be initialized directly with constructor. They are usually initialized with options from the preset JSON by a container such as an EffectList. gl is the webgl context connected to this instance of webvs. main is a reference to the Webvs.Main instance that contains this component. parent is the parent Component of this Component. options is a hash of paramters for the component. Pass options.id to initialize the component with a unique id.

id component.id
A unique string identifier for this component. If not initialized from opts.id a random unique id is generated.

main component.id
Reference to the Webvs.Main that manages this component.

parent component.id
Reference to the Webvs.Main that manages this component.

gl component.gl
Reference to the WebGLContext.

opts component.opts
A Hash containing the component options. This hash Read-Only. Component options have to be modified with the set method.

onChange component.onChange
A hash mapping option names to method names. The method will be called when the corresponding option changes value. Components can use this hash to listen for changes to options and change the state of the component appropirately. A special "*" can be used to listen to changes in any options.

get component.get(key)
Returns the option key

set component.get(key, value, [options])
Sets the options key with the value value.

toJSON component.toJSON()
Returns a JSON representation of the component's options. This object can be is put into preset JSONs.

setParent component.setParent(newParent)
Sets newParent as the parent of this component. This is useful when moving a component around in the component hierarchy.

destroy component.destory()
Cleans up componenet. Override this function to implement additional cleanup tasks. Remember to call the super destroy function.

init component.init()
This function is called once when the component is initialized. Override and implement initializations. Most of the component initialization code is typically implemented here instead of the constructor.

draw component.draw()
This function is called once every frame. Override this function and implement rendering code.

Webvs.Container

Containers are derived from Components are meant to be used as base classes for components that may contain subcomponents. It provides basic functionality to maintain subcomponents.

constructor new Webvs.Container(gl, main, parent, opts)
Webvs Container initializes all the sub-components using the opts.component field. This field must be an array containing hashes. Each of these hash should have a type field, that contains the string name of the Component type. The sub-components are initialized using their corresponding hash as the opts argument.

components container.components
This is a list containing all the sub-components. Should be treated as read-only. Use addComponent or detachComponent methods to manipulate the sub-components.

addComponent container.addComponent(componentOpts, [pos], [options])
Adds a new component into this container. componentOpts is the options hash for the new component. If an integer pos is given then the component will be added at the specified position in components list. The options hash if provided, will be passed to the addComponent event handlers.

detachComponent container.detachComponent(pos, [options])
Detaches a sub-component from this container and returns it. pos should be the index in the components array or the string id of the component to be detached.

findComponent container.findComponent(id)
Searches for a component with the given id in the component sub-tree rooted at this Container.

Webvs.ShaderProgram

Webvs components draw directly using shaders. ShaderProgram provides a featureful abstraction for writing shaders in Webvs. All Shaders extend ShaderProgram or one of its subclasses.

constructor new Webvs.ShaderProgram(gl, opts)
gl is the WebGLContext on which the shader will be rendering. The fragment shader and vertex shader shources should be passed as an array of strings in the opts.fragmentShader and opts.vertexShader fields. Pass opts.blendMode to set how the output of this shader is blended with exisitng output. Set opts.swapFrame to true to trigger a frame swap before rendering. This option can be used if the shaders wants to access the current buffer (particularly trans effects). Set opts.copyOnSwap option to true to trigger a copy before frame swaps. This may be required by shaders which do not touch all the pixels eg: rendering polygons. The opts.blendValue value specifies the blend amount in ADJUSTABLE blend mode. ShaderProgram tries to compiles the blending equations intor the shader program when possible, this means blending modes are not always changeable after instantiation. Setting opts.dynamicBlend to true ensures that blending modes are changeable after instantiation.

run shader.run([fm], [blendMode], [args, ...])
Runs this shader. If FrameBuffer manager instance is passed in fm, then it is used for this rendering. Otherwise the currently set render targets are used. blendMode can be used if a different blending mode should be used for this rendering. This works only if dynamicBlend is enabled. args are passed to the draw function and may include shader specific drawing options.

draw shader.draw([args, ...])
This function should never be called directly, it is called implicitly whenver a call to run function is made. Subclasses should implement drawing operations in this function.

setUniform shader.setUniform(name, type, value, [...])
Helper function to bind value to a uniform. name is the name of the variable. type should be a string indicating the type of the shader variable. Currently supported types are "texture2D", "1f", "2f", "3f", "4f", "1i", "2i", "3i", "4i", "1fv", "2fv", "3fv", "4fv", "1iv", "2iv", "3iv", "4iv".

setVertexAttribArray shader.setVertexAttribArray(name, array, size, type, stride, offset)
Binds data to the vertex attribute array. name is the name of the attribute variable. array should be and array of values containing the data to be bound. size, type, normalized, stride and offset have the same meaning as the arguments to the WebGL vertexAttribPointer function.

setElementArray shader.setElementArray(array)
Binds the element array buffer. array should be an array of values.

destroy shader.destroy()
Cleans up all resources used by this shader.

Webvs.ShaderProgram GLSL Reference

ShaderProgram inserts additional GLSL code into shaders in order to provide consistent interface and additional features. Here we discuss some of these macros and variables.

u_resolution vec2 u_resolution
Uniform variable containing the screen resolution

v_position vec2 v_position
A 0-1 normalized varying of the vertex position.

u_srcTexture sampler2D u_srcTexture
The source texture from the swapped out frame. Enabled only when swapFrame is set to true in the ShaderProgram initialization.

setPosition setPosition(vec2 pos)
Sets the gl_Position and the v_position variable. This macro is accessible only inside vertex shader.

getSrcColorAtPos getSrcColorAtPos(vec2 pos)
Returns the pixel value at the corresponding position in u_srcTexture.

getSrcColor getSrcColor()
Same as getSrcColor but returns the color at v_position. This macro is accessible only inside the fragment shader.

setFragColor setFragColor(vec4 color)
Sets the correctly blended color value in gl_FragColor. This macro/function should be used instead of setting value in gl_FragColor directly. Accessible only inside fragment shader.

Webvs.QuadBoxProgram

A common kind of shader in Webvs is one where the vertices only make two triangles that span the entire screen and the bulk of the shader code is in the fragment shader. QuadBoxProgram is a base class for shaders that makes it easy to write such shaders. QuadBoxProgram itself is derived from ShaderProgram.

constructor new Webvs.QuadBoxProgram(gl, options)
All the parameters are the same as ShaderProgram. The options.vertexShader can be omitted a default vertex shader that draws the box will be supplied.

draw program.draw([args, ...])
The draw method sets the vertex attribute array with a Box spanning the entire screen. Override this and implement additional drawing code. Remember to call the super draw function.

Webvs.FrameBufferManager

FrameBuffer manager maintains a set of named/unnamed render targets and enables switching between them. Many Webvs Components have to read pixels from previous frames or from the output of other components. WebGL doesnt allow reading and writing to the same render target at the same time. To overcome this, Webvs maintains two sets of render targets and then switches between them for each frame, using the previous render target as source for the second. FrameBufferManager provides the functionality to perform this switching. Additionally FrameBufferManager is also used for Buffers and offscreen rendering in EffectLists.

FrameBufferManager has a current texture which is used as the render target by default. Switching textures changes the current texture. In most cases a framebuffer manager reference can be obtained from the Component parent component.parent.fm.

constructor new Webvs.FrameBufferManager(gl, copier, [textureOnly], [texCount])
Instantiates a new FrameBufferManager. gl should be a reference to the WebGl Context. copier should be an instance of a Webvs.CopyProgram to be used when copying over textures. Pass textureOnly with true to prevent creation of FrameBuffers. ie. only textures will be maintained. Pass an integer in texCount to indicate the number of textures to be created initially, by default 2 textures are create on initialization.

addTexture fm.addTexture([name])
Adds a new texture to the FrameBufferManager. If a string name is passed then the texture created will be associated with the name. If a texture with the same name already exists then a new texture is not created. However, a reference count is incremented so that textures that go out of use can be cleaned up.

removeTexture fm.removeTexture(arg)
Removes and cleans up a texture maintained by the FrameBufferManger. If arg is an integer then the texture with that index is removed. If it is a string then the texture with that name is removed (if it is not being references elsewhere).

setRenderTarget fm.setRenderTarget([texName])
Saves the current render target and sets the current texture in this FrameBufferManager as the render traget. If the FM has a FrameBuffer then a FrameBuffer switch is also performed. Pass texName to set a specific texture as the target instead of the current texture.

restoreRenderTarget fm.restoreRenderTarget()
Restores the render target previously saved with a call to setRenderTarget

getCurrentTexture fm.getCurrentTexture()
Returns the current texture set in the framebuffer manager.

copyOver fm.copyOver()
Copies the previous texture to the current texture.

switchTexture fm.switchTexture()
Changes the current texture to the next texture in the framebuffer manager.

resize fm.resize()
Resizes the all the texture to gl.drawingBufferWidth and gl.drawingBufferHeight. This is typically called when components recieve a resize event.

destroy fm.destroy()
Cleans up all textures and FrameBuffers maintained by this FrameBufferManager.

Webvs.ResourceManager

ResourceManager manages external file resources that maybe used by the preset. ResourceManager maintains name to uri mapping. The URI could point to a remote file, in which case the file will be loaded asynchronously or it could be a base64 encoded data.

constructor new Webvs.ResourceManager([packs])
Initializes a resource manager. Pass packs is an array of resource packs to register with the ResourceManager. packs[i].name should be the name of the pack, packs[i].prefix, a prefix that will be appended to filenames and packs[i].fileNames should be an array of file names in the pack.

registerUri rsrcMan.registerUri(fileName, uri)
fileName is the name of the resource, by which it will be references in the presets. uri is the uri to the resource.

toJSON rsrcMan.toJSON()
Returns a JSON representation of the resources.

clear rsrcMan.clear()
Clears the state, uri mappings and caches of the ResourceManager.

destroy rsrcMan.destroy()
Cleans up all resources.

getImage rsrcMan.getImage(filename, [success], [error], [context])
Loads an image resource in the ResourceManager. filename is a registered filename in the resource manager. success and error are callback that will be called when the image load is successfull and errored respectively. context is the this context for the callbacks.

Change Log

1.5.0 - Feb 6, 2014

This release is a major overhaul since the last release. So many things have changed that it made sense to skip directly to 1.5. Some of the changes are as below.