Naivi: Studio 80 Year Of The Horse VJ / Light Performance

DSC7963-1024x965

As NAIVI we performed with our custom build LED panel and self written software at Year Of The Horse in Studio 80 (Amsterdam). The organisation wanted to do something different with their light setup and we (NAIVI) were invited to make that happen.

The LED panel is build from the ground up from high brightness LED’s shipped in from China. The panel receives it’s data through a Teensy Microcontroller. I wrote a custom C++ library to convert any sort of image (or video) data to a stream of bytes the micro controller can work with. On top of the library we wrote a vj application that samples and decomposes a video and audio stream to procedurally generate the image data that is send over. These images are synced to the music based on a modulation matrix that accepts LFO, Recorded Channel and Audio data. The modulation matrix consists of a set of application effects (controls) that are used to render a set of images.

You could think of the application as a video synthesizer that accepts any sort of device input to generate an image using a specific render interface (OpenGL).

The Application can also control any already existing light in the club. Most lights accept a DMX signal that is send out by a control panel. The panel in studio 80 accepts midi data that can be mapped to various controls to manipulate the brightness or movement of a light. The already existing modulation matrix (in the software application) is used to calculate a new light value based on the image, audio and lfo data available in the running application.This enabled us to control both the lights and visuals from the same software environment, merging two distinct worlds that are most often controlled seperately.

All the software is written using OpenFrameworks in C++. The teensy code is native C.

Programmable LED wall

After performing with our custom VJ software last month, I noticed that the video stream felt constrained by the bounds of the canvas and did not light up the venue in a way that was desirable or expected. Although we received very positive feedback regarding the visuals itself, I wanted to get rid of the beamer and move towards a more custom / scalable light emitting video projection source.

After reading up on custom LED panels I decided to give that a go. This would render the lights in a club in-effective and could display the visuals in a more tangible way. The added benefit of the hardware is that there would be no more fighting over what gets priority, the lights or the visuals. This turned out to be my main point of frustration that night we performed.

The final wall (combined) should be approx 4 meters wide and 2 meters high. The LED screen can be broken up in to various parts (panels), all of them addressable through a custom GL interface that renders parts of the video stream to a sub-set of the screen. This enables us to place multiple panels throughout a possible venue / space,

But buying that much equipment at once costs a lot of money. I decided to build a small prototype first, using components recommended online. Further down you can find the various parts I used. The microcontrollers arrived within a week, as well as the programmable led strips, cables, plugs and adapters.  All of the equipment was shipped from the United States using UPS.

I ordered 2 Teensy 3.1 microcontrollers, together with the OctoWS2811 Adaptor board to address the individual led strips. In combination with the OctoWS2811 LED library this setup can address thousands of LED’s, perfect for building a project of this size. I had to solder the components together and used a standard CAT cable to drive the individual strips. The wiring of the strips to the board was rather straight-forward. For this prototype 500 individual LED’s were fed a video stream using only 4 outputs out of the 8 available on every Teensy board. Every 2 strips (approx 300 leds) were given their own 10 amp, 5 volts power supply. The PWM signal was send from one teensy board. Multiple Teensy boards can be connected to keep the video signal in sync. The led strips were ordered at Adafruit.

The Led Panels are addressed through a custom C++ interface, compatible with OpenFrameworks. Using this interface I can drive multiple LED panel using various applications, including my VJ application. The vj app is controlled using an IPad and Akai APC20 (midi controller). The amount of light emitted is (to say the least) exceptional. Perfect for lighting up a medium sized venue, especially once all the hardware is combined in to a blistering light source.

Now there’s a working prototype, we can order the additional led strips, controllers and adaptors. 8 teensy boards will be used to drive the individual LED panels. Every teensy board wil feed approximately a 1×1 meter led panel. All of these panels can be combined to create a big wall, or placed throughout a venue to generate a more spacial effect. More on that soon.

grab_onegrab_threegrab_fourgrab_five

One finished led panel dancing to the music and lighting up the club @ Year of the Horse, Studio 80, Amsterdam

led wall test wip (using OpenFrameworks)

2nd programmable video led wall test

App Controllers

Akai APC20 and Ipad, used for controlling the application that drives the panels.

DSC_0230

The 2 teensy controllers soldered on to the Octo adapter (running)

DSC_0241

16 meter of led strips controlled by one Teensy board, using 4 out of the 8 available PWM channels. 2 strips use one 10A 5V power supply (300 leds)

Kala Processing Library (C++)

Started a new generative visuals project, where a video stream is layered and fragmented in real-time, using a custom library build upon OpenFrameworks. This library will be used to develop various filters and algorithmic effects that can be combined with video streams, images and geometric data. The current version runs a stable 60+ fps, where the source data is sampled from a video stream.

The framework will be used the 17th of May (2014) to render the visuals for the upcoming club night: The Year Of The Horse. The trailer (shown below) is recorded live and used a custom midi interface to control the layers. The midi interface hooks in to an IPad and Akai APC-20

blur

glitch_oneglitch_three

glitch_two

Video Analyzer / Sampler

I’m working on a program that analyzes and samples large video files (movies). First implementation is shaping up nicely, where there is an option to rip individual frames from a stream (in sequences) and analyze shots. The idea is to use this information to create spacial relationship maps based on the information found in a video stream.

The application is written in C++ and fully threaded. Multiple operations can be performed at the same time on the same video stream: for example sampling, analyzing and browsing. The interface is straightforward and should be of help when trying to find the right sample settings.

At some point I hope to be able to share the application and release the source code.

video_analyzer_screen

QTreeView / QModel Example (PyQT)

Quick tutorial on how to populate a QTreeView with a QModel. This example was build using PyQT4 and has been tested with the default 2.6.4 python distribution. The package contains a PyDev project and all the necessary modules. Simply run the main.py file to show the dialog together with the QTreeView

The model is constructed with a set of viewable items, derived from a set of base components. This divide might not be necessary but I prefer a strict division between items that can be viewed and the data that is made visible.

This is a typical example of show (better yet, try) don’t tell. So go ahead and download the package. As always, feel free to ask questions!

tree_view

Liminal Universe Shaper

This piece of software allowed people to design their own cover for the album Liminal by Exivious. Every cover is stored on a server, and is used as the starting point for a new design. By using this tool, all fans contributed to the final cover. After a month almost 1500 unique designs were submitted (saved), rendered, printed and shipped together with the album

But instead of me trying to explain the concept behind this piece of software, I’ll leave it to the guys of the band to do that for me:

With the artwork for Liminal, the band tried to illustrate this stage of transformation in an abstract way. They worked together with Coen Klösters and Pieter Willems to develop an interactive environment called the Liminal Universe Shaper. A 3D environment displaying a network of nodes and connections illustrating the stage of transformation.

With the help of this software the crowdfunding backers who bought the ‘Personalized Unique Edition’ were able to shape the liminal universe to their liking. The design could then be saved and was printed on a poster that shipped with the album.

Also, every time the Liminal Universe Shaper would be opened, it would automatically load the previous customer’s creation so the next customer could continue designing from that point on. After all customers designed their artwork, the band ended up with a piece of artwork that was shaped by all ‘Personalized Unique Edition’ backers. This piece of generative community artwork was then used to create the final album artwork.

The Liminal Universe Shaper application:

Now the album has been released, you can try to design something yourself. The game / software is rather light-weight and should run fine on most computers. The download and save functionality should still work. After downloading and extracting the package make sure to read the instructions or watch the demonstration video above.

Demonstration Video:

The real time component (game) is written in C# and build with Unity. The (final) rendering is done with Houdini using Mantra. I have made the Unity and Render code publicly available. For security reasons all server references have been removed and save and load functionality has been adjusted to be local (to your computer). It’s easy to override this behavior by adjusting the code in a couple of places. The package contains a UnityPackage (holding all the code, assets etc), a Houdini package (containing scripts, hip files and otls) and a Manual to get you started.

The code and data comes without any sort of warranty. Suggestions on how to improve the tool are welcome. I have done my best to include all the code, dependencies and assets. If something is missing and you can’t get the game to compile or run, let me know.

I’d like to thank Tymon Kruidenier for pushing me to get this done, Pieter Willems  for the network component (and production) and the band Exivious for their trust!

Some Designs:

Final Box Art:

Creating a VEX DSO: Adding Photoshop PSD Support to Houdini

This post will try to explain how to write and install your own VEX DSO plugin for Houdini, written in C++. The included example project creates a plugin called VexImageReader. This plugin can be used to read all sorts of images, including psd and dds files. The reader can be used in all vex context layers and is added as a function called: readimage.

The function takes as input arguments a U and V coordinate, an input string (image name) and a wrap mode. By calling the function Houdini evaluates the image at that specific coordinate and returns an interpolated color value (RGBA). The reader supports most popular image formats and reads 8, 16 and 32 bit grayscale and color images.

Before we start, download the necessary files right HERE. Inside this package you will find:

  • The source code and Visual Studio 2008 project file (code)
  • The FreeImage library source code (freeimage/Source)
  • A compiled x64 windows DSO (build)
  • A compiled x64 windows FreeImage library (freeimage/Dist)
  • An otl that wraps the Vex function call in to a VOP (otl)
  • An example HIP file (hipfile)
  • The VEXdso include file, used by Houdini to add the VEX plugin to the houdini DSO table.

As mentioned above, I included the compiled plugin. This means you (probably) don’t have to compile your own if you run Houdini 12.5 on a Windows x64 machine. I have tested the setup with Houdini version 12.5.427, on a windows 7 x64 machine. If you have a different Houdini version or use a different operating system, you will have to compile it yourself. On windows you can use the VexImageReader Visual Studio project. For more information on how to write and build DSO’s using the HDK, please read: Setting up the HDK for Houdini 12 with Visual Studio.

The code should be cross-platform compatible. For this tutorial all code, compile and project references are relative to Visual Studio 2008.

Notes on compiling the VexImageReader DSO

Compiling a VEX DSO is pretty much the same as compiling any other DSO. The only thing worth pointing out is that (in this case) we need to link against the FreeImage library. Again, if running the configuration mentioned above, you can use the precompiled FreeImage library (freeimage/Dist). Otherwise you need to compile your own. I have included the header and source files, but you probably want to get the latest version of FreeImage right here. FreeImage is a great Open Source library project for developers who would like to support popular graphics image formats like PNG, BMP, PSD, JPEG, TIFF and others as needed by today’s multimedia applications. FreeImage is easy to use, fast, multithreading safe, compatible with all 32-bit or 64-bit versions of Windows, and cross-platform (works both with Linux and Mac OS X).

To ensure the FreeImage library is included and linked properly, point to the right header files and library by editing the Visual Studio project properties:

After compiling the project VexImageReader.dll should be stored in the Output directory specified for the project. In my case that is: $(OutDir)\vex\VexImageReader.dll. Note the vex intermediate directory. This is of great importance for Houdini in order to pick it up on launch. Why this is will be discussed in the next section.

Loading the VEX DSO in Houdini

I found it rather difficult to get Houdini to pick up the VEX plugin. After reading through the Side Effects forum, Od-Force and HDK documentation I managed to create a setup that makes Houdini import the plugin on launch.

The HDK documentation states that, in order to install the DSO you need to: create a table in the Houdini path so that the VEX engine can find your plug-ins. The table lists the location of the dynamic link objects which should be loaded by the VEX engine. In order to do this we need to know how we can add objects to this table that are recognized by the VEX engine, and therefore included.

As you might know, Houdini uses the HOUDINI_PATH variable to look for releavant data to load or initialize. It also checks some pre-determined subdirectories of the directories included in the HOUDINI_PATH. One of those is called vex. In this subdirectory it expects to find a file called VEXdso. This VEXdso file lists additional VEX dso’s that need to be added to the VEX engine. This is where I added the VexImageReader.

The package you downloaded includes the VEXdso file that is required for loading the plugin (dependencies\vex\VEXdso). Copy the VEXdso file to a location that is included in the HOUDINI_PATH and make sure it’s in a subfolder of that path called “vex”. Houdini will parse the file and now look for a plugin called VexImageReader.dll.

The path to the VEXdso file should read: %HOUDINI_PATH%\vex\VEXdso

Now Houdini knows to look for a plugin with the name VexImageReader.dll, it will try to find it using the CUSTOM_DSO_PATH variable. Again, for VEX extensions Houdini appends “vex” to paths defined in there.

The path to the plugin should read: %CUSTOM_DSO_PATH%\vex\VexImageReader.dll

I created two user variables in Windows to automate the lookup. Both variables point to the same directory and contains a “vex” subfolder. This folder will contain the VEXdso file and all the vex plugins. This way everything is in one place and easier to edit or look after.

  • VEX_PATH = (YOUR PATH HERE)
  • CUSTOM_DSO_PATH : %VEX_PATH%\HoudiniDSO
  • HOUDINI_PATH : %VEX_PATH%\HoudiniDSO;&

The last thing to take care of is to include the location of the FreeImage.dll in the PATH variable. This way Houdini can find the FreeImage library when the function is evaluated.

If all went well Houdini is now able to add the plugin to the VEX engine. To verify that the plugin loaded correctly, type in a Houdini Textport the following: dsoinfo -vq This should print all the custom functions available to VEX and their signature.

Wrapping the function in an OTL

The plugin itself only adds a function to the VEX instruction set, accessible through code. It’s always nice to have a vop to access that functionality, so you can use it inside (for example) a vop sop. By using an inline VOP it’s easy to wrap the function in a vop operator and create an otl out of it. This otl becomes available to all the vop operators in Houdini (wow, imagine reading this without having any knowledge of the Houdini architecture. I truly hope people don’t).

When opening the example file (hipfile\image_sample.hip) you can see the function used in a SOP, COP and SHOP context. This means you can use it to manipulate geometry, composite images, build shaders and drive particles. Vex is a truly wonderful language and instruction set. Implementing your own vex extensions can be a bit tricky, and the setup is far from easy. Luckily the same settings apply for all compiled dso’s and the same setup can be used for all plugins.

Famous Last Words

The code and data included in this tutorial comes without any sort of warranty. Suggestions on how to improve the plugin are welcome! The only filter algorithm implemented is bilinear. Feel free to add other ones, this should be ‘relatively’ easy with the way the project is setup. The plugin is (should be) thread safe, did not encounter any issues with the number of threads set to 1. Images are cached and released based on the active operator.

Links

SESI HDK Vex Operators

Mario Marengo Saves The Day

Extracting DSO Information

 

Pacemaker Game Sketch

Last weekend I participated in the Global Game Jam 2013. For me it was the first time making a game from scratch and I liked the idea of returning to my student days of no sleep and sipping beer behind a monitor early in the morning. In the end we had a working platformer where all the elements are synced to a global pulse (beat) that changes over time in the level. The theme was: heartbeat.

As the bpm increases, so does the difficulty level (objects rotate, move.and disappear at a different interval). It also causes the audio to slowly change from a heartbeat to a more aggressive hardcore kick.

Unfortunately we had no real level / game designer and for that reason not a completely polished game. Everything came together 3 hours before the deadline. But it’s playable and does illustrate what we tried to accomplish.

I used Unity for the first time and must say I’m impressed. The component based  object model is flexible and almost everything you can think of is there: Audio, Shaders, Lights, Scripting / Coding environment, VS integration, etc. Definately worth trying out.

If you feel like playing it, I uploaded the game right: HERE. Use the arrow keys to move and press space to jump. Works better with an actual Controller (tried xbox 360).

 

Setting up the HDK for Houdini 12 with Visual Studio 2008

Thought I’d share my Visual Studio setup for creating and compiling custom DSO’s for Houdini 12. Included are a Solution, 2 project files, documented examples and a houdini file.

Get the package HERE

The reason I use Visual Studio is because of it’s great debugger and clear way of structuring (and maintaining) code. Unfortunately there isn’t much help on how to set VS up in order to use it with the Houdini development kit. This guide should have you up and running in no time. After completion you should be able to compile and load the compiled dso’s in Houdini and inspect them using the debugger.

Quick note on the HDK: The HDK is a comprehensive set of C++ libraries. These are the same libraries that the Side Effects programmers use to develop the Houdini family of products. With the HDK, you can create plugins that define nodes, commands etc. I often use the HDK to optimize earlier prototypes written Python and sometimes Vex. But this isn’t always necessary. Only when speed becomes an issue I rewrite some older operators to gain some responsiveness.

First things first. Houdini 12 is compiled for windows using MSVC9, shipped with Visual Studio 2008. For that reason it’s only possible to compile Houdini plugins with that specific Visual Studio compiler. So try to get your hands on a complete Visual Studio 2008 installation. I believe the express edition won’t work because you can’t link to dll’s, but never tried. When running a 64 bit version of Houdini, make sure to install the x64 compiler as well. This option is turned off by default. For this tutorial I will focus on the x64 version of Houdini.

After installation the compiler can be found here: C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64
If you haven’t included the x64 compiler during installation this directory is missing…

You should have a Houdini 12 version installed (duh ;)). This particular setup is tested with Houdini 12.1.33×64 and 12.1.92×64. Both versions worked fine. It’s important to remember that rather significant code changes can happen between Houdini releases, even daily builds. Therefore your code might stop working or won’t compile when switching to a different version. I try to stick with a specific release for as long as possible (most often Production builds), and only switch when necessary (new major release, a bug fix etc.)

Now Houdini and Visual studio are installed, it’s time to set up the Windows environment. We’ll do two things here:

  • Set up some environment variables that will be referenced in the visual studio solution and projects
  • Specify a new DSO path where the plugins will be stored

Open up the environment variables dialog in windows and add the following variables:

  • H12_PATH : C:\Program Files\Side Effects Software\Houdini %H12_VERSION% where Houdini is installed
  • H12_VERSION: 12.1.33 the version you are using
  • HOUDINI_DSO_PATH: %CUSTOM_DSO_PATH%;& extends the default dso path
  • CUSTOM_DSO_PATH: G:\WorkCoen\HoudiniDSO where the custom dsos are stored (customize to your liking)

For the second step we want to create a Solution that contains a project that can be compiled in x64. In the package provided you will find a solution that can be opened in Visual Studio 2008. This solution contains 2 projects that define 2 different types of operators: the well know SOP Star and my own UV Interpolator. After setting up the projects we can build them and debug the operators using VS.

First we want to check that each project is linked against the correct Houdini library. Open the project properties and browse to Linker > General > Additional Library Directories. This value for this field should state:

  • $(H12_PATH)\custom\houdini\dsolib

The Output File parameter controls where the dll will be stored. This directory needs to be included in the HOUDINI_DSO_PATH.
For that reason I pointed it to the CUSTOM_DSO_PATH variable.

  • $(CUSTOM_DSO_PATH)\$(ProjectName)_$(ConfigurationName).dll

Now for the actual header files. Houdini ships with a lot of them and they be found (in all their glory) right here:

  • $(H12_PATH)\toolkit\include
Make sure the Additional Include Directories parameter points to that directory

The command line arguments are important. They define how the operator is build by the compiler. Houdini uses a lot of arguments and I don’t know them by heart. What I do know is that the -O argument is an optimzation preventing me to debug my code. When you want to dive in and look for the source of a problem there are two important arguments to set / remove:

  • Remove -Ox (disables optimalization)
  • Add -d (enables debugging)
I currently use the following arguments:
  • -TP
  • -Zc:forScope
  • -DVERSION=”$(H12_VERSION)”
  • -DI386
  • -DWIN32
  • -DSWAP_BITFIELDS
  • -D_WIN32_WINNT=0x0501
  • -DWINVER=0x0501
  • -I .
  • -I “$(VS2008_PATH)/VC/include”
  • -I “C:/Program Files/Microsoft SDKs/Windows/v6.0A/Include”
  • -I “$(H12_PATH)/toolkit/include”
  • -MD
  • -EHsc
  • -GR
  • -DSESI_LITTLE_ENDIAN
  • -DNEED_SPECIALIZATION_STORAGE
  • -DAMD64
  • -DSIZEOF_VOID_P=8
  • -DMAKING_DSO

If we want to launch Houdini from within Visual Studio to debug the dso’s, it’s important to set up the Debugging properties. The Command property should point to the Houdini version you’d like to launch. In this case that’s:

  • $(H12_PATH)\bin\hmaster.exe (houdini master)

To ensure the dso is loaded when Houdini is launched, the CUSTOM_DSO_PATH variable is added to the HOUDINI_DSO_PATH variable, specified in the Environment property

  • HOUDINI_DSO_PATH=$(CUSTOM_DSO_PATH);&

With all of the above in place you should be able to build the projects and start Houdini by pressing F5 (launching the current active project). Everything should be loaded and an empty session should appear. From here on try to create the node (Star or UV Interpolator) and put down a break point somewhere in the source code. When the node is cooked the break point should be hit (if it’s in the right place). Happy debugging!

 

Some useful external resources:

Miko Live Demo

I’ve uploaded a video to Vimeo that shows what Miko is all about. The demo was recorded last week and ran about 60-150 fps.
To view the demo in HD, make sure to follow this link to the vimeo website, or click on the video itself.

Just a quick recap on what Miko is about:

Every image is decomposed in to key elements and recreated using various fields of data.

The sampled pixel data is used to drive a fluid simulation, with image recognition layered on top. Custom vector maps are derived to let elements `search` each other. The results can be displayed using fields, particles , geometry and image data.

A midi device can be used to control parts of the simulation and application. Various controls can be mapped such as particle decay rate, feature selection, image selection, vector map selection, particle count etc. A bpm counter is included to sync images and triggers to a feed of music.

My plan is to build an interface around the various elements that control the simulation and compositor. After that I’d like to start looking into OpenCL (or Cuda) and other forms of image control. If you’d like to help me out with the code or get a build, send me a mail: coen.klosters@gmail.com

This version is compiled using MSVC 2010 and runs stable on Windows 7, but I’d like to compile a version for OSX. Again, if you’re interested in helping me out here, let me know! I’d love to support multiple platforms.

Other thoughts? Send them over!

And for the non believers out there, a couple of quick pictures taken at Nachtwerk, where I had my first test run: