Resonance Repeat Project
Resonance Repeat is a VST3 plugin that applies user hand movements to control delay effects. It features motion control over the delay write-envelope, loop function, and delay parameter adjustments. Additionally, movement features like speed, the spread of fingers, and the duration of movements can be mapped directly to effect parameters. The resultant details in control signals allow for endless opportunities in sound design. Resonance Repeat is easy to use and speeds up the process of creating unique effects.
Architecture |
Resonance Repeat relies on an app called Resonance Sidekick that runs on the same machine as the plugin, outside of the plug-in’s host application. The Sidekick app interfaces with the webcam, processes image frames, derives normalized kinematic data, and passes results through an IPC thread to the plugin. The app is built for Mac Silicon and uses the Unix Domain Socket interface to achieve this transfer of data between the two processes.
The decision to side-load the application was made primarily to circumvent challenges with camera access through host applications that do not include the necessary attributes in their signatures. An extra benefit to sideloading the image analysis pipeline is sandboxing the heavy processing from the audio buffer to prevent delays in processing times during the callback period. A series of buffers were used in the IPC exchange to allow for graceful bypass of latent or dropped frames from the webcam processing.
Another major factor in deciding to side-load the image processing was the conflicting build systems required for dependencies of the project. The inclusion of the MediaPipe framework to derive hand landmark data from webcam frames complicated inclusion in a CMAKE build. In order to create a functional callback library that I could use in the standalone app, I sandboxed all of the MediaPipe processing and built my recognition processes in an altered clone of the MediaPipe repo. This allowed me to leverage the native Bazel build system for the MediaPipe callback to generate a dynamic library to use in the standalone app.
Build Stack |
Resonance Sidekick and Repeat are both built in C++. As mentioned, the Sidekick app uses the MediaPipe framework to interface with the camera and generate the initial hand landmarks. A parallel MediaPipe recognition app is also being used to generate pose landmarks(full body) which are used in normalization and the z-axis derivation from standard rgb webcam input. Windowing for the Sidekick app is provided by the GLFW library. I used the GLM library in Sidekick and Repeat for its vector members and ready-built math functions on vectors. DearImGui was used in both softwares to provide quick access to UI elements and an easy way to load textures and draw simple shapes for metering. I’m using LibSodium to encrypt the IPC data in Sidekick, and you guessed it, decrypt the data in the VST3. JUCE is used in the Sidekick app to load image files as binary data. GLAD is used as a shader compiler for the openGL processing in the Sidekick app.
The VST3 plugin uses JUCE more extensively to provide access to cross-platform deployment and a number of members that assisted greatly in quick development of the audio processing. I especially appreciated access to the envelope class which made quickly navigating the indexing of a target ramp value over time super simple.
Kinematic Analysis |
Recognized landmarks from the webcam image frame are returned from the ML pipeline as triplets of normalized values from 0.0 - 1.0 representing each landmark’s x,y, and z positions in the image frame. Sidekick performs a number of processes on the values to prepare usable data for the downstream plugin. Using the pose(body) landmarks, the origin for new position values is established at the mid-point between the shoulders.
While there are Z approximations generated in the recognition process, I found them to be inconsistent and wrote a simple z axis approximation helper function to create new z values based on comparisons of the lengths of rigid body distances. The preferred comparison is between the width of the head and width of the palm. As the distance between the palm landmarks grow and shrink relative tothe head width, the z value will increase and decrease respectively. If the head is not in the shot, the palm will be compared against the distance between the shoulders instead. This comparison against a consistent distance allows the system to work better when the user is at different distances from the webcam. To compensate for hand orientation relative to the camera, I ended up using the area of the palm between three points rather than a 1D distance. This allows the value to remain present when the hand is rotated even though the length of one of the three sides might approach zero.
I employed a number of different filtering methods in my processing of the position and derivative rates of change. I found the EMA (Exponential Moving Average) filter to be especially valuable in quickly smoothing out quick moving data points.
//simple EMA filter //c++ int lastX{}; float alpha = 0.5f; lastX = (lastX * alpha) +((1.0f-alpha) * newX); Additionally I used moving window averages on more important processes to get a more immediate view of the value without influence of the persistent memory of past values that exist in EMA filters. What really helped move my smoothing to a new level was the application of Kalman filters. After the first two consecutive positions are obtained it is simple to derive velocity and use it to predict the next position point. Averaging between the new collected position and the predicted position leverages the trajectory of past input in more closely approximating the true new position.
// Kalman filter for
// adaptive smoothing
// c++
vec3 lastPos{};
vec3 lastVel{};
// Skip Comparison to Init Reference Member
if(lastPos == vec3{0.0}){
lastPos = newPos;
return;
}else{
// Obtain Instantaneous Velocity
if(lastVel == vec3{0.0}){
lastVel = newPos - lastPos;
lastPos = newPos;
return;
}else{
// Combine Current Vel with Last Pos Meeting
// to Predict Current Pos
vec3 estPos = lastPos + lastVel;
lastVel = newPos - lastPos;
// Average Prediction With Reading to Obtain New Value
float alpha = 0.5f;
lastPos = (newPos*alpha) + ((1.0-alpha) * estPos);
return;
}
}I found it extremely helpful to use windowing to confirm various kinematic data and circumvent noisy signals and phantom recognition from the ML models. By requiring repeated conditional confirmation of a change of state I could achieve more stable and usable signals.
Control Constructs | Exploring Paradigms
My initial developments in the space were theremin-like in nature. I tied the absolute position of my hands directly to effect parameters and found plenty of success using the absolute positioning to control the effects. The precision required in this approach made operating the software tiresome and I began to look for ways to facilitate easier motion control.
The fore-mentioned kinematic analysis system stemmed from my curiosity in applying velocity and further derivatives of position over time to the effect control. I found that the relative position over time (speed) gave an easier way to affect control over the system as I wasn’t required to focus as much on my hands exact location in the camera frame.
Further experimentation led to the development of a high level state rotation system. This ‘Cue’ system also circumvents the Midas Touch problem and filters out unintentional movement from effecting state changes. The system requires a prep movement to be confirmed before an active movement in a distinct direction can be used to affect the system. In the motion control paradigm, this looks like a pull towards the body to enter the prep state followed by a push away from the body to enter the active state. After the system is calibrated, the active state is able to act as a flag that can be used to trigger envelopes similar to a MIDI-on message. In addition to being wicked fun to use, the paradigm allows a wealth of nuanced kinematic data to be derived from the high-level gesture that is connected to the intended user expression.
Another relative control that allows for immediate result are the implementations of static signs. The MediaPipe pre-trained recognizer task outputs a number of simple signs like closed_fist or thumbs_up by default. I found these to be useful in transitioning the state of the program so long as the sign is distinct enough to NOT be detected by mistake. I settled on a tri-sign as a ready method of invoking a rotational change mode. I’m actually using proximity confirmation between fingertips and the palm to derive this detection, but the ML pipeline can also be configured to recognize additional signs. I prefer DSP to circumvent accumulation of recognition error.
Of course, I would be remiss not to include the fan-favorite ‘pinch’ in the control paradigm though I must admit it was a late addition on my part. Eventually I found the various pinches useful in cycling through pre-determined values for delay and loop times, as well as selecting desired effect destinations for the rotation sign mode.
Control Constructs | Backend Parameter Arithmetic
These ideas are all well and good in practice, but it did take a bit of restructuring in the initial build to find a swift way of stacking parameter contributors to arrive at a final write to the APVTS tree that JUCE uses to track parameter values.
I decided to track effect parameters in the APVTS along with Cue and Loop active flags. While it would be simple enough to have every movement feature tracked by the APVTS, the number of kinematic values sent over the IPC thread was daunting and I didn’t want to risk overloading the plugin with processing or distract the user from core usable parameters. Tracking the resultant effect parameters and the cue/loop active and record states was enough to allow replay of the sounds after recording the motion control automation into the DAW.
Arriving at the resultant effect parameters required a system for accumulating contributing control signals at the advent of each process block. I created a simple ParamUtility class that handled stacking a number of contributing control sources and applying the resultant calculations towards a similar destination to the APVTS. I created a table in the UI that allows for addition of these contributors and routing from select movement features to effect parameters. The ParamUtility receives UI add/remove calls and handles the state of a pre-allocated array of Contributor structs to determine how many(if any) of the Contributors will be applied and to which destinations.
A separate GestureControl class was created to organize the focus of the macro control signals and provide a debounce to prevent accidental trigger within a cooldown timeframe. The Control sources were itemized in an enum (Cues, pinches, rotation sign, etc.) and precedence was given in order of most to least obvious control. The more obvious tri-sign is less likely to be unintentionally triggered so I gave it the priority when detected and allowed the other sources to follow suit in decreasing order of stability in detection. I devised a number of control functions that I found to adequately describe how the control was to be applied to parameter values. They fell into the main groups of momentary and continuous, dependent on whether control needed to be monitored over the duration of its activity.
Audio DSP |
The actual audio processing of the Repeat plugin is straight-forward. The motion control determines the writing of the current audio sample to a delay buffer which is mixed into the output signal a set period of time after it’s written. The feedback controls how much of this delayed signal is written back into the delay buffer along with any new delay writes. In an earlier version of the plugin, I began to heavily utilize the 100% feedback setting to use the device as a looper. This turned into such a useful feature that I added a second buffer specifically for the looping. This ‘freeze’ buffer can either be written to from a right index pinch, or from the normal delay cast once the freeze is active.
The freeze and delay buffers are both able to be replayed with a simple pitch modulation. The plugin alters the delay read index from the baseline delay by a sample read from an LFO table that is scaled by the pitch mod amount and incremented using the pitch mod rate.
I found it useful to create an ‘insert’ mode that plays back the dry signal at full volume and ducks it when a delay or freeze amplitude is present. This allows a DJ style use for the plugin when a consistent dry/wet mix is not advantageous. A diffusion network of cascaded All Pass Filters is available to be applied to the wet mix or the full insert output. Additionally, there are simple Low and High Pass filters available for the same destination.
As previously discussed in the ParamUtility section, all of these effect parameters can be controlled by movement features. I find it especially useful to map the duration of the cue to the feedback amount. This creates an easy way to shape the sound with natural movement. Other definite winners are spread of the hand to stereo spread, and average velocity to mod depth. There are a variety of kinematic values able to be mapped from each hand or both hands acting in unison.
In practice and with the combined result of all of the movement controls, I’ve found the plugin to be exceptional at quickly creating complex arrangements in parameter automations. From triggering loops to writing varying delay passes, it’s quick to move the sound.
Graphics |
I utilized a mixture of DearImGui classes and custom OpenGL pipelines to generate the graphics for the VST3 and application. The Sidekick application features a 3D model of the filtered landmarks, associated velocity vectors, and generated high level cues. Once the data is formatted into glm::vec3 data structures, rendering the connecting lines and background grid for context is as simple as creating some OpenGL boilerplate and applying some matrix transformations to obtain the correct viewpoint and scale of the rendered vertexes.
After initial publish of app demo’s, it was apparent that a webcam feed embedded into the main app GUI would be a welcome addition. After working out the trade-off of the webcam feed that was already being utilized by the ML pipeline, I decided to add some image overlays and texture manipulation to highlight the kinematic analysis and high level gesture analysis. The images were simple enough to add as overlays and rotate using built in dearImGui functions and some helpers for geometric transformations. The image filtering was a bit more complex to line up as I hadn’t worked with glsl formats before, but I found the syntax to be similar enough to C and quickly lined up some processes that altered color, warped geometry, and changed mask dependent on the focused control source and related kinematic data.
The Repeat plugin offered a unique opportunity to incorporate parameter metering in the displayed hand recognition display. I spent a bit of time coming up with a design centered around concentric rings and got some help from a graphic designer to spruce up the vector images a bit. I found combinations of arcs and rotating lines around the center of the hand landmarks that helped me see things like feedback amount or mod rate without having to turn my gaze back to the parameter label and moving slider on the slide. I landed on a collection of icons revolving around the right ‘Cue’ hand that show the major parameters of the effects sections. In hindsight it would appear that I’m missing a ‘freeze active’ and ‘record’ meter, but the waveform display that I generated at the top of the main display viewport do well enough to alert the user of current looping and delay information.
Performance |
When I first hooked up the gesture recognition across the IPC thread, it was a great joy to find that the latency on the plugin was low enough to be readily usable. I’m running the build on a Macbook M2 Silicon machine, and it can easily run at a 32 sample buffer size in Ableton Live. It is able to keep up with a substantially loaded mix, but I try to keep most of my demos down to 5 or so stems and minimize processing so that I can confidently run my video recording app in the background of the gesture detection app and DAW.
I’ve worked the DSP into a good spot and since it’s a basic audio processing project, I’m able to get clear and inspiring effects using the gesture control. During initial setups, I was worried that refreshing the gesture control data once every audio callback would hinder precision in control detail. Seeing as I do most of my use at low buffer sizes, it has not been a problem. If I were to continue to push the envelope for performance across larger buffer sizes, I’d certainly implement some interpolation of the control data to avoid staircasing across sample points and look into updating the gesture data multiple times per callback to accommodate updated controller information. That being said, if the sample rate is 44100 Hz and the webcam frame rate is set to 30 or even 60 fps, the control data transfer rate really only passes the buffer callback rate at around 1024 samples per buffer(30 fps is one frame every 33 milliseconds and the callback would be about every 24 milliseconds). I’m not accustomed to expect performance latency levels at that buffer size so the app won’t usually be in a situation where the frames are being captured faster than the audio callback gets called.
The features of the app really grew over the July-August timeframe. From my initial performance video in mid-July, I added the freeze buffer, routing matrix, control focus, improved GUI, preset savings, and refined a number of interface methods and DSP. I must admit, it took me a bit too long to get the snap to tempo right on the phase of the tremolo off of the PPQ value returned from the host. I got there eventually.
All around, I’ve found the project to be very inspiring to use, and a great workout as an added benefit! I can’t say I’ve really lost weight from waving my arms in the air, but I can confidently say my shoulders are looking plenty toned these days. ++
Takeaways |
Although I began work in gesture recognition in 2024, the Resonance Repeat project only began to take shape this year. Over the past eight months, I’ve really been learning a lot of the software development skills that I’ve been using on the project. Looking back on some of my earlier code, there are a number of things I’d do differently. One of the biggest lessons that I’ll apply moving forward is the importance of coding for scalability. While it does often take less time up front just to hack the processes together, a bit of extra time on the front end of the class and system design seem to pay out in dividends when maintaining or expanding on core features of an application. Along the same lines I have a newfound love of normalized values and implemented standards across process interfaces. Knowing how the value is going to be utilized downstream of generation is paramount in determining how it should be output from its current module for use. Lastly, a word on feature creep. It does absolutely burn to see the pages of the calendar flip by while a project goes un-published.(I can’t say I’ll ever understand how Da Vinci released so few works in his lifetime) I’m eager to continue cultivating my ability to prioritize and select the finest ideas to follow through on. That of course is the easy part. The harder flip-side of that skill is being able to cut the things you spend time on that yield the least return: even when they are deeply ingrained habits. Alas, all good things come in time and I’m grateful to be able to move forward with this intention from now on.
I hope you enjoyed learning a bit about my Motion Control Audio Effect project, Resonance Repeat. I’d love to hear your thoughts on the concept or implementation. Feel free to send an email at your convenience via the contact button below. Here’s to hoping you’re respecting your intuition and exploring this lovely world on your day to day. Til’ next time!
David M Allen ©August 23rd, 2026