Long ago, before the Wii came amount or was ever even mentioned, in 2007 I came up with a very dirty hack that could in theory act just like the Wii controller. I almost used it in an Insectoid performance but that never came about.
The way that it would work is that the user would have a device that would act just like an mouse. When the user left-clicks the computer starts recording the users movements and when they let go of the left mouse button an action takes place dependent on the gesture they made. The overall system uses a combination of OpenSebJ and StrokeIt software (Windows only), together with a mouse-like input device.
StrokeIt is gesture recognition software for Windows that does exactly what it says on the tin. The good thing about it is that gestures are programmable. So, even if I drew an S, I could have it execute any command I wanted to. The program worked in the background, so it would send commands to the active window, which is quite important.
The second program involved in making this work was OpenSebJ. Even as early as 2006 I was involved and inspired by Open Source software, so there’s no guessing why I chose this program. Essentially it is a multitrack audio editor but with one feature that I really needed: A sampler. I couldn’t find a free program anywhere that enabled me to map keyboard keys to a sound. Actually, there was one that was purpose built, though I can’t remember the name of it. Why I didn’t use this program was because I could only play one sound at a time. Hitting any other key would kill the previous sound. Not with OpenSebJ.
So, now that I had a way of tracking gestures and playing sounds according to gestures I needed to introduce the free movement. This is the part where my experiment becomes suddenly expensive!
You would’ve thought that after the invention of the mouse the air mouse would’ve been released straight afterwards. Not so. Presenter buttons only have to listen for a mouse click whereas air mice have to track movement, which is an especially trickier task. The technology behind some of the air mice I found was a gyroscope. I know very little about its construction but what I do know is that it let you track your movement… at a cost. Air mice were/are being sold at about £100.
The way I planned to use this was just to map the users’ gesture on left clicking and then execute an action based on this. Whether the software would accurately be able to detect movement at any angle is debatable, as I never got to try it out, but in principal it works.
The application for this could’ve been wide reaching. Performers could use it as a visual aid in their work, toy makers could use it for new exciting way to interact and even in education it has potential to teach people handwriting by giving them aural or visual feedback.
Then the Wii came out… Now this kind of thing is pretty easy to do:
I think I still will continue to explore this avenue. If I can get a modded Wii I may try this prototype again but with its controllers and then explore where I can take it next. Any thoughts are welcome.