May 11
Infrared Trackin’
icon1 Zach | icon2 Development | icon4 05 11th, 2009| icon31 Comment »

Aight, so I’m going to go back in time to when I started working on our tracking system. The year is 1923, alcohol is illegal. That must have sucked. Fast forward to 2008. I had a decent tracking system working using color tracking with the JMyron library for processing, a webcam, and glow sticks but decided that it wasn’t reliable enough because the tracking system kept loosing the points. The distance and angle of the glow stick changed the color that the webcam was seeing.

Infrared seemed like the next logical step. Communication with another team working on New Media Team Project lead to the use of another processing library. Wrj4P5 grabs the image from the infrared camera in a wiimote and tracks ir points in a processing sketch. Combining this with the work I had already done with JMyron resulted in a prototype that was used in a presentation.

What next? Make it work better. I was running into trouble with the wiimote swapping points causing the paddle to flip. I spent some time devising a system that could tell point A from point B. The wiimote assigns the points to an array based on the x and y values of the point starting from the top left corner, so I wrote a class that each paddle end could be assigned to. This class keeps track of its last known point and goes through the wiimote IR points array and its current point is assigned to which ever point is closest to its last point. This seemed to have fixed the flipping problem for the most part. But there was still some funny business going on.

When you point the wands away from the wiimote the sketch loses the point and when it comes back it likes to flip. You can’t expect the user to be aware of pointing the wand directly at the wiimote. I added some more if statements that check how far the new point is from both ends of the paddle and a boolean that checks if the points array is less then two and thats were we are today, ever closer to a perfected tracking system.

There are still more bugs to be fixed. I tend to be a perfectionist when it comes to my development, especially when I’m working on something that I am excited about and interested in doing. I will continue to tweak the tracking engine in my spare time until I feel that its done.

Feb 14

enginetest02

Another engine test can be found here! This guy comes complete with two-point functionality.

Use W S A and D to change the offset of the two points.

http://myo.cias.rit.edu/discovery/processing/test02/

Feb 10

The other day, Linzi had a great idea, which I think it’s fair to say, has benefited all of the groups involved in New Media Team project this year. Originally, Adam’s plan was to pretty much not have lectures spring quarter, since there isn’t that much material to present. Instead of skipping lectures on the days when Michelle and Adam don’t have a lot to present to us, Linzi wanted to try sending representatives from the teams to other teams for feedback.This has deinitely benefitted team MYO: today marked an important turning point for us thanks to another fellow team. Among other feedback, they suggest that we try using IR tracking instead of color tracking for our game, since the color tracking was a bit finicky and spotty. Dun dun dun! Stay tuned for more on this from Zach.

Feb 4

enginetest01

As far as development goes, we’ve made some progress. We have created our first bare bones engine test with some of the games core functionality. A Java based version can be found here.

enjoy :D

Jan 26

A quick demo of glowsticks being tracked to create images on the screen. This is pretty much the system we would be using in our game.

Jan 23

To accomplish our goals we will need to develop a few basic game engines that will be used with each of our mini-games. We will be using Processing and a few Java-based libraries, most likely JMyron and Box2D, to accomplish this. We plan on building at least 2 different engines to base our games off of.

The “Color Engine” will use JMyron to help detect the players as a “blob.” This shape will be used to position any avatar or overlay graphics we may want to apply to each user. Users will also wear distinct colors, probably either glow-sticks or bright colored stickers, on specific body parts to not only help distinguish one player from the other but to also allow the game to track individual points of interaction (i.e. hands, feet, or head as cue points). Each game utilizing this engine will allow users to interact with game elements using their extremities.

The “Silhouette Engine” will utilize the silhouette that a user makes against the lit backdrop to interact with the game.

The silhouette will either be used with hit detection on game elements or to detect a user’s body position. An example of the latter being: a user is presented with a shape and they then need to fit their body/silhouette into this shape.

Mixing these two engines to create a third engine is also a possibility. Game physics will utilized Box2D to create a rich gaming experience for each user.

Problem Areas:
A considerable amount of the computer’s processing power will be devoted toward the camera tracking. Thankfully, Processing allows for OpenGL support. This lets the bulk of the graphics processing to be done by the computer’s graphics card. Provided we do not go overboard with our game assets, rendering them should not be an issue.

Problems may occur when the source resolution of the camera is set too high. If the invisible rendering of the source camera is too large, the computer will not be able to go through all of the pixels from the video feed fast enough, and the game will experience a considerable slowdown. The trick will be to find a “sweet spot” where we can successfully get all of the data we need from the camera (i.e. silhouettes, and trackable colors from the glowsticks) and still maintain a level of performance that is acceptable for gameplay. Ideally we would like to maintain 30 frames per second (FPS), this would mean our application would have to grab the silhouettes / key colors from the video 30 times every second.

The performance of the end product will mostly depend upon our ability to optimize both the resolution of the source video, and the efficiancy of our engine.

Jan 22
The development team has been busy researching and evaluating the different technologies we might be able to use in this project. This is what we’ve come up with so far.

AS3 vs. Processing

Actionscript

  • There seems to be a lot more existing motion tracking projects using AS3
  • Possible multi-use
  • Web deployable

Processing

  • Open source
  • Hardware accelerated
  • Uses openGL
  • Much greater performance
  • built in 3D
[end_columns]
Basic color tracking method

For Human Machine
Captured images are run through a few filters in order to more easily detect the center of the objects. First is a mean filter that causes the image to be blurred, reducing noise. An RGB filter is then used to remove all objects from the image except for those areas where the requested RGB values exist. The image is then run through a threshold filter that reduces it to a black and white representation of that image. The white areas are where the requested color exists and the black areas are where the color is absent. With this image areas of white can be clumped together to track multiple objects and the center of each of these objects can be calculated. Here is a tool that can help you see what I’m talking about. Follow the directions using the orange circle on the image. You will see that it can easily be picked out of the rest of the image. So with this method if we have fast processing we can track multiple objects, ie. glow sticks, by running each frame though this type of filter. Fun stuff!

For Snowball Fight
There are two ways we can go about interacting with this game. The first would be the above method. This would involve some type of glove or band that had a very unique color. The second would be with motion tracking. Most motion tracking that I have research functions pretty much the same way. The web cam sends images to the program and the code finds the difference in color of pixels between the current frame and the previous frames. These areas are where motion is taking place. The pixels that have changed are plotted and the center of the motion is determined from there. Here is an example of motion tracking in action along with its explanation.

I think the most effect way of motion tracking would be with color. It seems to be more reliable because the motion of your body will throw off a motion tracking system where in color tracking only the colors are registered. I am also interested in using the processing language because of its performance advantages and because I am interested in learning it. And that’s all I have to say about that.
Jan 22

As ideas came up during our brainstorms, our development team researched the possibilities for implementation of said ideas. At this point we’ve pretty much determined that we want to focus on video motion capture technology for our three ideas to allow the users to interact with our game. There are numerous ways to do that, from silhouette tracking to color tracking, etc. What resulted is a library of links to various videos, websites, etc, as well as some qualified assessments and evaluations of the use of technologies research for our needs.

Motion Tracking using a webcam and AS3


With Color…


[end_columns]