QueTwo's Blog

thouoghts on telecommunications, programming, education and technology

Integrating the Microsoft Kinect into your Flex application

One of the topics I will be talking about during my MAX session this year is integrating game controllers into your Flex/AIR or ActionScript games in order to make more naturally interactive games.  There are a lot of examples of using the Wii controller online already — and plus with the next generation of video game consoles coming out soon, I thought it would be best to spend my time on the latest and greatest — the Microsoft Kinect.

For those who don’t know much about the Kinect, it is a pair of Web Cams — one RGB (like a normal Web Cam), and one that sees only infrared.  The combination of the two (along with a bunch of other electronics and software) allow the Kinect to have real depth-preception.  This means that being able to track people, hands and everything else become trivial.  The Microsoft Kinect was actually developed by PrimeSense, who originally made the device for PCs, rather than game consoles.  This means that the devices are actually compatible with PCs, rather than relying on a whole stack of hacks to get them to work. 

The biggest difficulty in getting the Kinect working is wading through all the pieces and parts that are needed to get it all working.  Documentation is in very short supply (and usually found within text files on GitHub).  Compounding the problem is that there are tens, if not hundreds of open-source projects revolving the Kinect — but unfortunately each depend on a different set of drivers, middleware and APIs — many which are NOT compatible with eachother. 

I will state right here that I am not an expert in all of these, but I did follow one path that made things work (after following MANY that didn’t).  Matt LeGrand and I are planning a separate website and blog that will explore these other paths more to help people make sense of what to choose and what not to choose.  My primary experience is getting the Kinect working with Windows 7, as I am well aware that my aging Mac laptop is not powerful enough to do much with the Kinect.  Yes, you need a modern computer in order to use the Kinect.

So, where would you start?  You obviously need the actual Microsoft Kinect controller.  There are two models in the market right now — one that is USB only, and one with a USB + Power supply.  The USB only version will ONLY work with the XBox, where the USB + Power Supply version works with PCs (the Kinect draws a few amp of power, which the XBox apparently can supply using USB).  The only difference between the two packages is the power brick and a USB power injection module.  If you end up with the wrong one, you are supposed to be able to buy the power brick and injection module, but I have no idea where one would pick that up.  I got my kit from Best Buy in the video game refurbished aisle (it was about $100, after an instant coupon).  The brand-new ones were about $140.

Before you plug it in, you need to install the drivers, middleware and APIs.  There are three well-known sets of drivers and middleware, but the ones I found worked for me are directly from PrimeSence.  The drivers and middleware are hosted on OpenNI’s website at http://www.openni.org/downloadfiles/opennimodules.  There are three downloads you need — the OpenNI Module (OpenNI Binaries), the PrimeSense NITE Middleware, and the PrimeSense Drivers (OpenNI Compliant Hardware Binaries).  Download only the 32-bit downloads, even if you are on a 64-bit Windows 7 install!!  I’ve found LOTS of issues with the 64-bit drivers that cause things to break.    Install the drivers, then the middleware, then the OpenNI APIs (in that order).

Finally, you will need the as3 modules.  There is an open-source project known as AS3OpenNI that makes programming towards the OpenNI APIs very simple.  Because AIR can’t directly talk to the APIs you have to use the included C++ application that proxies the driver calls to a TCP/IP connection.  I’m sure this will become easier in future versions of AIR.   AS3OpenNI takes all the hard word of processing the blob data that comes back from OpenNI and gives you either skeletal data (as a class), RGB, Depth, or “multitouch blob” data.  With this data you can read back data from multiple users and track their hands, head, neck, legs, etc.  I built all of my stuff on version 1.3.0, which is really stable.

Take a look at the examples in the AS3OpenNI project — they are pretty descriptive, and once you get all the parts working together, they work really, really well :) 

So, what did I make using the Kinect?  One of my first games I threw together was a simple version of Space Invaders.  I have AS3OpenNI track the right hand to move the player left and right, and when the right hand moves over the center point of the neck, I fire a missile.  The following code is all that is really needed (minus the setup of the player, etc.) :

protected function gotShipMove(event:ONISkeletonEvent):void
{
 var rightHand3D:NiPoint3D = event.rightHand; // get the right hand's x,y,z
 var rightHand:NiPoint2D = NiPoint3DUtil.convertRealWorldToScreen(rightHand3D, this.stage.width, this.stage.height);
    
 shipIcon.x = rightHand.pointX;
 if ((event.skeleton.rightHand.pointY > event.skeleton.neck.pointY) && canFire)
 {
  if (!fireInProgress)
  {
   fireInProgress = true;   // prevent 5,000 missiles from firing at once...
   var missile:MissileItem = new MissileItem(spaceInvadersCluster);
   missile.x = rightHand.pointX;
   missile.y = height - 64;
   missile.addEventListener("MissileFireComplete", missileFireComplete);
   missile.addEventListener("MissileFireHit", missileFireComplete);
   addElement(missile);
  }
 }
 else
 {
  fireInProgress = false;
 }
}

Watch my twitter stream for more information about our (Matt and I) new website dedicated to the Kinect and AS3 :)  I will also cover the game a lot more during MAX, so make sure to come!

About these ads

8 responses to “Integrating the Microsoft Kinect into your Flex application

  1. Pingback: Cool Stuff with the Flash Platform - 8/30/2011 | Remote Synthesis

  2. Pingback: Kinect and Flash and Unity3D « Jufa

  3. aidan mack September 16, 2011 at 4:58 am

    Hey,
    Im trying to follow your guide. But even after installing the required programs/drivers….
    Sensor-Win32-5.0.3.3-Redist.msi
    NITE-Win32-1.4.0.5-Dev.msi
    OpenNI-Win32-1.3.2.1-Dev.msi

    … It still asks me for the location of the drivers for the xbox nui ect ect.

    Do you no where I can get them from?

    Thanks
    Aidan

  4. Greg September 17, 2011 at 4:50 am

    Hi,
    A small correction:
    “Install the drivers, then the middleware, then the OpenNI APIs (in that order).” This won’t work in this order. If you try to install drivers first, installer will throw an error saying it needs OpenNI framework 1.2 or later. So you need to start out installing OpenNI API first.

    Now a tiny question:
    Do you need to set any parameters to make the C++ app talk to AIR? How do you set up this in AIR?
    Because when I launch AS3OpenNI_Alpha_1.3.0.exe and then I launch my AIR app, I don’t seem to get any data.

    thanks
    Greg

    • quetwo September 17, 2011 at 12:35 pm

      When I was talking about the drivers — I was meaning the Windows Drivers… I linked to those in my last comment.

      There are a set of parameters that need to be passed to the C++ app. If you use the as3OpenNI package, it actually will launch it with the proper parameters for you. You need to pass in if you are looking for depth maps, blob data, multitouch data, etc.

      The key is getting the proper path in the openNI.binaryPath property before you init. If you are having issues, create a folder C:\OpenNI\ and place the .exe and .dll in there. Then set the binaryPath property to “c:/OpenNI/AS3OpenNI_Alpha_1.3.0″, it as3OpenNI should launch it for you (it launches it hidden, so you won’t see it)

  5. Greg September 17, 2011 at 4:03 pm

    Thanks a lot quetwo! I got it working!

    Greg

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 28 other followers

%d bloggers like this: