The Kinect [Mac-Only]

Field 23+ has support for old-school Microsoft Kinect sensors, which, among other things, produce a depth map. Support is two-fold. First, there’s support in the Stage for a layer that happens to have the depth-map as a texture. Secondly there’s support for interrogating the depth map at various points directly.

Installation — Mac

Install Homebrew from here if you have not already done so — you’ll need the root password of the machine you are on (for the Logan center lab machines just ask). Then ask Homebrew, in a terminal, to go install the latest version of “FreeNect” for you:

brew install libusb libfreenect

Depth-map as texture

The Stage already supports textured layers with texture material from a variety of sources — images, ‘videos’, a ‘webcam’ etc. To that growing list, we’ll add the Kinect:

// make a layer
var k = _.stage.withKinectDepth()

// draw a filled rectangle over the whole
// stage
var f = new FLine()
f.rect(0,0,100,100)
f.filled=true
f.color=vec(1,1,1,1)

k.bakeTexture(f)
k.lines.f = f

// keep updating the window forever
while(_.stage.frame());

Option-up to begin this box yields:

Depth-map as data source

The code above is good for quickly putting something on the screen. More handy is writing code that accesses the depth map directly, and draws it in a different way:

var KinectDriver = Java.type("trace.input.KinectDriver")

// we'll reuse the same layer
var k = _.stage.withKinectDepth()

k.is3D = true

// lets make sure we can drive
// the camera around in 3d with the keyboard
k.makeKeyboardCamera()

// a little motion blur
_.stage.background.a = 0.3

var data = []
while(_.stage.frame())
{
	var f = new FLine()

	// draw 10000 random points
	for(var i=0;i<10000;i++)
	{
		x = 100*Math.random()
		y = 100*Math.random()
		
		// but at the 'correct' depth
		
		z = -KinectDriver.depthAt(x,y)*100*3
		
		f.moveTo(x,y,z)
	}

	f.pointed=true
	f.color=vec(1,1,1,0.8)
	f.pointSize=1

	k.bakeTexture(f)

	k.lines.f = f

    // bonus --- draw stripchart with motion info	
	// instantanous difference between successive depthmaps
	data.push(KinectDriver.difference)
	
	// plot it
	if (data.length>50)
	{
		data.splice(0, 1)
	}
	
	var f = new FLine()
	for(var i=0;i<data.length;i++)
	{
		f.lineTo(i*10 + _.frame.x+ _.frame.w, -data[i]/100 + _.frame.y + _.frame.h)
	}
	
	_.lines.f = f
}

Option-up to begin this box yields: