This essay was for an anthology on contemporary mapping, a topic that led to what I now regard as the unfortunate title given to the piece. “Mapping” is a pervasive and for the most part pernicious strategy in digital art, from which our work dissociates itself. The body of the essay still stands, however; perhaps the opening section on childhood is the most interesting.
If you cast your mind back to childhood, you will soon discover our topic — motion-mapping — most clearly and imaginatively revealed. There in the games and pursuits of your earliest years you can distinguish this phenomenon in its three most basic forms. You can also follow each form forward in time to find out what they grew into — forms of art.
You might start by bringing back the movement games you played so ardently as a child – in particular the games of imitation and mimicry, like copycat, follow the leader, charades, Simon says, and the innumerable variants you and your friends would improvise. These games had you map the movements of others onto your own body, sometimes in the simplest one-to-one transformations (as when you followed the leader), but sometimes more intricately, as when you mimicked a lion, or a horse, or a gladiator – or even the rain, or the wind, or an explosion. These are the games whose rules and roles lead to choreography and dance.
Next you might summon the countless hours you spent bringing inanimate objects to life – the freest kind of playing. Breaking a twig off a tree, for example, you’d make out that these little branchings could be limbs, and this little protuberance a head – and that the yard you now lay down on could be the vast world of this little creature, as you moved it about – and you yourself its god. Make-believe of this sort points forward in many directions, from science to literature, but for our purposes let us single out puppetry and figure animation.
Now try recalling something harder to remember because more distant in time – perhaps most easily recovered not from memory but from observation, if you happen to have a two-year-old nearby. The activity is drawing – but drawing in its purest state, which is scribbling, where the point is not the depiction of something, but rather the feeling of the motion itself and the delight at the lines of movement mapped on the paper below. This turns most obviously into the rapid marks of Zen calligraphy or of Jackson Pollock, but I also see it heading off more obscurely towards the technology of motion capture, where instead of a crayon it is a body inscribing itself not on a static piece of paper but on a flowing stream of data.
The three collaborative artworks I want to describe here all evolved, I now realize, from these deep experiences of childhood; and while my main topic is motion-mapping (as made possible by motion-capture), you might notice a second sort of mapping constantly present here: the mapping of artistic disciplines one onto another. Dance, drawing, puppetry, and filmmaking can be set into all sorts of reciprocal relationships when simulated – mimicked – mapped on a PC.
Ghostcatching, which I made with Bill T. Jones and Shelley Eshkar in 1999, can serve us first by illustrating the process of motion capture and its subsequent transformations. Here we can see our work illustrated in its six initial stages.
First, dancer/choreographer Bill T. Jones improvised different sequences of movement according to motifs and rules we formulated together.
The reflective markers worn by Jones at key points of his body were all that the infrared cameras encircling the motion capture space recorded. Each camera fed into a central workstation, which combined these inputs into a single data file.
Simple dots in 3D space represented the motion-capture data most conveniently. These showed not the markers themselves, but rather the corresponding points in the performer’s body as interpolated by the motion capture software.
The “biped,” a kinematic model created in Discreet’s character studio program, has an intricate structure of dependencies and constraints like that of the human body; it is like the most sophisticated marionette imaginable. But in this case its motions came not from a puppeteer pulling its strings, but rather from Jones dancing, for when mapped onto the biped, the motion capture file animated it.
An alternate body made of mathematical curves (splines) was modeled in 3D by Shelley Eshkar. He used the underlying biped as an invisible armature for this new body, which resembled a kind of wire sculpture. But the “wires” were flexible, their deformations and movements tied to the underlying motions of the biped.
Sampled charcoal and other scans were texture-mapped to the splines of the new body, so that when rendered it looked like a gesture drawing – but a gesture drawing inhabiting a 3D space and moving to Jones’ dance. Each frame of the final animation was a new drawing, created by the juncture of 3D model, motion-capture position, and virtual camera viewpoint.
This, then, was our process, but to what end? The idea that Eshkar and I had advanced came in part from understanding Jones’ usual choreographic practice, in which he first generated new movements by dancing them himself and then set them onto the members of his company, making adjustments for their disparate physiques, styles, and souls. Now we pictured Jones spinning off his movements not onto real dancers but rather onto virtual selves that we could bring to metaphoric life on the computer screen.
After studying Jones’ solo dancing more closely, we asked him to isolate and intensify certain aspects of his style that we had identified. In the resulting improvisations, Jones danced like a man possessed – possessed in turn by eight or nine distinct selves, whom we gave nicknames. There was “Dog,” for example, whose frenetic movements were parallel to the floor and whose attention was always outward, rapidly shifting from one spot to the next. There was “Ancestor,” who cycled slowly and endlessly through six fixed poses derived from a famous set of photos that Jones had made in his younger days with Tseng Kwong Chi and Keith Haring.
Jones was spooked a bit by all this, especially when his movements were first captured. He felt as if the machines were trying to steal his soul, what primitive people first suspected photographers of doing. He said we were “ghostcatching,” which gave us our title.
The 3D spaces that Eshkar and I subsequently constructed grew entirely out of the fragments of captured movements. Around the Ancestor’s fixed cycle, for instance, we drew a large upright box, which contained him like a dancer’s kinesphere — or a stranger’s telephone booth — or a grandfather’s coffin. When that figure spawned the next, breaking out of its frame, it started inscribing lines of its own, freely and ecstatically, in the surrounding space. These were the trajectories of its movement, which could be drawn not just by finger or toe, but also by shoulder or elbow or knee. After a time, however, the lines multiplied to the point where they choked off the space once more, establishing the recurring pattern in Ghostcatching of being captured and then breaking free — to be captured again, and again to break free.
In our next work, Eshkar and I used a similar technique of mapping motion-captured dance phrases to “hand-drawn” 3D figures, but with several differences. For one thing, our context had shifted: our projections now accompanied a live dance performance choreographed by Merce Cunningham, who had titled his piece BIPED after the software simulation we used. In our BIPED stage design, we had frontal projections from the balcony hitting a transparent scrim that covered the entire front of the proscenium.
This produced a strangely ambiguous perceptual zone in which our virtual figures seemed to float not only in front of the dancers on the stage, but even at times (impossibly) among and behind them. The mingling of projected figures and real dancers (never synchronized) in the same field of view made the audience ever more conscious of our motion-mapping. Since our captured phrases were drawn from the same choreography danced on stage, the movements of our projected figures either foreshadowed or recalled those of the real dancers, depending on the order determined by Cunningham’s chance operations. This was not simply a delayed visual duplication, however, for Eshkar and I had started to experiment more radically with our depiction of the figure, at times moving beyond the more recognizable hand-drawn forms of our previous works.
It’s not that we deviated from an early decision never to distort the underlying movements we motion-captured. We knew of course that we could make a virtual figure perform impossible leaps or spins or contortions, but we had no interest in these cartoon-like exaggerations. Lost in such special effects is not only the subtlety of movement, but also the crucial identification and alignment of viewer and dancer. Isn’t the most important mapping in dance the imagined projection of the viewer’s seated body into the dancer’s moving body? For this is what transports dance beyond the visual into the kinesthetic and the proprioceptive — the audience gets into the dance not just with their eyes, but also with their flesh, muscle, bone, and blood. It’s like dreaming, where the body is put under the benign and protective paralysis of sleep, but still shifts and twitches a bit to the tumult of mental action. Evolution and social upbringing have made our sensitivity to human movement greater than to any other kind — the speed, intricacy, and precision with which we read someone else’s motion (of facial expression, hand gesture, or whole body) is near miraculous. To sacrifice that intensity of perception, we had long since decided, would be foolish.
But it is that very intensity of perception that can enable an odd and compelling form of abstraction. Strip the visible signs of movement to the barest of minimums, and not only will you still distinguish the walk or the run or the leap or the tumble, but you will even still sense something of the size, the weight, the age, and the intention of the body behind it.
It was this idea that Eshkar and I now pursued. After BIPED’s premiere in Berkeley, we revised roughly a third of our sequences, shifting them towards this more minimal abstraction. Taking one of our gigantic hand-drawn figures, for example, we made it visible only in its intersection with a sparse composition of thin blue vertical lines, which lightened momentarily at the point of intersection. The nearly invisible figure seemed to loom even larger this way, the physical projection filled in and somehow magnified by the viewer’s interpolation.
We shifted not just the visibility but also the structure of our figures. Where in Ghostcatching they resembled drawings of a real body, in BIPED some started looking like invented anatomies, with new kinetic characteristics. These often read as figures only when set into motion. One, for instance, was a cluster of sticks, which were flung up into the air for a leap, then gathered back in on landing. This unreal figure didn’t distort the original movement, but rather emphasized it — and the viewer never lost the sense of the actual traced body, ghost-like and invisible, moving underneath.
What we hadn’t yet questioned was the hierarchy of our motion capture interpolations. Remember that the biped simulation we were using strictly maintained the constraints and dependencies of the body. If a shoulder shrugged, then the corresponding arm and hand moved with it (the term for this is forward kinematics); while, conversely, if a finger lifted up to point at something, the elbow and the shoulder joints rotated in turn, lifting the fore- and upper-arm with them (inverse kinematics). We had previously assumed that our figures should faithfully mirror or at least suggest these reciprocal linkages, which are as fundamental to the human body as they are to its simulation, but after BIPED we began to wonder about other possibilities.
We soon had the ideal motion data to explore these ideas. Long having desired a lasting record of Merce Cunningham’s own dancing, we had been blocked from motion-capturing him by his advanced age (he had just turned 80) and by the severe arthritis that now hobbled him. New advances in motion-capture technology, however, gave us a way around this obstacle. Rather than capturing his whole body, we could now focus in on just his hands, which remained marvels of virtuosity. (Later I came across Arlene Croce’s apt remark: “Cunningham’s hands are like chords of music; full articulation flows straight to the electric extremities. He really does seem to have in his little finger more than most dancers have in their whole bodies.” ) Merce still performed variations on a dazzling solo for his hands called Loops, which we were now able to capture.
The resulting data files tracked 21 points on each of the hands. What Eshkar and I now started pondering was how to connect them, with four distinct ways coming to mind.