Imagine an artwork that is site-specific in the sense that it adapts to its surroundings, but flexible enough that it can adapt just as specifically to other sites. With Toddler, we planned to make such a piece, first as an interactive tabletop installation and eventually as a very large public art installation that could occupy the facades of buildings in the process of being either erected or taken down.
Start
In 2003 we were invited to tour the World Trade Center area, with the idea of suggesting possible public artworks there.
We were struck by the instability all around us: reconstruction was already underway, but massive tasks of deconstruction still loomed as condemned and contaminated buildings had to be disassembled with extreme care.
Afterwards we discussed making a work that could adapt to this instability. We imagined a piece consisting of multiple stations (tiny, wirelessly networked computers coupled with bright LED screens) that workmen could keep repositioning in disparate windows on different floors of a building as they worked on it. An external webcam, focussed on the facade, would detect the position of each screen, and the artwork would reconfigure itself to any changes to the configuration of its “body.”
This idea of a loosely networked body brought to mind a long-held interest we had in the movement of toddlers. They, too, occupy an unstable world that they must struggle to balance.
Later, we proposed making a tabletop version of the piece so that we could develop it on a small scale first. What follows is the gist of that proposal.
Toddler movement
Perfect beauty lies in the two extremes of human motion.
In one direction, beauty’s outer limit is explored by virtuoso dancers, whose grace derives from a smooth mastery of an interlocking vocabulary of movements. Such movements have become second nature to dancers after years of study, practice, and performance. The memories stored in their very muscles allow them to execute intricate sequences of choreography without hesitation or even deliberation.
In the other direction we find toddlers on the brink of movement. For them nothing is yet second nature. As they start figuring out how to crawl, stand, and walk, they have to think through every move. This thinking is clearly visible in the whole of their bodies, not hidden away in the confines of their skulls.
You can see that their ideas are forming right there in their bodies: in the hand reaching out for support or in the foot suddenly become the teetering pivot of a full step forward.
Installation
In Toddler, we match the child’s uncertain movement and uncertain knowledge with that of an artificially intelligent display system that has to work just as hard to make sense of an ever-shifting outside world.
This outside world consists of a tabletop on which 16 small LCD displays are haphazardly placed. The software program must figure out how best to form an image on those screens it can properly identify. This identification process is not so easy, for the tabletop world is constantly unsettled (or “disequilibriated”) by viewer interaction. People are free both to move any or all of the displays and to change the position and angle of the video camera.
Facing this uncertainty, the code does its best to figure out where its displays are now positioned. It relies on two types of inputs:
- movement sensing: accelerometers are attached to each screen;
- computer vision: a small DV camera attached to a flexible neck points at the tabletop.
When a viewer moves a display, its accelerometer alerts the code to the fact of that movement (a crude measure, which only distinguishes between stasis and movement, but tells nothing of trajectory.) This alert prompts the program to re-examine its video feed to see how it has changed. The program assigns a slow flicker to the display (or displays) in question, which serves to show whether it or they are still within its field of vision (which may have been altered by the viewer’s having moved the video camera as well).
Once a screen has been re-identified, the program displays a series of simple lines on it to help it deduce the display’s orientation (is it now in portrait or in landscape position?) Finally, when the code has completed this process and knows which screens are now at its disposal, it stops sending graphics to the screens outside its view and concentrates instead on forming a coherent representation in the new display configuration.
Its goal is to move like a toddler. To do so, it draws on a large library of toddler movements, pure 3D data obtained by means of optical motion capture that has effectively recorded the movement but not the appearance of a real human child. The program must now examine that library to see which of the movement phrases it can play out most coherently on the display configuration.
In action
A concrete example might help here. If for instance the viewer has clustered eight of the displays together, and set two others towards the edge of the field of view, the program might first choose a toddler motion that puts its body into poses more or less matching the configuraton of the main display cluster.
Next, it might search for a reaching or a stepping phrase that would allow that main body to extend itself out to the two displays at its periphery.
Bear in mind that all of this is happening in abstract graphics (a constructivist visual vocabulary of points, lines, and planes). Oddly, however, the human presence of the toddler transmits itself clearly to the viewer, whose whole cultural and genetic heredity has sensitized him or her to human movement above all other visual input (solely excepting facial expression).
Algorithms
The main challenge to be solved by the underlying algorithms is this: how to divide an abstract body into parts (limbs, joints, etc) that can take on the child’s motions.
The code accomplishes this work of differentiation by mimicking the cell differentiation studied in embryology (how poetically appropriate: the biological algorithms of foetal development enable the AI algorithms of a surrogate toddler body!)
These operations run in sequence:
A kind of mind
In the end, of course, it’s the viewer’s encounter with the artwork rather than the brilliance of its code that counts. Our hope is to create a deeply interactive work that goes far beyond the one-to-one interaction that tends to typify new media. In Toddler the viewer has direct control over key aspects of the work (both the displays and the camera), but the work’s response to those actions is extraordinarily complex. The viewer doesn’t simply trigger a series of automatic and more-or-less independent responses, but rather causes the artwork to rethink itself.
In this sense, the viewer doesn’t interact with a set of media materials (the usual case), but rather with a kind of mind. And this mind is just beginning to learn and to operate in the outside world, much like the child that is its inspiration and its source.
