This thesis — Choreographing the Extended Agent: performance graphics for dance theater — came pouring out of me in late spring and summer 2005, as I rushed to complete my degree at the MIT Media Lab and to devote myself entirely to art-making. Of course, the ideas and the projects I wrote up had evolved over a much longer span of time than that — I’d been at the Lab since 1998, and even then I’d brought some of the ideas for The Musical Creatures project with me from Cambridge University.
My aim in the thesis was to give a full account of my efforts to create a new AI-based effort for making art. This entailed not only conceptual but also highly technical work, which for non-programmers will make part of the going tough. But feel free to skip the forbidding technical sections; I hope you’ll find the more general analyses and descriptions rewarding. (My colleague Paul Kaiser promises to provide [soon!] a guided tour to the non-technical sections of particular interest to digital artists, scholars, and audience members.)
The thesis provides what I consider to be a devastating critique of existing methods in interactive art, which has been led into a terrible blind alley by out-moded tools and concepts (chief among them, the idea of “mapping”). It suggests that there are places better than analog synthesizers to look for structuring ideas for interactive art — on the one hand modern dance and on the other embodied artificial intelligence.
It then articulates this agent-based approach to creating art and gives detailed examples of projects generated in this fashion. These include several of our collaborative dance works here at OpenEnded, principally Loops, how long does the subject linger on the edge of the volume…, and 22. Two musical projects also receive considerable attention: The Musical Creatures and Loops Score.
The technical thread running through the thesis addresses agent-based artificial intelligence as a software engineering problem (rather than as a machine-learning problem, a robotics problem or a math problem). A software engineering problem necessitates solutions for actual software engineers. It is insufficient (but not uninteresting) to simply create new algorithms or machine-learning approaches if these developments cannot be incorporated into large, messy frameworks, or rapidly applied to odd, diverse domains (for example, art).
Therefore my thesis reviews and critiques the software tools and approaches I developed for the Synthetic Characters Group at MIT — which were at the time leading examples of large, collaborative and messy AI systems — and then presents the authoring environment I created for the later artworks. It is this software that we will be releasing as open source later this fall, so the thesis serves also as background for the software’s eventual users.
0 — Introduction (link- 1.1MB)
Includes abstract and introduction.
1 — Context (link- 7.1MB)
_Contextualizes the thesis, locating it between the three areas it is in contact with — choreography, artificial intelligence and computer graphics — and outlines its main arguments, both technical and conceptual. _
**On computation and dance, 1 **
A Computational Sensibility, 3
**On “mapping”, 12 **
Toward the agent, 20
**The agent, 23 **
Authorship and AI, 28; Emergence, Artificial Life and Digital Art, 33; Toward an aesthetics and a practice of the agent-based, 42; “Non-photorealism” and computer graphics, 43; Live computer graphics and the stage, 46; Towards ambiguous computational graphics, 49; Concluding remarks — looking forward, 52
2 — Beginnings (link- 13.1MB)
_Lays the groundwork for the agent framework, and surveys the particular starting point for the agents constructed for this thesis. It indicates how the agent-based might fit into an art practice, and what kinds of work AI architectures need to meet the requirements of a practicing artist. _
**C5 — An agent toolkit, 55 **
Hierarchical structure, 58; Complex value, 59; Dynamic structure, 61
**The generic pose-graph motor system, 63 **
The generic pose-graph, 69
**Critiquing c5, alphaWolf, 70 **
Locating c5, 73; alphaWolf, a large c5 installation, 77
**3 — Loops ** (link- 13MB)
About _Loops , an abstract portrait of Merce Cunningham — an artwork that critically develops a response to artificial life’s “anti-methodologies” of emergence.
An overview of the artwork, 87
Distributing change, 90
The motor systems, 91; Signals and expectations, 96; Naming, 98; Rendering Loops, 105
Concluding remarks — authorship and emergence, 107
**4 — The Music Creatures ** (link- 14.6MB)
Presents the sound-image installation The Music Creatures. This artwork, while not drawing upon human movement, helps define several aspects of Downie’s agent-based aesthetics and sharpens some of the strategies that it offers in dealing with the uncertainties of interaction.
An overview of the artwork , 125
“Bio-musicology” and agent based AI, 126; Bird song — ontogeny, 128
Narrative descriptions of exchange, network, line and tile , 130
exchange, 132; tile, 134; network, 136; line, 138
“Tactical” learning, 139
Long term learning and persistence, 139; The methodologies of learning, 144
Advanced flow control — coroutines, radial-basis channels, and an introduction to the “language interventions,” 147 > Authoring the passage of time, 151; Adapting bodies, 156; The generic radial-basis channel 1 — flow blending, 158;The generic radial-basis channel 2 — rediscovering action-selection, 164
Concluding remarks — The Music Creatures ‘ aesthetics, 169
**5 — The b-tracker framework & distance mapping ** (link- 4.2MB)
Discusses two general frameworks for constructing the perception systems for agents in complex worlds, which will be of specific use in the dance theater works. Also contains the most focused technical rebuke of “mapping,” a diffuse and dangerous concept in widespread use in interactive art.
The perception system, 174
The b-tracker “design pattern,” 177
The distance mapping algorithm, 190
The motion-scrubbing solution, 197; Concluding remarks, 199
**6 — The Diagram framework & Loops Score ** (link- 5.5MB)
Introduces _Loops Score , and more importantly collects the extensions to the agent framework, based on the lessons learnt in making The Music Creatures and Loops into a new agent toolkit: the Diagram framework. This framework is designed to offer new forms of authorial involvement in the creation and maintenance of agents.
Complex assemblages — the inversion (of the inversion) of control, 202
The Context Tree — a new “working memory” for agents, 206
The uses of the context tree, 212
Context-tree container classes, 214; Programming in the interstices — code injection, 216; Storing parts of the context tree — the technical support for naming, 219; Authoring systems that change over time — the inverted context-tree list, 224; The context-tree and system creation — sub-classable complex systems, 229
An annotation tag library for context-tree use in Java, 234
Execution orderings, 236
The Diagram framework — the channel / marker representation, 239
Action selection in the Diagram framework, 24
Loops Score — live computational music for Loops, 246
The open, process score, 247; Generator stacks, 255; Generator-level operations —the abstract balance, 257; — filtration / perceptual partial re-tracking, 259; Channel-level operations — the rolling culler, 260; — the fusion filter, 261; — modified time view, 266; —continuation momenta, 266; — opportunistic alignment, 268; Concluding remarks, 270
**7 — 22 & how long does the subject linger on the edge of the volume… ** (link- 28.4MB)
Presents _how long…_and _22, both pieces for interactive dance theater. _22_provides a focused attempt to reform computer graphics’ “non-photoreal” with new rendering techniques, while _how long…_represents a sustained effort to create a collaboration between digital visual imagery and choreography in a live setting.
**22, an overview, 275 **
The re-projection renderers, 277; Distorted color spaces, 288; 22, video / geometry motor system, 290; Concluding remarks, 291
how long does the subject linger on the edge of the volume…, an overview, 292
The problems of real-time motion-capture data , 294 > Unlabeled data, 296; Use of the b-tracker framework in real-time motion capture, 299; The dancer-level tracker, 303; “Tracking” higher level features, 304; Blendable body framework structures — point, line, plane, point, 308; Line acceptor stack language— more programming in the context-tree 314; Triangular topology vertex-positioning system — fast “alpha blending” in time and space, 319 , The spaces on stage, 325
The agents deployed in how long… , 330
the triangle — the trace of movement, 332 ; parachutes & accumulation — coordination without specification, 337; tree / stage machine / forest fire, 350 ; memory score — the trace of perception, 356; weaving — a hidden body acting with a simple diagram combiner, 359
Concluding remarks, 361
8 — Fluid, an environment for digital art-making (link- 9.8)
_Concludes the main body of this thesis with a description of a parallel thread — the custom graphical environment that allowed the agent-based approach to meet the realities of collaboration, rehearsal and improvisatory choreographic practice. _
A critique of existing environments, 365
Fluid, an overview , 375
Code in every box, 380; Runners and execution, 381; A (persistent) plug-in architecture, 385; Connectivity, 387; Multiplicative extensions — Alternative layers, 391; Fast visualization for the agent toolkit, 393; Expressing history, 398; A network of text: copy & paste as a version control system, 403; The flow of time — more controllable time markers, 406; Closing remarks 416
**Contributions & future work ** (link- 2.4MB)
Summarizes thesis contributions and describes two subsequent projects.
Horizon , 2005- , 421
Enlightenment , 2005-6, 422