using photoshop is writing code
Shelley and I spent a wonderful week and a half holed up in Chicago making the source materials for Point A to B. This is an atypical project for us, and a very unusual project for me, because rather than some technical high wire stunt involving multiple computers composing images under the constant pressure of real-time, it’s being delivered as a finished animation — a Quicktime movie. This means that I’ve been spending time with a broader range of software much more than I normally do, eager as we are to afford ourselves of the rare opportunity to actually edit, hand keyframe and tweak a finished piece that will just play back the same way each time. Simultaneously I’ve been preparing our codebase for its open-source debut, so I’ve been thinking about my code’s relationship with the other applications on my hard drive.
timelines
I’ve been struck and dismayed by how brittle and cumbersome our animation workflow is now that it’s much more conventional, and how compromised my carefully architected “open” code becomes when surrounded by the big standard software suites. Conventional “creative” software, it turns out, isn’t in a very good place. Worse, it’s all in the same place, and has been for a decade.
AfterEffects, Premiere, Motion, Final Cut, Soundtrack, and Logicare all built around timelines; Shake has one, and a keyframe editor; its repackaged remnants, Color, has one. Quicktime Player has an editing timeline (although many miss it in its attempt to look like a car radio). Maya has at least two timelines that I can find and one suspects there are more lurking in its turgid depths, written by an intern and left in a distant corner of the shipping product. 3d Studio Max has 3, I believe. The new version of iTunes got one for editing ringtones. One of the handful of new features in Photoshop CS3 is in fact a timeline of sorts, can it be long until Illustrato and InDesign get theirs? Perhaps they already do. Recently I’ve discovered WireTap Studio, and it has a timeline editor for editing audio that it captures from applications which have their own timelines.
Obviously you can argue this is a significant waste of engineering and QA resources, a waste of UI learning time, and a waste of money in general — $10,000s worth of the same idea. After a final round of consolidation in the market, these products are mainly made by one of the “three A’s”, Apple, Adobe or Autodesk. It doesn’t have to be like this, and if you were to generate screenshots of the scrollbars of each of the above applications you’d find only two species (sadly not one, there’s Apple’s and the undocumented Apple “pro”) both largely provided by the OS. Apple, themselves, are ideally placed to componentize an hierarchical timeline editor into the OS itself just as they have done with the scroll bar (and list view, a tree view and a side scrolling column view &c).
integration as dissolution
These arguments are surely playing out somewhere, but they largely miss the point. These products, that sometimes ship in the very same box as each other, are converging while simultaneously scrambling to maintain their independence. It’s not that, as a programmer I want a standardized timeline “widget” so that my applications can look as cool as Apple’s, nor do I, as an artist, want all the timelines to look and handle the same. As a programmer and as an artist, I want all the timelines to be the same. I want the same timeline across applications, I want a level of “integration” where the “applications” being “integrated” lose their identities. I need to be able to use MotionBuilder to integrate optical markers with sequences inside Maya, programmatically create Soundtrack markers for our sound designer, track changes in source material from Maya, synchronize video from the MoCap shoot, revise rendering keyframes in my own development environment Field (which, of course, has its own ideas about time editing); this needs to happen on the same timeline and I need to version control the whole thing across each of our machines in Chicago and New York. I can’t do any of this. And there’s no code that I can write that will.
In more technical terms I don’t care that much about the views’ design, I need live, distributed access to the models, I need my code and all the proprietary codes I use to be able to talk to and write controllers without instantiating or licensing or even being aware of anybody else’s favorite views. No amount of JavaScript / AppleScript / LADSPA / AudioUnits / OSC / SoundFlower and what-have is the answer — they don’t even help in formulating the solution — they are all interstitial technologies that do nothing to dissolve the actual problem. Nor is this a file format issue, this is not something that will be solved with more XML (for example, Collada). This is not a standards problem, or an interchange problem, it’s a problem that stems from the existence of things that treat interchange as special, and single application development as the norm.
I am old enough to remember OLE (which begat ActiveX), and OpenDoc (which begat a hasty Jobsian retreat) and their accompanying canonical vision: a Word document with a real live Excel graph in it. Both ideas went astray — ActiveX revolutionized nothing but the security industry and OpenDoc went the way of the Newton. But it was a exciting time, but mainly in a bad way. Microsoft sought to replace the monolithic Microsoft Word application with two or more monoliths — say, Word and Excel — tied together at the ankle. The resulting three-legged race through your poor machines’ virtual memory was simply too dangerous enter into with data you cared about. Back then, my best friend’s utterly tricked-out PC felt like it was the only machine in the world fast enough to almost load my 100 page high school physics thesis. Two years later the PC room at the Cavendish Laboratory was filled with terrified, desperate students who had just had all of their embedded Excel spreadsheets — model, view, controller and this year’s final grade — turn to black “X”’s by a mistimed Office network upgrade. As it recedes this technology transitions from enabling a densely connected, multi-view, multimedia document creation (what Office was supposed to become) to powering multimedia document consumption in a discrete lumps in a browser window (Internet Explorer). A much simpler problem, a much less significant domain.
The more radical OpenDoc project — which sought to evaporate the monolithic application completely — collapsed, quite possibly under the weight of its own engineering. But it’s not too much of a stretch to think that it threatened the still dominant business model of software companies who are ultimately in the business of making, testing, advertising, supporting, upgrading and selling monoliths. Too bad: in the mid 90s such a disruptive technology could only have come from a hardware manufacturer willing to use unique software API’s to sell tangible objects; in short, it could only have come from Apple. It didn’t. Instead we have a largely empty choice between Motion and AfterEffects, Pages or Word. It’s not about doing more so much as it’s about crashing less.
open source, open software
Today, the open source movement ought to have given this story a happy ending. It hasn’t. This ongoing attempt to duplicate Adobe/Autodesk/Apple’s software products button for button, feature for feature is regrettable, but it’s also inevitable once you accept the way the problem has been framed in the first place. GIMP has plugins and scripting as much as Photoshop, Blender as much as Maya, Open Office as much as Microsoft Office, and no more, no deeper, no differently.
This is upside down, and open source software should be different; the sheer thrill of a triumphant ./configure && make && sudo make install (the magic command line incantation that takes the very source that is “open” and turns it into a program that you can use) should be diffused throughout the software product until the very idea of product as separate from source or from use has been disassembled. It would look like Perl’s CPAN / Ruby’s Gems / Python’s PyPi (popular and dependable online repositories for shared, modular code); it would look a little like Eclipse (a plugin-centric development environment for programmers); it would look a little like Mathematica (a mathematicians ‘notebook’); it might look like Field; part Smalltalk, part Lisp Machine, it might feel, but hopefully not look, like Emacs (a superpowered text-editor that thinks it’s an operating system). But it wouldn’t have a name, or the branding of Microsoft Word. It would be hard to have an ad campaign for it. Losely coupling views would be gathered together to hew things that look like timelines and documents and spreadsheets out of fantastically deep shared ground of controllers and standardized models available for download. Code would be mobile, languages used diverse. Grandparents, students and asian animation farms would use different distributions prepackaged by downstream support companies. The language and ideas here are pure open source programming, pure Bazaar to Microsoft’s Cathedra, but they seldom ever reach the surface or open source applications. Richard Stalman talks of protecting the “freedom to tinker”, the fundemental freedom to reconfigure and alter software guaranteed by free software — but why try and make this freedom disappear once the code is compiled?
Here’s the reality: doing anything creative with an application is writing code. Layers in Photoshop / AfterEffects / Motion / Final Cut / Illustrator execute to produce an image; timelines are rules for producing edits; paragraphs and their associated style information compile to produce layout and pagination. Sometimes the domain, representation and “syntax” fit (layers in Photoshop, versus channel operations), sometimes they don’t (what does “convert for smart filters” in Photoshop even mean ?). Most of the time the layers palette is a better interface than a text editor full of code, but many times it isn’t — there should be both. Sometimes the programming pokes through anyway: I’ve debugged Word documents having trouble with their footnotes; I’ve had recourse to expressions in AfterEffects; and the excitement over web 2.0 is fundamentally about programability (the “mashup”) not about network latency reduction strategies. There are good, deep and convenient ways of manipulating code and there are bad, tedious and error prone ways; there there are domain appropriate languages and syntaxes that are just not up to the task. But above all, it’s better to have as many different ways of writing, manipulating, and invoking code as possible. From this perspective the boundaries of applications and the duplication that these boundaries necessitate seem downright petty. My code talks to other people’s code all the time, why can’t the airbrush in Photoshop talk to the Live Paint brush in Maya? Why can’t I provoke that conversation with a well turned page of code? What proprietary secrets are these brushes hiding from each other? From me?
Are we getting there? Applications are incorporating real programming languages. Photoshop is doing Matlab and has had JavaScript for a while, and Autodesk is doing Python now — a move away from the grotesquery that is either MEL or MaxScript that I endorse wholeheartedly — but it’s absurd for me to be as excited by this latter move as I am. It has let me inject code into MotionBuilder that streams joint angles across a network socket to codebase that I control, so that we can use their motion capture skeleton matching algorithms in our live renderer. It works like a charm, but doing this felt like I was exploiting a security hole, stealing information, spying on the insides of an application, doing something that I shouldn’t be doing — something that will break on an upgrade. In telling it, people inside Autodesk have looked at me like I was a little crazy. I’m doing this because I want to use their FK/IK engine, I want to pay them for the privilege of not having to write it myself and I already have it on my hard drive. But we’ve seen artists go so far as to try to make realtime artworks inside Maya just to get access to their mocap plugins (this didn’t end well). Rather than integrating Python into their products Autodesk should be disintegrating their products into a collection of Python libraries (or C, or something, anything). If they did, I’d write maintain and distribute the Java bindings for free.
In this imaginary world I would be free to right my code in, on or with Photoshop and make things that Adobe, Apple and Autodesk can’t imagine. A grand vision, but one out of step with the market. Today I read with disbelief that Apple has proudly announced that their new OS, the eagerly awaited Leopard, will allow you to record menu selections and keypresses and play them back!
“Integration” has a long way to go, and I swear our next animation will be edited in software I wrote.
—10/07