A month after finishing Housebound, I am housebound again myself, following the second of two minor foot surgeries in two years.

Day after day on the sofa will have you yearning for the world outside your window, so when your next doctor’s appointment finally comes around again, it’s time for a little happiness.

Which is how it was that last Thursday, after the cab I’d caught to 80th and Columbus had got me there a quarter of an hour early, I hobbled inside a café for an iced tea, then sat myself down on the folksy wooden bench set on the sidewalk just outside its door. The day was bright and warm and dry, and for all of eight minutes I basked in the sunlight, trying to take everything in.

Across Columbus Avenue was the the little park that wraps around the back of the Natural History Museum, but closer at hand was the play of the sidewalk where I found so many things beckoning for my attention — a toddler riding regally by in her stroller, a tattooed furniture mover rolling a brown leather armchair on a diminutive wooden dolly, a high-heeled young beauty teetering past, the medallion at the end of a long chain around her neck striking her chest with every step.

It was actually a scrap of white paper that held my attention longest, skittering across the sidewalk at my feet. A page torn from …? — no, most likely a discarded sandwich wrapper from the bodega a few doors down. A whimsical wind had it going first one way and then another, lifting off briefly into the air then scraping across the rough concrete again, traversing the shadows of a little sapling planted near the curb, with its dark little leaves quivering madly on the leashes of their stems…

And all this part of the swirling dance of detritus on the street, which charmed me with its too-often-overlooked intricacy. I spotted a piece of transparent cellophane that had maintained a bit of its fragile shape, a folded corner suggesting the oblong volume of the original cigarette pack… And then, when the wind picked up, the cellophane went its own merry way while I was distracted by the discarded pages of yesterday’s news spreading broadsheet or tabloid wings to soar sideways a step or two before crumpling back down to earth again…

 

Peripheral vision

A childhood fascination with peripheral vision, which persists to this day, came to me from the stories my father would tell about manning the look-out post on the conning tower of his submarine. He’d always had keen eyesight, but in the Navy he was taught that in the dark of night you can sometimes see more out of the corner of your eye than directly.

Later on, I concentrated on the metaphorical truth of this observation, but more recently I’ve circled back to its literal meaning, which helps explain some of the more interesting aspects of stereopticon imaging.

 

Convergence

Take for example this right-eye shot from Housebound, in which among other things a scrap of paper skitters across the busy intersection in the direction indicated by the added arrow:

In editing the stereo footage, you have to displace the two images slightly to trigger the illusion of depth, tricking your eyes into converging as if onto a real object in real space. You can’t get the right displacement for the whole expanse of the image, but instead have to pick a point to optimize — a point you hope will throw the whole picture into the most compelling perspective.

For this shot, it was that scrap of paper that we picked, and indeed your eyes do resolve it into perfect 3D. However, more interesting is what this does to the rest of the image, much of which you now have to take in peripherally. All of a sudden the slightly blurred pedestrian activity and the nearby play of sunlight assert their presence more forcefully, somehow more lifelike because you apprehend them at the edge of your vision, precisely as you do when navigating a real crosswalk yourself.

 

Adjustable illusion

Our eyes crave meaning, finding it even where there is none (in clouds, in splotches of ink, in the flames of a fire). So too with the illusion of depth, which our eyes conjure up on the scantiest of evidence.

In looking at a stereo image, especially after a little practice, you can force disparate parts of the scene into depth by deliberately shifting the triangulation of your two eyes. Since two sectors of the image rarely have the same displacement of right and left layers, it’s only with this kind of ocular adjustment that you can move around within the space of the image, which feels a bit like traversing the space both of the scene and of your skull.

Nowhere was this made clearer in Housebound than in this shot of pigeons crisscrossing our window view at unexpected speeds, elevations, and depths. It is the birds’ movement that initially makes them visible (the added arrows are needed to make them out in the still).

Your eyeballs keep readjusting to snap a particular flight into focus.

 

Whirlwinds & parades

In New York City, people toss basketballs and they toss trash, sometimes in the same place. Too bad that during the shooting of Housebound it proved so hard to align and focus our ungainly stereo rig, for had it been a quicker process we’d have captured a surprising spectacle that unfolded above the school playground pictured in the shot above (an area just below the view framed by the window in the shot further above).

Strong winds swirled across this wide open space, which was bounded to the west by a large school building and to the east only by Broadway and the subway line that runs aboveground for 13 blocks there. The winds had curved together to form a real whirlwind that picked up a dense array of old newspapers, plastic bags, advertising circulars, and other debris, arcing through the air many feet above the ground in the most amazing fashion.

This manifestation of urban nature put me in mind of the ticker-tape parades in the Canyon of Heroes downtown:

—Ticker-tape in name only nowadays, since stock market updates have long since migrated from paper to computer screen.

Like the chain of so many New York City associations, this one leads back to September 11th, when the jetliner collisions that brought down the twin towers of the World Trade Center also released a storm of paper from all the collapsing offices there.

 

Homing

But let me not end this there.

The other day my daughter and I gazed out from our front window over the Harlem plain towards the East River, which could be made out only in the form of the towers of the Triboro Bridge rising above it.

What arrested our attention in the slanting late-afternoon light was a flock of birds that kept circling between two high-rises in the distance. There must have been 50 or more of these birds — pigeons presumably — and they flew as tightly packed together as a cluster of midges on a summer’s day.

As they arced back and forth through the air, they flashed white and then dark as they were caught in sunlight and then in shadow — like the dot-dash of some long forgotten code, elaborated in the old days when pigeon fanciers schooled their rooftop flocks on their tenement roofs and then dispatched them to the skies above the city.

An old trick of optics — that of the camera obscura — once conjured up an unexpectedly enthralling vision.

This was when I’d stumbled upon The Magic Eye at the Cliff House in San Francisco, and found its revolving periscope-style lens taking in Ocean Beach and Seal Rock at the edge of the Pacific.

The landscape reflected within the disc of the mirror below floated like an apparition — so much more vivid somehow than the real thing itself, as it had presented itself to my eyes just moments before outside. Seagulls banked into view, then hovered nearly motionless: suspended there by the wind and by the reflection simultaneously, it somehow seemed, near enough to touch if only one could.

But perhaps it’s precisely because I couldn’t touch this vision that it seemed so preternaturally real.

from “Hic et Ille” in The Dyer’s Hand

Auden once explained why:

As seen reflected in a mirror, a room or a landscape seems more solidly there in space than when looked at directly. In that purely visual world nothing can be hailed, moved, smashed, or eaten, and it is only the observer himself who, by shifting his position or closing his eyes, can change.

 

Forced perspective

Looking through a magnifying glass or the wrong end of a telescope can give rise to similarly powerful visions, as if by intensifying the artifice of vision we also intensify our experience of it.

Two weeks ago, at Marc’s place in Chicago, we were playing with various stereo viewers in our first attempt to set the texts and diagrams of Other Bodies in 3d. Marc said something surprising would happen if I were to look at the world through the largest of these viewers, which effectively more than doubled the distance between my eyes:

And indeed peering down their hallway there, everything stood out in wonderfully exaggerated perspective, as if depth had asserted itself much more powerfully over its cousins height and width. I felt like an altogether different creature, browsing the fresh world around me.

 

More mirrors

The stereoscopic illusion of Housebound depends upon mirrors to put an images for the left and for right eyes at the same distance apart as between your eyes. This can be seen from Shelley’s installation diagram below and by my snapshot of Marc peering into the jerry-rigged viewer he’d set up on his dining table.

The effectiveness of stereoptic vision comes from how oddly it combines sensations of extreme realism with extreme unreality. The act of gazing into such illusions forces you back onto yourself, as you grapple with the whole apparatus of perception that has suddenly become palpable to you.

(I’ve written about this before in two related posts: eye as object and 3d viewers.)

The issues that I raised in my two previous posts about software — both posts essentially about different views of the “openness” of software — came into sharp focus last month during the shoot for Depth.

We’ve always had an improvisatory approach to workflow — partly out of necessity (there’s seldom any time to properly “rehearse” projects) and partly through choice (pieces are often only found “in the making of them” rather than by careful planning, and conversely rehearsals tend to become pieces). Depth is a series of works that marks a departure for OpenEnded — they are all films of sorts, the first, Housebound, even has a screenplay. That said, they each aim to invent a new way of capturing the world photographically, the series concerns inventing cameras, which, in turn, necessitates inventing workflows.

 

In Stereo

As such, Housebound was recorded in stereo using this elaborate contraption:

A moment of shear gear-headedness: the mount is borrowed from Alain Derobe, a 3d-film specialist in Paris, and uses a half silvered mirror to allow you to effectively get the cameras closer together than their bodies would allow (we continue to be confused by images found online of rigs for sale that have with very wide baselines — are these for landscapes?). The cameras are a pair of Sony XDCAM-EX1s that send HD-SDI out to a pair of BlackMagic Decklink cards in a pair of Mac Pros each with their own three disk Raid 0 array to record the whole affair direct to glorious uncompressed 1080p30, 10-bit 4:2:2. And I do mean glorious, these cameras with this recording technique has an “authority” that to date I’ve only associated with film.

Regardless, the piece calls for the careful, 3d insertion of text fragments into the footage that seem to hover in space, but in real space (that is, they obey the physics of our hand-held camera material).

Attaching text to an object in the video footage is a relatively straightforward point tracking problem — you have the typical redundant non-choice of tools: Motion, After Effects, Nuke, didn’t Maya have a point-tracker somewhere? I had vaguely assumed (that is: was planning to improvise around) that we could make the tracking super-easy by introducing markers into the scene and then removing them in post. I had assumed that this would be straightforward (despite the absence of a background plate given our moving camera) because after all, we know where the marker is in the scene (we’ve tracked it). How I was planning on removing the marker from the scene turned out to be a little less well planned.

 

How to make a healing brush

Adobe Photoshop’s “Healing Brush”, if you haven’t tried it, is a genuine marvel. While almost every other thing that Photoshop does is pretty much obvious to anybody well versed in 2d image manipulation (quick: try to guess the number of unique gaussian blur implementations that Apple alone ship with their OS) this brush that magically removes blemishes while creating exactly the right sense of texture truly does seem to do something magical. And it wasn’t obvious to my eyes or brain just how it was doing it (at least it wasn’t before Siggraph 2003). It’s more than up to the task of removing our tracking markers from a video scene.

Knowing that Adobe had so solidly solved the marker removal problem I synthesized, in the back of my preproduction mind, a completely incorrect assumption: that the most recent version of After Effects would also have this wonderful feature.

It doesn’t. Incredible! Adobe ought to own the wire, scratch, marker removal and intelligent matting problem with this technology, and would do had the code that’s in Photoshop travelled down the hall to After Effects.

This omission is understandable within and only within the “large monolithic product” framework, where each product is a separate sovereign, competing state. But Adobe itself has been amongst the most aggressive in pushing a slightly more enlightened viewpoint: product as plugin-host. After Effects already has the UI for offering the healing brush (it possesses a conventional clone tool), but they remain pathetically trapped in a software framework of their own making. A prediction: just as Yahoo plan to “turn themselves inside out” into a “re-mixable” set of services, Adobe will ultimately be forced to do the same.

But for now neither their After Effects team, nor I, get their wonderful engineering in the form that we actually want it: an API that you can call from code, any old code, your code, anywhere.

What next? What any post-academic would do: read some papers, attempt to find out how they did it, implement it yourself.

We’re in a surprisingly good spot here: Adobe seems to have been permitting their senior hard-math figure Todor Georgiev to publish actual academic papers rather than just patents. Even better (and perhaps precipitating Georgiev’s release) Andrew Blake and friends published it first (Blake is a member of a small circle of computer graphics people whose papers are, strictly, always worth a read).

Still, both papers suffer from a two habits shared by academia and industry alike: Firstly they are a tease: they stop short of telling precisely you how to do it and switch to selling you on just how good their approach is. Georgiev prefaces his paper with an inspiring, and utterly un-SIGGRAPH theory of digital image representation that draws upon math that I recognize only because I once studied General Relativity; Pérez, Gangnet and Blake, in Poisson Image Editing (now there’s a Google-proof title), take you almost all the way there and leave you with the following sentence “results in this paper have been computed using either Gauss-Seidel iteration with successive over-relaxation or V-cycle multigrid”. What they actually mean is that they solved their equation 7 by doing pretty much the first thing that pops into your head if you’ve understood the problem this far. Why not say this? Perhaps it leaves room for the inevitable follow up paper “GPU accelerated Poisson Image Editing” where the intern / talented-undergrad reveals that Gauss-Seidel / SOR for this particular problem is essentially equivalent to the much easier to accelerate and understand “blurring the image a lot” algorithm.

 

How to write about the healing-brush

But the real crime is this: neither paper comes with source code. There’s no good reason for this, and plenty of bad reasons. Space isn’t a good reason — who reads proceedings in paper form? Relevance isn’t one either — this is a self contained problem with well-defined inputs and no architectural issues. In the last academic field I sincerely was a part of — large scale, messy, integrative AI — there really was an argument for not distracting people by sharing your code with them. What you want to convey with a “systems paper” are design decisions, tradeoffs and perhaps a certain philosophy of approach. None of these things are well captured by the language of code, and nobody takes seriously the scientific criterion of reproducibility when it comes to writings about software design. But in this case here, what the authors are at least pretending that they want to convey is exactly what they did in order to get the results they show. In this case, no special infrastructure is required: their algorithms take two images (original image and mask) and produce a third — you can use your favorite c++ image class for all I care — an image is an image. Their code is the ground truth for exactly what they did, any other level of description, for such small, self contained problems, is icing on the cake.

To not share the very code that they have written (and clearly extensively polished) to make their very accomplished demos would seem extremely odd if it weren’t so common-place. Ultimately it’s nothing less than two-faced: it claims to be committed to the exchange of ideas but not too committed. It’s hard to know how we got here — is it the hope of commercialization? fear of academic competition? laziness?

I am bound by no such desires or fears. And to provide a satisfying punch-line to this argument I’m placing a Field based open-source implementation of something very Healing-Brush-ish available for download by anybody (anybody, that is, that’s willing to accept the terms of the license) from the Field development site. It’s probably less than a SIGGRAPH paper column-long. Heck, it’s even GPU accelerated. Here’s it working:

Take it, go do something interesting with it.

 

The academic game

My frustration with academic pseudo-publishing continued for the rest of the week with another surprise. Due to the half silvered mirror in our stereo-rig, our camera pair doesn’t provide the same image when presented with the same scene. The mirror seems to reflect more green than it transmits, and the 300MB/s second pair of video that we obtain need to be carefully color matched with each other.

Unlike “healing brushes” there’s hardly any interesting maths to color matching — just user interface. For this problem, we find ourselves again a little far from the mainstream. But it’s just as much fun to build your own interactive color matching tool in Field:

That wasn’t a problem. But had I thought about this before hand I would surely have realized that this was going to cause problems with my dense optical flow code — algorithms that try to tell you what each bit of one image corresponds to in another. Generally such code depends on a pixel in one image being the same-ish color as a pixel in the other frame, in order to find the correspondences. Not to worry, I would have thought, had I been thinking about it at all: optical flow has been a topic of research in computer vision since David Marr was inventing the field itself in the 70s — there’s plenty of research that I can draw upon.

Well yes and no. This too is a problem with low infrastructural complexity — take two images of a scene separated slightly by space or time and produce another — an image of “arrows” that point from a pixel in one image to a pixel in the other. But this too is a place where hardly anyone is sharing actual honest code. Worse, since this is a far older problem than image in-painting, it has had time to shed its SIGGRAPH “ain’t my movie cool” sparkle and has now become a serious game played by serious players.

But the emphasis is very much on game. There are standard images for you to fight on, and, unbelievably, even a league table. Optical flow algorithms are being evaluated with rules that might be slightly less open than the Netflix prize — freely available training data with hidden test data, online submission, public rankings, anonymous submitters and, in this case, few rules concerning publication. When I checked, topping the charts this conference season are three or four utterly unpublished algorithms by masked authors. There’s a web 2.0 interface so you can browse just how good they are, but you can’t get hold of them yourself. What is the prize I wonder? Tenure? A job at MSR? Weta Digital? Surely not, at this point, a video compression start-up?

Spurious arguments about intellectual property have no place here — you are either committed to sharing ideas or not; you are either working on your dissertation or you are working on your startup. Too many people are essentially getting the intellectual equivalent of patents published by the ACM and IEEE, and these organizations’ willingness to add a quicktime to any paper only encourages this misdirection. Sure, they don’t have the legal weight or the financial future of a patent, but they mark something just as precious for the tenure track faculty or the career researcher: territory.

The text for Housebound was put to paper over the course of six weeks’ service on a Grand Jury convened at the Special Drug Court in lower Manhattan.This was several springtimes ago.

The long afternoons in the courtroom there entailed much sitting around, a confinement completely negligible compared to what was in store for most of the accused, soon to join the truly staggering number of prisoners that the United States, alone in the world, has been busily putting behind bars for decades now (as I read again in the Times three days ago: see “Inmate Count in U.S. Dwarfs Other Nations’“).

Still, our confinement was boring enough, propelling many of us towards inward escapes: crossword puzzles for some, writing for me.

From time to time we’d stir to collective life long enough to follow and then to rubberstamp a prosecutor’s invariably unchallenged indictment of some small-time drug dealer or abuser. Various undercovers (as they were called) came to testify, and it was these faces of the War on Drugs that proved most surprising — had we been anywhere else, I’d have have never guessed they weren’t truly the college student or junkie or handyman that they impersonated so expertly (or were so close to actually being).

Looking out the windows of my apartment in the evenings afterwards, I’d wonder who in the parade of pedestrians on Amsterdam Avenue below were themselves undercovers — or, for that matter, the petty criminals they pursued (since not a few of the arrests chronicled downtown were made in or near our neighborhood, notwithstanding its accelerating gentrification).

I work at home, which all too often means that I pretend to work: idly looking out the streaked windows or gazing at the pockmarked walls — hoping that at least a few of my ideas actually take wing without immediately escaping me. Naturally it was this apartment — these surfaces, these spaces, these shadows, and these views — that I pictured when piecing together Housebound’s sentences and shot-list. Still, when I took out a camera and peered at the place through an actual viewfinder, I discovered its shortcomings. The shots I framed were inevitably too cluttered with family possessions, and no composition was quite clean enough, by which I mean emptied out.

An alternative location came to us almost immediately upon Shelley’s placing an ad on Craigslist — a nearly identical prewar apartment thirteen blocks north of mine, with the same southeastern exposure, but with more dramatic views over Broadway rather than Amsterdam. The family living there vacated the two front rooms for us, and this is where we performed our complicated stereoscopic shoot.

This location proved superior to my place in every respect but one: its walls, unlike mine, had been re-sanded, re-plastered, and re-painted in someone’s recent memory. Thus they lacked the singularity of the walls I am used to — walls that bear traces not only of the many lives lived within them over the years, but even (or so it seems to me) of the innumerable thoughts that have bounced off or been absorbed by them. In short, such walls as these are true the way that a man or woman’s wrinkles are.

 

trued

to true a wall, however, means quite the opposite of what it should.

I remember being struck by the phrase one night in the early 90s when I was still new to the city. Perched uncomfortably on a Frank Gehry cardboard chair, I was mutely following the dinner conversation of middling but affluent artists and their partners. We were gathered at a MacDougal Street townhouse bought by our host (a painter) in the flush of the 80s art market.

The talk was less of art than of home improvement, though the latter phrase was not one they’d have used — and in any case the two categories proved nearly interchangeable here. truing the walls and corners of any given room was evidently a near-sacred task, as were a host of other aesthetic interventions that I can no longer recall.

These artists aspired to live in a kind of spatial (and decidedly materialist) utopia, as if they might inhabit the stripped-down spaces of Minimalist sculpture (of Donald Judd’s, especially). One spouse summed up the whole evening with the glib remark that all artists, no matter how impoverished, instinctively master the art of fine and simple living.

trued walls, indeed.

(Note: compile a list of linguistic cousins: spun, finessed, and what others?)

 

Settling

This old building keeps settling into its foundation in much the same way that I keep settling into my chair, trying to find just the right position for my creaky skeleton, worn out from hours of reading and daydreaming.

Just as by law all non-French films must be voiced over before being projected in French cinemas, so too must all foreign programs be dubbed before being broadcast on television. Though the government also limits the proportion of such shows, you do come across a fair number of them as you idly change channels — Law and Order and CSI Miami rendered peculiarly disembodied by the robust French voicings and the mismatched lip syncs. Only once have I had the reverse experience, back in the 70s when I saw Chabrol’s «Le Boucher» dubbed into hearty American English, a distraction that destroyed my viewing of it.

This was brought back to mind just now, on my return flight to New York, when my eye was caught by a related headline in the European edition of the Wall Street Journal:

On Polish TV, familiar voices. Viewers like dialogue done deep and flat; husky ‘Housewives.’

American shows aren’t dubbed by actors mimicking the original, English-speaking actors. A lektor, the Polish term for voiceover artists, reads all the dialogue in Polish. Viewers hear the original English soundtrack faintly in the background. ¶ The approach is popular in Poland, where viewers still feel comfortable with a style deeply rooted in the country’s communist past. Lektors, traditionally men with husky voices, pride themselves on their utterly emotionless delivery, their craft honed through thousands of hours in recording studios. Fans appreciate the quality of voices, often tempered by years of cigarette smoking.

A better solution to the problem of voice-overs, unwittingly Brechtian.

 

Bucharest

This article in turn took me back to my time in Bucharest, where I lived for a year after college. There I devoted myself to writing pieces for my tape-recorded voice, a long effort that yielded only one decent result, a 20-minute composition called Thoughts on erasing blank tape, a few fragments of which later re-emerged in Trace.

It was oddly appropriate, then, that by some odd circumstance I was pressed into service as a voice-over actor for a mamaliga western («mamaliga» being the Romanian national dish of polenta). With me were three other Americans — the Air Force attaché and two Marine corporals serving in the embassy guard.

A mini-bus took us to the cramped movie studio, where the director, speaking through a translator, explained the gist of the film: Romanian cowboys emigrate to the Wild West.

My character was an American cowboy whose edgy rashness was to end his life almost immediately. I stood at the microphone watching the looped projection of my double, who reclined in a barber’s chair, his face covered with a huge lather of white shaving cream. In the facing mirror he spotted his Romanian adversary entering the barber-shop, upon which I was to deliver my two lines of American insult. These triggered my —or rather his — immediate death in the next shot (so to speak).

Little did the Romanian director or his eventual viewers realize that this cowboy’s accent was a far cry from John Wayne’s. No, it was with a diffident prep-school accent that my voice performed its odd imposture.

 

Subtitling

Fortunately, in America foreign films are subtitled rather than dubbed. The quiet of Bresson and Mizoguchi is almost heightened by the act of silent reading with which we translate the music of their unfamiliar tongues.

My many hours of watching subtitled movies must have led to the idea of Housebound, one of the two sections of Depth we’re making during our residency at Le Fresnoy this year. Here dialogue and even action are dispensed with, for everything is told through the lines of text.

The twist is that these lines are not flat superimpositions on the bottom of the image. Instead, they hover within the wavering space of each hand-held shot, as if both word and picture are being taken in by the same subjective gaze.