Embedding GUI Elements inside code

First off, you should read the introduction to the TextEditor. Everything described here takes place inside it. Secondly, the GUI Embedding features are also covered in this downloadable tutorial.

The central problem is one of speed. Field demands code when other environments let you drag a slider, push a button and so on. While writing code is undeniably powerful, it's sometimes not very fast. Typing out your numbers long form is from a bygone age. But the philosophy of Field is that code is everywhere, and dumbed down, pre-canned interfaces aren't. How can these two competing views of interaction be combined?

Field's text editor offers an alternative way of making editing code as a fast as interacting with a graphical user interface — by simply quoting graphical user interface elements directly:

Here we set a variable "equal" to a slider. Slide the slider, and re-execute that line, and the variable has changed (and see below). You get a menu of these with a right click. These elements flow within the body of the text itself and the text insertion point simply skips over them; they can be copied and pasted like text (#20), deleted like text etc.. The rule is simple: the graphical object can be as graphical and as interactive as it needs to be, but in return it must to be able to convert itself to text on demand — Python is still a text-based language after all, and, as stated previously, we choose to operate under the constraint that this language remain completely standard. So, at some level this text editor (and anything else in the delegation graph) still believes that it's editing text. To help maintain this charade, the graphical object gets to convert itself to both text and a small persistent "cookie" (a property, of course) which helps with the conversion back to UI element if needs be. Importantly the UI element needs to be dedicated to "being text" convincingly since should this element not be present in the editor, the cost of instantiating UI elements is never paid.

The GUI Element Zoo

Once we've broken free of our text-only shackles we have another two axes of extension in Field. Firstly, the lexicon of graphical elements that we can embed in the code, and secondly, the kinds of "objects" that these UI elements masquerade as when they are text.

This first expansionary front is simple to understand — the vocabulary of UI elements is well known, and many if not most of them are useful when embedded inside code:

(A "Point Picker", which selects a Vector2 or a 2-tuple)

(A "Graph widget", which makes some function over 0->1)

Obviously we can keep going through the standard set of graphical user elements: we we can have buttons that do things (because they are given callbacks in the code itself); popup color-wells that select a color; combo-boxes that set and edit strings; and so on. Right-click in the text editor for the complete list.

Having established a basic vocabulary of UI elements with in the appropriate internal interfaces, they can migrate to other areas of Field, decorating other, traditionally text, components: the output window of the text editor can contain all of these things (again a handful of patterns emerge: it's sometimes useful to print, or indeed yield(), some buttons that offer up choices of what to do next in what becomes a very simple way of making step-by-step "wizards"). And, of course, these elements can be embedded not in the otherwise textual code inside visual elements but directly on the canvases themselves, effectively exposing a choice set of parameters, a visual front to the code inside. See EmbeddingGuiOnCanvas for a brief tutorial.

But perhaps more powerful, and less obvious, than adding to the standard library of graphical widgets that claim to be text, is thinking about what these graphical widgets can evaluate to. Here the malleability of Python's dynamism comes to the fore.

The simplest thing is to have the slider evaluate to something like "0.75" (that is literally the string "0.75", excluding the quotes, which is textual way of indicating the number 0.75. It's important to remember that UI objects have to evaluate to a text fragment, not an object — they can evaluate to "Vector2(1,2)" but not some specific Vector2 instance.).

But once we execute this assigning a value to a variable it loses its dynamism — any trace of those sliders etc. are erased, their values "baked". When we set and execute x=((slider))*5 and the slider is at 0.5, x is forever 2.5 even if we change the slider later. This is the unfortunate result of having graphical elements "evaluate to text". We can avoid this by making sliders evaluate to a (string that creates a) dynamic proxy that transforms numerical operations into operations that are evaluated lazily. In this way the dynamic UI penetrates deep into the structures set up by the code, rather than just its surface, and by changing the slider things "far away" can automatically update. See LazyFunctionalHelpers for a close up of this magic.

Block Markers

In addition to this growing set of GUI elements directly embedded inside code, there is one more class of graphical object in Field, related to the task of rapidly exploring code — the block marker:

A block marker is just a normal text field, but embedded, in turn, into the text editor. The convention here is that the label (number).name' starts a block and(number).name' ends it. In our original formulation of UI element evaluates to text, these simply evaluate to non-executed comments, but in Field's text editor, pushing command-(number) when the text cursor inside the block all of that block is automatically updated. Now one can slide around the control knob of a slider all the while pressing `command-1′ to execute the block it's in. Very fluid experimentation of code is now possible.

But again, we've added a small amount of additional structure to Field and seek to mine a diverse range of possibilities from it. Now that there is a convention that marks a block of code in the editor, we need to open this possibly back out to code itself. A new set of possibilities suggest themselves: firstly we can use the presence of the block markers to indicate that we'd like to transform the block of text itself.

Three illustrations of these blocks follow, both exploiting this structure and doing so with almost no pre-arranged support from a hidden Field "standard library". Rather, these transformations are created in-situ, and spontaneously.

The first example is from Forest, a 5 screen, 5 GPU / CPU artwork. In Forest we introduce a block transformation that can cause the text "transformed" to be executed on a different Field process, running on a different computer. This uses networking code that's entirely standard to the Python library. The second example is integration between Field and Maya, which, in the case of its proprietary scripting language "MEL" allows commands to be sent across a telnet port. Using a text transformation we can embed this foreign language directly into Field's text editor:

Note in this example the text transformation automatically uses Python's string formatting constructs to inject local variables into the foreign language, allowing in the above example for MEL code to be looped over by a Python for-loop. See MayaIntegration for more ideas. We'll simply note here that this sub-language executes not in Field but in the Maya runtime. Similar bridges to Apple's "Open Scripting Architecture" allow the embedding of AppleScript, which in turn has enabled Field's use as an asset manager and workflow organizer — calling upon other software applications to process and update files. Thirdly, and finally, we note a recent and exciting addition to Field — the Scala language — can be currently embedded inside Field using block transforms.

Prompts

Finally, a recent, work in progress, addition that opens up a new class of GUI elements for Field — the Stepper:

Most simply the prompt is a button that masquerades as a python generator / java iProvider that "blocks" until you push it (it doesn't really block in the threading sense — everything else gets to go on executing — it blocks in the coroutine sense). If the above code is executed with command-shift-return or inside a generator-transformation block, or inside a plain old generator, the code doesn't advance until you click the button — one advance per click, unless you hold down option-click (just like executing a visual element from The Canvas or a text area from the execution ruler). A few design elements: the prompt turns red when it's actually holding up execution, green when it's been option-clicked. In the future, we should be able to call the whole execution off (by throwing an exception inside PythonGeneratorStack). Finally, we note that, with no UI to interact with the prompt defaults to advancing.

To read more about where this idea comes from and ends up, see GeneratorExamples.