Embedding WebKit

The latest version of Field can embed a WebKit based web-browser directly into it's drawing canvases — both the 2d canvas and the fullscreen 3d canvas. You can have as many virtual browsers as you like; they are created and destroyed on demand; you can inject JavaScript directly into the pages that you load and, perhaps most excitingly of all, mouse interaction with them works (even in 3d).

The interface to this is a new set of properties on FLine's (Field's cleaned up drawing API). In Field to draw a line in you do something like this (in a spline drawer box):

line = FLine().moveTo(0,0).lineTo(500,0).lineTo(500,500)

line += Vector2(30,30) # -- move it over and down a bit

_self.lines.clear()
_self.lines.add(line)

Yields:

If we add these two properties to the line:

line.browser_url = "http://en.wikipedia.org"
line.browser_dimensions = Vector2(500,500)

We get this instead:

Go on: click on one of those links. Yes, they work just fine. Field has installed a .eventHandler property for you (see here)

The two magic properties on these lines make the magic go: * .browser_url — sets the url of the page to draw as you might expect. This is the indication to Field that you want to build a web browser for this line. * .browser_dimensions — the size, in pixels, of the browser to create (and thus the texture map to create it).

In the example above you can see that we draw a line three control nodes — an upside down 'L' shape. We go from top-left to top-right to bottom-right. If we modify that:

line = FLine().moveTo(0,0).lineTo(500,140).lineTo(500,500)

we get things like this:

The properties .color and .totalOpacity are honored as well, so things can quickly get a little out of hand:

_self.lines.clear()

index = 5
for n in "AEIOU":
    line = FLine().moveTo(0,0).lineTo(500,0).lineTo(500,1000)
    
    line += CFrame(r=0.1*index)
    line += Vector2(130*index,30*index)
    
    line.browser_url = "http://en.wikipedia.org/wiki/%s" %n
    line.browser_dimensions = Vector2(500,1000)
    line.color=Color4(Math.random(),1,Math.random(),0.5)
    _self.lines.add(line)
    
    index-=1

Yields:

Browsing the Web in 3d

Because they are extensions to the FLine drawing system, they work on the 3d GLSLang-shader-based "fullscreen" renderer.

Here's the code:

canvas = makeFullscreenCanvas()

shader = makeShaderFromElement(_self)

canvas << shader

lines = shader.getOnCanvasLines(canvas)

# this 0 means 'no' to use rectangular textures
lines.installWebbrowserSupport(0)

# Field uses texture id 1 for it's text and webbrowser system
shader.tex=1

shader << DisableDepthTest()
shader << SetBlendAdd()

If you've followed the documentation the only new line here is installWebbrowserSupport(useRectTextures). Confused about the difference between square and rectangular textures? see here. The call to shader.getOnCanvasLines(canvas) is the key one here — it gets us a FLine drawing system just line the one we have in the canvas but one that's shaded using shader instead.

Our shader for this demo needs to do some texture mapping — since the web page is a texture. The vertex shader:

varying vec4 vertexColor;
varying vec4 texture;

attribute vec4 s_Color;
attribute vec4 s_Texture;

void main()
{
    gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
    vertexColor = s_Color;
    texture = s_Texture;
}

And fragment shader:

varying vec4 vertexColor;
varying vec4 texture;
uniform sampler2D tex;

void main()
{
    vec4 c = texture2D(tex, texture.xy);
    gl_FragColor = c.xyzw;
    gl_FragColor.w = 0.5;
}

Don't forget to hit "refresh shader" once you've made the edits:

Now we can execute this code here:

lines.submit.clear()
index = 0
allLines = []
for n in "AEIOU":
    line = FLine(browser_url="http://en.wikipedia.org/wiki/%s"%n, browser_dimensions=Vector2(1000,1500))
    line.moveTo(0,0,0).lineTo(10,0,0).lineTo(10,15, 0).lineTo(0,15,0)
    line+=CFrame(r=Quaternion(Vector3(0,1,0), 0.2*index))
    line+=Vector3(0,0,-10*index)
    line(color=Vector4(1,1,1,1), thickness=1)
    lines.submit.add(line)
    allLines += [line]
    index += 1

    canvas.camera.position = Vector3(-35, 10, -65)
canvas.camera.up =Vector3(0,-1,0)

Draws (you might have to move the camera around with the keyboard):

And yes, clicking on any of those links (and scrolling with a scroll wheel) does in fact still work). Add a back button and you actually can use this as a web-browser.

Injecting JavaScript

One last trick with these embedded browsers (these are actually full-on Web-kit affairs with plugins and the like, not some half-baked Java-based HTML parser): running JavaScript.

Let's do this by example:

box1 = {}
canvas.lines().clear()
    
for line in allLines:
    b = line.browser
    b.executeJavaScript("elements=[]")
    b.executeJavaScript("""
        jQuery("a").each(function()
        {
            elements.push([jQuery(this).text(), jQuery(this).offset().left, jQuery(this).offset().top, jQuery(this).width(), jQuery(this).height()]);
        })
        """)
    rects =  b.executeJavaScript("return elements")
    offset =  b.executeJavaScript("return [window.pageXOffset,window.pageYOffset]")
        
    box2 = {}
    
    pts = line.browser_pageToSpace
    normal = Vector3().cross(pts(Vector2(1,0))-pts(Vector2(0,0)), pts(Vector2(0,1))-pts(Vector2(0,0))).normalize()
    for n in rects:
        m = FLine()
        rr = Rect(*n[1:])
        rr.x = rr.x+offset[0]
        rr.y = rr.y-offset[1]
        if (not(rr.y>0 and rr.y<1500)): continue
        m.moveTo(*pts(Vector2(rr.x, rr.y)))
        m.lineTo(*pts(Vector2(rr.x+rr.w, rr.y)))
        m.lineTo(*pts(Vector2(rr.x+rr.w, rr.y+rr.h)))
        m.lineTo(*pts(Vector2(rr.x, rr.y+rr.h)))
        m.lineTo(*pts(Vector2(rr.x, rr.y)))
    
        m+=Vector3(normal)*-0.0
        m(color=Vector4(1,1,1,0.05), filled=1)
        canvas.getOnCanvasPLine().submit.add(m)
        box2[n[0]]= m
    
    for n in box2: 
        if (n in box1):
            f = FLine()
            for i in range(len(box2[n].events)):
                a = box2[n].events[i].position()
                b = box1[n].events[i].position()
                f.moveTo(*a).lineTo(*b)(color=Color4(1,1,1,0.05))
                            
            canvas.lines().add(f)            
    
    box1 = box2
  • b=line.browser — Field set's the property browser on the line when the browser is created (in response to rasterizing something that has a browser_url set). This property contains a TextureBackedWebBrowser instance which has convenience methods on it for doing things; as usual, check the autocomplete.
  • executeJavaScript() — including executing javascript. The Javascript here builds an array of bounding rectangles for all of the 'a' elements (links, you know <A HREF= ... >). It exploits the fact that wikipedia already has jQuery installed on its pages.
  • line.browser_pageToSpace — when it sets line.browser in a 3d context Field also sets browser_pageToSpace which is a convience function that converts page space in pixels to world 3d space. This lets you draw a box around a piece of text in 3d.

The rest of the code is just math. The results, all links get boxes around them, and all links that link to the same page get linked across space: