A pen that sees

The learned pens know only how they moved. They can carry a habit, but they cannot see the line they drew a moment ago, so they cannot follow it, stop when they meet it, or close a shape. A pen that senses the ink around its tip can do all three. The two on this page were made with Henri Michaux's mescaline drawings in mind: dense fields of fine, trembling lines, each one echoing the line beside it.

The quickest way in is the starting point. Open Starting points on the welcome screen and search for a pen that sees. Its sixth box is a drawing panel: draw a line and the pen hatches beside it. Browse the starting point ↗

What it sees

At every point, a sensing pen is told how close the nearest ink is in each of sixteen directions, out to 3 model units (24 pixels at the usual scale), and which way, and how far, the middle of all the ink so far lies. It ignores its own last eight points, which are always just behind it. That is all: eighteen numbers alongside the usual three. Here the seeing Michaux pen is stopped at four moments while it hatches beside a wavering line. seeing() draws what it sees: the faint circle is its reach, and each ray runs to the nearest ink in one direction, brighter the closer it is.

:::static
import afield.ai.learning.pen.*

val yours = fline {
    breakNext()
    for (i in 0..120) {
        val y = i * 2.0
        val x = 14 * sin(y / 40) + 5 * sin(y / 13)
        lineTo(x, y)
    }
}
val writer = PenLine(yours, model = PenModel.bundled("michaux"), seed = 1)
val moments =
    [60, 180, 320, 520].map { until ->
        writer.grow(until - writer.history.size)
        [
            yours.copy().apply {
                color = vec(1.0, 0.85, 0.3, 1.0)
                thickness = 1.5
            },
            writer.drawn.copy().apply {
                color = vec(0.9, 0.92, 1.0, 1.0)
                thickness = 1.0
            },
        ] + writer.seeing()
    }
lines(sideBySide(moments, gap = 40.0), w = 520.0, h = 300.0, fitToSize = true)

The rays reach sideways, to the line it is following, and that is what it steers by.

Beside a line

Given a line to continue, the seeing pen turns at its end and comes back beside it, pass after pass, each following the undulation of the one before. Its blind twin was trained on exactly the same drawings, with the same settings, and was never shown what is around it. It knows the habit, long trembling strokes, but has nothing to follow. The yellow line is the primer; the seeing pen is white, the blind one red.

:::static
import afield.ai.learning.pen.*

val yours = fline {
    breakNext()
    for (i in 0..120) {
        val y = i * 2.0
        val x = 14 * sin(y / 40) + 5 * sin(y / 13)
        lineTo(x, y)
    }
}
fun beside(model: String, seed: Long) = [
    yours.copy().apply {
        color = vec(1.0, 0.85, 0.3, 1.0)
        thickness = 1.5
    },
    yours.continueWith(PenModel.bundled(model), steps = 1500, seed = seed).apply {
        color = if (model.endsWith("blind")) vec(1.0, 0.6, 0.6, 1.0) else vec(0.9, 0.92, 1.0, 1.0)
        thickness = 1.0
    },
]
lines(
    sideBySide(
        [
            beside("michaux", 1),
            beside("michaux", 2),
            beside("michaux", 3),
            beside("michaux-blind", 1),
            beside("michaux-blind", 2),
        ],
        gap = 30.0,
    ),
    w = 520.0,
    h = 440.0,
    fitToSize = true,
)

Cracks that stop at a line

The cracks pen was trained on webs of cracks inside wobbling outlines, every crack ending where it meets a line. Given an outline of ours, the seeing pen cracks it from the inside and its cracks stop on lines; the blind pen leaves the outline, because it cannot tell where it is.

:::static
import afield.ai.learning.pen.*

val outline = fline {
    breakNext()
    for (i in 0..90) {
        val a = i * 2 * PI / 90
        val r = 60 * (1 + 0.15 * sin(3 * a) + 0.08 * cos(5 * a))
        lineTo(cos(a) * r, sin(a) * r)
    }
}
fun inside(model: String, seed: Long) = [
    outline.copy().apply {
        color = vec(1.0, 0.85, 0.3, 1.0)
        thickness = 1.5
    },
    outline.continueWith(PenModel.bundled(model), steps = 700, seed = seed, liftAtEnd = true).apply {
        color = if (model.endsWith("blind")) vec(1.0, 0.6, 0.6, 1.0) else vec(0.9, 0.92, 1.0, 1.0)
        thickness = 1.0
    },
]
lines(
    sideBySide(
        [
            inside("cracks", 1),
            inside("cracks", 2),
            inside("cracks", 3),
            inside("cracks-blind", 1),
            inside("cracks-blind", 2),
        ],
        gap = 30.0,
    ),
    w = 520.0,
    h = 300.0,
    fitToSize = true,
)

It is not perfect. The cracks crowd towards the middle of the shape, and now and then the pen leaves the shape to draw an outline of its own, which is what its corpus begins with. liftAtEnd = true hands it the outline as a finished stroke, which matters: in its corpus, a finished outline is followed by a new stroke, not more outline.

Seeing against blind, measured

Two numbers say what sight adds. Following is the share of a pen's points that have older ink within 8 pixels, following(line); joins is the share of its strokes that end on ink, joins(line). Each is averaged over twenty seeds, from the primers above.

:::static
import afield.ai.learning.pen.*

val yours = fline {
    breakNext()
    for (i in 0..120) {
        val y = i * 2.0
        val x = 14 * sin(y / 40) + 5 * sin(y / 13)
        lineTo(x, y)
    }
}
val outline = fline {
    breakNext()
    for (i in 0..90) {
        val a = i * 2 * PI / 90
        val r = 60 * (1 + 0.15 * sin(3 * a) + 0.08 * cos(5 * a))
        lineTo(cos(a) * r, sin(a) * r)
    }
}
fun mean(f: (Long) -> Double) = ((0 until 20).map { f(it.toLong()) }.average()).f2
val hatching = { m: String -> mean { s -> following(yours.continueWith(PenModel.bundled(m), steps = 1500, seed = s)) } }
val cracking = { m: String ->
    mean { s -> joins(outline.continueWith(PenModel.bundled(m), steps = 700, seed = s, liftAtEnd = true)) }
}
markdown(
    "measures",
    """
| | seeing | blind |
|---|---|---|
| Michaux beside a line: following | ${hatching("michaux")} | ${hatching("michaux-blind")} |
| cracks in an outline: joins | ${cracking("cracks")} | ${cracking("cracks-blind")} |
"""
        .trimIndent(),
)

The seeing pens also predicted their validation drawings better than the blind ones, but by much less than these numbers suggest: a pen being shown a drawing one point at a time has the drawing to lean on either way. The difference is in what a pen does on its own.

A field of lines

Left to run from one short stroke, the Michaux pen lays down columns of trembling lines, each pass following the last, a little changed, a field that accumulates.

:::static
import afield.ai.learning.pen.*

val first = fline {
    breakNext()
    for (i in 0..60) {
        val y = i * 2.5
        val x = 6 * sin(y / 25)
        lineTo(x, y)
    }
}
val writer = PenLine(first, model = PenModel.bundled("michaux"), seed = 7)
writer.grow(5000)
lines(
    writer.line.apply {
        color = vec(0.9, 0.92, 1.0, 1.0)
        thickness = 0.8
    },
    w = 520.0,
    h = 320.0,
    fitToSize = true,
)

Drawing with it

A sensing pen can see ink it did not draw. PenLine(..., surroundings = lines) shows it lines before it starts, as though they were already on the page, and sense(lines) adds more later. In the starting point's sixth box, your newest stroke is the primer and everything else you have drawn is surroundings, so the pen hatches beside your line and around the rest.

:::static
import afield.ai.learning.pen.*

val border = fline {
    moveTo(0, 0)
    lineTo(500, 0)
    lineTo(500, 300)
    lineTo(0, 300)
    lineTo(0, 0)
    color = vec(1, 1, 1, 0.2)
}
val starter = fline {
    moveTo(120, 30)
    cubicTo(150, 100, 100, 170, 130, 270)
    color = vec(1.0, 0.85, 0.3, 1.0)
    thickness = 2.0
}
val canvas = lines(border, starter, channel = "draw here", w = 520.0, h = 320.0, editable = true, drawing = true)!!
val michaux = PenModel.bundled("michaux")
var writer: PenLine? = null
var shown = listOf<FLine>()
var seen = 1
frames {
    while (frame()) {
        val drawn = canvas.currentLines()
        if (drawn.size > seen) {
            seen = drawn.size
            val yours = drawn.last().copy().apply {
                color = vec(1.0, 0.85, 0.3, 1.0)
                thickness = 2.0
            }
            shown = drawn.drop(1).dropLast(1).map { it.copy().apply { color = vec(1.0, 0.85, 0.3, 0.6) } } + yours
            writer = PenLine(yours, model = michaux, seed = seen.toLong(), surroundings = drawn.drop(1).dropLast(1))
        }
        val w = writer ?: continue
        // six points a frame; it runs past the end of your line before it turns back
        if (w.history.size < 2400) w.grow(6)
        lines(
            [border] +
                shown +
                w.drawn.apply {
                    color = vec(0.9, 0.92, 1.0, 1.0)
                    thickness = 1.0
                },
            channel = "pen",
            w = 520.0,
            h = 320.0,
        )
    }
}

The seventh box does the same with the cracks pen: draw a closed shape and it cracks it.

Hatching around a drawing

The Michaux pen hatches beside one long line, because that is what its corpus showed it. A word is many short strokes in every direction. A pen trained on hatching around such strokes (the echoes corpus: letter strokes, curves and blobs, each followed by passes laid beside it) learned the hand but not the job: on its own it wanders off after a pass or two. So for hatching, Field steers.

A guide chooses among the pen's proposals. At every point the pen proposes 32 next points, sampled from its own prediction, and a FollowGuide favours the one that stays a set gap from the ink beside it while avoiding crossings and jumps. The pen keeps its tremor, its rhythm and the way it turns; the guide only decides which of its ideas to draw. Hatcher hands the pen each stroke of a drawing in turn, with everything else as surroundings, so every stroke gets its passes and they keep clear of each other:

:::static
import afield.ai.learning.pen.*

val word = HersheyFont.load("futural").text("FIELD", size = 90.0).apply {
    color = vec(1.0, 0.85, 0.3, 1.0)
    thickness = 2.0
}
val hatching = Hatcher([word], PenModel.bundled("echoes"), passes = 10, gap = 3.5, seed = 1).finish()
hatching.forEach {
    it.color = vec(0.9, 0.92, 1.0, 1.0)
    it.thickness = 0.7
}
lines([word] + hatching, w = 520.0, h = 220.0, fitToSize = true)

Any pen can be steered this way; PenLine.guide takes a guide for any drawing, not only hatching. The echoes pen gives the closest contour lines. The Michaux pen, guided the same way, gives a wilder, broken hatching. The handwriting pen barely manages, because its habit is to set off sideways for the next letter.

The starting point's ninth box hatches whatever you draw, as you watch:

:::static
import afield.ai.learning.pen.*

val border = fline {
    moveTo(0, 0)
    lineTo(500, 0)
    lineTo(500, 300)
    lineTo(0, 300)
    lineTo(0, 0)
    color = vec(1, 1, 1, 0.2)
}
val starter =
    [
            fline {
                moveTo(90, 60)
                cubicTo(200, 20, 260, 220, 400, 160)
            },
            fline {
                moveTo(150, 250)
                cubicTo(180, 150, 300, 300, 330, 230)
            },
        ]
        .onEach {
            it.color = vec(1.0, 0.85, 0.3, 1.0)
            it.thickness = 2.0
        }
val canvas = lines([border] + starter, channel = "draw here", w = 520.0, h = 320.0, editable = true, drawing = true)!!
val hands = ["echoes", "michaux"]
var hand = 0
var choice = 0
checkbox("Michaux's hand", channel = "hand") { choice = if (it) 1 else 0 }
var hatcher: Hatcher? = null
var drawing = listOf<FLine>()
var seen = -1
frames {
    while (frame()) {
        val drawn = canvas.currentLines().drop(1)
        if (drawn.size != seen || choice != hand) {
            seen = drawn.size
            hand = choice
            drawing = drawn.map {
                it.copy().apply {
                    color = vec(1.0, 0.85, 0.3, 1.0)
                    thickness = 2.0
                }
            }
            hatcher = Hatcher(drawing, PenModel.bundled(hands[hand]), passes = 8, gap = 4.0, seed = seen.toLong())
        }
        val h = hatcher ?: continue
        if (!h.done) h.grow(24)
        lines(
            [border] +
                drawing +
                h.hatching.map {
                    it.apply {
                        color = vec(0.9, 0.92, 1.0, 1.0)
                        thickness = 0.8
                    }
                },
            channel = "pen",
            w = 520.0,
            h = 320.0,
        )
    }
}

Training a pen that sees

What a pen sees is computed the same way in Python, where it is trained, and in Kotlin, where it draws (fieldpen/sense.py and PenSense, held together by a test). For training, prepare adds it to every point of a corpus:

uv run --project . --group pen python -m fieldpen.prepare mine/train train.npz --scale 8 --sense 16
uv run --project . --group pen python -m fieldpen.prepare mine/valid valid.npz --scale 8 --sense 16
uv run --project . --group pen python -m fieldpen.train --train train.npz --valid valid.npz --out runs/mine \
    --hidden 256 --epochs 200 --lr 2e-3 --batch 64 --crop 256 --samples 2048 --input-noise 0.02 --device mps

train reads the sense settings from the corpus and makes a pen with eighteen extra inputs. --init with the handwriting pen widens it with those inputs set to zero, so fine-tuning starts from exactly the pen it was. Export and load as for any pen (Teaching a pen); Kotlin works out what the pen sees as it draws, from the points it has drawn and any surroundings.

A corpus for a seeing pen has to need sight. The Michaux corpus draws each column as one unbroken stroke, down and back up beside itself, so at every point the line beside it says where to go; PenDrawings.michaux and PenDrawings.cracks are the two generators.

Source files