Skip to content

Video

Video sources, shared GPU frames and live diffusion sessions, with controls for motion and temporal output.

14 source file(s), 192 public symbol(s), 192 of them carrying KDoc. Everything below is the documentation that lives beside the code — the same text the completion documentation panel shows in the editor. A blank summary means the declaration has no KDoc yet, not that this page dropped it.

SharedGpuSlot.kt

One slot of the GPU memory a diffusion session shares with its worker: a device-local buffer and a timeline semaphore, both exported to the worker as file descriptors.

A StreamDiffusionSession allocates a ring of these for the frames it sends and another for the results it reads back, passes their descriptors over a Unix socket, and afterwards exchanges only slot indices and semaphore values with the worker. Pixels are written and read by the GPU at both ends: a slot is a dedicated device-local allocation and is never mapped for the CPU. Its barrier moves the buffer between Field's graphics queue and the external queue CUDA works on, and the device UUID it reports names the GPU the worker is to import onto. Construction asks the driver whether it can export this buffer and this kind of semaphore, and fails rather than handing over a descriptor the worker cannot use.

src/jvmMain/kotlin/afield/media/video/diffusion/SharedGpuSlot.kt — read the source

SpinningDiffusionDemo.kt

A complete demo: SpinningDiffusionDemo draws five rotating curves into a source texture, runs them through a StreamDiffusionView, and shows source and stylized output side by side.

It also supplies a renderer-derived motion provider for its ribbons, so it exercises display reprojection and history blending. Set angle to turn the curves.

src/jvmMain/kotlin/afield/media/video/diffusion/SpinningDiffusionDemo.kt — read the source

SpinningDiffusionDemo

class SpinningDiffusionDemo(val renderer: Renderer, config: StreamDiffusionConfig, val whiteOnBlack: Boolean = false, val name: String = "spinning-diffusion") : AutoCloseable

Native Field geometry for the video study's five curves. Left: live source; right: causal output.

Declaration Summary
val renderer: Renderer The renderer that draws the curves and carries the comparison to the screen.
val whiteOnBlack: Boolean Draw light curves on a dark ground instead of dark ones on pale paper, which is what the white-on-black prompts in StreamDiffusionPresets expect to be given.
val name: String The name the demo and its view register under; two demos need two names.
val source The 512 by 512 target the curves are drawn into, and the only thing the worker is shown.
val view: StreamDiffusionView The view over source, with the demo's own motion provider. Change the prompt or the strength while it runs through its reset, and read its counters for what the worker is doing.
var angle: Float Where the curves are in their turn, in radians; it also tilts them out of the plane. Set it once a frame to animate, which is what gives the worker something to follow.
override fun close() Takes the comparison off the screen, stops the worker, and releases the demo's targets.

SpinningDiffusionMotion.kt

The motion provider behind the spinning-curves demo, which derives per-pixel correspondence from the geometry Field drew rather than from the two images.

SpinningDiffusionDemo's five curves are ribbons carrying a surface id per vertex, and the rotation that drew each frame is kept against that frame's id. To answer a request this projects every pixel's surface point back through the reference frame's rotation, which is the position that point had then; an id pass rendered at the reference angle says whether the surface visible there is the same one, so a ribbon that has turned behind another is rejected instead of carrying the wrong colour across. The correspondence ribbon is widened well past the drawn line, so the broader marks the model returns are covered too, and confidence falls off toward its edge.

The fallback pass covers what the mask rejected. Where the current frame shows bare paper it fills from paper near that pixel in the reference image, so the rejected band carries no bright impression of the reference frame's curves into empty paper, and it leaves every pixel the current source actually drew.

src/jvmMain/kotlin/afield/media/video/diffusion/SpinningDiffusionMotion.kt — read the source

StreamDiffusionConfig.kt

Settings for the optional streaming-diffusion backend: where it is installed, the prompt and strength, the seed, frame rate and the temporal options.

StreamDiffusionConfig validates its ranges on construction and is handed to a StreamDiffusionView. StreamDiffusionPresets holds named prompts (ink, charcoal, wire, and white-on-black variants); change prompt, strength, seed or noise advection on a running view with view.reset(...).

src/jvmMain/kotlin/afield/media/video/diffusion/StreamDiffusionConfig.kt — read the source

Used in: Streaming diffusion

StreamDiffusionConfig

data class StreamDiffusionConfig(val root: File, val python: File = File(root, ".venv/bin/python"), val worker: File = File("optional/streamdiffusion/worker.py"), val prompt: String = StreamDiffusionPresets.INK, val strength: Double = 0.60, val seed: Int = 0, val fps: Int = 16, val useTensorRT: Boolean = true, val offloadTextEncoder: Boolean = true, val backend: String = "diffusion", val startupTimeoutSeconds: Long = 240, val displayReprojection: Boolean = false, val historyBlend: Double = 0.0, val noiseAdvection: Double = 0.0)

Optional installed backend. No Python/CUDA dependency is loaded by ordinary Field startup.

Declaration Summary
val root: File The StreamDiffusionV2 installation, holding the configs, ckpts and wan_models it needs.
val python: File The interpreter the worker runs in; it needs the backend's own Torch and CUDA environment.
val worker: File Field's worker script, by default relative to the directory Field itself was started in.
val prompt: String What the model is asked to draw. StreamDiffusionPresets holds the ones the study used.
val strength: Double How far the result may leave the input, 0.2 to 0.9; it sets the worker's noise scale and its denoising steps, so low values keep the drawing and high ones redraw it.
val seed: Int The worker's sampling seed, applied when it starts and again on every reset.
val fps: Int Frames per second to capture and to present, 1 to 60, whatever rate the renderer runs at.
val useTensorRT: Boolean Asks for TensorRT. Whether one was built is StreamDiffusionSession.tensorRtActive.
val offloadTextEncoder: Boolean Moves the text encoder to the CPU once the prompt is encoded, freeing its GPU memory; later prompt changes are then encoded there.
val backend: String A GPU-only identity/invert mode useful when composing a new RTT graph.
val startupTimeoutSeconds: Long How long the session waits for the worker to load its weights and report ready.
val displayReprojection: Boolean The view's starting StreamDiffusionView.displayReprojection, which needs a motion provider.
val historyBlend: Double The view's starting StreamDiffusionView.historyBlend, which needs a motion provider.
val noiseAdvection: Double How much of the latent noise the worker carries along the motion field, 0 to 1; with no correspondence for a frame it draws fresh noise instead.

StreamDiffusionPresets

object StreamDiffusionPresets

Presets match the expressive study; edit the prompt/strength live through view.reset(...).

Declaration Summary
const val FINE_INK One fine ink stroke per input line, on pale paper: the closest of these to the source.
const val FINE_CHARCOAL The same restraint in charcoal: a narrow rough stroke per line, pale paper, little grain.
const val WIRE Barbed steel wire following each line, barbs along it, pale background.
const val INK The default: wet India ink, with pools, splatter and bleeding edges around the curves.
const val CHARCOAL Thick velvety charcoal: crumbly edges, loose dust and blended smudges around the strokes.
const val WHITE_WIRE The wire prompt for a dark source: white-painted wire and barbs on matte black.
const val WHITE_INK Opaque white ink on black paper, for white lines on a dark source.
const val WHITE_CHARCOAL White charcoal on black paper, the dark-source companion to the charcoal prompt.

StreamDiffusionMotion.kt

The interface a renderer implements to give streaming diffusion per-pixel motion: StreamDiffusionMotionProvider, with the frame and request types it receives.

A provider answers each StreamDiffusionMotionRequest with an RGBA32F texture mapping pixels of the current frame back to a reference frame, plus confidence and rejection channels. The view uses it for display reprojection, history blending and noise advection.

src/jvmMain/kotlin/afield/media/video/diffusion/StreamDiffusionMotion.kt — read the source

Used in: Streaming diffusion

StreamDiffusionFrame

data class StreamDiffusionFrame(val id: Long, val timeNanos: Long, val generation: Long)

IDs refer to source render frames, including frames not accepted by inference.

Declaration Summary
val id: Long Counts every frame the view rendered, whether or not it was sent to the worker.
val timeNanos: Long That frame's sourceTimeNanos reading, which the worker returns with its result.
val generation: Long The session generation the frame belongs to; it changes on every reset.

StreamDiffusionMotionUse

enum class StreamDiffusionMotionUse

What a correspondence is wanted for: putting the newest result on the frame being rendered, blending that result with the one before it, or carrying the worker's latent noise along the motion instead of drawing it afresh.

Constants: DISPLAY, HISTORY, NOISE.

StreamDiffusionMotionRequest

data class StreamDiffusionMotionRequest(val current: StreamDiffusionFrame, val reference: StreamDiffusionFrame, val use: StreamDiffusionMotionUse)

One question for the provider: where in an earlier frame were the pixels of a later one? A view asks it up to three times a frame, once for each use, and never for a frame it has not announced.

Declaration Summary
val current: StreamDiffusionFrame The frame to map from: the one being rendered now, or a result that has just arrived.
val reference: StreamDiffusionFrame The frame to map to, which is always one the provider has already seen.
val use: StreamDiffusionMotionUse Which of the three the answer is for; they differ in how far StreamDiffusionMotionRequest.reference lies behind.

StreamDiffusionMotionProvider

interface StreamDiffusionMotionProvider

Optional renderer-owned backward correspondence. RGBA32F, same size as the source: RG = absolute normalized UV in reference, B = confidence [0,1], A = reactive rejection [0,1]. UV uses the same orientation as sampling the source RTT; pixel centres are (xy+.5)/extent. Confidence zero means unknown/disoccluded. Reject occluded surfaces using depth/object IDs. Implementations may record a target here, reuse it between requests, or return null for unavailable history. No CPU pixels. The returned target must be sampled and transfer-source capable.

Declaration Summary
fun onFrame(frame: StreamDiffusionFrame) Called after every source draw, before requests; retain only bounded frame state.
fun record(command: CommandBuffer, request: StreamDiffusionMotionRequest): RenderToTexture? Records the work that fills a correspondence texture for one request, and returns it, or null when this pair of frames cannot be mapped. The target is checked against the source extent and the RGBA32F format before anything samples it, so a wrong one is refused rather than read.
fun recordFallback(command: CommandBuffer, request: StreamDiffusionMotionRequest, source: RenderToTexture, referenceImage: RenderToTexture): RenderToTexture? Optional display hole fill, called immediately after a successful DISPLAY record. Return a sampled single-layer/single-sample RTT matching source extent and colour format, or null to use the current source. referenceImage is the resolved stylized result at request.reference; it is borrowed only for this call. The provider owns its fallback target.

StreamDiffusionSession.kt

The worker process behind a StreamDiffusionView: StreamDiffusionSession starts the Python worker, shares its rings of GPU buffers and semaphores with it, and exchanges JSON control messages.

Pixels never pass through the protocol; the messages carry slot indices, semaphore values and frame ids. status, ready, capabilities and lastStats report what the worker last said.

src/jvmMain/kotlin/afield/media/video/diffusion/StreamDiffusionSession.kt — read the source

StreamDiffusionSession

class StreamDiffusionSession internal constructor(private val renderer: Renderer, val width: Int, val height: Int, val format: Int, val config: StreamDiffusionConfig, val motionEnabled: Boolean = false) : AutoCloseable

Owns one isolated worker and its bounded GPU rings. Pixels never enter this protocol.

Declaration Summary
val width: Int Width of every shared slot, fixed for the life of the session.
val height: Int Height of every shared slot, fixed for the life of the session.
val format: Int The Vulkan colour format of the frames in the rings, taken from the source target.
val config: StreamDiffusionConfig The settings this worker was started with; StreamDiffusionSession.reset can change prompt, strength and seed.
val motionEnabled: Boolean Whether each input slot carries a motion texture after its colour, which makes it larger.
val isClosed: Boolean True from the first call to StreamDiffusionSession.close, before the worker process has finished exiting.
val generation: Long Counts StreamDiffusionSession.reset calls from 1. Results tagged with an earlier one are discarded by the view.
var ready True once the worker has imported the shared slots and loaded its models.
var status A sentence to show a reader: the worker's own progress, then Ready, or why it stopped.
var capabilities What the worker says it is running: the backend, its decoder, and TensorRT. Every event it sends carries this, so it stays current for the life of the process.
var lastStats The last timing report: seconds for that batch of frames, and CUDA memory in use.
val tensorRtActive: Boolean? Whether a TensorRT engine actually executed, as the worker reports it; null until it says. Asking for TensorRT (StreamDiffusionConfig.useTensorRT) does not make it active, so when it was asked for and is not, StreamDiffusionSession.status says so.
val logFile The worker's stderr, in a private temporary directory. StreamDiffusionSession.status names it when startup fails.
fun reset(prompt: String = config.prompt, strength: Double = config.strength, seed: Int = config.seed, noiseAdvection: Double = config.noiseAdvection) Reset prompt/strength/seed at a causal boundary; old outputs are discarded by the view.
override fun close() Stops the worker and gives its shared memory back. Call it from any thread and as often as you like: a second caller waits for the first to finish. The process is asked to exit and killed if it has not after three seconds, and the slots are freed from the renderer's own cleanup, since CUDA's mappings have to go before the Vulkan allocations under them.

StreamDiffusionSupport.kt

The preflight check for local streaming diffusion: whether this machine can run it, and a sentence saying what is missing when it cannot.

probe reports the first absent piece, looking at the platform, the renderer, the selected Vulkan device's external memory and semaphore FD support, an NVIDIA device that can export buffers and timeline semaphores, the backend root and its Python environment, the Field worker script, the model files a diffusion backend needs, and the native handle-passing helper. It allocates nothing on the GPU, starts no Python and reads no weights, so box code can call it and show the reason instead of a view. StreamDiffusionView and StreamDiffusionSession each check it as they are constructed and fail with the same sentence; CUDA, the Python packages, the weights and TensorRT are checked later, when the worker starts.

configuredRoot is where the installation path comes from: FIELD_STREAMDIFFUSION_ROOT, or the one line in ~/.afield/streamdiffusion-root.txt, which is read on every call so a Field that is already running finds a backend installed after it started.

src/jvmMain/kotlin/afield/media/video/diffusion/StreamDiffusionSupport.kt — read the source

StreamDiffusionSupport

data class StreamDiffusionSupport(val available: Boolean, val reason: String)

Availability without creating GPU allocations, starting Python, or loading model weights.

StreamDiffusionTemporalOutput.kt

The temporal half of a diffusion view: it holds the last two results the worker returned and renders the texture the view publishes.

Results arrive late and in groups, so geometry has moved on by the time a frame can be shown. Given a StreamDiffusionMotionProvider, a StreamDiffusionView routes its output through this file to settle both ends of that delay; without one it publishes the worker's output directly. Each arriving result is written into one of two history targets while the other is read, blended with the previous result where the provider's correspondence agrees and both came from the same worker generation, and the carried colours are clamped to the 3x3 neighbourhood of the fresh ones so a stroke that moved does not trail. The display target reprojects the newest history onto the current frame and mixes it with the live source where confidence runs out; with no correspondence for the current frame it shows that history as it stands.

src/jvmMain/kotlin/afield/media/video/diffusion/StreamDiffusionTemporalOutput.kt — read the source

StreamDiffusionView.kt

A render target that sends a source RenderToTexture through the streaming-diffusion worker and returns the result as an ordinary sampled texture, StreamDiffusionView.output.

The frames stay on the GPU throughout: the source is copied into shared buffers, the CUDA worker writes its result into others, and the view copies that back into output. Construct and close a view between frames. With renderSource = true the view schedules the source itself; with false the caller renders the source earlier in the same command buffer. An optional StreamDiffusionMotionProvider enables display reprojection and history blending.

src/jvmMain/kotlin/afield/media/video/diffusion/StreamDiffusionView.kt — read the source

StreamDiffusionView

class StreamDiffusionView(val renderer: Renderer, val source: RenderToTexture, val config: StreamDiffusionConfig, val name: String = "stream-diffusion", val renderSource: Boolean = true, val sourceTimeNanos: () -> Long = System::nanoTime, val motion: StreamDiffusionMotionProvider? = null) : Target, AutoCloseable

GPU RTT -> optional CUDA worker -> ordinary sampled RTT. Construct/close between frames. With renderSource=true this target owns scheduling of source; do not also register source. With renderSource=false the caller renders source earlier in the same command buffer.

Declaration Summary
val renderer: Renderer The renderer whose graphics queue hands the frames over, and whose cleanup frees them.
val source: RenderToTexture What is stylized: a single-layer, single-sample, transfer-source RGBA8 or BGRA8 target, with both dimensions multiples of 32 for the diffusion backend.
val config: StreamDiffusionConfig Where the backend is and how it draws.
val name: String The view's key in the renderer's pre-target list; two views need two names.
val renderSource: Boolean Whether this view draws StreamDiffusionView.source itself. Pass false, and render it earlier in the same command buffer, when something else already schedules it.
val sourceTimeNanos: () -> Long The clock stamped on each captured frame and returned with its result, for a caller whose source does not advance in real time.
val motion: StreamDiffusionMotionProvider? Per-pixel correspondence from the renderer, which is what display reprojection, history blending and noise advection need; without one the worker's output is published as it is.
val output: RenderToTexture The texture to sample: an ordinary sampled target holding the newest stylized frame.
val session: StreamDiffusionSession The worker process behind this view, with its status, readiness and timings.
var displayReprojection Warp the newest result onto the current frame before showing it, so the picture follows the geometry while the worker is still behind. Needs StreamDiffusionView.motion.
var historyBlend How much of the previous result to carry into a new one along the correspondence, 0 to 1. Steadies the image between results; too much of it smears a fast-moving stroke. Needs StreamDiffusionView.motion.
val displayReprojectionRequests: Long Display correspondences asked of StreamDiffusionView.motion so far; zero without one.
val historyBlendRequests: Long History correspondences asked of StreamDiffusionView.motion so far; zero without one, or at zero blend.
var noiseMotionFrames Captured frames that also carried a motion texture for the worker's noise.
var capturedFrames Source frames sent to the worker, which is at most StreamDiffusionConfig.fps a second.
var skippedCaptures Captures given up because every input slot was still with the worker.
var presentedFrames Results copied into StreamDiffusionView.output.
var droppedOutputs Results thrown away: from a generation before the last reset, or behind a backlog.
var lastSourceFrameId Which source frame the newest presented result came from, or -1 before the first one.
var lastSourceTimeNanos The StreamDiffusionView.sourceTimeNanos reading of that frame, so a caller can measure the round trip.
override fun render(command: CommandBuffer, index: Int) One graphics frame of the exchange, called by the renderer rather than by box code: present a finished result if one is due, then capture the source again if the interval has passed and the worker has a free slot. Until the first result arrives, StreamDiffusionView.output carries a copy of the source, so there is a picture to draw from the moment the view is made.
fun reset(prompt: String = config.prompt, strength: Double = config.strength, seed: Int = config.seed, noiseAdvection: Double = config.noiseAdvection) Change artistic controls; preserves the last valid image during causal warm-up.
override fun close() Unregisters the view, stops the worker and releases its targets. Call it between graphics frames: there is a command buffer open during one, and closing then is refused.

particleflow.kt

Tracked particles from a video, turned into a velocity field that can be measured.

ParticleVideo follows thousands of points through a movie and leaves a cloud of samples per frame. This file splats those velocities onto a regular grid, one FlowFrame per frame, so every cell carries a velocity whether or not a particle landed in it. ParticleFlow.at hands one frame back as an Img, u in channel 0 and v in channel 1, which is the form the vector-field topology and streamline code reads.

A tracked video's flow often describes the camera more than its subject, and on old film the whole frame shifts and turns a little from one frame to the next. So each frame also carries the camera motion measured from its own particles: a median shift, and an affine fit that picks up a slight zoom or rotation as well. CameraMotion chooses how much of that to subtract when a frame is read back. A flow is far smaller than the tracks it came from, and smaller again gzipped and quantised, so ParticleFlow.save and loadParticleFlow let a page or a test begin from a saved flow instead of running the tracker.

src/jvmMain/kotlin/afield/media/video/particleflow.kt — read the source

Used in: Vector field topology

Top-level functions

Declaration Summary
fun loadParticleFlow(path: String): ParticleFlow Reads a flow written by ParticleFlow.save.
fun ParticleVideo.flow(cell: Int = 4, sigma: Double = 1.5): ParticleFlow Every frame's particle velocities spread onto a grid of cell-pixel cells, with a Gaussian of sigma cells, and the camera motion of each frame measured: see ParticleFlow.

CameraMotion

enum class CameraMotion

What to take out of a flow before using it: nothing (KEEP), the frame's median velocity (SHIFT), or an affine motion fitted to every particle and refitted without the outliers (AFFINE), which also removes a camera's slight zoom and rotation, or old film weaving in the gate.

Constants: KEEP, SHIFT, AFFINE.

FlowFrame

class FlowFrame(val u: FloatArray, val v: FloatArray, val w: FloatArray, val shift: DoubleArray, val affine: DoubleArray)

One frame of a ParticleFlow: the particle velocities splatted onto the grid (weighted sums u, v and total weight w per cell), and the camera motion measured from the particles, as a median shift (vx, vy) and an affine fit (vx = a0 + a1 x + a2 y, vy = b0 + b1 x + b2 y, in processed-image pixels).

ParticleFlow

class ParticleFlow(val width: Int, val height: Int, val cell: Int, val fps: Double, val frames: List<FlowFrame>)

The tracked particles of a ParticleVideo turned into a dense velocity field: each frame's particle velocities spread onto a grid of cell-pixel cells with a Gaussian of sigma cells.

Declaration Summary
fun at(frame: Int, window: Int = 1, camera: CameraMotion = CameraMotion.AFFINE, prior: Float = 0.5f): Img The flow at frame, averaged over window frames either side, with camera motion removed. Cells with no particles near them fade towards zero (the average is divided by the weight plus prior).
fun save(path: String) Writes the flow, gzipped and quantised: each cell's mean velocity to 1/64 pixel per frame and its weight to 1/256, far finer than the tracking itself, which keeps a few hundred frames to a few megabytes.

particletracker.kt

Runs the optional offline particle tracker (RAFT optical flow, via PyTorch) over a video and caches the result as a PVTRACK1 file.

particleTracker makes a ParticleTracker for the tools in optional/particle-tracker; its track call is a suspend function meant for a box's start { ... } coroutine, and it returns a ParticleTrackingRun whose load hands the file to particlevideo.kt. ParticleTrackingOptions holds every algorithm setting, and each one is part of the cache key. Invalid options fail before any process starts (ParticleVideoTest).

src/jvmMain/kotlin/afield/media/video/particletracker.kt — read the source

Used in: Video particles

Top-level functions

Declaration Summary
fun particleTracker(tools: Path = Path.of(System.getProperty("afield.particleTracker.tools") ?: System.getenv("FIELD_PARTICLE_TRACKER_TOOLS") ?: "optional/particle-tracker"), python: String = System.getProperty("afield.particleTracker.python") ?: System.getenv("FIELD_PARTICLE_TRACKER_PYTHON") ?: tools.resolve(".venv/bin/python").toAbsolutePath().toString(), cache: Path = Path.of(System.getProperty("afield.particleTracker.cache") ?: System.getenv("FIELD_PARTICLE_TRACKER_CACHE") ?: Path.of(System.getProperty("user.home"), ".afield/particle-tracker").toString())): ParticleTracker Creates an optional PyTorch/C++ launcher. Nothing is installed, compiled or downloaded until ParticleTracker.track is called. Prepare the environment in optional/particle-tracker first, or point python at an existing matched Torch/Torchvision environment.

ParticleTrackingOptions

data class ParticleTrackingOptions(val width: Int = 0, val fps: Double? = null, val frames: Int = 0, val particles: Int = 3000, val minTrack: Int = 10, val grace: Int = 5, val spacing: Double = 0.0, val confidence: Double = .25, val seedConfidence: Double = .45, val colorLimit: Double = .18, val stillSpeed: Double = 0.0, val mergeDistance: Double = 0.0, val mergeAge: Int = 30, val mergeVelocity: Double = 1.5, val seedBudget: Int = 16, val seed: Long = 1, val iterations: Int = 20, val weights: String = "C_T_SKHT_V2", val device: String = "cuda", val threads: Int = 4)

Options for bidirectional RAFT-large video particles. Width 0 retains source resolution; frame limit 0 processes the whole clip. All distances and velocities use processed pixels and frames.

Declaration Summary
val width: Int Resize the video to this width before tracking, keeping its aspect ratio. Zero keeps the source resolution, and every distance below is in the pixels that choice produces.
val fps: Double? Resample the video to this frame rate; null tracks it at its own average rate.
val frames: Int Stop after this many frames. Zero runs to the end of the clip.
val particles: Int How many particles to keep alive. A frame that cannot be filled is left underfilled.
val minTrack: Int Discard a trajectory that never reached this many samples.
val grace: Int How many frames in a row a condition has to hold before it retires the particle.
val spacing: Double The closest two particles may be born, in pixels. Zero spaces them for the target count and the image area, which is usually what you want.
val confidence: Double The score a match needs to move a particle. Below it the particle holds its position.
val seedConfidence: Double The score a new particle needs where it is proposed, stricter than ParticleTrackingOptions.confidence by default.
val colorLimit: Double How far the colour under a particle may drift from its birth colour before it is retired, measured as an RGB root mean square. Use 1.0 to switch this off for recoloured or relit footage; the colours recorded with each sample still come from the current frame.
val stillSpeed: Double Retire a particle moving slower than this, in pixels per frame. Zero keeps them all.
val mergeDistance: Double Coalesce two particles that stay this close, in pixels. Zero never merges.
val mergeAge: Int How many frames old both particles must be before they are allowed to merge.
val mergeVelocity: Double How much two particles' velocities may disagree and still count as one, in pixels a frame.
val seedBudget: Int How hard to work at filling a gap: this many batches of eight candidate positions for each missing particle, every batch keeping the best-spaced position it found.
val seed: Long Seeds the placement of new particles, so the same video and options give the same run.
val iterations: Int Refinement steps the flow network runs on each frame pair; more of them cost more time.
val weights: String Which pretrained RAFT-large checkpoint to load, by its torchvision name.
val device: String Where the flow network runs: cuda, cuda:1, cpu or mps. The particle bookkeeping is on the CPU either way, and a different backend is a different cache entry.
val threads: Int Threads for the per-particle work, which runs in its own process alongside the network.

ParticleTrackingRun

data class ParticleTrackingRun(val key: String, val cacheHit: Boolean, val tracksFile: Path, val manifest: Path, val log: Path)

A finalized cached tracking result. The binary can be copied and loaded without the launcher.

Declaration Summary
val key: String The content key this run is filed under: the video, every option, the tools and the environment they ran in. Two calls that agree on all of it are one run.
val cacheHit: Boolean Whether a finished run was already on disk, so no inference happened on this call.
val tracksFile: Path The PVTRACK1 file, which ParticleTrackingRun.load reads and which can be copied anywhere.
val manifest: Path The receipt beside it: what was asked for, what came out, and checksums of both.
val log: Path Everything the tracker printed while producing it.
fun load(): ParticleVideo Memory-map and validate the recorded tracks; this does not run inference.

ParticleTracker

class ParticleTracker internal constructor(val tools: Path, val python: String, val cache: Path)

Cached offline video processing, intended to be called inside a box's start { ... } coroutine. The suspend call owns the Python process: cancellation terminates the wrapper and its inference descendants. Successful runs publish a checksum-validated receipt atomically; failures never become cache hits. Locks share concurrent identical requests across boxes and worktrees.

Declaration Summary
val tools: Path The optional/particle-tracker directory holding the wrapper, the engine and its tests.
val python: String The interpreter to run it with, which needs a matched Torch and Torchvision.
val cache: Path Where finished runs are kept, by default under ~/.afield/particle-tracker.
suspend fun track(input: String, options: ParticleTrackingOptions = ParticleTrackingOptions(), offline: Boolean = false, force: Boolean = false): ParticleTrackingRun Tracks a local video, or returns the identical cached run. offline refuses a missing pretrained checkpoint without downloading it. force recomputes this key while preserving an existing successful result if inference fails. Logs and provenance are retained in the cache.
suspend fun track(input: Path, options: ParticleTrackingOptions = ParticleTrackingOptions(), offline: Boolean = false, force: Boolean = false): ParticleTrackingRun Path overload of track. No shell expansion or string-command interpolation is used.

particlevideo.kt

Reads particle-tracking results (PVTRACK1 files) as scrubbable frames and per-identity tracks, with no Python or Torch at load time.

loadParticleVideo memory-maps a finalized file (a gzip-compressed copy is inflated into memory instead) and validates every lifecycle before returning a ParticleVideo. ParticleVideo.frame decodes one recorded frame; ParticleVideo.at scrubs by time and interpolates only between samples of the same identity. Each ParticleSample carries position, velocity, sampled RGB and ParticleFlags; each ParticleTrack records one trajectory and why it was retired.

The files come from particletracker.kt and are drawn by particlevideoview.kt. Ordering, interpolation, rejection of broken files and the compressed round trip are covered by ParticleVideoTest.

src/jvmMain/kotlin/afield/media/video/particlevideo.kt — read the source

Used in: Video particles

Top-level functions

Declaration Summary
fun loadParticleVideo(path: String): ParticleVideo Loads a finalized PVTRACK1 file (or a gzip-compressed copy), without Python or Torch.
fun loadParticleVideo(file: File): ParticleVideo Loads and validates complete particle lifecycles, indexing samples by frame and identity.

ParticleFlags

object ParticleFlags

Sample and retirement flags. Several reasons can apply to the same sample or retirement, and the four conditions a sample can carry retire the particle once they have held for the run's grace frames, which is why they appear in ParticleTrack.endFlags as well.

Declaration Summary
const val BORN The first sample of a trajectory. Its velocity is zero, there being no earlier position.
const val UNCERTAIN The flow estimate for this frame scored too low to use, so the particle held its position with zero velocity rather than jumping to a bad match.
const val CROWDED An older particle is nearer than the birth spacing. Of a crowded pair only the younger one carries this, so the flags count encounters rather than particles.
const val COLOR The colour under the particle has drifted further from its birth colour than the run allows.
const val STILL The particle is moving slower than the still-speed threshold, which is off unless set.
const val LOST A retirement: the particle reached the edge of the image.
const val MERGED A retirement into a nearby identity that has lived at least as long, which mergedInto names; the younger of the two is always the one that goes.
const val EOF A retirement at the end of the clip, which says nothing about how the tracking went.

ParticleSample

data class ParticleSample(val id: Long, val frame: Int, val flags: Int, val x: Float, val y: Float, val vx: Float, val vy: Float, val red: Float, val green: Float, val blue: Float, val confidence: Float, val forwardBackwardError: Float, val residual: Float, val texture: Float, val colorDelta: Float)

One identity at one processed video frame. Position and velocity use pixels, with y down.

Declaration Summary
val id: Long The identity this sample belongs to, stable for as long as the particle lives.
val frame: Int The processed frame it was taken on, counted from zero.
val flags: Int The ParticleFlags bits set for this sample.
val x: Float Position across the processed image, in pixels.
val y: Float Position down the processed image, in pixels, measured from the top.
val vx: Float Horizontal displacement since the previous frame, in pixels.
val vy: Float Vertical displacement since the previous frame, in pixels.
val red: Float Red under the particle on this frame, 0 to 1, as the video stores it rather than linearised.
val green: Float Green under the particle on this frame, 0 to 1.
val blue: Float Blue under the particle on this frame, 0 to 1.
val confidence: Float How much the estimator trusts this match, 0 to 1: a heuristic built from ParticleSample.forwardBackwardError, ParticleSample.residual and ParticleSample.texture rather than a calibrated probability. Below the run's threshold the sample is marked uncertain.
val forwardBackwardError: Float How far tracking forward and then back again misses the starting point, in pixels.
val residual: Float Brightness difference between the two matched windows, as a root mean square over 0 to 1.
val texture: Float How much structure the window had to match on, as the weaker gradient direction's strength per pixel; flat regions score near zero, and the neural tracker records -1 for all of them.
val colorDelta: Float Distance from the particle's birth colour, by the same measure the colour limit uses.
val position: Vec3 Fresh Field vector; mutating it does not modify the mapped track data.
val velocity: Vec3 Displacement per processed frame, not pixels per second. Birth and uncertain samples have zero velocity.
val uncertain: Boolean Whether this sample held its position instead of following a flow estimate.

ParticleTrack

data class ParticleTrack(val id: Long, val firstFrame: Int, val lastFrame: Int, val samples: Int, val endFrame: Int, val endFlags: Int, val mergedInto: Long?)

A committed trajectory, including its retirement reason and optional surviving merge identity.

Declaration Summary
val id: Long The identity, which no later particle reuses.
val firstFrame: Int The frame the particle was born on.
val lastFrame: Int The frame of its last recorded sample.
val samples: Int How many samples the file holds for it, never fewer than the run's minimum track length.
val endFrame: Int The frame its retirement was recorded on, which is ParticleTrack.lastFrame or a later frame.
val endFlags: Int Why it was retired, as ParticleFlags bits.
val mergedInto: Long? The identity it was merged into, or null if it ended some other way.

ParticleColor

enum class ParticleColor

Ways to color the same geometry: source RGB, deterministic identity, or heuristic confidence.

Constants: SAMPLED, IDENTITY, CONFIDENCE.

ParticleFrame

data class ParticleFrame(val index: Int, val seconds: Double, val samples: List<ParticleSample>)

A frame of sampled particles. Retired and newly born identities can both appear at a transition.

Declaration Summary
val index: Int The recorded frame this came from, and the earlier of the two when samples are interpolated.
val seconds: Double Movie time in seconds: the scrubbed time for an interpolated frame, ParticleFrame.index over the frame rate for a recorded one.
val samples: List<ParticleSample> The particles present, in the order the file records them.
fun points(scale: Double = 1.0, pointSize: Double = 2.0, color: ParticleColor = ParticleColor.SAMPLED): FLine Point geometry for node-color-aware drawing, including Vulkan. Use ParticleVideo.image for 2D output.

ParticleVideo

class ParticleVideo private constructor(val file: File, val width: Int, val height: Int, val fps: Double, val frameCount: Int, val seed: Long, val recordCount: Long, private val data: ByteBuffer, private val byFrame: Array<IntArray>, private val byTrack: Map<Long, IntArray>, val tracks: List<ParticleTrack>)

Read-only, memory-mapped video tracks with frame and identity indexes. No neural dependencies.

Declaration Summary
val file: File The mapped file, as an absolute path.
val width: Int Width of the processed image, which is the frame these coordinates are in.
val height: Int Height of the processed image.
val fps: Double Processed frames per second, which is what turns a frame index into movie time.
val frameCount: Int How many frames were processed, whatever the source clip's own length.
val seed: Long The seed the run was given, recorded so the same particles can be laid down again.
val recordCount: Long Samples and retirements in the file together, which is not a count of particles.
val tracks: List<ParticleTrack> Every committed trajectory, in the order the particles were retired.
val durationSeconds: Double Playback duration including the last frame's display interval.
val lastSampleSeconds: Double Time of the final recorded sample, for inclusive scrub controls.
fun frame(index: Int): ParticleFrame Decodes just one frame. Empty frames remain valid.
fun at(seconds: Double, interpolate: Boolean = true): ParticleFrame Scrubs recorded movie time, clamped to its sample range. Interpolation joins only matching identities in adjacent frames; births and retirements never crossfade into another identity. RGB, position and velocity interpolate; status/confidence diagnostics belong to the earlier frame.
fun trajectory(id: Long, fromFrame: Int = 0, throughFrame: Int = frameCount - 1): List<ParticleSample> A single recorded trajectory in time order; an unknown identity returns an empty list.
fun trails(frame: Int, length: Int = 12, stride: Int = 1, scale: Double = 1.0, color: ParticleColor = ParticleColor.IDENTITY): List<FLine> Trails ending at frame, one path per currently visible identity, in processed coordinates. stride selects stable identities by ID modulo, useful for drawing dense runs without flicker.

particlevideoview.kt

Draws a loaded ParticleVideo as images and as a scrubbable box output.

ParticleVideo.image rasterizes one recorded frame as per-identity trails and optional points, colored by sampled RGB, identity or confidence. particleVideoView puts that drawing in a box's output with controls for movie time, colors and trails. Both work from the loaded tracks and never run inference.

src/jvmMain/kotlin/afield/media/video/particlevideoview.kt — read the source

Used in: Video particles

Top-level functions

Declaration Summary
fun ParticleVideo.image(frame: Int, scale: Int = 2, pointRadius: Double = .45, color: ParticleColor = ParticleColor.SAMPLED, trailLength: Int = 24, drawPoints: Boolean = false, stride: Int = 1, lineWidth: Double = .6): BufferedImage Rasterizes an actual recorded frame, with sampled RGB or stable identity/confidence colors. pointRadius uses source pixel coordinates; scale magnifies the complete image and dots. trailLength draws recent consecutive samples as a polyline per identity, without bridging lifecycles. stride selects stable IDs, so subsampling cannot flicker between unrelated trajectories. Points are optional; line-only drawing is the default. lineWidth uses source pixels.
fun Box.particleVideoView(video: ParticleVideo, channel: String = "particle video", initialSeconds: Double = 1.0, scale: Int = 2, pointRadius: Double = .45, trailLength: Int = 24, stride: Int = 1, identityColors: Boolean = true) Scrubbable Field output: movie time, identity colors and trails are ordinary controls. Drawing runs in Kotlin from the loaded tracks; using this view never launches Python inference.
suspend fun ParticleVideo.renderMovie(path: Path, scale: Int = 2, pointRadius: Double = .45, color: ParticleColor = ParticleColor.SAMPLED, trailLength: Int = 24, drawPoints: Boolean = false, stride: Int = 1, lineWidth: Double = .6): Path Encodes Kotlin-rasterized tracks to MP4 with FFmpeg, without rerunning inference. The output is staged beside path and replaced only after successful encoding. Intended for offline exports.

videoFrames.kt

Reading a video file frame by frame as Field images, through ffmpeg.

videoFrames is the lazy sequence of frames for a path on disk, and videoSize answers the extent they will arrive at. ffmpeg does the decoding, the frame-rate conversion and the scaling; FFMPEG is the binary, and the ffprobe beside it reports a clip's native size. Frames arrive as Img, with channel values in 0..1, so Field's imaging and sampling work on them without a conversion step.

src/jvmMain/kotlin/afield/media/video/videoFrames.kt — read the source

Top-level functions

Declaration Summary
fun videoFrames(path: String, width: Int = 0, fps: Double? = null, start: Double = 0.0, duration: Double? = null, gray: Boolean = false): Sequence<Img> The frames of a video file as Imgs, decoded by ffmpeg (FFMPEG says which).
fun videoSize(path: String, width: Int = 0): Pair<Int, Int> The (width, height) frames come out at for width (0 = native), via ffprobe beside FFMPEG.