OpticalFlowOperation.kt

src/jvmMain/kotlin/afield/media/vision/OpticalFlowOperation.kt · 77 lines

package afield.media.vision

import afield.graphics.vulkan.Renderer
import afield.graphics.vulkan.targets.RenderToTexture
import afield.graphics.texture.HasImageView

The shared contract for dense optical flow in Field: the OpticalFlowOperation interface, the OpticalFlowBackend choice, and the opticalFlow factory that picks a backend for a renderer.

Both backends, HardwareOpticalFlow and PyramidOpticalFlow, take two images of one renderer and write the same RGBA32F flow texture, so code that samples OpticalFlowOperation.output does not depend on which one ran.

enum class OpticalFlowBackend

Which implementation computes a OpticalFlowOperation.

enum class OpticalFlowBackend {

HARDWARE when the device has VK_NV_optical_flow enabled, PYRAMID everywhere else.

    AUTO,

HardwareOpticalFlow: the dedicated VK_NV_optical_flow engine (NVIDIA Ada and later).

    HARDWARE,

PyramidOpticalFlow: coarse-to-fine Lucas–Kanade in ordinary shader passes; any Vulkan device, MoltenVK included.

    PYRAMID;

fun resolve(renderer: Renderer): OpticalFlowBackend

The backend AUTO would pick on this renderer; other values are returned unchanged.

    fun resolve(renderer: Renderer): OpticalFlowBackend =
        if(this==AUTO) (if(renderer.device.supportsOpticalFlow) HARDWARE else PYRAMID) else this
}

interface OpticalFlowOperation : AutoCloseable

A dense motion field between two same-sized images of one renderer, kept on the GPU.

Every backend honours one output contract, so a consumer never needs to know which one ran: output is an RGBA32F RenderToTexture of flowWidth × flowHeight = ceil(width/grid) × ceil(height/grid). .xy is first → second displacement in INPUT pixels, X right and Y down; .z is a matching cost where lower is better (its scale is backend-specific, see backend); .w is 1.

Call evaluate between Field frames, after the producers of both inputs have submitted; it returns once output is ready to sample. The inputs are borrowed — keep them alive, and at their construction size, until close.

val renderer: Renderer

val first: HasImageView

val second: HasImageView

val grid: Int

val width: Int

val height: Int

val flowWidth: Int

val flowHeight: Int

val backend: OpticalFlowBackend

The backend that actually runs, never OpticalFlowBackend.AUTO.

val output: RenderToTexture

Sampled RGBA32F motion field and cost, owned by this operation.

val milliseconds: ArrayDeque<Double>

Wall time of recent evaluate calls, including submission and the final fence.

val gpuTimings: Map<String,List<Double>>

Named GPU timestamp samples (milliseconds) for this backend's stages; empty lists without timestamps.

val allocatedBytes: Long

Bytes of the images and buffers this operation allocated (driver-private memory excluded).

fun evaluate(reset: Boolean=false)

Compute output from the current contents of first and second. reset marks a cut: no temporal hints.

fun opticalFlow(renderer: Renderer,first: HasImageView,second: HasImageView,grid: Int=4,

Dense optical flow from first to second on whichever backend backend resolves to on this device.

val flow = opticalFlow(renderer, previousImage, currentImage, grid = 4)
prop<RenderToTexture>(motion) = flow.output
flow.evaluate()                 // between frames, after both images were rendered

fun opticalFlow(renderer: Renderer,first: HasImageView,second: HasImageView,grid: Int=4,
    backend: OpticalFlowBackend=OpticalFlowBackend.AUTO): OpticalFlowOperation = when(backend.resolve(renderer)) {
    OpticalFlowBackend.HARDWARE -> HardwareOpticalFlow(renderer,first,second,grid)
    else -> PyramidOpticalFlow(renderer,first,second,grid)
}