SkinnedMeshFromDisk.kt

src/jvmMain/kotlin/afield/graphics/scene/SkinnedMeshFromDisk.kt · 294 lines

@file:Suppress("unused")

package afield.graphics.scene

import afield.graphics.render.before

import afield.geometry.mesh.TriMesh
import afield.geometry.mesh.TriMeshBuilder
import afield.math.linalg.Vec2
import afield.math.linalg.Vec3
import afield.math.linalg.Matrix4f
import org.lwjgl.system.MemoryUtil.memFree

Posing a rigged glTF on the CPU: into a fresh TriMesh, or back into the vertex buffer a loaded model already owns.

Which of the two you want depends on what happens next. skinnedGltfToTriMesh loads the file, deforms it at one time and hands back geometry, which suits measuring a pose, feeding the software rasteriser, or treating a frame as data. CpuSkinner wraps a mesh that is already loaded for drawing and overwrites its positions and normals in place, so the stock glTF shaders draw a moving character without knowing that anything is being skinned.

posedGltfToTriMesh is skinnedGltfToTriMesh with the joint palette supplied by the caller, for a clip blend or a pose an IK solver produced, and gltfClipNames reads the clip names out of a file without deforming anything.

The blend itself is GltfSkinning's; what this file adds is the plumbing on either side of it. GpuSkinner moves the same work into the vertex shader, which is the path to take when the character has to animate every frame.

fun skinnedGltfToTriMesh

A rigged file, deformed at a time, as a TriMesh you can render, measure and look at.

assimpToTriMesh is the rigid sibling of this: it applies each node's own world matrix and hands back the bind pose. This one ignores the mesh node's transform — glTF 2.0 §3.7.3.2 says a skinned mesh node's transform is not applied — and instead runs GltfSkinning.skin with the joint palette at seconds, so what comes back is the deformed surface in scene space.

A mesh in the file with no skin is not dropped: it goes through the rigid path, so a scene that is part character and part prop arrives whole.

val bent = skinnedGltfToTriMesh(RiggedFixture.locate(), clip = "bend", seconds = 1.0f)
showMesh(bent)

@param clip which animation to evaluate, by name. null is the rest pose — which, for a file whose first keys are the bind transforms, is the bind pose and should be identical to assimpToTriMesh's output vertex for vertex. @param seconds the time within the clip. Clamped at both ends; this does not loop, because a caller that wants looping wants to choose the phase itself.

fun skinnedGltfToTriMesh(
    path: String,
    clip: String? = null,
    seconds: Float = 0f,
    keepColors: Boolean = true,
    options: GltfSkinning.SkinningOptions = GltfSkinning.SkinningOptions(),
): TriMesh = posedGltfToTriMesh(path, keepColors, options) { skeleton ->
    skeleton.palette(clip?.let { skeleton.clip(it) }, seconds)
}

fun posedGltfToTriMesh

The same, with the palette chosen by the caller — for a clip blend, a pose driven by an IK solver, or anything else that is not "clip x at time t".

A different name rather than an overload of skinnedGltfToTriMesh, deliberately. Box code is compiled by the K2 REPL against a different import set from jvmMain, and an overload pair that differs only in a trailing lambda is exactly the shape that resolves in the library and goes ambiguous in a box — where no Gradle gate can see it.

@param palette called once per skinned mesh with that mesh's skeleton; must return one matrix per entry of GltfSkinning.Skeleton.bones, in that order. GltfSkinning.Skeleton.palette and GltfSkinning.Skeleton.blendedPalette are the two ready-made ones.

fun posedGltfToTriMesh(
    path: String,
    keepColors: Boolean = true,
    options: GltfSkinning.SkinningOptions = GltfSkinning.SkinningOptions(),
    palette: (GltfSkinning.Skeleton) -> List<Matrix4f>,
): TriMesh {
    val model = GltfSupport.loadGltfBasic(path, GltfSupport.GltfLoadConfig(loadTextures = false))
    try {
        var out: TriMesh? = null
        for (instance in model.meshes) {
            val b = instance.buffers
            if (b.indexCount < 3) continue
            val floats = b.vertexData.asFloatBuffer()

            // The packed layout the loader writes: [pos.xyz][normal.xyz][uv.xy][color.rgba].
            val positions = FloatArray(b.vertexCount * 3)
            val normals = FloatArray(b.vertexCount * 3)
            var whiteEverywhere = true
            var anyUv = false
            val rows = ArrayList<FloatArray>(b.vertexCount)
            for (v in 0 until b.vertexCount) {
                val o = v * 12
                val row = FloatArray(12) { floats.get(o + it) }
                positions[v * 3] = row[0]; positions[v * 3 + 1] = row[1]; positions[v * 3 + 2] = row[2]
                normals[v * 3] = row[3]; normals[v * 3 + 1] = row[4]; normals[v * 3 + 2] = row[5]
                if (row[8] != 1f || row[9] != 1f || row[10] != 1f) whiteEverywhere = false
                if (row[6] != 0f || row[7] != 0f) anyUv = true
                rows.add(row)
            }

            val skin = instance.skin
            val deformedP: FloatArray
            val deformedN: FloatArray
            val mirrors: Boolean
            if (skin != null) {
                val skinned = GltfSkinning.skin(positions, normals, skin.influences, palette(skin.skeleton), options)
                deformedP = skinned.position
                deformedN = skinned.normal ?: normals
                // A skinned mesh's winding is decided per triangle by the blended matrix, not by one
                // scene determinant, and a joint palette of rigid transforms never flips it. Left
                // alone rather than guessed at; a mirroring *joint* would need per-triangle work.
                mirrors = false
            } else {
                // Rigid fallback, matching assimpToTriMesh exactly: the node's true world matrix, and
                // the inverse transpose for the normal.
                val m = instance.bindMatrix ?: instance.bindPose.toMatrix()
                val nm = m.get3x3(afield.math.linalg.Matrix3f()).invert().transpose()
                deformedP = FloatArray(b.vertexCount * 3)
                deformedN = FloatArray(b.vertexCount * 3)
                val p = afield.math.linalg.Vector3f()
                val n = afield.math.linalg.Vector3f()
                for (v in 0 until b.vertexCount) {
                    p.set(positions[v * 3], positions[v * 3 + 1], positions[v * 3 + 2]); m.transformPosition(p)
                    n.set(normals[v * 3], normals[v * 3 + 1], normals[v * 3 + 2]); nm.transform(n)
                    val len = kotlin.math.sqrt(n.x * n.x + n.y * n.y + n.z * n.z)
                    deformedP[v * 3] = p.x; deformedP[v * 3 + 1] = p.y; deformedP[v * 3 + 2] = p.z
                    if (len > 1e-12f) {
                        deformedN[v * 3] = n.x / len; deformedN[v * 3 + 1] = n.y / len; deformedN[v * 3 + 2] = n.z / len
                    } else {
                        deformedN[v * 3 + 1] = 1f
                    }
                }
                mirrors = m.determinant() < 0f
            }

            val emitColors = keepColors && !whiteEverywhere
            val builder = TriMeshBuilder()
            for (v in 0 until b.vertexCount) {
                val row = rows[v]
                builder.vertex(
                    deformedP[v * 3], deformedP[v * 3 + 1], deformedP[v * 3 + 2],
                    normal = Vec3(deformedN[v * 3], deformedN[v * 3 + 1], deformedN[v * 3 + 2]),
                    uv = if (anyUv) Vec2(row[6], row[7]) else null,
                    color = if (emitColors) Vec3(row[8], row[9], row[10]) else null,
                )
            }
            val indices = b.indexData
            var t = 0
            while (t + 2 < b.indexCount) {
                val a = indices.get(t); val bb = indices.get(t + 1); val c = indices.get(t + 2)
                if (a < b.vertexCount && bb < b.vertexCount && c < b.vertexCount) {
                    if (mirrors) builder.triangle(a, c, bb) else builder.triangle(a, bb, c)
                }
                t += 3
            }
            val part = builder.build()
            out = out?.plus(part) ?: part
        }
        return out ?: TriMesh.empty
    } finally {
        for (instance in model.meshes) {
            memFree(instance.buffers.vertexData)
            memFree(instance.buffers.indexData)
            instance.buffers.tangentData?.let { memFree(it) }
        }
    }
}

fun gltfClipNames(path: String): List<String>

The clip names in a rigged file, in file order. Empty for a file with no animations.

fun gltfClipNames(path: String): List<String> {
    val model = GltfSupport.loadGltfBasic(path, GltfSupport.GltfLoadConfig(loadTextures = false))
    try {
        return model.meshes.firstNotNullOfOrNull { it.skin?.skeleton?.clips?.map { c -> c.name } } ?: emptyList()
    } finally {
        for (instance in model.meshes) {
            memFree(instance.buffers.vertexData)
            memFree(instance.buffers.indexData)
            instance.buffers.tangentData?.let { memFree(it) }
        }
    }
}

class CpuSkinner

CPU skinning straight into the GPU vertex buffer a loaded MeshInstance already owns — so the bundled glTF shaders draw a deforming character with no shader change at all.

GltfSupport.MeshInstance.buffers.vertexData is plain host memory in the fixed 12-float layout `pos.xyzuv.xy, andapplyToMesh` uploads it. Overwriting the position and normal fields in place before each upload is therefore the whole of GPU-rendered skeletal animation, at GltfSkinning's measured 17.7 M vertices/s. A real GPU skinning path — a joint palette buffer and a blend in the vertex shader — saves the upload and is the obvious next step, but it cannot produce a picture this does not.

The bug this class exists to make impossible

Skinning in place is not idempotent if you read what you last wrote: the second frame would skin an already-skinned mesh and the character would fly apart over a few seconds — slowly enough to look like a physics problem rather than a bookkeeping one. So the bind-pose positions and normals are snapshotted at construction and every pose call reads that snapshot, never the buffer. CpuSkinnerTest.posingTwiceWithTheSamePaletteGivesTheSameAnswer is the assertion.

val (module, models) = gltfModelHere("character.glb") { }
val skinner = CpuSkinner(models[0].meshes[0])
loop {
    skinner.pose("walk", time)
    models[0].meshes[0].applyToMesh(mesh, camera)   // but see the warning below
}

One thing in the way of that loop, read off the code and not measured on a device

MeshInstance.applyToMesh calls defaultBuffer { … }.buffer for its vertex and index buffers. defaultBuffer builds a fresh BufferParameters on every call, and BufferParameters.result — the cache that makes toBuffer() return the same allocation twice — lives on that object. So a per-frame applyToMesh allocates a new FrameBufferGroup and a new VulkanBuffer every frame. Fine for the once-at-setup call the documentation shows; not fine for a sixty-times-a-second re-upload.

Making this loop right therefore wants the two BufferParameters hoisted onto the MeshInstance so their result caches across frames, and applyToMesh writing into them rather than rebuilding them. That has not been done, because it cannot be verified without a device and this branch had no GPU verification path; the deformation itself is exercised through the CPU rasterizer and the path tracer instead. Deforming into the buffer — the part this class does — is independent of that and is unit-tested by CpuSkinnerTest.

@throws IllegalArgumentException if the instance has no skin — a rigid mesh needs no skinner, and silently doing nothing would be the wrong answer to a typo in a filename.

class CpuSkinner(
    val instance: GltfSupport.MeshInstance,
    val options: GltfSkinning.SkinningOptions = GltfSkinning.SkinningOptions(),
) {

val skin: GltfSkinning.MeshSkin

The mesh's skin; non-null by construction.

    val skin: GltfSkinning.MeshSkin = requireNotNull(instance.skin) {
        "MeshInstance '${instance.name}' has no skin; CpuSkinner has nothing to deform"
    }

val clips: List<String> get() = skin.skeleton.clips.map { it.name }

Clip names available on this mesh's skeleton, in file order.

    private val vertexCount = instance.buffers.vertexCount
    private val bindPositions = FloatArray(vertexCount * 3)
    private val bindNormals = FloatArray(vertexCount * 3)

init

    init {
        val f = instance.buffers.vertexData.asFloatBuffer()
        for (v in 0 until vertexCount) {
            for (k in 0..2) {
                bindPositions[v * 3 + k] = f.get(v * 12 + k)
                bindNormals[v * 3 + k] = f.get(v * 12 + 3 + k)
            }
        }
    }

fun pose(palette: List<Matrix4f>): GltfSkinning.SkinnedVertices

Deform to palette — one matrix per entry of skin.skeleton.bones — and write the result into the instance's vertex buffer, overwriting only the position and normal fields.

    fun pose(palette: List<Matrix4f>): GltfSkinning.SkinnedVertices {
        val out = GltfSkinning.skin(bindPositions, bindNormals, skin.influences, palette, options)
        val f = instance.buffers.vertexData.asFloatBuffer()
        for (v in 0 until vertexCount) {
            for (k in 0..2) {
                f.put(v * 12 + k, out.position[v * 3 + k])
                f.put(v * 12 + 3 + k, (out.normal ?: bindNormals)[v * 3 + k])
            }
        }
        return out
    }

fun pose(clip: String?, seconds: Float): GltfSkinning.SkinnedVertices

Deform to clip clip at seconds. null is the rest pose.

    fun pose(clip: String?, seconds: Float): GltfSkinning.SkinnedVertices =
        pose(skin.skeleton.palette(clip?.let { skin.skeleton.clip(it) }, seconds))

fun reset()

Put the bind pose back, bit for bit — the snapshot is written straight through.

    fun reset() {
        val f = instance.buffers.vertexData.asFloatBuffer()
        for (v in 0 until vertexCount) {
            for (k in 0..2) {
                f.put(v * 12 + k, bindPositions[v * 3 + k])
                f.put(v * 12 + 3 + k, bindNormals[v * 3 + k])
            }
        }
    }
}