Audio AI

AnalysisOperations.kt

Typed constructors for model-free musical analysis and learned speech representations.

Each one is a ModelOperation factory and nothing more — it validates its arguments here in the JVM, names the backend that serves it, and packs the parameters into the snake_case keys that backend expects. Submitting it, waiting for it and drawing it are FieldAudioClient's job.

Every constructor asks for a single candidate, because an analysis of a fixed window with fixed parameters has one answer; the seed family in styleCandidates has no counterpart here. All three also produce an .npz of full-resolution arrays plus, for a drawing, a downsampled AnalysisPreview — see analysisVisualizationHtml.

Which to reach for: multiScaleBeatAnalysis when you want when, and at several time scales at once, because a tempo estimate is only meaningful relative to a window length; monophonicPitchAnalysis when you know there is one voice and want a continuous F0 with a voiced probability; polyphonicPitchAnalysis when there are chords and you want discrete note events (and MIDI) rather than a curve.

src/jvmMain/kotlin/afield/audio/ai/AnalysisOperations.kt — read the source

Used in: Multi-instrument transcription with MuScriptor, Visualizers, Displays, and Little Instruments, Beat and pitch analysis

Top-level functions

Declaration	Summary
fun multiScaleBeatAnalysis(source: AssetRegion, scalesSeconds: List<Double> = listOf(2.0, 4.0, 8.0, 16.0), hopLength: Int = 256, minBpm: Double = 30.0, maxBpm: Double = 300.0, backend: String? = "analysis.librosa", requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Multi-scale local tempo trajectories plus a discrete beat grid.
fun spectralEmbeddingAnalysis(source: AssetRegion, hopLength: Int = 882, nFft: Int = 2048, nMels: Int = 64, nMfcc: Int = 20, fminHz: Double = 30.0, fmaxHz: Double? = null, backend: String? = "analysis.librosa", requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Short-time spectral shape for mel-transport, log-mel, and cepstral spaces.
fun monophonicPitchAnalysis(source: AssetRegion, fminHz: Double = 55.0, fmaxHz: Double = 1760.0, frameLength: Int = 2048, hopLength: Int = 256, resolution: Double = 0.1, backend: String? = "analysis.librosa", requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Frame-wise monophonic F0 and voiced probability from probabilistic YIN.
fun polyphonicPitchAnalysis(source: AssetRegion, onsetThreshold: Double = 0.5, frameThreshold: Double = 0.3, minimumNoteLengthMillis: Double = 127.7, minimumFrequencyHz: Double? = null, maximumFrequencyHz: Double? = null, multiplePitchBends: Boolean = false, melodiaTrick: Boolean = true, backend: String? = "analysis.basic_pitch", requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Note events, activation tensors, pitch bends, and MIDI from Spotify Basic Pitch.
fun musicTranscriptionAnalysis(source: AssetRegion, backend: MuscriptorBackend = MuscriptorBackend.SMALL, instruments: List<MuscriptorInstrument> = emptyList(), useSampling: Boolean = false, temperature: Double = 1.0, cfgCoefficient: Double = 1.0, beamSize: Int = 1, preludeForcing: Boolean = true, batchSize: Int? = null, strictEos: Boolean = false, tempoDetection: MuscriptorTempoDetection = MuscriptorTempoDetection.BEST_EFFORT, includeQuantizedMidi: Boolean = true, requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Multi-instrument note transcription with immutable MIDI and structured-note artifacts.
fun AnalysisResult.musicTranscriptionMidi(): AssetRepresentation	The performance-timed MIDI emitted by musicTranscriptionAnalysis.
fun AnalysisResult.musicTranscriptionQuantizedMidiOrNull(): AssetRepresentation?	Beat-quantized MIDI, present only when requested and a usable subdivision was detected.
fun AnalysisResult.musicTranscriptionMessagePack(): AssetRepresentation	Full onset/offset/pitch/instrument event document in the versioned binary interchange format.
fun speechEmbeddingAnalysis(source: AssetRegion, backend: SpeechEmbeddingBackend, wavlmLayer: Int = 6, chunkSeconds: Double = 20.0, contextSeconds: Double = 1.0, mdsDimensions: Int = 4, mdsLandmarks: Int = 384, mdsSmoothingMillis: Double = 80.0, requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	A 50 Hz learned speech trajectory, explicit voice activity, novelty, and a reproducible four-axis MDS map.
fun emotion2VecEmbeddingAnalysis(source: AssetRegion, chunkSeconds: Double = 20.0, contextSeconds: Double = 1.0, mdsLandmarks: Int = 384, mdsSmoothingMillis: Double = 80.0, requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Emotion/prosody-oriented speech embedding convenience constructor.
fun wavlmBasePlusEmbeddingAnalysis(source: AssetRegion, layer: Int = 6, chunkSeconds: Double = 20.0, contextSeconds: Double = 1.0, mdsLandmarks: Int = 384, mdsSmoothingMillis: Double = 80.0, requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	WavLM Base+ speech embedding convenience constructor.

MuscriptorInstrument

enum class MuscriptorInstrument(val id: String)

Instrument groups learned by MuScriptor 0.3.0 and accepted as hard transcription constraints.

Constants: ACOUSTIC_PIANO, ELECTRIC_PIANO, CHROMATIC_PERCUSSION, ORGAN, ACOUSTIC_GUITAR, CLEAN_ELECTRIC_GUITAR, DISTORTED_ELECTRIC_GUITAR, ACOUSTIC_BASS, ELECTRIC_BASS, VIOLIN, VIOLA, CELLO, CONTRABASS, ORCHESTRAL_HARP, TIMPANI, STRING_ENSEMBLE, SYNTH_STRINGS, VOICE, ORCHESTRA_HIT, TRUMPET, TROMBONE, TUBA, FRENCH_HORN, BRASS_SECTION, SOPRANO_AND_ALTO_SAX, TENOR_SAX, BARITONE_SAX, OBOE, ENGLISH_HORN, BASSOON, CLARINET, FLUTES, SYNTH_LEAD, SYNTH_PAD, DRUMS.

MuscriptorBackend

enum class MuscriptorBackend(val modelId: String)

Revision-pinned MuScriptor model exposed behind the model-neutral music-transcription operation.

Constants: SMALL, MEDIUM, LARGE.

MuscriptorTempoDetection

enum class MuscriptorTempoDetection

What MuScriptor should do when deriving MIDI tempo and meter from the selected audio.

Constants: REQUIRED, BEST_EFFORT, OFF.

SpeechEmbeddingBackend

enum class SpeechEmbeddingBackend(val modelId: String)

The two revision-pinned learned speech spaces exposed behind one model-neutral operation.

Constants: EMOTION2VEC, WAVLM_BASE_PLUS.

AnalysisVisualization.kt

Analysis viewers and dashboards for beat, pitch, and note results, with zoom, pan, and a shared time cursor.

analysisVisualizationHtml turns an AnalysisResult into a zoomable interactive display; the browser decides which of the three layouts to draw from AnalysisPreview.operation: onset strength and multi-scale tempo for a beat analysis, pYIN frequency and voicing probability for a monophonic pitch analysis, or a Basic Pitch piano roll with activation curves for a polyphonic one. analysisPreviewVisualizationHtml takes an already-fetched AnalysisPreview and works without any service round trip.

Dashboard variants (analysisDashboardHtml, analysisPreviewDashboardHtml) stack panels with a synchronized cursor, making beat/pYIN/Basic Pitch disagreements inspectable side by side. All panels support wheel zoom, drag pan, hover inspection, and click-to-seek, synchronized across panels that share a syncGroup.

src/jvmMain/kotlin/afield/audio/ai/AnalysisVisualization.kt — read the source

Used in: Visualizers, Displays, and Little Instruments, Multi-instrument transcription with MuScriptor

Top-level functions

Declaration	Summary
fun analysisVisualizationHtml(result: AnalysisResult, serviceBaseUrl: String = "/field-audio", width: Int = 1100, height: Int = 360, points: Int = 1600, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	The analysis viewer: one function that turns an AnalysisResult into a zoomable beat, pitch or note display for a box output.
fun analysisPreviewVisualizationHtml(preview: AnalysisPreview, width: Int = 1100, height: Int = 360, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Render a deterministic or already-fetched preview without contacting the service.
fun analysisDashboardHtml(results: List<AnalysisResult>, serviceBaseUrl: String = "/field-audio", width: Int = 1100, panelHeight: Int = 270, points: Int = 1600, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Stack several live results into one board. Zoom, pan and hover cursors are synchronized, which makes beat/pYIN/Basic Pitch disagreements inspectable.
fun analysisPreviewDashboardHtml(previews: List<AnalysisPreview>, width: Int = 1100, panelHeight: Int = 270, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Model-free counterpart used by documentation, fixtures and cached previews.

AudioAssetVisualization.kt

The one-line bridge from a Field Audio AssetRepresentation to Field's general-purpose waveform player.

It exists so that playing a service asset needs no URL arithmetic: the extension resolves the binary path and the server-side waveform endpoint against the same-origin /field-audio prefix and hands both to AudioVisualizer, which then fetches only the envelope of whatever window the user is looking at.

src/jvmMain/kotlin/afield/audio/ai/AudioAssetVisualization.kt — read the source

Top-level functions

Declaration	Summary
fun AssetRepresentation.audioVisualizerHtml(serviceBaseUrl: String = "/field-audio", label: String? = role, options: AudioVisualizerOptions = AudioVisualizerOptions()): String	Render any Field Audio WAV representation with range-aware server-side waveform summaries.

CandidateBoard.kt

The candidate board: the instrument for choosing between the alternatives a generation produced.

One function, one HTML fragment, three things a listener actually needs — a player per candidate, the source-into-candidate splice preview beside it so a take can be judged in place rather than in isolation, and a rating, selection flag and note that are written straight back to the service. The provenance line under each player states the sample rates and the conversion, because whether you are hearing the model or a resampler is not a detail.

src/jvmMain/kotlin/afield/audio/ai/CandidateBoard.kt — read the source

Top-level functions

Declaration	Summary
fun candidateBoardHtml(candidateSet: CandidateSet, serviceBaseUrl: String = "/field-audio"): String	A self-contained Field output-area candidate browser.

CodecVisualization.kt

The EnCodec reconstruction microscope: what a neural codec kept, what it threw away, and what it wrote down in between.

A codec round trip is three things that have to be looked at together — the audio in, the discrete tokens, the audio out — and this file is the rendering half of that view. It is browser-free below the top layer: the spectrograms, the waveform and the token map are drawn into PNG bytes on the JVM, so the same code produces a live box output, a test assertion and the committed picture on the EnCodec microscope page.

Everything here needs the bytes locally, which is what FieldAudioClient.codecVisualizationHtml is for — it downloads the three representations off Field's runloop and then renders.

src/jvmMain/kotlin/afield/audio/ai/CodecVisualization.kt — read the source

Used in: EnCodec round trip

Top-level functions

Declaration	Summary
fun FieldAudioClient.codecVisualizationHtml(result: CodecReconstructionResult, cacheDirectory: Path): CompletableFuture<String>	Download the three binary assets off Field's runloop, then render one self-contained output view.

CodecVisualizationData

data class CodecVisualizationData(val source: FloatArray, val reconstruction: FloatArray, val sampleRate: Int, val tokens: NpyIntTensor, val metrics: ReconstructionMetrics, val frameRate: Double, val cardinality: Int, val codecLabel: String, val sourceWav: ByteArray? = null, val reconstructionWav: ByteArray? = null)

Everything the EnCodec microscope draws, already decoded into the JVM.

Declaration	Summary
val source: FloatArray	The model input: the windowed, resampled audio the codec was given, as mono float samples.
val reconstruction: FloatArray	The decoded output at the same rate and length, so the two can be compared sample for sample.
val sampleRate: Int	Sample rate shared by source and reconstruction; load refuses inputs that disagree.
val tokens: NpyIntTensor	The discrete codes the encoder produced, indexed codebook then frame.
val metrics: ReconstructionMetrics	How far reconstruction is from source, as the backend measured it.
val frameRate: Double	Token frames per second, from the codec metadata; 50.0 when the backend did not say.
val cardinality: Int	Codebook size, from the codec metadata; 2048 when the backend did not say. Sets the token colour scale.
val codecLabel: String	Model name for the heading, from the codec metadata; "EnCodec" when the backend did not say.
val sourceWav: ByteArray?	The original WAV bytes of source, kept for browser playback.
val reconstructionWav: ByteArray?	The original WAV bytes of reconstruction, with the same purpose as sourceWav.

CodecVisualization

object CodecVisualization

Rendering shared by live Field output, tests, and the public executable documentation.

Declaration	Summary
fun load(result: CodecReconstructionResult, modelInputWav: Path, reconstructionWav: Path, tokenNpy: Path): CodecVisualizationData	Read the three downloaded files into one CodecVisualizationData.
fun renderHtml(data: CodecVisualizationData, width: Int = 1200): String	The whole microscope as one self-contained HTML fragment: two players, two log-frequency spectrograms, and the token map with a hover readout.
fun renderWaveform(samples: FloatArray, width: Int, height: Int): ByteArray	A min/max waveform as PNG bytes, one column per pixel.
fun renderLogSpectrogram(samples: FloatArray, sampleRate: Int, width: Int, height: Int): ByteArray	A log-frequency STFT as PNG bytes: 1024-point Hann windows, 256-sample hop, 35 Hz to Nyquist, 90 dB of range below the clip's own peak.
fun renderTokens(tensor: NpyIntTensor, cardinality: Int, width: Int, height: Int): ByteArray	The token tensor as PNG bytes: time across, codebooks down, coarse to fine, one hue per code.

CorpusClip.kt

Turns a window of a recording in the Field audio corpus into a box that plays it.

corpusClip fetches a recording from the Field audio service, caches its WAV, and cuts the requested window into a CorpusClip, whose CorpusClip.sampler plays it. The rest of the file places a neighbour found by the corpus microscope (corpusMicroscopeHtml) as a box of its own: corpusNeighbourName and corpusNeighbourCode give the box its name and its code, and placeCorpusNeighbour creates it beside an existing box.

src/jvmMain/kotlin/afield/audio/ai/CorpusClip.kt — read the source

Top-level functions

Declaration	Summary
fun corpusClip(assetId: String, startSeconds: Double, endSeconds: Double, client: FieldAudioClient = FieldAudioClient()): CorpusClip	The window startSeconds..endSeconds of the recording assetId, fetched from the Field audio service. The recording's WAV is cached under ~/.afield/cache/field-audio, so placing several windows of one recording downloads it once. This blocks until the bytes are here.
fun corpusNeighbourName(neighbour: CorpusNeighbour): String	The name a placed neighbour's box gets: its recording and window.
fun corpusNeighbourCode(neighbour: CorpusNeighbour): String	The code a placed neighbour's box gets: ordinary box code that plays the window.
fun placeCorpusNeighbour(info: SessionInformation, beside: Box, neighbour: CorpusNeighbour, index: Int = 0): Box	Make the box for neighbour beside beside: named after the window, its code corpusNeighbourCode. index stacks the boxes of several placed neighbours below one another.
fun Box.placeCorpusNeighbour(neighbour: CorpusNeighbour, index: Int = 0): Box	placeCorpusNeighbour beside this box, in its own document: what a card's place button does.

CorpusClip

class CorpusClip(val assetId: String, val startSeconds: Double, val endSeconds: Double, val sample: AudioSample)

One window of a corpus recording, in memory: what a neighbour the corpusMicroscopeHtml found becomes when it is placed as a box of its own.

Declaration	Summary
val assetId: String	The recording the window is cut from.
val startSeconds: Double	Where the window starts in that recording, in seconds.
val endSeconds: Double	Where it ends, in seconds.
val sample: AudioSample	The window's samples, the recording's left channel.
fun sampler(): Sampler	A Sampler over the window, already triggered, so it plays as soon as it is launched.
override fun toString(): String	The recording and the window, with both times to two decimal places: CorpusClip(sha256:…, 21.00–27.00 s).

CorpusMicroscope.kt

The audition surface for a corpus nearest-neighbour query: the query window and every window the search returned, as players on one shared relative timeline.

corpusMicroscopeHtml turns the CorpusPreview an analysis.find_neighbours job leaves behind into a single self-contained section of HTML, and it refuses a preview from any other operation. Nothing in it reads audio: each AssetRepresentation in the preview becomes a media URL and a waveform URL handed to AudioVisualizer, which does the fetching, the drawing and the shared-cursor arithmetic.

A card's header prints two readings of the same number. One is the raw CorpusNeighbour distance; the other is a percentage that maps distance 0 to 100% and saturates at 2, so a neighbour further away than that reads as no similarity at all. The per-group bars apply the same mapping to each entry of groupDistances and keep the raw distance in the tooltip, so a bar and the header can be read against each other. The weights above the grid are taken from the preview, which means they arrive with the result rather than from the CorpusDistanceWeights the query was written with.

The footer carries what the players cannot. A retrieved window is a stretch of some other recording, so the card keeps its absolute start and end, its scale in seconds, and whether CorpusSegment records a previous and a next window. A neighbour that sits at the very beginning or the very end of its recording shows up there.

src/jvmMain/kotlin/afield/audio/ai/CorpusMicroscope.kt — read the source

Used in: Corpus microscope: from recordings to related windows

Top-level functions

Declaration	Summary
fun corpusMicroscopeHtml(preview: CorpusPreview, queryAudio: AssetRepresentation, serviceBaseUrl: String = "/field-audio", width: Int = 1100, cardWidth: Int = 510, syncGroup: String = "corpus-${preview.query.assetId.takeLast(12)}-${preview.query.startSeconds}", placeControlIds: List<String>? = null): String	A source-relative audition board for CorpusPreview.neighbours.
fun Box.showCorpusMicroscope(preview: CorpusPreview, queryAudio: AssetRepresentation, channel: String = "microscope", width: Int = 1100, cardWidth: Int = 510)	corpusMicroscopeHtml in channel, with a place button on every neighbour card: pressing it makes a box beside this one whose code plays that neighbour's window (placeCorpusNeighbour).

CorpusOperations.kt

Multi-scale corpus operations: segment a source into overlapping windows, embed each window with grouped audio descriptors, and retrieve neighbours by weighted distance.

The three operations form a pipeline that runs server-side. segmentCorpus enumerates deterministic time windows at several scales without computing any descriptors. embedCorpus computes and persists the five grouped descriptor families (harmony, pitch, texture, rhythm, dynamics) for every window. findCorpusNeighbours retrieves the closest embedded windows to a query using CorpusDistanceWeights, which controls how much each group contributes to the distance. All three produce a ModelOperation that FieldAudioClient.submit executes.

src/jvmMain/kotlin/afield/audio/ai/CorpusOperations.kt — read the source

Used in: Corpus microscope: from recordings to related windows

Top-level functions

Declaration	Summary
fun segmentCorpus(source: AssetRegion, scalesSeconds: List<Double> = listOf(1.5, 3.0, 6.0, 12.0), overlap: Double = 0.5, backend: String? = "analysis.librosa", requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Enumerate deterministic source-adjacent windows without computing descriptors.
fun embedCorpus(source: AssetRegion, scalesSeconds: List<Double> = listOf(1.5, 3.0, 6.0, 12.0), overlap: Double = 0.5, hopLength: Int = 512, nFft: Int = 2048, backend: String? = "analysis.librosa", requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Compute and persist grouped descriptors for every multi-scale source window.
fun findCorpusNeighbours(source: AssetRegion, scaleSeconds: Double = source.durationSeconds, limit: Int = 12, weights: CorpusDistanceWeights = CorpusDistanceWeights(), includeSameAsset: Boolean = true, hopLength: Int = 512, nFft: Int = 2048, backend: String? = "analysis.librosa", requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Retrieve indexed windows by an explicit weighted descriptor distance.

CorpusDistanceWeights

data class CorpusDistanceWeights(val harmony: Double = 1.0, val pitch: Double = 0.7, val texture: Double = 1.0, val rhythm: Double = 0.8, val dynamics: Double = 0.45)

Explicit weights for the five interpretable descriptor groups used by corpus retrieval. Each scales how much its own group counts towards the distance between two windows, a zero takes that group out of the comparison, and at least one of the five has to be positive.

Declaration	Summary
val harmony: Double	Mean and variation of the twelve chroma classes: the chord and key colour of a window.
val pitch: Double	Dominant-pitch centre, dominant-pitch spread, and how much of the window is voiced.
val texture: Double	MFCCs with spectral centroid, bandwidth, flatness and rolloff: timbre rather than notes.
val rhythm: Double	Onset strength, peak density and local tempo statistics.
val dynamics: Double	RMS centre, spread and range: the loudness shape, independent of what is making it.

FieldAudioClient.kt

The verbs of Field Audio: one client object whose every method is a request to the model service, plus the typed constructors for the two operations that ship with real controls.

The protocol types — AssetRegion, ModelOperation, JobRecord and the rest — are the vocabulary; this is what you can do with it. The shape to hold on to is that nothing here blocks and nothing here runs a model in the JVM: a call returns a CompletableFuture that completes on the client's own executor, and the model work happens in a Python worker that may take minutes. Composing futures (thenCompose, thenAccept) is therefore the only correct way to use this from a box — a .get() on Field's runloop freezes the editor for as long as the model takes.

Argument validation happens here, in the JVM, before anything is sent: require failures name the constraint. Everything the service rejects instead arrives as a FieldAudioException carrying the service's own StructuredError, and a job that fails after being accepted arrives as a FieldAudioJobException from one of the await… helpers.

src/jvmMain/kotlin/afield/audio/ai/FieldAudioClient.kt — read the source

Used in: Field Audio AI, EnCodec round trip, Gesture board: drawing toward a sound

Top-level functions

Declaration	Summary
fun styleCandidates(source: AssetRegion, prompt: String = "", durationSeconds: Double = 12.0, seeds: LongRange = 0L until 8L, backend: String? = null, requestedBy: String? = null, controls: MusicGenStyleControls = MusicGenStyleControls(), forceRegenerate: Boolean = false): ModelOperation	Ergonomic live-code constructor for the source-window style instrument.
fun codecReconstruction(source: AssetRegion, numCodebooks: Int? = null, backend: String? = "audiocraft.encodec_musicgen_style_32khz", requestedBy: String? = null, forceRegenerate: Boolean = false): ModelOperation	Exact EnCodec encode/decode round trip for one source selection.

FieldAudioException

class FieldAudioException(val statusCode: Int, val responseBody: String, val error: StructuredError? = null) : RuntimeException(error?.let { "${it.code}: ${it.message}" } ?: "Field Audio HTTP $statusCode: ${responseBody.take(500)}")

The service said no, and said why.

Declaration	Summary
val statusCode: Int	HTTP status of the response that failed.
val responseBody: String	The response body verbatim, for the cases error could not be decoded from it.
val error: StructuredError?	The service's structured error, when the body contained one. See StructuredError.code.

FieldAudioJobException

class FieldAudioJobException(val jobId: String, val state: JobState, val error: StructuredError?) : RuntimeException(error?.let { "${it.code}: ${it.message}" } ?: "audio job $jobId ended as $state")

A job was accepted and then did not finish successfully.

Declaration	Summary
val jobId: String	The job that ended badly; still fetchable with FieldAudioClient.inspect.
val state: JobState	The terminal state it ended in — anything but COMPLETE.
val error: StructuredError?	Why, when the job recorded a structured error. Null for a plain cancellation.

FieldAudioTransportException

class FieldAudioTransportException(val baseUri: URI, val requestPath: String, message: String, cause: Throwable?) : RuntimeException(message, cause)

There was no conversation: the service could not be reached, or did not answer in the protocol.

Declaration	Summary
val baseUri: URI	The client's base address — the thing to check is running, or to point somewhere else.
val requestPath: String	Path of the request that failed, e.g. /v1/assets/import.

FieldAudioClient

class FieldAudioClient(baseUrl: String = "http://127.0.0.1:8765", private val callbackExecutor: Executor = DEFAULT_EXECUTOR)

Non-blocking, backend-neutral client for Field's audio model service.

Declaration	Summary
val baseUri: URI	Where this client points, with any trailing slash removed so relative paths resolve cleanly.
val json: Json	The shared Field Audio wire format. Prefer FieldAudioClient.json when no client is needed.
fun health(): CompletableFuture<String>	Liveness only: the raw body of GET /health. It starts nothing and loads no model.
fun models(): CompletableFuture<List<ModelCapability>>	Every backend the service knows about, including the ones it cannot currently run.
fun importWav(path: Path): CompletableFuture<AudioAsset>	Upload a WAV file and get the asset it became.
fun describe(assetId: String): CompletableFuture<AudioAsset>	Fetch one asset and all its representations by content address.
fun assetByFilename(pattern: String, preferredRole: String? = "canonical_project"): CompletableFuture<AudioAsset>	The one asset whose original filename matches pattern — and an error if that is not exactly one.
fun assetsByFilename(pattern: String, preferredRole: String? = "canonical_project"): CompletableFuture<List<AudioAsset>>	Every asset whose original filename matches pattern, preferring preferredRole representations.
fun waveform(representationId: String, startSeconds: Double = 0.0, durationSeconds: Double? = null, points: Int = 1200): CompletableFuture<WaveformSummary>	A min/max envelope of one representation's window, for drawing.
fun submit(operation: ModelOperation): CompletableFuture<JobAccepted>	Submit an operation and get the job it created — the start of every piece of model work.
fun compileGesture(board: GestureBoard, operation: String = "generate.style", parameters: JsonObject = JsonObject(emptyMap())): CompletableFuture<GestureCompilation>	Ask the authoritative service to expose exactly how a board reaches this backend.
fun inspect(jobId: String): CompletableFuture<JobRecord>	The durable state of one job right now. This is the authority, not the progress stream.
fun jobs(state: JobState? = null, limit: Int = 100): CompletableFuture<List<JobSummary>>	Recent jobs, newest first, optionally filtered to one state. limit must be within 1..500.
fun cancel(jobId: String): CompletableFuture<JobRecord>	Ask for a job to stop, and get its state back.
fun candidateSet(id: String): CompletableFuture<CandidateSet>	Fetch a stored candidate family by id — as taken from JobRecord.candidateSetId.
fun codecResult(id: String): CompletableFuture<CodecReconstructionResult>	Fetch a stored codec round trip by id — as taken from JobRecord.codecResultId.
fun analysisResult(id: String): CompletableFuture<AnalysisResult>	Fetch a stored analysis by id — as taken from JobRecord.analysisResultId.
fun analysisPreview(id: String, points: Int = 1200): CompletableFuture<AnalysisPreview>	The plot-sized view of an analysis: at most points samples per series, 16..4096.
fun corpusPreview(id: String): CompletableFuture<CorpusPreview>	Fetch bounded segment/neighbour metadata and playable handles for a corpus analysis result.
fun judge(candidateSetId: String, candidateIndex: Int, judgement: CandidateJudgement): CompletableFuture<String>	Write a rating, selection flag and note back onto one candidate.
fun download(representation: AssetRepresentation, destination: Path): CompletableFuture<Path>	Fetch a representation's bytes to destination, creating parent directories, and return the path.
fun downloadBytes(representation: AssetRepresentation): CompletableFuture<ByteArray>	Fetch a binary representation into memory without routing it through JSON or a temporary file.
fun await(jobId: String, pollEvery: Duration = Duration.ofMillis(200)): CompletableFuture<JobRecord>	Poll durable job state without occupying Field's runloop.
fun awaitCandidates(accepted: JobAccepted): CompletableFuture<CandidateSet>	Poll a generate.* job to the end and hand back its candidate family.
fun awaitCodec(accepted: JobAccepted): CompletableFuture<CodecReconstructionResult>	Poll a codec.* job to the end and hand back its reconstruction result.
fun awaitAnalysis(accepted: JobAccepted): CompletableFuture<AnalysisResult>	Poll an analysis.* job to the end and hand back its analysis result.
fun watch(jobId: String, onError: (Throwable) -> Unit = {}, onEvent: (ProgressEvent) -> Unit): CompletableFuture<WebSocket>	Subscribe to progress history plus live events. The server closes the socket after a terminal event. The returned socket can be used to cancel the subscription.

MusicGenStyleControls

data class MusicGenStyleControls(val useSampling: Boolean = true, val topK: Int = 250, val topP: Double = 0.0, val temperature: Double = 1.0, val cfg: Double = 3.0, val cfgBeta: Double? = 5.0, val twoStepCfg: Boolean = false, val styleQuantization: Int = 1, val styleExcerptSeconds: Double? = null, val styleDownsampleFactor: Int? = null, val styleEncodecCodebooks: Int? = null)

Global controls exposed by the pinned MusicGen-Style checkpoint.

Declaration	Summary
val useSampling: Boolean	Sample from the distribution rather than taking the argmax. Off makes output deterministic and dull.
val topK: Int	Keep only the k most likely tokens at each step; 0 disables the cut. Must be non-negative.
val topP: Double	Nucleus sampling threshold within 0..1; 0 disables it. Use one of topK and topP, not both.
val temperature: Double	Softmax temperature, strictly positive. Above 1 wanders, below 1 commits.
val cfg: Double	Classifier-free guidance strength, non-negative. Higher follows the conditioning harder and varies less.
val cfgBeta: Double?	Balance between the text prompt and the style excerpt, non-negative.
val twoStepCfg: Boolean	Run guidance as two forward passes instead of one batched pass: slower, occasionally different.
val styleQuantization: Int	Size of the style bottleneck, strictly positive. Larger keeps more of the source's identity; smaller abstracts it into gesture.
val styleExcerptSeconds: Double?	How much of the source window the style encoder actually reads, in seconds.
val styleDownsampleFactor: Int?	Temporal downsampling of the style representation, strictly positive. Null leaves the model's default.
val styleEncodecCodebooks: Int?	How many EnCodec codebooks the style path uses, strictly positive. Fewer is a coarser style.

FieldPersistence.kt

How a Field document remembers audio work: ids and judgements in a box property, and not one sample of audio.

The rule this file enforces is that Python is a worker and Field is the authority. The service owns immutable, content-addressed assets; the document owns which of them it refers to and what you thought of them. That is why saved state is a FieldAudioProjectState of ids rather than audio, and why reopening a document and re-fetching those ids gets the same bytes back.

src/jvmMain/kotlin/afield/audio/ai/FieldPersistence.kt — read the source

FieldAudioPersistence

object FieldAudioPersistence

Persist only protocol data and asset IDs; Python remains a worker, never project authority.

Declaration	Summary
const val DEFAULT_PROPERTY	The box property the project state lives in unless you name another one.
fun install(property: String = DEFAULT_PROPERTY)	Register property as persistent and JSON-sidecar-separated on this document.
fun load(box: Box, property: String = DEFAULT_PROPERTY): FieldAudioProjectState	Read the saved state out of box, or an empty FieldAudioProjectState if there is none yet.
fun save(box: Box, state: FieldAudioProjectState, property: String = DEFAULT_PROPERTY)	Write state into box as pretty-printed JSON, replacing whatever was there.

GestureBoard.kt

Temporal sketches for source-conditioned generation, with prompt regions, onset events, and parameter curves that together express a generation intent.

Build a GestureBoard with the gestureBoard{} DSL and pass it to gestureCandidates to produce a multi-seed ModelOperation. Prompt regions are compiled into the model's global text condition; onset events and parameter curves are evaluative: the service scores each generated candidate against them rather than passing them as native model controls.

compileGestureBoard previews the protocol locally and returns a GestureCompilation that lists how each board element is wired to the backend. Example:

val board = gestureBoard(8.0, "acoustic guitar, warm") {
    prompt(0.0, 4.0, "sparse, arpeggiated")
    event(3.5, "accent")
    curve("energy", GestureCurveKind.ACTIVITY) {
        point(0.0, 0.2); point(4.0, 0.8); point(8.0, 0.3)
    }
}
val operation = gestureCandidates(source, board, seeds = 0L until 4L)

src/jvmMain/kotlin/afield/audio/ai/GestureBoard.kt — read the source

Used in: Gesture board: drawing toward a sound

Top-level functions

Declaration	Summary
fun gestureBoard(durationSeconds: Number, prompt: String = "", block: GestureBoardBuilder.() -> Unit = {}): GestureBoard	A live-codeable temporal sketch for source-conditioned generation.
fun validateGestureBoard(board: GestureBoard): GestureBoard	Checks a board against the version-1 rules and returns it, so it can be used inline. It enforces timing (regions and points inside the duration, curve times strictly increasing), ranges (values and strengths in 0..1, weights in (0, 10]), unique curve names, and that each element's delivery is the one protocol version 1 defines for it. The gestureBoard{} builder validates on the way out and every function here that takes a board validates on the way in, so this is the entry point for a board assembled by hand or one that arrived over the wire.
fun compileGestureBoard(board: GestureBoard, operation: String = "generate.style", parameters: JsonObject = JsonObject(emptyMap())): GestureCompilation	Immediate local preview of the service's version-1 compilation contract.
fun gestureCandidates(source: AssetRegion, board: GestureBoard, seeds: LongRange = 0L until 8L, backend: String? = null, requestedBy: String? = null, controls: MusicGenStyleControls = MusicGenStyleControls(), forceRegenerate: Boolean = false): ModelOperation	Source-style generation plus an inspectable gesture sketch and candidate scoring.

GestureBoardDsl

annotation class GestureBoardDsl

The DSL scope marker shared by the two gesture-board builders. It stops a nested curve{} block from reaching past its own receiver to call prompt or event on the board around it.

GestureCurveBuilder

class GestureCurveBuilder internal constructor(private val name: String, private val kind: GestureCurveKind, private val weight: Double)

Collects the breakpoints of one curve inside a curve{} block. Points are kept in the order they are written, and strictly increasing times, values in 0..1 and a minimum of two points are all checked when the enclosing board is built, not as each point arrives.

Declaration	Summary
fun point(timeSeconds: Number, value: Number)	Adds one breakpoint to the curve: timeSeconds along the board, value in 0..1.

GestureBoardBuilder

class GestureBoardBuilder internal constructor(private val durationSeconds: Double, private val basePrompt: String)

The receiver of a gestureBoard{} block: prompt regions, events and curves accumulate here in the order they are written, and the finished board is validated once, when the block returns.

Declaration	Summary
var metadata: JsonObject	Free-form JSON that travels with the board to the service. Neither validation nor compilation reads it, so it is somewhere to record where a sketch came from.
fun prompt(startSeconds: Number, durationSeconds: Number, text: String, weight: Number = 1.0)	Adds a timed text region. Protocol version 1 has no timed text condition, so compileGestureBoard folds each region into the one global prompt, worded from As to Bs: text, in the order the regions were written; a region has to fit inside the board's duration and its weight has to lie in (0, 10].
fun event(timeSeconds: Number, label: String, strength: Number = 1.0, weight: Number = 1.0)	Marks a labelled onset at timeSeconds. Events are evaluative: nothing is conditioned on them, and a generated candidate is rewarded for an accent near the marked time. strength is 0..1 and weight is in (0, 10].
fun curve(name: String, kind: GestureCurveKind, weight: Number = 1.0, block: GestureCurveBuilder.() -> Unit)	Opens a named lane of one GestureCurveKind and collects its points. Names have to be unique within a board and a curve needs at least two points. Like an event, a curve is measured against the generated waveform rather than sent as a model control, and the pinned contract has four kinds to measure against: density, activity, register and brightness.

GestureBoardVisualization.kt

A gesture board drawn as a picture of itself: prompt regions, onset events and parameter curves on one timeline, each mark labelled with how far into the model it reaches.

gestureBoardHtml renders a GestureBoard as a single self-contained section, with inline styles, one SVG and a few lines of script for the hover cursor. There is no asset to fetch and no library to install. Every rule and the script are scoped to an id derived from the board's hash, so one document can show several boards at once. Prompt regions become blocks on the prompt lane, events become dashed needles at their marked time, and each curve gets a lane of its own coloured by its GestureCurveKind.

The chips above the timeline are the bindings of a GestureCompilation, and they separate a mark the backend is conditioned on from a mark it is only scored against. By default the compilation is the local compileGestureBoard preview of the service's version-1 contract rather than a record of what some particular job sent, and the effective prompt printed beside the chips is the time-worded text that preview produces.

Passing a CandidateSet adds a ranking strip under the chips: a row per candidate, ordered by gesture score where the service returned one and by the candidate's own score otherwise, with a bar for each evaluated lane. A board is validated before anything is drawn, so a sketch the service would reject fails here rather than being rendered.

src/jvmMain/kotlin/afield/audio/ai/GestureBoardVisualization.kt — read the source

Top-level functions

Declaration	Summary
fun gestureBoardHtml(board: GestureBoard, compilation: GestureCompilation = compileGestureBoard(board), candidates: CandidateSet? = null, width: Int = 1080): String	Draws board as a timeline section: the top-level name for GestureBoardVisualization's renderer, with the same defaults, and the one a box calls.

GestureBoardVisualization

object GestureBoardVisualization

Interactive, dependency-free timeline renderer for a GestureBoard.

Declaration

Summary

fun renderHtml(board: GestureBoard, compilation: GestureCompilation = compileGestureBoard(board), candidates: CandidateSet? = null, width: Int = 1080): String

Draws board over width pixels of timeline, inside a card 38 pixels wider than that. width must be at least 640 and the SVG keeps it, so a narrower output area scrolls the timeline horizontally instead of compressing it. compilation defaults to the local preview; passing the result of FieldAudioClient.compileGesture instead shows how the live service wires this same board. candidates adds the ranking strip and changes nothing else on the timeline.

JobMonitor.kt

The job monitor: the service's own live view of every durable job, embedded in a box output.

It is an <iframe> rather than a Kotlin-rendered table, because the page is served by the same service that owns the jobs, so it cannot disagree with them, and it needs no polling code on the Field side. Same-origin through the /field-audio prefix, so nothing here reaches across origins.

src/jvmMain/kotlin/afield/audio/ai/JobMonitor.kt — read the source

Used in: Beat and pitch analysis, Corpus microscope: from recordings to related windows, Short-time spectral spaces

Top-level functions

Declaration	Summary
fun fieldAudioJobMonitor(serviceBaseUrl: String = "/field-audio", heightPixels: Int = 420): String	An embeddable, same-origin view of all durable Field Audio jobs.
fun fieldAudioJobStatusHtml(accepted: JobAccepted, event: ProgressEvent? = null): String	A small per-operation status card suitable for replacing one named Field output as events arrive.
fun fieldAudioJobStatusHtml(job: JobRecord): String	Render durable inspected state, including the structured error when a job failed.

MusicTranscription.kt

Typed decoder for the versioned note document a multi-instrument transcription job leaves behind.

musicTranscriptionAnalysis in AnalysisOperations.kt builds the request, and three accessors beside it locate what came back: performance MIDI, optional beat-quantized MIDI, and the MessagePack event document. This file reads the last of those. decodeMusicTranscriptionMessagePack turns its bytes into a MusicTranscription, and FieldAudioClient.musicTranscription is the one-call path from a finished AnalysisResult.

MessagePack rather than JSON keeps the note times binary, so an onset reaches the JVM as the float64 the worker wrote. The decoder rejects a payload whose format is not field.audio.music-transcription.v1, rejects trailing data after the root map, and range-checks every note: non-negative onset, offset at or after onset, MIDI pitch inside 0..127. A malformed artifact fails at the decode rather than at the first drawing that reads it.

Most of the document is optional. MusicTranscription.beatGrid is null when the artifact carries no beat_grid or a nil one, and inside a grid beatsPerBar, onsetDelaySeconds and beatSubdivision are each independently absent. The four provenance fields come from the artifact's metadata map and are null when the worker recorded nothing there. instrument is a string naming one of the worker's own instrument groups, which is a model's hypothesis about a mixed recording and not a reading of what played it. See Multi-instrument transcription with MuScriptor for how far that has been checked against a known source.

src/jvmMain/kotlin/afield/audio/ai/MusicTranscription.kt — read the source

Used in: Multi-instrument transcription with MuScriptor

Top-level functions

Declaration	Summary
fun decodeMusicTranscriptionMessagePack(bytes: ByteArray): MusicTranscription	Decode the complete source-aligned note document without routing numerical data through JSON.
fun FieldAudioClient.musicTranscription(result: AnalysisResult): CompletableFuture<MusicTranscription>	Download and decode the complete typed note document retained by musicTranscriptionAnalysis.

MusicTranscriptionNote

data class MusicTranscriptionNote(val startSeconds: Double, val endSeconds: Double, val midi: Int, val instrument: String)

One source-aligned note recovered by a multi-instrument transcription worker.

Declaration	Summary
val startSeconds: Double	Where the note begins, in seconds. The decoder rejects a negative onset.
val endSeconds: Double	Where the note ends, in seconds. Never earlier than startSeconds; the decoder rejects a pair that is.
val midi: Int	MIDI note number, 60 being middle C. The decoder rejects anything outside 0..127.
val instrument: String	The worker's own name for the instrument group it put this note in. It is a model's reading of a mixed recording rather than a record of what played the note.

MusicTranscriptionBeatGrid

data class MusicTranscriptionBeatGrid(val bpm: Double, val beatsPerBar: Int?, val firstDownbeatSeconds: Double, val onsetDelaySeconds: Double?, val beatSubdivision: Int?)

Tempo, meter, and the measured model-onset correction retained with a transcription.

Declaration	Summary
val bpm: Double	Tempo in beats per minute. The decoder rejects a grid whose bpm is not positive.
val beatsPerBar: Int?	Beats to the bar, or null where the artifact records no meter.
val firstDownbeatSeconds: Double	Where the first downbeat falls, in seconds.
val onsetDelaySeconds: Double?	The onset correction the worker recorded, in seconds, or null where it recorded none.
val beatSubdivision: Int?	How many parts the worker divided each beat into, or null where the artifact records no subdivision.

MusicTranscription

data class MusicTranscription(val notes: List<MusicTranscriptionNote>, val beatGrid: MusicTranscriptionBeatGrid?, val backend: String?, val modelId: String?, val modelRevision: String?, val checkpointSha256: String?)

Typed contents of the versioned MuScriptor MessagePack artifact.

Declaration	Summary
val notes: List<MusicTranscriptionNote>	Every note in the artifact, in the order it lists them.
val beatGrid: MusicTranscriptionBeatGrid?	Tempo and meter, or null where the artifact carries no beat grid or an explicitly nil one.
val backend: String?	Which transcription backend the worker ran, from the artifact's metadata. Null where it recorded nothing.
val modelId: String?	The model the backend loaded, from the artifact's metadata. Null where it recorded nothing.
val modelRevision: String?	The revision of that model, from the artifact's metadata. Null where it recorded nothing.
val checkpointSha256: String?	SHA-256 of the checkpoint weights, from the artifact's metadata. Null where it recorded nothing.

NpyIntTensor.kt

A dependency-free reader for the integer NPY tensors the audio service emits — the codec token grids, in practice.

numpy.kt already reads .npy and .npz generally; this is the narrow, self-checking version for the one shape that matters here (codebooks × frames of small integers), so the codec views can index tokens by codebook and frame without a general tensor type in between. It refuses what it cannot represent, rather than guessing: Fortran order, an unsupported dtype and a multi-item batch are all errors with a message.

src/jvmMain/kotlin/afield/audio/ai/NpyIntTensor.kt — read the source

NpyIntTensor

data class NpyIntTensor(val shape: IntArray, val values: LongArray)

Small, dependency-free reader for the C-order integer NPY tensors emitted by the audio service.

Declaration	Summary
val shape: IntArray	Array dimensions as the NPY header gave them; the last two are codebooks and time.
val values: LongArray	All codes, C-order (row-major), widened to Long whatever the on-disk integer width was.
val codebooks: Int	Size of the second-to-last axis: how many codebooks the encoder used.
val frames: Int	Size of the last axis: how many token frames, and so the time extent.
operator fun get(codebook: Int, frame: Int): Long	One code, by codebook and frame. Unchecked: an out-of-range index is an array error, not null.

Protocol.kt

The wire vocabulary of Field Audio — one Kotlin type per JSON object in the pinned service contract, and nothing else.

Field Audio runs models out of process: a long-lived Python service owns inference, and a box talks to it over HTTP and one WebSocket through FieldAudioClient. These are the values that cross that boundary. Every SerialName here is the snake_case name the service uses, and the committed, model-free snapshot of the same contract is published as the Field Audio service reference — if the two disagree, the service is right and this file is stale.

Three facts that explain the shapes

• Results are assets, not samples. Everything the service produces is written once, named by the SHA-256 of its bytes (sha256: followed by 64 hex digits) and referred to afterwards by id. That is why a result type carries an AssetRepresentation rather than a FloatArray: the audio is still on the service, and FieldAudioClient.download is how bytes reach the JVM.
• Work is a job, not a call. Submitting an operation returns a JobAccepted immediately; the outcome arrives later as a JobRecord you poll or a stream of ProgressEvents you watch.
• Identical requests are not repeated. The service hashes the request (ignoring ModelOperation.forceRegenerate) into JobRecord.requestHash and returns the stored result when it has seen that hash before, with cacheHit = true and a job that is already complete.

Times, ids and timestamps

All times and durations are seconds as Double unless the property name says otherwise. createdAt, updatedAt and ProgressEvent.at are ISO-8601 UTC timestamp strings, kept as text because nothing in Field parses them and a string round-trips through project persistence unchanged. Ids are opaque and prefixed by kind: sha256: for assets, rep_ for representations, job_, op_, cset_, codec_ and analysis_ for the rest.

The client decodes with ignoreUnknownKeys, so a newer service that adds a field will not break an older Field; the reverse is not true, because the service rejects unknown fields it is sent.

src/jvmMain/kotlin/afield/audio/ai/Protocol.kt — read the source

Used in: Field Audio service reference, Corpus microscope: from recordings to related windows, Visualizers, Displays, and Little Instruments

AssetRegion

data class AssetRegion(@SerialName("asset_id") val assetId: String, @SerialName("start_seconds") val startSeconds: Double, @SerialName("duration_seconds") val durationSeconds: Double)

A time window inside one asset — the unit of "what should the model listen to".

Declaration	Summary
val assetId: String	Content address of the asset to read, of the form sha256: plus 64 hex digits.
val startSeconds: Double	Offset of the window from the start of the asset, in seconds; must be non-negative.
val durationSeconds: Double	Length of the window in seconds; must be strictly positive.

AssetRepresentation

data class AssetRepresentation(val id: String, @SerialName("asset_id") val assetId: String, val role: String, @SerialName("media_type") val mediaType: String, @SerialName("sample_rate") val sampleRate: Int? = null, val channels: Int? = null, @SerialName("frame_count") val frameCount: Long? = null, @SerialName("duration_seconds") val durationSeconds: Double? = null, @SerialName("native_sample_rate") val nativeSampleRate: Int? = null, @SerialName("resampling_method") val resamplingMethod: String? = null, @SerialName("binary_url") val binaryUrl: String, val metadata: JsonObject = JsonObject(emptyMap()))

One concrete encoding of one asset's bytes: what format they are in, and where to fetch them.

Declaration	Summary
val id: String	Stable representation id: rep_ followed by a digest of the asset, role and format.
val assetId: String	Content address of the asset these bytes belong to.
val role: String	What this representation is for, as one of the role names described above.
val mediaType: String	IANA media type of the bytes, e.g. audio/wav or application/x-npy.
val sampleRate: Int?	Sample rate of these bytes in Hz, or null when the representation is not audio.
val channels: Int?	Channel count of these bytes, or null when the representation is not audio.
val frameCount: Long?	Length in sample frames, or null when the representation is not audio.
val durationSeconds: Double?	Length in seconds, or null when the representation is not audio.
val nativeSampleRate: Int?	The rate the producing model actually worked at, when sampleRate is a resample of it.
val resamplingMethod: String?	How the resample to sampleRate was done, when one happened; null when nothing was resampled.
val binaryUrl: String	Service-relative path to the bytes: /v1/representations/{id}/content.
val metadata: JsonObject	Free-form provenance the producing step attached — backend id and revision, operation and request hash, seed, source asset, decoded WAV header. Nothing in Field interprets it.

AudioAsset

data class AudioAsset(val id: String, @SerialName("byte_size") val byteSize: Long, @SerialName("created_at") val createdAt: String, val representations: List<AssetRepresentation>, val metadata: JsonObject = JsonObject(emptyMap()))

An immutable blob of audio on the service, plus every representation of it.

Declaration	Summary
val id: String	Content address of the bytes: sha256: followed by 64 hex digits.
val byteSize: Long	Size of the underlying bytes, in bytes.
val createdAt: String	ISO-8601 UTC timestamp of the first time these bytes were stored.
val representations: List<AssetRepresentation>	Every materialised view of this asset, oldest first. See AssetRepresentation.role.
val metadata: JsonObject	Free-form provenance recorded at import or generation time; not interpreted by Field.
val durationSeconds: Double?	Duration of the first representation that declares one, in seconds, or null if none does.
val sampleRate: Int?	Sample rate of the first representation that declares one, with the caveat on durationSeconds.
val channels: Int?	Channel count of the first representation that declares one, with the caveat on durationSeconds.

WaveformSummary

data class WaveformSummary(@SerialName("representation_id") val representationId: String, @SerialName("start_seconds") val startSeconds: Double, @SerialName("duration_seconds") val durationSeconds: Double, @SerialName("sample_rate") val sampleRate: Int, val channels: Int, @SerialName("point_count") val pointCount: Int, val minimum: List<List<Float>>, val maximum: List<List<Float>>)

A min/max envelope of one representation over one time window — what a waveform display draws.

Declaration	Summary
val representationId: String	The representation that was summarised.
val startSeconds: Double	Start of the summarised window, in seconds from the start of the representation.
val durationSeconds: Double	Length of the summarised window, in seconds.
val sampleRate: Int	Sample rate of the underlying audio in Hz — the envelope itself is not sampled at this rate.
val channels: Int	Number of channels, and so the outer length of minimum and maximum.
val pointCount: Int	Points per channel. Never more than the window's frame count, so a short window returns fewer.
val minimum: List<List<Float>>	Per-channel minima, indexed [channel](simulation.md#model-kt--point), nominally in −1..1.
val maximum: List<List<Float>>	Per-channel maxima, indexed [channel](simulation.md#model-kt--point), nominally in −1..1.

GestureDelivery

enum class GestureDelivery

How one part of a gesture board reaches the sound, which the board always says out loud.

GestureCurveKind

enum class GestureCurveKind

What a GestureCurve traces over the length of a board. Each is scored on a normalized 0..1 scale.

GesturePoint

data class GesturePoint(@SerialName("time_seconds") val timeSeconds: Double, val value: Double)

One breakpoint of a GestureCurve.

Declaration	Summary
val timeSeconds: Double	When this breakpoint falls, in seconds from the start of the board, and inside its duration.
val value: Double	What the curve is worth there, between zero and one.

GestureCurve

data class GestureCurve(val name: String, val kind: GestureCurveKind, val points: List<GesturePoint>, val weight: Double = 1.0, val delivery: GestureDelivery = GestureDelivery.EVALUATIVE)

A shape asked of one quality of the sound over the length of a board.

Declaration	Summary
val name: String	What to call this curve. Must not be blank, and no two curves on a board may share it.
val kind: GestureCurveKind	Which quality it traces.
val points: List<GesturePoint>	The breakpoints, at least two of them, in strictly increasing time order.
val weight: Double	How much this curve counts against the others when candidates are ranked. In (0, 10].
val delivery: GestureDelivery	Fixed at GestureDelivery.EVALUATIVE in protocol version 1: candidates are measured against the curve, not conditioned on it.

GestureEvent

data class GestureEvent(@SerialName("time_seconds") val timeSeconds: Double, val label: String, val strength: Double = 1.0, val weight: Double = 1.0, val delivery: GestureDelivery = GestureDelivery.EVALUATIVE)

A moment the sound is asked to mark.

Declaration	Summary
val timeSeconds: Double	When the accent is wanted, in seconds from the start of the board, and no later than its duration.
val label: String	What to call this event. Must not be blank.
val strength: Double	How pronounced the accent should be, between zero and one.
val weight: Double	How much this event counts against the others when candidates are ranked. In (0, 10].
val delivery: GestureDelivery	Fixed at GestureDelivery.EVALUATIVE in protocol version 1: a candidate is rewarded for an onset near GestureEvent.timeSeconds.

GesturePromptRegion

data class GesturePromptRegion(@SerialName("start_seconds") val startSeconds: Double, @SerialName("duration_seconds") val durationSeconds: Double, val text: String, val weight: Double = 1.0, val delivery: GestureDelivery = GestureDelivery.COMPILED)

Words asked of one stretch of the board.

Declaration	Summary
val startSeconds: Double	Where the stretch begins, in seconds from the start of the board. Not negative.
val durationSeconds: Double	How long it runs, in seconds. Positive, and it must end within the board's duration.
val text: String	What to ask for over that stretch. Must not be blank.
val weight: Double	How much this region counts against the others. In (0, 10].
val delivery: GestureDelivery	Fixed at GestureDelivery.COMPILED in protocol version 1. The times are written into the global prompt as words, so the model is told about them rather than timed by them.

GestureBoard

data class GestureBoard(@SerialName("duration_seconds") val durationSeconds: Double, @SerialName("base_prompt") val basePrompt: String = "", @SerialName("prompt_regions") val promptRegions: List<GesturePromptRegion> = emptyList(), val events: List<GestureEvent> = emptyList(), val curves: List<GestureCurve> = emptyList(), val metadata: JsonObject = JsonObject(emptyMap()))

Everything asked of one generated sound, with each part's route to the audio attached.

Declaration	Summary
val durationSeconds: Double	How long the generated sound is to be, in seconds. Positive, and every element must fall inside it.
val basePrompt: String	Words that apply throughout, sent to the model as its global text condition.
val promptRegions: List<GesturePromptRegion>	Words that apply to particular stretches, folded into the prompt in time order.
val events: List<GestureEvent>	Moments to mark. They score candidates rather than condition them.
val curves: List<GestureCurve>	Shapes to follow, used the same way. Their names must be unique across the board.
val metadata: JsonObject	Anything else to carry alongside. Field does not read it; it travels with the board.

GestureBinding

data class GestureBinding(val name: String, val kind: String, val delivery: GestureDelivery, val target: String, val explanation: String)

One line of the account a compilation gives of how a board's element reaches the sound.

Declaration	Summary
val name: String	The element this line is about, by its own label, curve name, or base prompt / prompt 1.
val kind: String	Which sort of element it was: text, prompt_region, event, or the curve's kind.
val delivery: GestureDelivery	The route it took.
val target: String	What it reached, such as backend text condition or candidate onset-flux score.
val explanation: String	A sentence saying what that route does, written to be shown to whoever built the board.

GestureCompileRequest

data class GestureCompileRequest(val board: GestureBoard, val operation: String = "generate.style", val parameters: JsonObject = JsonObject(emptyMap()))

A board put forward for compilation, with the operation and parameters it is meant for.

Declaration	Summary
val board: GestureBoard	The board to compile.
val operation: String	Which operation it is destined for.
val parameters: JsonObject	Parameters to merge into the compiled request. A prompt here leads the effective prompt.

GestureCompilation

data class GestureCompilation(val board: GestureBoard, val operation: String, @SerialName("effective_prompt") val effectivePrompt: String, val parameters: JsonObject, val bindings: List<GestureBinding>)

What a board turns into, next to an account of how each part of it got there.

Declaration	Summary
val board: GestureBoard	The board that was compiled, unchanged.
val operation: String	The operation it was compiled for.
val effectivePrompt: String	The prompt the model will actually see: the caller's own prompt, then the base prompt, then each region as from 1.5s to 3s: …, joined with ; then.
val parameters: JsonObject	The parameters as sent, with the effective prompt, the duration and the compilation version added.
val bindings: List<GestureBinding>	One entry per element of the board, saying where it ended up.

GestureEvaluation

data class GestureEvaluation(@SerialName("overall_score") val overallScore: Double, @SerialName("lane_scores") val laneScores: Map<String, Double> = emptyMap(), @SerialName("feature_summary") val featureSummary: Map<String, Double> = emptyMap())

How well one generated candidate answered the board it was asked for.

Declaration	Summary
val overallScore: Double	The single number candidates are ranked by.
val laneScores: Map<String, Double>	A score per scored element, keyed by its event label or curve name.
val featureSummary: Map<String, Double>	The measurements those scores were computed from, keyed by feature name.

ModelOperation

data class ModelOperation(val operation: String, val inputs: List<AssetRegion> = emptyList(), val parameters: JsonObject = JsonObject(emptyMap()), val seeds: List<Long> = emptyList(), @SerialName("candidate_count") val candidateCount: Int = 8, @SerialName("backend_preference") val backendPreference: String? = null, @SerialName("requested_by") val requestedBy: String? = null, @SerialName("gesture_board") val gestureBoard: GestureBoard? = null, @SerialName("force_regenerate") val forceRegenerate: Boolean = false)

A request for the service to do something — the one type you build by hand, or nearly.

Declaration	Summary
val operation: String	The dotted operation name, e.g. generate.style; an unknown one is UNSUPPORTED_OPERATION.
val inputs: List<AssetRegion>	Source windows to work from. Every operation that ships today reads inputs[0] and no more.
val parameters: JsonObject	Operation-specific settings in the service's snake_case keys. The typed constructors fill this.
val seeds: List<Long>	One RNG seed per candidate, or empty to let the service choose.
val candidateCount: Int	How many alternatives to generate, 1..64. Analysis and codec operations use 1.
val backendPreference: String?	Which backend should serve this, e.g. analysis.librosa or audiocraft.encodec_musicgen_style_32khz. Null lets the service resolve one, and the choice it made comes back as JobAccepted.resolvedBackend.
val requestedBy: String?	Free-form attribution, carried through to JobSummary.requestedBy; useful once a job list gets busy.
val gestureBoard: GestureBoard?	The gesture board this request was compiled from, when it came from one.
val forceRegenerate: Boolean	Bypass the result cache and run the model again.

JobState

enum class JobState

Where a job has got to. Most of these are progress narration; four of them are the end.

Declaration	Summary
val terminal: Boolean	True once the job will not change again: COMPLETE, CANCELLED, COMPLETED_BEFORE_CANCELLATION or FAILED.

JobAccepted

data class JobAccepted(@SerialName("job_id") val jobId: String, val state: JobState, @SerialName("request_hash") val requestHash: String, @SerialName("resolved_backend") val resolvedBackend: String, @SerialName("cache_hit") val cacheHit: Boolean = false)

The immediate answer to submitting a ModelOperation: a job exists, and here is how to follow it.

Declaration	Summary
val jobId: String	Id of the job to poll (FieldAudioClient.inspect) or watch (watch): job_ plus a UUID.
val state: JobState	State at acceptance: QUEUED for real work, COMPLETE when cacheHit is true.
val requestHash: String	Hash of the submitted request; identical requests share it, which is how caching happens.
val resolvedBackend: String	The backend the service actually chose, whether or not one was preferred.
val cacheHit: Boolean	True when this request had been served before and no model was run.

StructuredError

data class StructuredError(val code: String, val message: String, val retryable: Boolean = false, val details: JsonObject = JsonObject(emptyMap()))

A failure the service could name, as opposed to an HTTP status.

Declaration	Summary
val code: String	A code from the service's fixed vocabulary — ASSET_NOT_FOUND, INVALID_TIME_RANGE, UNSUPPORTED_OPERATION, MODEL_NOT_AVAILABLE, MODEL_LOAD_FAILED, MODEL_OUT_OF_MEMORY, GENERATION_FAILED, ENCODING_FAILED, DECODING_FAILED, ANALYSIS_FAILED, RESAMPLING_FAILED, JOB_CANCELLED, WORKER_UNAVAILABLE, PROTOCOL_VERSION_MISMATCH, CACHE_CORRUPTION. Typed as a String so an unrecognised future code still decodes.
val message: String	Human-readable detail. Used as-is in the exception message; not machine-parseable.
val retryable: Boolean	The service's own hint that retrying unchanged might work. A job that fails mid-run sets it false.
val details: JsonObject	Extra machine-readable context, e.g. the worker exception type.

JobRecord

data class JobRecord(val id: String, @SerialName("operation_id") val operationId: String, @SerialName("request_hash") val requestHash: String, val state: JobState, @SerialName("resolved_backend") val resolvedBackend: String, val progress: Double, @SerialName("created_at") val createdAt: String, @SerialName("updated_at") val updatedAt: String, val error: StructuredError? = null, @SerialName("candidate_set_id") val candidateSetId: String? = null, @SerialName("codec_result_id") val codecResultId: String? = null, @SerialName("analysis_result_id") val analysisResultId: String? = null, @SerialName("cache_hit") val cacheHit: Boolean = false)

The durable state of one job — the thing polling returns.

Declaration	Summary
val id: String	The job id: job_ plus a UUID.
val operationId: String	The stored operation this job ran, op_ plus a UUID. Several jobs share one on a cache hit.
val requestHash: String	Hash of the request; the cache key.
val state: JobState	Where the job has got to. Check JobState.terminal before reading a result id.
val resolvedBackend: String	The backend that ran, or is running, this job.
val progress: Double	Fraction of the job done, 0..1.
val createdAt: String	ISO-8601 UTC timestamp of when the job was accepted.
val updatedAt: String	ISO-8601 UTC timestamp of the last state or progress change.
val error: StructuredError?	Why the job failed, when it did. Null in every other state.
val candidateSetId: String?	Set when a generate.* job completed: fetch it with FieldAudioClient.candidateSet.
val codecResultId: String?	Set when a codec.* job completed: fetch it with FieldAudioClient.codecResult.
val analysisResultId: String?	Set when an analysis.* job completed: fetch it with FieldAudioClient.analysisResult.
val cacheHit: Boolean	True when this job was answered from the result cache without running a model.

JobSummary

data class JobSummary(val id: String, @SerialName("operation_id") val operationId: String, @SerialName("request_hash") val requestHash: String, val state: JobState, @SerialName("resolved_backend") val resolvedBackend: String, val progress: Double, @SerialName("created_at") val createdAt: String, @SerialName("updated_at") val updatedAt: String, val error: StructuredError? = null, @SerialName("candidate_set_id") val candidateSetId: String? = null, @SerialName("codec_result_id") val codecResultId: String? = null, @SerialName("analysis_result_id") val analysisResultId: String? = null, @SerialName("cache_hit") val cacheHit: Boolean = false, val operation: String, @SerialName("requested_by") val requestedBy: String? = null)

A JobRecord as it appears in the job list, with the two fields that make a list readable: what was asked for, and by whom.

Declaration	Summary
val id: String	The job id: job_ plus a UUID.
val operationId: String	The stored operation this job ran.
val requestHash: String	Hash of the request; the cache key.
val state: JobState	Where the job has got to.
val resolvedBackend: String	The backend that ran, or is running, this job.
val progress: Double	Fraction of the job done, 0..1; see JobRecord.progress.
val createdAt: String	ISO-8601 UTC timestamp of when the job was accepted.
val updatedAt: String	ISO-8601 UTC timestamp of the last state or progress change.
val error: StructuredError?	Why the job failed, when it did.
val candidateSetId: String?	Set when a generate.* job completed.
val codecResultId: String?	Set when a codec.* job completed.
val analysisResultId: String?	Set when an analysis.* job completed.
val cacheHit: Boolean	True when this job was answered from the result cache.
val operation: String	The dotted operation name that was requested — the column a job list is really scanned by.
val requestedBy: String?	Whatever ModelOperation.requestedBy was set to, or null.

Candidate

data class Candidate(val asset: AudioAsset, @SerialName("operation_id") val operationId: String, val index: Int, val seed: Long, @SerialName("native_representation_id") val nativeRepresentationId: String, @SerialName("proxy_representation_id") val proxyRepresentationId: String, @SerialName("splice_preview_representation_id") val splicePreviewRepresentationId: String? = null, @SerialName("native_representation") val nativeRepresentation: AssetRepresentation? = null, @SerialName("proxy_representation") val proxyRepresentation: AssetRepresentation? = null, @SerialName("splice_preview_representation") val splicePreviewRepresentation: AssetRepresentation? = null, val score: Double? = null, @SerialName("gesture_evaluation") val gestureEvaluation: GestureEvaluation? = null, val rating: Double? = null, val selected: Boolean = false, val annotation: String? = null)

One alternative in a generated family, with its provenance and your verdict on it.

Declaration	Summary
val asset: AudioAsset	The model-native output asset, described in full so no second request is needed.
val operationId: String	The stored operation that produced the whole family.
val index: Int	Position in the family, from zero. The candidate board displays it one-based.
val seed: Long	The RNG seed this alternative was generated with — resubmit it to get this take back.
val nativeRepresentationId: String	Representation id of the model's own output, at the model's own sample rate.
val proxyRepresentationId: String	Representation id of the 48 kHz WAV to audition. This is the one to play.
val splicePreviewRepresentationId: String?	Representation id of the source crossfaded into this candidate, when the backend made one.
val nativeRepresentation: AssetRepresentation?	Inline metadata for nativeRepresentationId, when the service inlined it.
val proxyRepresentation: AssetRepresentation?	Inline metadata for proxyRepresentationId, when the service inlined it.
val splicePreviewRepresentation: AssetRepresentation?	Inline metadata for splicePreviewRepresentationId, when there is one.
val score: Double?	A backend-computed score, if that backend computes one. Not comparable across backends.
val gestureEvaluation: GestureEvaluation?	How the gesture board scored this candidate, when one produced it.
val rating: Double?	Your rating, as last written back with FieldAudioClient.judge. The board's slider spans −1..1.
val selected: Boolean	Your selection flag, as last written back.
val annotation: String?	Your note, as last written back.

CandidateSet

data class CandidateSet(val id: String, @SerialName("operation_id") val operationId: String, @SerialName("request_hash") val requestHash: String, val candidates: List<Candidate>, @SerialName("created_at") val createdAt: String)

The whole family a generate.* job produced, in one immutable, re-fetchable value.

Declaration	Summary
val id: String	Id of the set: cset_ plus a UUID.
val operationId: String	The stored operation that produced it.
val requestHash: String	Hash of the request; two identical requests resolve to the same stored set.
val candidates: List<Candidate>	The alternatives, in Candidate.index order.
val createdAt: String	ISO-8601 UTC timestamp of when the set was written.

ReconstructionMetrics

data class ReconstructionMetrics(val rmse: Double, @SerialName("snr_db") val snrDb: Double? = null, @SerialName("spectral_convergence") val spectralConvergence: Double? = null)

How far a codec round trip's output is from its input, by three different measures.

Declaration	Summary
val rmse: Double	Root-mean-square error in normalised sample units; 0 is a perfect reconstruction.
val snrDb: Double?	Signal-to-noise ratio in dB, when the backend reports one. Higher is better.
val spectralConvergence: Double?	Relative spectral distance, when the backend reports one. Lower is better.

CodecReconstructionResult

data class CodecReconstructionResult(val id: String, @SerialName("operation_id") val operationId: String, @SerialName("request_hash") val requestHash: String, val source: AssetRegion, @SerialName("model_input") val modelInput: AssetRepresentation, val tokens: AssetRepresentation, @SerialName("reconstruction_native") val reconstructionNative: AssetRepresentation, @SerialName("reconstruction_proxy") val reconstructionProxy: AssetRepresentation, val metrics: ReconstructionMetrics, val codec: JsonObject, @SerialName("created_at") val createdAt: String)

Everything a codec.reconstruct job produced: what went in, the discrete tokens, what came back out, and how far apart the two ends are.

Declaration	Summary
val id: String	Id of the result: codec_ plus a UUID.
val operationId: String	The stored operation that produced it.
val requestHash: String	Hash of the request.
val source: AssetRegion	The source window that was reconstructed.
val modelInput: AssetRepresentation	The audio the codec actually saw: the window, resampled to the model's rate. Compare against this.
val tokens: AssetRepresentation	The discrete codes, as a C-order integer .npy tensor — read it with NpyIntTensor.
val reconstructionNative: AssetRepresentation	The decoded reconstruction at the model's own sample rate.
val reconstructionProxy: AssetRepresentation	The same reconstruction resampled to 48 kHz for audition.
val metrics: ReconstructionMetrics	How close reconstructionNative is to modelInput.
val codec: JsonObject	The backend's description of the codec itself. CodecVisualization.load reads frame_rate, cardinality and model_name out of here, falling back to EnCodec's 50 Hz / 2048 defaults.
val createdAt: String	ISO-8601 UTC timestamp of when the result was written.

AnalysisResult

data class AnalysisResult(val id: String, @SerialName("operation_id") val operationId: String, @SerialName("request_hash") val requestHash: String, val source: AssetRegion, val data: AssetRepresentation, val artifacts: List<AssetRepresentation> = emptyList(), val summary: JsonObject = JsonObject(emptyMap()), val analysis: JsonObject = JsonObject(emptyMap()), @SerialName("created_at") val createdAt: String)

Everything an analysis.* job produced — as stored assets, not as numbers.

Declaration	Summary
val id: String	Id of the result: analysis_ plus a UUID.
val operationId: String	The stored operation that produced it.
val requestHash: String	Hash of the request.
val source: AssetRegion	The source window that was analysed.
val data: AssetRepresentation	The main product: an .npz of the analysis arrays, with an analysis_… role.
val artifacts: List<AssetRepresentation>	Secondary products — transcribed MIDI or a speech_embedding_messagepack, for example.
val summary: JsonObject	Backend-chosen headline numbers, e.g. an overall tempo estimate. Free-form.
val analysis: JsonObject	The parameters the analysis actually ran with, as the backend resolved them. Free-form.
val createdAt: String	ISO-8601 UTC timestamp of when the result was written.

AnalysisSeries

data class AnalysisSeries(val name: String, val label: String, val unit: String? = null, @SerialName("times_seconds") val timesSeconds: List<Double?>, val values: List<Double?>, val metadata: JsonObject = JsonObject(emptyMap()))

One named curve inside an AnalysisPreview: a time axis, a value axis, and what to call them.

Declaration	Summary
val name: String	Machine name, e.g. onset_strength, frequency_hz, tempo_0_bpm.
val label: String	Human label for a legend, e.g. tempo · 4 s. Falls back to name.
val unit: String?	Unit of values where one applies, e.g. Hz or bpm; null for a dimensionless curve.
val timesSeconds: List<Double?>	Sample times in seconds from the start of the analysed window; null where non-finite.
val values: List<Double?>	The curve itself, parallel to timesSeconds; null where non-finite.
val metadata: JsonObject	Per-series extras, e.g. the scale_seconds a multi-scale tempo track was measured at.

AnalysisNote

data class AnalysisNote(@SerialName("start_seconds") val startSeconds: Double, @SerialName("end_seconds") val endSeconds: Double, val midi: Int, val amplitude: Double, val instrument: String? = null)

One transcribed note from polyphonic analysis: when, how high, how loud.

Declaration	Summary
val startSeconds: Double	Onset, in seconds from the start of the analysed window.
val endSeconds: Double	Offset, in seconds from the start of the analysed window.
val midi: Int	MIDI note number — 60 is middle C.
val amplitude: Double	Note amplitude as the transcriber reported it; the built-in viewer draws it as a 0..1 opacity.
val instrument: String?	MuScriptor instrument group; null for transcribers that do not assign instruments.

AnalysisPreview

data class AnalysisPreview(val operation: String, @SerialName("duration_seconds") val durationSeconds: Double, val series: List<AnalysisSeries>, @SerialName("beat_times_seconds") val beatTimesSeconds: List<Double> = emptyList(), val notes: List<AnalysisNote> = emptyList(), val summary: JsonObject = JsonObject(emptyMap()), val analysis: JsonObject = JsonObject(emptyMap()))

A plot-sized view of an AnalysisResult, built server-side so the browser never sees the .npz.

Declaration	Summary
val operation: String	The dotted operation this preview came from — the discriminator for everything below.
val durationSeconds: Double	Length of the analysed window in seconds; the natural x-axis extent.
val series: List<AnalysisSeries>	The curves, already downsampled. See AnalysisSeries for the null convention.
val beatTimesSeconds: List<Double>	Detected beat positions in seconds, for analysis.beats; empty otherwise. Not downsampled.
val notes: List<AnalysisNote>	Transcribed notes, for analysis.pitch_poly and analysis.transcribe_music; empty otherwise.
val summary: JsonObject	The same headline numbers as AnalysisResult.summary.
val analysis: JsonObject	The same resolved parameters as AnalysisResult.analysis.

CorpusSegment

data class CorpusSegment(val id: String, @SerialName("asset_id") val assetId: String, @SerialName("start_seconds") val startSeconds: Double, @SerialName("duration_seconds") val durationSeconds: Double, @SerialName("scale_seconds") val scaleSeconds: Double, @SerialName("previous_segment_id") val previousSegmentId: String? = null, @SerialName("next_segment_id") val nextSegmentId: String? = null, @SerialName("descriptor_representation_id") val descriptorRepresentationId: String? = null, val metadata: JsonObject = JsonObject(emptyMap()))

One immutable, source-adjacent window in the indexed audio corpus.

Declaration	Summary
val id: String	The index's own identifier for this window, which previousSegmentId and nextSegmentId refer to.
val assetId: String	The recording the window was cut from.
val startSeconds: Double	Where the window starts in that recording, in seconds.
val durationSeconds: Double	How long the window is, in seconds.
val scaleSeconds: Double	The window length this part of the index was built at. One recording is usually indexed at several scales at once, and a window is only ever compared with others of its own.
val previousSegmentId: String?	The window before this one in the same recording at the same scale, or null at the start.
val nextSegmentId: String?	The window after it, or null at the end. Following these is how a splice keeps going.
val descriptorRepresentationId: String?	The representation holding this window's descriptor vector, where the index kept one.
val metadata: JsonObject	Anything else the indexer recorded about the window.

CorpusNeighbour

data class CorpusNeighbour(val segment: CorpusSegment, val distance: Double, @SerialName("group_distances") val groupDistances: Map<String, Double> = emptyMap(), val audio: AssetRepresentation, @SerialName("splice_preview") val splicePreview: AssetRepresentation? = null)

A retrieved corpus window together with its grouped distance and playable representations.

Declaration	Summary
val segment: CorpusSegment	The window that was found.
val distance: Double	Its weighted distance from the query. Smaller is closer, and the results arrive in this order.
val groupDistances: Map<String, Double>	The distance within each descriptor group on its own, keyed harmony, pitch, texture, rhythm and dynamics. These say what the window was matched on, which a single number cannot: two neighbours at the same distance can agree on rhythm and disagree on harmony.
val audio: AssetRepresentation	The window's own audio, which is what makes a result auditionable rather than only rankable.
val splicePreview: AssetRepresentation?	The query with this window spliced into it, where the service rendered one.

CorpusPreview

data class CorpusPreview(val operation: String, val query: AssetRegion, @SerialName("scale_seconds") val scaleSeconds: Double? = null, @SerialName("indexed_segment_count") val indexedSegmentCount: Int = 0, val segments: List<CorpusSegment> = emptyList(), val neighbours: List<CorpusNeighbour> = emptyList(), val weights: Map<String, Double> = emptyMap(), val summary: JsonObject = JsonObject(emptyMap()))

Bounded metadata for a corpus index or nearest-neighbour result.

Declaration	Summary
val operation: String	Which corpus operation this came from, and so which list below is filled: analysis.embed populates segments, analysis.find_neighbours populates neighbours.
val query: AssetRegion	The window that was indexed or searched with.
val scaleSeconds: Double?	The scale the operation worked at, where it worked at a single one.
val indexedSegmentCount: Int	How many windows the index holds, which is the pool a retrieval searched.
val segments: List<CorpusSegment>	The windows an indexing run produced. Empty for a retrieval.
val neighbours: List<CorpusNeighbour>	What a retrieval found, nearest first. Empty for an indexing run.
val weights: Map<String, Double>	The descriptor-group weights the distances were measured with, keyed as in CorpusNeighbour.groupDistances.
val summary: JsonObject	Headline numbers for the run, for showing above the results.

ProgressEvent

data class ProgressEvent(val sequence: Long, @SerialName("job_id") val jobId: String, val state: JobState, val progress: Double, val message: String? = null, @SerialName("candidate_index") val candidateIndex: Int? = null, val at: String)

One frame of a job's live narration, delivered over the WebSocket FieldAudioClient.watch opens.

Declaration	Summary
val sequence: Long	Monotonically increasing within one job; use it to order or de-duplicate replayed events.
val jobId: String	The job this event belongs to.
val state: JobState	The state the job entered.
val progress: Double	Fraction done, 0..1; see JobRecord.progress.
val message: String?	A short human-readable note, e.g. "making 48 kHz audition proxy". Null when there is nothing to say.
val candidateIndex: Int?	Which candidate this event is about, for a family generated one at a time.
val at: String	ISO-8601 UTC timestamp of the event.

ModelCapability

data class ModelCapability(@SerialName("model_id") val modelId: String, val revision: String, val operations: List<String>, @SerialName("native_sample_rates") val nativeSampleRates: List<Int>, val channels: List<String>, @SerialName("max_duration_seconds") val maxDurationSeconds: Double? = null, @SerialName("supports_seed") val supportsSeed: Boolean, @SerialName("supports_cancellation") val supportsCancellation: Boolean, val conditioning: List<String>, val available: Boolean, @SerialName("unavailable_reason") val unavailableReason: String? = null)

What one model can do, as reported by FieldAudioClient.models — the authoritative answer to "can I ask for this?" before submitting anything.

Declaration	Summary
val modelId: String	Backend id: the same string ModelOperation.backendPreference takes.
val revision: String	Checkpoint revision, recorded into the provenance of everything this model produces.
val operations: List<String>	The dotted operations this backend serves.
val nativeSampleRates: List<Int>	Sample rates the model works at natively, in Hz. Output is resampled to 48 kHz for audition.
val channels: List<String>	Channel layouts supported, each mono or stereo.
val maxDurationSeconds: Double?	Longest output the model will produce, in seconds; null when unbounded or unknown.
val supportsSeed: Boolean	Whether ModelOperation.seeds does anything here. If false, results are not reproducible.
val supportsCancellation: Boolean	Whether a cancel request can actually stop it, or will end as CANCEL_UNSUPPORTED.
val conditioning: List<String>	The conditioning inputs this model accepts, e.g. a text prompt or a style excerpt.
val available: Boolean	False when the model is known but cannot run right now.
val unavailableReason: String?	Why available is false — a missing weight file, an unavailable worker environment.

CandidateJudgement

data class CandidateJudgement(val rating: Double? = null, val selected: Boolean = false, val annotation: String? = null)

Your verdict on one candidate, written back with FieldAudioClient.judge. Every field is optional.

Declaration	Summary
val rating: Double?	How good it is. The candidate board's slider spans −1..1 in steps of 0.1; the service stores what it is sent.
val selected: Boolean	Whether this take is the keeper. Nothing enforces one selection per set.
val annotation: String?	A free-text note.

SavedAudioOperation

data class SavedAudioOperation(val request: ModelOperation, val accepted: JobAccepted, val job: JobRecord? = null, @SerialName("candidate_set") val candidateSet: CandidateSet? = null, @SerialName("codec_result") val codecResult: CodecReconstructionResult? = null, @SerialName("analysis_result") val analysisResult: AnalysisResult? = null, val judgements: Map<Int, CandidateJudgement> = emptyMap())

One operation and everything that came of it, as saved into a Field document.

Declaration	Summary
val request: ModelOperation	The request as submitted — enough to resubmit it exactly.
val accepted: JobAccepted	What the service said when it accepted it.
val job: JobRecord?	The last job state seen, if one was recorded before the document was saved.
val candidateSet: CandidateSet?	The candidate family, for a generate.* operation that finished.
val codecResult: CodecReconstructionResult?	The codec result, for a codec.* operation that finished.
val analysisResult: AnalysisResult?	The analysis result, for an analysis.* operation that finished.
val judgements: Map<Int, CandidateJudgement>	Ratings and notes by Candidate.index, as last saved.

FieldAudioProjectState

data class FieldAudioProjectState(val version: Int = 1, val assets: Map<String, AudioAsset> = emptyMap(), val operations: List<SavedAudioOperation> = emptyList())

The whole Field-side audio project: which assets it refers to, and what was done with them.

Declaration	Summary
val version: Int	Format version of this saved state. Bump it when the shape changes incompatibly.
val assets: Map<String, AudioAsset>	Assets the project refers to, keyed by AudioAsset.id.
val operations: List<SavedAudioOperation>	Operations in the order they were performed.

RenderedAudioAnalysis.kt

The render-to-analysis pipeline: render Field synthesis to a WAV, upload it once, and fan out durable analysis jobs.

RenderedAudioFile is a WAV produced by Field code, tagged with a generator name and a provenance map that identify what produced the bytes. Six render functions cover the synthesis systems: renderPatWav (Pat rhythmic patterns), renderDirtWav (Pat through the Dirt sample library), renderDoughWav and renderDoughStereoWav (Dough signal graphs), renderFaustWav and renderFaustWavAsync (Faust programs), and recordSonicPiWav (real-time Sonic Pi capture).

analyzeRenderedAudio uploads the WAV once, submits a fan-out of analysis jobs (beats, monophonic pitch, and polyphonic pitch by default), watches progress over WebSocket, and completes with a RenderedAudioAnalysis. The fan-out is all-or-nothing: a failed leg abandons the others rather than returning a partial result. RenderedAudioAnalysis.visualizationHtml renders the waveform and all analysis lanes on a shared, bidirectional time cursor.

val wav = renderPatWav(tmp, pattern, cycles = 4, cps = 1.5)
val analysis = client.analyzeRenderedAudio(wav).get()
display(analysis.visualizationHtml())

src/jvmMain/kotlin/afield/audio/ai/RenderedAudioAnalysis.kt — read the source

Used in: Render Field music, then analyse it, Visualizers, Displays, and Little Instruments

Top-level functions

Declaration	Summary
fun standardAudioAnalyses(region: AssetRegion, requestedBy: String? = null, includeBeats: Boolean = true, includeMonophonicPitch: Boolean = true, includePolyphonicPitch: Boolean = true, forceRegenerate: Boolean = false): List<NamedAudioAnalysis>	The standard fan-out used by the transcript; callers can freely supply a different list.
fun FieldAudioClient.analyzeRenderedAudio(render: RenderedAudioFile, startSeconds: Double = 0.0, durationSeconds: Double? = null, requests: (AssetRegion) -> List<NamedAudioAnalysis> = { standardAudioAnalyses(it) }, onAccepted: (NamedAudioAnalysis, JobAccepted) -> Unit = { _, _ -> }, onProgress: (NamedAudioAnalysis, JobAccepted, ProgressEvent) -> Unit = { _, _, _ -> }, onProgressError: (NamedAudioAnalysis, Throwable) -> Unit = { _, _ -> }): CompletableFuture<RenderedAudioAnalysis>	Import render, submit a fan-out of analysis jobs, watch them, and complete with all results.
fun renderPatWav(path: Path, pattern: PatPattern<String>, cycles: Int, cps: Double = 1.0, voice: PatVoiceSpec = patVoice(), sampleRate: Int = PatSoundDocs.SR, normalizePeak: Float? = null, provenance: Map<String, String> = emptyMap()): RenderedAudioFile	Offline-render a Pat pattern to an immutable-analysis-ready WAV.
fun renderDirtWav(path: Path, pattern: PatPattern<String>, cycles: Int, cps: Double = 1.0, library: DirtSampleLibrary? = DirtRender.libraryOrNull(), voice: PatVoiceSpec = patVoice(), sampleRate: Int = PatSoundDocs.SR, normalizePeak: Float? = null, provenance: Map<String, String> = emptyMap()): RenderedAudioFile	Offline-render Pat through the Dirt sample library, falling back exactly as Pat normally does.
fun renderDoughWav(path: Path, signal: Signal, seconds: Double, sampleRate: Int = 48_000, normalizePeak: Float? = null, provenance: Map<String, String> = emptyMap()): RenderedAudioFile	Offline-render a mono Dough Signal to WAV.
fun renderDoughStereoWav(path: Path, node: StereoRenderable, seconds: Double, sampleRate: Int = 48_000, normalizePeak: Float? = null, provenance: Map<String, String> = emptyMap()): RenderedAudioFile	Offline-render a stereo Dough node without folding away its spatial image.
fun renderFaustWav(path: Path, program: FaustProgram, seconds: Double, sampleRate: Int = 48_000, normalizePeak: Float? = null, provenance: Map<String, String> = emptyMap()): RenderedAudioFile	Render an already-compiled Faust program to a stereo WAV.
fun renderFaustWavAsync(path: Path, name: String, source: String, seconds: Double, sampleRate: Int = 48_000, compilerArguments: String = FaustEngine.DEFAULT_ARGS, normalizePeak: Float? = null, provenance: Map<String, String> = emptyMap()): CompletableFuture<RenderedAudioFile>	Compile and render Faust away from Field's run loop (the first compiler warm-up can be slow).
fun recordSonicPiWav(sonicPi: SonicPi, path: Path, source: String, seconds: Double, workspace: String = "field_audio_analysis", startSettleMillis: Long = 300, tailMillis: Long = 900, stopSettleMillis: Long = 500, saveTimeoutMillis: Long = 30_000, provenance: Map<String, String> = emptyMap()): CompletableFuture<RenderedAudioFile>	Record a Sonic Pi program in real time, then return the same artifact the offline renderers do.

RenderedAudioFile

data class RenderedAudioFile(val path: Path, val generator: String, val sampleRate: Int? = null, val channels: Int? = null, val durationSeconds: Double? = null, val provenance: Map<String, String> = emptyMap())

A finite sound made by Field code and written to a real WAV file.

Declaration	Summary
val path: Path	The WAV on disk. It must exist, end in .wav, and be longer than the 44 bytes of a bare header, all checked when this value is made rather than when the upload starts.
val generator: String	Which Field system wrote the bytes. The render functions here use Pat, Dirt, Dough, Faust and Sonic Pi; anything non-blank is accepted.
val sampleRate: Int?	Frames per second, positive, or null where it is not known. RenderedAudioFile.existing reads it from the header.
val channels: Int?	Channel count, positive, or null where it is not known. RenderedAudioFile.existing reads it from the header.
val durationSeconds: Double?	Length in seconds, positive, or null where it is not known. RenderedAudioFile.existing reads it from the header.
val provenance: Map<String, String>	Render parameters to carry beside the bytes, for a box drawing a provenance line. Each render function adds keys of its own first, such as cycles and cps for Pat, so a key given here replaces the function's.

NamedAudioAnalysis

data class NamedAudioAnalysis(val label: String, val operation: ModelOperation)

One named analysis request, retained beside its accepted job and final result.

Declaration	Summary
val label: String	Short name for this leg. standardAudioAnalyses uses beats, monophonic pitch and polyphonic pitch. It reaches CompletedAudioAnalysis and the progress callbacks so a caller can tell the legs apart; RenderedAudioAnalysis.visualizationHtml is handed only the results, and names its lanes from those.
val operation: ModelOperation	The request this leg submits.

CompletedAudioAnalysis

data class CompletedAudioAnalysis(val label: String, val accepted: JobAccepted, val result: AnalysisResult)

A completed leg of a rendered-audio analysis fan-out.

Declaration	Summary
val label: String	The NamedAudioAnalysis.label this leg was requested under.
val accepted: JobAccepted	What the service said when it took the job, including the job id progress was watched on.
val result: AnalysisResult	The durable result the job finished with.

RenderedAudioAnalysis

data class RenderedAudioAnalysis(val render: RenderedAudioFile, val asset: AudioAsset, val sourceRepresentation: AssetRepresentation, val region: AssetRegion, val analyses: List<CompletedAudioAnalysis>)

The complete round trip: local render, immutable service asset, exact analysed window, jobs and durable results. This is suitable for storing on a Field node without making Python authoritative.

Declaration	Summary
val render: RenderedAudioFile	The local WAV that was uploaded, with the generator name and provenance it was made with.
val asset: AudioAsset	The immutable service asset those bytes became.
val sourceRepresentation: AssetRepresentation	The asset's WAV representation the analyses ran against: its canonical_project WAV where there is one, otherwise the first WAV it has.
val region: AssetRegion	The window actually analysed. Its duration is clamped to what is left of the render after AssetRegion.startSeconds, so it can be shorter than the one asked for.
val analyses: List<CompletedAudioAnalysis>	One entry per leg of the fan-out, in the order the requests were listed.
fun visualizationHtml(serviceBaseUrl: String = "/field-audio", width: Int = 1100, waveformHeight: Int = 150, panelHeight: Int = 250, points: Int = 1600, syncGroup: String = "render-analysis-${asset.id.removePrefix("sha256:").take(16)}"): String	Source waveform and every analysis lane on one shared, bidirectional time cursor.

SpectralEmbeddingProjection.kt

Spectral embedding: decode, pool, and project the short-time spectral frames stored in an analysis artifact.

A spectral analysis job writes a binary MessagePack artifact. decodeSpectralEmbeddingMessagePack reads it into a SpectralEmbeddingFrames value containing mel distributions, log-mel spectrograms, and MFCC vectors, optionally bounded to a time window, temporally pooled, and sampled to a target frame count. Call .mds() or .tsne() on the result to get a SpectralEmbeddingProjection with exact frame times, then pass it to withSpectralProjection to attach the axes to an AnalysisPreview as named time series.

Five distance metrics are available via SpectralEmbeddingDistance. The default, mel Wasserstein, treats each frame as a probability distribution over perceptual mel-bin positions and computes earth-mover distance. The other four apply cosine or Euclidean distance to log-mel or MFCC vectors directly. Both projection methods delegate to Smile via the smileMds and smileTsne families.

Temporal pooling and hop length are independent: overlapping perceptual windows can still sample at display rate. Sampling to maxFrames happens after pooling and retains time order. Example:

val frames = client.spectralEmbeddingFrames(result).get()
val projection = frames.mds()
val preview = preview.withSpectralProjection(projection)

src/jvmMain/kotlin/afield/audio/ai/SpectralEmbeddingProjection.kt — read the source

Used in: Short-time spectral spaces, Visualizers, Displays, and Little Instruments

Top-level functions

Declaration	Summary
fun AnalysisResult.spectralEmbeddingMessagePack(): AssetRepresentation	Locate the dense spectral artifact without confusing it with the plot-sized NPZ preview.
fun decodeSpectralEmbeddingMessagePack(bytes: ByteArray, maxFrames: Int = 600, startSeconds: Double = 0.0, durationSeconds: Double? = null, poolMillis: Double? = null, hopMillis: Double? = null): SpectralEmbeddingFrames	Decode, temporally pool, and finally sample spectral frames before quadratic projection.
fun melWassersteinDistance(a: FloatArray, b: FloatArray): Double	Normalized 1-D Wasserstein distance on a shared, uniformly-spaced mel-bin support.
fun SpectralEmbeddingFrames.mds(dimensions: Int = 4, distance: SpectralEmbeddingDistance = SpectralEmbeddingDistance.MEL_WASSERSTEIN, positiveCorrection: Boolean = true): SpectralEmbeddingProjection	Project the chosen spectral metric with classical MDS.
fun SpectralEmbeddingFrames.tsne(dimensions: Int = 4, distance: SpectralEmbeddingDistance = SpectralEmbeddingDistance.MEL_WASSERSTEIN, perplexity: Double = 30.0, learningRate: Double = 200.0, earlyExaggeration: Double = 12.0, maxIterations: Int = 750, seed: Long = 184_921L): SpectralEmbeddingProjection	Project the chosen spectral metric with fixed-seed exact t-SNE.
fun AnalysisPreview.withSpectralProjection(projection: SpectralEmbeddingProjection): AnalysisPreview	Attach locally computed projection axes to the service's lightweight spectral curves.
fun FieldAudioClient.spectralEmbeddingFrames(result: AnalysisResult, maxFrames: Int = 600, startSeconds: Double = 0.0, durationSeconds: Double? = null, poolMillis: Double? = null, hopMillis: Double? = null): CompletableFuture<SpectralEmbeddingFrames>	Download a finished spectral job's artifact and decode it, with the same windowing, pooling and sampling arguments decodeSpectralEmbeddingMessagePack takes.
fun FieldAudioClient.spectralEmbeddingSmilePreview(result: AnalysisResult, maxFrames: Int = 600, startSeconds: Double = 0.0, durationSeconds: Double? = null, dimensions: Int = 4, distance: SpectralEmbeddingDistance = SpectralEmbeddingDistance.MEL_WASSERSTEIN, positiveCorrection: Boolean = true, previewPoints: Int = 1800, poolMillis: Double? = null, hopMillis: Double? = null): CompletableFuture<AnalysisPreview>	The service's lightweight preview curves with locally computed MDS axes attached, in one call.
fun FieldAudioClient.spectralEmbeddingSmileTsnePreview(result: AnalysisResult, maxFrames: Int = 600, startSeconds: Double = 0.0, durationSeconds: Double? = 10.0, dimensions: Int = 4, distance: SpectralEmbeddingDistance = SpectralEmbeddingDistance.MEL_WASSERSTEIN, perplexity: Double = 30.0, learningRate: Double = 200.0, earlyExaggeration: Double = 12.0, maxIterations: Int = 750, seed: Long = 184_921L, previewPoints: Int = 1800, poolMillis: Double? = null, hopMillis: Double? = null): CompletableFuture<AnalysisPreview>	The same preview with t-SNE axes instead, arriving as tsne_0, tsne_1, … series.

SpectralEmbeddingDistance

enum class SpectralEmbeddingDistance(val label: String)

The interchangeable short-time representations and metrics retained by one spectral job.

Constants: MEL_WASSERSTEIN, LOG_MEL_COSINE, LOG_MEL_EUCLIDEAN, MFCC_COSINE, MFCC_EUCLIDEAN.

SpectralEmbeddingFrames

data class SpectralEmbeddingFrames(val timesSeconds: DoubleArray, val melDistributions: Array<FloatArray>, val logMelDb: Array<FloatArray>, val mfcc: Array<FloatArray>, val melFrequenciesHz: FloatArray, val sampledFrameIndices: IntArray, val sourceFrameCount: Int, val pooledFrameCount: Int, val frameRateHz: Double, val poolMillis: Double, val hopMillis: Double, val backend: String, val libraryVersion: String)

A bounded and optionally pooled range decoded from a spectral MessagePack artifact.

Declaration	Summary
val timesSeconds: DoubleArray	When each retained frame sits in the recording, in seconds, read from the artifact's own clock.
val melDistributions: Array<FloatArray>	One mel distribution per retained frame, each renormalized to sum to one so it can be read as a distribution over mel-bin position. Columns line up with melFrequenciesHz.
val logMelDb: Array<FloatArray>	One log-mel spectrum per retained frame, in decibels, pooled by mean. Columns line up with melFrequenciesHz.
val mfcc: Array<FloatArray>	One MFCC vector per retained frame, pooled by mean.
val melFrequenciesHz: FloatArray	The frequency in hertz the artifact gives for each mel bin, one per column of melDistributions.
val sampledFrameIndices: IntArray	For each retained frame, its index in the artifact's own frame numbering.
val sourceFrameCount: Int	Frames in the whole artifact, before any time window was applied.
val pooledFrameCount: Int	Pooled rows the chosen window produced, before sampling down to maxFrames. The retained frame count is this or maxFrames, whichever is smaller.
val frameRateHz: Double	The artifact's native frame rate, which sets the smallest pool and hop that can be asked for.
val poolMillis: Double	The pooling window actually used, in milliseconds. A request is rounded to a whole number of native frames and held to at least one, so this can differ from what was asked for.
val hopMillis: Double	The hop actually used, in milliseconds, rounded the same way as SpectralEmbeddingFrames.poolMillis.
val backend: String	Which analysis backend wrote the artifact.
val libraryVersion: String	The version of that backend's own library, as the artifact records it.

SpectralEmbeddingProjection

data class SpectralEmbeddingProjection(val timesSeconds: DoubleArray, val coordinates: Array<DoubleArray>, val sampledFrameIndices: IntArray, val sourceFrameCount: Int, val pooledFrameCount: Int, val poolMillis: Double, val hopMillis: Double, val backend: String, val libraryVersion: String, val distance: SpectralEmbeddingDistance, val method: String, val scores: DoubleArray = doubleArrayOf(), val proportion: DoubleArray = doubleArrayOf(), val perplexity: Double? = null, val seed: Long? = null, val cost: Double? = null)

A Field-owned spectral projection with exact frame times for synchronized audition.

Declaration	Summary
val timesSeconds: DoubleArray	When each projected frame sits in the recording, in seconds, carried over from the frames it came from.
val coordinates: Array<DoubleArray>	One point per projected frame, each as long as the number of dimensions asked for.
val sampledFrameIndices: IntArray	For each projected frame, its index in the artifact's own frame numbering.
val sourceFrameCount: Int	SpectralEmbeddingFrames.sourceFrameCount of the frames this was projected from.
val pooledFrameCount: Int	SpectralEmbeddingFrames.pooledFrameCount of the frames this was projected from.
val poolMillis: Double	The pooling window those frames were built with, in milliseconds.
val hopMillis: Double	The hop those frames were built with, in milliseconds.
val backend: String	Which analysis backend wrote the artifact underneath.
val libraryVersion: String	The version of that backend's own library.
val distance: SpectralEmbeddingDistance	Which of the five metrics the distances were measured with.
val method: String	MDS or t-SNE. withSpectralProjection names its series from this.
val scores: DoubleArray	The MDS eigenvalue scores. Empty for a t-SNE projection.
val proportion: DoubleArray	The share of the total each MDS dimension retains. Empty for a t-SNE projection.
val perplexity: Double?	The perplexity t-SNE ran with. Null for an MDS projection.
val seed: Long?	The seed t-SNE ran with, which is fixed by default so a projection repeats. Null for MDS.
val cost: Double?	The divergence t-SNE finished at, lower being a closer fit. Null for an MDS projection.

SpeechEmbeddingMds.kt

Speech embedding: decode, pool, and project the learned speech frames stored in an analysis artifact.

speechEmbeddingAnalysis submits the analysis.embed_speech job, and the worker keeps two things: the full float16 frame tensor, and its own four-axis MDS preview over cosine distance. This file is the second route to a picture. decodeSpeechEmbeddingMessagePack reads the tensor into a SpeechEmbeddingFrames bounded to a time window, then .mds() or .tsne() fits the projection in Field through Smile's smileMds and smileTsne. Changing the range, the pooling, the distance, the dimension count, the perplexity or the seed re-fits locally and never re-runs the speech model.

The payload stores float64 times and float16 vectors as little-endian MessagePack binary fields, and a private half-float expansion reads them, so nothing here needs JVM float16 support. The decoder rejects any format other than field.audio.speech-embedding.v1, and any dtype pair other than those two.

Pooling and hop are independent. poolMillis is the width of a centred mean window and hopMillis is the spacing between the vectors that come out of it. Both round to the frame rate recorded in the payload, and omitting them preserves the model's own frames one for one. Pooling runs as a rolling sum, so a window covering many source rows costs one pass over the range instead of one pass per output row, and maxFrames sampling happens after pooling and keeps time order.

client.speechEmbeddingSmileTsnePreview(result).thenAccept { preview ->
    display("speech space", speechEmbeddingPreviewVisualizationHtml(preview))
}

withSpeechMds and withSpeechTsne replace a preview's projection axes with the locally fitted ones and stamp the owner, library and method into both the series metadata and the preview's analysis object, which is how a Field-fitted view is told apart from the worker's cached one on the page. Both require at least four dimensions, because speechEmbeddingPreviewVisualizationHtml draws axes 0/1 and 2/3 as two phase portraits.

Every projection result carries the frame times and source frame indices it was fitted from alongside the backend, model id, model revision and WavLM layer, so a trajectory on screen can be traced back to the rows and the checkpoint behind it. The coordinates themselves are navigational: axis numbering is specific to one fit, and comparing coordinate 0 between two backends or two independently fitted recordings has no meaning. Use the stored vectors for that. See Speech-reactive learned spaces.

src/jvmMain/kotlin/afield/audio/ai/SpeechEmbeddingMds.kt — read the source

Top-level functions

Declaration	Summary
fun AnalysisResult.speechEmbeddingMessagePack(): AssetRepresentation	Locate the binary frame-vector artifact emitted by the speech workers.
fun decodeSpeechEmbeddingMessagePack(bytes: ByteArray, maxFrames: Int = 384, startSeconds: Double = 0.0, durationSeconds: Double? = null, poolMillis: Double? = null, hopMillis: Double? = null): SpeechEmbeddingFrames	Decode a MessagePack speech tensor, retaining at most maxFrames pooled rows from a time range.
fun SpeechEmbeddingFrames.mds(dimensions: Int = 4, distance: SmileMdsDistance = SmileMdsDistance.COSINE, positiveCorrection: Boolean = true): SpeechEmbeddingMds	Run Field's Smile MDS over this bounded speech-frame sample.
fun SpeechEmbeddingFrames.tsne(dimensions: Int = 4, distance: SmileMdsDistance = SmileMdsDistance.COSINE, perplexity: Double = 30.0, learningRate: Double = 200.0, earlyExaggeration: Double = 12.0, maxIterations: Int = 750, seed: Long = 184_921L): SpeechEmbeddingTsne	Run a fixed-seed, local-neighbourhood t-SNE over this speech-frame range.
fun AnalysisPreview.withSpeechMds(projection: SpeechEmbeddingMds): AnalysisPreview	Replace a worker preview's MDS axes with coordinates computed locally in Field by Smile.
fun AnalysisPreview.withSpeechTsne(projection: SpeechEmbeddingTsne): AnalysisPreview	Replace a worker preview's MDS axes with a Field-owned Smile t-SNE projection.
fun FieldAudioClient.speechEmbeddingFrames(result: AnalysisResult, maxFrames: Int = 384, startSeconds: Double = 0.0, durationSeconds: Double? = null, poolMillis: Double? = null, hopMillis: Double? = null): CompletableFuture<SpeechEmbeddingFrames>	Download and decode the binary embedding artifact without blocking Field's runloop.
fun FieldAudioClient.speechEmbeddingSmilePreview(result: AnalysisResult, maxFrames: Int = 384, startSeconds: Double = 0.0, durationSeconds: Double? = null, dimensions: Int = 4, distance: SmileMdsDistance = SmileMdsDistance.COSINE, positiveCorrection: Boolean = true, previewPoints: Int = 1800, poolMillis: Double? = null, hopMillis: Double? = null): CompletableFuture<AnalysisPreview>	Fetch the lightweight analysis curves, project the binary frame vectors with Smile in Field, and return one preview ready for speechEmbeddingPreviewVisualizationHtml.
fun FieldAudioClient.speechEmbeddingSmileTsnePreview(result: AnalysisResult, maxFrames: Int = 600, startSeconds: Double = 0.0, durationSeconds: Double? = 10.0, dimensions: Int = 4, distance: SmileMdsDistance = SmileMdsDistance.COSINE, perplexity: Double = 30.0, learningRate: Double = 200.0, earlyExaggeration: Double = 12.0, maxIterations: Int = 750, seed: Long = 184_921L, previewPoints: Int = 1800, poolMillis: Double? = null, hopMillis: Double? = null): CompletableFuture<AnalysisPreview>	Fetch lightweight analysis curves and attach a fixed-seed Smile t-SNE computed in Field.

SpeechEmbeddingFrames

data class SpeechEmbeddingFrames(val timesSeconds: DoubleArray, val vectors: Array<FloatArray>, val sampledFrameIndices: IntArray, val sourceFrameCount: Int, val pooledFrameCount: Int, val sourceDimensions: Int, val frameRateHz: Double, val poolMillis: Double, val hopMillis: Double, val backend: String, val modelId: String, val modelRevision: String, val wavlmLayer: Int?)

A bounded, time-preserving frame range decoded from a speech embedding MessagePack artifact.

Declaration	Summary
val timesSeconds: DoubleArray	When each retained frame sits in the recording, in seconds, read from the payload's own clock.
val vectors: Array<FloatArray>	One model vector per retained frame, each sourceDimensions long, pooled by centred mean.
val sampledFrameIndices: IntArray	For each retained frame, its index in the payload's own frame numbering.
val sourceFrameCount: Int	Frames in the whole payload, before any time range was applied.
val pooledFrameCount: Int	Pooled rows the chosen range produced, before sampling down to maxFrames. The retained frame count is this or maxFrames, whichever is smaller.
val sourceDimensions: Int	The width of the model's own vectors, and so of every row of vectors.
val frameRateHz: Double	The model's native frame rate, which sets the smallest pool and hop that can be asked for.
val poolMillis: Double	The pooling window actually used, in milliseconds. A request is rounded to a whole number of native frames and held to at least one, so this can differ from what was asked for.
val hopMillis: Double	The hop actually used, in milliseconds, rounded the same way as SpeechEmbeddingFrames.poolMillis.
val backend: String	Which speech worker produced the payload.
val modelId: String	The model it ran.
val modelRevision: String	The revision of that model, so a trajectory can be traced back to one checkpoint.
val wavlmLayer: Int?	Which WavLM layer the vectors were read from, or null where the backend recorded none.

SpeechEmbeddingMds

data class SpeechEmbeddingMds(val timesSeconds: DoubleArray, val coordinates: Array<DoubleArray>, val sampledFrameIndices: IntArray, val sourceFrameCount: Int, val pooledFrameCount: Int, val poolMillis: Double, val hopMillis: Double, val backend: String, val modelId: String, val modelRevision: String, val wavlmLayer: Int?, val distance: SmileMdsDistance, val scores: DoubleArray, val proportion: DoubleArray)

Smile MDS coordinates attached to the exact speech-frame times that were projected.

Declaration	Summary
val timesSeconds: DoubleArray	When each projected frame sits in the recording, in seconds, carried over from the frames it was fitted from.
val coordinates: Array<DoubleArray>	One point per projected frame, as long as the number of dimensions asked for. These axes are for navigating a single fit: the numbering belongs to this fit alone, so coordinate 0 of two separately fitted recordings are not comparable. Compare SpeechEmbeddingFrames.vectors instead.
val sampledFrameIndices: IntArray	For each projected frame, its index in the payload's own frame numbering.
val sourceFrameCount: Int	SpeechEmbeddingFrames.sourceFrameCount of the frames this was fitted from.
val pooledFrameCount: Int	SpeechEmbeddingFrames.pooledFrameCount of the frames this was fitted from.
val poolMillis: Double	The pooling window those frames were built with, in milliseconds.
val hopMillis: Double	The hop those frames were built with, in milliseconds.
val backend: String	Which speech worker produced the payload underneath.
val modelId: String	The model it ran.
val modelRevision: String	The revision of that model.
val wavlmLayer: Int?	Which WavLM layer the vectors were read from, or null where the backend recorded none.
val distance: SmileMdsDistance	The metric the distances were measured with.
val scores: DoubleArray	The eigenvalue scores of the fit, one per dimension.
val proportion: DoubleArray	The share of the total each dimension retains, one per dimension.

SpeechEmbeddingTsne

data class SpeechEmbeddingTsne(val timesSeconds: DoubleArray, val coordinates: Array<DoubleArray>, val sampledFrameIndices: IntArray, val sourceFrameCount: Int, val pooledFrameCount: Int, val poolMillis: Double, val hopMillis: Double, val backend: String, val modelId: String, val modelRevision: String, val wavlmLayer: Int?, val distance: SmileMdsDistance, val perplexity: Double, val seed: Long, val cost: Double)

Smile t-SNE coordinates attached to the exact speech-frame times that were projected.

Declaration	Summary
val timesSeconds: DoubleArray	When each projected frame sits in the recording, in seconds, carried over from the frames it was fitted from.
val coordinates: Array<DoubleArray>	One point per projected frame, as long as the number of dimensions asked for. As with MDS these axes are for navigating a single fit, and t-SNE adds its own caution: the distances between well-separated clusters carry much less than the grouping does.
val sampledFrameIndices: IntArray	For each projected frame, its index in the payload's own frame numbering.
val sourceFrameCount: Int	SpeechEmbeddingFrames.sourceFrameCount of the frames this was fitted from.
val pooledFrameCount: Int	SpeechEmbeddingFrames.pooledFrameCount of the frames this was fitted from.
val poolMillis: Double	The pooling window those frames were built with, in milliseconds.
val hopMillis: Double	The hop those frames were built with, in milliseconds.
val backend: String	Which speech worker produced the payload underneath.
val modelId: String	The model it ran.
val modelRevision: String	The revision of that model.
val wavlmLayer: Int?	Which WavLM layer the vectors were read from, or null where the backend recorded none.
val distance: SmileMdsDistance	The metric the distances were measured with.
val perplexity: Double	The perplexity the fit ran with, which sets how large a neighbourhood each frame is placed against.
val seed: Long	The seed the fit ran with. It is fixed by default, so the same frames give the same picture.
val cost: Double	The divergence the fit finished at, lower being a closer one.

SpeechEmbeddingVisualization.kt

Phase portrait renderers for speech and spectral embedding spaces, synchronized with a shared audio cursor.

Two kinds of space are supported. Speech embeddings come from learned models such as WavLM and use voice activity as the opacity modulator. Spectral embeddings come from SpectralEmbeddingProjection and use spectral flux instead. Both draw two phase portraits (axes 0/1 and 2/3) above an activity strip, with hover audition and click-to-seek wired into the same field-time-* event bus as the audio player.

Each kind has three entry points: a single-panel live variant that fetches from an AnalysisResult, a single-panel preview variant that works from a cached AnalysisPreview, and a multi-panel dashboard that stacks several panels with a shared cursor. The spectral variants accept the MDS or t-SNE preview that withSpectralProjection attaches to an AnalysisPreview.

src/jvmMain/kotlin/afield/audio/ai/SpeechEmbeddingVisualization.kt — read the source

Used in: Visualizers, Displays, and Little Instruments

Top-level functions

Declaration	Summary
fun speechEmbeddingVisualizationHtml(result: AnalysisResult, serviceBaseUrl: String = "/field-audio", width: Int = 1100, phaseHeight: Int = 300, points: Int = 1800, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Draw a speech result as two synchronized phase portraits: MDS axes 0/1 and 2/3.
fun speechEmbeddingPreviewVisualizationHtml(preview: AnalysisPreview, width: Int = 1100, phaseHeight: Int = 300, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Model-free counterpart for cached previews, documentation, and hand-built trajectories.
fun spectralEmbeddingPreviewVisualizationHtml(preview: AnalysisPreview, width: Int = 1100, phaseHeight: Int = 300, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Draw a Field-projected spectral space with the same shared time-cursor protocol as audio.
fun spectralEmbeddingPreviewDashboardHtml(previews: List<AnalysisPreview>, width: Int = 1100, phaseHeight: Int = 280, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Stack alternative spectral distances while keeping one bidirectional audition cursor.
fun speechEmbeddingDashboardHtml(results: List<AnalysisResult>, serviceBaseUrl: String = "/field-audio", width: Int = 1100, phaseHeight: Int = 280, points: Int = 1800, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Stack learned speech spaces while preserving one shared audio cursor.
fun speechEmbeddingPreviewDashboardHtml(previews: List<AnalysisPreview>, width: Int = 1100, phaseHeight: Int = 280, syncGroup: String? = null, timeOffsetSeconds: Double = 0.0): String	Stack already-fetched speech previews without contacting the service.

TrackedAnalysis.kt

One explicit analysis request, followed from submission to durable result.

FieldAudioClient has four separate verbs for the stages of that, each returning its own future: submit, watch, awaitAnalysis and inspect. trackAnalysis composes them into a single future owned by the calling run, with a callback at each stage, which is what a box asking one question of the service usually wants. Ownership divides along the way: the subscription and the polling belong to the run and are released when it stops, while the job belongs to the service and goes on, since another client may be waiting on the same result.

src/jvmMain/kotlin/afield/audio/ai/TrackedAnalysis.kt — read the source

Top-level functions

Declaration

Summary

fun FieldAudioClient.trackAnalysis(request: ModelOperation, onAccepted: (JobAccepted) -> Unit = {}, onProgress: (JobAccepted, ProgressEvent) -> Unit = { _, _ -> }, onCompleted: (JobRecord) -> Unit = {}, onResult: (AnalysisResult) -> Unit = {}, onError: ((Throwable) -> Unit)? = null): CompletableFuture<AnalysisResult>

Submit one explicit analysis request, watch progress, await its result and inspect final job state. Publication and resources belong to the calling run through runAsync. Stop/rerun cancels local polling and closes the watch, including a handshake that finishes late. Durable service jobs and shared clients are retained: abandoning a subscription does not cancel work another client may use. Progress transport failures fail this operation; final job state comes from FieldAudioClient.inspect.

UnderfitOperations.kt

Request builder for source-conditioned Stable Audio 3 generation, with LoRAs named by digest.

underfitCandidates is a ModelOperation factory and nothing more: it validates its arguments here in the JVM, names underfit.stable_audio_3 as the backend, and packs everything into the snake_case keys that backend expects. Submitting it, waiting on it and auditioning the candidate board are FieldAudioClient's job. Stable Audio 3 and Underfit themselves stay in an optional Python worker, so the Kotlin here knows a wire operation name and a parameter vocabulary rather than a model.

The source AssetRegion is sent as init audio, and initNoiseLevel on UnderfitGenerationControls sets how far generation may travel from it. One candidate is requested per seed, so the default 0L until 8L asks for eight. UnderfitOperation chooses between the generate.style and generate.variations wire operations, and UnderfitModel carries the duration ceiling of each checkpoint pack, which underfitCandidates checks durationSeconds against before building anything.

A LoRA is identified by a path relative to the worker's configured LoRA root plus the lowercase SHA-256 of its .safetensors bytes, and both travel with the request. So a request names one exact set of weights rather than a filename that may have been retrained since. UnderfitLoraCheckpoint.fromFile computes that digest from a local file, which assumes Field and the worker share a filesystem. UnderfitLoraCheckpoint's own init block checks what it can before a request exists: the path stays inside the root and ends in .safetensors, the digest is exactly 64 lowercase hex digits, strength is finite and non-negative, and the LoRA's denoising interval satisfies 0 <= intervalMin <= intervalMax <= 1. UnderfitGenerationControls validates its sampling knobs the same way, independently of any one request.

src/jvmMain/kotlin/afield/audio/ai/UnderfitOperations.kt — read the source

Used in: Field Audio AI

Top-level functions

Declaration

Summary

fun underfitCandidates(source: AssetRegion, prompt: String = "", negativePrompt: String? = null, durationSeconds: Double = source.durationSeconds, checkpoints: List<UnderfitLoraCheckpoint> = emptyList(), model: UnderfitModel = UnderfitModel.SMALL_MUSIC, operation: UnderfitOperation = UnderfitOperation.STYLE, seeds: LongRange = 0L until 8L, requestedBy: String? = null, controls: UnderfitGenerationControls = UnderfitGenerationControls(), forceRegenerate: Boolean = false): ModelOperation

Build a source-conditioned Stable Audio 3 request using base weights or checksummed Underfit LoRAs.

UnderfitModel

enum class UnderfitModel(val id: String, val maximumDurationSeconds: Double)

Stable Audio 3 checkpoint packs supported by the pinned Underfit worker.

Constants: SMALL_MUSIC, SMALL_MUSIC_BASE, SMALL_SFX, SMALL_SFX_BASE, MEDIUM, MEDIUM_BASE.

UnderfitOperation

enum class UnderfitOperation(val wireName: String)

Source-conditioned operation exposed by the Underfit worker.

Constants: STYLE, VARIATIONS.

UnderfitLoraCheckpoint

data class UnderfitLoraCheckpoint(val workerPath: String, val sha256: String, val strength: Double = 1.0, val intervalMin: Double = 0.0, val intervalMax: Double = 1.0, val layerFilter: String = "")

One immutable Underfit LoRA as the worker sees it.

Declaration	Summary
val workerPath: String	Path relative to FIELD_AUDIO_UNDERFIT_LORA_ROOT on the worker host.
val sha256: String	Lowercase SHA-256 of the .safetensors file.
val strength: Double	LoRA contribution. Zero disables it without changing the request's checkpoint identity.
val intervalMin: Double	Lowest denoising sigma at which this LoRA is active.
val intervalMax: Double	Highest denoising sigma at which this LoRA is active; lower values skip early structural steps.
val layerFilter: String	Optional Underfit layer-selection expression; blank applies the LoRA to every trained layer.

UnderfitGenerationControls

data class UnderfitGenerationControls(val steps: Int = 8, val cfg: Double = 1.0, val initNoiseLevel: Double = 0.65, val samplerType: String? = null, val cfgIntervalMin: Double = 0.0, val cfgIntervalMax: Double = 1.0, val cfgRescale: Double = 0.0, val cfgNormThreshold: Double = 0.0, val durationPaddingSeconds: Double = 6.0, val apgScale: Double = 1.0, val chunkedDecode: Boolean? = null)

Stable Audio 3 sampling controls honored by the pinned Underfit worker.

Declaration	Summary
val steps: Int	How many denoising steps the sampler takes. Between 1 and 1000. Sent as steps.
val cfg: Double	Classifier-free guidance strength. Non-negative and finite. Sent as cfg.
val initNoiseLevel: Double	How far generation may travel from the source audio, between zero and one: zero preserves the source and one approaches a fresh text-conditioned sample. Sent as init_noise_level.
val samplerType: String?	Names one of the worker's samplers. Null leaves the key out of the request, so the worker picks; a blank string is rejected here. Sent as sampler_type.
val cfgIntervalMin: Double	Start of the denoising range over which guidance applies. The pair must satisfy 0 <= cfgIntervalMin <= cfgIntervalMax <= 1. Sent as cfg_interval_min.
val cfgIntervalMax: Double	End of that range, at or after cfgIntervalMin and at most one. Sent as cfg_interval_max.
val cfgRescale: Double	Guidance rescaling. Non-negative and finite. Sent as cfg_rescale.
val cfgNormThreshold: Double	Guidance norm threshold. Non-negative and finite. Sent as cfg_norm_threshold.
val durationPaddingSeconds: Double	Duration padding, in seconds. Non-negative and finite. Sent as duration_padding_seconds.
val apgScale: Double	APG guidance scale. Non-negative and finite. Sent as apg_scale.
val chunkedDecode: Boolean?	Whether the worker decodes in chunks. Null leaves the key out of the request, so the worker picks. Sent as chunked_decode.