Corpus microscope: from recordings to related windows

Pick a passage from a recording and the corpus microscope finds other stretches that sound like it. It compares harmony, pitch, texture, rhythm and dynamics, and you can choose how much each matters to the match.

To prepare a collection, it measures overlapping windows of 1.5, 3, 6 and 12 seconds; your audio files stay unchanged. Here we'll search for six seconds of Nancy in Final Out at Candlestick Park.

To run the examples on this page, open the audio_ai_corpus_microscope starting point.

The first two neighbours of the six-second Nancy query, with their source timestamps and separate distances for harmony, pitch, texture, rhythm and dynamics.

What each window is measured by

Each group is pooled over the window, and the five together are the vector a query is matched against.

Submit and monitor the index

Segmentation and descriptor extraction are ordinary cached analysis.* jobs. This box indexes Nancy and Final Out at Candlestick Park. Keep the index, query and placement fragments below in the same box. Run the index first and wait for Index ready before running the query. The box properties publish only successful results for the later fragments:

import java.util.concurrent.CancellationException
import java.util.concurrent.CompletableFuture

val client = cached { FieldAudioClient() }
prop<AudioAsset>(nancy)
prop<CorpusPreview>(preview)
prop<AssetRepresentation>(canonical)
display("all jobs", fieldAudioJobMonitor(heightPixels = 330))

:::fold durable job tracking
fun tracked(run: RunScope?, label: String, request: ModelOperation): CompletableFuture<AnalysisResult> = RunScope.withCurrent(run) {
    client.trackAnalysis(
        request,
        onAccepted = { display("$label job", fieldAudioJobStatusHtml(it)) },
        onProgress = { job, event -> display("$label job", fieldAudioJobStatusHtml(job, event)) },
        onCompleted = { display("$label job", fieldAudioJobStatusHtml(it)) },
        onError = {}, // The enclosing operation reports failure once.
    )
}

:::endfold

runAsync<AudioAsset>(
    onResult = { source ->
        nancy = source
        markdown("index", "Index ready. Run the query fragment below.")
    },
    onError = { failure -> markdown("index error", "Index failed: ${(failure.message ?: failure.toString()).md}") },
) {
    // Each HTTP continuation must re-enter the run that owns its tracked job.
    val run = RunScope.current()
    val lookups = ["*Nancy*", "*Candlestick Park*"].map { client.assetByFilename(it) }
    onCancel { lookups.forEach { it.cancel(true) } }
    CompletableFuture.allOf(*lookups.toTypedArray()).thenCompose {
        if (future.isDone) throw CancellationException("run ended")
        val assets = lookups.map { it.join() }
        val jobs = assets.mapIndexed { index, asset ->
            val whole = AssetRegion(asset.id, 0.0, requireNotNull(asset.durationSeconds))
            tracked(run, "source ${index + 1} segments", segmentCorpus(whole, requestedBy = "field:corpus/index"))
                .also { pending -> onCancel { pending.cancel(true) } }
                .thenCompose {
                    if (future.isDone) throw CancellationException("run ended")
                    tracked(
                            run,
                            "source ${index + 1} descriptors",
                            embedCorpus(whole, requestedBy = "field:corpus/index"),
                        )
                        .also { pending -> onCancel { pending.cancel(true) } }
                }
        }
        CompletableFuture.allOf(*jobs.toTypedArray()).thenApply { assets.first() }
    }
}

segmentCorpus records deterministic windows and their previous/next source neighbours. embedCorpus pools the five descriptor groups over those exact windows. Re-running either unchanged request returns its saved result immediately.

Query, await and visualize

A query is a job too, since the query passage has to be measured. Once it completes, corpusPreview loads the neighbours and corpusMicroscopeHtml shows them with players you can listen to. This query leaves out Nancy itself, so every result comes from Candlestick; set includeSameAsset = true to search the query's own recording as well:

val query = AssetRegion(nancy.id, startSeconds = 18.0, durationSeconds = 6.0)

runAsync<CorpusPreview>(
    onResult = { result ->
        preview = result
        canonical =
            nancy.representations.firstOrNull { it.role == "canonical_project" }
                ?: nancy.representations.first { it.mediaType in ["audio/wav", "audio/x-wav"] }
        display("microscope", corpusMicroscopeHtml(preview, canonical))
    },
    onError = { failure -> markdown("query error", "Query failed: ${(failure.message ?: failure.toString()).md}") },
) {
    val run = RunScope.current()
    val neighbours = tracked(
        run,
        "Nancy neighbours",
        findCorpusNeighbours(
            query,
            limit = 12,
            includeSameAsset = false,
            weights = CorpusDistanceWeights(
                harmony = 0.8,
                pitch = 0.55,
                texture = 1.35,
                rhythm = 0.7,
                dynamics = 0.35,
            ),
            requestedBy = "field:corpus/query",
        ),
    )
    onCancel { neighbours.cancel(true) }
    neighbours.thenCompose { result ->
        if (future.isDone) throw CancellationException("run ended")
        client.corpusPreview(result.id).also { pending -> onCancel { pending.cancel(true) } }
    }
}

Each neighbour card shows where the window comes from, a bar for each of the five distances, and a player that crossfades from the query into the neighbour. The query and neighbour players both start at zero, so hovering, zooming, clicking and playback stay aligned between the two recordings. Each card also lists the window's timestamps in its source and the ids of the windows before and after it.

Place a neighbour as a box

Show the microscope with showCorpusMicroscope and each neighbour card gets a place button. After the query finishes, press it to make a box that plays that window. The new box is named for the recording and window; edit its times and run it to move or stretch the excerpt.

showCorpusMicroscope(preview, canonical)

placeCorpusNeighbour(neighbour) does the same from code. This box places the first neighbour and shows the code the new box was given:

:::static
import java.io.File

val folder = File("public/docs/assets/audio_ai_corpus_microscope")
val preview = FieldAudioClient.json.decodeFromString<CorpusPreview>(File(folder, "corpus.json").readText())
:::fold show the placed box's name and code
val placed = placeCorpusNeighbour(preview.neighbours.first())
markdown("placed", "**${placed.name}**\n\n```kotlin\n${placed.properties["code"]}```")

corpusClip fetches the recording from the Field audio service once, keeps it under ~/.afield/cache/field-audio, and cuts out the window; clip.sampler() plays it from the start.

See also: the EnCodec round trip for the token representation, and Short-time spectral spaces.