StdlibReferenceGenerator.kt
src/jvmMain/kotlin/afield/tools/docs/StdlibReferenceGenerator.kt · 2809 lines
package afield.tools.docs
import afield.runtime.completion.KotlinSourceIndex
import java.io.File
import kotlin.system.exitProcessFIELD_TEST id: docs.codebase-reference.generate runner: jvm-main entrypoint: afield.tools.docs.StdlibReferenceGenerator tags: [smoke, docs, reference, stdlib, verification] timeoutSec: 360 expect: exitCode: 0 stdoutContains: - "StdlibReferenceGenerator ok"
Emits Field's codebase reference — the browsable answer to "what is already in the box?" — from the KDoc that lives next to the code.
Why this exists, and why it is not Dokka
Dokka was removed in 20e64da5, because the durable facts belong in KDoc beside the code and a
generated wall of one-page-per-symbol HTML is not documentation anyone reads. Field already has
two interactive ways into that KDoc — the completion panel's recursive code browser
(public/docs/completion_doc_browser.md) and the semantic index — but both need you to already
know the name you are looking for. Nothing answered the browsing question. This generator does,
with ordinary Zensical markdown inside the existing docs site: a page per completion category,
a section per source file, the file's own KDoc as the orientation prose, and a table per type.
One parser, not two
Declarations come from KotlinSourceIndex.parseTopLevelDefinitions — the same scan that feeds the in-app completion doc browser — so a symbol listed here is by construction a symbol the doc panel can resolve. Only the signature text is re-read from source here (the index stores names and lines, not signatures); see signatureAt.
What it will not invent
- Every summary is the first paragraph of a real KDoc comment; a symbol with no KDoc is listed with an empty summary rather than a guessed one.
- A KDoc reference in square brackets is linkified only when the target is a type this reference actually emits — the same "links only appear when they resolve" rule the completion panel follows. Everything else degrades to plain inline code.
- The "see also" links from a reference section to a worked-example page are evidence-based:
a page is linked only when it literally mentions
MIN_EVIDENCE_HITSor more distinct public symbol names from that source file (seeevidenceLinks). No hand-maintained topic map to rot.
File overview rule
The first KDoc block in a file is treated as the file's overview when it ends before the first declaration and is separated from that declaration by at least one blank or comment line — and it is then not also shown as that declaration's own summary. A block written flush against its declaration stays that declaration's own KDoc, of which only the first paragraph is published; the file's "what it is" cell on the index then has nothing to fill it with.
Running it
Regenerate the committed pages (needs no Field server, no GPU, no network):
./gradlew generateCodebaseReferencedocs.codebase-reference.generate
FIELD_TEST does, so an API or KDoc change that nobody regenerated fails the build. This mirrors
afield.examples.AgentTranscriptMaterializer and the tools/linalg-generator --check mode that
guards public/docs/linear_algebra_api.md.
The pages are committed rather than built on the fly: the Zensical docs build must not gain a JVM step, and a generated reference is only trustworthy if an API change shows up as a reviewable diff.
object StdlibReferenceGenerator
Emits public/docs/reference/ from the KDoc in src/, or drift-checks the committed pages.
object StdlibReferenceGenerator {Repo-relative home of the committed reference pages.
private const val REFERENCE_REL = "public/docs/reference"Repo-relative home of the KDoc worklist.
dev_docs/ and not public/docs/: this is a ranked list of what is missing, which is a
standing engineering task rather than something a reader of the site wants. The published
pages already carry the honest coverage numbers.
private const val WORKLIST_REL = "dev_docs/kdoc_worklist.md"How many individual gaps the worklist names before it stops and says how many it dropped.
private const val WORKLIST_ENTRIES = 120How many distinct entry points a docs page must call before it earns a "Used in" link.
private const val MIN_EVIDENCE_HITS = 2At most this many docs pages are linked from one source file's section.
private const val MAX_EVIDENCE_LINKS = 3Every JVM Kotlin source file belongs in the catalog, including implementation and harnesses.
private val SOURCE_ROOTS = listOf("src/jvmMain/kotlin")internal fun sourceRoots(): List<String> = SOURCE_ROOTS
SOURCE_ROOTS, for the one other generator that has to agree with it.
SourcePageGenerator renders a docs page per source file and used to carry its own identical copy of this list. Identical and independent is the worst of both: a reference row's "read the source" link is emitted from the file's own partition rather than from that corpus, so a root added here and not there ships a page of links to pages nobody rendered.
Known build/verification helpers remain listed, but are visually marked as implementation.
private val IMPLEMENTATION_PREFIXES = listOf(
"src/generatedLinalg/kotlin/afield/math/linalg/",
"src/jvmMain/kotlin/afield/audio/synth/AbletonSetSmokeTest.kt",
"src/jvmMain/kotlin/afield/audio/synth/DocsAudioProvenance.kt",
"src/jvmMain/kotlin/afield/audio/synth/DoughDspPlotGallery.kt",
"src/jvmMain/kotlin/afield/audio/synth/DoughDspSmokeTest.kt",
"src/jvmMain/kotlin/afield/audio/synth/FaustPolySmokeTest.kt",
"src/jvmMain/kotlin/afield/audio/synth/FaustSmokeTest.kt",
"src/jvmMain/kotlin/afield/audio/synth/PrettyPrint.kt",
"src/jvmMain/kotlin/afield/audio/synth/SonicPiActivity.kt",
"src/jvmMain/kotlin/afield/audio/synth/SonicPiPatternsSmokeTest.kt",
"src/jvmMain/kotlin/afield/audio/synth/SonicPiSmokeTest.kt",
"src/jvmMain/kotlin/afield/audio/synth/SoundRenderPathDocs.kt",
"src/jvmMain/kotlin/afield/audio/synth/SynthCookbookCompileDemo.kt",
"src/jvmMain/kotlin/afield/simulation/kratos/KratosProtocol.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/FabricatePassGraphTestMain.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/FabricateRenderPlanTestMain.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/FabricateSceneGraphTestMain.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/FabricateSceneTestMain.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/PathTraceGltfTestMain.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/PathTraceTestMain.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/PathTraceGltfGpuSmoke.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/PathTraceTemporalGpuSmoke.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/RayTracingDocsHarness.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/RectSLGalleryHarness.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/StackSLGalleryHarness.kt",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/StandardRendererDocsHarness.kt",
"src/jvmMain/kotlin/afield/data/query/datalog/DatalogSmokeTest.kt",
"src/jvmMain/kotlin/afield/lang/rectsl/RectSLFitFigure.kt",
"src/jvmMain/kotlin/afield/lang/rectsl/RectSLSoftRasterFigure.kt",
"src/jvmMain/kotlin/afield/lang/stacksl/cpu/StackSLCpuGallery.kt",
"src/jvmMain/kotlin/afield/lang/stacksl/cpu/StackSLCpuContactSheet.kt",
)internal const val LANGUAGES = "Languages"
The category the small-language packages are published under, and the one label in this file
that no source file carries. Named as a constant because it is the join between
CATEGORY_BY_PATH, CATEGORY_BLURBS and documentedPackages, and because the plural is
load-bearing: afield.runtime.completion.KotlinCompletionSupport normalises the singular "language" to
"Kotlin", so the tidier-looking spelling would name a different category.
internal const val RENDERERS = "Renderers"
The category afield.graphics.scene.fabricated is published under — the second label in this file that
no source file carries, and the second one whose spelling is load-bearing rather than
chosen.
The obvious name is Rendering, and it is unusable.
afield.runtime.completion.KotlinCompletionSupport.normalizedCompletionCategory maps "rendering" onto
"Graphics", so a page label spelled that way would be filed under an existing, unrelated
category the moment anything normalised it. This is exactly the Language → Kotlin trap
that made the plural in LANGUAGES load-bearing, arriving through a different word, and the
same when branch takes "shader", "shaders", "drawing", "geometry" and "vulkan"
with it. "renderers" matches no branch, so it titlecases to itself.
LanguagesCategoryTest pins both directions, as it does for Languages.
The existing Graphics page was the other candidate and is the wrong home on two counts.
The kept half of this root is 9207 lines, which would nearly triple graphics.md — already
the second largest page here — and drown its hand-written drawing helpers under a GPU frame
graph. And the Graphics blurb describes "everything that ends up as an FLine, a colour or
a shader", which is a fair description of afield.stdlib's drawing surface and not of a
path tracer, a splat renderer and a pass library. A category is a promise about what a reader
will find; these are two different promises.
Note that completion already answers Graphics for these files by a different route —
afield.runtime.completion.KotlinCompletionSupport infers it from the path /afield/four/ — and this label does
not change that. Nothing here is annotated, so the popup's view is untouched; see
CATEGORY_BY_PATH.
The category a file lands in when it carries no @file:CompletionCategory — matched on the
repo-relative path, first entry wins, UNCATEGORISED when nothing does.
This map is read by this generator and by nothing else, and that is the entire point. The
alternative — annotating the files — was rejected for the reason spelled out at length in
IMPLEMENTATION_PREFIXES: @file:CompletionCategory is not inert. It is read by
afield.runtime.completion.KotlinCompletionSupport.isHiddenCompletionCategory, on the strength of which one
spelling of it removes candidates from the completion popup; KotlinSourceIndex records every
declaration in an annotated file under that file's label with no visibility filter; and
KotlinSourceIndex.getCompletionCategoryForSymbol ends in a name-only
categories.firstOrNull(), so a label on one file can answer for a same-named declaration
declared somewhere else entirely. Nothing in these thirteen packages is annotated today, and
publishing them here does not change that: KotlinSourceIndex.fileCompletionCategory matches
only a literal @file: annotation, so with none present it keeps returning null for every one
of these files and the completion popup cannot tell the difference. A documentation change must
not be able to alter completion, and the way to guarantee that is for the label to live where
completion cannot see it.
The mechanism is not new — the afield/visualization/ entry below is the original hardcoded
if this map generalises, and it has assigned a category by path since the reference shipped.
LanguagesCategoryTest pins both halves: that the map's labels stay unhidden and unannotated,
and that adding language roots left every pre-existing page byte-identical.
Languages is plural on purpose. afield.runtime.completion.KotlinCompletionSupport normalises the singular
"language" to "Kotlin", so a file (or a future entry here) spelled Language would be
quietly filed under the Kotlin category instead. The test pins the plural.
Simulation is here for a weaker reason than Languages, and the difference matters.
Languages is a label that exists nowhere but this map. Simulation is a real completion
category that eight files in afield.simulation.kratos and afield.geometry.mesh.tetgen already carry, so the two
entries below are not inventing a label — they are extending one the source chose to the
files beside it that never got annotated. Which is why the two entries are not three: every
file in afield.geometry.mesh.tetgen annotates itself, so a prefix for it would label nothing today, and
if a third file ever lands there unannotated the UNCATEGORISED build failure telling its
author to "add the annotation its package siblings carry" is correct advice in that
package. In afield.simulation.kratos it would not be — three of its nine files are unannotated and
always were — so the prefix does real work there, and it does all of it in
afield.simulation.physics, where nothing is annotated at all.
RENDERERS is back to the Languages shape, and it is the stronger of the two cases.
No source file anywhere in src/ carries Renderers, and no file under
afield/four/fabricated/ carries a category annotation of any kind, so the prefix below is
the only thing in the tree that knows the label exists. LanguagesCategoryTest holds that
property for every path-assigned category no source file declares, rather than for
Languages by name, so this entry joined that guard without being added to it.
The safety argument is the same one and is unchanged by the label being real: this map is
still read by the generator and by nothing else, and this branch adds no annotation to
any file, so KotlinSourceIndex.fileCompletionCategory keeps returning exactly what it
returned before for every file under both prefixes and the completion popup cannot tell the
difference. The counterfactual — annotating the 18 unannotated files instead — was measured
rather than waved at, and the numbers are in dev_docs/kdoc_reference_contract.md.
source — 64 lines, from line 240
private val CATEGORY_BY_PATH: List<Pair<String, String>> = listOf(
"src/jvmMain/kotlin/afield/ui/visualization/" to "Visualization",
"src/jvmMain/kotlin/afield/simulation/physics/" to "Simulation",
"src/jvmMain/kotlin/afield/simulation/kratos/" to "Simulation",
"src/jvmMain/kotlin/afield/graphics/scene/fabricated/" to RENDERERS,
"src/jvmMain/kotlin/afield/lang/clojure/" to LANGUAGES,
"src/jvmMain/kotlin/afield/data/query/datalog/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/forth/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/fieldml/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/morpho/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/nano/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/python/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/rectsl/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/remotejs/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/stacksl/" to LANGUAGES,
// Package ownership also supplies a docs-only fallback after the split.
// Explicit file categories still win; completion annotations are unchanged.
"src/jvmMain/kotlin/afield/ai/retrieval/" to "Machine learning",
"src/jvmMain/kotlin/afield/animation/interpolation/" to "Graphics",
"src/jvmMain/kotlin/afield/animation/keyframes/" to "Graphics",
"src/jvmMain/kotlin/afield/app/diagnostics/" to "Workspace",
"src/jvmMain/kotlin/afield/document/context/" to "Workspace",
"src/jvmMain/kotlin/afield/document/graph/" to "Workspace",
"src/jvmMain/kotlin/afield/document/properties/" to "Workspace",
"src/jvmMain/kotlin/afield/document/workspace/" to "Workspace",
"src/jvmMain/kotlin/afield/geometry/path/" to "Graphics",
"src/jvmMain/kotlin/afield/graphics/color/" to "Graphics",
"src/jvmMain/kotlin/afield/graphics/render/" to "Graphics",
"src/jvmMain/kotlin/afield/graphics/shader/" to "Graphics",
"src/jvmMain/kotlin/afield/graphics/text/" to "Graphics",
"src/jvmMain/kotlin/afield/graphics/vulkan/buffers/" to "Graphics",
"src/jvmMain/kotlin/afield/io/osc/" to "Networking",
"src/jvmMain/kotlin/afield/math/functions/" to "Math",
"src/jvmMain/kotlin/afield/math/linalg/" to "Math",
"src/jvmMain/kotlin/afield/media/image/" to "Imaging",
"src/jvmMain/kotlin/afield/media/video/" to "Video",
"src/jvmMain/kotlin/afield/runtime/execution/" to "Workspace",
"src/jvmMain/kotlin/afield/runtime/languages/" to "Workspace",
"src/jvmMain/kotlin/afield/runtime/lifecycle/" to "Workspace",
"src/jvmMain/kotlin/afield/ui/controls/" to "Visualization",
"src/jvmMain/kotlin/afield/ui/output/" to "Visualization",
"src/jvmMain/kotlin/afield/lang/lua/" to LANGUAGES,
"src/jvmMain/kotlin/afield/lang/brief/" to "Solvers",
"src/jvmMain/kotlin/afield/lang/solidsl/" to "Graphics",
"src/jvmMain/kotlin/afield/lang/lambda/" to "Function",
// Complete domain fallbacks, after the more specific package choices above.
"src/jvmMain/kotlin/afield/ai/" to "Machine learning",
"src/jvmMain/kotlin/afield/animation/" to "Graphics",
"src/jvmMain/kotlin/afield/app/" to "Workspace",
"src/jvmMain/kotlin/afield/audio/" to "Audio",
"src/jvmMain/kotlin/afield/data/" to "Data",
"src/jvmMain/kotlin/afield/document/" to "Workspace",
"src/jvmMain/kotlin/afield/examples/" to "Examples",
"src/jvmMain/kotlin/afield/geometry/" to "Graphics",
"src/jvmMain/kotlin/afield/graphics/" to "Graphics",
"src/jvmMain/kotlin/afield/io/" to "Networking",
"src/jvmMain/kotlin/afield/math/" to "Math",
"src/jvmMain/kotlin/afield/media/" to "Imaging",
"src/jvmMain/kotlin/afield/runtime/" to "Workspace",
"src/jvmMain/kotlin/afield/simulation/" to "Simulation",
"src/jvmMain/kotlin/afield/study/" to "Search",
"src/jvmMain/kotlin/afield/tools/" to "Workspace",
"src/jvmMain/kotlin/afield/ui/" to "Visualization",
)internal fun pathAssignedCategories(): Set<String> = CATEGORY_BY_PATH.map { it.second }.toSet()
The distinct categories CATEGORY_BY_PATH can assign — the set LanguagesCategoryTest checks.
internal fun pathAssignedPrefixes(): List<String> = CATEGORY_BY_PATH.map { it.first }
The path prefixes CATEGORY_BY_PATH assigns a category to, for the same test.
internal fun pathAssignedPrefixesFor(category: String): List<String>
Just the prefixes CATEGORY_BY_PATH files under category.
internal fun pathAssignedPrefixesFor(category: String): List<String> =
CATEGORY_BY_PATH.filter { it.second == category }.map { it.first }internal fun categoryBlurb(category: String): String? = CATEGORY_BLURBS[category]
category's page blurb, or null when it has none — which would publish a page with a heading
and no description. LanguagesCategoryTest refuses that for any path-assigned category.
Shared coverage description for the reference and KDoc worklist.
private fun documentedPackages(): String = "all JVM Kotlin packages in `src/jvmMain/kotlin`"The last-resort bucket for a file carrying no @file:CompletionCategory, and a failure —
never a page. It has no CATEGORY_BLURBS entry, so a file landing here is published under a
heading called "Uncategorised" with no description, which is worse than not publishing it.
Enforcing that here rather than in the drift guard is deliberate. The drift guard compares the
committed pages against what the source produces, so it certifies faithfulness, not
correctness: once someone regenerates and commits an uncategorised.md, source and artifact
agree and every guard goes green while the docs ship an instrument on a page named after the
fact that nobody categorised it. That is exactly what happened between 2026-08-08 and
2026-08-14 with the gesture board. This check cannot be silenced by regenerating, because it
fails on the source condition.
private const val UNCATEGORISED = "Uncategorised"page lints
Two checks over the rendered page text, run from main in the failures list before
the drift guard. The ordering is the whole design, and it is the same argument
UNCATEGORISED makes above: the drift guard only certifies that the committed pages equal
what the source produces, so any defect that survives regeneration is certified rather than
caught. A lint placed after it could always be silenced with
./gradlew generateCodebaseReference. Placed before it — and the drift guard is gated on
failures.isEmpty() — a planted defect reports the lint alone, with no drift noise, which
is also what makes a red-first demonstration legible.
A markdown inline-link target that could name something.
Anything else is a link markdown will render but that can resolve to nothing — the shape
a Kotlin sample produces when it lands in prose, because pat("[bd cp](../../../../../../reference/3,8)") is a
perfectly good markdown link to the path 3,8.
Measured over the 20 committed pages at 4546d72b5: 12675 ](…) occurrences, 11324 distinct
targets, of which 1 fails this pattern — 3,8, on patterns.md:2508, at indent 8. See
impossiblePathLinkFailures for why that one is not a failure.
private val PLAUSIBLE_LINK_TARGET =
Regex("""^(?:#|https?://|mailto:)|^[A-Za-z0-9_./-]+(?:#[A-Za-z0-9_.-]+)?$""")
private val INLINE_LINK_TARGET = Regex("""\]\(([^)\n]*)\)""")Pages whose prose renders a link that can resolve to nothing.
The predicate is narrower than it first looks, and the narrowing is the measurement. A
](…) with an impossible target only matters where markdown will actually make a link
out of it, which means: not inside a ``` fence, and at an indent below 4 — at 4 or more the
line is an indented code block and no link is produced.
Both halves were established by counting rather than by reasoning. Over the 20 committed
pages the impossible-target set has exactly one member, patterns.md:2508
(val groove = pat("[bd cp](../../../../../../reference/3,8) hh!2")), and it sits at indent 8. That line is the
fixed form of the very defect this lint exists for: the bug was the same sample rendered
at indent 3, where it did become a link. So a predicate without the indent clause would
fail on the base commit over content that is already correct — and the floor this lint
holds is 0, not 1.
Deliberately not grandfathered: the count is 0, so there is nothing to grandfather, and a ledger with no entries is the right state for a ratchet.
private fun impossiblePathLinkFailures(pages: Map<String, String>): List<String> {
val failures = mutableListOf<String>()
pages.forEach { (name, text) ->
var inFence = false
text.lineSequence().forEachIndexed { index, line ->
if (line.trimStart().startsWith("```")) {
inFence = !inFence
return@forEachIndexed
}
if (inFence) return@forEachIndexed
if (line.length - line.trimStart(' ').length >= 4) return@forEachIndexed
INLINE_LINK_TARGET.findAll(line).forEach { match ->
val target = match.groupValues[1]
if (target.isNotEmpty() && PLAUSIBLE_LINK_TARGET.containsMatchIn(target)) return@forEach
failures += "$name:${index + 1}: renders a link to \"$target\", which can name " +
"nothing — a code sample reached prose. Indent the sample by 4, or fence " +
"it, so markdown stops reading `](…)` as a link: ${line.trim()}"
}
}
}
return failures
}Fenced lines carrying a literal backtick, grandfathered by exact text with an owner.
A backtick inside a fence renders as a backtick, so it is almost always KDoc inline-code
markup that escaped its prose and landed in a sample. Measured over the 20 committed pages
at 4546d72b5: 416 lines sit inside a fence and 1 carries a backtick — the entry
below. The brief for this ratchet stated the rule as "no generated page contains a literal
backtick inside a fence", and that is false at the base commit; it is implemented with the
one real site named, rather than by widening the rule until it passes.
Graded backwards as well, like KNOWN_UNRESOLVED in tools/check_doc_site_links.py: an
entry that no longer reproduces is a failure, or the ledger becomes a list of problems that
used to exist.
private val GRANDFATHERED_FENCE_BACKTICKS: Map<String, String> = linkedMapOf(
"// Forward `line` into your output channel / renderer." to
"A Kotlin *comment* inside a real ```kotlin fence, naming the local `line` it just " +
"bound — src/jvmMain/kotlin/afield/lang/forth/ForthInterpreter.kt:81, published at " +
"languages.md:886. Correct as authored: a comment that mentions an identifier is " +
"how Kotlin is written, and the fence is deliberate. Owner: whoever next edits " +
"that KDoc, and the fix if one is ever wanted is to drop the backticks from the " +
"comment, not to relax this lint.",
)
private fun fenceBacktickFailures(pages: Map<String, String>): List<String> {
val failures = mutableListOf<String>()
val seen = mutableSetOf<String>()
pages.forEach { (name, text) ->
var inFence = false
text.lineSequence().forEachIndexed { index, line ->
if (line.trimStart().startsWith("```")) {
inFence = !inFence
return@forEachIndexed
}
if (!inFence || '`' !in line) return@forEachIndexed
val trimmed = line.trim()
if (trimmed in GRANDFATHERED_FENCE_BACKTICKS) {
seen += trimmed
return@forEachIndexed
}
failures += "$name:${index + 1}: a literal backtick inside a ``` fence — KDoc " +
"inline-code markup renders as a backtick in a code block: $trimmed"
}
}
(GRANDFATHERED_FENCE_BACKTICKS.keys - seen).forEach {
failures += "GRANDFATHERED_FENCE_BACKTICKS entry no longer reproduces, delete it: $it"
}
return failures
}Page order, and the prose that introduces each category page.
source — 246 lines, from line 459
private val CATEGORY_BLURBS: Map<String, String> = linkedMapOf(
"Examples" to "Runnable examples, galleries and verification programs. Read these for usage patterns; their entry points are not a promise of a reusable box API.",
"Internal" to "Implementation details explicitly marked Internal by their source. They remain available to browse here, but are not intended as entry points for code written in a Field box.",
"Lighting" to "Light sources, fixtures and control protocols for working with physical and rendered lighting.",
"Networking" to "OSC messages, discovery and query interfaces for connecting Field to other applications and devices.",
"Video" to "Video sources, shared GPU frames and live diffusion sessions, with controls for motion and temporal output.",
"Math" to "Numerics: signal processing in both domains, estimation and fitting, " +
"interpolation, statistics, randomness and units. Everything here is plain " +
"`Double`/`Float` Kotlin with no GPU and no window, so it runs in a box, a test or a " +
"headless docs capture equally well.",
"Data" to "Structures and processes over data rather than over pixels: clustering and " +
"nearest-neighbour search, generative and constraint systems, graph dynamics, and the " +
"I/O helpers (files, JSON, subprocesses, OSC-adjacent devices) that get data into a box.",
"Graphics" to "Everything that ends up as an `FLine`, a colour or a shader: drawing " +
"helpers, contouring, geometry, colour spaces, keyframing and the small shading " +
"languages' host side.",
RENDERERS to "Turning a scene into pixels on a Vulkan device. Three renderers live in " +
"`afield.graphics.scene.fabricated`, and they disagree about what a frame costs.\n\n" +
"**`StandardRenderer` is the one that runs at frame rate.** A small deferred " +
"renderer — g-buffer, shadow map, ambient occlusion, lighting composite — built on " +
"the Field 4 primitives rather than hiding them, so you feed it meshes and wire " +
"a `RenderTarget` around the passes it hands back. `FabricateFrameGraph` is that pass " +
"list made ordered and named, and it does not hide the underlying `Target` objects: " +
"inserting a pass and manipulating one directly are both available. " +
"`FabricatePostStack` and `FabricatePassChain` are the two post-processing surfaces, " +
"and they differ in one thing: a `FabricatePostStack` pass inherits its source's " +
"resolution and format, while a `FabricatePassChain` lets every pass declare its own, " +
"which is what a half-resolution reduce or a downsample pyramid needs.\n\n" +
"**`PathTracingRenderer` is the one that converges.** A progressive GPU path tracer " +
"with a stackless software BVH that works on every Field Vulkan device and KHR ray " +
"queries where the device has them. It accumulates: the image improves while nothing " +
"moves, camera motion resets or reprojects that history, and a lighting edit always " +
"starts again. Its scene and environment are snapshots — `PathTraceScene` and " +
"`PathTraceEnvironment` are read once at construction, so editing a material means " +
"building a new tracer rather than mutating a live one. Output is a fixed-resolution " +
"linear HDR render target, and `blitToScreen` is a separate call because exposure and " +
"tone mapping are a presentation decision. `PathTraceGltf` loads glTF into it, " +
"sRGB-decoding base colour while leaving metallic-roughness linear.\n\n" +
"**`GaussianSplatRenderer` is the one that renders something measured rather than " +
"modelled.** Splats from a capture, sixteen floats each, sorted and blended rather " +
"than rasterized as triangles; `SplatSequence` reads a directory of " +
"`.compressed.ply` frames in any order through a bounded most-recently-used cache, " +
"because scrubbing a shot is random access and a forward-only reader would be no use " +
"for working.\n\n" +
"Underneath all three sits `FabricateScene`, a Three.js-shaped scene graph that owns " +
"no Vulkan resources at all: nodes carry transforms, renderables carry material and " +
"rendering intent, and `FabricateScenePlanner` turns that into the opaque, " +
"alpha-masked and back-to-front transparent draw queues a renderer consumes. That " +
"independence is what lets the scene graph, the pass graph and the render plan be " +
"exercised without a GPU: `FabricateSceneGraphTestMain`, `FabricatePassGraphTestMain` " +
"and `FabricateRenderPlanTestMain` are plain `main`s that do, carrying their own " +
"assertions. They are not in the `FIELD_TEST` smoke catalog, so run them by hand.\n\n" +
"Two constraints apply to the renderers themselves. A renderer is constructed, " +
"installed and closed on Field's render thread, like any other Field 4 module " +
"— installing " +
"under a name already in use retires the previous occupant, so there is no coroutine " +
"lifetime bookkeeping to do. And a device is required: nothing here that draws does " +
"so into an `FLine`, or in a headless test, the way the `Math` page's contents do. " +
"For the same ground with pictures, see [The Fabricate renderer](" +
"../fabricate_renderer.md), [Path tracing](../path_tracing.md) and " +
"[Field 4 graphics](../afield_four_graphics.md).",
"Polytopes" to "Polyhedra as topology first and coordinates second. An abstract incidence " +
"complex you can build and validate; the Conway operator algebra — `dual`, `ambo`, " +
"`kis`, `truncate`, `gyro` and the composites written over them — that rewrites it, " +
"with per-element provenance and lineage carried through every step; and three " +
"realization verbs that hand the result coordinates: `realize` for a first embedding, " +
"`relax` for one that satisfies constraints you declare, and `canonicalize` for the " +
"classical edge-tangent form you would recognise as a solid. Everything is immutable " +
"and CPU-only; `FieldAdapters.kt` is where a polytope becomes `FLine`s or a " +
"`TriMesh`. 4-polytopes, their Schlegel and slice projections, and an OFF importer " +
"live here too. Files are listed alphabetically below; a reader new to the package " +
"wants `Topology.kt`, then `Transforms.kt`, then `Conway.kt`, then `Canonical.kt`, " +
"then `FieldAdapters.kt` — and [Polytopes: topology first, coordinates later]" +
"(../polytope_examples.md) for the same ground with pictures.",
"Solvers" to "Nonlinear least squares, aimed at geometry. A constraint vocabulary — " +
"planarity, colinearity, fixed distances, perpendicularity, dihedral angles, panel " +
"areas, positions held hard or held softly — assembled into a program, compiled once " +
"and solved by Ceres across a foreign-function bridge; the architectural layer above " +
"it for gridshells and panel families; and the reuse-oriented half that re-solves the " +
"same program every frame while a user drags. Residuals you cannot express in the " +
"vocabulary can be written in Kotlin and differentiated exactly by Ceres through the " +
"expression VM. These entry points need the native Ceres library present, and nothing " +
"here draws — a solve returns points, and drawing them is your business. Start from " +
"`CeresGeometry.kt`, which is the vocabulary everything else is written in; " +
"`ArchitecturalGeometry.kt` is the layer most calls go through.",
"Simulation" to "Making a shape behave. Three stacks live here and they answer different " +
"questions.\n\n" +
"**`afield.simulation.physics` is the one that moves.** It is a CPU solver you step inside " +
"your own `loop { }`, at frame rate, with a person's hand on it: " +
"`SimdVerlet` holds some thousands of particles in flat float arrays and pulls them " +
"together with distance constraints, and on top of that sit `Cloth`, `Strand`, " +
"`Wind`, tearing and a collider set. `JoltPhysicsWorld` is the rigid-body half — " +
"boxes, spheres, meshes, joints and character controllers over the Jolt engine — and " +
"`Dyn4jScene` is the same idea flat, in 2D. Nothing here owns a renderer: every one of " +
"them hands back `FLine`s and where those go is your business. Nothing here needs a " +
"GPU either. It is fast and it is approximate, and the number it gives you is not a " +
"number you would put in a report.\n\n" +
"**`afield.simulation.kratos` is the one that is right.** It is finite elements, out of process: " +
"`KratosSession` starts a Python worker — a local venv, a container, or one already " +
"running — and talks to it over an authenticated socket, so nothing in Field's own " +
"startup imports a solver and a machine without the optional environment simply says " +
"so. A solve is seconds, not milliseconds, which is why the Kratos examples put it " +
"behind a button. What you get back is a `SimulationResult`, which describes its own " +
"fields rather than declaring them, so one class carries a scalar temperature and a " +
"velocity–pressure pair without knowing about either. The layer above it is the " +
"reason to care: a `StudySpace` varies the design, a `measure` reduces each answer to " +
"one number, and `KratosStudy` reports the landscape and where its optimum lies. " +
"Because the embedded formulations reclassify one fixed background grid instead of " +
"remeshing, **geometry is an ordinary study axis** — sweeping an arm length costs what " +
"sweeping a conductivity costs. `KratosStudyFigure` draws the result as pixels a box " +
"can display.\n\n" +
"**`afield.geometry.mesh.tetgen` is the one in between.** Kratos has no embedded structural element, " +
"so stress needs a mesh whose boundary really is the part's surface, sharp edges " +
"included. `Tetgen.tetrahedralize` takes a closed triangle surface — a " +
"[Manifold](../manifold.md) solid, usually — and fills it with tetrahedra that " +
"reproduce it exactly, optionally quadratic, in Kratos's own node order. " +
"`TetMeshViews` is how you look at the inside of the result rather than at its skin.\n\n" +
"Where the neighbours are. [Solvers](solvers.md) is Ceres, which finds the shape you " +
"*want* by minimising residuals you declare; simulation tells you what that shape " +
"then *does*. A gridshell relaxed " +
"in `Solvers` and a bracket solved in Kratos are two halves of the same design loop, " +
"not two ways of doing one thing. Everything above returns geometry, so " +
"[Graphics](graphics.md) is where it gets drawn.\n\n" +
"The prose, with pictures: " +
"[Physics: particles, constraints and cloth]" +
"(../physics.md) for the real-time stack — read it before touching `stiffness`, " +
"because it and `iterations` are not two knobs — and " +
"[Filling a solid: tetrahedral meshing with TetGen](../tetgen.md) for the meshing and " +
"the route into Kratos. Four loadable Kratos documents (thermal, flow, structural and " +
"the parameter study) and the cloth starting point are in the " +
"[Agent Transcript Gallery](../agent_transcripts.md); " +
"[Frames and the Animation Loop](../animation_time_model.md) explains why an integrator " +
"wants a fixed `dt`. `dev_docs/TETGEN.md` carries the mesher comparison that chose " +
"this route.",
"Imaging" to "Rasters and computer vision: convolution and the filter family, gradients " +
"and edges, corners, blobs and morphology, multiscale pyramids, warps and optical " +
"flow. Dependency-free CPU float math built on one value type, `Img`, with the " +
"separable-convolution core vectorized to roughly OpenCV's single-threaded speed. " +
"Every detector hands back geometry, so results go straight to `lines()`.",
"ALife" to "Spatial artificial life: grid cellular automata (life-like, cyclic and " +
"Wolfram's elementary rules), continuous systems (reaction-diffusion, Turing patterns, " +
"Lenia), agent populations (boids, Physarum, diffusion-limited aggregation) and " +
"structural growth (L-systems, space colonization), plus the measurement layer every " +
"one of them makes its claims through.",
"Function" to "The function algebra — Field's first-class, evolvable, graph-rewritable " +
"notion of a function. Build them, compose them, fit them, mutate them, and rewrite " +
"their graphs with user-authored patterns.",
"Search" to "Optimisation over a space you can only sample. You say how to make a genome " +
"and how to vary one, you say what a genome scored, and a driver spends a budget of " +
"*evaluations* against that pair. Nothing in the drivers knows what a genome is, so " +
"it can be a cellular-automaton rule, an L-system, an `Fn` graph or a design vector. " +
"One file holds all five: random sampling, hill climbing, a (µ+λ) evolutionary loop, " +
"novelty search and MAP-Elites. The callers are on three other pages: " +
"[ALife](alife.md) evolves rules and L-systems, [Function](function.md) evolves " +
"function graphs, and [Simulation](simulation.md)'s `KratosStudy` searches a design " +
"space. `afield.lang.morpho` keeps a search of its own. Read `PopulationSearch.kt` for why " +
"`random` is reported as a control arm, and why the budget is evaluations rather than " +
"generations: both decide whether a result means anything.",
"Visualization" to "Turning a value into something you can look at, without leaving the box.",
"Synthesis" to "Field's own signal graph — the part of the audio stack that makes samples " +
"rather than arranging them. One value type carries the whole thing: `Signal` is a " +
"block-rendered mono source, `SignalProcessor` is a `Signal` that reads another one, " +
"and `/` chains the two, so `saw / OnePole(800.p)` is a filtered sawtooth and nothing " +
"else had to be declared. Control values are `Param`, which a plain number becomes " +
"with `.p`, and which can itself be driven by a signal. `audio { … }` closes a mono or " +
"stereo graph into an `AudioPatchSpec`; return one as a box's last expression and the " +
"box stays lit for as long as it is making sound. Underneath is a shared OpenAL " +
"engine, one process-wide, with warm-up passes available so the first buffer is not " +
"also the JIT's first look at your graph. On top of the oscillator / filter / effect / " +
"envelope families sit three larger things that are not Field's own DSP: **Faust**, " +
"whose compiler is embedded as WebAssembly and turns a `#faust` box into a running " +
"node in the same graph; the **dough** ports, hand-rolled float math whose " +
"inaccuracy is part of the sound; and the **spectral** " +
"and **granular** layers, which are Field's own. `Visualizers.kt` is how you look at " +
"any of it — every function there is an extension on a type declared in this " +
"category. Start from `synth.kt`, which is the terse surface a box actually imports; " +
"[Sound: from a pattern to something you can hear](../sound.md) is the prose, " +
"[Faust](../faust.md) is the embedded compiler with worked examples, and " +
"[DoughDSP: hear the primitives](../dough_dsp_sound.md) plays the ports.",
"Patterns" to "Musical time, as a value. A `PatPattern<A>` is a function from a span of " +
"cycles to the events inside it, so it is queried rather than stored, and a pattern " +
"that never repeats costs no more to hold than one that does. Time is kept as an exact " +
"rational (`PatQ`, `BigInteger` over `BigInteger`) all the way through the transforms " +
"and is only converted to seconds at the boundary where something has to be " +
"scheduled, which is why a third of a cycle survives `fast`, `rev` and `every` without " +
"drifting. The notation and much of the vocabulary is Tidal's — `pat(\"bd [hh hh] " +
"sd\")`, `stack`, `jux`, `off`, `degradeBy`, the `|+|` operator family — but an atom " +
"here means whatever you want it to mean, not a sample name, which is what lets the " +
"same pattern drive a note process, a Field `Fn`, a set of signal-graph parameters or " +
"an arbitrary side effect. `Sequencing.kt` is the note-and-piano-roll layer patterns " +
"bridge into, and `BeatLaunch.kt` is `.onBeat { … }`, which holds any launchable value " +
"until the next musical boundary. " +
"[Patterns: writing rhythm as a function of time](../pattern_language.md) is the prose " +
"for all of it.",
"Audio" to "Sound that arrives from, or leaves for, somewhere other than Field's own " +
"signal graph: analysis products written by an offline tool, the network protocols " +
"that carry control data, the Dirt sample library on disk, an Ableton Live set read " +
"off the filesystem, and Sonic Pi driven as an owned child process with no window of " +
"its own. `SoundPathMap.kt` is the one to read first: the declared, " +
"re-derived-from-the-code map of every route Field has from a box to something " +
"audible, which [Making a sound in Field: every path there is](../sound_paths.md) " +
"renders as a page. [Sonic Pi](../sonic_pi.md) covers owned mode in full.",
"Audio AI" to "The client half of Field Audio: a separate HTTP/WebSocket service that runs " +
"generative and analysis models over audio, and the typed vocabulary a box talks to it " +
"in. Nothing here loads a model or does inference in the JVM — a box builds a " +
"`ModelOperation`, submits it through `FieldAudioClient`, and gets back a job it can " +
"poll or watch; the results are immutable server-side *assets* referenced by id, which " +
"is why the protocol types carry representation ids rather than samples. Every call is " +
"asynchronous and completes off Field's runloop and off the audio callback, so nothing " +
"here may be `.get()`-ed from a box's frame. The rest of the category is what you do " +
"with a result inside a box: the candidate browser, the EnCodec reconstruction " +
"microscope, the beat/pitch analysis viewer and the job monitor, each of them one " +
"function returning a self-contained HTML fragment for `display(…)`. Start from " +
"`Protocol.kt` for the vocabulary and `FieldAudioClient.kt` for the verbs; " +
"[Field Audio service reference](../field_audio_service_reference.md) is the " +
"service-side contract the types mirror.",
LANGUAGES to "Field's small languages, from the host side. A box can be *written* in one " +
"of them — a per-box property picks the language and a `BoxExecution` runs it, which is " +
"how `datalog`, `forth`, `nano`, `morpho`, `clojure`, `python`, FieldML and the two " +
"`remotejs` runtimes work — or a program can be *embedded*, a triple-quoted Kotlin " +
"string whose first line names the language and whose contents are compiled on the JVM, " +
"which is how RectSL and StackSL work. What is documented below is the Kotlin that " +
"either shape hands you: the compiler you give a program to, the result and the " +
"diagnostics it gives back, and the bridge a running program calls host values through. " +
"The notation itself is on the cheatsheets, and that is where the pictures are. Read " +
"`Datalog.kt` first — it is the smallest complete example of the shape all of these " +
"have, a value you add source to, query, and then ask `why` of — then " +
"`DatalogEngine.kt` for what the implementation underneath actually costs, `StackSL.kt` " +
"for the compile-and-diagnose surface both embedded languages share, " +
"`ForthInterpreter.kt` and `NanoVM.kt` for the two implementations you can read end to " +
"end in a sitting, and `HostBridge.kt` for FieldML's reflective JVM bridge — the file " +
"to read before wiring any language to Kotlin values. `dev_docs/` carries the map of " +
"what a new language has to touch on the way in, in `small_language_anatomy.md`, " +
"which is also the guarded list of which languages exist. The prose, with pictures: " +
"[Language extensions](../language_extensions.md) for the route decision, then " +
"[Datalog](../datalog.md), [RectSL](../rectsl_cheatsheet.md), " +
"[StackSL](../stacksl_cheatsheet.md), [FieldML](../fieldml_cheatsheet.md), " +
"[MorphoHDL](../morpho_examples.md), [Nano](../nano_examples.md), " +
"[Forth](../forth_examples.md) and [Python](../python.md).",
"Workspace" to "The box's own surroundings: properties, persistence, and the handful of " +
"names every box already has in scope.",
"Machine learning" to "Models small enough to train inside a box, in plain Kotlin, while " +
"you watch. Flow matching carries one 2D or 3D point cloud onto another and hands back " +
"the deformation between them; the multi-output regression head it trains is usable " +
"on its own. See [Flow Matching](../flow_matching.md).",
)Docs pages that are not worked examples and would only add noise as "see also" targets.
private val EVIDENCE_PAGE_BLOCKLIST = setOf(
"index.md", "installation.md", "survivalGuide.md", "funFeatures.md", "aiFeatures.md",
"staging.md", "building_and_testing.md",
)fun main(args: Array<String>)
body — 115 lines, from line 712
@JvmStatic
fun main(args: Array<String>) {
val repo = locateRepoRoot()
val explicitOut = args.firstOrNull()?.takeIf { it.isNotBlank() }
?: System.getProperty("afield.codebaseReference.out")?.takeIf { it.isNotBlank() }
val outRoot = explicitOut?.let { File(it) }
?: File.createTempFile("codebase-reference", "").let { probe ->
probe.delete(); probe.mkdirs(); probe
}
outRoot.mkdirs()
val sources = collectSources(repo)
if (sources.isEmpty()) {
System.err.println("StdlibReferenceGenerator: no sources found under ${repo.absolutePath}")
exitProcess(1)
}
val documents = sources.map { readDocument(repo, it) }
fileAnchors = documents.groupBy { it.category to slug(it.fileName) }.values.flatMap { group ->
group.sortedBy { it.relativePath }.mapIndexed { index, doc ->
doc.relativePath to if (index == 0) slug(doc.fileName)
else "${slug(doc.fileName)}-${slug(doc.relativePath.substringBeforeLast('/'))}"
}
}.toMap()
val typeAnchors = buildTypeAnchorIndex(documents)
val evidence = buildEvidenceIndex(repo, documents)
val pages = linkedMapOf<String, String>()
val categories = CATEGORY_BLURBS.keys.filter { category -> documents.any { it.category == category } } +
documents.map { it.category }.distinct().filter { it !in CATEGORY_BLURBS.keys }.sorted()
categories.forEach { category ->
val inCategory = documents.filter { it.category == category }.sortedBy { it.relativePath }
pages["${slug(category)}.md"] = renderCategoryPage(category, inCategory, typeAnchors, evidence)
}
pages["index.md"] = renderIndexPage(categories, documents, typeAnchors, evidence, File(repo, LOC_HERO_REL))
pages.forEach { (name, text) -> File(outRoot, name).writeText(text) }
// The worklist is committed outside the reference dir, so it needs its own destination:
// beside the pages when they are going to a scratch directory (so a probe run is
// self-contained), and to its real home only when the committed pages are being rewritten.
val worklist = renderKdocWorklist(documents, evidence)
val writingCommitted = explicitOut != null &&
outRoot.canonicalFile == File(repo, REFERENCE_REL).canonicalFile
val worklistOut =
if (writingCommitted) File(repo, WORKLIST_REL) else File(outRoot, "kdoc_worklist.md")
worklistOut.parentFile?.mkdirs()
worklistOut.writeText(worklist)
println("StdlibReferenceGenerator: KDoc worklist -> ${worklistOut.absolutePath}")
val symbolCount = documents.sumOf { it.symbolCount() }
println(
"StdlibReferenceGenerator: ${pages.size} page(s), ${documents.size} source file(s), " +
"$symbolCount public symbol(s) -> ${outRoot.absolutePath}"
)
categories.forEach { category ->
val inCategory = documents.filter { it.category == category }
println(
" ${slug(category)}.md — ${inCategory.size} file(s), " +
"${inCategory.sumOf { it.symbolCount() }} symbol(s)"
)
}
val failures = mutableListOf<String>()
if (symbolCount < 500) failures += "only $symbolCount symbols emitted — the parse looks broken"
documents.filter { it.category == UNCATEGORISED }.forEach {
failures += "${it.relativePath}: no @file:CompletionCategory, so it would be published " +
"on a page called \"$UNCATEGORISED\" — add the annotation its package siblings carry"
}
pages.forEach { (name, text) ->
if (text.length < 200) failures += "$name is suspiciously short (${text.length} chars)"
}
// Before the drift guard on purpose — see the comment above [PLAUSIBLE_LINK_TARGET]. The
// populations these two select, measured over the 20 committed pages at 4546d72b5:
// 12675 `](…)` occurrences (11324 distinct targets), 0 rendering an impossible link; and
// 416 lines inside ``` fences, 1 carrying a backtick and grandfathered by exact text.
failures += impossiblePathLinkFailures(pages)
failures += fenceBacktickFailures(pages)
// Drift guard: the FIELD_TEST path (no explicit output dir) additionally asserts that the
// committed pages are exactly what the current source produces.
if (failures.isEmpty() && explicitOut == null) {
val committed = File(repo, REFERENCE_REL)
if (committed.isDirectory) {
println("StdlibReferenceGenerator: drift-checking committed pages at ${committed.absolutePath}")
pages.forEach { (name, text) ->
val existing = File(committed, name)
when {
!existing.isFile -> failures += "$name: missing from $REFERENCE_REL"
existing.readText() != text -> failures += "$name: committed page is out of date"
}
}
committed.listFiles { f -> f.isFile && f.name.endsWith(".md") }
?.map { it.name }
?.filter { it !in pages.keys }
?.forEach { failures += "$it: committed but no longer generated — delete it" }
val committedWorklist = File(repo, WORKLIST_REL)
when {
!committedWorklist.isFile -> failures += "$WORKLIST_REL: missing"
committedWorklist.readText() != worklist ->
failures += "$WORKLIST_REL: committed worklist is out of date"
}
if (failures.isNotEmpty()) failures += "Regenerate with: ./gradlew generateCodebaseReference"
} else {
println("StdlibReferenceGenerator: no committed pages at ${committed.absolutePath} — skipping drift check")
}
}
if (failures.isNotEmpty()) {
System.err.println("StdlibReferenceGenerator FAILED with ${failures.size} problem(s):")
failures.forEach { System.err.println(" - $it") }
exitProcess(1)
}
println("StdlibReferenceGenerator ok — $symbolCount symbol(s) across ${pages.size} page(s)")
}model
One public declaration, ready to render.
private data class Symbol(
val kind: String,
/** Qualified as the parser saw it: `Owner.member` for members, bare for top level. */
val qualifiedName: String,
val simpleName: String,
val owner: String?,
val signature: String,
val summary: String,
val line: Int,
)One source file's worth of reference material.
source — 56 lines, from line 843
private data class SourceDocument(
val relativePath: String,
val fileName: String,
val category: String,
val implementation: Boolean,
val overview: String?,
val types: List<TypeEntry>,
val topLevelFunctions: List<Symbol>,
val topLevelValues: List<Symbol>,
/**
* The lines [SourcePageGenerator] will give a heading — and therefore an `#L…` anchor — on
* this file's source page.
*
* Held rather than assumed because the two parsers do not agree on every line, and it is
* cheaper to ask than to publish a link that scrolls nowhere. A `data class`'s constructor
* properties are the common case: `KotlinSourceIndex` reports each as a declaration on its own
* line, the source page renders them as one signature, so only the class itself has an anchor.
*/
val sourceAnchors: Set<Int>,
/** The file's `package`, empty for the default package. */
val packageName: String = "",
/** The file's `import` targets, as written (`a.b.C`, `a.b.*`); aliased imports are dropped. */
val imports: List<String> = emptyList(),
) {
fun allSymbols(): List<Symbol> =
types.map { it.symbol } + types.flatMap { it.members } + topLevelFunctions + topLevelValues
fun symbolCount(): Int = allSymbols().size
/** How many of this file's public symbols actually carry a KDoc summary. */
fun documentedCount(): Int = allSymbols().count { it.summary.isNotBlank() }
/**
* Search patterns that, if found in a docs page's *code*, mean the page really uses this
* file's API.
*
* Only entry points count — top-level functions, types and values, never members. A member
* name like `transform`, `origin` or `previous` is an ordinary English word that turns up in
* unrelated example code; a top-level call `stft(` or a type `Spectrogram` does not.
*
* [GENERIC_NAMES] is excluded on top of that, because "exactly one file defines it" is
* not the same as "seeing it means that file". It applies to **values as well as calls**:
* `synth.kt`'s whole design is that a bare lowercase `val` is a fresh unit generator, so it
* declares top-level `square`, `pulse`, `white`, `brown` and `impulse`, and `square` was
* already in the set — as a call name only, which is not where it was doing any harm.
*/
fun evidenceTokens(): Set<Pair<String, String>> = buildSet {
topLevelFunctions.forEach {
if (it.simpleName.length >= 3 && it.simpleName !in GENERIC_NAMES) add("call" to it.simpleName)
}
types.forEach { if (it.symbol.simpleName.length >= 4) add("type" to it.symbol.simpleName) }
topLevelValues.forEach {
if (it.simpleName.length >= 5 && it.simpleName !in GENERIC_NAMES) add("name" to it.simpleName)
}
}
}Names that are never evidence of anything, however uniquely a file owns them.
The evidence index counts a name only when exactly one documented file declares it — which
looked like enough until CeresExpression.kt was published. That file declares top-level
exp, cos, tan, min, max, abs, sqrt, pow and atan2 as expression builders, and
it is the only file in the corpus that declares them at top level, so it "uniquely owned" the
whole kotlin.math vocabulary. Every docs page with arithmetic in a code fence matched two or
more, and functions_examples.md — which has nothing whatsoever to do with the Ceres
expression VM — was published as a worked example of it.
afield.audio.synth is the second instance of the same class, and a broader one, because a
live-coding audio API is deliberately written in one-word English. Measured (see
AFIELD_EVIDENCE_DEBUG in buildEvidenceIndex) on the day the package was added:
synth.kt was published as a worked example of polytope_examples.md on the strength of
plate(, square and white — a panel, a face and a colour — and of functions_examples.md
on square and pulse. PatSignals.kt matched petri_nets.md on run( and range(;
PatAlgebra.kt matched three unrelated pages on with(, which is a Kotlin scope function.
Uniqueness is a claim about the corpus; this set is a claim about the reader. Domain words
that happen to be short or ordinary-looking stay in (kis, dual, stft, zaw, pulze,
dirt, plateReverb): seeing one really does mean the page is about that file. Removing a
word degrades gracefully rather than deleting a link — a page that genuinely uses a file
matches several specific names as well, and MIN_EVIDENCE_HITS is only two, so the true
relationships in this corpus survive losing off and scale while the false ones do not.
private val GENERIC_NAMES = setOf(
"abs", "acos", "asin", "atan", "atan2", "ceil", "cos", "cosh", "exp", "floor", "hypot",
"ln", "log", "log10", "log2", "max", "min", "pow", "round", "sign", "sin", "sinh", "sqrt",
"tan", "tanh",
// Not kotlin.math, but words an unrelated code fence uses about anything.
"constant", "square", "reciprocal",
// Kotlin scope functions and ordinary member names that a top-level declaration shadows.
// Every one of these was verified to be matching something else: `with(` is Kotlin's,
// `net.run(` is a Petri net, `event.num(` is a member of `PatScheduledEvent`, and
// `rig.params(…).scale(` and `fn.scale(` are two other `scale`s entirely.
"with", "run", "num", "scale",
// `synth.kt`'s one-word unit-generator vals, which on a geometry, function or imaging page
// mean a panel, a beat, a filter response and two colours.
"plate", "pulse", "impulse", "white", "brown",
// The archetypal member names, which `stdlib.kt` happens to also declare at top level. Once
// the `\b` anchor stopped `set(` matching `offset(`, these alone were enough to publish
// `afield_four_graphics.md` and `ar_reference.md` as worked examples of `stdlib.kt` on the
// strength of `mesh.set(`, `list.add(` and `map.get(`. What is left for `stdlib.kt` is
// `Delegation` / `Inheritance` / `InheritanceFromContext` in `running_code.md`, which
// is the one page that really is about it.
"set", "add", "get", "end", "start", "invoke", "begin",
)
private data class TypeEntry(
val symbol: Symbol,
val members: List<Symbol>,
val enumConstants: List<String> = emptyList(),
)reading
source — 69 lines, from line 957
private fun collectSources(repo: File): List<File> =
SOURCE_ROOTS
.map { File(repo, it) }
.filter { it.isDirectory }
.flatMap { root -> root.walkTopDown().filter { it.isFile && it.extension == "kt" }.toList() }
.sortedBy { it.absolutePath }
private fun readDocument(repo: File, file: File): SourceDocument {
val rel = repo.toPath().relativize(file.toPath()).toString().replace('\\', '/')
val text = file.readText()
val lines = text.split("\n")
val definitions = KotlinSourceIndex.parseTopLevelDefinitions(text)
val overview = fileOverview(lines, definitions.minOfOrNull { it.line })
val overviewOwner = overview?.second
val sourceAnchors = try {
SourcePageGenerator.headedDeclarationLines(KotlinSourcePartition.partition(text))
} catch (t: Throwable) {
// A partition failure costs links, not pages. `checkSourcePartition` is where that gets
// to be a build failure; here it degrades to the unlinked reference this page has always
// been.
emptySet()
}
fun toSymbol(def: KotlinSourceIndex.ParsedDefinition): Symbol {
val summaryRaw = if (def.line == overviewOwner) null else def.kdoc
return Symbol(
kind = def.kind,
qualifiedName = def.name,
simpleName = def.name.substringAfterLast('.'),
owner = def.owner,
signature = signatureAt(lines, def.line, def.kind),
summary = firstParagraph(summaryRaw),
line = def.line,
)
}
val visible = definitions.filter { it.visibility == "public" }
val topLevel = visible.filter { it.owner == null }
val types = topLevel
.filter { it.kind in TYPE_KINDS }
.map { def ->
val members = visible
.filter { it.owner == def.name }
.filterNot { it.simpleName() == "Companion" }
.sortedBy { it.line }
.map(::toSymbol)
val symbol = toSymbol(def)
val constants =
if (symbol.signature.contains("enum class")) enumConstantsAt(lines, def.line) else emptyList()
TypeEntry(symbol, members, constants)
}
val functions = topLevel.filter { it.kind == "fun" }.sortedBy { it.line }.map(::toSymbol)
val values = topLevel.filter { it.kind == "val" || it.kind == "var" }.sortedBy { it.line }.map(::toSymbol)
return SourceDocument(
relativePath = rel,
fileName = file.name,
category = fileCategory(text) ?: inferredCategory(rel),
implementation = isImplementation(rel, text),
overview = overview?.first,
types = types,
topLevelFunctions = functions,
topLevelValues = values,
sourceAnchors = sourceAnchors,
packageName = Regex("""(?m)^\s*package\s+([\w.]+)""").find(text)?.groupValues?.get(1).orEmpty(),
imports = Regex("""(?m)^\s*import\s+([\w.]+(?:\.\*)?)\s*$""").findAll(text).map { it.groupValues[1] }.toList(),
)
}The URL of doc's source page, relative to a reference page, or null when it has none.
Reference pages live in reference/ and source pages in reference/source/, so the whole
relative path is the source/… suffix. Written in the directory-URL form rather than as a .md
link because these end up inside a markdown link whose text is already raw HTML, and zensical
rebases the prefix either way.
Null for a file outside SourcePageGenerator's corpus. Those exist — the reference covers a few files whose category excludes them from the source-page selection — and a link that resolves to nothing is worse than a name that never claimed to be one.
private fun sourcePageUrl(doc: SourceDocument): String? {
if (doc.sourceAnchors.isEmpty()) return null
val page = SourcePageGenerator.pagePathFor(doc.relativePath)
return page.removePrefix("reference/").removeSuffix(".md") + "/"
}text as a link to line on doc's source page, or unchanged when that anchor is not there.
private fun linkToSource(doc: SourceDocument, line: Int, text: String): String {
val url = sourcePageUrl(doc) ?: return text
if (line !in doc.sourceAnchors) return text
return "[$text]($url#L$line)"
}
private fun KotlinSourceIndex.ParsedDefinition.simpleName(): String = name.substringAfterLast('.')
private val TYPE_KINDS = setOf("class", "interface", "object", "typealias")
private fun fileCategory(text: String): String? =
Regex("""(?m)^\s*@file:(?:[\w.]+\.)?CompletionCategory\s*\(\s*"([^"]+)"""")
.find(text)?.groupValues?.getOrNull(1)?.trim()?.takeIf { it.isNotEmpty() }internal fun isImplementation(relativePath: String, text: String): Boolean
Conservative presentation hint; public renderers, textures, audio and language APIs stay prominent.
internal fun isImplementation(relativePath: String, text: String): Boolean {
if (fileCategory(text) == "Internal") return true
if (IMPLEMENTATION_PREFIXES.any { relativePath.startsWith(it) }) return true
val plumbing = listOf(
"app/server/", "app/protocol/", "app/commands/",
"runtime/kotlin/", "runtime/completion/",
"document/persistence/", "document/history/", "tools/",
)
if (plumbing.any { relativePath.startsWith("src/jvmMain/kotlin/afield/$it") }) return true
val name = relativePath.substringAfterLast('/').removeSuffix(".kt")
return name.endsWith("SmokeTest") || name.endsWith("TestMain") ||
name.endsWith("DocsHarness") || name.endsWith("GpuSmoke")
}Last-resort bucket for a file that carries no @file:CompletionCategory.
private fun inferredCategory(relativePath: String): String =
CATEGORY_BY_PATH.firstOrNull { (prefix, _) -> relativePath.startsWith(prefix) }?.second
?: UNCATEGORISEDThe file's own opening KDoc, plus the line of the declaration it would otherwise have been attached to (so that declaration does not repeat it as its summary).
Returns null unless the block ends before firstDeclarationLine and at least one blank or
comment line separates the two — a KDoc flush against a declaration documents that
declaration, not the file.
private fun fileOverview(
lines: List<String>,
firstDeclarationLine: Int?,
): Pair<String, Int?>? {
val start = lines.indexOfFirst { it.trimStart().startsWith("/**") }
if (start < 0) return null
var end = start
while (end < lines.size && "*/" !in lines[end]) end++
if (end >= lines.size) return null
// The block must precede the first declaration in the file.
if (firstDeclarationLine != null && firstDeclarationLine <= end + 1) return null
// …and must not be flush against it.
var probe = end + 1
var sawGap = false
while (probe < lines.size) {
val trimmed = lines[probe].trim()
if (trimmed.isEmpty() || trimmed.startsWith("//") || trimmed.startsWith("@")) {
if (trimmed.isEmpty() || trimmed.startsWith("//")) sawGap = true
probe++
continue
}
break
}
if (!sawGap) return null
val body = lines.subList(start, end + 1).joinToString("\n")
val cleaned = dropBlockTags(stripKDocMarkers(body))
if (cleaned.isBlank()) return null
// The KDocExtractor attaches this same block to the next declaration; remember which one so
// the declaration's own row stays empty rather than echoing the file overview.
val attachedTo = (probe + 1).takeIf { probe < lines.size }
return cleaned to attachedTo
}text with its trailing KDoc block tags (@see, @param, @throws, …) removed.
A symbol's summary is only ever its first paragraph, so tags never reach the page that way; a
file overview is printed whole, and imaging_simd.kt's @see lines were rendering literally as
body text. Only tags at the end of the block are dropped, which is where KDoc puts them —
anything else would silently swallow prose.
private fun dropBlockTags(text: String): String {
val lines = text.split("\n").toMutableList()
var cut = lines.size
for (i in lines.indices.reversed()) {
val trimmed = lines[i].trim()
if (trimmed.isEmpty()) continue
if (BLOCK_TAG.containsMatchIn(trimmed)) { cut = i; continue }
break
}
return lines.subList(0, cut).joinToString("\n").trimEnd()
}
private val BLOCK_TAG = Regex("""^@(param|return|throws|exception|see|since|sample|author|constructor|receiver|property|suppress)\b""")internal fun stripKDocMarkers(block: String): String
internal fun stripKDocMarkers(block: String): String =
block.split("\n").joinToString("\n") { raw ->
var s = raw.trim()
if (s.startsWith("/**")) s = s.removePrefix("/**").trim()
if (s.endsWith("*/")) s = s.removeSuffix("*/").trim()
if (s.startsWith("*")) s = s.removePrefix("*").let { if (it.startsWith(" ")) it.drop(1) else it }
s
}.trim()internal fun signatureAt(lines: List<String>, line: Int, kind: String): String
The declaration's own text, read back from source starting at line (1-based).
The index stores where a declaration is, not what it looks like, so the header is recovered
here by accumulating source until the brackets balance and a body marker ({, =) or the
end of the header is reached. Leading annotations are dropped; whitespace is collapsed.
Comments inside the header are skipped, and do not count against
MAX_SIGNATURE_LINES. A data class that documents each of its primary-constructor
properties — the natural place to put "seconds, not samples" — otherwise publishes its own
KDoc inside its rendered signature, and a long one is then truncated mid-comment with no
closing bracket. AudioVisualizerSource shipped in exactly that state.
body — 88 lines, from line 1168
internal fun signatureAt(lines: List<String>, line: Int, kind: String): String {
if (line - 1 !in lines.indices) return ""
// A typealias *is* its right-hand side, so `=` opens the interesting part rather than ending
// the header. Everywhere else `=` starts an initializer or expression body and is the stop.
val stopAtAssign = kind != "typealias"
val builder = StringBuilder()
var depth = 0
var index = line - 1
var done = false
var codeLines = 0
var inBlockComment = false
// Only a raw literal can still be open at a line break, so it is the only literal state
// carried between lines; [literalEnd] closes the others within the line that opens them.
var inRawString = false
while (index < lines.size && !done && codeLines < MAX_SIGNATURE_LINES) {
val raw = lines[index]
var i = 0
var contributed = false
while (i < raw.length) {
val ch = raw[i]
val previous = raw.getOrNull(i - 1)
if (inRawString) {
val close = raw.indexOf("\"\"\"", i)
val stop = if (close < 0) raw.length else close + 3
builder.append(raw, i, stop)
contributed = true
inRawString = close < 0
i = stop
continue
}
if (inBlockComment) {
if (ch == '*' && raw.getOrNull(i + 1) == '/') {
inBlockComment = false
i += 2
} else {
i++
}
continue
}
when {
// A literal is copied through verbatim before either comment rule can look
// at it: a default value's own text is not code, and reading it as code is
// what published a truncated glob for a default whose real value was a
// recursive json glob. See [literalEnd].
ch == '"' || ch == '\'' -> {
val end = literalEnd(raw, i)
val stop = if (end < 0) raw.length else end
builder.append(raw, i, stop)
contributed = true
inRawString = end < 0
i = stop
continue
}
ch == '/' && raw.getOrNull(i + 1) == '/' -> { i = raw.length; continue }
// Kotlin block comments nest, but a nested one inside a *signature* is not a
// thing that happens; the flat scan is enough and cannot run away.
ch == '/' && raw.getOrNull(i + 1) == '*' -> { inBlockComment = true; i += 2; continue }
ch == '(' || ch == '[' -> { depth++; builder.append(ch); contributed = true }
ch == ')' || ch == ']' -> { depth--; builder.append(ch); contributed = true }
ch == '{' && depth <= 0 -> done = true
ch == '=' && depth <= 0 && stopAtAssign &&
raw.getOrNull(i + 1) != '=' &&
previous != '=' && previous != '!' && previous != '<' && previous != '>' -> done = true
else -> {
builder.append(ch)
if (!ch.isWhitespace()) contributed = true
}
}
if (done) break
i++
}
if (done) break
builder.append(' ')
if (contributed) codeLines++
index++
// A complete header at bracket depth zero ends the scan; a continuation line is only
// consumed while a bracket is still open or the next line clearly continues the header.
if (depth <= 0) {
val next = lines.getOrNull(index)?.trim().orEmpty()
val continues = next.startsWith(":") || next.startsWith(",") ||
next.startsWith("where ") || builder.trim().endsWith("=") ||
(kind == "class" && Regex("""^(?:(?:public|private|protected|internal)\s+)?constructor\b""")
.containsMatchIn(next))
if (!continues) break
}
}
return tidySignature(builder.toString())
}The index just past the string or character literal that starts at start in line, or -1
when a raw """ literal is still open at the end of the line.
signatureAt has to know where literals are before it can recognise a comment, because a default value can spell a comment marker inside a string. The glob default of afield.animation.keyframes.keyFilesMatching did exactly that: a recursive wildcard followed by a slash and a second wildcard opens a block comment as far as a scan that does not know it is inside a string, and the published signature lost the rest of the default silently — it advertised a glob the code does not have. A default URL spells the line-comment marker the same way.
Escapes are honoured, so a literal whose last character is an escaped quote does not close
early. An unterminated ' is treated as an ordinary character rather than swallowing the
line, since a lone quote is far likelier to be broken input than a real char literal.
private fun literalEnd(line: String, start: Int): Int {
if (line.startsWith("\"\"\"", start)) {
val close = line.indexOf("\"\"\"", start + 3)
return if (close < 0) -1 else close + 3
}
val quote = line[start]
var i = start + 1
while (i < line.length) {
when {
line[i] == '\\' -> i += 2
line[i] == quote -> return i + 1
else -> i++
}
}
return if (quote == '\'') start + 1 else line.length
}
private const val MAX_SIGNATURE_LINES = 40internal fun enumConstantsAt(lines: List<String>, line: Int): List<String>
The constants of an enum class declared at line, in source order.
An enum's constants are its API — a reference that lists enum class WindowType and stops
has told the reader nothing. Reads the body between the opening brace and the first ; (or
the closing brace), taking the leading identifier of each depth-one comma-separated entry, so
HANN, HAMMING(2) and a trailing-comma list all come out right.
Comments are stripped first, block comments included, and that is the whole reason this is
not a one-liner. A constant carrying its own KDoc is ordinary style in this codebase, and
KDoc prose is full of commas — so a body that still contains it makes parseEnumEntries read
the first word after every comma as a constant name. Until 2026-09-17 only // was removed,
and the published reference carried lines like Constants: which, because and
Constants: or, and, counted, when for enums whose real constants it never saw. Not a
truncation a reader can spot: a list of plausible-looking lowercase words where the API
should be. EnumConstantCommentTest pins it, and a block comment opened on one line and
closed on another is why the flag lives outside the per-line loop.
(The markers are named in words rather than spelled here, and neither is a glob written over
a reference path, because Kotlin block comments nest: an opener anywhere in a KDoc
block opens a comment that the block's own terminator does not close, and the rest of the
file then parses as declarations inside a comment. Both mistakes were made while writing
this, and the second one — a glob after a slash — is the one that looks completely innocent.)
internal fun enumConstantsAt(lines: List<String>, line: Int): List<String> {
var index = line - 1
if (index !in lines.indices) return emptyList()
val body = StringBuilder()
var depth = 0
var started = false
var inBlockComment = false
while (index < lines.size && index < line - 1 + MAX_ENUM_LINES) {
val (code, stillOpen) = stripComments(lines[index], inBlockComment)
inBlockComment = stillOpen
for (ch in code) {
when {
ch == '{' -> { depth++; if (depth == 1) { started = true; continue } }
ch == '}' -> { depth--; if (depth == 0 && started) return parseEnumEntries(body.toString()) }
ch == ';' && depth == 1 -> return parseEnumEntries(body.toString())
}
if (started && depth >= 1) body.append(ch)
}
body.append('\n')
index++
}
return if (started) parseEnumEntries(body.toString()) else emptyList()
}internal fun stripComments(line: String, blockCommentOpen: Boolean): Pair<String, Boolean>
line with its comments removed, and whether a block comment is still open at the end of it.
Deliberately not string-literal aware, unlike signatureAt's scan: an enum body has no
business containing a string that spells a comment marker, where a parameter default very much
does — that is the hazard SignatureLiteralTest exists for. Keeping this simple means it
cannot mis-handle a case it will never see; if an enum ever does carry such a literal, the
constant names still come out right, because a literal only ever appears inside a constant's
argument list and parseEnumEntries takes the leading identifier.
internal fun stripComments(line: String, blockCommentOpen: Boolean): Pair<String, Boolean> {
val kept = StringBuilder(line.length)
var open = blockCommentOpen
var i = 0
while (i < line.length) {
when {
open -> if (line.startsWith("*/", i)) { open = false; i += 2 } else i++
line.startsWith("/*", i) -> { open = true; i += 2 }
line.startsWith("//", i) -> return kept.toString() to open
else -> { kept.append(line[i]); i++ }
}
}
return kept.toString() to open
}
private const val MAX_ENUM_LINES = 200
private fun parseEnumEntries(body: String): List<String> {
val out = mutableListOf<String>()
var depth = 0
val current = StringBuilder()
fun flush() {
val entry = current.toString().trim()
current.setLength(0)
val name = Regex("^([A-Za-z_][A-Za-z0-9_]*)").find(entry)?.groupValues?.get(1) ?: return
if (name.isNotEmpty() && name !in out) out += name
}
for (ch in body) {
when {
ch == '(' || ch == '[' || ch == '{' -> { depth++; current.append(ch) }
ch == ')' || ch == ']' || ch == '}' -> { depth--; current.append(ch) }
ch == ',' && depth == 0 -> flush()
else -> current.append(ch)
}
}
flush()
return out
}
private val LEADING_ANNOTATION = Regex("""^@\w[\w.]*\s*(\([^)]*\))?\s*""")transform applied to every stretch of signature that is not inside a string or
character literal, with the literals copied through verbatim.
The rewrites in tidySignature are about the shape of a declaration — collapsing the runs
of whitespace a multi-line header leaves behind, closing up the space after an open bracket.
A default value that happens to contain those same spellings is not that shape, and rewriting
it changes what the page says the code does: prefix: String = "( " published as "(" is a
different default. Same argument as literalEnd, one layer later.
private fun outsideLiterals(signature: String, transform: (String) -> String): String {
val out = StringBuilder(signature.length)
val plain = StringBuilder()
var i = 0
while (i < signature.length) {
val ch = signature[i]
if (ch == '"' || ch == '\'') {
val end = literalEnd(signature, i)
val stop = if (end < 0) signature.length else end
out.append(transform(plain.toString()))
plain.setLength(0)
out.append(signature, i, stop)
i = stop
} else {
plain.append(ch)
i++
}
}
out.append(transform(plain.toString()))
return out.toString()
}
private fun tidySignature(raw: String): String {
var s = outsideLiterals(raw) { it.replace(Regex("\\s+"), " ") }.trim()
while (true) {
val m = LEADING_ANNOTATION.find(s) ?: break
if (m.range.first != 0) break
s = s.substring(m.range.last + 1).trimStart()
}
s = outsideLiterals(s) {
it.replace("( ", "(").replace(" )", ")").replace(" ,", ",").replace(",)", ")")
}
// `val x: Int get() = …` stops at the `=`, leaving a dangling accessor: it is not part of
// the declaration a caller writes.
s = s.removeSuffix("get()").trimEnd()
s = s.trimEnd().trimEnd(':').trimEnd()
// A trailing comma survives when a parameter list is cut short by the line budget.
if (s.endsWith(",")) s = s.dropLast(1)
return declegate(s.trim())
}Property delegates whose single type argument is the property's type.
private val PROPERTY_DELEGATES = setOf("Delegation", "Inheritance", "InheritanceFromContext")Rewrites var FLine.color by Delegation<Vector4f> { … } as var FLine.color: Vector4f.
Which delegate stores a box property is an implementation detail; the caller writes
line.color = … and needs the type. For the delegates in PROPERTY_DELEGATES the single
type argument is exactly that type, so this loses nothing and drops a lot of noise. A property
that already states its type, or that uses any other delegate, is left alone.
private fun declegate(signature: String): String {
val byIndex = signature.indexOf(" by ")
if (byIndex < 0) return signature
val head = signature.substring(0, byIndex)
if (':' in head) return signature
val delegate = signature.substring(byIndex + 4).trimStart()
val open = delegate.indexOf('<')
if (open <= 0) return signature
if (delegate.substring(0, open) !in PROPERTY_DELEGATES) return signature
var depth = 0
for (i in open until delegate.length) {
when (delegate[i]) {
'<' -> depth++
'>' -> {
depth--
if (depth == 0) {
val type = delegate.substring(open + 1, i).trim()
return if (type.isEmpty()) signature else "$head: $type"
}
}
}
}
return signature
}The KDoc's first paragraph, as a single line of markdown.
Stops at the first blank line or the first block tag (@param, @return, …) — the same
"summary sentence" convention KDoc itself uses. Never synthesises text: no KDoc means "".
private fun firstParagraph(kdoc: String?): String {
if (kdoc.isNullOrBlank()) return ""
val out = mutableListOf<String>()
for (raw in kdoc.split("\n")) {
val s = raw.trim()
if (s.isEmpty()) { if (out.isNotEmpty()) break else continue }
if (s.startsWith("@")) break
if (s.startsWith("```")) break
out += s
}
return out.joinToString(" ").replace(Regex("\\s+"), " ").trim()
}overload coalescing
One table row: a canonical declaration, plus the sibling overloads it stands in for.
@param alsoAccepts alternative parameter types the collapsed overloads accept, first seen first @param extraOverloads how many further overloads this row speaks for (0 when it is alone)
private data class Row(
val symbol: Symbol,
val summary: String,
val alsoAccepts: List<String> = emptyList(),
val extraOverloads: Int = 0,
)Collapses an overload family that only exists to walk Kotlin's numeric tower into one row.
Field's signal API is written three times over — once taking Fn, once Number, once Any —
and again for Domain vs ClosedRange<*>. Six near-identical 20-parameter signatures for
pulseTrain tell the reader nothing the first one did not, and they were the tallest rows on
the page.
Two declarations join a family only when everything except the parameter types matches: same name, same parameter names in the same order, same return type. That is deliberately strict — real overloads differ in arity or in what they name their parameters, so they stay separate rows.
The family is not collapsed when its members carry two different KDoc summaries: that means someone documented them apart, so they are not interchangeable after all.
The canonical member is the one with the most defaulted parameters, then the longest signature, then source order — a total order, so the output is stable for the drift guard.
Most-defaults rather than most-general-types, which is the tempting rule: List<Fn>.sequence
takes overlap: Number = 0.0 in one overload and a mandatory overlap: Fn in the other, and
showing the Fn one would tell the reader that overlap is required when it is not. A wrong
default is a worse error than a narrow type, and the badge names the other types anyway. Where
the general overload is also the fullest — pulseTrain, and most of this corpus — this picks
it regardless.
private fun coalesce(symbols: List<Symbol>): List<Row> {
val groups = LinkedHashMap<String, MutableList<Symbol>>()
symbols.forEachIndexed { index, symbol ->
// Only functions overload; anything unparseable gets a unique key and stays alone.
val key = if (symbol.kind == "fun") overloadKey(symbol.signature) else null
groups.getOrPut(key?.let { "$it@${symbol.owner}" } ?: "unique#$index") { mutableListOf() } += symbol
}
return groups.values.map { family ->
val summaries = family.map { it.summary }.filter { it.isNotBlank() }.distinct()
if (family.size == 1 || summaries.size > 1) {
return@map family.map { Row(it, it.summary) }
}
val canonical = family.withIndex().maxWithOrNull(
compareBy({ defaultedParameterCount(it.value.signature) }, { it.value.signature.length }, { -it.index })
)!!.value
val canonicalTypes = parameterTypes(canonical.signature)
val alternatives = LinkedHashSet<String>()
family.filter { it !== canonical }.forEach { other ->
parameterTypes(other.signature).forEachIndexed { position, type ->
if (type.isNotBlank() && canonicalTypes.getOrNull(position) != type) alternatives += type
}
}
listOf(
Row(
symbol = canonical,
summary = summaries.firstOrNull().orEmpty(),
alsoAccepts = alternatives.toList(),
extraOverloads = family.size - 1,
)
)
}.flatten()
}Identity of an overload family: name, parameter names in order, return type. Types erased.
private fun overloadKey(signature: String): String? {
val (head, params, tail) = splitSignature(signature) ?: return null
val names = params.map { it.substringBefore(':').substringBefore('=').trim() }
if (names.any { it.isEmpty() }) return null
return "${head.trim()}(${names.joinToString(",")})$tail"
}
private fun defaultedParameterCount(signature: String): Int =
splitSignature(signature)?.second?.count { '=' in it } ?: 0
private fun parameterTypes(signature: String): List<String> =
splitSignature(signature)?.second?.map {
if (':' in it) it.substringAfter(':').substringBefore('=').trim() else ""
} ?: emptyList()internal fun parameterNames(signature: String): Set<String>
The parameter names of one signature, as a KDoc written above it refers to them.
vararg, crossinline, an annotation and a constructor's val/var are all stripped, so
fun f(vararg xs: Int) gives xs; a backticked name is unwrapped. See
TypeAnchors.resolveReference for what the set is for.
Only a segment carrying its own : counts, which is what keeps a function type out. On
val handler: (Box) -> Unit the first ( splitSignature finds belongs to the type, so its
one segment is the bare Box; reading that as a parameter would take Box out of scope for
the whole summary.
internal fun parameterNames(signature: String): Set<String> =
splitSignature(signature)?.second.orEmpty().mapNotNull { param ->
if (':' !in param) return@mapNotNull null
param.substringBefore(':')
.trim().substringAfterLast(' ').trim('`')
.takeIf { it.matches(Regex("""[A-Za-z_]\w*""")) }
}.toSet()A signature split into everything before the parameter list, the top-level parameters, and
everything after — respecting nesting, so (Fn) -> Double and Map<String, Int> survive.
The > of an arrow closes nothing, which has to be said explicitly because the same scan
counts the > of a generic argument. Reading an arrow as a closing bracket brings the depth
to zero at the first function-typed parameter and cuts the list off there:
keyframeSlider(…, get: () -> Double, set: (Double) -> Unit) reported get as its last
parameter and tryAll reported three of its eight.
private fun splitSignature(signature: String): Triple<String, List<String>, String>? {
val open = signature.indexOf('(')
if (open < 0) return null
val params = mutableListOf<String>()
val current = StringBuilder()
var depth = 0
for (i in open until signature.length) {
val ch = signature[i]
when (ch) {
'(', '[', '<' -> { depth++; if (depth > 1) current.append(ch) }
')', ']', '>' -> {
if (ch == '>' && i > open && signature[i - 1] == '-') current.append(ch)
else {
depth--
if (depth == 0) {
if (current.isNotBlank()) params += current.toString()
return Triple(signature.substring(0, open), params, signature.substring(i + 1))
}
current.append(ch)
}
}
',' -> if (depth == 1) { params += current.toString(); current.setLength(0) } else current.append(ch)
else -> current.append(ch)
}
}
return null
}cross-references
internal data class AnchorEntry
One published declaration as TypeAnchors sees it: where it lives, what a KDoc reference
may call it, and the page.md#anchor its row carries.
@param owner the type a member is declared in (Owner of Owner.member), null at top level
@param receiver an extension's receiver type, simple name, null for anything else
@param isType a type heading rather than a table row; only these resolve through an unknown
capitalised qualifier, see TypeAnchors.resolveReference
internal data class AnchorEntry(
val file: String,
val name: String,
val owner: String?,
val receiver: String?,
val isType: Boolean,
val target: String,
val pkg: String = "",
)internal data class FileScope(val pkg: String, val imports: List<String>)
What a file can see without qualifying: its own package and its imports.
internal class TypeAnchors(entries: List<AnchorEntry>, private val scopes: Map<String, FileScope> = emptyMap())
Where a KDoc [Symbol](../../../../../../reference/source/jvmMain/afield/document/persistence/edn.md#L45) reference is allowed to point, resolved in two steps.
A reference resolves inside the file that wrote it first, and against the whole corpus only
if that fails. Both halves earn their place. The corpus-wide half is what lets one file's KDoc
link a type another file declares, which is most of the linking on these pages. The file-local
half is what NodeRef means when the file saying it declares a NodeRef: Kotlin resolves a
KDoc reference in the scope it was written in, so a name that is ambiguous across the corpus is
not ambiguous there, and treating it as ambiguous silently downgrades a correct link to plain
code.
That is not hypothetical, it is why this class exists. afield.stdlib/interactionnets.kt
declares a NodeRef and so does afield.lang.rectsl; the moment the second one was published, five
already-resolved links in interactionnets.kt's own KDoc turned back into inline code on
visualization.md — a page nobody had touched. Corpus-wide-only resolution makes every new
source root a possible silent un-linking of the pages already shipped.
Why a qualified reference is checked against its qualifier
Before every row had an anchor only top-level types did, so a KDoc reference to a nested one —
Outer.Nested — had nothing to resolve to, and linkify used to fall back to the last segment alone. That is
a false link waiting for a name collision, and adding the simulation roots supplied one:
afield.stdlib/tuio.kt documents TuioReceiver.Frame, TuioReceiver.Frame is a nested
data class with no anchor, and afield.simulation.physics/CSVReader.kt declares a top-level
Frame for motion-capture samples. The name matched, so a sentence on data.md about a TUIO
cursor frame acquired a link into the physics page — on a page nobody had edited, from a
change in a different package.
resolveReference closes it with the narrowest rule that does: when a reference is qualified
and this reference knows the qualifier, the target has to live in the same source file the
qualifier does, because a nested type cannot live anywhere else. An unknown qualifier — a
package name, a JDK type — still falls back to the last segment, so [java.nio.ByteBuffer](../../../../../../reference/source/jvmMain/afield/math/linalg/stdlib.md#L219)
behaves exactly as before.
internal class TypeAnchors(entries: List<AnchorEntry>, private val scopes: Map<String, FileScope> = emptyMap()) {Every top-level declaration of an ambiguous bare name, for scoped.
private val candidates = mutableMapOf<String, MutableList<AnchorEntry>>()
private val corpus = mutableMapOf<String, AnchorEntry>()
private val byFile = mutableMapOf<String, MutableMap<String, AnchorEntry>>()
private val qualifiedCorpus = mutableMapOf<String, AnchorEntry>()
private val qualifiedByFile = mutableMapOf<String, MutableMap<String, AnchorEntry>>()init
body — 48 lines, from line 1710
init {
val ambiguous = mutableSetOf<String>()
val ambiguousQualified = mutableSetOf<String>()
val fileMembers = mutableMapOf<String, MutableMap<String, MutableSet<String>>>()
// Types first, then top level, then members, so within one file a type outranks the
// factory function sharing its name and a top-level name outranks a member's.
val ordered = entries.filter { it.isType } +
entries.filter { !it.isType && it.owner == null } +
entries.filter { !it.isType && it.owner != null }
ordered.forEach { e ->
val local = byFile.getOrPut(e.file) { mutableMapOf() }
if (e.owner == null) {
// A bare name resolves corpus-wide only for top-level declarations: Kotlin never
// resolves `[member]` from outside the member's own scope.
val prior = corpus.putIfAbsent(e.name, e)
if (prior != null && prior.file != e.file) ambiguous += e.name
local.putIfAbsent(e.name, e)
val list = candidates.getOrPut(e.name) { mutableListOf() }
if (list.none { it.file == e.file }) list += e
} else {
fileMembers.getOrPut(e.file) { mutableMapOf() }
.getOrPut(e.name) { mutableSetOf() } += e.owner
local.putIfAbsent(e.name, e)
}
val qualifier = e.owner ?: e.receiver
if (qualifier != null) {
val key = "${qualifier.substringAfterLast('.')}.${e.name}"
val prior = qualifiedCorpus.putIfAbsent(key, e)
if (prior != null && prior.file != e.file) ambiguousQualified += key
qualifiedByFile.getOrPut(e.file) { mutableMapOf() }.putIfAbsent(key, e)
}
}
// A bare member name two types of one file share is ambiguous inside that file too.
fileMembers.forEach { (file, names) ->
names.filterValues { it.size > 1 }.keys.forEach { name ->
val local = byFile[file] ?: return@forEach
if (local[name]?.owner != null) local.remove(name)
}
}
// A name that two files both define cannot be linked unambiguously from a third, so from
// a third it is not linked at all. From either of the two it still resolves.
ambiguous.forEach { corpus.remove(it) }
candidates.keys.retainAll(ambiguous)
ambiguousQualified.forEach { qualifiedCorpus.remove(it) }
}
private fun lookup(name: String, file: String?): AnchorEntry? =
file?.let { byFile[it]?.get(name) } ?: corpus[name] ?: scoped(name, file)An ambiguous bare name, settled the way Kotlin would settle it from file: a declaration in
the file's own package, else an explicit import of it, else a star import of its package.
Each step links only when it picks exactly one; otherwise the name stays unlinked.
private fun scoped(name: String, file: String?): AnchorEntry? {
val scope = file?.let { scopes[it] } ?: return null
val all = candidates[name] ?: return null
val steps = listOf<(AnchorEntry) -> Boolean>(
{ it.pkg == scope.pkg },
{ "${it.pkg}.$name" in scope.imports },
{ "${it.pkg}.*" in scope.imports },
)
for (step in steps) {
val hits = all.filter(step)
if (hits.size == 1) return hits.single()
if (hits.size > 1) return null
}
return null
}fun resolveReference(raw: String, file: String?, locals: Set<String> = emptySet()): String?
A whole KDoc reference — Symbol, Owner.member, Receiver.extension, Outer.Nested or
some.package.Symbol — resolved to its page.md#anchor, or null.
Null both when nothing matches and when the qualifier contradicts the match; the caller renders inline code either way.
locals are the names the referring KDoc settles in its own scope before any index is
consulted — in practice the parameter names of the declaration it sits above. Kotlin
resolves a parameter name to the parameter, so a name in this set resolves to nothing and
publishes as plain code.
Without it a param reference reaches the index at all, because a parameter is not a
declaration in any file: it misses byFile and corpus answers with whatever top-level
symbol happens to share the name. leniaFromLifeRule(rule: LifeRule) in
alife_continuous.kt names its own parameter in its summary, and the row published on the
alife page sent that name to rewritedsl.kt's rewrite-rule builder on the workspace page.
885 published links across 25 pages pointed off their own page this way.
The set applies to a bare reference only: Owner.name still names a member, whatever
the row's parameters are called.
fun resolveReference(raw: String, file: String?, locals: Set<String> = emptySet()): String? {
val simple = raw.substringAfterLast('.')
if (simple == raw) return if (raw in locals) null else lookup(simple, file)?.target
val qualifier = raw.substringBeforeLast('.').substringAfterLast('.')
val key = "$qualifier.$simple"
(file?.let { qualifiedByFile[it]?.get(key) } ?: qualifiedCorpus[key])?.let { return it.target }
val target = lookup(simple, file)?.takeIf { it.owner == null } ?: return null
val qualifierOwner = lookup(qualifier, file)?.takeIf { it.isType }?.file
if (qualifierOwner != null) return target.target.takeIf { qualifierOwner == target.file }
// An unknown qualifier. A package (`afield.foo.bar`) is lowercase and falls back to the
// last segment as it always has; an unknown *type* (`List.add`, `ByteBuffer.flip`) names
// a member of something this reference does not emit, so only a type may match it.
return target.target.takeIf { target.isType || qualifier.first().isLowerCase() }
}
}internal fun receiverOf(signature: String, name: String): String?
An extension's receiver type, simple name, from its signature, or null for a non-extension.
fun <T> List<T>.foo( gives List; val Box.bar gives Box.
internal fun receiverOf(signature: String, name: String): String? {
val m = Regex("""\b(?:fun|va[lr])\s+(?:<[^=(]*?>\s+)?([\w.<>?,* ]+?)\.`?""" + Regex.escape(name) + """`?\s*[(:=<]?""")
.find(signature) ?: return null
val receiver = m.groupValues[1].trim().removeSuffix("?").replace(Regex("<.*>"), "").trim()
return receiver.substringAfterLast('.').takeIf { it.matches(Regex("""[A-Za-z_]\w*""")) }
}Every row anchor of one file, keyed by owner to name: the first row of an overload family
carries it, so name lands on the family. A slug collision (case, _ vs -) is resolved by
a -2, -3 suffix in source order, so an anchor changes only when its own colliding twin moves.
private fun rowAnchors(doc: SourceDocument): Map<Pair<String?, String>, String> =
rowAnchorCache.getOrPut(doc.relativePath) {
val taken = doc.types.map { typeAnchor(doc, it.symbol) }.toMutableSet()
val out = LinkedHashMap<Pair<String?, String>, String>()
(doc.topLevelFunctions + doc.topLevelValues + doc.types.flatMap { it.members }).forEach { s ->
val key = s.owner to s.simpleName
if (key in out) return@forEach
val base = if (s.owner == null) "${fileAnchor(doc)}--${slug(s.simpleName)}"
else "${fileAnchor(doc)}--${slug(s.owner)}--${slug(s.simpleName)}"
var anchor = base
var n = 2
while (!taken.add(anchor)) anchor = "$base-${n++}"
out[key] = anchor
}
out
}
private val rowAnchorCache = mutableMapOf<String, Map<Pair<String?, String>, String>>()
private val emittedRowAnchors = mutableMapOf<String, MutableSet<Pair<String?, String>>>()Every published declaration -> the page.md#anchor that documents it, for KDoc [Symbol](../../../../../../reference/source/jvmMain/afield/document/persistence/edn.md#L45) linkification.
private fun buildTypeAnchorIndex(documents: List<SourceDocument>): TypeAnchors {
val entries = mutableListOf<AnchorEntry>()
documents.forEach { doc ->
val page = "${slug(doc.category)}.md#"
doc.types.forEach { type ->
entries += AnchorEntry(doc.relativePath, type.symbol.simpleName, null, null, true,
page + typeAnchor(doc, type.symbol), doc.packageName)
}
val anchors = rowAnchors(doc)
(doc.topLevelFunctions + doc.topLevelValues + doc.types.flatMap { it.members }).forEach { s ->
val anchor = anchors[s.owner to s.simpleName] ?: return@forEach
// A nested type listed as a member keeps its member row: `[Outer.Nested]` lands there.
entries += AnchorEntry(doc.relativePath, s.simpleName, s.owner,
if (s.owner == null) receiverOf(s.signature, s.simpleName) else null, false, page + anchor,
doc.packageName)
}
}
return TypeAnchors(entries, documents.associate { it.relativePath to FileScope(it.packageName, it.imports) })
}For each source file, the worked-example pages that demonstrably use it.
Evidence, not opinion. A page qualifies only when at least MIN_EVIDENCE_HITS distinct symbol
names from that file appear in the page's code — fenced blocks and inline code spans, not
prose — and only names that are unique to one source file in the whole corpus count. Both
filters matter: matching prose made every page look like it was about Functions.kt (the word
"function"), and shared names like update or points matched everywhere.
The consequence is that this can never claim a relationship the pages do not have, and a new example page starts being linked the moment it uses the API.
source — 51 lines, from line 1888
private fun buildEvidenceIndex(repo: File, documents: List<SourceDocument>): Map<String, List<Pair<String, Int>>> {
val docsDir = File(repo, "public/docs")
val pages = (docsDir.listFiles { f -> f.isFile && f.extension == "md" } ?: emptyArray())
.filter { it.name !in EVIDENCE_PAGE_BLOCKLIST }
.sortedBy { it.name }
.map { file ->
val text = file.readText()
pageTitles[file.name] = headingOf(text) ?: prettifyFileName(file.name)
file.name to codeTextOf(text)
}
// A name defined by more than one documented file cannot identify any of them.
val tokenOwners = mutableMapOf<Pair<String, String>, MutableSet<String>>()
documents.forEach { doc ->
doc.evidenceTokens().forEach { tokenOwners.getOrPut(it) { mutableSetOf() } += doc.relativePath }
}
// Set `AFIELD_EVIDENCE_DEBUG` to a path suffix (`synth/synth.kt`, or `.kt` for everything) to
// print which tokens each page matched. "Used in" is a published factual claim about a page,
// and this is how you check one without re-deriving the rule by hand — which is how
// `synth.kt` was found being published as a worked example of `polytope_examples.md`.
val debugFor = System.getenv("AFIELD_EVIDENCE_DEBUG")?.trim()?.ifBlank { null }
return documents.associate { doc ->
val kept = doc.evidenceTokens().filter { tokenOwners[it]?.size == 1 }
val patterns = kept.associateWith { (kind, name) ->
// Two tightenings, each fixing a measured false claim:
// - `\b` on the call form. Unanchored, `ply(` matched `apply(`, `run(` matched
// `rerun(`, `range(` matched `arrange(`, `cosine(` matched `cosineInterp(` and
// `hslSet(` matched `okhslSet(` — where the last one let a single call in
// `spatial_geometry_examples.md` count as two hits and clear the threshold alone.
// - no whitespace before the `(`. A fenced block's *comments and string literals*
// are code text as far as this index is concerned, and English prose inside them
// ("seed cosine (grey) → target sine", "full range (chebyshev clamps)") was
// matching. Kotlin allows `f (x)`; this corpus never writes it.
EvidenceTokenMatcher(kind, name)
}
if (debugFor != null && doc.relativePath.endsWith(debugFor)) {
println("EVIDENCE-DEBUG ${doc.relativePath} unique tokens: $kept")
pages.forEach { (pageName, code) ->
val matched = kept.filter { patterns.getValue(it).containsMatchIn(code) }
if (matched.isNotEmpty()) println("EVIDENCE-DEBUG $pageName <- $matched")
}
}
val hits = pages.mapNotNull { (name, code) ->
val count = patterns.values.count { it.containsMatchIn(code) }
if (count >= MIN_EVIDENCE_HITS) name to count else null
}.sortedWith(compareByDescending<Pair<String, Int>> { it.second }.thenBy { it.first })
.take(MAX_EVIDENCE_LINKS)
doc.relativePath to hits
}
}internal class EvidenceTokenMatcher(kind: String, private val name: String)
Exact evidence matching, with a cheap necessary condition before scanning with a regex.
internal class EvidenceTokenMatcher(kind: String, private val name: String) {
private val pattern = Regex.escape(name).let { escaped ->
if (kind == "call") Regex("\\b$escaped\\(") else Regex("\\b$escaped\\b")
}fun containsMatchIn(code: String): Boolean = name in code && pattern.containsMatchIn(code)
private val FENCED_BLOCK = Regex("(?s)```.*?```")
private val INLINE_CODE = Regex("`[^`\n]+`")
private val FILENAME_SPAN = Regex("^`[\\w./-]+\\.(kt|md|kts|py|json|yaml|yml|glsl)`$")Just the code in a markdown page: fenced blocks plus inline spans, prose discarded.
Inline spans that are only a filename are dropped. Naming a source file is not using its API — and since the worked-example pages now carry a back-link that names the very file this index is scoring, counting them would make the evidence partly self-referential.
private fun codeTextOf(markdown: String): String {
val fences = FENCED_BLOCK.findAll(markdown).map { it.value }.toList()
val prose = FENCED_BLOCK.replace(markdown, " ")
val inline = INLINE_CODE.findAll(prose).map { it.value }.filterNot { FILENAME_SPAN.matches(it) }.toList()
return (fences + inline).joinToString("\n")
}Docs page filename -> its own # heading, so "see also" links read as the site reads.
private val pageTitles = mutableMapOf<String, String>()
private fun headingOf(markdown: String): String? =
markdown.lineSequence().firstOrNull { it.startsWith("# ") }
?.removePrefix("# ")
// A few page headings carry inline HTML for typographic effect; a link label must not.
?.replace(Regex("<[^>]*>"), "")
?.replace(Regex("\\s+"), " ")
?.trim()
?.takeIf { it.isNotEmpty() }
private fun evidenceLinks(doc: SourceDocument, evidence: Map<String, List<Pair<String, Int>>>): String? {
val hits = evidence[doc.relativePath].orEmpty()
if (hits.isEmpty()) return null
return hits.joinToString(", ") { (page, _) ->
"[${escapeCell(pageTitles[page] ?: prettifyFileName(page))}](../$page)"
}
}rendering
source — 98 lines, from line 1990
private fun renderCategoryPage(
category: String,
documents: List<SourceDocument>,
typeAnchors: TypeAnchors,
evidence: Map<String, List<Pair<String, Int>>>,
): String = buildString {
appendLine(BANNER)
appendLine("# $category")
appendLine()
CATEGORY_BLURBS[category]?.let { appendLine(it); appendLine() }
appendLine(
"${documents.size} source file(s), ${documents.sumOf { it.symbolCount() }} public symbol(s), " +
"${documents.sumOf { it.documentedCount() }} of them carrying KDoc. Everything below is the " +
"documentation that lives beside the code — the same text the " +
"[completion documentation panel](../completion_doc_browser.md) shows in the editor. A blank " +
"summary means the declaration has no KDoc yet, not that this page dropped it."
)
appendLine()
appendLine("<div class=\"codebase-reference\" markdown>")
appendLine()
documents.forEach { doc -> append(renderFileSection(doc, typeAnchors, evidence)) }
appendLine("</div>")
}
private fun renderFileSection(
doc: SourceDocument,
typeAnchors: TypeAnchors,
evidence: Map<String, List<Pair<String, Int>>>,
): String = buildString {
appendLine("## ${doc.fileName} { #${fileAnchor(doc)} }")
appendLine()
// demoteHeadings before fencesToDiffstatic: the latter rewrites an opening fence to
// "```diffstatic {data-lang=…}", which FENCE_OPENER no longer matches, so a demotion run
// afterwards mistakes the *closing* fence for an opening one and stops demoting from there on.
doc.overview?.let {
appendLine(fencesToDiffstatic(demoteHeadings(linkify(indentedBlocksToFences(it), typeAnchors, doc)))); appendLine()
}
val source = sourcePageUrl(doc)
if (source == null) {
appendLine("`${doc.relativePath}`")
} else {
appendLine("`${doc.relativePath}` — [read the source]($source)")
}
appendLine()
evidenceLinks(doc, evidence)?.let {
appendLine("**Used in:** $it")
appendLine()
}
if (doc.topLevelFunctions.isNotEmpty()) {
appendLine("### Top-level functions { #${fileAnchor(doc)}-functions }")
appendLine()
append(renderTable(doc.topLevelFunctions, typeAnchors, doc))
}
if (doc.topLevelValues.isNotEmpty()) {
appendLine("### Top-level values { #${fileAnchor(doc)}-values }")
appendLine()
append(renderTable(doc.topLevelValues, typeAnchors, doc))
}
doc.types.forEach { type ->
appendLine("### ${type.symbol.simpleName} { #${typeAnchor(doc, type.symbol)} }")
appendLine()
appendLine(linkToSource(doc, type.symbol.line, highlightSignature(type.symbol.signature)))
appendLine()
if (type.symbol.summary.isNotBlank()) {
appendLine(fencesToDiffstatic(linkify(indentedBlocksToFences(type.symbol.summary), typeAnchors, doc)))
appendLine()
}
if (type.enumConstants.isNotEmpty()) {
appendLine("Constants: ${type.enumConstants.joinToString(", ") { "`$it`" }}.")
appendLine()
}
if (type.members.isNotEmpty()) append(renderTable(type.members, typeAnchors, doc))
}
}
private fun renderTable(
symbols: List<Symbol>,
typeAnchors: TypeAnchors,
doc: SourceDocument,
): String = buildString {
appendLine("| Declaration | Summary |")
appendLine("| --- | --- |")
val anchors = rowAnchors(doc)
// Per file, not per table: a `fun pulse` and a `val pulse` share one key and one anchor.
val emitted = emittedRowAnchors.getOrPut(doc.relativePath) { mutableSetOf() }
coalesce(symbols).forEach { row ->
val signature = row.symbol.signature.ifBlank { row.symbol.simpleName }
val declaration = linkToSource(doc, row.symbol.line, highlightSignature(signature))
val key = row.symbol.owner to row.symbol.simpleName
val anchor = anchors[key]?.takeIf { emitted.add(key) }?.let { "<a id=\"$it\"></a>" }.orEmpty()
appendLine(
"| $anchor$declaration${overloadBadge(row)} | " +
"${escapeCell(linkify(row.summary, typeAnchors, doc, parameterNames(signature)))} |"
)
}
appendLine()
}A package tree whose destinations come from the files actually emitted on category pages.
source — 161 lines, from line 2090
private fun renderPackageMap(documents: List<SourceDocument>): String = buildString {
val roots = documents.map { it.relativePath.substringBeforeLast('/') }.distinct().sorted()
val prefix = "src/jvmMain/kotlin/afield/"
appendLine("<div class=\"reference-map\" markdown=\"1\">")
appendLine()
appendLine("<div class=\"reference-map-root\">afield</div>")
appendLine()
appendLine("<div class=\"reference-map-branches\" markdown=\"1\">")
appendLine()
fun branch(paths: List<String>, parent: String, depth: Int) {
paths.groupBy { it.removePrefix(parent).substringBefore('/') }.forEach { (name, children) ->
val path = parent + name
val destinations = if (path in roots) documents
.filter { it.relativePath.startsWith("$path/") }
.groupBy { it.category }.toSortedMap() else emptyMap()
val links = destinations.map { (category, files) ->
val first = files.minBy { it.relativePath }
val label = if (destinations.size == 1) name else category
"[$label](${slug(category)}.md#${fileAnchor(first)})"
}
val label = when {
links.size == 1 -> links.single()
links.isNotEmpty() -> "*$name* — ${links.joinToString(" · ")}"
path == "${prefix}math/linalg" -> "[$name](../linear_algebra_api.md)"
else -> name
}
val rendered = if (destinations.isNotEmpty() && destinations.values.flatten().all { it.implementation })
"<span class=\"reference-internal\">$label</span>" else label
appendLine("${" ".repeat(depth)}- $rendered")
val descendants = children.filter { it != path }
if (descendants.isNotEmpty()) branch(descendants, "$path/", depth + 1)
}
}
roots.groupBy { it.removePrefix(prefix).substringBefore('/') }.forEach { (group, paths) ->
appendLine("<div class=\"reference-map-group\" markdown=\"1\">")
appendLine()
appendLine("### $group")
appendLine()
val direct = documents.filter { it.relativePath.substringBeforeLast('/') == "$prefix$group" }
direct.groupBy { it.category }.toSortedMap().forEach { (category, files) ->
appendLine("- [$category](${slug(category)}.md#${fileAnchor(files.first())})")
}
branch(paths.filter { it != "$prefix$group" }, "$prefix$group/", 0)
appendLine()
appendLine("</div>")
appendLine()
}
appendLine("</div>")
appendLine()
appendLine("</div>")
appendLine()
}
private fun renderIndexPage(
categories: List<String>,
documents: List<SourceDocument>,
typeAnchors: TypeAnchors,
evidence: Map<String, List<Pair<String, Int>>>,
locHero: File,
): String = buildString {
appendLine(BANNER)
appendLine("# What's already in the box")
appendLine()
if (locHero.isFile) {
append(locHero.readText())
appendLine()
}
appendLine(
"Explore Field's JVM codebase by package. Follow a branch to its declarations " +
"and the documentation that sits next to them in the source. Where a package spans " +
"several reference pages, each page is linked separately."
)
appendLine()
append(renderPackageMap(documents))
appendLine("## Reference pages")
appendLine()
val total = documents.sumOf { it.symbolCount() }
val documented = documents.sumOf { it.documentedCount() }
appendLine(
"**$total public symbols across ${documents.size} source files**, $documented of them " +
"(${documented * 100 / total}%) carrying a KDoc summary. Pages are grouped by the " +
"source's `@file:CompletionCategory` where present, with package-based categories " +
"for unannotated files. Public visibility describes Kotlin access; it does not " +
"necessarily mean an API intended for box code."
)
appendLine()
appendLine(
"The \"documented\" column counts declarations carrying a KDoc summary. Where a summary " +
"column below is blank, the declaration has none yet."
)
appendLine()
appendLine("| Category | Files | Symbols | Documented | Covers |")
appendLine("| --- | --- | --- | --- | --- |")
categories.forEach { category ->
val inCategory = documents.filter { it.category == category }
val blurb = CATEGORY_BLURBS[category]?.let { firstSentence(it) } ?: ""
val symbols = inCategory.sumOf { it.symbolCount() }
val documentedHere = inCategory.sumOf { it.documentedCount() }
appendLine(
"| [$category](${slug(category)}.md) | ${inCategory.size} | $symbols | " +
"$documentedHere (${if (symbols == 0) 0 else documentedHere * 100 / symbols}%) | " +
"${escapeCell(blurb)} |"
)
}
appendLine()
appendLine("## Every file")
appendLine()
appendLine("Every Kotlin file under `src/jvmMain/kotlin` is listed, including files without public declarations. " +
"Dimmed entries are identifiable internal plumbing or build/test harnesses; they remain linked and searchable. " +
"Low-level APIs that a box can use, such as renderers and textures, retain full emphasis.")
appendLine()
appendLine("<div class=\"codebase-reference codebase-file-index\" markdown>")
appendLine()
appendLine("| File | Category | Symbols | Documented | What it is | Used in |")
appendLine("| --- | --- | --- | --- | --- | --- |")
documents.sortedBy { it.fileName.lowercase() }.forEach { doc ->
val link = "[${doc.fileName}](${slug(doc.category)}.md#${fileAnchor(doc)})"
val sourceDirectory = doc.relativePath.substringBeforeLast('/') + "/"
val sourcePath = escapeCell(sourceDirectory.replace("&", "&")
.replace("<", "<").replace(">", ">")).replace("/", "/<wbr>")
val style = if (doc.implementation) " reference-internal" else ""
val fileCell = "<small class=\"reference-file-path$style\">$sourcePath</small> $link"
// linkify before stripLinks: `linkify` is what turns a KDoc `[Symbol]` into either a link
// or inline code, and without it the raw brackets reach the page. `stripLinks` then flattens
// the resolved ones, because this column is a one-line gist sitting between two columns
// that are already all links.
val gist = doc.overview
?.let { overviewGist(it) }
?.let { stripLinks(linkify(it, typeAnchors, doc)) }
.orEmpty()
val examples = evidenceLinks(doc, evidence).orEmpty()
appendLine(
"| $fileCell | ${doc.category}${if (doc.implementation) " · internal" else ""} | ${doc.symbolCount()} | ${doc.documentedCount()} | " +
"${escapeCell(gist)} | $examples |"
)
}
appendLine()
appendLine("</div>")
appendLine()
appendLine("## Scope")
appendLine()
appendLine("This catalog covers the JVM Kotlin source tree. Shared protocol and browser source sets, " +
"private personal code and generated vector algebra are separate from that inventory. " +
"The generated `afield.math.linalg` API has its own " +
"[reference](../linear_algebra_api.md), with [Linear algebra in Field](../linear_algebra.md) for the intuition.")
appendLine()
appendLine("## How to read a symbol you find here")
appendLine()
appendLine(
"Type its name in a box and press `ctrl-space`: the " +
"[completion documentation panel](../completion_doc_browser.md) shows the same KDoc " +
"with every resolvable `[Symbol]` reference turned into a link you can follow, plus " +
"a jump straight to the definition's line in the source. This reference and that " +
"panel read declarations through the same parser. Entries marked internal here may " +
"be hidden from autocomplete; use their source links to inspect the implementation."
)
appendLine()
appendLine(
"To search rather than browse, see [The Semantic Index](../semantic_index.md)."
)
}signature highlighting
Kotlin keywords a signature can contain. Deliberately not the whole language — a signature has no statements — so anything unrecognised falls through to the name rules below.
private val SIGNATURE_KEYWORDS = setOf(
"fun", "val", "var", "class", "interface", "object", "typealias", "enum", "data", "sealed",
"annotation", "value", "companion", "inner", "abstract", "open", "final", "override",
"inline", "noinline", "crossinline", "suspend", "tailrec", "operator", "infix", "external",
"const", "lateinit", "vararg", "reified", "out", "in", "where", "by", "init", "constructor",
"public", "private", "internal", "protected", "expect", "actual",
)Literal keywords Pygments colours as constants rather than as keywords.
private val SIGNATURE_CONSTANTS = setOf("true", "false", "null", "this", "super", "it")internal fun highlightSignature(signature: String): String
Renders a signature as pre-highlighted HTML using Pygments' own class names, wrapped in a
.highlight container.
The docs theme already ships a Pygments stylesheet mapping k/nf/nc/n/o/p/mf/s
to its --md-code-hl-*-color palette, so borrowing those class names colours these signatures
with the same palette as every fenced code block on the site and needs no stylesheet of its
own. Doing it here rather than with a fenced block is what keeps a signature inside a table
cell, where a fence cannot go.
Hand-written rather than delegated to a real lexer because the output has to be byte-stable for the drift guard, and because a signature is a small enough grammar to tokenise exactly: keyword, declaration name, type (capitalised), parameter name, literal, operator, punctuation.
body — 72 lines, from line 2283
internal fun highlightSignature(signature: String): String {
val out = StringBuilder("<code class=\"highlight\">")
var i = 0
var sawDeclarationName = false
var depth = 0
while (i < signature.length) {
val ch = signature[i]
when {
ch.isWhitespace() -> { out.append(' '); i++ }
ch == '`' -> {
val end = signature.indexOf('`', i + 1)
val stop = if (end < 0) signature.length - 1 else end
out.span("n", signature.substring(i, stop + 1))
i = stop + 1
}
ch == '"' -> {
var j = i + 1
while (j < signature.length && (signature[j] != '"' || signature[j - 1] == '\\')) j++
val stop = minOf(j, signature.length - 1)
out.span("s", signature.substring(i, stop + 1))
i = stop + 1
}
ch.isDigit() -> {
var j = i
while (j < signature.length && (signature[j].isLetterOrDigit() || signature[j] == '.')) j++
val text = signature.substring(i, j)
out.span(if ('.' in text) "mf" else "mi", text)
i = j
}
ch.isLetter() || ch == '_' -> {
var j = i
while (j < signature.length && (signature[j].isLetterOrDigit() || signature[j] == '_')) j++
val word = signature.substring(i, j)
val followedByCall = signature.getOrNull(j) == '('
val cls = when {
word in SIGNATURE_KEYWORDS -> "k"
word in SIGNATURE_CONSTANTS -> "kc"
// The declaration's own name: the first call-shaped identifier at top level.
!sawDeclarationName && depth == 0 && followedByCall -> { sawDeclarationName = true; "nf" }
word.first().isUpperCase() -> "nc"
followedByCall -> "nf"
else -> "n"
}
out.span(cls, word)
i = j
}
else -> {
if (ch == '(' || ch == '[') depth++
if (ch == ')' || ch == ']') depth--
// `->`, `?`, `*`, `=` read as operators; brackets and separators as punctuation.
val isArrow = signature.startsWith("->", i)
when {
isArrow -> { out.span("o", "->"); i += 2 }
ch == '=' || ch == '?' || ch == '*' || ch == '!' || ch == '&' || ch == '|' ->
{ out.span("o", ch.toString()); i++ }
else -> { out.span("p", ch.toString()); i++ }
}
}
}
}
out.append("</code>")
return out.toString()
}
private fun StringBuilder.span(cls: String, text: String) {
append("<span class=\"").append(cls).append("\">").append(htmlInTableCell(text)).append("</span>")
}Escapes text for raw HTML that sits inside a markdown table cell.
Both layers bite: </>/& are HTML, and | would end the cell while *, _, `,
[ and ] are markdown inline syntax that the parser still applies around inline HTML —
Map<*, *> would otherwise come out italicised. Numeric entities render identically and dodge
both.
private fun htmlInTableCell(text: String): String {
val out = StringBuilder(text.length)
for (ch in text) {
when (ch) {
'&' -> out.append("&")
'<' -> out.append("<")
'>' -> out.append(">")
'"' -> out.append(""")
'|' -> out.append("|")
'*' -> out.append("*")
'_' -> out.append("_")
'`' -> out.append("`")
'[' -> out.append("[")
']' -> out.append("]")
'\\' -> out.append("\")
else -> out.append(ch)
}
}
return out.toString()
}The +N overloads · also … badge that stands in for a coalesced family.
private fun overloadBadge(row: Row): String {
if (row.extraOverloads == 0) return ""
val plural = if (row.extraOverloads == 1) "overload" else "overloads"
val also = row.alsoAccepts
.takeIf { it.isNotEmpty() }
?.joinToString(", ") { htmlInTableCell(it) }
?.let { " · also $it" }
.orEmpty()
return "<span class=\"sig-overloads\">+${row.extraOverloads} $plural$also</span>"
}KDoc worklist
Members Kotlin generates for a data class, or that every type has anyway.
A worklist that tells someone to go and write KDoc for component1 is a worklist nobody
finishes. These are excluded from the gap counts entirely — they are not documentation debt.
private val BOILERPLATE_MEMBERS = setOf(
"toString", "equals", "hashCode", "copy", "compareTo",
)
private fun isBoilerplate(symbol: Symbol): Boolean =
symbol.owner != null &&
(symbol.simpleName in BOILERPLATE_MEMBERS || symbol.simpleName.matches(Regex("component\\d+")))The undocumented declarations of one file that are worth someone's afternoon, highest first.
"Entry point" is the same notion SourceDocument.evidenceTokens uses: a type, or a top-level
function or value. A member is only reachable once you already have the thing it hangs off, so
the type's own KDoc is what orients a reader and is counted first. Overload families are
counted once — coalesce has already merged them, so documenting pulseTrain closes one gap,
not six.
private fun undocumentedEntryPoints(doc: SourceDocument): List<Symbol> {
val types = doc.types.map { it.symbol }
val topLevel = coalesce(doc.topLevelFunctions).map { it.symbol } + doc.topLevelValues
return (types + topLevel).filter { it.summary.isBlank() && !isBoilerplate(it) }
}
private fun undocumentedMembers(doc: SourceDocument): List<Symbol> =
doc.types.flatMap { type -> coalesce(type.members).map { it.symbol } }
.filter { it.summary.isBlank() && !isBoilerplate(it) }A ranked, actionable list of the KDoc that is missing — the other half of the coverage column.
The reference pages report that 10% of this surface carries KDoc; that number is only useful if it comes with a next action. Ranking is by payoff, and every signal is one this generator already computes:
- Files a worked example already demonstrates come first.
buildEvidenceIndexknows which docs pages call which file's API, so these are the entry points readers demonstrably reach for — an undocumented symbol here costs the most. - Types before functions before values, entry points before members. A type's KDoc orients everything inside it, so it is the single highest-leverage thing to write.
- Bigger gaps before smaller ones, so an hour spent lands where the surface is widest.
Regenerate with the pages: ./gradlew generateCodebaseReference.
source — 103 lines, from line 2448
private fun renderKdocWorklist(
documents: List<SourceDocument>,
evidence: Map<String, List<Pair<String, Int>>>,
): String = buildString {
fun demonstrated(doc: SourceDocument) = evidence[doc.relativePath].orEmpty().isNotEmpty()
val totalSymbols = documents.sumOf { it.symbolCount() }
val totalDocumented = documents.sumOf { it.documentedCount() }
val gaps = documents.associateWith { undocumentedEntryPoints(it) }
val memberGaps = documents.associateWith { undocumentedMembers(it) }
val totalEntryGaps = gaps.values.sumOf { it.size }
val totalMemberGaps = memberGaps.values.sumOf { it.size }
appendLine(
"<!-- DO NOT EDIT — generated by afield.tools.docs.StdlibReferenceGenerator alongside " +
"public/docs/reference/. -->"
)
appendLine("<!-- Regenerate: ./gradlew generateCodebaseReference -->")
appendLine("# KDoc worklist")
appendLine()
appendLine(
"The other half of the coverage column on " +
"[the codebase reference](../public/docs/reference/index.md): $totalDocumented of " +
"$totalSymbols public symbols in ${documentedPackages()} carry a " +
"KDoc summary, and this is what is missing, most valuable first."
)
appendLine()
appendLine(
"**$totalEntryGaps undocumented entry points** (types, top-level functions and values) " +
"and $totalMemberGaps undocumented members. Overload families count once — " +
"documenting `pulseTrain` closes one gap, not six — and the members Kotlin generates " +
"for a `data class` are not counted at all."
)
appendLine()
appendLine(
"Ranking is by payoff, from signals the generator already has: files that a worked " +
"example page demonstrably calls come first, then types before functions before " +
"values, then the widest gaps. Write the KDoc next to the declaration, then run " +
"`./gradlew generateCodebaseReference` — the reference pages and this list both " +
"follow from the source."
)
appendLine()
appendLine("## Where the debt is, by file")
appendLine()
appendLine("| File | Category | Used by examples | Undocumented entry points | Undocumented members | Coverage |")
appendLine("| --- | --- | --- | --- | --- | --- |")
documents
.sortedWith(
compareByDescending<SourceDocument> { demonstrated(it) }
.thenByDescending { gaps.getValue(it).size }
.thenByDescending { memberGaps.getValue(it).size }
.thenBy { it.relativePath }
)
.filter { gaps.getValue(it).isNotEmpty() || memberGaps.getValue(it).isNotEmpty() }
.forEach { doc ->
val pages = evidence[doc.relativePath].orEmpty()
val used = if (pages.isEmpty()) "—" else pages.joinToString(", ") { it.first.removeSuffix(".md") }
val coverage =
if (doc.symbolCount() == 0) 0 else doc.documentedCount() * 100 / doc.symbolCount()
appendLine(
"| [${doc.fileName}](../${doc.relativePath}) | ${doc.category} | " +
"${escapeCell(used)} | ${gaps.getValue(doc).size} | " +
"${memberGaps.getValue(doc).size} | $coverage% |"
)
}
appendLine()
appendLine("## The highest-value individual gaps")
appendLine()
val ranked = documents.flatMap { doc -> gaps.getValue(doc).map { doc to it } }
.sortedWith(
compareByDescending<Pair<SourceDocument, Symbol>> { demonstrated(it.first) }
.thenBy { kindRank(it.second) }
.thenBy { it.first.relativePath }
.thenBy { it.second.line }
)
if (ranked.isEmpty()) {
appendLine("Nothing — every entry point in this surface carries a KDoc summary.")
} else {
// The name and where it lives, not the signature: a 400-character `data class` header is
// unscannable in a worklist, and the reference page already carries it in full.
appendLine("| Declaration | Kind | Where |")
appendLine("| --- | --- | --- |")
ranked.take(WORKLIST_ENTRIES).forEach { (doc, symbol) ->
appendLine(
"| `${escapeCode(symbol.qualifiedName)}` | ${symbol.kind} | " +
"`${doc.relativePath}:${symbol.line}` |"
)
}
appendLine()
val dropped = ranked.size - minOf(ranked.size, WORKLIST_ENTRIES)
if (dropped > 0) {
appendLine(
"…and **$dropped more** undocumented entry points not listed here. This section " +
"is capped at $WORKLIST_ENTRIES so it stays a worklist rather than a dump; " +
"the per-file table above is complete."
)
} else {
appendLine("That is all of them.")
}
}
}Types first, then top-level functions, then values — the order KDoc pays off in.
private fun kindRank(symbol: Symbol): Int = when (symbol.kind) {
"interface" -> 0
"class" -> 0
"object" -> 1
"typealias" -> 1
"fun" -> 2
else -> 3
}small helpers
The lines-of-code chart tools/loc_chart.py --docs writes; inlined as the index's hero.
private const val LOC_HERO_REL = "public/docs/assets/loc/loc_hero.html"
private const val BANNER =
"<!-- DO NOT EDIT — generated by afield.tools.docs.StdlibReferenceGenerator from KDoc in src/jvmMain/kotlin. -->\n" +
"<!-- Regenerate: ./gradlew generateCodebaseReference -->"internal fun demoteHeadings(text: String): String
Re-levels the headings in a file overview so its shallowest one becomes ### — a child of the
## SomeFile.kt section the prose is printed inside — with deeper headings keeping their
relative depth.
A file overview is written as free-standing prose and its author reasonably starts at # or
##. Printed verbatim, a # competed with the page's own title and a ## became a sibling
of the file heading: reference/polytopes.md rendered 45 h2s of which only 17 were files, and
the page's table of contents read as a flat list alternating "Canonical.kt" with "Two
iterations, and why there are two". Normalising to ### puts them at the same level as the
### TypeName sections that follow in the same file, which is where they belong, and means an
overview reads the same whichever convention its author used.
Lines inside a fenced block are left alone — # is a comment in several of the languages the
overviews quote. Levels are clamped at ######, markdown's deepest.
internal fun demoteHeadings(text: String): String {
if ('#' !in text) return text
val lines = text.split("\n")
var fenced = false
val headingLevels = mutableListOf<Int>()
lines.forEach { line ->
if (FENCE_OPENER.matchEntire(line) != null) { fenced = !fenced; return@forEach }
if (fenced) return@forEach
HEADING.matchEntire(line)?.let { headingLevels += it.groupValues[1].length }
}
val shallowest = headingLevels.minOrNull() ?: return text
val shift = 3 - shallowest
if (shift == 0) return text
fenced = false
return lines.joinToString("\n") { line ->
if (FENCE_OPENER.matchEntire(line) != null) { fenced = !fenced; return@joinToString line }
if (fenced) return@joinToString line
val match = HEADING.matchEntire(line) ?: return@joinToString line
val level = (match.groupValues[1].length + shift).coerceIn(1, 6)
"${"#".repeat(level)} ${match.groupValues[2]}"
}
}
private val HEADING = Regex("""^(#{1,6})\s+(.*)$""")internal fun fencesToDiffstatic(text: String): String
Rewrites the markdown code fences in KDoc-derived prose to the docs site's diffstatic fence.
KDoc is written with ordinary triple-backtick kotlin fences, which is what IDEs and Field's own
in-app documentation panel expect, so the sources stay exactly as they are. The published
reference, though, is a Zensical page where every other code block on the site is a diffstatic
fence rendered by Field's own highlighter — a passed-through kotlin fence gets the site's
default highlighter instead and reads as a foreign element (different type colours, a line-number
gutter, a copy button). Normalising at emission rather than in the KDoc keeps one convention in
the source and one on the page, and covers every overview written after this without asking
anyone to remember.
A fence with no language is treated as Kotlin, which is what those blocks have always been in
practice. Closing fences are left alone, and an already-diffstatic fence is passed through.
An indented code block — the shape most file overviews use, see indentedBlocksToFences — is
fenced first, so it gets the same treatment.
internal fun fencesToDiffstatic(text: String): String {
val fenced = indentedBlocksToFences(text)
if (!fenced.contains("```")) return fenced
var open = false
return fenced.split("\n").joinToString("\n") { line ->
val match = FENCE_OPENER.matchEntire(line) ?: return@joinToString line
if (open) {
open = false
line
} else {
open = true
val lang = match.groupValues[2].ifBlank { "kotlin" }
if (lang.startsWith("diff")) line else "${match.groupValues[1]}```diffstatic {data-lang=$lang}"
}
}
}internal fun indentedBlocksToFences(text: String): String
Rewrites markdown indented code blocks as bare triple-backtick fences, dedented by the block's
smallest indent, so fencesToDiffstatic can give them the diffstatic treatment.
A block is a run of lines indented four or more spaces that follows a blank line (or opens the text), with blank lines allowed inside it. Markdown's own rules are kept where they bite: an indented run after a list item is list continuation, not code, and nothing inside an existing fence is touched. Trailing blank lines stay outside the new fence.
internal fun indentedBlocksToFences(text: String): String {
val lines = text.split("\n")
val out = ArrayList<String>(lines.size)
fun indent(l: String) = l.length - l.trimStart(' ').length
val listItem = Regex("""^\s*([-*+]|\d+[.)])\s+.*""")
var inFence = false
var inList = false
var i = 0
while (i < lines.size) {
val line = lines[i]
if (FENCE_OPENER.matchEntire(line) != null || line.trimStart().startsWith("```")) {
inFence = !inFence
out += line; i++; continue
}
if (inFence) { out += line; i++; continue }
val prevBlank = out.isEmpty() || out.last().isBlank()
if (line.isNotBlank() && indent(line) >= 4 && prevBlank && !inList) {
var end = i
var j = i
while (j < lines.size && (lines[j].isBlank() || indent(lines[j]) >= 4)) {
if (lines[j].isNotBlank()) end = j
j++
}
val block = lines.subList(i, end + 1)
val strip = block.filter { it.isNotBlank() }.minOf { indent(it) }
out += "```"
block.forEach { out += if (it.isBlank()) "" else it.substring(strip) }
out += "```"
i = end + 1
continue
}
if (line.isNotBlank()) {
if (listItem.matches(line)) inList = true
else if (indent(line) == 0) inList = false
}
out += line; i++
}
return out.joinToString("\n")
}A whole-line triple-backtick fence: leading indent, then an optional bare language word.
private val FENCE_OPENER = Regex("""^(\s*)```\s*([A-Za-z0-9_+-]*)\s*$""")Turns KDoc [Symbol](../../../../../../reference/source/jvmMain/afield/document/persistence/edn.md#L45) / [display](../../../../../../reference/source/jvmMain/afield/document/persistence/edn.md#L45) references into links, but only when the
target is a type this reference emits — mirroring linkifyKDocSymbols' rule that an
unresolved reference stays literal rather than becoming a dead link. Unresolved references
become plain inline code so the raw brackets never leak into the page.
doc is the file the text was read from, and it is what makes a reference to a type declared
in that same file resolve even when the name is ambiguous elsewhere. Null only where the text
has no one declaring file. See TypeAnchors.
An ordinary markdown link written in a KDoc is left alone: the lookahead for ( is what stops
[Lygia](https://lygia.xyz) becoming `Lygia`(https://lygia.xyz) with the URL dangling as
prose. SourcePageGenerator.linkifySymbols draws the same line for the same reason.
locals names what this text resolves for itself — the parameters of the row it summarises.
See TypeAnchors.resolveReference.
source — 46 lines, from line 2715
private fun linkify(
text: String,
typeAnchors: TypeAnchors,
doc: SourceDocument?,
locals: Set<String> = emptySet(),
): String {
if (text.isBlank()) return text
// Code in a fence is code: `destination[i]` must not become a link to a symbol named `i`.
if (text.contains("```")) {
val segments = mutableListOf(mutableListOf<String>())
var inFence = false
text.split("\n").forEach { line ->
val isFence = line.trimStart().startsWith("```")
if (isFence && !inFence) segments += mutableListOf<String>()
segments.last() += line
if (isFence && inFence) segments += mutableListOf<String>()
if (isFence) inFence = !inFence
}
return segments.filter { it.isNotEmpty() }.joinToString("\n") { seg ->
val joined = seg.joinToString("\n")
if (seg.first().trimStart().startsWith("```")) joined else linkify(joined, typeAnchors, doc, locals)
}
}
val alternate = Regex("""\[([^\]\[]+)]\[([A-Za-z_][\w.]*)]""")
val simple = Regex("""\[([A-Za-z_][\w.]*)](?!\()""")
var out = alternate.replace(text) { m ->
val target = typeAnchors.resolveReference(m.groupValues[2], doc?.relativePath, locals)
if (target != null) "[${m.groupValues[1]}]($target)" else "`${m.groupValues[1]}`"
}
out = simple.replace(out) { m ->
val raw = m.groupValues[1]
val target = typeAnchors.resolveReference(raw, doc?.relativePath, locals)
if (target != null) "[`$raw`]($target)" else "`$raw`"
}
return out
}
private fun stripLinks(text: String): String =
Regex("""\[([^\]]*)]\([^)]*\)""").replace(text) { it.groupValues[1] }
private fun firstSentence(text: String): String {
val flat = text.split("\n").takeWhile { it.isNotBlank() }.joinToString(" ")
.replace(Regex("\\s+"), " ").trim()
val stop = Regex("""[.!?](\s|$)""").find(flat) ?: return flat
return flat.substring(0, stop.range.first + 1)
}The index's "What it is" cell for a file overview that opens with a Markdown heading.
firstSentence takes the overview's first paragraph as written, so an overview whose first
line is #### What this file is published its hashes into the table cell, markup and all.
Dropping the marker keeps the heading's own words, which is what a one-line title already is;
taking the paragraph below it instead would promote prose written as a body, and several of
those read as a sentence fragment or open on a colon.
The heading stays in the page body, where demoteHeadings re-levels it under the generated
file heading.
private fun overviewGist(text: String): String =
firstSentence(text.replaceFirst(Regex("""^\s*#{1,6}\s+"""), ""))Escape markdown (links, inline code preserved) for a table cell.
private fun escapeCell(text: String): String = text.replace("|", "\\|").replace("\n", " ")Escape a raw signature that is about to be wrapped in an inline-code span in a table cell.
private fun escapeCode(text: String): String =
text.replace("|", "\\|").replace("\n", " ").replace("`", "'")
private fun slug(text: String): String =
text.lowercase().replace(Regex("[^a-z0-9]+"), "-").trim('-')
private var fileAnchors: Map<String, String> = emptyMap()
private fun fileAnchor(doc: SourceDocument): String = fileAnchors[doc.relativePath] ?: slug(doc.fileName)
private fun typeAnchor(doc: SourceDocument, symbol: Symbol): String =
"${fileAnchor(doc)}-${slug(symbol.simpleName)}"
private fun prettifyFileName(page: String): String =
page.removeSuffix(".md").split('_', '-').joinToString(" ") { part ->
part.replaceFirstChar { c -> c.uppercaseChar() }
}Walk up from the working directory until the repo root (the one with public/docs) is found.
private fun locateRepoRoot(): File {
var probe: File? = File(System.getProperty("user.dir")).absoluteFile
while (probe != null) {
if (File(probe, "public/docs").isDirectory && File(probe, "src/jvmMain/kotlin").isDirectory) return probe
probe = probe.parentFile
}
return File(System.getProperty("user.dir")).absoluteFile
}
}