Kina.ai

Specification

Phase 0 / Phase 1 Addendum: Ecosystem Architecture

Companion to the Phase 0 specification and a forward commitment for Phase 1. Architectural patterns derived from RAPIDS that shape the runtime, the public ABI, and the long-term ecosystem strategy.

1. Framing

The CUDA-native walkthrough committed to CUDA being a first-class concern in the type system and IR. This addendum extends that with three further commitments, drawn from RAPIDS' architectural patterns:

  1. A shared GPU memory manager (RMM-style) underlying all GPU allocations.
  2. A canonical tensor ABI with cross-framework interop (DLPack-compatible, with Apache Arrow paths for tabular data when domain libraries arrive).
  3. A library-of-libraries architecture, in which the compiler, runtime, and type system are infrastructure that domain-specific libraries (analogs of cuDF, cuML, cuGraph) sit on top of.

These are not Phase 0 spec changes in the strict sense — Sections 1–4 of Phase 0 don't need modification. They are Phase 0/1 engineering commitments: decisions about how the runtime is built, what the public ABI looks like, and how the project's scope extends beyond "an ML language" to "typed-GPU-computation infrastructure."

The RAPIDS comparison is the operating reference. RAPIDS proved at scale that the patterns below produce a substantially different ecosystem than the bolted-on alternative. This addendum commits us to learning that lesson rather than re-discovering it.

2. Pattern 1: Shared GPU memory manager

2.1 What RMM does

The RAPIDS Memory Manager is the substrate that makes RAPIDS' library-of-libraries model viable. Specifically:

The result: any RAPIDS library writes idiomatic allocations (auto buffer = rmm::device_buffer(size, stream)) and inherits all the policy decisions (which allocator, how aggressively to pool, whether to log) from a centrally-configured resource. Library authors don't reinvent allocation; users tune one knob and all libraries adapt.

2.2 Our analog

A Rust trait, in the runtime crate, modeled on DeviceMemoryResource:

pub trait MemoryResource: Send + Sync {
    fn allocate(
        &self,
        size: usize,
        stream: Stream,
    ) -> Result<DevicePtr, AllocError>;

    fn deallocate(
        &self,
        ptr: DevicePtr,
        size: usize,
        stream: Stream,
    ) -> Result<(), DeallocError>;

    fn supports_streams(&self) -> bool;
    fn allocator_name(&self) -> &'static str;
}

With initial implementations:

The runtime exposes a current resource per-thread (or per-context), with a global default. Library code writes idiomatic allocations:

let buf = runtime::device_alloc(size, stream)?;  // uses current resource

And the compiler's lowering passes emit code that uses runtime::device_alloc rather than direct cudaMalloc. This means a user who wants to switch to a pool allocator (or add logging, or use managed memory) does so by configuring the resource — all generated kernels and all libraries pick it up automatically.

2.3 Why this is Phase 1, not Phase 5

Without a shared memory manager, every domain library built on our infrastructure would need to invent its own allocation story. Worse, two libraries used together would compete for GPU memory without coordination. The RAPIDS lesson is that this must be infrastructure, not per-library.

The cost of getting this right in Phase 1 is small (a Rust trait, four or five implementations, integration with the compiler's lowering). The cost of not getting it right is rebuilding every library on top of our project once a second library appears. Phase 1 commits to this.

2.4 Phase 1 deliverable

3. Pattern 2: Canonical tensor ABI with cross-framework interop

3.1 What RAPIDS does with Arrow

RAPIDS' ecosystem strategy depends on Apache Arrow as a shared columnar format:

The win: a cuML model fit on a cuDF dataframe doesn't pay a serialization cost. The data lives in one canonical format throughout.

3.2 The right format for tensors

Arrow is row/columnar-oriented and works for tabular data. For tensors specifically, the dominant cross-framework standard is DLPack — a small C struct describing a multi-dimensional array, used by PyTorch, JAX, TensorFlow, CuPy, NumPy, MXNet, and several others for zero-copy tensor exchange.

The DLPack struct (simplified):

typedef struct {
    void* data;
    DLDevice device;        // device type and id
    int32_t ndim;
    DLDataType dtype;
    int64_t* shape;
    int64_t* strides;       // optional; NULL for compact
    uint64_t byte_offset;
} DLTensor;

DLPack is what RAPIDS uses for tensor interop with PyTorch and JAX (e.g., from cuML to PyTorch) — Arrow is for tabular data, DLPack is for tensors. They're complementary, not alternatives.

3.3 Our commitment

The tensor type at the runtime level has a stable C ABI compatible with DLPack. Specifically:

Concretely, our Rust runtime's tensor:

#[repr(C)]
pub struct Tensor {
    data: *mut u8,
    device: Device,
    ndim: i32,
    dtype: DType,
    shape: *const i64,
    strides: *const i64,    // null if compact
    byte_offset: u64,
    // Internal fields (lifetime, allocator handle) follow,
    // but the prefix above is DLPack-compatible.
}

The internal fields (which allocator owns the memory, which stream the tensor is bound to, refcount or lifetime metadata) live after the DLPack-compatible prefix. Code that consumes Tensor as a DLTensor reads the prefix; code that knows the full type uses the runtime's full API.

3.4 What this enables

3.5 Phase 5 deliverable (with Phase 1 prep)

The full DLPack and Arrow interop story lives at Phase 5 (where the tile IR and lowering work consolidates). What Phase 1 must commit to:

This is a five-line commitment in the runtime's source plus a written-down ABI policy. Cheap to do correctly now; expensive to retrofit later.

4. Pattern 3: Library-of-libraries architecture

4.1 The RAPIDS ecosystem model

RAPIDS is not one library — it's a deliberate ecosystem:

Each is domain-specific, with its own API surface and its own engineering team. They all sit on shared infrastructure: RMM for memory, libraft for shared GPU primitives, Arrow for tabular data exchange, DLPack for tensor exchange, common build tooling and CI infrastructure.

The unification is at the infrastructure layer, not the API layer. cuDF's API is shaped like pandas; cuML's is shaped like sklearn; cuGraph's is shaped like NetworkX. They share none of these surface APIs — they share the underlying CUDA primitives, memory manager, and data formats.

4.2 Our commitment

The project is not "an ML language." It is a language and runtime that domain-specific libraries can sit on top of. The phases we have planned (algebraic IR, AD via effects, sharding, sparsity, hardware capabilities, verification) are the infrastructure layer. Domain libraries are downstream concerns — possibly built by us, possibly by others, but planned-for from the architecture upward.

Anticipated domain libraries, in rough priority order:

  1. Tensor / array primitives (the core; built in Phase 1).
  2. Classical ML (linear regression, decision trees, k-means, PCA — the cuML analog). Phase 5+.
  3. Typed dataframes (cuDF analog with refinement-typed schemas — schema refinements as first-class types). Phase 5+.
  4. Graph analytics (typed graph algorithms — graph-as-typed-data-structure). Phase 5+.
  5. Vector search (typed embeddings, ANN indices). Phase 5+.
  6. Signal processing (typed FFT, convolution, filter design). Phase 5+.
  7. Differential equations / scientific computing (typed ODE/PDE solvers). Phase 5+.

Each of these is its own multi-month project. The point of listing them is to motivate the architectural choices: the public ABI of the runtime, the type system extensibility points, and the documentation strategy must all be designed for library authors as a target audience, not just for compiler internals.

4.3 What library authors need from us

Five things, each a specific deliverable somewhere in the phase plan:

  1. A stable public API for the type system. Library authors define new typed operations; the type-checker accepts them via a documented extension mechanism. This is how a typed-dataframes library adds a GroupBy operation that the rest of the type system reasons about correctly. Phase 5 work item.

  2. A stable public API for the IR. Library authors construct typed IR programs programmatically (rather than via the surface language). This is how a library exposes high-level operations that lower to the algebraic IR. Phase 5 work item.

  3. A stable public API for the runtime. Memory manager, stream management, kernel launch, error handling. From §2 above. Phase 1 work item.

  4. A custom-kernel authoring path via the tile IR. Library authors write performance-critical kernels at the tile level, with the same type system properties as the algebraic level. Phase 5 work item.

  5. An interop story for cross-framework data exchange. DLPack for tensors, Arrow for tabular data when applicable. From §3 above. Phase 1 prep, Phase 5 deliverable.

These are the surfaces that determine whether the project can support an ecosystem at all. Designing them poorly limits the project to "the things we built ourselves"; designing them well opens up the cumulative growth model that made RAPIDS a real platform.

4.4 Implications for Phase 0 and Phase 1

The phase plan grows by one architectural concern: public-ABI stewardship. Every Phase 1+ deliverable that touches a runtime or compiler API has to consider whether that API will be exposed to library authors and what the stability guarantees are. Concretely:

The "library author's guide" is a substantial deliverable that doesn't exist in the original phase plan. Adding it is a deliberate choice — without it, the architectural pattern collapses back to "tools we wrote ourselves."

5. What this changes about Phase 0

Strictly speaking, the Phase 0 spec sections 1–4 are unchanged. The commitments here affect Section 5 (Implementation Infrastructure) and the engineering plan downstream.

5.1 Updates to Section 5 of Phase 0

Section 5's runtime story grows from "minimal: tensor heap, dlopen, run" to:

These are additions to the existing Section 5 runtime work, not replacements. The cost is small — most of the work is documentation and a few extra design constraints. The compounding benefit is that everything built on top of the runtime, from Phase 1 onward, respects these constraints rather than working around them.

5.2 Updates to the Phase 1 milestone

Phase 1 was originally framed as "the shape-typed core compiled end-to-end through MLIR to LLVM." The CUDA-native commitment added "and runs on a GPU via existing library kernels (cuBLAS / cuDNN dispatch)." This addendum adds:

Phase 1's scope grows by perhaps 2–4 weeks of focused work for these. The tradeoff is that Phase 1's deliverable is no longer just "we have a working forward-pass compiler"; it's "we have a working forward-pass compiler that other libraries can plausibly build on." That's a much stronger position from which to do Phase 2 and onward.

6. Open issues introduced

  1. API stability vs. design flexibility. Public ABIs are a tax — every breaking change is a paperwork-and-coordination cost. The discipline of "design for library authors from day one" must be balanced against "the project is in early stages and learning rapidly." Suggestion: every public-API decision in Phase 1 ships behind a #[unstable] annotation that requires explicit opt-in, with a documented stabilization path.

  2. Coordination with DLPack's evolution. DLPack's spec evolves (most recently with bfloat16 and FP8 support). We pin a version, document upgrade paths, and accept that some interop will lag the latest DLPack features by months. Same model as the LLVM/MLIR pinning policy from Section 5.

  3. The Apache Arrow C Data Interface for tabular data isn't free. When a typed-dataframes library lands (Phase 5+), Arrow integration becomes a real piece of work. Worth noting now so Phase 5 planning anticipates it.

  4. Memory manager performance vs. simplicity. RMM has had years of tuning; our initial implementation will be naive. The trait is the right shape; the implementations will need iteration. Initial benchmark target: within 3× of cuMalloc for the raw resource, within 1.5× of CUDA async allocator for the pool resource. Improve over time.

  5. What goes in the runtime vs. what goes in a library? A blurry line. Tensor primitives (matmul, elementwise ops) are clearly runtime. Random number generation could be either. Linear algebra solvers (LU, QR, eigendecomposition) are probably libraries. The principle: the runtime is what every library needs; everything else is a library. Boundary cases get individual ADRs.

7. Summary

Three patterns from RAPIDS, committed to as Phase 0/1 architectural decisions:

Pattern What When
Shared GPU memory manager (RMM-style) MemoryResource trait + 4 implementations, single-pool semantics Phase 1
Canonical tensor ABI (DLPack-compatible) #[repr(C)] prefix with DLPack layout; full interop deferred Phase 1 prep, Phase 5 deliverable
Library-of-libraries architecture Public APIs designed for library authors; first domain library validates Phase 5+

The combined effect: the project's scope is no longer "an ML language." It is typed-GPU-computation infrastructure — a foundation that domain libraries (the cuDF, cuML, cuGraph analogs) can be built on, with the type system and verification as the differentiator from RAPIDS itself.

The commitments are largely engineering, not theoretical. The Phase 0 spec sections 1–4 are unchanged. What changes is the runtime's design constraints, the public-ABI discipline, and the explicit anticipation of an ecosystem rather than a single language artifact.

This is the right scope for the long-term ambition. The patterns are proven; the engineering is bounded; the integration point with the existing GPU/ML world (DLPack, Arrow, RMM-style allocation) is well-understood. Adopting them now means not having to retrofit them later.

End of ecosystem architecture addendum.