Future Track: Apache Arrow, Typed Dataframes, and the Observability Stack

Forward-looking design document. Not Phase 0 work; not Phase 1 work; commits to a track of future research and engineering that the architecture should anticipate.

1. Framing

The ecosystem-architecture addendum committed to Arrow as the canonical tabular-data format when domain libraries arrive. This document goes further: typed dataframes built on Arrow are the first non-ML domain library worth building, the schema-refinement-typing approach is a real research contribution, and the observability stack is the concrete use case that grounds the whole track.

This is deliberately a "future track" rather than a near-term commitment. Phases 0 and 1 don't need this work. But the design decisions Phases 0 and 1 are making do affect whether this is buildable later, and the architecture should be deliberately compatible with the eventual deliverable rather than accidentally so.

The structure of this document:

• §2 establishes why Arrow is more central than the addendum implied.
• §3 sketches the typed-dataframes design — schema as refinement type.
• §4 grounds it in the observability use case (NebuCloud's stack as the target).
• §5–6 cover what Phase 0/1 must do and what Phase 5+ delivers.
• §7 covers the paper-shaped research artifact.
• §8–9 wrap up.

2. Arrow's actual centrality

The previous treatment positioned Arrow as "the format we use when a dataframes library happens." That undersells it. Arrow has become the de facto standard for analytical-data interchange, and the surrounding ecosystem has grown around it in ways that make it more than just one library's choice:

• The PyData stack: pandas (via PyArrow), Polars (Arrow-native), DuckDB (Arrow-native).
• Analytics databases: ClickHouse, BigQuery, Snowflake, Spark, Velox, Dremio.
• Storage formats: Parquet (Arrow's persistent counterpart), Iceberg, Delta Lake.
• GPU computing: RAPIDS (cuDF, cuVS), NVIDIA Morpheus (cybersecurity), Theseus.
• Observability: OpenTelemetry (recent OTel-Arrow protocol), Grafana (Arrow-aware dashboards in development).
• ML feature stores: Feast, Tecton, increasingly Arrow-native.

The pattern: Arrow is becoming the system bus for analytical data. Crossing systems used to mean serialization; with Arrow, it increasingly means just sharing a buffer pointer through the C Data Interface.

For our project, this means Arrow integration is not just "a feature for the dataframes library." It is a strategic interop boundary — the boundary at which our typed-GPU-computation infrastructure connects to the existing analytical ecosystem. Same role DLPack plays for tensor interop with the ML world.

3. Typed dataframes: schema as refinement type

The technical thesis. Existing dataframe libraries (pandas, Polars, cuDF) have schemas, but the schemas are runtime objects: a df.select("nonexistent_col") is a KeyError at runtime, not a compile-time error. A df.join(other, on="id") is checked at runtime. Operations produce new schemas that are determined dynamically.

In our refinement type system, schemas can be compile-time objects with refinements. The dataframe type carries its schema; operations are typed as transformations on schemas.

3.1 The basic shape

Dataframe[Schema {
  age:    Int32   refines (0 < age < 150),
  name:   Utf8,
  email:  Utf8    refines (matches(email_pattern)),
  region: Dict<Int8, Utf8>,
  signup: Timestamp(Microsecond, "UTC"),
}]

A Schema is a structural type: an ordered list of (name, dtype, refinements) triples. The refinement language is the same Presburger-with-uninterpreted-functions from Phase 0, extended for string predicates (matches, contains) where the underlying SMT solver supports them.

3.2 Operations as schema transformations

Each dataframe operation has a typing rule that transforms the schema:

The interesting cases:

• filter strengthens refinements. After df.filter(col("age") >= 18), the age column has refinement 18 <= age < 150. The compiler tracks this. Downstream code that requires age-of-majority gets the guarantee for free.
• join requires schema compatibility on the join key. Two dataframes joining on id must both have id of compatible types; the compiler discharges this as an SMT obligation.
• select is a type-level filter on the schema. A select that names a column not in the schema is a compile-time error with a helpful message.

This is the Polars / cuDF API surface, type-checked.

3.3 A concrete example

let process_users [N]
    (df : Dataframe[Schema {
        id: Int64,
        age: Int32 refines (0 <= age < 150),
        signup: Timestamp(Microsecond, "UTC"),
        country: Utf8
    }, rows = N])
    : Dataframe[Schema {
        id: Int64,
        age_group: Dict<Int8, Utf8>,
        country: Utf8
    }, rows ≤ N] =

  df
    |> filter (col "age" >= 18)
    // age now: 18 <= age < 150
    |> with_column "age_group" (
         case col "age"
           | (age < 30) -> "18-29"
           | (age < 50) -> "30-49"
           | _          -> "50+"
       )
    |> select ["id", "age_group", "country"]
    // schema: id:Int64, age_group:Dict<Int8,Utf8>, country:Utf8

Notable:

• The output row count is rows ≤ N because filter can only reduce rows. This is a refinement on the dataframe's row dimension.
• The age_group column is typed as a dictionary-encoded categorical with three possible values, derived from the case expression's exhaustivity.
• The column references col "age" are checked at compile time; a typo col "agee" is a type error at parse-elaboration time.
• Drop-the-source-columns happens implicitly via select.

3.4 Why this is genuinely new

Polars has good runtime schema validation. cuDF inherits pandas' duck typing. Arrow's Java/C++ libraries have schema introspection but no compile-time integration. No production dataframe library does compile-time schema-refinement-typed operations.

The closest existing work is in functional programming research:

• Records and row types for SQL queries (Ur/Web, Links, some Haskell research). Type-safe SQL with row-polymorphic records. Very close in spirit; nothing adopted by the analytical ecosystem.
• TyCC and StaticSQL (research languages with type-checked SQL). Limited to SQL; no GPU; no tabular operations beyond SQL.
• Polars' LazyFrame and Polars-style queries in Rust. Static schema but no refinements.

The combination — refinement-typed Arrow schemas, tabular operations as schema transformations, GPU-resident data, refinement-flow through filters and joins — is novel. Publishable as a paper (PLDI / ICFP / OOPSLA shape) before anyone else combines these pieces.

4. The observability stack as the validation target

Why this track grounds in observability specifically.

4.1 The data shapes are perfect refinement-type targets

Observability data is structured but variable:

• Traces (OpenTelemetry spans): trace_id (16 bytes), span_id (8 bytes), parent_span_id, service_name, operation_name, start_time, end_time, status (enum), attributes (map of string → string|int|float|bool), events (list of struct with timestamp + attributes), links.
• Metrics: metric name, value (float), timestamp, dimensional labels (map of string → string), value type (gauge / counter / histogram).
• Logs: timestamp, severity (enum), body (string or structured), service, attributes.

Each of these has a fixed outer structure with variable attribute payloads. This is exactly the case where:

• The outer structure is a strict schema (compile-time check).
• Attribute access is gradual — you can know a particular attribute exists in a particular pipeline by refining the schema.
• Operations on these (filter spans by service, group by operation, compute percentiles by attribute) are schema-transforming in well-defined ways.

A typed-dataframes library that handles this cleanly is far more useful than the existing alternatives — the existing alternatives all treat attribute payloads as opaque blobs of dynamic types.

4.2 OpenTelemetry is going Arrow-native

The OTel-Arrow protocol (formerly OTel/Arrow) was accepted as an OpenTelemetry experimental protocol in late 2023 and has been moving toward stable. The motivation:

• OTLP (OpenTelemetry's default protocol) uses protobuf, which has serialization overhead that becomes significant at high cardinality and volume.
• OTel-Arrow uses Arrow as the wire format. Significantly lower bandwidth and CPU overhead at scale.
• Major observability vendors (Datadog, New Relic, Splunk, Honeycomb, Lightstep) are adopting it on the ingest path.
• ClickHouse can ingest Arrow natively, so Arrow → ClickHouse is zero-copy.

This means: by the time our typed-dataframes library lands (Phase 5+), the observability data flowing through systems like NebuCloud's stack will be increasingly Arrow-native end-to-end.

4.3 Specific applications in the NebuCloud context

NebuCloud's observability stack (per the project description: OTel Collector, VictoriaMetrics, ClickHouse, Tempo, Grafana) plus the NebulaIQ analytics layer is exactly the use case this track is designed for. Concrete applications:

• Trace anomaly detection (currently NebulaIQ's territory, in Python). A typed-dataframes library would express anomaly-detection pipelines with compile-time schema checks; the GPU lowering would run them at the rates needed for high-cardinality services.
• Span-level ML features (extracting features from trace spans for downstream classifiers). Schemas guarantee the feature pipeline matches the model's expected input.
• Real-time aggregation pipelines (windowed metric computation from trace data). Schema typing catches mismatches between incoming traces and aggregation logic at compile time.
• Cross-source analytical queries (joining traces from Tempo with metrics from VictoriaMetrics with logs from ClickHouse). Each source has its own schema; joins type-check the union; the result is a typed dataframe.

The observability stack is therefore not a hypothetical use case but a directly aligned target: the language's first non-ML domain library could plausibly serve infrastructure that's already in development, by an existing user (you), with a clear progression from "the language exists" to "the language is useful" to "the language is in production for at least one project."

This is a substantial alignment. Most language projects struggle for years to find a real first user; here, there's a credible one already in the project's surrounding work.

4.4 Beyond NebuCloud

The observability use case also generalizes. The same architecture applies to:

• Security event analysis (SIEM-type workloads). Schemas are well-defined; ML on top is increasingly normal; GPU acceleration is increasingly desired.
• Financial market data (tick data, order book reconstruction). Strict schemas; high volume; GPU-friendly workloads.
• IoT telemetry (sensor data pipelines). Variable schemas per device class; refinement-typing handles the variability cleanly.
• Game telemetry (player behavior analytics). Similar shape to observability.

These are all places where pandas / Polars / Arrow-based pipelines exist today, and where GPU acceleration plus type safety would be a real win.

5. What Phase 0/1 must do to keep this track open

Minimal but real:

1. The runtime's MemoryResource trait (Phase 1, from the ecosystem-architecture addendum) must allocate raw bytes, not tensor-shaped buffers. The trait is correct as designed; the documentation needs to make clear that columnar buffers (Arrow's variable-length data, validity bitmaps, offset arrays) are valid clients.

2. The tensor ABI's #[repr(C)] prefix (also Phase 1) sits in the runtime's Tensor struct. The type system should be designed so that Dataframe is a different type, not a special case of Tensor. The two share infrastructure (memory manager, stream allocation, device placement) but have different layouts and operations. Phase 0's grammar already separates Tensor from Scalar and Unit; adding Dataframe[Schema] later is additive.

3. The compiler's IR design (Phase 0/1) should not assume tensor-only. The IR's representation of values should be polymorphic over the value type so adding a new value kind (Dataframe) doesn't require restructuring the IR. This is already the case in any well-designed compiler IR; just worth flagging as a constraint to verify.

4. A short ADR committing to "Arrow as canonical tabular-data format from Phase 5+" so the constraint is recorded and visible to future PRs. One page; goes in decisions/ (repo root; Phase 0 Section 5).

These are commitments worth a few hours of design attention now and zero implementation effort. They're cheap to do correctly; expensive to retrofit.

6. Phase 5+ deliverables

When the dataframes track lands:

1. A Dataframe[Schema, rows] type in the IR with the schema-as-refinement-type semantics from §3.
2. Core dataframe operations (select, filter, with_column, groupby, agg, join, concat, sort, limit, distinct).
3. Arrow C Data Interface bindings in the runtime — zero-copy import from PyArrow, Polars, DuckDB, ClickHouse client libraries, etc.
4. An OTel-Arrow ingest path as a flagship demo: receive an OTel-Arrow stream, type-check the schema against expected, run a typed analytical pipeline, emit results.
5. Performance benchmarks against cuDF and Polars on representative workloads. Goal: within 1.5x of cuDF on GPU, faster than Polars on CPU due to compile-time elimination of schema-validation overhead.

This is roughly a year of focused work, similar in scope to the typed-tensor compiler from Phase 1. The architecture decisions in §5 keep the cost bounded — the Phase 5 work is "build the dataframes library on infrastructure that already supports it," not "retrofit infrastructure to support dataframes."

7. The research artifact

Independent of whether the project ships, the schema-refinement-types-on-Arrow design is a paper. Specifically:

• Title shape: "Refinement-Typed Schemas for Columnar Analytical Data" or similar.
• Venues: PLDI, ICFP, OOPSLA, ECOOP. Possibly DBLP (database community) but PLDI is the better fit because the contribution is a type system.
• Core technical content: schema as a refinement type, operations as schema transformations, soundness theorem (well-typed pipelines produce well-typed results), implementation in our runtime as the validation.
• Related work: row types for SQL (Links, Ur/Web), Polars' lazy schema validation, cuDF's runtime types, Arrow's existing schema language.
• Novel contribution: the integration. Schema refinements + Arrow + GPU + compile-time checking = a configuration nobody has shipped or proposed in published form.

The paper can land before the full Phase 5 implementation. A research prototype implementing a subset of operations (select, filter, with_column, simple aggregations) is enough to validate the core design and produce empirical evidence for the paper. This is plausibly a 6-month focused effort, parallelizable with the rest of Phase 5 work, possibly led by a collaborator if the project gains one.

8. Open issues

1. Variable-length schemas (the OpenTelemetry attributes case) are existential types — the schema includes "a map of string to value" where the value type is union-variant. Refinement typing over union variants is non-trivial. The schema language probably needs a small extension. Worth thinking about before committing to "schemas as Phase 0's existing refinement language" verbatim.

2. Schema evolution. Real-world data pipelines have schemas that change over time. A typed-dataframes library needs an answer for "this column was renamed in v2 of the data; existing pipelines should still work." Approaches: schema migration as a typed transformation, dual-typing with deprecated columns, runtime fallbacks. Open.

3. Interaction with sharding (Phase 4 work). A dataframe sharded across devices has both a schema axis and a sharding axis. The composition is additive — same as for tensors — but the row-axis sharding interacts with operations like join in non-trivial ways. Defer to Phase 5+; flag now.

4. Exposing pandas/Polars-like surface ergonomics. The compile-time-typed API is precise but more verbose than df.filter(df.age >= 18). Surface syntax (Phase 7) needs to recover ergonomics without sacrificing type-checking. Same problem as for the tensor surface; same answer (elaboration with implicit refinement-context-passing).

5. The "small data" use case. Refinement-typed dataframes are most valuable for large analytical workloads — the kind where compile-time checking saves runtime cost. For small dataframes (a few hundred rows), the type-checking cost may exceed the runtime cost. The library should be opt-in for small data, mandatory for large.

9. Summary

Three commitments derived from the Arrow + observability angle:

The track has unusually good alignment — the architecture supports it cleanly, the technical contribution is genuinely novel, the ecosystem (Arrow, OTel-Arrow, RAPIDS) is moving in the same direction, and the validation target (your existing observability stack) is real rather than hypothetical.

This document does not commit to doing the work. It commits to not foreclosing the work with Phase 0/1 decisions, and to recording the architectural opportunity so it isn't accidentally lost. If the dataframes track gets picked up later — by you, by a collaborator, by anyone — the foundation will be ready.

End of future-track design.