Execution Engine Xaccel Extension — CM Coprocessor Offload Specification

Document version: 0.1 (draft) Status: Initial design capture — provisional; depends on the DeMon CM5 PCIe link, which is not yet characterised for latency or bandwidth Parent document: Execution Engine ISA Companions: Xctx Extension, Xmath Extension, Accelerator — AntOS Compatibility Layer

1. Overview

The Xaccel extension lets native EE code hand work to the DeMon compute module (Raspberry Pi CM4 / CM5) as a coprocessor, over the PCIe link that already connects the FPGA to the CM. The CM's quad-core ARM, Linux userland, and VideoCore GPU take jobs that are too heavy or too resource-hungry for the FPGA soft core — heavy math, codecs, ML inference, decompression, rendering — while the EE stays the real-time core throughout. An offload is an instruction, not an OS call: the bus and the driver stack are hidden below the ISA.

There are two shapes of offload, chosen by whether the EE needs a result back:

• Posted offload — fire-and-forget. The EE writes a job, or streams data, into a PCIe-mapped shadow window in CM memory and does not wait. Used for rendering and streaming; results, if any, return out of band (e.g. the CM's HDMI display-overlay layer, composited by the EE). It is bandwidth-bound, not latency-bound, and never stalls.
• Reply offload — the job produces a value the EE consumes. The destination register is scoreboarded exactly like a long-latency multiply or memory load: register-unrelated instructions keep running, and the pipeline stalls only if it reads the result before the CM has returned it.

The two-axis rule: bandwidth is plentiful either way (stream all the data you like), and latency only matters when you wait for a reply — and even then the EE's existing out-of-order scoreboard hides it behind independent work.

1.1 Wins

• Each engine does what it is best at. The fabric is superb at deterministic, beam-locked, parallel chipset work but weak at long sequential/complex algorithms; the CM is the reverse. Xaccel couples them without a driver wall in the programmer's way.
• No new stall machinery. The reply path reuses the EE's shallow-out-of-order scoreboard (the same mechanism behind the DSP-block multipliers), so overlapping PCIe latency with independent work is free.
• Rendering on a real GPU. Posted offload turns the VideoCore GPU into a render coprocessor — 3D, complex compositing, shader effects, video — output composited by the EE as an ordinary display layer, double-buffered and swapped on the system vsync.
• Full-rate streaming. Posted writes move data at link rate (≈ 400 MB/s on one CM4 Gen2 lane, more with pciex1_gen=3) — a full display list or scene per 60 Hz frame, one-way, with no stall.

1.2 Non-Goals

• Not real-time. The CM runs Linux; offload latency is non-deterministic. Xaccel is for work that tolerates that — hidden behind the scoreboard or streamed one-way — never for beam-locked chipset timing.
• Not a replacement for Xmath. Short DSP / game math stays in fabric via Xmath; Xaccel is for work whose size dwarfs the PCIe round-trip.
• Not coherent shared memory. The shadow window and the mailbox are explicit, software-managed staging areas, not a cache-coherent view of CM RAM.
• Not guaranteed delivery. A reply offload can time out (CM error, reset, service crash); the extension faults rather than hanging, but delivery is best-effort.

2. Relationship to Standard RISC-V

Xaccel adds a small number of instructions in the wide-mode-only 0x7F escape space (ISA §7.5), so it is available in wide mode only and vanilla RV64GC never encounters it. The posted-write path needs no new opcode at all: the shadow window is a plain memory-mapped region (§4), so ordinary stores into it are the posted-write mechanism. Only the reply path and the job-control primitives are new instructions.

Per the §7.5 escape discipline, Xaccel must define its format-bit value ([35]) and a dispatch sub-field within the 29-bit payload that does not collide with the existing Xcrisp PIC allocations — Xcrisp uses both format bits ([35]=0 for PC-relative, [35]=1 for register-indirect), so Xaccel selects a free sub-encoding under one of them and registers it in the Xcrisp Reserved Spaces table. The specific sub-encoding is TBD (§11).

Xaccel adds no CSRs to the base ISA namespace beyond the configuration block in §5.

3. The Two Modes

3.1 Posted offload (fire-and-forget)

The EE writes into the shadow window — a region of CM memory exposed in the EE's address map through the PCIe endpoint. Because PCIe posted writes require no completion, these writes never stall: The EE produces (a display list, a scene, a command stream), the CM consumes at its own pace. Two idioms:

• Streaming: plain stores (or a DMA/block-memory burst) into the window. No Xaccel instruction required.
• Descriptor job: OFFPOST (§6) rings the CM's doorbell with a job descriptor, for a one-shot posted job (e.g. "render this scene"), still with no reply.

The result, if the job has one, returns out of band — most often as the CM's HDMI overlay layer, which the EE already composites. Double-buffer the shadow region and swap on vsync and the producer/consumer pipeline hides the whole latency: The EE fills frame N+1 while the CM renders frame N.

3.2 Reply offload (scoreboarded)

OFFLOAD rd, rs1 issues a job whose result the EE will read. rd is marked pending in the register scoreboard the instant the job is issued, exactly as the destination of a long-latency multiply or load would be. The EE keeps issuing and retiring every instruction that does not depend on rd. When the CM completes, a PCIe write lands the result and clears the pending bit; if an instruction reaches a read of rd before then, the pipeline stalls until it lands. Schedule independent work between the OFFLOAD and the first use of rd and the latency is hidden for free.

When the latency is too long to stall on — no independent work available, and a long dependency stall is unacceptable — issue the offload and HALT the Xctx context instead; the CM's doorbell raises an interrupt whose handler issues RESUME for the halted context (Xctx's "cheap interrupt-driven I/O" pattern), while other real-time contexts run meanwhile. Rule of thumb: scoreboard when you have work to overlap; Xctx when you would rather run something else entirely.

Backends — near (A25) and far (CM)

(Section numbering to be normalised when the draft firms up.)

Concept stage. The near (A25) backend below is exploratory — a design sketch, not a committed mechanism. It reuses hardware that already exists (the hard core, BSRAM, the scoreboard), but the plumbing is unbuilt and unmeasured; treat all of it as provisional.

Two coprocessors answer the same Xaccel instructions; a job's descriptor selects which:

• Near — the on-die Andes A25. The GW5AST-138's hard RISC-V core (AE350 subsystem, 800 MHz, RV32 with a hardware double-precision FPU and P-extension DSP/SIMD, 32 KB I/D cache) sits one AXI hop from the fabric. It is the low-latency, deterministic tier: run it bare-metal and a job's turnaround is predictable, measured in hundreds of nanoseconds rather than the CM's microseconds.
• Far — the DeMon CM5 over PCIe. The bulk / Linux / GPU tier described above.

The two differ in what runs: the near A25 is RISC-V, so it executes RISC-V kernels directly; the CM is ARM, so a far offload invokes a native ARM service, not shipped RISC-V code. Either way OFFLOAD only scoreboards a result — it is a normal instruction that pends its destination register and lets the EE run on, never an illegal-opcode trap, so there is no trap overhead and no loss of parallelism.

Pick by job: DP-FP or DSP work that is latency-sensitive → near; bulk, complex, or library/GPU work → far.

The near backend in detail

The A25's working memory is a region of fabric BSRAM exposed as a slave on its AE350 bus and writable by the EE — the mailbox and any per-job code/data live there, on-die, with no DDR3 round-trip. Because the A25 caches (32 KB I/D), the mailbox region is mapped non-cacheable on the A25 side (via its PMA), so it always sees the code and datan EE just wrote — no stale-cache hazard. (Hot, fixed routines can instead live in a cacheable library loaded once; only the per-job mailbox then needs to be non-cacheable.)

A near offload runs like this:

1. The EE stages the job in the BSRAM mailbox — routine pointer, operands, and the target: which register rd, in which Xctx context — and marks rd pending in the scoreboard.
2. The EE raises an interrupt to the A25; its handler runs the routine bare-metal (no OS jitter).
3. The A25's result write is plumbed straight into the EE's register-file writeback. Writing the result drives the (rd, value) write port and clears the scoreboard pending bit in the same action — so the A25 is, in effect, a remote long-latency functional unit whose writeback is the completion, exactly like the DSP-block multiplier. No mailbox read-back, no EE-side ISR for the result.

Because the target carries the context, the writeback lands in the correct context's rd and can wake it if it HALTed on the result (§3.2). A scalar result lands in rd directly; a buffer result lands in the mailbox and rd receives a status word over the same writeback path — either way the writeback completes the scoreboard entry. A fault or timeout rides the same path, writing rd a fault marker that traps on use rather than hanging.

This tight completion is the near backend's edge: the on-die A25 injects its writeback directly into the scoreboard, so its completion is cheaper than the far backend's, where the CM (across PCIe) raises a doorbell that the Xaccel logic must turn into the writeback.

Granularity — SIMD and function-kernels, not single scalar ops

The near backend is not used to offload individual scalar FP instructions: the per-op dispatch overhead would dominate, and dependency-chained FP (fmul.d, then fadd.d on its result) would stall on almost every use. It is used at two coarser grains, where the compute dwarfs the dispatch:

• SIMD / vector ops. A single vector op already does batch work, so the dispatch cost is small against the compute — a natural fit for the A25's P-extension SIMD.
• Function-kernels. Whole routines — a DP transcendental, an FFT, a matrix multiply, a physics step, a codec block, a crypto primitive — compiled as A25 code resident in BSRAM and invoked as a unit. Each is, in effect, a custom complex instruction whose implementation is an A25 function: The EE issues one offload naming the kernel's handle, its operands, and the destination (context, rd); the A25 runs the whole routine and completes through the direct writeback. One scoreboard entry covers the entire kernel, so its (longer) latency hides behind independent work and the dispatch overhead is amortised across the routine. The dependency chain lives inside the kernel, not across the EE↔A25 boundary.

This is a powerful extensibility path: a new accelerated operation is a routine you load and register a handle for, not fabric you build or a soft-core instruction you add. The hardened FPU / SIMD do the work; the fabric stays free for the chipset.

It also settles the cache question. A kernel library is fixed code, so it can be cacheable (loaded once, runs fast); only the per-call operand/result mailbox need be non-cacheable — and the kernel manages its own FP state (fcsr etc.) internally, returning only the result and, if needed, a status word. The A25's own register save/restore lands in BSRAM too, and is minimal and per-kernel: RISC-V leaves all register saving to the trap handler (hardware writes only mepc / mcause / mstatus on entry), so a compiled kernel's stub saves only the GPRs it clobbers — and FP registers only when mstatus.FS shows it dirtied them, letting an integer-only kernel skip the FP context entirely. Mechanically this is nothing new: a function-kernel call is a scoreboarded reply-offload on the near backend, completing through the direct writeback above. The refinement is a rule about what is worth dispatching, not new hardware.

Kernel reach — the FPGA register space

A near kernel is not limited to the operands staged in its mailbox. If the fabric's register space (the EE chipset registers and memory-mapped I/O) is exposed as a slave on the A25's AE350 bus, a kernel can read and write chipset state directly. That gives two flavours of kernel:

• Pure — operands in, result out through the writeback; no side effects. The clean scoreboard case.
• Side-effecting — the kernel reaches into the register space and acts on the chipset: a batch sprite-attribute update, copper-list construction, blit setup, audio-parameter writes. It may return a value or just a completion status; either way it runs in parallel with the EE, which stalls only when it reads the return register — the completion status doubling as the sync point.

This makes the A25 a genuine parallel agent on the chipset, not merely a value-returning functional unit — a second worker touching the same register space alongside the EE. Two things that reach implies:

• Arbitration. Two masters on the chipset register bus (the EE and the A25) need arbitration, and software must avoid conflicting concurrent access — e.g. don't have both rewriting the same sprite in the same frame. Ordinary shared-bus discipline, but it has to be designed in.
• Timing. The A25 is not beam-locked the way the EE is. Beam-critical, mid-scanline register changes stay with the EE and the Copper; the A25's register reach is for batch or between-frame work — setting up the next frame's state, not hitting a register at an exact pixel.

4. Address Map — Shadow Window, Mailbox, Doorbell

Three CM-facing regions appear in the EE's MMIO map (base addresses set by the CSRs in §5; sizes provisional):

A job descriptor in the mailbox is a small fixed record (fields provisional):

For a reply job whose result is a single word (a computed scalar, or a completion/error code), the CM writes it straight into rd; larger results land in result_ptr and rd receives a status word.

5. Configuration (CSRs)

(CSR numbers TBD — coordinate with the other the EE CSR allocations in ISA §4.8.)

• xaccel_winbase — shadow-window base / size
• xaccel_mbase — mailbox base / size
• xaccel_timeout — reply-offload timeout, in cycles (0 = no timeout, not recommended)
• xaccel_cap — read-only capability/status: link up, endpoint width/gen, service bitmap advertised by the CM
• xaccel_status — outstanding-job count, last error

6. Instructions

(Encoding provisional — all in the 0x7F wide-mode escape space.)

Posted data streaming uses ordinary stores (or Xcrisp block-memory / DMA) into the shadow window — no instruction here.

7. Trap Causes

(Cause numbers TBD — coordinate with ISA §4.9 / §5.6; Xstack 24–31, Xcond 32–34, Xctx 40–42, Xlate 32–34 are already allocated.)

• Offload timeout — a pending rd was read (or xaccel_timeout elapsed) and the CM never completed. The trap lets software recover rather than hang; the register is left in a defined faulted state.
• Offload error — the CM reported the job failed (bad service_id, service crash, argument fault). Delivered on the read of rd, or asynchronously if the descriptor requested it.
• Link down — an offload was issued while the PCIe endpoint was not up.

8. Examples

8.1 Render a scene on the CM (posted, no reply)

    ; write the frame's display list into the shadow window, then post the job
    ; (streaming stores omitted — plain SW/DMA into [xaccel_winbase])
    la    a0, render_desc        ; service_id = RENDER, arg = window range
    OFFPOST a0                   ; fire-and-forget; returns immediately
    ; ... The EE continues; CM renders; result arrives as an HDMI overlay layer

8.2 Heavy transform with a value back (reply, scoreboarded)

    la    a0, solve_desc         ; service_id = SOLVE, args in mailbox
    OFFLOAD a5, a0               ; a5 marked pending; CM works over PCIe
    ; --- independent work runs here, hiding the latency ---
    <dozens of instructions not touching a5>
    add   a6, a5, a7             ; first *use* of a5 — stalls only if not yet ready

8.3 Long job, switch contexts instead of stalling

    la    a0, infer_desc
    OFFLOAD a5, a0               ; a5 pending
    HALT                         ; Xctx: yield this context; run other real-time work
    ; context resumes on the CM doorbell, with a5 populated

9. Interaction with Other Extensions

• Xctx — the escape hatch for latencies too long to stall on: HALT on issue; the doorbell interrupt's handler issues RESUME (the Xctx interrupt-driven-I/O pattern — HALT, ISR RESUMEs on completion). A halted context is a natural fit for a pending offload.
• Xlate — posted writes to the shadow window are ordinary stores, so any write-translator configured on the source register applies. Use identity translation for raw streaming; a byteswap translator is available for free should a CM-side service ever want a different byte order (both ends are little-endian by default, so normally none is needed).
• Xmath — the boundary. Short, deterministic DSP/game math stays in fabric via Xmath; Xaccel is only for jobs whose size dwarfs the PCIe round-trip.
• Xcrisp — block-memory / DMA primitives are the efficient way to fill the shadow window for streaming.
• Xwide — Xaccel is wide-mode only (it lives in the 0x7F escape space).

10. Detection

The CM link and its advertised services are read from xaccel_cap (§5). Software must check link-up before issuing an offload; a job issued with the link down traps (§7). Because the CM advertises a service bitmap, code can fall back to a fabric or DDR3 implementation when a given service is unavailable (CM absent on a minimal build, or a service not installed).

11. Open Items

This extension is early and depends on hardware not yet measured. Provisional throughout; the larger unknowns:

• PCIe endpoint characterisation. The Gowin PCIe hard-IP endpoint's real latency and sustained bandwidth are unmeasured. These set the minimum worthwhile job size for reply offload and the achievable streaming rate for posted offload. Everything quantitative here is provisional until bring-up.
• A25 near backend. Several pieces need working out: the BSRAM budget for the mailbox / code and per-kernel context saves (working-set-only, not a full context per job, but still competing with sprite buffers, copper lists, scratchpad); the non-cacheable-region setup on the A25 (PMA config, or the cached-library split); the context-targeted writeback plumbing (how (context, rd) reaches the register-file write port and clears the scoreboard); and the fact that the A25 currently runs the EE debug-monitor firmware — whether it is dedicated to compute, time-shares, or runs both as one resident image.
• Instruction encoding. The instructions live in the 0x7F wide-mode escape (ISA §7.5); the format bit and dispatch sub-field are TBD and must be allocated per the §7.5 discipline — non-colliding with Xcrisp PIC, and recorded in the Xcrisp Reserved Spaces table.
• CSR and trap-cause numbers. Placeholders; to be assigned alongside the other extensions (Xstack 24–31, Xcond/Xlate 32–34, Xctx 40–42 are the current suggested allocations).
• Descriptor format and service ABI. The field layout in §4 and the CM-side service interface (how AntOS on the CM registers and dispatches services) need a companion spec.
• Doorbell mechanism. PCIe MSI vs a mapped doorbell register vs reuse of the EE→DeMon return path — to be settled with the interconnect.
• Result delivery for reply jobs. Register-direct vs buffer-plus-status, and the completion write's ordering relative to the doorbell.
• Security / isolation. A personality (via the compatibility layer) and native code share the CM; the service ABI must not let one reach another's state.
• Relationship to the compatibility layer. accelerator.md redirects a personality's accelerator calls to CM services; Xaccel is the native path to the same services. Both should resolve to one CM-side service model.