RISC-V/RVV at the Edge: From PyTorch to vsetvli

A SpacemiT board lands on your desk. Somewhere on it, silicon markets itself as RVV 1.0. You have a PyTorch checkpoint. Somewhere between the two sits a genuinely interesting question that most MLOps content skips entirely: how does a tensor operation you wrote in Python actually become a vector instruction on a RISC-V core?

Not “can you run inference on RISC-V” — you can, apt install and go. The real question is what happens in between: which compiler owns that translation, whether OpenAI’s Triton has any business being in this conversation yet, and what the vector extension itself is actually doing differently from the CUDA model most of us think in by default.

This piece is that path, traced end to end, for the IMLG crowd.

The room RISC-V hasn’t entered yet

Let’s be honest about where we are before we get excited about where we’re going. Data-center AI inference in 2026 is still an NVIDIA market — Blackwell and Hopper-class GPUs hold the overwhelming majority of it. AMD’s MI300-series is the credible second. Inside China, a domestic stack has grown fast under export-control pressure: Huawei’s Ascend line, Alibaba’s in-house silicon, Biren, Moore Threads, Cambricon, Hygon’s CPU+DCU combination — all real, all shipping, none of it RISC-V-based at the accelerator level.

That’s the honest picture, and it’s not the subject of this post. Where RISC-V is moving — fast, and specifically with the RVV vector extension — is edge and IoT inference: traditional ML, RNNs, CNNs, vision, and now small/quantized LLMs running on boards that cost less than a mid-range phone.

One data point is worth flagging before we leave the data center behind, because it complicates the tidy “RISC-V stays at the edge” story: in March 2026, Alibaba’s T-Head division unveiled the XuanTie C950, a licensable RISC-V core IP built on a 5 nm process at 3.2 GHz, scoring over 70 on SPECint2006 — roughly triple its C920 predecessor and, T-Head claims, a new record for the architecture. It pairs a Vector Acceleration Engine with a Matrix Acceleration Engine (T-Head’s own AME, Attached Matrix Extension) under unified addressing, and is positioned to run models like Qwen3-235B-A22B and DeepSeek V3-671B natively on CPU — no GPU in the loop. CNX Software’s own framing for it is telling: “a powerful RVA23-compliant 64-bit RISC-V core for Edge AI computing.” The line between “edge” and “data center” is exactly where RISC-V is currently pushing hardest, and C950 sits right on it.

With that flagged, let’s go where the interesting engineering actually is for most of us: the edge.

RVV, not RISC-V — the vector model in one register file

Skip the ISA history. What you need is the vector extension’s actual mental model, because it’s genuinely different from what SIMD means on x86 or ARM.

RVV gives you 32 vector registers, v0 through v31. Their width — VLEN — is not part of the instruction set. It’s a silicon decision, made by whoever designs the core. A tiny embedded core might implement VLEN=128; the SpacemiT K3’s AI-oriented cores implement VLEN=1024. The instruction encoding is identical either way.

Three knobs turn that fixed register file into something flexible at runtime:

• SEW (Selected Element Width) — how wide each element is: 8, 16, 32, or 64 bits.
• LMUL (Length Multiplier) — how many physical registers get grouped to act as one logical vector register, from ⅛ up to 8. Group four registers with LMUL=4 and v8 through v11 behave as a single wide register for that instruction.
• vl — the actual vector length: how many elements the next instruction will touch. Set by the vsetvli instruction, and bounded by VLMAX = LMUL × VLEN / SEW.

That third one is the point of the whole design. A loop compiled once will ask the hardware, at runtime, “how many elements can you give me this time?” — and the same binary runs correctly whether it lands on a VLEN=128 microcontroller or a VLEN=1024 AI core, simply processing more elements per instruction on the wider chip. This is the vector-length-agnostic (VLA) programming model, and it’s the single biggest conceptual departure from fixed-width SIMD like AVX or NEON, where the instruction set itself bakes in the width.

The diagram below shows this literally: the same vsetvli line executing on a SpacemiT K1 (VLEN=256) processes 8 elements per instruction at SEW=32; on a K3 (VLEN=1024), the identical instruction processes 32 — four times the throughput, zero source changes.

The other piece worth internalizing is tail handling. Real loops rarely divide evenly by vector width. Ask for 5 elements (AVL=5) on hardware that can process 8, and older SIMD models force you to write a scalar cleanup loop for the remainder. RVV’s vsetvli carries a tail policy (ta = tail-agnostic, tu = tail-undisturbed) and a parallel mask policy (ma/mu) that tell the hardware exactly what to do with the leftover lanes — agnostic meaning “don’t care, may be garbage,” undisturbed meaning “leave whatever was already there.” Either way, the compiler doesn’t need to emit a separate scalar remainder path. That’s a small detail with a large payoff: it’s a big part of why auto-vectorized RVV loops look cleaner in generated assembly than their AVX equivalents.

Two ways to go wide: RVV’s SIMD vs CUDA’s SIMT

If you’ve spent your career thinking about parallelism through a CUDA lens, RVV requires a genuine model switch, not just new syntax.

CUDA’s GPUs are SIMT — Single Instruction, Multiple Threads. A warp of 32 threads executes the same instruction in lockstep, but each thread carries its own program counter and register state (conceptually — they reconverge after divergence). The scaling unit is the thread: an SM juggles many warps concurrently and hides memory latency by swapping between them. Branch divergence is handled by executing both paths and masking off the inactive threads on each, then reconverging — a real cost, but one the hardware absorbs transparently.

RVV is classic SIMD — Single Instruction, Multiple Data — but with the runtime-configurable width described above. There’s one program counter. One instruction touches vl elements of one wide register in the same core that’s running your scalar code. There’s no warp scheduler, no thread reconvergence, no occupancy tuning. The tail/mask policy plays the role that predication plays in SIMT, just at the instruction level instead of the thread level.

Neither model is “better” in the abstract — they’re solving different placement problems. SIMT amortizes control logic across thousands of threads and is superb at hiding latency behind massive occupancy; that’s what makes GPUs win at training and batch inference. RVV amortizes control logic across the width of one register and is superb at low-power, low-latency, single-stream inference — exactly the profile of a camera, a mic array, or a vibration sensor running one model, one frame at a time, on a power budget measured in watts, not hundreds of them.

From model.pt to vsetvli: how the toolchain actually gets there

Here’s the part IMLG readers usually never see, because MLOps tooling hides it well: your PyTorch or TensorFlow graph doesn’t reach RISC-V silicon through one obvious path. There are two roads today, and they’re at very different levels of maturity.

The Triton road: real, but early

You heard right that Triton — OpenAI’s kernel DSL — has some story here, but it’s important to be precise about what exists versus what’s roadmap. There is no native, OpenAI-maintained RISC-V backend in Triton today.

What does exist: Triton-CPU, a community-driven backend that lowers Triton kernels to LLVM IR targeting CPU rather than NVIDIA GPUs. RISC-V International and SpacemiT published a working demonstration running Triton-CPU on a K1 development board — proof that the path is viable, not that it’s production-grade. Separately, PyTorch’s own RISC-V support roadmap (tracked publicly as GitHub issue #171659) lists a native RISC-V/RVV backend for torch.compile’s Inductor stack as planned work, not shipped work. And at the more experimental end, individual engineers have hand-retargeted Triton’s AMD GPU backend to emit RVV instructions, getting a softmax kernel running correctly on the Spike ISA simulator — a genuinely clever proof of concept, and a clear signal of how far this still is from something you’d put in a production pipeline.

If your team is evaluating Triton-on-RISC-V today, evaluate it as “worth prototyping, not worth depending on.”

The LLVM road: why it’s the stronger bet

This is where the real infrastructure already lives, and it’s worth understanding in more depth — both because it’s more mature and because it’s the more strategically defensible bet for anyone building edge-inference pipelines over the next few years.

The reason LLVM matters more than any single backend is architectural: Clang/LLVM’s pipeline is target-agnostic by design. Source goes through lexing, parsing, and semantic analysis into an AST, then CodeGen lowers that AST into LLVM IR — a single intermediate representation shared across every backend LLVM supports. The backend stage (SelectionDAG or GlobalISel, then MC) is what’s target-specific, and LLVM already ships backends for x86, ARM, PowerPC, RISC-V, and — notably — NVPTX (NVIDIA GPUs) and AMDGCN (AMD GPUs). That’s not a coincidence worth glossing over: the same IR infrastructure spans the data-center accelerators dominating the space today and the RISC-V edge silicon this post is about. Betting on LLVM fluency is a transferable skill in a way betting on any single vendor’s stack isn’t.

For RVV specifically, Clang represents vectors using LLVM’s scalable vector types — written <vscale x n x ty>, where vscale is a runtime multiplier the hardware resolves at execution time rather than compile time. This is the IR-level expression of the same vector-length-agnostic idea from vsetvli: the compiler emits code that’s correct across VLEN implementations, and the actual width is resolved on the target. If you want to write RVV explicitly rather than rely on auto-vectorization, Clang exposes it through the riscv_vector.h intrinsics header, with types like vint32m1_t or vfloat32m4_t — the suffix encoding element type and LMUL directly in the type name. A -mrvv-vector-bits=<zvl|N> flag controls whether codegen stays fully scalable or targets a fixed width for extra optimization headroom when you know your deployment target.

For the common case — you didn’t write intrinsics, you wrote a plain loop — two LLVM passes do the heavy lifting: the Loop Vectorizer, which targets whole-loop vectorization, and the SLP (Superword-Level Parallelism) Vectorizer, which finds vectorizable patterns in straight-line code that isn’t a loop at all (adjacent scalar operations that can be fused into one vector op). Together they’re what let ordinary C/C++ — and by extension, whatever your ONNX runtime or graph compiler emits as C/C++ or MLIR-lowered-to-LLVM-IR — walk away with RVV instructions without anyone writing intrinsics by hand.

One honest caveat, because this space is genuinely still maturing on both fronts: recent academic benchmarking comparing GCC 15 against LLVM/Clang 21 on RVV auto-vectorization has found GCC ahead on some kernels. LLVM’s advantage right now isn’t “always generates better code” — it’s ecosystem breadth: MLIR sits on top of it, Triton’s own backends are built on it, and every RVV silicon vendor from SiFive to SpacemiT to T-Head ships vendor extensions through it. That breadth is why it’s the better long-term investment even where code-gen quality is still a live horse race.

The silicon zoo: who’s actually building RVV chips

The “RISC-V ecosystem” framing undersells how different these implementations are from each other. Some highlights, roughly ordered from smallest to largest vector width:

Two names deserve a specific callout because they didn’t stay in the RISC-V edge/vector story: Ventana Micro (Veyron V2, a genuinely server-class RVA23 vector core) was acquired by Qualcomm in December 2025, and Rivos was acquired by Meta in 2025. Both are signals that big compute buyers see real value in RISC-V vector cores — just not always as an open ecosystem play. Tenstorrent remains independent and continues developing RISC-V CPU IP (Ascalon) alongside its own AI accelerator chiplets. Akeana is a newer entrant licensing vector-capable application cores.

The practical takeaway for anyone shopping boards: SpacemiT K1 and K3 are the two you’ll actually find on a desk today — K1 in Banana Pi BPI-F3 and Milk-V Jupiter-class boards around $100–150, K3 in Pico-ITX-format “mini AI computer” boards with 60 TOPS of AI compute and full RVA23 compliance, cheap enough to homelab.

RVV 0.7 vs RVV 1.0 — and what’s shipping right now

This version split isn’t academic — it’s a real compatibility trap if you’re buying boards without checking.

If a board or IP core you’re evaluating only lists “RVV support” without a version number, assume you need to check — a 0.7.1-compiled binary will not run correctly on 1.0 silicon and vice versa. Nearly everything shipping new in 2026 targets 1.0, but there’s still a long tail of cheap 0.7.1-era boards in circulation from the early RISC-V dev-board wave, and they’re a genuine trap for anyone assuming “RVV” is a single target.

On tape-outs to watch: SpacemiT’s K3 is the most concrete recent one — announced mid-2025, now shipping in Pico-ITX boards through 2026, and the first widely available RVA23-compliant platform running mainstream Linux distributions out of the box. T-Head’s C950 (discussed above) is a licensable core rather than a shipping chip yet, but with partners already lined up, expect silicon carrying it within the next design cycle. Watch RISC-V International’s own profile announcements too — RVA23 is where the interesting mandatory-extension action is (vector goes from optional to mandatory under RVA23U64, along with a set of vector sub-extensions covered in the extensions section below).

Cheap, weird, and effective: cost innovations at the edge

The economic case for RVV at the edge isn’t subtle. A K1-class board runs a fraction of the cost of anything with a discrete GPU, draws single-digit watts, and — critically — doesn’t need one. A K3-class board, at roughly 35–40 W under full AI load, is running quantized 30B-parameter-class models at double-digit tokens/second entirely on RISC-V vector cores, no accelerator card, no PCIe GPU, in a chassis the size of an old Intel NUC.

The pattern worth internalizing for cost-conscious deployments: quantization (INT8/INT4) plus a wide-enough VLEN closes most of the gap that used to require a dedicated NPU or GPU for anything beyond toy models. SpacemiT’s own A100 AI cores explicitly support INT4/INT8/FP8/FP16/BF16 in hardware specifically to make this trade-off work — you’re not fighting the silicon to get low-precision throughput, it’s designed in from the start. For fleets of sensors running one model each — vision, keyword spotting, anomaly detection, vibration analysis — this is usually a better economic shape than a shared GPU server plus a network round-trip: no inference-serving infrastructure, no batching complexity, no round-trip latency, and a bill of materials an order of magnitude cheaper per node.

MQTT/EMQX: the nervous system tying it together

Here’s the piece that’s easy to miss if you’re thinking purely in “compiler → silicon” terms: a fleet of RVV edge nodes running independent local inference is only useful if something coordinates them, and the pattern that’s actually working at scale for this is old, boring, and exactly right for the job — MQTT.

MQTT’s publish/subscribe model with a 2-byte minimum header, long-lived connections, and QoS levels tuned for lossy networks was built for exactly this shape of problem: many low-power devices, intermittent connectivity, small payloads, no need for a request/response round trip per message. EMQX, one of the more widely deployed open-source MQTT brokers, runs natively across ARM, x86, and — explicitly — RISC-V, meaning the broker itself can sit on the same class of edge hardware as the inference nodes, not just in the cloud.

The architecture that falls out of this is simple and scales well: each edge node runs its model locally and publishes only the result — a classification, an anomaly score, a transcript fragment — to a topic. A broker (EMQX, self-hosted or as an edge instance) fans that out to a dashboard or Unified Namespace for live visibility, to a rule engine or lightweight AI agent that can act on the payload directly, and optionally escalates ambiguous cases upstream to a larger cloud LLM for a second opinion. Commands and model updates flow back down the same channel via subscription, so a fleet of thousands of nodes can be updated or redirected without anyone building a custom control-plane protocol.

The reason this matters more than it might first appear: it decouples the inference problem (solved locally, in silicon, per node) from the coordination problem (solved centrally, over a protocol designed for exactly this traffic pattern). You get the cost and latency benefits of on-device inference without losing fleet-level visibility or control.

India’s RISC-V moment — and the gap nobody’s filled yet

This deserves its own section because the story is genuinely different from the SpacemiT/T-Head narrative above, and it’s the part most relevant to where IMLG readers can actually contribute.

India’s RISC-V effort is real and has institutional depth. IIT Madras’s SHAKTI processor family, now commercialized through InCore Semiconductors, spans a range from the Azurite and Calcite CPU cores to the Axon series of deep-learning accelerators. C-DAC’s VEGA processor line and the THEJAS32/THEJAS64 cores round out a public-sector effort now formalized under MeitY’s Digital India RISC-V (DIR-V) program, with newer cores like DHRUV64 targeting 1 GHz dual-core designs. IIT Bombay’s AJIT project and ISRO-SCL’s VIKRAM (built for space-grade radiation tolerance) show the base-ISA and control-core work is genuinely broad. On the commercial IoT side, Mindgrove Technologies ships a Shakti-derived secure IoT MCU in silicon, with named industrial partnerships.

What’s honestly still missing: a widely known, shipping, RVV-class vector core out of India, comparable to what SpacemiT, SiFive, or Andes are shipping. The current strength is scalar, control-plane, and secure-IoT silicon — genuinely solid ground — but the vector-extension layer that’s driving the edge-AI story in this post is, as far as public information shows, still an open space. That’s not a criticism; it’s a gap, and gaps are opportunities. India has the base-ISA design experience (SHAKTI/InCore), the fabrication and program support (DIR-V, C-DAC), and now a very clear, very well-documented target architecture to build against (RVA22/RVA23 profiles, RVV 1.0, the whole LLVM toolchain this post just walked through). What’s missing is someone shipping the vector core.

Three RVV extensions worth watching

Quick hits, because the vector ISA isn’t standing still:

1. Zvfbfa — full BFloat16 arithmetic support, building on the more minimal Zvfbfmin/Zfbfmin extensions that RVA23U64 already makes mandatory. BF16 is the precision AI training and a growing share of inference has standardized on; this closes the gap between “can convert BF16” and “can compute in BF16 natively” in the vector unit.
2. OFP8 support (Zvfofp8min/Zvfmx8min) — 8-bit minifloat formats (E5M2/E4M3-style) aimed squarely at the low-precision inference trend already visible in production LLM serving. Expect this to matter a lot for exactly the quantized-edge-LLM use case discussed above.
3. Matrix extensions — a live standards race. There are two competing proposals rather than one settled answer: the community-driven IME (Integrated Matrix Extension — already appearing in shipping silicon, SpacemiT’s K3 lists IME-1.0 support) and T-Head’s own proprietary AME (Attached Matrix Extension, used in XuanTie C950 to interface with its TPE accelerator). Which one — or whether both — becomes the standard matrix story for RVV is genuinely unresolved, and worth tracking if you’re making multi-year architecture bets.

The ask

If you’re an MLOps or LLMOps engineer reading this on the IMLG blog, here’s the direct version: most of the tooling conversation in this community still assumes GPU-shaped inference, because that’s where the data center’s attention is, correctly, for training and large-batch serving. But a meaningful share of deployment — sensors, cameras, gateways, anything that has to run cheap, low-power, and offline-capable — is quietly becoming a RISC-V/RVV story, and the compiler infrastructure to target it (LLVM, today; Triton, soon) is maturing fast enough that “we’ll deal with it later” is starting to be the wrong call.

Go get a K1 or K3 board. Compile something with -march=rv64gcv. Read the vsetvli your compiler emits. And if you’re in India specifically — there’s real, unclaimed ground in RVV-class silicon itself, not just in software targeting it. Someone in this community should be building that core.

Further reading

• RISC-V International — RVV specification and ratified profiles (RVA20/22/23)
• Alexey Bataev, LLVM Compiler for RISC-V Architecture: A Unique Approach to Vectorization (Apress) — the deepest single treatment of Clang/LLVM’s RVV codegen path
• PyTorch RISC-V architecture support roadmap, issue #171659 — tracks native Triton/Inductor RISC-V backend work
• EMQX — MQTT broker with native RISC-V support
• CNX Software — ongoing coverage of SpacemiT K1/K3, XuanTie C950, and most new RVV silicon as it lands