Laya-MLX as a Sub-15 ms Decision Layer for Orchestrating Specialised SLMs on Apple Silicon

Research paper · 22 September 2026

Abstract

Multi-model systems built from specialised Small Language Models (SLMs) need a component that decides, for every request, which model should answer. Using a generative language model for that decision spends latency, tokens and parsing effort on what is fundamentally a classification problem.

This paper examines Laya-MLX, an open-source native MLX runtime for Convai Innovations' Laya typed decision models, as a routing and gating layer for local-first SLM orchestration on Apple Silicon. Laya evaluates a state against declared, typed questions (choice, score and noul) in a bidirectional forward pass and returns calibrated probabilities with zero output tokens.

Using the port's published M3 Max measurements — 13.42 ms P50 for the 421M English checkpoint, 7.39 ms for the 322M multilingual checkpoint, peak allocation below 1 GiB — we propose a three-primitive routing architecture: a noul guardrail gate, a score-based complexity escalation ladder and a choice-based domain router. We give a reference asynchronous implementation against the laya-mlx 0.2.0 API, a latency and memory budget, and the limitations that bound the approach, chiefly that zero-shot accuracy of the base checkpoint is modest and that routing quality depends on task-specific specialisation and calibration.

Keywords: model routing, small language models, typed decisions, MLX, Apple Silicon, bidirectional encoders, on-device inference

1. Introduction

A growing class of AI systems replaces one large generalist model with several smaller specialists: a code model, a mathematics model, a general conversational model, and an escape hatch to a frontier model for the hardest requests. The economics favour this design when most traffic is easy, and the privacy case favours it when data must stay on the device.

The weak point is the router.

If the router is itself a generative model, every request pays for token-by-token decoding before any useful work starts, and the application must parse free text and recover when the router answers conversationally instead of emitting a strict route tag.

Laya-MLX [1] offers a different primitive. Rather than generating an answer, it scores a fixed, application-declared set of answers. The output space is closed by construction: a choice question can only return one of the labels it was given. This paper argues that this property, combined with single-digit to low-double-digit millisecond latency and a sub-gigabyte footprint, makes Laya-MLX a natural fast "System 1" layer [8] sitting in front of slower, more capable "System 2" SLMs.

Our contributions are:

1. A description of Laya-MLX's decision primitives as routing operators.
2. A three-stage gate, escalate and route architecture.
3. A reference asynchronous router written against the published package API.
4. An explicit statement of the evidence and its limits, separating the port's measured figures from design estimates.

2. Background

2.1 Laya and typed decisions

Laya is an open-weight, non-autoregressive decision model from Convai Innovations [4][5]. It takes a state — text, a JSON object or a conversation — and one or more typed questions, and computes answers through a bidirectional encoder followed by a decision Transformer, a scoring head and an action head [2]. The pipeline is:

state + typed question → bidirectional encoder → decision heads → probabilities

The English checkpoint is built on ModernBERT-large [6]; the multilingual checkpoint on mmBERT-base [1].

2.2 Laya-MLX

Laya-MLX is an independent Apache-2.0 port by mizorewww that reimplements the encoder, decision Transformer, scoring head and action head in Apple's MLX framework [1][7]. It does not require PyTorch, the Transformers runtime or a cloud API at inference time; after the first checkpoint download, inference is fully local. It retains upstream weights, prompt formatting, temperature calibration and output schema, and is explicitly not an official Convai Innovations release [1][2]. Pre-converted FP16 checkpoints are published on Hugging Face [2], and the package is distributed on PyPI [3]. A Rust port running on CPU, CUDA and Metal also exists [11].

Table 1. Checkpoints supported by Laya-MLX [1]. Context covers instructions, options and state together.

2.3 Related work on routing

Learned routing between models of different cost has been studied as preference-trained routers (RouteLLM [9]) and as cost-aware cascades that escalate only when a cheaper model is insufficient (FrugalGPT [10]). Both show that most of the saving lies in sending easy queries to cheap models.

The contribution of Laya-MLX to this line of work is operational rather than algorithmic: it makes a general-purpose, schema-bounded decision model cheap enough to run locally in front of every request.

3. Decision primitives as routing operators

The laya-mlx 0.2.0 Agent.predict call returns, per question, a dictionary with a confidence value and an action.act_probability, plus a type-specific payload [3]. Each type maps directly onto a routing role:

Table 2. Typed decision outputs in laya-mlx 0.2.0 and their role in an orchestrator.

Three properties matter for orchestration.

First, the output is closed: the router cannot invent a route that does not exist. Second, the output is a distribution rather than a single label, so the orchestrator can threshold on margin and treat low-confidence cases differently. Third, the reported token usage has output_tokens: 0; cost scales with input length and the number of questions, not with answer length.

One constraint shapes the design. Each question row is encoded jointly with the state, so the runtime does not encode the state once and reuse it across arbitrary questions [1]. Three questions therefore cost three encoder rows, although they are batched into a single forward pass up to batch_size.

4. Proposed architecture

We place Laya-MLX between the user prompt and the model pool and ask it three questions in one batched call.

                          User prompt
                               │
  ┌────────────────────────────▼────────────────────────────┐
  │  Laya-MLX decision layer (System 1, one forward pass)    │
  │                                                          │
  │    noul            score              choice             │──► P>t ──► Refuse /
  │  guardrail gate  complexity 0-4    domain route          │           short-circuit
  └───────┬──────────────────┬───────────────────┬───────────┘
        0-1                2-3                   4
          │                  │                   │
          ▼                  ▼                   ▼
  ┌───────────────┐  ┌─────────────────┐  ┌──────────────────┐
  │ Tier 1: ~1B   │  │ Tier 2: ~7B     │  │ Tier 3: frontier │
  │ fast, local   │  │ pool, local     │  │ API, cloud       │
  └───────────────┘  └────────┬────────┘  └──────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
        Coding SLM       Maths SLM      General SLM
     choice=coding     choice=maths    choice=general

Figure 1. Gate, escalate and route. A single batched Laya-MLX call answers all three questions; only the selected SLM is woken.

• Gate (noul). Propositions such as "the request is unsafe or out of scope" or "answering requires live web data" are evaluated first. If P(true) exceeds a threshold, the pipeline short-circuits to a refusal, a retrieval step or a canned response without loading any heavy model.
• Escalate (score). A five-level complexity rubric yields an expected level in [0, 4]. Levels 0 to 1 go to a ~1B model, 2 to 3 to a ~7B specialist, and 4 escalates to a frontier API. Using the expected value rather than the argmax gives a smooth signal that can be thresholded and tuned.
• Route (choice). Within the local tier, a choice question over specialist identifiers — for example coding, maths, general — selects the model. Label descriptions can be supplied as a dictionary, which gives the model more to condition on than bare names.

Low-margin outcomes are first-class. If the top two choice probabilities are close, the orchestrator can send the request to the more general model, or escalate one tier, rather than trusting a near coin flip. Laya-MLX clamps fitted calibration temperatures to [0.5, 5.0] precisely because an unclamped bucket would report a coin flip as near certainty [1].

5. Reference implementation

The listing below targets laya-mlx 0.2.0. MLX calls are blocking, so the decision step runs in a single-worker thread executor to keep the event loop free; the downstream model clients are placeholders for whichever local servers host the SLMs. Thresholds and model identifiers are illustrative and should be fitted on the deployment's own traffic.

import asyncio
from concurrent.futures import ThreadPoolExecutor
import laya_mlx as laya

agent = laya.load("aac6fef/laya-mlx", dtype="float16", batch_size=16)
_pool = ThreadPoolExecutor(max_workers=1)   # one MLX caller at a time

QUESTIONS = {
    "unsafe": {"type": "noul",
               "instructions": "Is the request harmful or outside the assistant's scope?"},
    "complexity": {"type": "score",
                   "instructions": "How hard is this request to answer well?",
                   "criteria": ["trivial lookup", "simple", "moderate",
                                "hard, multi-step", "expert, open-ended"]},
    "domain": {"type": "choice",
               "instructions": "Which specialist should answer the request?",
               "criteria": {"coding":  "writing, reviewing or debugging code",
                            "maths":   "calculation, proofs, quantitative reasoning",
                            "general": "everything else"}},
}
LOCAL = {"coding": "qwen-coder-7b", "maths": "phi-4", "general": "llama-3b"}

async def decide(prompt: str) -> dict:
    loop = asyncio.get_running_loop()
    res = await loop.run_in_executor(_pool, agent.predict, {"request": prompt}, QUESTIONS)
    return res["answers"]

def plan(a: dict, gate=0.7, margin=0.15) -> str:
    if a["unsafe"]["noul"] >= gate:
        return "refuse"
    level = a["complexity"]["score"]          # expected level in [0, 4]
    if level >= 3.5:
        return "frontier-api"
    p = sorted(a["domain"]["probabilities"].values(), reverse=True)
    domain = a["domain"]["choice"] if p[0] - p[1] >= margin else "general"
    return "tiny-1b" if level < 1.5 else LOCAL[domain]

async def handle(prompt: str) -> str:
    target = plan(await decide(prompt))
    return await call_model(target, prompt)   # your SLM / API client

async def call_model(target: str, prompt: str) -> str:   # placeholder
    return f"[{target}] would answer: {prompt[:40]}"

Listing 1. Asynchronous gate, escalate and route orchestrator against the laya-mlx 0.2.0 API.

For repeated workloads, laya.load(..., compile=True, pad_to_multiple=16, cache_prompts=True) enables compilation and a bounded prompt-prefix cache; in the port's Snake benchmark this path was about 6.5% faster than its eager control [1]. For mixed-language traffic, the package's Router sends non-English input to the multilingual checkpoint and keeps at most max_loaded checkpoints resident [1].

6. Latency and memory budget

All figures below are the port author's measurements on an M3 Max (40 GPU cores, 128 GiB), FP16, end to end including prompt preparation, tokenisation, inference, calibration and formatting, but excluding model loading [1].

Table 3. Published Laya-MLX performance [1].

Two consequences follow for orchestration. The router's resident cost is below 1 GiB, leaving most of a Mac's unified memory for the downstream SLMs. And a three-question decision (Listing 1) costs three encoder rows in one batched pass; its latency will exceed the one-question P50 and should be measured on the target machine rather than assumed. An independent community report cites roughly 11 decisions per second for a three-question Snake loop with a planner on an M4 Max [12], while the port's optimised Snake path reports 75.40 moves per second on M3 Max [1]; workload shape dominates.

By comparison, a generative router must at minimum complete a prefill and emit a route token or short JSON object. We estimate that this commonly adds a few hundred milliseconds to around a second on local hardware, depending on model size, prompt length and output format. This comparison is a design estimate, not a measurement reported here.

Fidelity of the port is well evidenced: all three checkpoints matched the upstream selected answer on 63/63 validation questions in both FP32 and FP16 (378/378 comparisons), with deterministic outputs and zero measured active-memory growth over 100 repeated calls [1][2]. This establishes that the port reproduces Laya; it does not establish that Laya routes a given deployment's traffic correctly.

7. Limitations

• Zero-shot accuracy. Independent analysis of the project's typed-decision evaluation reports the base model only slightly above a random baseline in zero-shot use, with large gains after task-specific fine-tuning [13]. Routing labels should be validated on labelled traffic, and fine-tuning — which lives in the upstream project, not the port — is likely required for production quality [1].
• Context budget. Questions and options share the context with the state: 512 tokens for the English checkpoint [2]. Long prompts must be truncated or summarised before routing, and large label sets need the port's opt-in predict_shortlist [1].
• Confidence is not correctness. Calibrated probabilities help thresholding, but the port itself notes that confidence does not guarantee accuracy [1].
• Platform. Laya-MLX targets Apple Silicon (Python 3.11+, macOS 14+); the published measurements were taken on macOS 27.2 with MLX 0.32.2, and older macOS versions were not tested [1]. Cross-platform deployments would need the Rust port [11] or upstream PyTorch.
• Provenance. Laya-MLX is a community port; model quality, calibration and language limitations are those of the upstream checkpoints [2].

8. Conclusion

Routing is overhead, and overhead should be cheap, bounded and predictable.

Laya-MLX makes a closed-vocabulary, calibrated decision in roughly 7 to 14 ms on Apple Silicon with under 1 GiB of memory, which is the right cost profile for a component that runs before every request. Its three primitives compose into a clean gate, escalate and route pipeline in which heavy models are woken only when needed.

The open question is not speed but accuracy on a given traffic distribution; the next step is a labelled routing benchmark comparing zero-shot Laya, a fine-tuned Laya checkpoint and a generative router on accuracy, latency and cost under identical hardware.

References

1. mizorewww. Laya-MLX: Native MLX runtime for Laya typed decision models. GitHub repository, Apache-2.0. github.com/mizorewww/laya-mlx (accessed 22 September 2026).
2. aac6fef. laya-mlx model card (FP16 MLX conversion of convaiinnovations/laya). Hugging Face (accessed 22 September 2026).
3. laya-mlx 0.2.0. Python Package Index (accessed 22 September 2026).
4. Convai Innovations and contributors. Laya. GitHub repository.
5. Convai Innovations. convaiinnovations/laya. Hugging Face.
6. Warner, B., Chaffin, A., Clavié, B., et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (ModernBERT). arXiv:2412.13663.
7. Hannun, A., Digani, J., Katharopoulos, A., Collobert, R. (2023). MLX: Efficient and flexible machine learning on Apple silicon. github.com/ml-explore/mlx.
8. Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.
9. Ong, I., Almahairi, A., Wu, V., et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665.
10. Chen, L., Zaharia, M., Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176.
11. laya-rs: Rust port of Laya-MLX. docs.rs (accessed 22 September 2026).
12. hqmank (Kai). Post on X describing the Laya-MLX Snake demo on an M4 Max. x.com/hqmank (accessed 22 September 2026).
13. Zima Store. Laya Model: Open-Source Decision AI You Can Run Locally. shop.zimaspace.com (accessed 22 September 2026).