Picking an MoE Model for a 24GB Mac with MLX
Total parameters decide whether an MoE model fits in 24GB; active parameters decide speed. File sizes for gpt-oss, Gemma 4, LFM2.5, Qwen and Nemotron.
A mixture-of-experts model has to keep every expert in memory, even though each token uses only a few of them. On a 24GB Mac, the number that decides whether an MoE model fits is total parameters × bits per weight. The “A3B” or “A4B” in the name (the active parameters) does not shrink the file. It only sets how fast tokens come out. You can check this against Hugging Face: Qwen3.6-35B-A3B activates 3B parameters per token, but its mlx-community 4-bit build is 20.4GB of weights. That is too large for a 24GB machine before you add any context.
This article follows on from Running Local LLMs on a 24GB M4 Mac, which covers dense models and the basic memory budget. Here the question is narrower: which current MoE open-weight model, at which quantization, should you run on a 24GB Mac with MLX, and when is llama.cpp or Ollama the better runner?
Short answer: the MoE models that fit are the 20B-26B-total ones. Use gpt-oss-20b when you want headroom, and Gemma 4 26B-A4B or LFM2.5-24B-A2B when you want more model. The 30B-35B “A3B” models fit only at 3-bit, after pruning, or after you raise the GPU memory limit and accept a short context.
Which MoE models fit in 24GB
Metal reports the default GPU memory cap as recommendedMaxWorkingSetSize, and llama.cpp prints it at startup. Apple does not document how the value is calculated. On a 24GB Mac it is 17.18GB on macOS 15 (an M4 Pro on macOS 15.6 logs “17179.89 MB” in this benchmark thread) and about 19.07GB (17.76GiB) on macOS 26 and later (mlx-video #54, MoltenVK #2890). The companion article rounds this to about 18GB. Check your own log for the exact value. A practical target is weights at or under about 16GB, so that 1-2GB remains for context and runtime buffers. That margin is a rule of thumb, not a measurement.
All sizes below are the sum of the weight files in each Hugging Face repo (safetensors or GGUF) as listed on 2026-11-18. Parameter counts come from the model cards.
| Model | Total / active params | Quant (repo) | Weights | Fits a 17–19 GB budget? |
|---|---|---|---|---|
| gpt-oss-20b | 21B / 3.6B | MXFP4 MLX (mlx-community) | 12.1 GB | Yes, about 5GB left for context |
| LFM2.5-24B-A2B | 24.0B / 2.45B | MLX 4-bit (LiquidAI) | 13.6 GB | Yes |
| Gemma 4 26B-A4B | 25.2B (+~0.55B vision) / 3.8B | QAT Q4_0 GGUF (Google) | 14.4 GB | Yes |
| Gemma 4 26B-A4B | same | MLX 4-bit (mlx-community) | 15.3 GB | Yes, tight |
| Qwen3-Coder-30B-A3B | 30.5B / 3.3B | MLX 3-bit (mlx-community) | 13.4 GB | Yes, at 3-bit quality |
| Qwen3-Coder-30B-A3B | same | MLX 4-bit (mlx-community) | 17.2 GB | macOS 15: only after raising the GPU limit (this thread shows Qwen3-30B-A3B-4bit hitting a Metal out-of-memory error on a 24GB M4 Pro, macOS 15.6). macOS 26+: tight, short context only |
| Nemotron 3.5 Lightning 30B-A3B | 30B / 3B | MLX 4-bit (mlx-community) | 17.8 GB | macOS 15: only after raising the GPU limit (same out-of-memory risk as the 30B row above). macOS 26+: tight, short context only |
| Qwen3.6-35B-A3B | 35B / 3B | MLX 4-bit (mlx-community) | 20.4 GB | No |
| Qwen3.6-35B-A3B | same | UD-Q3_K_M GGUF (unsloth) | 16.6 GB | Borderline |
| Qwen3.6-35B-A3B, 50% experts pruned | 18.8B / 3B | OptiQ 4-bit REAP-19B (mlx-community) | 13.2 GB | Yes, with a quality cost |
| Qwen3.8-Flash-Next | 125B (+51B n-gram embeddings, +4B MTP) / 6B | MLX 4-bit (mlx-community) | 111.5 GB | No |
| Dense reference: Qwen3.8-27B | 27B / 27B | MLX 4-bit (mlx-community) | 16.1 GB | Yes, tight, but dense (slow) |
Three things in this table are easy to miss:
- The quant name does not tell you the size. For Nemotron 3.5 Lightning, the bartowski Q4_K_M GGUF is 25.5GB, while the MLX 4-bit build is 17.8GB. For Qwen3.6-35B-A3B, Q4_K_M is 22.1–22.3GB depending on the quantizer (Unsloth UD-Q4_K_M 22.1GB, bartowski 22.3GB) and MLX 4-bit is 20.4GB. Different quantizers keep different layers at higher precision. Check the file size in the repo, not the label.
- Qwen 3.8 has no small MoE. The open-weight Qwen3.8 releases on the Qwen Hugging Face org are the 27B dense model, Flash-Next, and a 2.4T-A95B model. If you want a Qwen-family model at about 13GB before context, the options are Qwen3-Coder-30B-A3B at 3-bit (13.4GB) or the expert-pruned Qwen3.6 REAP-19B (13.2GB). Qwen3.8-27B at 4-bit is 16.1GB, and because it is dense it is slow (see the speed table below).
- Pruning removes knowledge, not speed. The REAP-19B card says 50% of the routed experts were removed and active parameters are unchanged, so decode speed stays the same. The same card reports MMLU falling from 83.7 to 62.3, while HumanEval and BFCL-V3 drop by 1.3 and 1.0 points (91.5 to 90.2 and 92.5 to 91.5). Use it for coding and tool calls, not for broad knowledge questions.
Why total parameters decide fit and active parameters decide speed
An MoE layer has many expert blocks and a router. For each token, the router picks a few experts. Qwen3.6-35B-A3B has 256 experts and uses 8 routed plus 1 shared per token. Gemma 4 26B-A4B has 128 experts, 8 active plus 1 shared. Any expert can be picked for the next token, so every expert must already be in memory. A runner cannot load only the “active” 3B. That is why the file size follows total parameters:
Weight size (GB) ≈ total parameters (billions) × effective bits per weight ÷ 8
“4-bit” in MLX is not exactly 4 bits. The mlx-community builds use affine quantization with a group size of 64 (see the quantization block in each repo’s config.json). MLX’s quantize stores a scale and a bias for each group. At 16 bits each, that adds 0.5 bits per weight, so 4-bit is about 4.5 bits effective. MXFP4, which gpt-oss uses for its expert weights, shares one 8-bit scale across 32 values, so it is about 4.25 bits.
Check the formula against the table:
- Qwen3.6-35B-A3B: 35.1B × 4.5 ÷ 8 = 19.7GB (actual 20.4GB; the router weights stay at 8-bit and the repo includes the vision encoder)
- Qwen3-Coder-30B-A3B: 30.5B × 4.5 ÷ 8 = 17.2GB (actual 17.2GB)
- LFM2.5-24B-A2B: 24.0B × 4.5 ÷ 8 = 13.5GB (actual 13.6GB)
- gpt-oss-20b: 20.9B × 4.25 ÷ 8 = 11.1GB (actual 11.2GB for the MXFP4-Q4 build, 12.1GB for MXFP4-Q8, which keeps attention and embedding weights at 8-bit)
The active parameter count does not appear in the formula at all. A 35B-A3B model and a 35B dense model need the same memory at the same bit width.
Speed is the other way around. Decoding one token is mostly limited by memory bandwidth: the chip must read every weight that the token uses. For an MoE model, that is only the active parameters. An upper bound is bandwidth ÷ bytes read per token. Apple lists 120GB/s for M4 and 273GB/s for M4 Pro.
| Model | Active params | Bytes read per token (4-bit) | Ceiling, M4 | Ceiling, M4 Pro |
|---|---|---|---|---|
| LFM2.5-24B-A2B | 2.45B | ~1.4 GB | ~87 tok/s | ~198 tok/s |
| Qwen3.6-35B-A3B / Nemotron 30B-A3B | 3B | ~1.7 GB | ~71 tok/s | ~162 tok/s |
| Qwen3-Coder-30B-A3B | 3.3B | ~1.9 GB | ~65 tok/s | ~147 tok/s |
| gpt-oss-20b (MXFP4) | 3.6B | ~1.9 GB | ~63 tok/s | ~143 tok/s |
| Gemma 4 26B-A4B | 3.8B | ~2.1 GB | ~56 tok/s | ~128 tok/s |
| Qwen3.8-27B (dense) | 27B | ~16 GB | ~7.5 tok/s | ~17 tok/s |
These are theoretical ceilings, not benchmarks. Real decode speed is lower, because the runner also reads the KV cache, runs the router, and spends time on compute and scheduling. The ratios are the useful part. Every MoE model in this list has a ceiling 7-12× higher than the dense 27B that takes similar memory. Among the MoE models that fit, the ceilings are within about 1.5× of each other. So on a 24GB Mac, the active count is not what should drive your choice. All of these models are fast enough. What separates them is whether they fit.
The runner-up factor: context reserve
After weight size, the next factor is how much memory the KV cache takes at the context length you need. MoE models from 2026 mostly use hybrid attention, which keeps this small:
- Qwen3.6-35B-A3B puts full attention in only 10 of 40 layers. The rest are Gated DeltaNet layers with a fixed-size state. With 2 KV heads of dimension 256 (config.json), the cache is 10 × 2 × 2 × 256 × 2 bytes = 20KB per token at fp16. That is about 0.67GB at 32K tokens and 2.7GB at 128K.
- gpt-oss-20b alternates full attention with 128-token sliding-window layers (config.json). The 12 full layers with 8 KV heads of dimension 64 give 24KB per token: about 0.8GB at 32K and 3.2GB at 128K.
- Gemma 4 26B-A4B uses a 1,024-token sliding window in 25 of its 30 layers (config.json), so most layers hold a fixed amount of cache.
- LFM2.5-24B-A2B uses short-convolution blocks in 30 of 40 layers and attention in 10 (model card).
This is where the choice can change. gpt-oss-20b at 12.1GB leaves room for a 128K context. Gemma 4 at 15.3GB leaves room for a moderate context only. Qwen3-Coder-30B-A3B at 4-bit (17.2GB) leaves almost nothing, and its own card suggests cutting the context to 32,768 tokens if you run out of memory. The Qwen3.6 card advises keeping at least 128K of context to preserve thinking, which a 24GB Mac cannot do with that model.
You can also shrink the cache itself. mlx-lm’s generate accepts --kv-bits and --max-kv-size, and llama.cpp accepts -ctk / -ctv for a quantized KV cache.
MLX vs llama.cpp vs Ollama for MoE on Apple Silicon
The size arithmetic is the same in every runner. What differs is which files you can load and how much setup each needs.
MLX (mlx-lm). The current release is mlx-lm v0.32.2 (2026-10-16). It has architecture files for gpt_oss, gemma4, qwen3_5_moe, lfm2_moe and nemotron_h, among others. Two points matter for MoE:
- Version pins matter. The LiquidAI MLX card states that LFM2.5-24B-A2B “requires mlx-lm 0.32.0 or later. Earlier versions load the model but route the experts incorrectly.” A wrong router does not raise an error; you just get worse output. Upgrade before you judge a model.
- Some repos are packaged as vision-language models. The mlx-community cards for Gemma 4 26B-A4B and Qwen3.6-35B-A3B tell you to use
mlx-vlm. The gpt-oss and LFM2.5 repos use plainmlx-lm.
llama.cpp. llama.cpp has --cpu-moe and --n-cpu-moe flags that keep expert weights in CPU memory. Those flags help on PCs with a small discrete GPU. On Apple Silicon, CPU and GPU share one physical pool of memory, so moving experts to the CPU does not create space. GGUF is still useful here for two reasons. Some quant mixes have no MLX equivalent, such as unsloth’s dynamic Q3 for Qwen3.6 (16.6GB). And Google ships an official QAT Q4_0 GGUF for Gemma 4 26B-A4B (14.4GB).
Ollama. Ollama announced an MLX engine in preview with version 0.19 on 2026-03-30. That post covered Qwen3.5-35B-A3B and said to “make sure you have a Mac with more than 32GB of unified memory.” Ollama 0.30 (2026-05-13) added a llama.cpp-based path that “augments the MLX engine on Apple Silicon.” The library tags show MoE sizes directly: gpt-oss:20b is 14GB, gemma4:26b-a4b-it-qat is 16GB and gemma4:26b-mlx is 18GB, and qwen3.6:35b-a3b is 23GB. On 24GB, the gpt-oss and Gemma QAT tags are the realistic MoE choices in Ollama.
oMLX. The “omlx” in some searches is oMLX, an MLX-based server with a menu-bar app. Its KV cache has a RAM tier and an SSD tier, so a long prompt that was processed once can be reused. The SSD tier stores cache, not weights. The model weights must still fit in RAM, so the table above still decides what you can run.
Which one to run
Pick by total size first:
- Most model that fits at 4-bit: Gemma 4 26B-A4B. Use Google’s QAT Q4_0 GGUF (14.4GB) in llama.cpp or Ollama, or the mlx-community 4-bit build (15.3GB) through mlx-vlm. Keep the context moderate.
- Headroom for long context, an IDE and a browser: gpt-oss-20b in MXFP4 (12.1GB with MLX, 14GB as the Ollama tag). It leaves about 5GB even on the macOS 15 budget of 17.18GB, which is enough for a 128K cache.
- Agent and tool-call work with the highest speed ceiling: LFM2.5-24B-A2B at MLX 4-bit (13.6GB) with mlx-lm 0.32.0 or later. Read the license first: commercial use is not licensed for organizations with annual revenue of US$10 million or more.
- Qwen coder at about 13GB: Qwen3-Coder-30B-A3B at MLX 3-bit (13.4GB), or the Qwen3.6 REAP-19B build if you accept the MMLU drop its card reports.
The condition that changes the answer is memory. If you raise iogpu.wired_limit_mb, close other apps and keep the context short, the 4-bit Qwen3-Coder-30B-A3B (17.2GB) and Nemotron 3.5 Lightning (17.8GB) builds become reachable. On a 32GB Mac running macOS 26 or later (about 26.8GB GPU budget, as reported in this thread), Qwen3.6-35B-A3B at 4-bit (20.4GB) fits with long context: 26.8 − 20.4 − 1.5GB of runtime reserve leaves about 4.9GB, or roughly 240K tokens at 20KB per token. On macOS 15 (22.91GB budget), the same arithmetic leaves about 1.0GB, or roughly 49K tokens. With macOS 26 or later, Qwen3.6-35B-A3B becomes the default pick.
FAQ
Can I run Qwen3.6-35B-A3B on a 24GB Mac?
Is there a Qwen 3.8 model that uses about 13GB before context?
Does an A3B model need less RAM than a dense 30B model?
Will llama.cpp's --cpu-moe flag let a larger MoE fit on a Mac?
Related tools
Some links above are affiliate links. We may earn a commission if you sign up. See our disclosure for details.
Related reading
2026-11-18
Local LLMs on a 16GB M4 Mac: What Fits and What Breaks
A 16GB M4 Mac lets the GPU use about 11.45 GB (macOS 15) or 12.7 GB (macOS 26+). Which 4B-14B models fit at 4-bit, and why 20B+ breaks.
2026-11-18
How Much Mac Memory a 30B or 70B Local Model Needs
Weights plus KV cache must fit the GPU share of unified memory. What 32GB, 48GB, 64GB and 128GB Macs run at 30B and 70B, with the context math.
2026-10-15
What Muse Spark Open's 816B size means for developers
Meta's new open-weight coding model runs like a 42B model but needs the memory of an 816B one. Here's what that means for running it.
2026-10-14
What Anthropic's Scion means for Git and AI coding
Anthropic swapped Git for a custom system called Scion after merges jumped to 30,000 a day. Here is the background and what it means for your workflow.
2026-09-07
Grok 4.6 Scores 26% and 88% on the Same Benchmark Line
xAI's own model card reports Terminal-Bench 3.0 at 26.0%. Artificial Analysis reports 88.4% on 2.1. Both are the same model. The model card is unusually honest about why — the problem is everyone quoting it drops the qualifiers.
Get the best tools, weekly
One email every Friday. No spam, unsubscribe anytime.