2 posts tagged with "mlx"

Running a 35B model on your laptop is impressive right up until you watch it type. Local inference on Apple Silicon is memory-rich and bandwidth-poor: every single token needs its own forward pass through the whole model, and that pass is dominated by shuffling weights from unified memory into the GPU. The math is fast. The waiting is not.

MTPLX attacks exactly that bottleneck, and it does it without the usual trade-offs. On a Mac mini it lands around 1.6x faster decode; on Qwen 3.6 27B the author measures up to 2.24x. And it does that while producing the same output your model would have produced anyway.

Cloud APIs are convenient, but they come with per-token costs, rate limits, and the nagging question of where your code actually goes. What if you could run the same frontier-class models (GLM, MiniMax, Kimi, DeepSeek) at full Q6 or even Q8 quality, entirely on hardware you already own?

With DevoxxGenie's refreshed Exo integration, you can. Pool the memory of multiple Apple Silicon Macs into a single inference cluster and run models that would never fit on one machine. No subscription. No token budget. Just electricity.