Qwen3.8-27B on an M1 Max Mac Studio: My Own Benchmarks

I’ve spent a bit of time over the past week looking at Qwen3.8-27B reviews and benchmarks, partly because the benchmark claims around it are difficult to ignore, but also because some of the early user reports seemed to tell a very different story.

On paper, this is a 27.8B dense model being pitched very heavily around coding, agentic work, and computer use, with some comparisons to much larger frontier models. At the same time, I kept coming across people talking about excessive reasoning, huge token usage and painfully slow responses unless you got the reasoning settings just right.

That second part was probably what interested me more because I have a Mac Studio M1 Max with 64 GB of unified memory sitting here, and I’ve been trying to work out what role local models genuinely have for me rather than just running them because I can. The machine is now four years old and uses the first-generation Mac Studio silicon, but with 400 GB/s of memory bandwidth and 64 GB available to the system, it can still comfortably load models of this size.

I’d already seen plenty of published numbers for newer M4 and M5 Macs, but very little for the original M1 Max Mac Studio. Rather than keep trying to extrapolate what somebody else’s benchmark might mean for my hardware, I decided to spend a Saturday testing Qwen3.8-27B properly for myself.

What I found was slightly different from what I expected. The model itself was very good throughout my testing, and I couldn’t reproduce much of the reasoning behaviour I’d been reading about, but that doesn’t mean I came away thinking it was a great interactive model for this machine.

The main problem turned out to be something that most benchmark tables barely talk about at all: how long the model spends processing the prompt before it gives you anything back.

My test setup

The machine is an original Mac Studio M1 Max with 64 GB of unified memory and 400 GB/s of memory bandwidth. For these tests, I ran qwen3.8:27b through Ollama using the Q4_K_M GGUF, which takes up around 17 GB on disk.

There was actually a small problem before I even got to the benchmarking. My Mac Studio tends to sit unattended and I discovered it was still running Ollama 0.30.10, which refused to pull Qwen3.8-27B with an HTTP 412 and a message saying the model required a newer version of Ollama. The machine hadn’t auto-updated because it normally sits at the login screen and the updater only runs within a logged-in GUI session, so after upgrading to Ollama 0.32.15 I could finally get started.

For most of these tests, I also dropped OLLAMA_NUM_PARALLEL from my usual setting of two down to one, mainly to reduce KV cache use and give the dense model a fairer run. I used a 65,536 context limit throughout, which becomes quite important once we get into the longer-context testing.

Around 10 to 12 tokens per second

I started with the obvious number because this is the one we tend to use whenever we talk about local LLM performance. I ran three fixed prompts covering a short conversation, code generation and a longer prompt, taking the timings directly from Ollama’s own eval_count, eval_duration, prompt_eval_count and prompt_eval_duration values rather than trying to estimate them from wall-clock timing.

I then repeated the same test to see whether the first run had been particularly favourable or unfavourable.

The generation numbers were remarkably consistent between runs, which gave me a reasonable amount of confidence that the 10 to 12 tokens per second range is representative of what this particular configuration can do.

It isn’t going to feel like a hosted frontier model and nobody is going to call it fast, but for a dense 27.8B model running locally on first-generation Mac Studio hardware, I didn’t think those numbers were particularly bad.

One of the more useful comparisons I found was a published M4 Max 128 GB result using Ollama and the same Q4_K_M quantisation, which came in at around 16.6 tokens per second. That is about 1.5 times faster than my M1 Max result, but given the difference in memory bandwidth between those generations of Apple Silicon, I don’t see anything particularly mysterious going on there.

This is also where I could quite easily have stopped and written a fairly positive article about the M1 Max still being capable of running a modern 27B local model.

The problem is that tokens per second only tells you how quickly the answer arrives once the model has actually started producing it, and as I increased the amount of context I was giving Qwen, that became less and less relevant to how the model actually felt to use.

The wait happens before the answer starts

To look at longer context properly, I built a simple needle-in-a-haystack test. Each run inserted a randomly generated override code into a synthetic block of text at either 10%, 50% or 90% depth, and Qwen had to find and return that exact code.

I tested four increasingly large context sizes and ran each at all three depths, giving me 12 retrieval tests in total.

The retrieval side of this was excellent. Qwen found all 12 needles and I saw no obvious lost-in-the-middle effect at all, with codes placed at 10%, 50% and 90% being retrieved correctly regardless of where they appeared in the context.

The timing is where my view of the model started to change. A context of around 1,900 tokens meant waiting about 24 seconds for the first token, which I could live with, while 7,400 tokens pushed that to around 85 seconds.

At roughly 22,000 tokens, I was waiting four and a half minutes, and once I got to around 55,000 tokens, the wait before the first generated token was between 12½ and 13½ minutes.

That is a very different performance characteristic from saying that the model generates at 11 tokens per second. Once it starts generating, 11 tokens per second is fine for me, but if I’ve already spent 13 minutes waiting for that first token, then the generation speed has become almost irrelevant to whether I want to use it interactively.

It is also worth putting that 55k figure into context because Qwen3.8-27B advertises a native context window of 262,144 tokens. I was using only around 21% of that native context and was already well beyond what I would consider a comfortable interactive workflow on this machine.

This probably became the biggest takeaway from the entire session for me. We talk endlessly about generation speed when benchmarking local models, but for larger context windows I’m much more interested in how long I’m going to be sitting there before the model starts talking back.

Accuracy wasn’t where I had a problem

The other side of this is that I found very little to complain about when it came to the actual quality of the responses. After fixing several problems in my own benchmark harness, Qwen3.8-27B eventually passed every valid scored task I gave it.

I may need to qualify that because 37/37 can easily look like a much bigger claim than I’m making. This isn’t a replacement for a proper independent model evaluation and most cells were only run once, but they were useful tests for the way I might actually use a model locally.

The coding tasks were checked by executing the generated functions against tests the model hadn’t seen, the reasoning questions had known answers, and the instruction tests checked hard constraints such as exact JSON output and word counts.

The longer-context result interested me more than the headline score because it showed that the model wasn’t simply accepting a big context window and then losing track of what was inside it. Even at around 55,000 tokens, it correctly retrieved information from the beginning, middle and end of the supplied context, so the limitation I was hitting wasn’t whether Qwen could use that information, but how long I was prepared to wait while it processed it.

I also wanted to see the agentic side for myself

Some of the strongest benchmark claims around Qwen3.8 are related to coding, terminal use and agentic behaviour, so I didn’t want the testing to consist entirely of synthetic reasoning questions.

I created a small mock fleet-management API with three tools for listing machines, fetching information about a machine and checking the size of a model, then gave Qwen tasks that required it to work out which tools it needed and use the returned data correctly.

On the tasks that involved checking several machines, it also fanned the tool calls out across the available machines rather than unnecessarily making each request sequentially. This is probably the part of the testing that made me more interested in using Qwen3.8 locally, because if I’m running an agent or a batch process where I’m not sitting there staring at the screen, the latency problem matters much less.

The reasoning behaviour wasn’t what I expected

Before running any of this, I thought the reasoning-effort test might end up being the main story. There had been enough discussion about Qwen3.8 burning huge amounts of tokens and disappearing into long thinking loops that I expected to see a fairly obvious relationship between reasoning effort, elapsed time and accuracy.

I tested that using three harder coding tasks at low, medium, high and max reasoning effort, with the generated code executed against hidden unit tests each time.

I didn’t reproduce the behaviour I’d been expecting at all. Accuracy was identical at every reasoning level; medium was actually the fastest setting overall and thinking-token use didn’t increase cleanly with the effort level either, with medium using fewer estimated thinking tokens than low.

Max certainly did more thinking and took longer, but even then the total runtime was around one and a half times the low setting rather than anything resembling some of the reports of simple tasks taking tens of minutes.

There are a few possible reasons for that, including changes between the original release and Ollama 0.32.15 or simply the behaviour being very dependent on the prompt, but based on my own tests I can only report what happened on my machine.

For the sort of verifiable tasks I used here, increasing reasoning effort bought me nothing. All four settings passed everything, so I’d take the lower latency rather than assume more thinking automatically gives me a better answer.

My benchmark was responsible for every failure

The part I found most interesting actually came from the tests that initially went wrong, because at several points Qwen appeared to fail in exactly the ways I had been expecting after reading other people’s experiences.

One instruction test asked the model to describe the sun in exactly five words. I capped the output with num_predict set to 128 and got an empty response, which initially looked like a straightforward model failure. When I dug into it, Qwen had spent the entire allowance reasoning and had no budget left to produce the final answer.

Running the identical prompt with progressively larger allowances made the problem fairly obvious.

Once the model had enough room to finish, it returned exactly five words and passed the test. The model hadn’t failed the instruction at all, but the harness had turned an output-budget problem into something that looked exactly like a failed model response.

That wasn’t the only issue. I initially used xhigh for the top reasoning level because that terminology appears in discussion around the model, but Ollama expects max, so three HTTP 400 responses were temporarily being counted as three failed tasks.

My first quality comparison also had Qwen3.8 running at think=low while the older comparison models were using their default reasoning settings, which made Qwen3.8 look much better simply because the other models were consuming more of their prediction allowance on thinking and sometimes returning truncated or empty answers.

I even found a more traditional coding-harness problem where my parser took the first Markdown fence from the response rather than finding the block that actually defined the function I was trying to test.

By the time I’d finished debugging it, every apparent Qwen failure from the session had turned out to be mine.

There is a wider benchmarking point in that which I think gets overlooked. I went into these tests having already read that Qwen3.8 could overthink, burn through tokens and sometimes behave badly; then my own benchmark started giving me empty answers and zero scores that appeared to confirm what I already expected. Had I stopped there and written up the first results, I could have produced a perfectly convincing article about Qwen behaving exactly as everybody said it did.

The problem was that the evidence would have been wrong.

A benchmark harness is still software. It has its own settings, limits, parsing assumptions and bugs, and with reasoning models in particular, some of those bugs fail quietly enough that they are very easy to mistake for weaknesses in the model itself.

So would I actually use Qwen3.8-27B on the M1 Max?

I think I would, but the use case is quite different from the one I had in mind when I first started looking at the model.

For unattended agentic work, batch jobs and local processes where I can give the model something to do and come back later, I came away impressed. It fits comfortably within the 64 GB available on the Mac Studio; the agentic tool use worked cleanly, the coding tests passed and I had no problems with retrieval even as I pushed the context past 55,000 tokens.

For interactive use, particularly coding workflows where a conversation gradually accumulates a large context, I think I would find it frustrating on this generation of Apple Silicon. The 10 to 12 tokens per second generation rate isn’t really the thing that bothers me, because once the answer starts arriving, that speed is perfectly readable.

Waiting several minutes for prompt processing before anything appears is a different matter, and at around 13 minutes for a 55k-token prompt, I would probably have stopped using it interactively long before reaching the limits of what the model can technically handle.

There are some limitations to my testing that I also want to be clear about. Most measurements are single runs, although repeating the basic speed test produced very similar numbers, and I haven’t tested the MLX build, which is probably the biggest missing piece for an Apple Silicon comparison. I also haven’t touched the vision or video capabilities yet, and the long-context retrieval test used synthetic filler text rather than the sort of messy mix of source code, documentation and conversation history you might see in a real workflow.

Even with those caveats, I think I’ve answered the question I originally had. My M1 Max Mac Studio can run Qwen3.8-27B and the model itself is capable enough that I can see genuine uses for it, particularly if I put that Mac Studio back to work as an unattended local inference machine.

What has changed is the benchmark number I’m interested in. Seeing 10, 12 or 16 tokens per second tells me something about how quickly a local model writes, but once the context starts growing, it tells me very little about how the model is actually going to feel to use.

For that, I want to know how long it takes before I get the first token. Has anyone else been benchmarking this model on an M1 Max?