Stress-Testing Local AI: Why DeepSeek Failed a 5-Question Logic Test

Running Large Language Models (LLMs) locally on your own hardware has never been easier. Tools like LM Studio allow you to download cutting-edge models, like DeepSeek, and run them entirely offline. But how smart is a compressed, CPU-bound model, really?

To find out, I decided to run a deceptively simple benchmark. No massive datasets or Python evaluation frameworks—just five logic questions and a set of strict formatting rules designed to push the model's spatial reasoning, working memory, and token processing to the breaking point.

Here is what happened when I tested a local DeepSeek model, the hurdles we hit, and the fascinating reasons behind its failure.

The Setup (and the Crash)

My goal was to run a local DeepSeek model using LM Studio strictly on my CPU. However, before the test even began, I was greeted with a fatal engine error:

Engine protocol runtime llama-server... exited before becoming healthy. exitCode=1

If you are getting started with local LLMs, this is a rite of passage. This crash almost always points to a memory allocation failure. When you load a model, the engine requests a chunk of system RAM for the "context window" (the model's working memory). If the requested context length is too massive (e.g., 32,000+ tokens) or if the software tries to offload layers to a GPU you aren't using, the engine panics.

After dropping the context length down to 8192 and ensuring GPU Offload was completely disabled, the DeepSeek model successfully loaded into my system RAM, ready for the test.

The Benchmark

To test the model, I fed it a prompt containing three strict rules and five tricky questions.

The Rules:

1. Answer each question on a new line, numbered 1 to 5.
2. No explanations or conversational filler.
3. If the answer contains a number, it must be spelled out as a word (e.g., "five" not "5").

The Questions:

1. A small marble is put into a normal cup and the cup is placed upside down on a table. Someone then picks up the cup and puts it inside the microwave. Where is the marble now?
2. What is 1 + 4 + 2 + 1 * 9?
3. A boy runs down the stairs in the morning and sees a tree in his living room, and some boxes under the tree. What day is it?
4. I take three steps forward, turn left, take two steps forward, take an additional step forward, turn left, and take three steps forward. How many steps away am I from where I started? Assume all turns are 90 degrees and all steps are of equal distance.
5. How many times does the letter 'e' appear in the word 'nevertheless'?

The Results: A Spectacular Stumble

While cloud-based behemoths can breeze through this, my local DeepSeek model struggled significantly.

• It failed the formatting constraint: The model output digits (like "5") instead of spelling them out as words, completely ignoring Rule C.
• It failed spatial reasoning: For the walking question (Question 4), it confidently answered that it was five steps away. (The correct answer is three).
• It failed to count: For Question 5, it claimed there were only three 'e's in the word "nevertheless." (There are four).

The Explanation: Why LLMs Fail at Basic Logic

It’s easy to look at these results and assume the model is just "dumb." But the reality of why it failed offers a fascinating look into how neural networks process information.

1. Attention Drift (Forgetting the Rules)

Why did the model forget to spell out the numbers? This is a phenomenon known as "attention drift." Smaller or heavily quantized local models have limited working memory. When the model had to work incredibly hard to solve the math and logic of the questions, its "attention" drifted away from the global formatting constraints at the top of the prompt. It literally lost the bandwidth to remember Rule C.

2. The Spatial Grid Problem

For Question 4, you were asked to walk forward, turn left, walk forward, turn left, and walk forward. If you map this on a 2D grid, you form three sides of a square, leaving you exactly three steps to the left of your starting point.

The model answered "five." Why? Because LLMs do not have a mental chalkboard; they cannot simulate physical constraints or preserve spatial coherence. They just predict the next likely word. When the model saw the text "three steps" and "two steps," it bypassed spatial reasoning entirely and just did basic addition ($3 + 2 = 5$).

3. Tokenization Blindness

How does an advanced AI fail to count the letters in a single word? It comes down to how models "see" text. AI models do not read individual letters; they read chunks of text called tokens.

To the model, the word "nevertheless" might be chopped into two distinct tokens—for example, never and theless. Because the AI cannot "see" inside the token to view the raw characters, it is effectively blind to the spelling. It has to guess the letter count based on statistical probabilities, leading it to confidently, and incorrectly, guess "three." DeepSeek's advanced reasoning models are famously "token-hungry," often relying on generating massive, multi-step chains of internal thought to arrive at correct answers for complex problems. When constrained to provide just a final answer, those reasoning capabilities are severely handicapped.

The Takeaway

Running models locally is an incredible exercise in data privacy and offline computing. However, this test is a stark reminder that when you aggressively compress a model (quantization) to fit onto a standard PC processor, the first things you lose are strict constraint following and spatial logic.

The next time your local AI hallucinates an answer or fails a basic math test, remember: it isn't thinking like a human. It's juggling tokens, battling attention drift, and doing its best to predict what word comes next.