A Small Model on My Laptop

The laptop fan was loud enough for my wife to notice from the bedroom. Our daughter had finally fallen asleep. I had chosen that particular moment to find out whether a small language model could run comfortably on the machine I already owned.

The answer to comfortably depended on who was being asked. The model was producing words. The laptop was warm. My wife stood in the doorway and pointed at the sleeping baby without saying anything. I stopped the program and moved the machine away from the bedroom wall.

I had imagined local inference as a quiet little arrangement: open a file, ask a question, get an answer without sending the material elsewhere. That remains an attractive possibility. My first evening also involved a large download, a confusing set of model variants and enough heat to remind me that computation takes place in physical objects.

Our daughter was three months old. My available experiment time came in short, uncertain pieces. This turned out to be useful. I couldn't spend a whole night admiring a demonstration. I needed to decide whether the setup could do one small job well enough to be worth opening again.

Choosing a job it might finish

My original idea was a general assistant for technical work. That description included almost every task I did and provided no useful way to judge success. I reduced it to extracting a few fields from short, invented maintenance notes.

I wrote examples that resembled ordinary notes without using anyone else's confidential material. A machine identifier, an observation and a proposed action appeared in different orders. Some notes omitted the proposed action. A few used Vietnamese abbreviations that I understood but hadn't explained.

The desired output was deliberately small. I wanted a structured record with an explicit missing value when information wasn't present. If the model added a plausible action that the note hadn't mentioned, that counted as an error even when the action sounded sensible.

This made the test less entertaining. It also made mistakes visible. A model could write a convincing paragraph about maintenance while quietly confusing a suggestion with work that had already been done. A compact record gave me fewer places to hide that confusion.

I used a small set to explore the setup and kept another set aside for checking changes. The sets were not large enough for a sweeping claim. They were large enough to stop me demonstrating the same friendly example until I believed the problem was solved.

My wife read one invented note while I was holding our daughter. She asked whether the phrase at the end meant the repair was finished or merely requested. I had written the example myself and still had to think about it. I marked it ambiguous rather than forcing a single answer and blaming the model later.

What small meant on my machine

Small is a relative word in this field. A model described with billions of parameters can still be called small because larger ones exist. That doesn't mean every laptop will run it quickly, or that every compressed version will behave in the same way.

The Mistral 7B paper from 2023 was one of the references I returned to while reading about this class of models. The number in a model name tells me something about scale. It doesn't tell me the complete memory requirement of the particular runtime and configuration I am about to use.

I wrote down the exact model file and the runtime version. Earlier, I had compared two outputs without noticing that I had changed both. The resulting opinion about which setting worked better was worth very little.

Quantized variants reduced the storage and memory burden in my experiment. They came with choices I needed to test rather than treating compression as free. The quality difference on my narrow task could be different from the difference on a broad benchmark, so I kept returning to the maintenance notes.

I measured the time until the first output and the total time for a completed record. Watching words appear quickly can create an impression of speed that doesn't match how long I wait for something usable. Long explanations were especially misleading because they kept the screen busy while delaying the result I needed.

The machine also had other work to do. A test with nothing else open wasn't representative of an ordinary afternoon. I repeated a few examples with my usual editor and documents running. The fan made its opinion clear before the numbers did.

I didn't buy hardware for the experiment. The useful question was whether the existing machine could support this limited task. Changing the budget would have changed the problem, and I had enough unfinished problems already.

An empty field that wouldn't stay empty

The most persistent failure concerned missing information. I had asked the model to leave a field empty when the note didn't support an answer. It often did. Then a familiar-looking note would tempt it into completing the story.

One example described a temperature observation and a request for inspection. The output supplied a repair action that sounded reasonable but hadn't appeared in the text. If I had read only the formatted result, I might have mistaken that addition for extracted information.

I tried shorter instructions, clearer examples and a stricter output format. Each change helped some cases. None justified assuming that a well-formed record was a faithful record. A parser could verify brackets and keys while missing the more important error.

An answer can fit the format and still contain something I never said.

I put that sentence beside the evaluation sheet. It kept me from counting successful parsing as successful understanding. The two measurements belonged in separate columns.

For a few fields, ordinary rules worked better. An identifier with a known prefix could be matched directly. A date in a consistent format didn't require a generated explanation. I let the model handle the less regular phrasing and used simpler checks where they were sufficient.

This reduced the apparent ambition of the tool. It also reduced the number of unsupported additions. I was happier with a small component inside a boring process than with an impressive assistant whose output required me to reread every source sentence anyway.

I added a route for uncertain cases. When the note was ambiguous, the result could be sent for manual review rather than forced into a complete record. This wasn't a magical confidence estimate. It was a practical way to preserve uncertainty that already existed in the source.

Local doesn't finish the privacy work

Running a model locally can help keep particular inputs off an external inference service. That is a useful property. It doesn't settle every question about the software, the files on the machine or the permissions attached to the material.

I checked where the runtime stored its files and what happened to prompts after a session. I looked for logs rather than assuming there were none. I also kept the test collection separate from work documents, because an accidental folder selection would be a very ordinary way to create an unnecessary problem.

The model file came from a source I could identify, and I read the relevant license before treating it as a component I might use beyond a personal experiment. A convenient download button wasn't enough information about permitted use.

I was careful with installation instructions too. A snippet copied from a discussion can be useful, but I wanted to understand what it would execute before running it. Local software still has access to local resources. The word local doesn't make an arbitrary program trustworthy.

For this experiment, invented maintenance notes were enough. I didn't need private customer material to investigate latency, missing fields or output structure. When someone asked whether the same arrangement could be used with real documents, my answer depended on the documents and the approved environment, not just on whether the network cable was connected.

There was also a simple household consideration. The laptop sat on a table that other people used. I closed it when I stopped working and kept the charging cable out of the route between the cot and the door. A technically private experiment could still be an irritating object in a shared room.

My wife appreciated the cable change more than the explanation of quantization. She was right about the cable being the more immediate improvement.

The second week

During the second week I stopped changing the model every evening. I kept one configuration long enough to understand its recurring mistakes. This was less exciting than comparing new downloads, but the results became much easier to interpret.

I added examples from the categories that had caused trouble, while keeping a separate check set. Negation mattered. A note saying that a part had not been replaced could produce a different failure from a note that merely omitted replacement information. Abbreviations mattered too, especially when the same letters could stand for more than one thing.

I asked a colleague to read a few examples without seeing my expected answers. We disagreed on two. I revised the task definition for one and marked the other uncertain. The model wasn't the only source of variability in the experiment.

I kept the generated output short. That improved turnaround time on my machine and made checking easier. A field extraction task didn't benefit from an introductory paragraph announcing that the requested information had been carefully analysed.

By Friday I had a small working pipeline: read an approved example, apply a few straightforward checks, request a structured output, validate it and show the original beside the result. It still required review. It also saved a little effort on the clearer cases.

The savings were modest enough that I could have exaggerated them without anyone noticing. I wrote down actual timings instead. Loading the model for one tiny task sometimes cost more time than doing the task manually. Keeping it available for a short batch made more sense, provided the laptop wasn't needed for something else.

This was the practical conclusion I had wanted at the beginning. I had just expected it to arrive wearing a more impressive name.

Closing the lid

I still use larger hosted tools for some work when the data and the terms permit it. I also keep this local experiment because it gives me a way to explore a limited task with direct control over the setup. The useful choice changes with the task rather than following a single rule about where computation ought to happen.

What I value most is being able to inspect a small process end to end. I know which notes went in, which configuration produced the output and which cases failed. That knowledge is more useful to me than a broad claim that the model is surprisingly capable for its size.

On Saturday morning I ran one final batch while our daughter slept in the next room. The output finished. I saved the results, closed the program and waited for the fan to settle. There was a bottle to wash and a cloth lying over the back of my chair.

Later, while I was folding that cloth, I remembered another example I wanted to add to the test set. I wrote it on paper rather than opening the laptop again. The experiment could wait until the next stretch of quiet, and the note was short enough that I could still understand it when I returned.

— John