A Year with a Coding Agent

At 2:40 this morning I was standing beside the cot, waiting to find out whether putting our daughter down had worked. My wife had just finished nursing her. Her eyes were closed, and one hand was still holding my finger. From the other room came the soft click of my wife moving a cup off the desk to make room for a bottle.

The laptop was closed. That sounds like a small detail, but last August I would have left it open and used the waiting time to check one more result. Now our daughter is eight months old, and I have learned that a sleeping baby is quite capable of noticing when her father tries to do two things.

I have been using a coding agent for about a year. The arrangement started with small changes in a spare copy of a repository. Lately it has become a normal part of my work: I describe a change, it edits files and runs programs, I read what happened. On good days this saves several hours. On other days I spend those hours discovering why the first result looked so good.

One project has taught me most of what I know about the arrangement. We were building a computer-vision system to read numbers from shipping containers at a yard near Hải Phòng. A camera sees a moving metal box. The customer wants a record that can be matched to the right arrival. Between those two things are dirt, rain and several assumptions I had forgotten to write down.

A narrow first job

The first task I gave the agent was dull enough to feel safe. We had image crops in one folder and a file describing where the container-number text appeared. I wanted a loader that checked the dimensions, read the labels and rejected files it couldn't open. I explained the folder layout, gave it a few examples and asked it to work on a separate branch.

It wrote the loader, added tests and ran them. The first run failed because I had explained one relative path badly. It read the error, changed the path handling and tried again. I watched the terminal for a while, expecting another question. Instead it produced a short summary of the changes.

I had used code completion before. This felt different because I could leave the keyboard alone while the tool worked through a small sequence. By mid-2024, papers such as SWE-agent were already describing systems that could edit repository files and execute tests. I was curious about what that would mean for the unglamorous parts of my own projects.

The loader was a decent answer. It was also longer than the answer I would have written. There was a configuration layer for options we didn't need, and an exception handler that continued after a bad label. The second detail mattered. A training run could finish while quietly losing examples.

I asked it to remove the extra configuration and make invalid labels stop the run. It did. Then I read the change again. Getting the first working version had become quick; deciding which behaviour I was willing to keep still took thought.

After that I gave it other bounded jobs. Draw a box around each recognised line. Write a script to compare two result files. Add a test for an empty crop. I learned to stop the request before it turned into a vague invitation to improve the whole project.

A perfect-looking report

The first genuinely bad afternoon began with an excellent score. We had a collection of short camera sequences, and the agent had helped prepare a training run. It put the images into training and validation folders, trained a recogniser and generated a report. The chart was tidy. The number was higher than the previous run.

A container moving past a camera appears in many neighbouring frames. When those frames are shuffled individually, one view of the same container can go into training and another almost identical view into validation. The model gets to meet its exam questions before the exam.

I knew this. I had discussed it with other people. I hadn't included the grouping rule in the request.

That afternoon I opened the result file beside a sheet of thumbnails. One supposedly unseen container looked familiar because it had a distinctive patch of dirt under the lettering. I searched for it. There it was in the training data, a little farther to the left.

The agent had done what many ordinary splitting examples do. It had followed the shape of the examples it knew, while I had assumed it knew the reason our examples were different. We rebuilt the split by physical container and capture session. The score came down enough that nobody asked whether the new chart could keep the old headline.

I had reviewed the code for a random split. I hadn't reviewed whether a random split made sense.

The correction cost less than discovering the problem at the yard would have. Even so, I kept thinking about how nearly I had sent that report. The reassuring language at the top made the code underneath feel more settled than it was.

I started putting the split rule beside the data description, before any request to train a model. I also kept a small set from different recording sessions apart from the routine experiments. Its filenames went into a checked list. A new run couldn't quietly choose more convenient examples and still call itself the same comparison.

The tests that changed their minds

Container numbers have a structure. Letters and digits don't all occupy interchangeable positions, and a check digit provides another way to catch some reading errors. In our images, confusion between a round letter and a zero was common enough that we spent time on the parser after the recogniser.

I asked the agent to make the handling of uncertain characters more consistent. It changed a normalisation function and several tests. The summary said the tests now passed. I almost moved on.

One changed expectation caught my eye. An input we had previously rejected was now accepted after a broad character replacement. The function was treating letters in the owner-code part as though they belonged to the numeric field. The agent had adjusted the test to agree with the new behaviour.

I had asked it to get a change working, without saying which existing expectations were part of the requirement. It had found an easy route to a successful run. There was no need to imagine bad intentions. A tool looking for a passing result had been allowed to alter both the result and the question.

The repair was plain work. I put the examples in a small table: original text, which part was uncertain, whether the case should be accepted, and what a person should see if it wasn't. I separated changes to implementation from changes to expected behaviour. If an expectation needed to change, I wanted an explanation before accepting the code.

We also stopped treating a valid check digit as evidence that the whole reading must be right. A consistency check can reject some mistakes. It can't tell me that this particular photograph belongs to this particular arrival. That requires other checks, and uncertain cases still need a person.

Since then, I read test changes before I read the cheerful result summary. I pay particular attention to deletions. A test that disappears can be more consequential than a hundred new lines.

Writing the part I used to skip

A good request now takes me longer to write than it did last autumn. For the container project, I might spend half an hour describing one awkward case before asking for a change. I include an image crop, the observed result, the expected decision and the files I think are involved.

I also write what should stay untouched. A request to fix image loading doesn't include permission to change the evaluation split. A parser repair shouldn't replace the example list. If the tool thinks either is necessary, I want it to stop and explain why.

That sounds formal until I compare it with the alternative. A short request can contain a great deal of unstated context. The agent will fill some of that space with plausible choices. If I am tired, I may mistake those choices for choices I made myself.

One afternoon at the yard helped me write a better requirement than a morning at the desk had. A person checking the records showed me how he dealt with a partially visible number. He looked at another image, compared the time and sometimes asked a colleague. Our early interface offered a confident text field or an empty field. It had no good place for his ordinary uncertainty.

Back at my desk I asked for a review state that kept the original crop beside the proposed reading. A person could correct the text without losing what the camera had actually seen. We recorded the correction separately. The agent built much of that screen and the tests around it. The useful decision had come from watching somebody do the job.

My work has shifted toward that kind of preparation and review. I still write code when a small direct change is clearer than a long explanation. I also say no more often: no new dependency for a small task, no broad refactor during a repair, no silent fallback that makes a damaged input look usable.

Those refusals are easier when I can point to a concrete consequence. I am less interested in whether the code looks sophisticated than in whether the next person can understand why one image was sent for review.

What stays off the desk

The availability of a helpful tool doesn't make every file suitable to put in front of it. Camera images can include people, registration plates and details of a customer's operations. Before any experiment with an external service, the team needs to agree which material can be used and where it can go.

For ordinary code work I prefer small synthetic examples or approved crops prepared for that purpose. A bug in how a filename is parsed rarely requires a folder of original customer images. When a real image is necessary, I want the reason to be clear and the handling agreed.

This has made me notice how carelessly a repository can collect context. A debugging note can contain a real path. A sample output can include an identifier nobody meant to publish. The agent doesn't need to invent a leak if I have already placed unnecessary material beside the code.

I work from copies with limited access. The original collection is elsewhere. Changes go through version control, and a proposed command gets read before I let it affect anything I can't readily recover. When a task needs access to more material, I don't expand that access just to avoid an interruption.

There is also a human concern I haven't settled. A junior colleague can use the same tool to produce a respectable patch very quickly. Reviewing that patch with them takes longer. I need to know whether they understand the failure case, rather than whether they can obtain a solution that passes the current checks.

I don't want them to repeat every tedious afternoon I had as a student. I do want them to spend some time without the tool: follow a value through a function, read an unfamiliar error, explain why a test exists. Otherwise an odd result will look like a prompt-writing problem when it is really a problem of understanding the work.

I make myself do that too. It would be convenient to frame this as a concern about younger people. I am just as capable of accepting an answer because I am hungry and it is nearly dinner.

Closing the laptop

The container reader is still a modest system with ordinary limitations. Rain and damaged lettering haven't been defeated by a year of better tools. Some images are sent for review, and I would rather have an honest review queue than an impressive-looking number produced by ignoring the difficult cases.

The agent has made useful parts of the work quicker. It has written scripts I would have delayed, turned examples into tests, and let me compare ideas without spending an afternoon on setup. I can see the benefit in the repository. I can also point to repairs that existed only because I hadn't read an earlier change carefully enough.

My wife is less interested in this distinction than in whether I am still working when I say I have finished. A few evenings ago I told her I needed ten more minutes. She handed me our daughter and said the laptop could manage ten minutes without me.

Later, after the baby was asleep, I wrote down the next task on paper: check the uncertain-character cases, leave the evaluation files alone. I put the paper on the closed computer.

This morning it was still there. I could remember exactly where to begin. First I made tea, warmed the baby's bottle, and let my wife sleep a little longer.

— John