Looking for the Note I Knew I Had

I remembered writing the answer. That was the irritating part. I could picture a paragraph about an image preprocessing problem, somewhere below a heading I could no longer remember. I searched for the library name and found six documents, including a shopping list in which I had pasted the library name by mistake.

My wife was sitting across the table checking a list of things we would need before the baby arrived in December. She asked whether I had found the measurement for the space beside our bed. I had not. I had been looking for the technical note for twenty minutes, and the room measurement was apparently another item I had considered sufficiently memorable to store badly.

There were two searches in progress. Hers concerned where a cot might fit. Mine concerned why a small image-processing experiment had produced different results after a change in input size. Both answers existed somewhere in files I had saved myself.

I decided to build a modest search tool for my notes. The word modest needs emphasis because, for the first evening, I was already imagining a system that could answer almost anything about my work. By Sunday I wanted it to find the correct document without inventing the missing half.

A folder with several versions of me

The folder held meeting notes, experiment logs and short explanations written for my future self. Some were in English, some in Vietnamese. A few started in one language and switched when I got tired. File names ranged from useful dates to variations on the word final.

I began by reading the collection rather than indexing it. This was slow and slightly embarrassing. Several notes assumed knowledge I no longer had. One said that the second approach worked better, without describing either approach. Another contained a screenshot of a result and no explanation of which dataset had produced it.

A search system could help retrieve these documents. It couldn't restore information I had never recorded. I put those cases in a separate pile instead of hoping a language model would supply the missing context.

There were duplicated notes too. I had copied a useful paragraph into a newer file and amended only part of it. The old and new versions disagreed about a parameter. Neither explicitly said which experiment it described. A system that found both might appear impressively thorough while leaving me with the original uncertainty.

I added a small header to the technical notes I actually used: when the experiment happened, what it used and whether the conclusion still applied. I resisted cleaning every document. That would have turned a search experiment into a month of filing, and I knew which activity I would abandon first.

For the room measurement, the solution was simpler. My wife found it in a message I had sent her. I copied it onto the paper beside her list and immediately understood why she preferred that list to my collection of clever tools.

Starting with words

The first version used ordinary word matching. I made a plain text index and searched for distinctive terms. It wasn't impressive to demonstrate. It was fast, and when it found a result I usually understood why.

That transparency helped me spot mistakes in my own preparation. A document I expected to find had been skipped because I was reading the wrong file extension. Another contained an identifier broken across two lines. Neither problem needed a more capable model.

Exact terms mattered more than I had expected. If I remembered a function name, a version string or an error message, literal matching often took me straight to the right place. A system that treated similar meanings as interchangeable could make those searches worse by returning related explanations without the exact detail I needed.

But my memory didn't always retain the useful word. I might remember an experiment as the one where small images looked strangely smooth, while the note described resizing and interpolation. In Vietnamese I might phrase the question differently again. Here the vocabulary gap was real.

I kept a list of twenty questions drawn from things I had genuinely tried to find. Some named an exact error. Others described a symptom. A few referred to an old experiment by a detail I remembered incorrectly. I wrote down the expected document when I could identify it.

The list stopped me testing only the questions that made a new method look good. It also exposed a case with no answer in the collection. That question belonged in the test. Otherwise I would only measure whether the tool could produce something, never whether it could admit that the necessary note was missing.

Similar meaning, different experiment

I added vector representations of the text so that a question could retrieve passages with related meaning. This helped with several vague recollections. A query about washed-out edges found a note about interpolation that my literal search had missed.

It also found the wrong experiment with striking confidence. Two documents discussed nearly identical symptoms in different data. One concerned grayscale inspection images; the other concerned photographs with uneven lighting. The relevant next step depended on which situation I meant.

I had split the notes into short passages because I wanted focused results. In one document the conditions appeared under a heading above the passage. My splitting had separated the claim from the conditions that made it true. The resulting text was easier to match and easier to misuse.

I started carrying the document title and nearby heading with each passage. I kept links back to the original file. When the tool returned a fragment, I wanted to open its surroundings with one click rather than treat the fragment as a complete answer.

The 2020 paper on retrieval-augmented generation had already described combining retrieval with language generation. Reading it again was useful, but my own little collection made the practical problem very tangible. Retrieving something relevant was a separate achievement from answering the question correctly.

I tried combining literal results with the vector results. The mixture was less elegant than choosing a single method, but it protected exact identifiers while helping with fuzzy recollections. I judged it using the same question list, looking at which useful documents appeared near the top.

My scores were small counts, not a claim about search in general. There were only twenty questions, chosen from my habits and my notes. When one change helped two questions and hurt three others, I could inspect all five. That was a useful scale for learning.

Letting the tool write a paragraph

Adding a generated summary made the tool feel immediately more helpful. Instead of scanning three results, I could read a short explanation. It was also the stage at which I became much more likely to trust it too quickly.

One answer combined a recommendation from an early experiment with a warning from a later one. The sentences fit together smoothly. In the actual notes, the later warning was the reason we had stopped following the earlier recommendation. Chronology had become a decorative detail.

I changed the display so that dates and source links sat close to the answer. I also asked for a short quotation from the retrieved passage, checked against the stored text. A source link could still point to an irrelevant document, so a row of links wasn't enough by itself.

Show me where I wrote it before telling me what I meant.

I wrote that above the test list. It was a reminder to myself as much as a requirement for the tool. When an answer sounded plausible, I had a tendency to stop investigating. Making the source visible gave me a small reason to continue.

The best answers became shorter. For a precise error message, a matching passage and its file name were often enough. For a broad question, I preferred several candidate notes with one sentence explaining the difference between them. For the missing answer, I wanted the tool to say that it hadn't found supporting material.

I still saw confident guesses. Changing the wording of a request reduced some of them without solving the general issue. I kept a separate set of examples where the summary had to be checked manually. A friendly interface didn't turn an uncertain collection into a reliable reference book.

One evening my wife asked what the tool said about the cot measurement. I told her I hadn't included our household messages. She looked at the tape measure beside me and suggested that I measure again. We had an answer in under a minute.

The small evaluation sheet

I kept the evaluation sheet deliberately plain. Each question had space for the first useful result, whether the summary was supported, and a note about what went wrong. I recorded the configuration so that I could compare changes without relying on memory.

A technically related result wasn't always useful. If I wanted the command from a past experiment, an introductory explanation of the same library didn't count as success. If I asked why a choice had changed, finding the final choice without its reason was incomplete.

I also checked how the tool behaved when I used Vietnamese phrasing. Some questions worked well. Others found English notes whose vocabulary resembled my query only loosely. I tried several real phrasings rather than translating one carefully polished sentence and calling the system bilingual.

The most helpful improvement came from repairing two notes. I added the missing experiment conditions and marked the old recommendation as superseded. Suddenly both ordinary search and the generated summary behaved better. I had been looking for a technical adjustment to compensate for an editorial omission.

That result changed what I considered maintenance. Updating the index mattered, but so did giving a conclusion enough context to survive being read six months later. I didn't need a formal template for every scrap of thought. I did need to record why a result mattered when I expected to rely on it again.

For sensitive work material, I kept the experiment separate. I used notes I could legitimately process in the setup I had chosen, with synthetic examples where necessary. A convenient search interface wasn't a reason to send someone else's material to a service without permission. The test didn't need those documents to teach me the basic problems.

By the end of the week I had a useful tool for a limited collection. It found several things faster than my old method, failed in ways I could describe, and required occasional repair when a document moved. That was enough to justify keeping it.

A place for unfinished thoughts

I still write rough notes. If I required every note to be complete before saving it, I would save fewer of the details that matter. The difference is that I now try to distinguish a passing idea from a checked conclusion.

Sometimes I add a sentence saying that I haven't tested something. Sometimes I put the actual command beside the result instead of assuming I'll remember it. These are small additions, but they make a retrieved passage much less likely to mislead me later.

I have also stopped expecting one search box to resolve every kind of uncertainty. Sometimes I need an exact file. Sometimes I need to compare several experiments. Sometimes I am asking a question that I haven't done the work to answer. Those requests can look similar when typed into the same rectangular box.

Yesterday I found the original preprocessing note. It was shorter than I remembered and had a spelling mistake in the heading. The important detail was a change to the input data, written in a paragraph I had almost omitted during an earlier cleanup.

I left the spelling mistake until after I had copied the useful detail into the new experiment log. Then I closed the laptop and moved the little table beside our bed. My wife held the tape measure at the other end. We wrote the new measurement on paper and put it with the list.

— John