← Back to indexNiranjan
Essay · IXJul 20234 min read

Behind on AI

Why we shipped a GPT-4 feature on a weekend a few months back, and the prompt I'm still not proud of.

We shipped our first AI feature a few months back. Every product I followed was launching something, and most of our competitors had said the word "AI" in a press release. We had not. I felt left behind, so I built something on a weekend so that we would have a thing to point at.

The thing I built was a button. The user records a few interactions in our recorder, clicks the button, and the API ships those interactions to GPT-4 with a prompt asking for up to 10 more test cases extrapolated from what was just recorded (positive and negative cases, streamed back to the dashboard). The mechanic was good. The recorder already captured a single happy path, and GPT-4 turning that into 10 was a small leap on a feature people wanted anyway.

The prompt I shipped is in a file called prompts.ts. A single constant with 14 numbered objectives. I'm looking at it as I type this.

1. Generate upto 10 possible positive and negative useful regression test cases...
2. Generate separate tests cases for each component...
...
7. Each test case should be
8. Focus on testing single components or functions...

Item 7 is the whole sentence. "Each test case should be." I never finished it. The prompt also says "tests cases" instead of "test cases" in 2 places. The output format was a custom CSV-ish thing with a TEST_CASES section and a STEPS section, IDs joined by hyphens, because I had tried JSON first and GPT-4 at the time could not be trusted to close a brace consistently. CSVs are more forgiving to parse.

The whole feature was about 157 lines. A raw OpenAIApi client, streaming via openai-ext, all of it under services/gpt/. No provider interface, no model abstraction. Rate limiting was whatever GPT-4 itself enforced.

It worked, the way LLM features in 2023 worked. It generated useful test cases most of the time and now and then produced one that referenced an element the user had not recorded, or made up an action verb that wasn't in our supported list. We shipped it anyway. Customers liked the demo.

I went back 3 months later and paid the debt. The single prompts.ts got split into a prompts/ directory because we were adding a second AI feature (generating validations from a plain-English description) and one constant in one file was no longer the shape of the problem. The validations feature shipped at the end of May.

The cleanup didn't touch the prompt itself. Item 7 still trails off and the typos are still in there. By the time I went back to clean up the code around it, the prompt was the part that worked, and I wasn't going to be the one to break it for the sake of a typo.

If I had to name what I learnt, it's that LLM features are weirdly forgiving of code quality, in a way nothing else in our codebase is. A typo in a prompt was a non-event, and item 7 trailing off did not stop the model from understanding what was wanted. The instincts I had built up over a decade of writing software, where every typo and every malformed line was a potential bug, did not transfer to the inside of a prompt. Half my engineering instincts were now operating on a thing that would shrug them off.

I shipped the feature out of fear, which is a bad reason to ship anything. I would like to say I learnt to ignore the FOMO after that. I did not.

❦
Thanks for reading. Questions, disagreements, or corrections,
.