← Back to indexNiranjan
Essay · XJan 20244 min read

Letting GPT drive

We let GPT-4 pick selectors and run actions on a real page during a test. The demo went great. Then we met a virtualized list.

A few months after the test-case generation feature shipped, we did the obvious next thing and let GPT drive the browser.

In November I shipped what we called an AI Action. At run time, the supervisor hands GPT-4 the page HTML, tells it what the user wants to do in plain English and lets it return a structured tool call. The tool was performActionOnPage, with a selector, an actionType of input, click, dblclick, keypress or scroll, an optional value, an optional keyValue for things like Enter or Shift, a scroll direction and amount for scrolls and a toBeCalledAgain flag if the model wanted another turn. The supervisor took the result, ran it on the real page through the existing executor and either kept going or asked the model again.

The prompt was 4 sentences. It's committed, and I'm looking at it as I type this.

You are a test automation bot that will run the actions provided
by the user on a HTML Web page. Output only in json. This is the
HTML of the webpage. Generat selectors based on this HTML. Ensure
the selectors are resilient.

"Generat" is missing an "e". I noticed it months later and left it. Same reason as the prompt in prompts.ts from earlier in the year. By the time I saw the typo, the prompt was the part that worked.

A few weeks after AI Action, I shipped a sibling, validateSingleElementOnPage. The user describes a check in English, GPT returns a javascript validate(variables, attributes, innerText) function as a string and the supervisor evaluates it against the real element. The prompt had 3 or 4 worked examples. The output was usually a 1 or 2 line function.

The first demos went the way the test-case generator's demos had gone. A user types "click the Submit button at the bottom", GPT returns a sensible CSS selector, the supervisor clicks it and the test moves on. The validation flow worked too. "Check that the price is greater than thirteen" came back as Number(attributes.value) > 13 and ran clean. Customers found it impressive, and the mechanic felt like the future for about 3 weeks.

Then it had to run for real, on the kinds of pages our customers test. Take a virtualized list. The model picks a selector for a row, the user scrolls, and the same selector now points at different content. Or a custom dropdown built out of divs with no semantic role, where actions only appear on hover and the selector the model returned belonged to an element that had already collapsed by the time the supervisor tried to use it. Every interaction changed the page, and the model was being asked to re-pick selectors against a target that had already moved by the time its previous answer arrived.

The other failure mode was self-inflicted. The toBeCalledAgain flag meant the model could ask for another turn, and on long flows it would, and every turn was the full HTML again. Token bills climbed faster than I had budgeted for. The validate functions GPT wrote sometimes did not compile, and when they did, they now and then referenced an attribute that wasn't on the element or hardcoded a value the user had written as a variable.

By December the tweaks had started. We dropped the temperature and added more examples to the prompts. None of it fixed the underlying thing, which is that putting the model in the execution path made every test as deterministic as the model was, and the model was not.

A few months earlier, the test-case generator had taught me that LLM features are forgiving of code quality. With the model now inside the execution loop I was getting the second half of that lesson, which was less tidy. LLM features are unforgiving of the things code is good at. A regular click in our recorder is as flaky as the selector and the page. Once GPT was picking the selector, the same click was also as flaky as the model and the prompt and the temperature and the example mix and how much of the HTML fit in the context window that day. I had spent a decade writing software that did the same thing twice when given the same input, and I missed it more than I expected to.

GPT-4V is out, and I have spent more than a few evenings wondering whether a vision model would understand the page better than the structural representation we're handing it. I'm also starting to doubt whether selectors are the right primitive at all. Even with the HTML we send heavily sanitized (attributes stripped, decorative tags removed), data-heavy pages still fill most of the context window before the prompt and the tool definitions have room to land. The accessibility tree is the other shape I keep circling. It's flatter than the DOM and describes each element by what it's for, and a framework's layout scaffolding mostly disappears.

We kept the AI Action and the AI validation in the product. They're better than nothing on flows the regular recorder can't handle. The version of November, where this was going to be the new shape of the product, has settled by now into something more measured. There's plenty more AI work to come. I have just stopped expecting any single piece of it to replace the deterministic parts of the system.

❦
Thanks for reading. Questions, disagreements, or corrections,
.