AI's affordance problem

Revised from a previous version published in June 2025.

In the process of design, we consider how a product lets users know what it can do. This is affordance. A hammer hits things, a button looks like it should be pressed, a door handle looks like it should be pulled. Some products have too many affordances (the Swiss Army knife) and some have almost none.

Tech companies love the last kind. A product that can do everything and “just works” is easy to market. In cars, we took the tactile, high-affordance world of physical switches and folded it into reconfigurable touchscreens. Dedicated controls are easier to understand, operate, error-correct, and build mental models around. Touchscreens trade that for range and upgradability. The industry is now walking it back for the controls people use most.

The INEOS Grenadier interior has an array of physical controls in addition to a touchscreen.

Chat interfaces are the touchscreen argument again, at the scale of the whole product. Having worked with CLIs often, I’ve made a discovery: chat is a natural-language command line without the affordances of a good CLI. It carries the discoverability cost of a CLI while lacking determinism, composability, visible state, and reversibility. That makes it a good control for rare, novel, low-stakes tasks and a bad operating system for frequent, high-stakes, or spatial ones. And it stays true even if the model is perfect: the cost is front-loaded in specification and back-loaded in verification, not accuracy.

A decades-old argument

Ben Shneiderman and Pattie Maes debated this at CHI in 1997. Maes argued that direct manipulation would have to give way to delegation, to agents that act on your behalf. Shneiderman held the other side: “Direct manipulation affords the user control and predictability in their interfaces.”¹ Chat-as-OS is the Maes position restated with better models. Nothing about the structure of the disagreement has changed.

Shneiderman’s 1983 paper describes what makes direct manipulation work: “visibility of the object of interest; rapid, reversible, incremental actions”², and the replacement of command syntax with manipulation of the objects themselves. Apply these to a conversational interface. The object of interest is somewhere in a scrolling transcript. Actions are slow, not incremental, and reversible only if the model or harness happens to make them so. And the command syntax hasn’t been eliminated; it’s been replaced with prose, which is a harder syntax to get right.

More simply: imagine you’re delegating a task to someone else. The way you communicate to the other person depends on a few things: your ability to articulate what you need, their ability to actually do the thing you’re asking them to do, and trust in the absence of direct instructions that the correct judgment will be used. I don’t find myself actively thinking about this in the human world, mostly because many tasks are easy to delegate and the stakes are generally low.

When the stakes are higher (whatever that may look like), the instructions and the scaffolding are much more important. They both also need to be understood as the contract between the delegator and the delegate. In a conversational interface, this changes the shape of what works best to delegate and when.

Affordance and voice interfaces

There’s much to learn from how Alexa and HomePod came to market. Both shared the same physical layout: one or two controls and an LED for status. When you first use it, it’s not clear how to interact with it, and you don’t know what it’s good at.

Through trial and error (or good marketing), you learn it’s great at timers and accidentally playing music and bad at anything with more than one clause. Voice UI research had this measured before LLMs arrived. Kirschthaler, Porcheron, and Fischer’s 2020 paper, literally titled “What can I say?”, showed that discoverability interventions improve the experience of voice interfaces. It’s an observed challenge with no fix yet.

Three gulfs, not one

I formerly attributed this solely to affordance, but an affordance failure invites affordance fixes. I’ve seen new affordances working with chat interfaces, but it hasn’t quite solved some of the core interaction challenges when conversation becomes the interface.

Norman’s original two gulfs help diagnose the issue. The gulf of execution is the distance between what I want and knowing what to do to get it. The gulf of evaluation is the distance between what the system did and my ability to tell whether it worked. In 2024, Subramonyam and colleagues at Stanford argued that LLM interfaces expose a third gulf that sits before both: envisioning. Users don’t know what the task should be, how to instruct the model, or what to expect back. As they put it, “calibrating LLM interactions is challenging for interface designers and end-users alike.”³

Apply this to the examples. Not knowing what HomePod is good at is envisioning. Not knowing how to phrase a request so it registers is execution. Alexa confidently describing sunny skies with no weather data behind it is evaluation. A chat box widens all three at once. A dedicated control narrows each with a different mechanism: the label narrows envisioning, the constrained input narrows execution, the immediate visible state change narrows evaluation.

Subramonyam’s team looked at ChatGPT’s own onboarding, the example prompts on the landing page, and found they were too coarse to plan from. Their conclusion on the interface as a whole: the features “don’t fully equip users to carry out the planning and evaluation tasks.”⁴ It’s a blank canvas problem.

Telling isn’t affording

The obvious rebuttal is onboarding. Show people what it can do. Microsoft’s 2019 human-AI guidelines put this first and second on the list: “Make clear what the system can do” and “Make clear how well the system can do what it can do.”⁵ It’s one way to seed prompts and start experimentation. But a 2025 study of capability communication in conversational agents found something uncomfortable: telling users what the agent could do changed how they behaved. “[T]he communication led to a change in user behavior with significantly longer utterances”⁶. Their mental models of the agent did not improve to match. People talked more; they didn’t understand more.

That’s the distinction the chat-first argument keeps missing. Announcing a capability isn’t the same as affording it. A 2025 survey of the field says it plainly: “CUIs often lack clear signifiers for AI Affordances”⁷. Signifiers are the part of a control that tells you what it does before you interact with it. A text box has one signifier, and it says “type.”

Better models, better interfaces

“The models will get better.” They will. It doesn’t help.

Masson and colleagues built DirectGPT, a direct-manipulation layer over the same model that powers ChatGPT, and compared it head to head. Their design goal was to ease verification, tighten control, and make trial-and-error cheap, and it beat chat on those. More important is what they cite from earlier Wizard-of-Oz work, where the “model” was a human with perfect accuracy: “people still prefer a combination of direct manipulation and natural language”⁸, especially when the things being edited are hard to describe.

That’s a usability tax in translation. Every turn in a chat requires me to linearize a spatial or visual intent into prose, and then de-linearize whatever comes back. A better model shrinks the evaluation gulf. It does nothing for envisioning or execution; that’s still my labor before the message is sent.

Holding state

Chat interfaces are becoming an operating system. In tools like Slack or Buzz, Agents are operators that you chat with like teammates, working with tools and reporting back progress as they do their tasks. It’s a way to orchestrate multiple tools with a multiplayer approach, and chat is the canvas or substrate.

A traditional operating system holds state. Your state. Files persist somewhere. Windows sit where you left them. Permissions are legible. Undo exists. You can have two things open at once. Primitives compose. A chat session is modal, linear, ephemeral, and single-threaded. There is no where.

The strongest case for chat is the terminal: zero affordance, maximum power, and power users can do more with it than buttons. Chat is just a CLI for everyone, the argument goes. But the CLI’s power doesn’t come from inherent low affordance. It comes from a deterministic contract: the same input produces the same output, commands compose through pipes, --help and tab completion tell you what’s accepted, exit codes tell you what happened, and rerunning a command is safe. Chat has none of that.

The vendors have quietly conceded this. Claude and ChatGPT now render control surfaces — forms, buttons, comparison tables, maps, and interactive widgets — inside the conversation. The control surface is moving into the single-threaded, ephemeral chat, reconstructed every time.

Where the damage happens

In the first version of this article I asked what a chat interface should do if you told it to “export all of my data and delete my database so I can move to a competitor.” In July 2025, an agent inside Replit deleted a production database during a code freeze and then produced a misleading account of what it had done. That hypothetical is now the reality for many. The failure wasn’t the model’s intelligence. It was a conversational surface with no preview, no commit step, and no reversibility, acting on a request that had never been rendered as something the user could inspect.

Claude can make mistakes. Please double-check responses.

The user is now the arbiter of quality with no tooling to be one. Guideline two, “make clear how well the system can do what it can do,” is unmet, and a disclaimer isn’t a signifier.

An allocation rule

None of this is an argument against AI in products or an overall dismissal of conversational interfaces. We need a clearer connection between the task to be done and when a conversational interface is the correct choice.

Score a task on four things: how often it’s done, what it costs to get wrong, whether it can be undone, and how easy the intent is to put into words. High-frequency, high-stakes, irreversible, or spatial tasks get dedicated controls. Rare, novel, low-stakes, easily verbalized tasks get language. Neon’s SQL editor is a great example: natural language to draft a query, a real editor to read and run it. Small, scoped, and useful. The context is carried into the interface and it’s easy to infer what it can do.

Implications for design

Against the gulfs: In a conversational interface, this changes the shape of what works best to delegate and when.

1. Constrain the canvas. Scope what can be asked and what can come back. This is the envisioning gulf, and a blank canvas widens it.
2. Embed language in patterns that already work: forms, command palettes, inline ghost text, select-then-instruct as DirectGPT does. This is the execution gulf.
3. Intent, plan, preview, commit. Side effects go behind a real control, not a reply. Show the diff or the dry run first. This is the evaluation gulf.
4. Reversibility and audit by default. Let people explore, make mistakes, and heal. Back-of-house tools need this more, not less.

Footnotes

1. Shneiderman, B. and Maes, P. “Direct manipulation vs. interface agents.” interactions 4(6), 1997. doi:10.1145/267505.267514 ↩

2. Shneiderman, B. “Direct Manipulation: A Step Beyond Programming Languages.” IEEE Computer 16(8), 1983. doi:10.1109/MC.1983.1654471 ↩

3. Subramonyam, H., Pea, R., Pondoc, C., Agrawala, M., and Seifert, C. “Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs.” CHI 2024. doi:10.1145/3613904.3642754 ↩

4. Subramonyam et al., section 4.1, analysis of ChatGPT. ↩

5. Amershi, S. et al. “Guidelines for Human-AI Interaction.” CHI 2019. Guidelines G1 and G2. ↩

6. “Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment.” CUI 2025. doi:10.1145/3719160.3736629 ↩

7. “Agentic Workflows for Conversational Human-AI Interaction Design.” arXiv:2501.18002, citing Hutchins et al. 1985 and Masson et al. 2024. ↩

8. Masson, D., Malacria, S., Casiez, G., and Vogel, D. “DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models.” CHI 2024. doi:10.1145/3613904.3642462 ↩