AI by Design — A Manifesto in 5 Acts

Where it comes from

I was educated as an industrial designer. That means I spent years learning to design by having conversations with physical materials — plastic, wood, metal, foam. The material resists in certain ways. It surprises you in others. You learn what's possible not by specifying it upfront but by making and responding to what you made.

When I started working seriously with AI, I had a recognition: this is the same relationship, but the material is entirely new. AI is non-deterministic. It has something like agency. It produces outputs you didn't fully anticipate, which teach you things you didn't know you needed to know. Donald Schön called this a "reflective conversation with materials" — the designer gets a response back from the medium, sees what is happening, and makes judgments about what to do next.

What Schön didn't anticipate — because he couldn't in 1996 — is that the conversation would become multimodal. The channel through which you speak to AI is itself a design decision. Text, image, and design system are not interchangeable — they surface different kinds of knowledge and produce fundamentally different kinds of output.

The three channels

Textual

Intent, constraints, tone, structure

Logic, copy, architecture, decision frameworks

Use when you know what you want but not how to get there. Text is the channel for articulated intent — when you can say what you're after, even imperfectly. The AI responds with structure. What comes back often reveals gaps in how you framed the question.

Visual

Sketches, screenshots, mood references, existing UI

Form variations, unexpected combinations, visual alternatives

Use when you're exploring or when words aren't enough. Sketching captures intent that language can't — gesture, proportion, compositional instinct. Feeding a sketch to AI is a different kind of question than describing what you want. The gap between what you drew and what comes back is where the design insight lives.

Systemic

Design tokens, component libraries, brand constraints

Constrained output that stays coherent at scale

Use when you need the output to stay inside a design system. Feeding AI your token library or component constraints changes the nature of what it produces — it stops exploring and starts executing within rules. This is the channel for delivery, not discovery. The constraint is the brief.

The key insightThe channel is the prompt. Choosing the wrong channel doesn't just limit the output — it asks the wrong question. Feeding a description when you should feed a sketch produces a plausible answer to a question you weren't actually asking.

The conversation

Schön's phrase "backtalk" describes the moment when the material surprises the designer — when what comes back doesn't match the intention, and that gap is generative rather than frustrating. AI backtalks constantly. An unexpected metaphor reveals something you didn't know you were looking for. A generated layout makes you realize your mental model of the problem was wrong. A tone mismatch reveals that you hadn't actually decided what you wanted the experience to feel like.

The skill is staying in the conversation long enough for backtalk to happen — not optimizing the prompt until the output matches your existing idea. That's not conversation, it's command execution. The designer who only accepts output that confirms their prior intent isn't conversing with a material. They're using a faster pen.

What practitioners sayJoel Lewenstein (Anthropic): "Claude is not a slavish executor of my vision — it should push back, be a sparring partner." Pablo Stanley: "AI can do volume and give you all the options, but taste will allow you to know what to ask from it." Jenny Wen (Anthropic): "The mocking up part is now 30–40%. The rest is pairing, polishing, last-mile work." Ammaar Reshi (Google DeepMind): demonstrates the multimodal switch in practice — Figma for visual exploration, prompts for behavior, real-world data as input. Channel choice is always intentional.

The moves

Choose the channel deliberately

Before starting, ask: am I articulating intent (text), exploring form (visual), or executing within constraints (systemic)? The answer determines the channel. Mixing modes mid-conversation is fine — but know which mode you're in.

Treat output as a sketch

Iterate fast. Throw most of it away. Annotate what you learn from the gap between what you expected and what came back. That annotation is the design thinking — not the output.

Stay for the backtalk

Don't stop at the first plausible output. Push until the conversation produces something that surprises you. If nothing surprises you, you may be commanding rather than conversing.

Apply taste to stop

Taste — Pablo Stanley's word — is the judgment that ends the loop. Not when the output is technically correct, but when it has earned its place in the work. That's a human judgment. AI can't make it.

The watch-outFluency without judgment. The conversation can become a refinement loop that produces polish without purpose. Schön called this "reflection without action" — the designer keeps adjusting without ever committing. The antidote is the stopping condition: what would it mean for this to be done?

Grounding

Donald Schön, Reflective Conversation with Materials (1996): "There is no direct path between the designer's intention and the outcome. As you work a problem, you are continually in the process of developing a path into it, forming new appreciations and understandings as you make new moves."

ACM DIS 2024 research (DesignPrompt): multimodal prompt input encourages designers to explore and express themselves more effectively, with interaction preferences shifting depending on whether they are seeking inspiration or a specific output — which maps directly to the textual/visual channel distinction above.

AI-assisted design is a reflective conversation with a new kind of material.

Sections

Submit

Colophon