It was about six weeks after Lehman Brothers collapsed. The coverage of his death chronicled his life’s work, including his book of voices from the Great Depression, first published in 1970. An analyst drew a parallel to the uncertainty we were entering.
So I went looking for it.
One idea turns up in that book more than once, as I remember it, and one telling of it stayed with me. I can’t give you the exact words. I’ve looked for the passage and couldn’t find it. So it goes something like this…
A little girl during the Depression. A hobo comes to the house. Her mother is generous with people in his position, as generous as limited means allow. He explains how he ended up where he is, and there’s some anger in it, or so I read it.
After he leaves, the girl asks her mother whether what he said was true.
Her mother says it was true from his point of view.
The girl carried that sentence long enough to hand it to Terkel on tape. I’ve carried it since.
I can’t vouch for the wording. I can vouch for the sentence.
That was 2008. On October 16, I relearned it, in a comment thread, over a quote I found in an old file box.
It was an SD Times from June 2019. A quote box stopped me cold. Theresa Lanowitz, saying that once Mercury Interactive went under, no vendor stood up for testers as a valuable part of the software engineering process.
Technically, Mercury didn’t go under. They were acquired by HP. But her point struck me as dead on, and I think I know why it struck me so hard.
I spent years at Empirix, one of Mercury’s competitors. We were rivals in the marketplace, but we stood on the same side of the industry, selling tools and making the case for the people who used them.
Mercury dominated that marketplace. They were the biggest, the loudest, and the most capable, and they had the megaphone. The rest of us could whisper and nudge, but we never had the platform.
That was the landscape I knew… the booths, the conferences, the user groups, the messaging. From where I stood, outside the companies we served, selling goods rather than services, the products and the stories around them were what drove the advocacy. Mercury was the proof. They advocated fiercely for the tester’s seat at the engineering table, flaws and all.
So I wrote it up the way I saw it. When Mercury disappeared, nobody picked up the megaphone.
The rest of us weren’t in a position to pick it up, because we were disappearing too. An industry-wide wave of consolidation swept up Mercury, Empirix, Segue, Compuware, and others. What had once been companies built around testing became product lines inside much larger organizations. Testing went from being a company’s whole reason for existing to a line on someone else’s price list.
The megaphone was gone. And one by one, so were the whispers.
That was my truth. It was true from where I stood.
Then Theresa Lanowitz showed up in the comments.
She did more than agree. She sharpened the point. The Mercury tools never went away, she said. The spirit of what the company stood for did… a company that put customers first, backed its partners, and gave testing professionals a voice. In a B2B market, she noted, that kind of commitment is rare.
Then she said something I hadn’t thought to say. She sees a lot of similarities between testing and cybersecurity, where she works now. The difference is that security made a case for itself and “has a CISO as an executive sponsor.” Testing, she said, had few companies with a C-level executive advocating for it.
And she voted Yes. Testers still need a seat at the table.
I had been looking at the vendor. She was looking at the org chart.
That was the part I hadn’t seen. From where she stood, looking back at testing through the lens of cybersecurity, the question was different: who sits at the table when the decisions get made?
I wrote back that this was the exact analogy I’d been missing. I meant it.
The mother’s answer comes back to me here. My view of Mercury was true from where I stood. Hers was true from where she stood. Neither of us was wrong, and it took her stepping outside, and me listening, to see the shape of the whole thing.
So who is testing’s CISO?
I don’t think the answer is a vendor, and I suspect that’s the quiet lesson of the Mercury years. A vendor can lend a craft its voice. It can’t give a craft a seat. And as the consolidation showed, a vendor’s voice lasts only as long as the vendor’s strategy does.
Security got its seat, as I read it, by changing what it talked about. It stopped describing itself as a technical activity and started describing itself as protection against systemic, existential business risk. That language earns an executive sponsor, and an executive sponsor earns a permanent place at the table, all the way up to the board.
Testing, too often, went the other way. It got pushed down into tactical delivery and treated as a cost center, or a bottleneck on a release schedule. Nobody takes a bottleneck to the board.
Now the ground is moving again. Autonomous tooling and generative models are producing code faster than anyone has seen. Deploying a system nobody has verified brings enterprise risk in just as fast as deploying insecure code does. That’s the Verification Tax, the line item that never appears on an executive dashboard until an outage forces the issue, coming due at machine speed.
Without someone at the executive level demanding rigorous verification, the belief that AI removes the need for practitioners could turn out to be a very expensive lesson for some organizations. That’s my read, and I hold it loosely.
Which brings me back to the doorstep.
A hobo at a mother’s door was one person telling one version of the truth, and a mother had one sentence to keep the moment honest. A feed is that doorstep multiplied a thousand times, and almost nobody gets the mother’s sentence. Worse, more of what arrives now is true from nowhere. Generated text has no standpoint to check, no history, nothing at stake.
The comment thread was the opposite. A named person, with a history and a stake, stood somewhere I couldn’t and told me what the view was like from there. I could check her. I could answer her. The sentence did its work again, nearly a century after a girl first heard it.
It reached me through an obituary, a book, a girl’s memory, and a mother’s answer. Every hand on that chain had a name. That may be how this kind of truth still travels best.
That same Friday, I posted a joke job listing under that thread. Advocate for the Software Tester. Must be completely unfunded. I thought I was kidding. The seat may have a different name than I thought, and the listing may need a sponsor more than an advocate.
My verdict, for now, and offered for debate: testing may not need a louder megaphone. It may need a seat, and someone senior enough to sit in it.
That’s true from where I stand. I’d like to hear what’s true from where you stand.
I would love to hear from you… rick@qameditations.com
]]>This post is about the two questions a sponsor asks before they fund any of it. You can’t claim a dividend without saying what the investment was, and you can’t say when you’ve earned it without knowing when to stop. Those questions tend to arrive in the same breath… How long is this going to take? And how will we know when you’re done?
Way back in 2013, Elizabeth Hendrickson, to her credit and a testament to her enduring craft, answers the first one honestly in her book Explore It!, and the honest answer is “it depends.”
Brainstorm a list of charters for the feature area, estimate how many sessions it takes to cover them, and accept that programming, checking, and exploring are so tangled together that better engineering makes every one of them faster, while shortcuts slow all three down at once. Session-based test management gave the craft real accountability on the other side of the ledger… time-boxed sessions and a clear account of where the hours went. But it reports what happened. It was never built to predict how many sessions a system needs before anyone starts. In practice, whenever a team lacked a structural picture of what it was testing, the answer to “how long” collapsed back into “it depends,” which sounds like a shrug to the person holding the checkbook. That’s a great way to not get funding.
I don’t think her answer was wrong then. I think it was the best available one. However, I think it may have finally expired…
The old estimate takes two completely different quantities and blends them into a single guess. How many charters does this system actually need? And how long does each one take to run?
Take an order-to-cash workflow that, for illustration, presents fourteen genuinely distinct risk interactions worth a charter each. Whether a tester and an agentic tool take forty-five minutes or three days to work through those fourteen is the execution variable. The fourteen is something else entirely. That number is anchored to the system, though how I model it, and which risks I decide matter, will push it up or down.
In 2013 there was little practical reason to pull those apart. The same human did the thinking and the doing, so one guessed time-box covered both. That’s no longer true. Execution has been separating from judgment for a while now, and the last few posts in this series have explored that exact idea. Tooling can compress the time it takes to run a charter, sometimes dramatically. What it cannot compress is the number of charters worth running, because that number isn’t governed by how fast anything executes. It is governed far more by how many genuinely different things the system can do.
Which brings back an old idea from this series: eleven loops, not eleven hundred. A system that looks like a thousand test cases can turn out to have eleven real execution paths, and the real number is the one that actually matters. Those paths aren’t charters, mind you. They feed the charter count, they aren’t the count itself.
Once you’ve done the work from Parts 6 and 7 (reconstructing the real dimensionality of a system from its inputs, code paths, observed behavior, and whatever documentation survived), part of the charter count stops being a brainstorm. Formal factor-and-level design gives you the minimal set of combinations covering the interactions that actually exist. That’s arithmetic on a structure you’ve already mapped. To be clear, those combinations aren’t charters yet. They’re the raw material charters get built from.
I’ll try to be exact (or at least pretty close) about what that arithmetic gets you, because any experienced practitioner will push back on it if I don’t. The factors and levels describe combinations; they don’t describe surprises. What they produce is the foundation… the baseline, structural reconnaissance charters that establish minimum coverage of everything the map says exists. That is real, countable, and estimable. It is not the whole of exploring. Hunting along boundaries, state transitions, and integration seams where a system behaves in ways nobody modeled is precisely what human judgment is for. That judgment is freed up because the foundation no longer requires a brainstorming session to construct. The count gives you the floor. Exploration is what you do standing on it.
There’s a second effect hiding in there, and it may be what sits underneath Hendrickson’s tangle. A team with solid regression coverage, a healthy build pipeline, and deterministic checks tends to resolve easily covered risk before an explorer ever shows up. That doesn’t make the system any simpler, but it does shrink the part of it that still needs exploring. “Better practices make exploring faster” is more than an intuition. There is a plausible structural mechanism behind it. Mature engineering may not just run the same sessions quicker; it may reduce the number of exploratory sessions that need to exist.
So the new estimate has two honest parts: a charter count anchored to the structure of the system, and a per-charter execution time that tools are free to shrink. I’d suggest not blending them. When a sponsor asks what it costs to speed things up, you can finally say which of the two numbers the money moves.
The second question is harder, and Hendrickson’s answer is a good one: Stop when your open questions are answered. Stop when you’re no longer learning anything new. Stop when more information wouldn’t change what anyone does next. Or stop when it’s clear the organization won’t act on what you find regardless.
Every one of those relies on an experienced practitioner reading the room, and they are sound. But to a CFO or a VP of Engineering under ship-date pressure, they sound indistinguishable from “whenever the tester feels like it.”
I’d like to offer something firmer, with an explicit boundary around it. For the portion of a system whose structure is actually mapped, you can state a stopping rule that doesn’t depend on anyone’s mood:
Notice what that rule does not claim. It does not claim there is nothing left to find. No explorer ever knows that. It claims the known questions are exhausted and the trail has gone cold, within territory you can explicitly name. That is a statement a sponsor can verify, and it stays honest about the dark corners outside the map. Those are what the next session is for, and the one after that.
One more boundary, and it matters. Finishing the exploring is not the same as approving the release. Our job is to surface what we found and be plain about what we left open, the fiduciary idea from Part 3. Deciding to live with that risk belongs to the people who own it.
The reason exploratory testing kept losing the “how long” and “how will we know” arguments was never that the answers didn’t exist. It’s that nobody had separated the question into its real parts. Count the foundation charters from the structure. Time the execution separately. Stop when the questions and the tensions are both cleared, for the territory you’ve actually mapped.
Sponsors don’t need a guarantee that a system is infallible. They need a record showing you looked everywhere you had a map for, answered every question on the table, and can tell them plainly where the map ran out. So, how many charters is enough? Enough to cover the floor, which you can calculate from the structure and be held to. Beyond that, it’s a risk-informed call about how much further to go, and it belongs to the people who own the risk. It’s a better answer than “it depends,” and it’s one you can stand behind.
I hold the hypothesis on engineering maturity reducing charter counts loosely, and I’d be interested to hear from teams who have watched that dynamic play out, or observed the opposite.
]]>I told the team I was talking to, in almost exactly these words, that it seemed like execution was what their platform actually automated, while the judgment behind what is worth testing, why it matters, and what constitutes an unacceptable outcome still remained in human hands… That I was, in effect, separating execution from judgment, and asking whether that was a fair read of where their own product actually draws the line.
“Right, exactly.”
I wasn’t fishing for a concession. I expected some version of “well, it’s more nuanced than that,” the kind of hedge a vendor gives when a question gets a little too close to a limitation they’d rather not talk about on a call. Instead I got two words, immediate, unhedged, from the people who actually build the thing. No marketing claim… an engineering team stating an accurate description of their own architecture.
None of this is a knock on what the platform does. Visual comparison across builds, orchestrating a check across a dozen browser and device combinations, catching a regression a human would blink past… that’s real, valuable, and something I wouldn’t want to do by hand even if I could, especially at my age, with these reading glasses. Execution isn’t the lesser half of the overall split.
What was more interesting in a way, was a detail one of their own engineers offered, unprompted, about why they don’t hand execution itself over to an LLM. Their reasoning wasn’t philosophical. It was operational: for this layer, they needed repeatable results. Ask a large language model to evaluate the same capture twice and you may not get the same answer. Their core diff engine gives you the same answer every time, because it isn’t guessing, it’s comparing. The people building AI-powered testing tools are, themselves, choosing deterministic mechanisms for the execution layer precisely because that layer needs to be trustworthy in a way generation isn’t built to be. That’s not a limitation they’re apologizing for. It’s the design decision.
The interesting part wasn’t that an AI testing company wasn’t using AI everywhere. It was that they knew where not to use it.
The same call surfaced two limitations they were unusually direct about, and I found those answers more useful than another feature list.
First: no built-in way to encode an absolute invariant (an Always/Never rule, in the language I’ve been using throughout this series) as a standing guardrail the platform checks against automatically. “Maybe on the roadmap” was the answer. Not yet.
Second, no path for bounded or directed exploration. Nothing in the platform today lets you hand it a negotiated risk hypothesis and have it explore around that specific boundary. It can crawl, it can map, it can compare what exists against what existed before. It doesn’t yet take a Quality Word and go looking for the ways it might be violated. Also “maybe on the roadmap.”
I’d rather have those two honest no’s than a confident, vague yes. A vendor who tells you where the edge of their own platform sits is a vendor whose yes you can actually trust when it comes.
It would be easy to hear what I am saying here as an argument that the platform “isn’t good enough yet”, and that once the roadmap items ship, the distinction dissolves. I don’t think that’s right, and I don’t think it’s what the “right, exactly” was actually recognizing.
The line isn’t a maturity gap that better engineering simply closes. It’s a structural one. A sufficiently capable system may generate its own hypotheses. It may identify anomalies, propose risks, and eventually conduct remarkably sophisticated exploration without waiting for a human to tell it where to look. But generating a hypothesis isn’t the same thing as establishing its consequence. Someone still has to determine what matters, what is acceptable, what must never happen, and where uncertainty carries enough risk to warrant investigation. The tool can get progressively better at acting on judgment, and even at proposing judgment. But the basis on which that judgment matters still has to come from somewhere.
If the line holds, and this conversation gave me more confidence that I’m looking in the right place, then the practical shift isn’t “wait for AI testing to mature and the judgment problem solves itself.” It’s that the judgment work becomes the scarcer, more valuable half of the job, precisely because the execution half is being absorbed, faster and more thoroughly than most teams have adjusted to yet. The Quality Words, the charters, the plausibility cross-reference, the work of deciding what’s worth pointing all this excellent execution machinery at, doesn’t get automated away by a more capable version of the same tool. It becomes the part all that machinery still depends upon.
Not a hedge against the technology… It’s a bet on where the actual leverage may be, made a little more confidently now that I’ve heard the people building the technology say so themselves.
]]>Most of my career has lived inside enterprise platforms that have been running for a decade or more, customized layer over layer, by people who’ve long since moved on to other companies, other careers, sometimes retired outright. Ask “how many real execution paths does this have” in that conference room, and you get silence, or worse, a confident guess from someone who’s been there three years and genuinely doesn’t know what they don’t know. There’s no original architect. There’s a system that works, mostly, and a pile of accumulated knowledge scattered across whoever’s left.
That’s the actual, common case, and it’s where most of this type of work really happens.
When there’s no one to ask, the real dimensionality of a system still exists, it’s just gone quiet. Finding it again means pulling from a handful of genuinely different sources. Each one captures a different kind of “truth” as you piece together the structure of the system, I think you need them all as one won’t substitute for another.
Formal factor and level design starts from what’s still enumerable even when nothing else is, the known inputs, the known configuration options, and works out mathematically the minimal set of combinations needed to cover their real interactions. You don’t need anyone’s memory for this one. You need the input space written down, and a method for not testing every combination when testing the right handful covers the same ground.
Code-path analysis goes straight to the most honest witness available, the code itself. Whatever anyone remembers or forgot, the actual execution paths are sitting right there, discoverable by tracing what the system can structurally do, independent of anyone’s account of what it’s supposed to do. I know, this one is probably the hardest to extract when every team is already too busy.
Behavioral observation is the one that surprises people most. Watching what a system actually does in live use, not what anyone believes it does, regularly turns up workarounds nobody documented and nobody would have thought to mention, because the people using them don’t think of them as unusual. They just think of them as how the job gets done. Those shadow paths are frequently exactly where the real risk concentrates, precisely because nobody’s watching them on purpose. By the way, this is a uniquely human endeavor.
Documentation archaeology is the least glamorous and often the most necessary, reconstructing entity relationships and data structure from whatever actually survives: old schemas, half-finished wikis, a ticket from four years ago that happens to explain a decision nobody remembers making. It’s slow, unglamorous work, and it’s frequently the only source available at all for a system old enough that nobody left was there when it was built.
None of these four is sufficient alone. A code-path analysis tells you what’s structurally possible without telling you what actually happens in production. Behavioral observation tells you what happens without telling you why the code allows it. Used together, they triangulate toward something close to the real number, the way four separate accounts of an intermittent bug, cross-checked against each other, get you closer to the truth than any single account manages alone.
There’s a sequencing discipline here, and getting it backward quietly wastes a lot of time and a good deal of the value these four sources provide.
Structural knowledge belongs upstream, feeding generation. Before you narrow anything down, you want the full, honest picture of what’s actually there, all four sources, gathered without yet asking which parts matter to anyone. Judgment belongs downstream, tempering what’s already been generated. Only after the real dimensionality is mapped do you filter it against what an organization actually cares about, which is a different, later question entirely.
Reverse that order, and the damage isn’t obvious right away. Filtering by priority before the structure is understood means you never see the full space in the first place, you only see the parts someone already thought to look for, which just recreates the exact problem eleven loops disguised as a thousand test cases was supposed to solve.
An AI system generating test coverage without any of this underneath it isn’t doing something categorically different from a human skipping the same step, it’s just doing it faster, and producing volume that looks like confidence.
Feeding a model the actual structural picture, the real paths, the real behavioral patterns, the real entities, before asking it to generate anything, isn’t a nice-to-have. It’s the difference between coverage that maps to something real and coverage that maps to whatever happened to be easy to generate. The architect who isn’t there can’t be replaced by more test cases. The four sources above might be the closest thing to bringing them back into that conference room.
]]>What actually broke it open was a single question put to an architect: how many genuinely different execution paths does this system actually have. Not customer profiles, not test cases, paths. The answer was eleven. Eleven real loops, dressed up in thousands of superficially different clothes, because different customer data walking through the same handful of code paths looks like variety and isn’t. A handful of transactions built specifically to exercise those eleven loops and their interactions found the defect that thousands of cosmetically distinct test cases had missed entirely.
That’s the question worth asking before any charter gets written, and it’s a harder question than it sounds. Apparent variety and real variety are not the same thing, and a charter aimed at the wrong one burns real attention re-testing the same underlying logic in different costumes.
The charter template itself, explore this, with these resources, to discover this, has already come up in this series. What hasn’t come up yet is where the content of a good one is actually supposed to come from, and the honest answer is: several places, none of them sufficient alone.
Stakeholders will tell you plenty if you ask the right shape of question, not “does this work” but “what would worry you if it didn’t.” Bug databases and old support tickets are a quieter version of the same thing, a record of what’s already gone wrong once, worth mining rather than starting fresh every time. A requirements or planning meeting is a genuinely efficient source too, since the conversation naturally produces both halves of a charter at once, the resources being discussed and the risk being raised, in the same breath, without anyone having to translate one into the other afterward.
And there’s the question worth asking anyone with reputational skin in the game: what’s the single failure that would force you to face a board or a furious customer base on Monday morning? It cuts straight through polite engineering optimism, surfacing the exact operational risks everyone quietly worries about but rarely puts into an official requirement.
One source deserves its own mention, because it’s easy to overlook. Marketing copy, packaging, and public claims are, technically, an oracle, someone wrote down what this thing is supposed to do, from a vantage point completely separate from engineering. Comparing actual behavior against that claim is a real, legitimate technique.
For a heavily customized enterprise platform, there’s rarely public marketing that describes your specific configuration, so the equivalent source has to be internal, the business case that got the customization funded in the first place, the SLA promised to the business unit it serves, the training materials written for the people who’d use it. Someone, somewhere, already wrote down a confident claim about what this thing was supposed to deliver. That claim is worth testing against, exactly the way a product’s marketing copy would be.
Years ago, when I was on the product side at Empirix, our QA manager, Joe Strazzere, used to drive me crazy. Whenever an edge case surfaced that felt entirely implausible, Joe had a flat, unblinking response that cut straight through whatever roadmap defense I was preparing: “Right. And when they do, how should the system respond?”
Not whether it would happen. What happens once it does.
At some point during that ongoing friction, Joe tossed a printout onto my desk. It had Michael Bolton’s name at the top and an exhaustive catalog of reasons a real user might do something a developer or product manager swears no sane person would ever attempt. I still have that paper somewhere. The URL printed along the footer faded out years ago, partially obscured by scribbled margin notes, dog-eared corners, and a couple of old coffee rings. Only Michael probably remembers the exact canonical link at this point.
The items themselves, though, haven’t aged a day: a slip of the fingers, plain curiosity, confusion caused by bad interface design, an assumption that because a sequence worked in one screen it should work here, or bulk data shoved through an API in patterns no human at a keyboard would ever produce.
On its own, that printout is just a catalog of human error and mischief, interesting, but directionless. But crossed against an organization’s negotiated priorities, it turns into an operating weapon.
Take a priority leadership has explicitly committed to protect, and ask, one by one, which of those behaviors could plausibly violate it. Not all of them will apply. But the ones that do become razor-sharp charters. One side supplies what actually matters to the business; the other supplies the messy, unscripted reality of how systems actually break.
That’s really the whole discipline … Don’t chase apparent variety, chase real dimensionality. Ground every candidate in something an organization has actually told you matters, not a generic assumption about what a well-built system should do. And when you’ve got a real priority and a real, plausible way it could break, that’s a charter worth running, not one of a thousand cosmetic variations on the same eleven questions.
]]>I don’t think it does. I think, examined closely, these four techniques turn out to be some of the best evidence for the distinction there is. Not because they’re weak. Because every one of them, examined at the level of how they actually work, needs a human to hand it something before the machinery has anything to chew on.
Start with the one that looks, on the surface, most like pure automated exploration. Chaos engineering deliberately injects failure into a running system, kills a service, adds latency, partitions a network, to see what breaks. That sounds like exactly the kind of unscripted, adversarial probing this whole series has been describing.
Look at the actual methodology, though, the one the field’s own founders wrote down. The first, mandatory step of any real chaos experiment is defining a steady-state hypothesis, a stated, specific claim about what “normal” looks like, before anything gets broken. You cannot run a meaningful experiment without first saying what you expect to still be true afterward. The automation doesn’t invent that expectation. It executes against one a human wrote.
There’s a detail in that same methodology worth mentioning and thinking about, because it isn’t my argument, it’s theirs. Netflix, the team that essentially invented modern chaos engineering, has said outright that a good steady-state metric is “more a business metric than a technical one,” and that business metrics tend to be more useful in this work than technical ones. That’s the people who built the practice saying, in their own words, that the thing driving a meaningful experiment isn’t something the tool discovers on its own. It’s a business-situated judgment a human supplies first. Chaos engineering is a remarkable execution mechanism for a hypothesis. It has never claimed to originate one.
Property-based testing generates hundreds or thousands of randomized inputs and hunts for the one that breaks a rule you stated, reversing a list twice returns the original, a balance is never allowed to go negative, that kind of thing. The generator is genuinely impressive. It’ll construct edge cases no human would think to type by hand, and when it finds a failure, it’ll shrink the input down to the smallest version that still breaks the rule, which is its own small miracle of usefulness.
But look at what it’s testing against. The property. Someone had to decide that reversing twice should return the original, or that a balance should never go negative. No tool I’m aware of derives that rule on its own, from first principles, without a human deciding it mattered. The property is providing the same basic function as a charter does, at least in part, it names the thing worth investigating. The generator and the shrinker are extraordinary execution machinery in service of that charter. They’re not a replacement for having one.
Fuzzing deserves to be split into two cases, because they’re not actually the same technique, and one of them is a genuinely harder case for my argument than the other.
Plain fuzzing throws malformed, random, or unexpected input at a system and watches for a narrow, mechanical signal, does it crash, does it hang, does it trip a memory-safety check. It’s superb at that specific job. It has no opinion whatsoever about a result that’s wrong without crashing. A parser that silently corrupts a customer’s data without ever throwing an exception is invisible to a fuzzer looking only for crashes, because crashing was the only thing it was ever told to watch for.
Coverage-guided fuzzing is the more interesting case, and I want to give it real credit rather than wave it away. Tools in this category genuinely adapt as they run, favoring mutations that reach code paths they haven’t exercised before. That’s a real learning loop, not a fixed script, and pretending otherwise would be dishonest. But look closely at what it’s learning toward. Code coverage. Structural reachability. Whether a branch has been executed before. That’s the exact same shape as a crawler systematically mapping every corner of a web application, just running one layer deeper, at the level of branches instead of URLs. It’s optimizing for “have I been here,” not “does anyone with a stake in this business care what happens here.” A coverage-guided fuzzer will spend enormous effort exploring a code path that handles an internal logging format nobody outside the team will ever see, with exactly the same enthusiasm it brings to a payment-processing branch, because coverage doesn’t know the difference and was never asked to.
Differential testing runs the same input against two implementations, or two versions of the same system, and flags anywhere they disagree. It’s a genuinely powerful, fully automated version of something this series has already named, self-consistency, the idea that a system’s own internal agreement with itself is a real, checkable signal even without an external oracle.
It has the same limit self-consistency always had, though. It can tell you two things disagree. It cannot tell you which one is right, or whether either one is what the business actually needs, unless a human already decided which side counts as the reference. Two implementations can agree with each other perfectly and both still be wrong about what a customer actually needed.
I find it genuinely persuasive that laying these four out side by side, and it’s not something I could have argued nearly as well from just one of them. Chaos engineering, property-based testing, fuzzing, and differential testing come from different corners of the field entirely. Nobody building any one of them was thinking about the others, and none of them were built with this series’ argument in mind. And every single one, when you look past the marketing and into the actual mechanism, needs a human to supply something first, a steady-state hypothesis, a property, a definition of interesting coverage, a reference implementation, before the automation can do anything at all.
That’s not four coincidences. That’s the same framework showing up in four unrelated places, because it isn’t a limitation of any particular tool. It’s a structural fact about what automation can and can’t originate on its own. These techniques don’t threaten the case for human-directed exploration. They’re some of the clearest, most independent confirmation of it currently available, offered by people who never intended to make that argument at all.
]]>Now put it next to a second fact from the same moment: an inventory count that should have decremented after a purchase didn’t. Neither one, on its own, is a bug report. A response that’s a little slow happens. A count that’s momentarily stale happens. But a tester who’s spent real time inside this system doesn’t see two small, separate facts. They see one thing, a payload that may be getting submitted twice, a payment that may be about to double-charge someone, and they abandon whatever they were doing to go find out.
Nobody wrote a test for that. Nobody could have. And I want to spend this piece on the thing that happens in that exact moment, because I think it’s the part of exploratory testing that’s hardest to explain to someone who’s never done it, and the part most at risk of being quietly assumed away as AI moves further into this work.
I called this the human fiduciary in the piece before this one, and I want to slow down and earn that word properly here, because I used it once and moved on, and it deserves more than that.
A fiduciary exists, as a legal concept, because you cannot write a contract detailed enough to cover every situation that might arise. So instead of trying, you entrust someone with an ongoing duty, to act in your actual interest when the unanticipated shows up, precisely because you couldn’t have anticipated it yourself. That’s not a loose metaphor for what happened with the four hundred milliseconds. It’s a precise description of it. Nobody’s charter said watch for this exact combination. The entire value of that moment lived in a standing judgment that recognized it mattered, live, and acted, without waiting for permission or a rewritten plan.
I want to be honest about where the metaphor ends, though, because overclaiming it would undercut the piece rather than strengthen it. A fiduciary in the legal sense carries formal liability. A tester doesn’t, and shouldn’t be described as though they do. What they actually carry is narrower, and I’ve come to think it’s more honest for being narrower: an obligation to observe faithfully, connect what they’re seeing to what the business genuinely depends on, and put that connection in front of the person who bears the consequence, without deciding on that person’s behalf, and without letting a distorted or incomplete picture stand in for what’s true.
There’s an old lesson in this field, one that predates almost everything else in this series, that says any stakeholder should be able to report a bug. On its face that sounds like it flattens the tester’s role into something anyone could do. I don’t think it does, and working through why sharpened this whole piece for me.
A stakeholder reporting something they noticed is contributing raw observation. That’s valuable, and it should be welcomed, not gatekept. But raw observation isn’t the same thing as information, and turning one into the other is real, skilled work. It means taking the four hundred milliseconds and the stale count and asking what they mean together, whether they’re connected, what they’d cost if they’re real, and whether they’re the kind of thing this particular business would consider serious or shrug off. That synthesis is the tester’s actual job. And once it’s done, once raw noise has become a clear, specific piece of information, it belongs in front of whoever owns the consequence of being wrong about it. Not because the tester lacks the standing to decide. Because deciding was never the job. Informing faithfully was.
I think that’s the cleanest answer to why this is fiduciary in spirit rather than in name. A fiduciary’s whole function is to act faithfully in someone else’s interest when a decision has to be made without them in the room. A tester’s whole function, in the moment I opened with, is exactly that, minus the final decision. Watch, synthesize, surface, faithfully. Let the person who actually bears the outcome decide what to do about it.
One thing worth being clear about, because it would be easy to read this as a title reserved for the fifteen-year veteran: it isn’t. The fiduciary posture is an attitude of observational fidelity, not a seniority grade. A junior tester who flags an odd UI flicker because it feels wrong for a real user is exercising the exact same muscle as a longtime architect connecting a latency spike to a pricing tier. What varies with experience is how much you catch and how quickly you can explain why it matters. What doesn’t vary is the obligation itself, and it belongs to anyone doing the watching, on day one or year fifteen.
This isn’t one obligation with one intensity, and I think treating it that way makes it sound more dramatic than it usually is, right up until the day it isn’t.
Most of the time, this plays out as an ordinary business disagreement. You think a behavior matters more than the team currently does, you say so, they weigh it against everything else on their plate, and a reasonable answer comes back that isn’t the one you’d have picked. That’s not a crisis. That’s just work.
Occasionally it’s sharper than that, a real professional confrontation, where you’re not just flagging a preference but pushing back hard on a decision you think is genuinely wrong, and the conversation gets uncomfortable because something real is actually at stake. I’ve been in rooms like that. They’re not comfortable, and they’re not supposed to be.
And every so often, rarely, it crosses into something else entirely, a genuine ethical line, not a disagreement about priorities but a matter of what shouldn’t be allowed to happen regardless of what anyone decides. That’s a different category of obligation, and it’s worth naming as distinct rather than blurring it into the other two, because collapsing all three into one dramatic gesture cheapens the rare cases that actually deserve the word.
Most days, this obligation looks like the first kind. The reason it’s worth naming at all is that you don’t get to choose in advance which kind a given moment will turn out to be. You only find out by paying attention closely enough to notice when it’s stopped being the first kind and become something else.
I used to think about this obligation almost entirely in terms of profit and loss, because that’s the currency most commercial software gets measured in, and it’s the language a budget conversation understands.
But the underlying principle isn’t really about dollars. It’s about whoever bears the actual consequence of an outcome, and dollars are just the currency that happens to apply in most commercial contexts. Change the context and the structure survives, only the currency changes. In a military or safety-critical system, the consequence isn’t profit, it’s mission success, operational readiness, and in the worst cases, someone’s life. That world already has a more explicit, doctrinally codified version of this same idea than most companies do, command responsibility, the principle that accountability for an outcome follows whoever held command over the systems and people involved, regardless of who executed the specific failing task. It doesn’t allow the kind of ambiguity a corporate postmortem often does, about whose fault a bad release really was. The chain is explicit by design.
Worth being precise about where the parallel actually sits, since it’s easy to blur two different roles together here. Command responsibility says the liability stops with whoever held command, not with the tester. That’s exactly right, and it’s not in tension with anything said earlier about a tester not carrying formal liability. The fiduciary role isn’t the commander’s, it’s closer to the intelligence officer’s: the one whose entire job is making sure the person who does carry that liability is never operating on a distorted, incomplete, or quietly sanitized picture of what’s actually happening.
I bring that up because I think it clarifies rather than complicates the point. The human fiduciary isn’t a commercial invention dressed up in serious language. It’s the same underlying obligation, observe faithfully, synthesize honestly, surface it to whoever actually bears what happens next, showing up in whatever currency the domain happens to be denominated in.
Of everything in this series so far, I think this is the idea I’d defend most stubbornly if someone tried to talk me out of it, because I think it’s the one most likely to get quietly assumed away as AI keeps moving into this work.
It would be easy to believe that a sufficiently well-built agent, given enough context and a good enough prompt, could eventually catch the four hundred milliseconds and the stale count too. In one narrow sense, it already can, and it’s worth being precise rather than defensive about that. Modern observability and anomaly-detection systems are genuinely good at cross-telemetry correlation, spotting that a latency spike and a database anomaly happened in the same window, often across thousands of concurrent requests, faster than any human watching a single screen ever could. That’s not a hypothetical capability. It’s real, and pretending otherwise would be the kind of claim that falls apart the moment someone who actually builds these systems reads it.
What that correlation-spotting can’t do is the part that actually mattered in the moment I opened with. A machine can tell you Event A and Event B happened together. It can’t tell you that, for this particular business, that combination means a specific high-value customer on a legacy pricing tier is about to be silently double-charged in a way that breaches a contractual promise nobody wrote into a monitoring rule. That evaluation isn’t a harder version of the same technical problem. It’s a different kind of problem entirely, business-situated, built from months or years spent learning what a particular system is actually supposed to protect. The gap isn’t in seeing that something happened. It’s in knowing what it would cost, to whom, and why anyone should care.
That’s worth holding onto, and so is this: none of it should live only in one person’s head. If this obligation is real, its value is only realized once it stops being locked inside a single tenured employee and gets transmitted, into charters other people can run, into a team’s shared sense of what actually matters here, into something the organization holds rather than something that walks out the door the day one person leaves. A practice that depends entirely on one irreplaceable person’s accumulated memory isn’t a fiduciary duty being well discharged. It’s a bus-factor problem that hasn’t been noticed yet. Part of the obligation is making sure what you’ve learned gets out of your own head before it has to.
That’s the piece worth holding onto as this series keeps going. Not as a defense of testers for its own sake, but because I think it’s true, and because losing sight of it quietly, one convenient assumption at a time, is exactly how something valuable disappears without anyone deciding to get rid of it.
]]>I understand the instinct. If a sprint runs long and the test plan never gets written, calling what happens next “exploratory testing” feels better than calling it what it usually is, which is someone clicking around under a deadline, hoping their instincts are good enough to cover for the plan that never existed. I’ve done it myself, early on. I suspect most of us have. And I think that quiet substitution, wandering standing in for a discipline, is the single biggest reason exploratory testing has such a credibility problem with the people who control the budget for it. They’ve watched it happen. They’ve seen “we’re doing some exploratory testing” turn into three unproductive hours and a shrug, and they’ve filed the whole discipline under the same heading as that shrug, without ever seeing the real thing.
So before this series goes any further, I want to draw the line carefully. Because the real thing is not wandering. It has a shape.
There’s a useful distinction that’s been sitting in the testing literature for longer than most people realize, and it’s worth stating plainly, because everything else in this piece depends on it.
Testing, all of testing, is really answering two different questions. The first is whether the software does what it’s already supposed to do. You wrote a requirement, you built a test around it, you run the test, you get a clean answer. That’s checking. It’s valuable, it’s necessary, and it will never once surprise you, because by design it only ever confirms or denies something you already thought to ask.
The second question is harder, and it’s the one checking structurally cannot answer: what else is true about this software that nobody thought to ask about in advance? That’s exploring. And here’s the part worth sitting with… neither of these is a lesser version of the other. A checking suite, however exhaustive, is a net with a fixed weave. It catches exactly what it was built to catch, and nothing it wasn’t. The finest net in the world still has holes shaped like the things nobody imagined needed catching. Exploring is what goes looking in the space between the threads.
If you know this territory well, you’re probably already thinking of the tools built specifically to search that space automatically… property-based testing, fuzzing, chaos engineering, differential testing, techniques that can throw millions of machine-generated permutations at a system in less time than it took me to write this sentence. I don’t think they refute anything I’m about to say. I think, looked at closely, they turn out to be some of the best evidence for it there is. That argument deserves its own room rather than a paragraph borrowed from this one, though, so I’ll make it properly in a companion piece later in this series, currently going by the working title “The Fuzzy Map Lines in Exploring.”
I’ve heard this stated more bluntly, and I like the bluntness. Scripted testing tells you what the system is supposed to do. Exploratory testing asks what the system actually does when a real person, impatient, unpredictable, skipping steps, gets their hands on it.
Here’s a distinction worth being precise about, especially for anyone who spends their days inside agentic tooling and might read the last few paragraphs and think, fair enough, but isn’t that just a better prompt?
It isn’t, and the gap matters enough to name directly. Imagine a session where a response comes back four hundred milliseconds slower than it should, and in the same breath, an inventory count that should have decremented didn’t. Nobody wrote a test for that specific combination. Nobody could have, because the two signals only mean anything together, and only to someone who already understands what this system is supposed to do when it’s healthy. A tester who catches that isn’t executing a step. They’re synthesizing, in real time, a hypothesis nobody handed them, abandoning whatever they’d planned to do next, and going after it instead. That pivot is the exploring. Everything before it was setup.
I want to be fair to what modern agentic tools actually do here, because the honest version of this argument is stronger than the convenient one. A tool-calling loop genuinely adapts within a run… it sees an error, sees an unexpected value, and changes its next action accordingly. That’s real, moment-to-moment steering, and pretending otherwise would be a claim any AI-native reader could take apart in a sentence. What that loop is steering by is the part worth being precise about. It’s reacting to structural, technical signal, an error code, a changed field, an output that breaks a pattern. It has no way to connect that signal to something like “this delay, paired with this stale count, means the payment might be double-charging someone,” because that connection isn’t technical. It’s business-situated, built from months or years spent understanding what this particular system is actually supposed to protect. A static prompt, however well written, can’t carry that kind of accumulated understanding into the moment it’s needed. It can only carry what was already known when it was written.
That’s the piece I’ve started calling the human fiduciary, and I mean the word carefully, not loosely. A fiduciary exists because you can’t write a contract detailed enough to cover every situation that might arise, so you entrust someone with an ongoing duty to act in your actual interest when the unanticipated shows up. Nobody’s charter said watch for this exact combination, because nobody could have written it in advance. The value was entirely in a standing judgment that recognized it mattered, live, and acted on it. I’ll say plainly that this isn’t a legal claim… a tester doesn’t carry a fiduciary’s formal liability. What they carry is narrower, and I think more honest: an obligation to observe faithfully, connect what they’re seeing to what the business actually depends on, and surface it to whoever bears the consequence, without deciding on that person’s behalf and without letting a distorted or incomplete picture stand in for the truth.
Here’s where I want to be careful, because there’s a lazy version of this argument I don’t want to make.
It would be easy to say the difference between a real session and wandering is practitioner discipline… that good testers stay focused and bad ones drift, and leave it there. I don’t think that’s honest, and I don’t think it would survive five minutes in front of anyone who controls a budget. If exploratory testing routinely degrades into unfocused clicking whenever it’s left to individual willpower under real deadline pressure, that’s not a character flaw in the people doing it. That’s a predictable, recurring failure mode of running the method without governance, and pretending otherwise, waving away every failure as “that wasn’t real exploratory testing”, is exactly the kind of unfalsifiable defense that makes a skeptical VP of Engineering trust the method less, not more.
So let me say this plainly instead. A real session has structure that doesn’t depend on anyone’s discipline holding up on a bad day, because it’s built as operational controls, not virtues.
The time box isn’t a display of focus. It’s a hard stop. A session that’s capped at ninety minutes cannot, mechanically, become the open-ended afternoon of clicking that erodes trust in this work. The charter isn’t a nice habit worth cultivating. It’s a reviewable artifact that exists before the session starts, a target, resources, and information sought, written down in advance, so nobody downstream has to take anyone’s word for what was supposed to happen. They can just read it. And the debrief isn’t good citizenship. It’s a required output, and a debrief that comes back empty or vague is not a private failure that quietly disappears, it’s a visible signal sitting in the session record, exactly the kind of thing an audit would catch.
I want to be honest about the limit of this, too, because overclaiming here would just be a quieter version of the same mistake. None of these controls make drift impossible. A charter can be written broadly enough to mean nothing. A debrief can technically exist and still say almost nothing real. Structure reduces the room for silent failure, it doesn’t eliminate the possibility of failure dressed up to look like structure. The honest claim isn’t that this method can’t degrade. It’s that, done this way, degradation stops being invisible. It becomes something a manager can actually see in the record, rather than something they have to take on faith.
That’s a different, and I think much more defensible, promise than “trust that our people are disciplined.” It’s closer to “here’s what you’d actually be able to audit.”
There’s a related question worth answering plainly too, because it’s the kind of thing a sharp reader will ask on their own if I don’t. If every session produces a charter and a debrief, isn’t a record of twenty sessions a sprint just the heavy documentation overhead of Waterfall, dressed up in charter language? I don’t think so, and the difference is worth stating precisely rather than assumed. Waterfall’s documentation was prescriptive, written before anyone knew anything, an attempt to specify every scenario in advance because the whole model assumed correctness could be planned rather than discovered. A charter is descriptive. It’s one to three lines, written to capture what’s actually being investigated, and the debrief records only what was actually found, not what might theoretically happen. One kind of document tries to predict reality before it happens. The other records reality after it’s already been observed. Those aren’t the same weight, and there’s real evidence behind the difference, not just argument… the same Helsinki research already cited in this series measured design effort directly, and found exploratory testing required less of it, not more. The honest caveat is the same one from a moment ago. An organization that turns a lightweight debrief into a heavy sign-off process, exhaustive templates, an approval gate before a session can even start, will absolutely recreate Waterfall under new branding. That was never a property of exploratory testing itself. It’s a property of what an organization does to any lightweight process left unguarded long enough.
I said in the last piece that something has shifted, and that the old argument for cutting this work quietly stops holding. I meant that. But I wanted to make sure the ground under that claim was solid first, because I think a lot of the conversation happening right now about AI and testing skips this step entirely. People are eager to talk about what artificial intelligence can automate, and much less eager to first agree on what the thing being automated actually is, and what would actually stop it from quietly rotting back into the wandering it’s meant to replace.
That’s a mistake, and I think it’s a costly one. If you don’t have a precise definition of exploratory testing, and a precise answer for how its governance actually holds under pressure, then you can’t tell the difference between a tool that genuinely reinforces that governance and a tool that’s just crawling your application very quickly and calling the map it draws “exploration.” Those are not the same thing, and the distinction matters more than almost anything else in this series.
That’s where we’re headed next.
]]>Then a new release shipped, and the checkboxes were simply gone.
I filed it as a regression. Development pushed back immediately, it wasn’t a defect at all, it was a deliberate simplification. In their view, those toggles were visual clutter confusing to ordinary users, so they’d swept the interface clean and deprecated the execution hooks underneath along with it. When I laid out how enterprise implementations were actually chaining these modules together in production, the engineering team was genuinely stunned. Nobody had been careless, and nobody had been malicious. The engineers had been solving for a clean, theoretical user journey. The practitioners were living inside an entirely different operational reality, one nobody had ever asked them about.
They eventually restored the capability, uncommenting the underlying logic and exposing a clunky programmatic workaround, but the clean toggles never came back. The real breakdown here was never a coding error. It was two groups operating for months on completely different assumptions about what the software was actually for, and discovering the unwritten contract between them only after someone had already broken it.
Elisabeth Hendrickson describes something structurally identical in her own book, a tester who kept finding what looked like bugs in how a parser handled malformed HTML, filed session after session, fixed without argument, until a developer finally objected that nobody had ever specified the parser needed to handle invalid HTML at all. Same failure, running in the opposite direction, a tester over-attributing meaning to unspecified behavior instead of engineers under-attributing it. I don’t think that’s a coincidence. I think it’s the same missing step, showing up on both sides of the same kind of silence.
Ron Jeffries, one of the people who signed the Agile Manifesto, has a blunter line for the tester-side version of this: I wish you testers would stop making stuff up. I understand the frustration behind it, and I think it’s aimed at something real. But my own story runs the other direction, and I don’t think the fix, either direction, is anyone making up less or assuming more confidently. I think the fix is making sure “what this is actually for” gets decided, on purpose, by someone with the standing to decide it, before anyone, tester or engineer, has to guess.
Quality isn’t a property sitting inside the codebase, waiting to be measured against some fixed standard. It’s a runtime contract, negotiated between user intent and architectural assumptions, and like any contract nobody wrote down, it can be broken by either side without either side meaning to.
Henrik Andersson has a useful name for the negotiated half of that: company quality words, an organization-specific vocabulary a practitioner has to go elicit, not read off a generic checklist. Should a lifecycle toggle stay in an interface because real practitioners quietly built production patterns around it, or come out because it looks, to whoever’s designing the next release, like clutter nobody needs? There’s no fact of the matter sitting inside the code. Removing it assumes an answer nobody gave. Insisting it can never change assumes an answer nobody approved either.
Most friction between development and QA over “that’s not a real bug” isn’t really an argument about the software. It’s a collision over a decision neither side realized they hadn’t made yet.
Elisabeth Hendrickson tells a story that sharpens this further, and it doesn’t involve a bug at all. A product manager and a lead developer had been discussing a feature for weeks, separately, each developing their own confident understanding of what it should do. Asked individually how to test it, they both gave the same deflecting answer, just check it against the spec. It wasn’t until someone got both of them in the same room, at a whiteboard, and asked the developer to use the feature as actually built to run one of the product manager’s own example scenarios, that the gap became visible. He couldn’t do it. Neither of them had known, until that exact moment, that they’d been building toward two different things the entire time.
Nothing was broken, technically. Two people had simply never checked their private definitions against the same concrete example. Same failure as the checkbox story, wearing different clothes, two definitions of correct, quietly diverging, with nothing forcing them to reconcile until the gap became impossible to ignore.
Stakeholders rarely announce their priorities directly. They surface sideways, in what people complain about and what they’re quietly proud of, in the war story that still gets brought up two years later, in the post-incident language a company used to apologize to its own customers, the rare moment an organization is forced to state, in plain English, what quality meant to them at the exact instant they failed to deliver it.
The words themselves matter, not just the idea behind them. If three people independently reach for the phrase “silent failure,” that’s not a coincidence worth paraphrasing away. That’s the organization’s actual vocabulary, worth capturing verbatim rather than translated into something that sounds more like a requirements document.
Priorities drift. A quality word load-bearing eighteen months ago might have quietly stopped mattering, and a static list captured once will miss it. This is a live, current gap, not a hypothetical one, I’ve watched practitioners online ask the honest question, a risk profile from six months ago can miss what matters today, how do you keep yours current, and watched the thread move on without really answering it.
The honest answer is that this has to be a recurring conversation, not a document, revisited on a real cadence rather than filed away after one interview. There’s a test worth applying that has nothing to do with software at all: whether something someone tells you matters is a should or a must. Plenty of things get nodded along to in a room without ever crossing into a priority anyone’s actually willing to act on. That gap is usually where the real answer lives.
None of this is abstract philosophy sitting apart from the practical work. Every quality word gathered this way becomes raw material for the next piece of this series, the actual charters that get run, the specific things worth spending a session’s attention on. A generic industry checklist can tell you what a well-built system usually handles gracefully. Only this can tell you what this one is actually going to be judged against.
That’s the piece worth carrying forward. Not “what would a well-designed system do here,” but “what has this organization already decided broken means,” and building the exploration around that instead of around a guess.
]]>Somewhere in the last decade, “regression testing” stopped meaning one thing and started meaning whatever the speaker needed it to mean. Ask a build engineer and you’ll get a pipeline stage. Ask a manager and you’ll get a line item in a sprint plan. Ask a tester steeped in Rapid Software Testing and you’ll get something closer to a stance… any testing motivated by the risk of change to a previously working product, whatever form that testing takes.
That last definition is the more honest one. It’s also the one I want to push on a little, not because it’s wrong, but because I think it quietly folds three different things into a single drawer, and the drawer’s label stops telling you what’s actually inside.
Start with what regression testing is actually checking. Something that already worked is supposed to keep working. The baseline exists. The question is whether the product has drifted away from it.
Primary testing, by contrast, doesn’t have a baseline to hold against. It’s aimed at something novel, or something changed enough that the old baseline no longer applies… new functionality, a modified workflow, a requirement that got refined mid-sprint. You’re not asking “did this stay the same.” You’re asking “does this now do what it’s supposed to do.” Whether that validation happens by hand, by script, or by an agent running through scenarios doesn’t change what’s being asked. It’s a different question aimed at a different target, and I don’t think motivation alone is enough to collapse it into the same category as regression.
This matters more than it sounds like it should, because of how we actually label testing work once it’s done. We don’t label an activity by what the tester meant to find when they sat down. We label it by what it produced. A check earns its name because it produces a binary, comparable result… pass or fail, green or red, matched or mismatched. Primary testing earns its name because it produces something else entirely… a mental model, a bug report, a new question nobody had thought to ask, evidence that a design intent was or wasn’t met.
If you define regression testing purely by motivation, “any test aimed at change-related risk,” you get a definition that’s true and also strangely unmeasurable. Motivation is invisible. Nobody reports “I was motivated by regression risk” in a sprint retro. They report what came out the other end. And what comes out the other end of a regression check looks nothing like what comes out the other end of primary testing, even when both were triggered by the same commit.
Here’s where I think the tidy four-way split, primary testing, retesting, check development, scripted checking, starts to wobble a little.
Retesting isn’t a peer to the other three. It’s a response.
Look at when retesting actually happens. It happens because a build cycle needs to reconfirm that prior coverage still holds after another iteration landed. It happens because deterministic checks are fragile… flaky results, environment drift, a framework quietly self-healing something that deserved a human’s attention instead. It happens because a bug got fixed and someone needs to verify the fix actually took, or reproduce the original report one more time to be sure.
Strip away the surface differences and those three triggers are the same mechanism wearing different hats. Something produced a result. The result needs confirming. That’s a feedback loop, not a discipline of its own. It doesn’t have its own target the way regression and primary testing do. It exists downstream of them, called into service whenever their outputs need a second look.
Putting retesting on the same shelf as primary testing and check development makes for a clean four-box diagram. But it also implies retesting has its own reason for existing, independent of what triggered it. I don’t think it does. I think it’s the loop that connects the other activities to each other, not a fourth activity standing beside them.
None of this is an argument for narrowing regression testing back down to “run the automated suite.” That version was always too small, and the people who’ve made that argument well have made it convincingly. Automated checks catch the easy regressions and nothing else, and pretending otherwise is how teams get burned by a green suite sitting on top of a silently broken deployment pipeline.
The argument here is smaller and, I think, more practical. If we define regression testing by motivation alone, we lose the axis we actually report against. Someone eventually has to answer “what did regression testing produce this cycle.” That answer is easier to give, and easier to defend in a planning meeting, when the labels track outputs instead of intentions. Checks produced a pass/fail signal. Primary testing produced a model and maybe a bug. Retesting closed a loop that something else opened.
Keep the targets separate. Keep the loop visible as a loop. The risk of change doesn’t need one drawer to live in. It needs a shelf, with the items on it labeled for what they actually gave you back.
]]>