So there I was, face to the whiteboard, deep in a market segmentation exercise. Segmentation is the domain of fuzzy logic, where a team’s patience and imagination are eventually rewarded with a crisp new market planning model.
We asked the AI clarifying questions, to test our assumptions, and so it replied in the perfectly structured and certain tone of an LLM. In the midst of the intellectual fog of the exercise, I was amazed and energised by how crisp and sure its answers were. I could feel a weight being lifted off the team as our questions were responded to with apparent fact.
It felt like we nailed it.
The danger of confident answers
Now, you can probably guess what was going on here, because I think we’ve all been there by now.
In the middle of this genuinely hard task, we were being offered an easy way out. A short cut. The answers, so clear and professionally expressed, didn’t invite doubt in any way. So as we lifted this human-AI cadence throughout the day, it didn’t occur to us that the AI might be mistaken in any way. And boy, was it wrong! Every answer that we didn’t challenge took us incrementally further away from the reality that we were seeking to understand.
It wasn’t the hallucinations
A year on, I think we’ve all experienced enough AI hallucinations to work with the tools in a different way. But the lesson from that day has stuck with me and neatly expresses the challenge we face as strategic problem solvers.
In market positioning we’re always working with hypothetical situations. It’s the OG of fuzzy logic in business planning, where we try to predict future outcomes using wildly disparate and disconnected sources of data or insight.
Hence my issue wasn’t that the facts got a bit jumbled by AI. It was that we didn’t bother to question them. And I want to really work out why. Because it’s not just the confidence of an LLM that does this to us. The confidence bestowed by hierarchy or personality are equally likely to lead us astray in the same way.
Confidence is not competence
When somebody says something strongly, when was the last time that you questioned the source? “Why exactly do you know that is true?” might be the most under-used question in business.
Now the psychologists amongst you will already be calling out why. The human brain, just like an LLM, seeks out patterns as it enables more energy efficient thinking. So when something is told clearly and definitively to us, it offers subconscious relief to our energy hungry brain. Our mind stops searching once we have an answer that fits our pattern or simply feels plausible enough for now.
Of course, this is severely compounded by our Authority bias, which nudges us to react more positively to senior figures in our social hierarchy. We are all part of this. We subconsciously protect our own status by making strong, authoritative statements that are unlikely to be challenged. We’re hard wired to enjoy being right, yet competence and confidence are not the same thing.
Beliefs masquerading as insight
