Strategy & experimentation
Video · 5 min read
The short answer
Watch: What is different about AI now, and how should that change the way leaders think about strategy?
Watch the explanation, then explore the practical implications below.
Treat an earlier AI test as evidence about a particular setup
A disappointing result can be a sound reason not to use a system for a task at that time. It is less useful as a permanent conclusion about every future system or workflow. Record what was tested, what information was supplied and what made the result unacceptable.
The video focuses on two developments leaders should examine: changing capabilities and the ability to assign more of a process to an agent. Neither means that every task is ready for delegation. Each requires evidence about the work your business actually needs done.
Distinguish answers from completed work
Summarising a document is different from reviewing several documents, identifying gaps and preparing questions for an owner. The second task includes intermediate choices about relevance, completeness and when further input is needed.
Assess those choices as part of the workflow. A system that produces a fluent final answer may still select the wrong source, overlook a missing document or continue when it should stop. Test the sequence and its boundaries, not just the last paragraph.
Understand why the previous attempt failed
Separate problems in capability from problems in the brief, source information, access or process design. If the system never received the approved specification, a new model alone will not establish which version should govern the work.
Conversely, a carefully designed process may still contain a task the chosen system cannot perform reliably enough. Keep those findings specific. That makes it possible to retest the relevant assumption without discarding useful lessons or repeating the entire pilot.
An example: revisiting document review
Imagine a business previously rejected AI-assisted review because important omissions were missed. Keep a small set of representative cases, including the omissions that matter and examples with conflicting or incomplete evidence. Record the expected checks and the review required before use.
When there is a reason to revisit the task, run the new setup against those cases under comparable conditions. Also include fresh cases, so the assessment is not limited to familiar examples. Measure missed issues, false alarms and reviewer effort. A more confident explanation is not a substitute for better results.
Make reassessment a bounded management activity
Keep a simple record of tested opportunities, failure causes, current constraints and an owner. Decide what should trigger a review: a material change in the system, improved source data, a changed business need or evidence from another controlled test.
There is no need to chase every new release. Focus on assumptions that affect an important business decision and can be tested economically. Compare the expected value of another test with the cost of leaving the workflow unchanged and the effort required to maintain it.
Change delegation only when the evidence supports it
Improved performance on one task should not silently widen permissions or remove approvals elsewhere. Define what the system may prepare, what it may execute and what still requires human authority. Test recovery and exceptions before extending live use.
At CSM, we would revisit the original business objective and use real work to evaluate the current setup. The outcome may be a useful implementation, a narrower task or a clear reason to wait. This keeps strategy responsive to evidence while preserving accountability for the decisions and commitments that follow.
Questions for your leadership team
Which AI assumptions are shaping our decisions?
Identify where an earlier test has become an unexamined limit on current plans.
What exactly failed in that test?
Separate model capability from missing context, poor evidence, access and workflow design.
Do we have representative cases to test again?
Include known difficult examples and fresh work, with clear review criteria.
What would justify revisiting the decision?
Agree a meaningful trigger and an owner rather than following every announcement.
What evidence is needed before expanding responsibility?
Assess quality, missed issues, review effort and controls before changing delegation.
Full video transcript
The full video transcript is included below for reference.
I think there are two things to pay attention to. One is that the capabilities keep moving. The second is that agents change the kind of work you can give these systems. You’re moving from asking for an answer towards asking for a piece of work to be carried out.
That’s an important change. If you ask AI to summarise a document, you’re still deciding every next step. If you ask it to review a set of documents, identify what’s missing and prepare the questions that need answering, you’re giving it responsibility for more of the process.
Now, that doesn’t mean it will get everything right. You still need to understand what it can do reliably, what information it needs and when it should come back to you. But it does change the question a leader should be asking.
Because a lot of organisations are still making decisions based on their first experience of AI. Someone tried it, got a disappointing answer and decided it wasn’t ready. That might have been a perfectly fair judgement about that task at that time. It isn’t a permanent assessment.
So I think you need a way of revisiting those judgements. What have we tested? What failed? Was it the model, the information we gave it or the way we set up the work?
Otherwise, you can have a sensible strategy built around a limitation that no longer applies. And nobody notices, because the organisation remembers the decision but has stopped checking the assumption behind it.
Put these questions to work in your business
Bring a real decision or workflow to a conversation with CSM. We can explore the evidence, responsibilities and practical steps needed to make AI useful for your team.