the models are only as good as the input. So. if you're moving quickly, is there a potential you're
missing inputs that could be disastrous down the line?
14:55
Michael Kratsios
You bring up a very good point. So, I mentioned this a little bit earlier, but I can't stress that enough.
The most important aspect of enterprise-grade large language model deployments is a robust
testing and evaluation regime. So, there are lots of vendors in the space, lots of people who are sort
of working in this. How do you fine tune a model to particular data of a company? And the most
important thing is, while you can sort of throw some data at it and hope that it works out, we've all
experienced moments when we've asked ChatGPT a question and it's hallucinated and given us an
answer that we never imagined possible or don't understand where it came from. And in most
enterprise use cases, that is pretty unacceptable most of the time.
So there has to be a lot of work on this test evaluation piece. That's something we think about.
That's something that the industry is trying to perfect the moment, and we may get to this in a bit,
but you even saw in President Biden's executive order last Monday, there was a lot of talk around
building standardized methodologies for testing and evaluation that the government can work on.
So, I think everyone recognizes that no matter where you deploy these things, having a standardized
way to think about test and eval is front and center important.
16:01
Amna Nawaz
And I want to get to the regulation question in just a moment because I think there's a whole
different conversation around that. But Simon, I want to get your take on what Michael just said,
especially about some of the concerns, potential harms is, of course, a willingness and a desire to
move quickly to do what's best for your business as a CEO. But how are they thinking about the
potential harms, the fact that employment, as you said, redistribution could be happening? Where
are those future jobs? Who are they looking to hire and bring on board to make sure that those
potential harms and disastrous consequences from not having the right inputs don't happen?
16:36
Simon Freakley
Right. Well, what I found just in my many conversations with client CEOs is that the very best leaders
are just really getting to the nub of the key questions for their business and their industry. What are
the key questions they need to answer to be able to make sure that their business remains not just
relevant in the future, but a high performing business in the future? What is the nub of the key
questions that they need to be in a position to answer to work out what they should do in terms of
next steps as they develop this technology within their business? And of course, most chief
executives, because of the tenure where they're at in their careers to be chief executive, haven't got
a digital background. Some have, but many don't. They're not digital natives. I mean, they work very
hard indeed to understand enough to ask relevant questions, but the very best leaders put people in
the room with them that really are digital natives and understand some of the key questions to
challenge the leaders with.
So, assembling the right team to be able to get to the nub of these questions is key. And then
working out from an investment profile the type of investments that are not just desirable but are
also affordable. I personally think that every corporation, major corporation, will have its own large