language model, probably several large language models, to do different things, that those models
are unlikely to be developed from scratch by them. They'll probably take a proprietary foundation
model and then adjust it, make it bespoke to their enterprise, train that model on their own data so
they can synthesize both information from public source and information behind their firewall. And
that's what I find most leaders are doing at the moment, working out how to navigate to that
position, how to do it safely with several choices, which would be the best choice for them, but
putting people in the room with them, that will likely enable them to get to the right answer,
because probably their own experiences are not going to be enough of an insight as they make these
critical choices.
18:42
Amna Nawaz
Simon, I'm just curious, most of the business leaders you've talked to, did they experiment with
ChatGPT and check it out when it was released?
18:51
Simon Freakley
Mostly, yes. I mean, as I chat with people, it's quite interesting to hear some of the stories that they
will use to illustrate the point, some of which, of course, are very much business questions. One I
spoke to recently used ChatGPT to make up bedtime stories for their children. Somebody else was
looking to sail quite a long distance on a boat and asked ChatGPT to tell them the best way to do it,
which was remarkably accurate, as it turned out, when they then compared it to a maritime service.
And so, people are experimenting, I think, to learn how to use it. And I think that as they experiment,
they realize that it's not like a Google search, that actually you can ask some really quite
sophisticated relative questions, not just absolute questions. And I think that this interaction that
people are having are helping them develop a frame of reference, but understanding they'll never be
an expert, and they need to bring the experts into the room to help them understand the real
questions they need to be asking.
19:50
Amna Nawaz
So, Michael, for those leaders you talked about fine tuning, rather Simon mentioned, starting with
some of the open source models and then fine tuning them to their own businesses and needs. How
does a company know they've reached a level of confidence that it's ready to be rolled out, it's ready
to be implemented? How should they be thinking about that?
20:11
Michael Kratsios
Yeah, I think it goes back to this test eval sort of question, and I think what we typically advise and
try to work with customers out of the gate is on sort of non-mission critical solutions. So, the first
implementation doesn't need to be something that is sort of like do or die for your business. You
have to be in an environment where you can imagine that the end user, the LLM, probably the
company will be fine if some hallucinations come through and those can be evaluated and
monitored and recorded, and the model can just improve over time. So, I think for us, there's a
battery of tests you can do that allow you to kind of check the performance of the model. You
obviously can have it running for a certain amount of time. And what we found is that the greatest
experts are actually people in the company themselves, in the customer themselves, that are using
this model every day, and they can provide extraordinarily valuable feedback on where the model is