Big companies want AI they can explain to regulators

Nikesh Arora, the chief executive of Palo Alto Networks, says the question that matters most to a bank or hospital is whether it can explain what the AI just did.

That is why his company prefers open source, or open weight, models. Open weight means the company can download the underlying settings of the model, adjust them, and retrain the system on its own data. More important, he says, it can interrogate the chain of thought, the step by step reasoning the model used to reach an answer.

Large enterprises will need what Arora calls explainability, auditability, and verifiability. To make the point, he offered an example of what will not pass in regulated businesses like financial services and health care: going to your boss and saying, "I don't know what happened, but the IT guys put an AI model in there. It did it." Arora says only open models currently let his team pull apart that reasoning and prove what happened.

That preference creates a second problem. To use an open model, a company needs someone to run it reliably all over the world, a service called inference. Arora says it is difficult to locate a worldwide, resilient, and highly available inference provider outside the big cloud companies.

In Arora's example, a startup can rent computing power in a single facility in one region of the United States, and live with an outage of a few hours. An enterprise that promises service that works 99.9 percent of the time cannot. He says providers that can deliver that kind of global reliability want to sell him their own model, rather than run an open model at a price that makes using open source economically useful.

So big companies are caught in the middle. They want open models they can audit, but they still need the giant computer networks that tend to come with closed ones.