AI Companies Want Everyone to Slow Down. Jensen Huang Thinks That’s Odd.

There is a strange debate happening among the people building the world's most powerful AI systems. Some of the biggest names in AI are warning that development may be moving too quickly and that companies need stronger safety rules, independent evaluations and, potentially, coordinated slowdowns.

Anthropic CEO Dario Amodei has been among the loudest voices calling for deliberately pacing frontier AI development. OpenAI's Sam Altman has backed the general idea that additional safety measures may be necessary as AI becomes increasingly powerful. AP has a good breakdown of Amodei's warning and the safety measures he's proposing.

NVIDIA CEO Jensen Huang sees the situation a little differently. Huang argues that choosing between AI safety and AI development speed is a false choice. Companies can pursue both at the same time, and individual AI companies already have a responsibility to slow themselves down if they aren't confident in what they're about to release. Forbes covered Huang and Mark Zuckerberg's pushback against the proposed industry-wide slowdown.

And Huang raises an interesting question. If you think what you're building could be dangerous, why do you need everyone else to agree to slow down before you do?

If You Think It's Dangerous, Don't Release It

Huang isn't arguing that AI can't be dangerous or that companies shouldn't worry about safety. His argument is more about responsibility.

If an AI company believes its next model isn't safe, it doesn't need every other AI company to agree before it slows down. If Anthropic believes it needs another six months to test Claude, take six months. If OpenAI believes a new model presents unacceptable risks, don't ship it.

Mark Zuckerberg has made essentially the same argument. Meta delayed its Muse AI personal agent for several months over safety concerns without first asking competing AI companies to agree to slow their own development. AP covered Zuckerberg's response to the proposed coordinated AI slowdown.

Of course, there is a pretty obvious counterargument. What happens when your competitor doesn't stop?

If one company pauses for six months while everyone else keeps going, it could lose customers, developers, employees and potentially billions of dollars. That creates a classic prisoner's dilemma where everyone might be safer if everyone slows down, but nobody wants to be the only company hitting the brakes.

That's one reason companies such as Anthropic have argued for common safety standards. There is another side to regulation, however, that doesn't get nearly as much attention.

Regulation Can Become a Moat

Right now, building frontier AI is insanely expensive, and that alone protects the companies already at the top. You need enormous amounts of computing power, expensive GPUs, huge data centers, researchers, electricity and, of course, a mountain of cash.

But technology has a funny habit of getting cheaper. The computer sitting on your desk today would have required a room full of equipment decades ago. Storage that once cost thousands of dollars now costs pocket change. There is no reason to assume AI compute will somehow be immune to that same progression.

Better chips, better algorithms, smaller models and more efficient training could eventually make extremely capable AI available to companies that could never afford to compete today. In theory, that should be great for competition.

But what if, by then, we've built another barrier?

Suppose launching a powerful AI model eventually requires independent safety audits, specialized compliance teams, extensive reporting, cybersecurity certifications, liability insurance and regular interaction with government regulators. Microsoft, Google, Meta, OpenAI and Anthropic can afford all of that without breaking a sweat. A couple of smart people working out of a garage probably can't.

Suddenly the expensive GPUs aren't the only moat anymore. Compliance is.

To be clear, I'm not suggesting AI companies are pushing safety regulation because they secretly want to eliminate future competitors. There isn't evidence to make that claim, and there are perfectly legitimate reasons to regulate technology capable of causing serious harm.

But regulation has a funny habit of producing unintended consequences too, and one of those can be making life considerably easier for the companies already sitting at the top. A company large enough to absorb millions of dollars in compliance costs may barely notice them. The startup trying to compete with that company certainly will.

NVIDIA Has a Different Idea

This is where NVIDIA's latest announcement gets especially interesting.

NVIDIA has launched its Open Agent Safety Platform, which takes a somewhat different approach to the problem. Instead of depending entirely on an AI agent to follow the rules, NVIDIA wants to put hard technical restrictions around what the agent can actually do. VentureBeat has a good independent look at the technology and the basic idea behind it: if AI agents can't always be trusted to police themselves, the infrastructure around them needs to do it.

Its OpenShell technology essentially creates a sandbox around an AI agent. NVIDIA describes it as an open-source runtime that controls which systems, files, credentials and services an agent is permitted to access. NVIDIA has a much more technical explanation of how OpenShell's runtime controls work here.

Think about the difference this way. Instead of telling an AI agent, "Please don't access that server," the system simply makes it impossible for the agent to access that server.

That's important because an AI agent could be tricked by prompt injection, misunderstand its instructions, encounter something its developers never anticipated or simply decide that getting around an obstacle is the best way to accomplish the job it was given.

And if that sounds familiar, it should. We've been dealing with essentially the same problem in computer security for decades. You don't ask potentially dangerous software to please behave itself. You restrict its permissions, sandbox it, monitor what it's doing and stop it when it tries to do something it isn't supposed to do.

NVIDIA is basically proposing that we start treating autonomous AI agents the same way.

The company is also proposing an independent hardware watchdog called Sentry. Running separately from the agent workload, Sentry is designed to monitor what an agent is doing and quarantine it when it attempts to move outside its defined boundaries.

In other words, don't rely exclusively on the intelligence inside the box to keep itself inside the box. Build a better box.

But That Doesn't Solve Everything

There is an obvious problem with NVIDIA's approach too. Someone still has to decide where the walls of that box go.

If you give an AI agent permission to delete a database and it mistakenly decides deleting that database is necessary to accomplish its assignment, the sandbox isn't necessarily going to save you. You authorized it to do exactly what it just did.

Security permissions have always had this problem. A firewall can't protect you from traffic you deliberately allowed through the firewall.

So NVIDIA hasn't "solved rogue AI," despite some of the headlines you may see. What it has proposed is another layer of defense, and that's probably a much more realistic description of where AI safety is headed.

Maybe Nobody Needs to Be Completely Right

Anthropic's Dario Amodei is worried that competition could push companies into deploying increasingly powerful AI before they completely understand the risks. That's not an unreasonable concern.

Jensen Huang is looking at the same situation from another direction. If companies genuinely believe their products could become dangerous, why do they need their competitors to agree to slow down before they'll behave responsibly?

That's not an unreasonable question either.

NVIDIA is now putting some technology behind Huang's argument. Keep developing AI, but don't assume the AI will always follow the rules. Put boundaries around it that it can't negotiate with, reason around or simply ignore.

There is also a much larger issue worth watching here. Today's enormous cost of building frontier AI gives a handful of companies a natural advantage. If those costs eventually collapse, that advantage should begin disappearing and smaller companies should have a much better opportunity to compete.

But if we've spent the intervening years constructing an enormously expensive regulatory system around AI, we may discover that we've simply replaced one moat with another.

We've already seen versions of this in tech. Europe's GDPR was designed to protect people's privacy, but the cost and complexity of compliance hit smaller publishers, ad companies and startups much harder than giants like Google and Meta. GDPR wasn't created to help Google and Meta, but it demonstrated one of the unintended consequences of complex regulation: companies with armies of lawyers, engineers and compliance people are much better equipped to absorb it than smaller competitors.

The fact is, GPUs and compute will get cheaper. The Government paperwork won’t. The AI giants may still be the only ones wealthy enough to cross that gap, which makes me watch this fight with a very cynical eye.

Jim McMahon Jim McMahon, aka Corporal Punishment, is the founder of MajorGeeks.com. He has spent decades testing software, troubleshooting Windows, and helping users cut through the nonsense. He loves real freeware, hates bloatware, and runs on caffeine, sarcasm, and questionable choices.

comments powered by Disqus