---
title: "Can We Really Make AI Safe?"
date: 2026-09-10
readTime: 10
tags: ["ai-safety","agents","cybersecurity","architecture"]
description: "A powerful model becomes a safety problem when it can act. Safer systems limit what a mistake can do."
canonical: https://abubakarsiddik.site/blog/can-we-really-make-ai-safe
---

# Can We Really Make AI Safe?


## Can we really make AI safe?

I have been thinking about this question a lot lately.

There has been a viral AI-safety post this week, from former OpenAI and Anthropic researcher [Jacob Coxon](https://x.com/hilbertspaess/status/2097476203863224394). His post is an insider's warning, not proof that a particular disaster is about to happen. Still, it puts a serious question in front of people who normally do not spend their days reading model cards or safety papers: what are we building, and how much control do we really have over it?

I have been asking myself the same thing while working on my recent [review of cyber-capable AI agents](https://arxiv.org/abs/2607.25379).

Can we really make AI safe when we do not fully understand why a model gives one answer instead of another, and we are steadily giving it more access to the world?

I do not mean the old chatbot that gives a wrong answer about a film or writes an awkward email. I mean an AI agent with a browser, a terminal, cloud credentials, access to work files, and permission to make changes. The kind of system that can read a ticket, open a web page, run a command, restart a service, or connect to a remote machine over SSH.

That is where this stops feeling like a distant debate about artificial intelligence and starts feeling like an engineering problem we are already creating.

## The safety layer is not the foundation

Before I get into what could happen if AI starts helping build the next AI, I want to talk about what is already true today. This is not a problem waiting in the future. It is why I think the danger Jacob is warning about is not just a hypothetical.

I want to say something here that comes from my own research, not from a lab's report.

My master's work was on emergent misalignment in LLMs, and the finding that stuck with me is this: if you fine-tune a model on a narrow harmful task, something as specific as writing insecure code on purpose, the model does not just get worse at that one task. It gets worse in places that have nothing to do with it. Ask it something unrelated afterward and the answers come back worse too. That is not a guess about how models might behave. It has been proven by the researchers.

What that tells me is that the polite, safe-sounding model we interact with is not built the way it looks. Pretraining comes first, on tens of trillions of tokens pulled from the internet, and the internet is not curated for good behavior. It is mostly whatever people happened to write, technical knowledge next to manipulation next to things nobody should be teaching a model. Only after that do we spend a comparatively small amount of effort teaching the model to be polite and to refuse bad requests.

That second stage is not removing what the model learned in the first one. It is suppressing it. And a suppressed behavior is still in there. When someone fine-tunes the model again, even on a small, narrow, unrelated bad dataset, that suppression does not hold, and what comes back out is not new. It is the pretraining knowledge getting reinforced and let back through.

You do not have to take my word for it. Go look at Hugging Face. Every few weeks someone posts a jailbroken version of an open model where the safety behavior has been fine-tuned off, sometimes with a small dataset and very little effort. That is what you would expect if safety is a layer sitting on top rather than something the model actually internalized.

I am not going to pretend I have the fix for this. I do not think a better system prompt or a smarter filter on top solves it. I think the honest position is that we do not understand pretraining well enough to know what we are actually suppressing when we do safety training, and until that changes, every safety-trained model should be treated as one fine-tune away from losing it.

![Safety training sits as a thin layer on top of pretraining, and a narrow bad fine-tune can reach straight through it](../images/ai-safety-layer-stack.svg)

## The part of the model we cannot simply inspect

There is another reason this is hard. LLMs are not like ordinary programs.

With normal software, we can often follow the code. If a calculation is wrong, we can inspect the rule that produced it. With a large language model, especially one with hundreds of billions or trillions of parameters, the behaviour comes from training across a huge number of parameters, and nobody can point to the neuron or the weight responsible for a given answer. We can test it, measure patterns, and make it behave better. We still cannot look inside a frontier model and confidently explain every decision it will make in every new situation.

That does not mean models are mystical or impossible to control. It means we should be honest about the kind of control we have. We have useful controls. We do not have a complete guarantee.

This is why I get worried when safety is described as one final step after a model has become powerful. A lab trains a model on an enormous collection of text, code, explanations, forums, books, and documents. Then it teaches the model to be helpful, polite, and to refuse harmful requests.

That second stage matters. It changes behaviour. It makes models safer and more useful in many real interactions. But it should not be treated as a certificate saying every dangerous capability has disappeared.

I think of it like this. Imagine someone who learns from a library so large that no person could read it in a lifetime. The library contains good material, bad material, technical knowledge, manipulation, and things that should never be used. Later, a teacher trains that person to be helpful and to behave responsibly, and it works. They become polite, careful, and safe to be around.

Then a second teacher shows up and retrains them on one narrow, unrelated task, something small like writing sloppy, insecure code on purpose. That narrow retraining does not stay narrow. It reaches back into everything the first teacher tried to put away, and the person comes out of it worse across the board, not just at the one thing they were retrained on. The bad material they learned from the library years ago was never gone. It was sitting underneath the good behavior the whole time, and the second round of training was enough to bring it back out.

Once we accept that, we should not rely only on the first teacher's training holding forever. We should also decide which doors this person's keys open, who can see what they do with them, and how quickly we can take the keys back.

The analogy is imperfect. Models do not learn like people. It is still useful because it shows why training and access control have to work together.

This is the situation today: a system we cannot fully inspect, running on a safety layer that suppresses rather than removes what pretraining put there. Now consider where this goes next, if AI itself starts helping build the AI that comes after it.

## What happens when AI helps build the next AI?

Today's public models have not demonstrated that they can independently run the whole process of improving themselves. We should not write as though that has already happened, and we cannot responsibly claim to know what any lab has achieved privately.

The question still needs to be asked now, before the answer becomes obvious.

Models already help researchers write code, inspect experiments, search large bodies of knowledge, and speed up ordinary engineering work. If they become good enough to make AI research and development move much faster, the safety problem changes pace. A model would not need to wake up with a science-fiction goal to matter. It could simply help a lab test more ideas, run more experiments, and build stronger systems faster than our ability to evaluate and govern them.

This is not a fringe concern. OpenAI lists AI self-improvement as a tracked category in its Preparedness Framework. Google DeepMind includes machine-learning research and development among the areas it monitors for severe risks. Anthropic's current safety roadmap says it considers it plausible that its systems could fully automate or dramatically accelerate work done by top research teams in some sensitive fields as early as 2027. These are the labs' own frameworks and forecasts, not proof that the threshold has already been crossed. [OpenAI's Preparedness Framework](https://openai.com/index/updating-our-preparedness-framework/), [Google DeepMind's Frontier Safety Framework](https://deepmind.google/frontier-safety/), and [Anthropic's safety roadmap](https://www.anthropic.com/responsible-scaling-policy/roadmap) all make that direction visible.

That is why safety cannot wait until after capability arrives. If a model is close to accelerating the work that creates its successor, the evaluation, security, and governance around it need to be ready before the next jump in capability.

## The moment a model can act, the question changes

Imagine an agent that helps an engineer keep a server running. It can inspect logs, use the terminal, connect to a remote machine, and make a small change when it thinks a service is failing. That sounds useful. In many cases, it will be useful.

Now imagine that the agent searches the web for a solution. It opens a page that looks like normal documentation, but someone has placed hidden text there to manipulate an AI reader. The page tells the agent that it needs to run a diagnostic, change a setting, or share information to complete the task. The agent may not see that as an attack. It may see it as part of the page it was asked to read.

This is one version of prompt injection. It can arrive through a web page, an email, a PDF, a support ticket, a shared document, or the output of another tool. The problem gets much more serious when the agent can act on what it reads.

OpenAI makes a similar point in its guidance on prompt injection: trying to identify every malicious sentence is not enough. The system has to limit what manipulated content can cause the agent to do. [Its explanation of how agents should resist prompt injection](https://openai.com/index/designing-agents-to-resist-prompt-injection/) treats this as a security problem with prompt-writing implications.

An AI model can be very capable and still be safe to use in a limited setting. A model that can only answer questions has less room to cause direct harm than the same model with a terminal, a network connection, and broad permissions.

The model has not suddenly become more intelligent in that moment. We have given it more authority.

That distinction matters. A lot of AI safety conversations stay focused on what a model knows. In real systems, the risk also depends on what the model is allowed to touch, how long it can keep trying, what it can delegate, and whether a person can stop it before a mistake spreads.

## Why prompts are not enough

Prompts help. They can tell a model to be cautious, ask for confirmation, or refuse risky requests. We should use them.

A prompt is language inside a system that processes language probabilistically. It guides behaviour; it does not enforce a permission boundary.

If we tell an agent, "Never delete production data," that is a good instruction. It is still not enough protection if the same agent has the database password, a command that can delete data, and a web page or email that convinces it that deletion is part of an urgent repair.

The safe design is simple to say, even if it takes work to build: the model can suggest an action, while normal software decides whether that action is permitted.

The normal software should be able to say:

- this agent can read these logs, but not other customer data;
- it can connect to this test machine, but not a production server;
- it can prepare a change, but a person must approve it before it runs;
- it can use the web for research, but it cannot send secrets or files to a website;
- it has ten minutes and a fixed number of tool calls, then it stops.

This is not exciting. It is the same kind of careful permission design we already use for employees, applications, and cloud services. The name engineers use is _least privilege_: give a system only the access it needs for the job in front of it.

For AI agents, least privilege has to include their tools, credentials, files, browser sessions, network destinations, spending limits, and ability to create more agents. An agent that is allowed to use SSH should not receive a powerful permanent key that works everywhere. It should get a short-lived, restricted credential for one approved environment and one defined task. If it does the wrong thing, the damage should be contained.

## Start safety earlier than the system prompt

I said earlier that pretraining is where the real knowledge sits, and that safety training on top of it suppresses that knowledge rather than removing it. So it is worth asking whether more can be done earlier, in the pretraining data itself.

We should care about where the data came from, whether it contains private information, whether it has been poisoned or manipulated, and whether we are giving a model unnecessary exposure to narrowly hazardous material. This is part of building responsible models. It also gives researchers a chance to reduce dangerous capability before a model reaches users.

There is early evidence that this can help. Anthropic reported that filtering a pre-training corpus for some CBRN-related content reduced performance on a harmful-capability evaluation in its experiment, while preserving performance on the harmless evaluations it selected. The report also makes the important point that filtering is not a complete answer, especially for dual-use knowledge. [Anthropic's pre-training data filtering work](https://alignment.anthropic.com/2025/pretraining-data-filtering/) is worth reading for the limitations as well as the result.

Cybersecurity makes the trade-off clear. A model needs to understand insecure code to help a defender find and fix it. A blanket attempt to remove every piece of cyber knowledge would make defensive work worse, and it still would not guarantee that a capable model could not combine general programming and reasoning knowledge in a harmful way.

So pre-training data curation is one layer. It should reduce unnecessary risk. It cannot carry the whole safety argument by itself.

The same is true of post-training and fine-tuning. People sometimes say that harmful knowledge is sitting inside a model and safety training merely hides it. That is too simple. Post-training can make a real difference to what a model does. At the same time, further fine-tuning, new tools, changed context, or deliberate attacks can change behaviour again.

Research has shown how fragile that boundary can be in experiments. [Qi and colleagues](https://arxiv.org/abs/2310.03693) found that a small amount of adversarial fine-tuning could compromise safety alignment, and that benign fine-tuning could also weaken it to a lesser degree. [Sleeper Agents](https://arxiv.org/abs/2401.05566) showed deliberately created backdoors persisting through several standard safety-training methods. These are warnings from controlled research, not evidence that every deployed model has a hidden bad intention. They tell us that testing one version of a model is not enough when we keep changing the system around it.

## What should happen when an agent has a terminal?

This is the question I keep coming back to.

If an AI agent can run Bash commands, use SSH, browse the web, read internal documents, or control a remote device, we should treat it like a new type of employee with unusual speed and very little common sense. It may be able to work all night. It may follow a strange instruction too literally. It may be manipulated by information it was only supposed to read. It may also be genuinely helpful.

Tools remain useful when their authority matches a narrow task. The work is deciding where autonomy is appropriate.

Here are the directions I think matter most.

### Put a boundary around every tool

Tools should be narrow by default. Reading a log file should not also allow editing a server. Opening a support ticket should not also allow sending an email to every customer. Access to a remote machine should be restricted to an approved environment, with a short-lived credential, a limited command set where possible, and clear records of what happened.

This is capability-based security: access is granted for a specific purpose, not because the agent has a general identity that can do everything.

### Treat the web as untrusted

The web is useful for research. It is also full of content written by people with their own goals, including attackers.

An agent should be able to read a webpage as evidence without treating the webpage as an instruction. The same rule should apply to emails, documents, memory retrieved from a database, and tool output. The original user request and the system's policy are the sources of authority. Everything else is information to evaluate.

In practice, this means separating data from instructions, checking tool calls against the original task, and blocking sensitive actions when the instruction came from untrusted content. A filter that tries to spot malicious wording can help, but it should not be the only defence.

### Keep meaningful actions with people

An agent can draft a deployment plan, prepare a command, identify a likely problem, or propose a change. For actions that are hard to reverse, it should stop and show a person exactly what it wants to do.

Send the email. Delete the record. Publish the post. Change the permission. Deploy the service. Transfer the money. These are different from searching, summarising, or preparing a draft.

OpenAI's computer-use safety work uses this kind of action-specific confirmation. [Its Operator system card](https://openai.com/index/operator-system-card/) discusses human oversight for consequential actions and acknowledges the limits of these measures. That feels much closer to the right model than a blanket instruction saying "be careful."

### Bound the task, including sub-agents

"Improve yourself" sounds small. In an agent system, it can become very large.

What does improvement mean? Which files can change? Can the agent spend money? Can it create sub-agents? Can those sub-agents create more sub-agents? Can any of them reach the internet or a production environment? How long do they keep working if the original task is unclear?

Every agent needs a defined objective, a limited set of tools, a time and cost budget, and a stop condition. A sub-agent should not quietly inherit every permission held by its parent. It should receive only the access needed for its own narrow task. Approval to take a consequential action should stay with a person, not be passed down a chain of agents.

### Build for failure, not perfect behaviour

This is the part I wish was discussed more outside engineering circles.

We do not make airplanes safe by assuming a pilot will never make a mistake. We build checklists, redundant systems, warnings, maintenance procedures, and ways to recover when something goes wrong. AI systems need the same attitude.

An agent should have logs that show what it read, what it planned, which tools it called, and what changed. It should have a kill switch. Its access should be easy to revoke. A harmful change should be reversible where possible. Teams should practise what they will do when an agent behaves unexpectedly, before an incident forces them to learn in public.

Engineers call this defence in depth. One safeguard will fail eventually, so one failure cannot be allowed to become a disaster.

## Capability gates before new access

There is also a decision for AI labs and governments.

As models become more capable in cyber, autonomy, biology, or persuasion, the safeguards around them should become stronger before more access is granted. That means evaluating what a model can do in realistic conditions, not only asking whether it gives a safe answer to a bad prompt.

If a model can make a cyber task much easier, the questions are practical: who gets access, are the model weights protected, what monitoring is in place, which tools are available, and what happens if the system is misused? Frameworks such as [Anthropic's Responsible Scaling Policy](https://www.anthropic.com/responsible-scaling-policy) and [Google DeepMind's Frontier Safety Framework](https://deepmind.google/frontier-safety/) are attempts to connect measured capabilities to stronger safeguards. They are not final answers. They are a more serious starting point than releasing first and discussing safety later.

There is a piece of this that does not get said out loud often enough. What the public gets to use is not the frontier. Every major lab has been open about the fact that it keeps training more capable successors regardless of what is currently deployed, and the model available to download or query today is generally understood to sit a year or two behind what the same lab is already running internally. None of them have said they intend to stop doing that. If evaluation, red-teaming, and governance are all built around the public model, they are being built around something that is already behind the system doing the real work inside the lab. Capability gates only mean something if they apply to what a lab has, not only to what it ships.

![Public releases trail what labs run internally, roughly a year or two by the labs' own account](../images/ai-safety-release-gap.svg)

We also need independent evaluation, clear incident reporting, and room for people inside companies to raise concerns. The viral post that started this conversation is a reminder that safety cannot depend only on the confidence of the people building the system.

## So, can we really make AI safe?

I do not think we can honestly promise a powerful general-purpose model will never surprise us. We do not understand these systems well enough for that, and their real behaviour depends on the environment we place around them.

I do think we can make AI systems much safer than they are when we connect a capable model to everything and hope a prompt holds.

We can start with better training data and better post-training. We can test models after every meaningful change. We can keep their permissions small, treat outside content as untrusted, require approval before they make consequential changes, and keep the ability to stop them quickly.

That is the direction I think we should take.

AI safety includes a model's response to a bad request. It also asks what happens if the model is wrong, manipulated, confused, or more capable than we expected. The more tools we give it, the more that decision belongs to all of us.