August 7, 2025
5 min read
Ed Sperling
Agentic AI introduces complex security challenges in chip design due to opaque decision-making and agent interactions.
Agentic AI: Lots Of Little Black Boxes
Security issues may be magnified by the number of AI agents and their potential interactions, not all of which may be obvious to chipmakers. AI is changing so quickly that it’s not always clear how much of a security threat it poses for semiconductor design, and that uncertainty increases as AI agents are introduced into the mix. So far, the use of AI in chip design has been highly targeted. Most of what is included in design tools is some version of machine learning, bounded by tight control loops. EDA and IP vendors, large chipmakers, and systems companies are researching and experimenting with various new flavors of AI, but they’re still unsure of how much freedom it should be given to accomplish specific and more general tasks. All have said publicly that their use of AI will increase over the next couple of years, with AI agents added into the mix, but exactly how, where, and when it gets implemented is still rather hazy. The reason for utilizing this technology is the sheer complexity of designing, testing, verifying, and manufacturing multi-die assemblies and sub-2nm SoCs. Being able to keep track of all the possible interactions and dependencies, while still delivering a chip on schedule — and one optimized for a particular workload or market segment — is an extraordinary engineering feat. But it’s expensive, labor-intensive, and it requires a team with a mix of engineering disciplines and deep domain knowledge, and there are not enough seasoned veterans to go around. AI agents can help by divvying up various tasks to get results done faster. They can work together or independently on a particular problem, and they can be assigned varying levels of freedom to achieve a goal. In addition, while those differing levels of freedom or abstraction are enabled by hardware, they can reach well beyond a single workstation or even an entire data center. But no matter how effective they are, AI remains largely opaque. Once AI begins working, it’s not obvious how it arrives at its final answer. There are numerous reports and technical papers about bias in data sets that generate erroneous results. On top of that, the training data can contain spyware or sleeper code that may not be immediately obvious. And while agents may be effective in accelerating time to results, they effectively add more black boxes into the mix.“The field is moving so fast that mistakes are going to be made,” said Preeth Chengappa, head of industry for semiconductor and EDA, mission engineering, at Microsoft. “The challenge we have is that we need enterprise-level capability before we send anything out into an enterprise to actually do things. It’s all good when we’re doing travel arrangements or planning a wedding, but when it’s doing a high-end chip design, we want to vet it. That’s when companies like Microsoft or Google or NVIDIA will be able to deploy this in their own environment and validate what they are doing. And companies like Microsoft will encourage this technology to be leveraged by Synopsys, Cadence, and Siemens for chip design itself. But it has to be done in a responsible way. Otherwise, it’s going to be the Wild West.”How quickly this happens is unknown at this point. Despite the massive investments in AI-enabled search and generative AI, using AI for mission-critical or safety-critical applications has no specific timetable yet.
AI Agents as a Security Risk
An adjacent concern is how AI can be used for the wrong purposes. That can happen more quickly than mass adoption, because in those cases, an agentic excursion or misalignment doesn’t matter. AI has taken a few major steps toward usefulness in the past couple of years. The rollout of ChatGPT in November 2023 made it tangible for the first time for anyone with an internet connection. That also raised questions about security, because every connected device is a potential source of new data. Expert systems — a precursor to AI as we know it today — have been in use for decades for things like customer service, financial analysis, and in virtual assistants like Siri and Alexa, but most of that data was well protected. These types of systems are still in use today, and able to create their own stimulus and measure the response based upon a given data set. But the relatively slow improvements in expert systems are dwarfed by the complexity, granularity, and speed of current AI technology, which along with huge advances in data mining, have enabled the current AI frenzy. Agentic AI takes that one giant step further.“Agentic software can be programmed to do anything,” said Scott Best, senior director for silicon security products at Rambus. “It can be programmed to talk to other agents, to initiate new web browser connections to go find new data, establish new connectivity, exchange data, get real-world information. It could be attached to microelectronic equipment that’s looking at a system in real-time on a bench under electrical test. And it could be configured to do something in real-time, including communicating with other parallel agents that are isolated, or use some sort of local or wide-area connection to communicate remotely with another agentic system that’s performing an attack on a similar system — and possibly sharing strategies.”One example of AI agents gone rogue is a simulation run by Anthropic, in which the AI threatened to expose an executive’s extramarital affair if he defunded the AI program. The technical term is “agentic misalignment,” and it shows just how wily AI can be in achieving its goal. In some cases, AI has created its own unique, highly efficient language for communication between agents.
“There are different ways to use agents and to design agents,” said Mike Borza, principal security technologies and scientist at Synopsys. “There are fairly simple agents that are simply given a task and expected to accomplish it, and maybe come back for some feedback or guidance from the user in the form of a response. Those forms seem to pose the least risk. The things the Anthropic report identified really have to do with cooperating agents. These are goal-oriented agents that may be using different AI backends, where it’s essentially a group of backends trying to get some kind of average consensus on the best way to answer a question. But those start to raise concerns about how these behaviors interact with each other, whether they can adapt to each other’s answers or behavior, or form an opportunity for a lot more of this behavior.”This becomes potentially more worrisome when agents become commercially available. When they are used internally, AI vendors can modify them as needed and limit what they do. But how agents will interact with other agents under real workloads that are defined by the end user is potentially much more problematic and needs to be clearly defined.
“Regardless of which tools the AI agent engineer is using, they should still follow the existing privacy policies and security rules assigned to that particular user,” said William Wang, CEO of ChipAgents. “For example, what are the files that this particular user can access? In big companies there are lots of different projects. Sometimes you’re not allowed to go to the repository. Sometimes you’re not allowed to see this project. But whatever privilege is assigned to this user of an agent, the understanding is that the agent should stick with the user’s privilege. It’s not okay for the agent to have a different set of rules than the human. Aligning the agentic AI’s privilege with the user’s privilege is very important.”
Limiting What Agents Can Do
One of the big advantages of using agents is that it allows multiple operations to be performed on whatever architecture makes the most sense. In effect, it overcomes the longstanding challenge of parallel programming for anything but embarrassingly parallel problems, which is why many applications are still single-threaded and run at essentially the same speed they have for more than a decade. Agents help to overcome that barrier, with the added benefit that they can work together (symmetrically) or independently/semi-independently (asymmetrically).“Agentic AI is an interesting concept, providing we are confident enough to trust a level of autonomy for the AI to go off and make active decisions,” said Mike Ellow, CEO of Siemens Digital Industries Software. “On one level, we can define a task with a set of boundary conditions and then let AI operate within that box in order to drive to a solution. But when you get to the agentic piece, basically you’re saying, ‘Here’s the problem. You think about the best way to execute it, you come up with that solution, and you go drive toward the desired outcome from an EDA perspective.’ This is how we look at the evolution of AI into our tools.”Boxing in AI is essential, and this is the tack taken by all the major EDA vendors today.
“It’s all self-contained,” said Paul Graykowski, director of product marketing at Cadence. “You can almost think of having a prompt for every tool. So instead of clicking switches, doing settings, and all that kind of stuff, you just say, ‘I want to run a high-performance simulation with mixed-signal design. That’s safe, because what we’re doing is pulling all of the documentation out of our database into RAG (retrieval augmented generation). Where we’re going next is to have more of a reasoning approach, but it will still be very restrictive. We’re super-constraining it.”One of the concerns here is that the technology is accelerating much faster than the rules to control it, and that affects where AI is being deployed. It’s largely a risk assessment of an unknown result. So using AI in verification is arguably much lower risk than the design of an SoC or multi-die assembly, for example, because there are multiple levels of testing that follow verification.
“In verification, things are much more carefully defined,” said ChipAgent’s Wang. “We can iterate this really fast with our enterprise users and get their feedback every day. There are so many messages about feature requests and bug reports. But throughout that iteration, it helps us understand what would be a best practice to set up these rules for agents to follow so that it can improve the productivity for design verification engineers.”That’s one piece of the security picture. In addition, the hardware itself also needs to be secured against cyberattacks, which can alter how an LLM or different agents behave.
“Hardware security has improved,” said Synopsys’ Borza. “It’s not perfect, but it’s better than it was, and it continues to get better. There’s also acceptance that it needs to continue to improve. And for devices that run LLMs and agents, correct and properly functioning hardware that’s working on data that’s authentic is the baseline of trust you need to have a compute infrastructure capable of running these things. Otherwise, the hardware itself can change the behaviors of the language models, whether it’s accidental or due to someone attempting to steer it with some kind of attack. But now you have the real problem, which is that the system that’s running on that hardware needs to be trusted and have some transparency, and be able to show you how it got to a conclusion. Those tools exist for people building the models and the tools, but they’re not generally available to users of the system to understand how an answer they got back was produced and give them at least a chance to assess how correct or suitable the answer was and what considerations went into that answer.”
What Comes Next?
Understanding these various flavors of AI and how they can be used to create an AI system that is robust, secure, and adaptive is well beyond the capabilities of the human brain. Machines are needed to make all the pieces work together — and to keep it working optimally throughout a device’s projected lifetime. And despite the hype and the massive investments, there is an enormous amount of work left to do in understanding what’s really going on inside an AI system.“Once AI starts something, it’s difficult to know when to stop it. There is currently no safety mechanism,” Best said. “There’s no human in the loop, so there is no conscience layer that is an automated system monitoring the decision-making of the reasoning model and saying, ‘You’ve wandered out of line. I understand you’re trying to achieve your goal, but we’re going to have to pull back your stimulus because it’s now crossing a line. There is no progress yet to that, and until there is, I’m not sure any reasonable person is going to just imbue trust in this closed-loop automated system knowing that it is purely goal-oriented with no qualms about how the goal is achieved.”In the short term, the best practice is experimentation, observation, and analysis, and the baseline for adoption is repeatability across as many use cases as possible.
“When we look at things like chip design, we always think about, ‘Okay, let’s validate everything the agent does,'” said Microsoft’s Chengappa. “Even before an agent is sent out, you validate everything it does. You monitor the exchange of information. You store all that, and you check that in the backend multiple ways before you say, ‘This agent is ready for prime time.’ If you look at Microsoft’s team, they’re working on all these things. We leverage those, and build on top of that. So we hope that we inherit some of the good things they do, and avoid some of the bad things they might miss initially. But hope is not a strategy. So we’re working really hard on this.”
Related Reading: How AI Will Impact Chip Design And Designers How AI is reshaping EDA, and how it will help chipmakers to focus on domain-specific solutions.
Source: Originally published at SemiEngineering on August 7, 2025.