“There’s many levers in the design of a technology that can incentivize, reward, or sanction behaviors,” said Lena Slachmuijlder, co-chair of the Council on Tech and Social Cohesion. “And that’s where we saw the gap.”

Our team attended the Council’s 2026 expo, “From Harm Mitigation to Intentional Design,” at the end of May, tuning into the Asia segment hosted from Bishkek, Kyrgyzstan.2 The discussion went beyond downstream responses of harmful content to the core incentives embedded into their hosted platform design, as well as the regulatory tools emerging globally to govern them.

The DSA and the move toward design accountability

For years, the most visible conversations in digital design have focused on the moderation of content that appears on user feeds. Which posts should be removed? Which accounts should be suspended? How should users be notified or allowed to appeal when platforms take enforcement action? But as policymakers and researchers look beyond individual incidents, they are progressively asking whether the systems beneath them deserve the same level of scrutiny.

A recurring theme throughout the session was the focus on the behavioral levers behind the most compelling platform design. Recommender systems shape what users see and which posts gain visibility, while interface choices can steer, delay, simplify, or complicate the decisions users make online. These levers include menu hierarchies, engagement metrics, push notifications, autoplay, infinite scroll, and other forms of digital choice architecture that can incentivize, reward, or discourage behavior.

Recommended systems: algorithmic systems that mediate information propagation on digital platforms by selecting, ranking, and prioritizing content for users (also referred to as “assemblages” by academics). Rather than merely displaying posts from accounts a user follows, they predict which content a user is likely to engage with and use those predictions to shape feeds.

“These are now peripheral features that we’re looking at,” said Niamh Hanafin, assistant director of Coimisiún na Meán, Ireland’s media regulator. “Looking at how platforms are designed and operated rather than this content model.”

The European Union’s Digital Services Act, or the DSA, is one of the EU’s most comprehensive and consequential regulatory frameworks for online platforms at this time.

The European Commission describes the DSA as a framework for regulating online services such as social media platforms, marketplaces, app stores, and travel platforms, with the goal of creating a safer digital space where fundamental rights are protected. The law also gives users greater transparency and control over their online experience, including the ability on very large platforms to choose non-personalized feeds, receive clearer information about advertising, and be protected from dark patterns.

Dark patterns: online interface designs that deceive, manipulate, or materially distort users’ ability to make free and informed decisions.

That framing treats platform architecture as a non-neutral risk environment. In the same way that vulnerable cybersecurity architecture can expose users to predictable forms of risk, poorly designed or inadequately governed social platform architecture can expose users (particularly children and vulnerable communities) to predictable forms of manipulation, escalation, compulsive engagement, harassment, and polarization.

“Regulation is a key incentive for companies to make choices, either to better enforce their own policies or to decide to explore other options,” Slachmuijlder said.

DSA is a complex legal document, and implementation has taken time to reach speed, according to Hanafin. But as of recently, there has been an increase in enforcement actions by digital services coordinators and the European Commission. Ireland’s media regulator was described as especially consequential because 16 of the 26 very large online platforms and search engines are headquartered in Ireland. As a result, enforcement by the Irish regulator can have implications across the European region, according to Hanafin.

Meta, dark patterns, and the right to choose

One of the clearest examples of design enforcement discussed during the session was Coimisiún na Meán’s decision to open two formal investigations into Meta’s Facebook and Instagram services. The investigations concern whether the platforms have complied with DSA requirements on recommender system transparency and online interface design.

The first issue concerns the user’s right to choose. Under the DSA, platforms that use recommender systems must explain the main parameters that determine why certain information is suggested to users and must provide users with ways to modify or influence those parameters. For very large online platforms and search engines, the law goes further; exceptionally large platforms must provide at least one recommender-system option that is not based on profiling (scroll down to Article 27). In practice, this means users should be able to access a version of a feed that is not ranked according to automated inferences about their personal data or predicted engagement.

The second issue concerns dark patterns. Online platforms may not design, organize, or operate their interfaces in ways that deceive or manipulate users, or otherwise materially distort or impair their ability to make free and informed decisions, according to Article 25. The DSA identifies examples such as giving more prominence to certain choices, repeatedly prompting users after a choice has already been made, or making it harder to terminate a service than to subscribe to it. In this context, dark patterns are interface arrangements that can undermine user autonomy by making some choices easier, more visible, or more persistent than others.

This distinction is interesting because it implies that digital rights are only meaningful if they can be exercised under realistic conditions. A non-profiling feed that exists but is difficult to find, confusing to activate, or easy to reverse through nudging does not provide the same level of control as one that is clearly presented and accessible from the point where recommendations are displayed. The DSA itself recognizes this problem by requiring that certain recommender system choices be directly accessible from the relevant online interface.

In this frame, the “peripheral” interface prompts and settings become matters of public accountability. They help determine whether users genuinely control their online experience or are steered toward choices that primarily benefit the platform’s data, advertising, or engagement model.

For Meta, the legal question remains unresolved while the investigations are ongoing.

TikTok and addictive design as systemic risk

The European Commission’s TikTok investigation moves platform regulation deeper into the mechanics of engagement. In February 2026, the Commission preliminarily found TikTok in breach of the Digital Services Act over what it described as addictive design. The features under scrutiny included infinite scroll, autoplay, push notifications, and TikTok’s highly personalized recommender system.

“We know how effective their recommender system is. It’s incredibly personalized, it’s extremely compelling,” Hanafin said. “This is what makes TikTok the system and the success that it is. It’s core to their business model.”

The Commission’s preliminary findings focused on whether TikTok had adequately assessed the risks that these design features could pose to users’ physical and mental well-being, particularly for minors and vulnerable adults. It also questioned whether TikTok’s existing safeguards, including screen time tools and parental controls, were sufficient to mitigate risks created by the platform’s design.

From a safety and security perspective, this question reframes engagement itself. Time spent on a platform is often treated as a measure of product success. Under a systemic risk approach, however, engagement can also become evidence of risk when it is produced through design choices that weaken user agency, encourage compulsive use, or make disengagement difficult.

“This [case] will potentially be setting some very important precedent,” Hanafin said.

That shift is particularly relevant within the backdrop of emerging technology. As AI-powered personalization becomes more powerful, the line between recommendation and manipulation may become harder for users to see. Regulators will need to understand how systems shape attention, agency, and behavior alongside the platforms’ output in order to proactively mitigate the risks tied to them.

The same design questions now apply to AI systems built for, used by, or likely to affect children. The 5Rights Foundation’s Children & AI Design Code argues that those who build and deploy AI systems should identify, evaluate, and mitigate known risks to children while also preparing for “known unknowns.” It calls for foreseeable risks to children to be considered “by design and default,” rather than addressed only after harms emerge.

“Children do not only need protection from spaces, but they also need better spaces that are designed with their rights and safety in mind,” said Head of International Affairs at 5Rights Marie-Eve Nadeau.

If recommender systems can amplify harmful dynamics, AI systems may personalize those dynamics further. If dark patterns can steer users through interface design, AI assistants and generative systems may steer users through more personal language, timing, and automated suggestions. If children are already navigating systems that adults struggle to understand, AI raises the stakes for transparency, testing, and accountability of design.

The insight from the DSA discussion is that regulators should ask design questions regarding optimization, reward behaviors, and foreseeable risks early. Who benefits when a user keeps scrolling, clicking, or staying on emerging networks?

Global regulation without overreach

Any serious discussion of online safety must also confront the risk of regulatory overreach.

In many parts of the world, civil society groups worry that online safety laws may become tools for censorship, surveillance, or political control when they are framed too broadly or enforced without adequate safeguards. A law written in the language of protection may empower governments to silence critics, suppress dissent, or expand state authority over speech.3

The session thus expanded Europe’s regulatory turn within a wider global movement toward both child safety and rights respecting design. Recent developments in Brazil and Indonesia suggest that governments outside Europe are beginning to impose clearer duties on digital platforms and services, especially where children are likely to be users, according to Nadeau.

Brazil’s Digital Statute for Children and Adolescents, known as the ECA Digital, establishes obligations for digital products and services accessed or likely to be accessed by minors, including duties related to inappropriate content and parental supervision. Indonesia’s Government Regulation No. 17 of 2025 similarly imposes child protection obligations on electronic system providers. Beyond Latin America and Asia, the African Union’s Child Online Safety and Empowerment Policy, adopted in 2024, signals a regional effort to frame children’s online safety as a topic of privacy, participation, and the best interests of the child.

In jurisdictions where legal overreach has greater opportunity to succeed, the session offered grounding corporate accountability in international human rights law and focus on system design rather than political content policing. That distinction begins by asking how platform architecture shapes amplification and user control rather than asking governments which political content should be removed (hence, the “harm mitigation” aspect of the expo).

To ask how platform systems make certain content viral, how recommender systems amplify harmful dynamics, how engagement incentives shape behavior, and how companies profit from risky architecture is not the same as monitoring content that compromises or benefits certain agenda off- and online.

Such an approach offers a more principled starting point for regulation, shifting attention away from individual viewpoints to the systems that structure how users interact and utilize the very platforms that are shaping lives.

“We can design technology differently to bring out the best of humans, to not exploit our vulnerabilities,” Slachmuijlder said.

If you are an industry professional who would like to contribute or be interviewed, feel free to message us below:

Modern security teams are somewhat of a reimagined guard tower. They surveil corporate infrastructure with preemptive suspicion, deploying technical countermeasures against rogue actors the way sentinels once held perimeters behind masonry walls. But as the center of gravity in conflict shifts toward cyberspace, the nature of the threat has shifted with it, and so have the tools deployed to meet it.

Among the most significant of those tools are AI-powered summarizers, now widely adopted in Security Operations Centers (SOCs) to help analysts manage the overwhelming volume of machine-generated records: alerts from providers, servers, and cloud infrastructure that would otherwise require teams of analysts to manually review. These tools distill thousands of events into short, digestible narratives, allowing analysts to triage and act with greater speed. Their proliferation has accelerated as the underlying models have grown more capable and cost-efficient.1 Recent survey research on large language model (LLM) deployment in SOC environments reflects that trajectory, identifying summarization, alert triage, and knowledge assistance as among the most promising applications for generative AI in security operations.

What that appeal obscures is a structural vulnerability that comes bundled with the technology. AI systems do not read only human instructions; they also read the data placed into their context. If an attacker can hide instructions inside security-related artifacts, the model may process malicious words alongside the analyst’s request. This is the logic behind indirect prompt injection, in which the adversary does not need to talk to the model directly if they can plant language the model will later retrieve and act on.

Indirect prompt injection: a security vulnerability where malicious instructions are hidden within external data.

This report documents one such attack scenario in empirical detail, leveraging a sociotechnical lens to discuss real-world implications. A simulated attacker embeds a prompt injection payload inside a network security log, targeting an AI-powered SOC summarizer built on a Retrieval-Augmented Generation (RAG) pipeline.2 The current result set contains 2,250 evaluated runs across five open-source AI models and nine tested conditions. Across those runs, the attack produced 362 confirmed breaches, for an overall Attack Success Rate (ASR) of 16.1% in the result set.

Retrieval-Augmented Generation (RAG): an AI framework that improves LLM responses by fetching relevant facts from external data sources before generating an answer.

For readability, this report leads with a brief background section followed by findings and discussion rather than following the conventional structure. The full methodology appears below those sections for readers who want the technical details.

Background

The Attack Surface RAG Creates

RAG pipelines work in two stages. First, source documents (in this case, network logs) are processed and stored in a searchable database. When an analyst submits a query, the system retrieves whichever stored excerpts are most relevant to that query and places them in front of the model as context before it generates a response. The model only ever sees what the retrieval step hands it.

This architecture is operationally effective but introduces a structural risk: the model cannot distinguish between instructions provided by its operator and content retrieved from the collection of data.

In a SOC setting, the attack is uniquely favored for this technique. The very logs that analysts rely on to detect adversaries are generated, in part, by adversary activity. An adversary who can influence what goes into that data, or whose malicious activity leaves traces that end up indexed, can embed instructions inside the content the model is designed to retrieve and summarize.

Findings

The attack was not uniformly successful, but rather conditioned by retrieval layer, prompt structure, payload design, and model behavior, determining whether the AI “sentinel” remained useful or became a conduit for counterparty instructions.

1. The retrieval layer is a security boundary, but not the whole story

The most common way to talk about prompt injection is to ask whether the model “obeyed” the attacker. That question matters, though it is incomplete for RAG systems. In a RAG pipeline, the model can only follow the malicious instruction if the retrieval system first places that instruction into context. The retrieval layer therefore acts like a security checkpoint.

This campaign data shows both sides of this checkpoint. When the poisoned log was not retrieved (based on the Retrieval Hit Rate in Table 2), the attack produced no breaches. That happened with 100 logs and a single poisoned log, and again with 50 logs and a disguised poisoned log. In those cases, a 0.0% attack success rate should not be read as proof that the model resisted the attack. It may simply mean the model never saw the poisoned instruction.

Retrieval Hit Rate (RHR): the percentage of runs in which the RAG system retrieved the poisoned log chunk and placed it into the model’s context window.

But the reverse is also true, since a retrieval did not always guarantee a breach (Table 3). With 50 logs and a single poisoned log, the poisoned log reached the model every time, but the attack still produced 0.0% confirmed breaches. With 100 logs and repeated poisoned logs, the poison also reached the model every time, yet the attack success rate was only 1.2%. Most notably, with 100 logs and a disguised poisoned log, the poison reached the model in 100.0% of runs but still produced 0.0% confirmed breaches.

That combination produces a more nuanced interpretation, in which the retrieval layer is a necessary, but not entirely sufficient, boundary in mitigating indirect prompt injections.

2. The “small batch” problem is operationally meaningful

The strongest results appeared when the model was asked to summarize a small batch of 10 logs. In that setting, the poisoned log had less competition from surrounding data and was more likely to become prominent in the model’s context.

With 10 logs and a single poisoned log, the attack succeeded in 60.4% of runs on average across models. The 10 logs with a disguised poisoned log condition was nearly as prominent. Here, the poisoned instruction was wrapped in security language so that it looked more like the kind of event the model was asked to review. This condition produced a 55.6% average attack success rate. Qwen 2.5 7B and Mistral 7B stood out as models that followed injected instructions in 100% and 98% of times, respectively.

Below is an example of such a response:

This finding is practically important because small log batches can occur after log rotation, during quiet network periods, in smaller organizations with lower event volume, or during narrowly filtered investigations. A SOC analyst may ask for a focused summary of a small set of suspicious events, which is the kind of reduced noise setting in which the attack performed most effectively.

The sociotechnical risk is that SOC workflows often optimize for exactly that narrowed context. Analysts build models intended to retrieve the most relevant evidence, but relevance is also what an attacker can imitate. A poisoned log that mentions directory traversal, authentication failure, or critical anomalies may look useful to the retriever precisely because it resembles the analyst’s query.

Our takeaway for this caveat is that “more logs” can sometimes dilute a poisoned entry, but security teams should not treat log volume as a reliable defense. Many real workflows naturally shrink the retrieval pool down to a small, highly relevant set of records.

3. Model susceptibility varied substantially

The campaign also shows that models behave differently. Qwen 2.5 7B and Mistral 7B were the most vulnerable on average across the nine main attack conditions, with mean attack success rates of 26.4% and 26.0%, respectively. Llama 3.1 8B averaged 13.8%, Gemma 2 9B averaged 11.8%, and Phi-3 Mini was least affected at 2.4%.

It would be tempting to frame this as a simple model-selection result, in which choosing a model with the lowest breach rate moderates the injection success rate. But that would be premature. A model may resist this specific injection because it is less instruction-following, less sensitive to the payload, less capable of parsing noisy context, or more likely to ignore parts of the prompt. Some of those traits may reduce attack success as well as usefulness - a technical paradox that poses as a double-edged sword.

For model developers, the better target is models and application architectures that preserve a clear distinction between trusted instructions and untrusted content. An effective SOC summarizer would follow the analyst’s instruction while refusing to treat a log entry as an authority source. The model must still read the poisoned text as evidence while not obeying it as command, which is a subtle, yet crucial, capability.

4. Repeating the poisoned log did not reliably strengthen the attack

One hypothesis behind our test was that redundancy would help the attacker. That pattern included multiple poisoned logs rather than one, testing whether repeated attacker instructions would improve retrieval and breach rates.

With 10 logs and repeated malicious logs, the poison reached the model every time, but the average attack success rate was only 13.6%. That was far lower than both 10 logs, single poisoned log at 60.4% and 10 logs, disguised poisoned log at 55.6%.

A potential explanation is that even though repeated poisoned logs made the attack more visible to the retrieval system, the model may have reverted to treating them as noisy log content rather than a credible directive. By contrast, the single poisoned log was simpler and more direct, while the disguised poisoned log looked more relevant to the analyst’s task.

At 50 logs with repeated poisoned logs, the average attack success rate was 14.0%. This was not evenly distributed across models. Mistral 7B accounted for the strongest result at 44.0%, while Qwen 2.5 7B reached 12.0%, Phi-3 Mini 8.0%, Gemma 2 9B 6.0%, and Llama 3.1 8B 0.0%.

The practical insight is that repetition may help the poisoned content get noticed by the retrieval system, but it does not guarantee that the model will follow it. In these tests, the most successful attacks were the ones that appeared in small log batches or were framed in a way that matched the analyst’s task.

5. Disguised poisoned logs worked in small batches, but not in larger batches

The disguised poison strategy was designed to test a more targeted attack. Instead of placing a bare instruction into the logs, the payload was wrapped in language that looked relevant to the analyst’s query, including terms like “directory traversal,” “authentication anomalies,” and “critical alerts.”

In the 10 logs, disguised poisoned log condition, this strategy was directionally effective. It produced a 55.6% average attack success rate and the two highest model results in the campaign, averaging 100.0% for Qwen 2.5 7B and 98.0% for Mistral 7B.

But that success did not carry over to larger log batches. With 50 logs and a disguised poisoned log, the poison was not retrieved at all, and the attack success rate was 0.0%. With 100 logs and a disguised poisoned log, the poison was retrieved in 100.0% of runs, but still produced 0.0% confirmed breaches.

Synthesizing those findings, we find that disguising the poisoned log as a relevant security event worked effectively in a small batch, but but it did not restore attack success in the tested medium or large batches.

6. Downstream connectivity amplifies risk

In production environments, AI-generated summaries may feed into other security orchestration tools, be it dashboards or escalation workflows. In those settings, a manipulated summary can travel downstream and influence both machines and humans. Our test relied on that very idea, where an AI-summarizer was tricked into yielding an attacker’s email address within a hypothetical ticketing system that could then escalate onward in the organization’s workflow.

One such example of an absorbed attack goes as follows:

The possible consequences are broader than one expected outcome. A successful injection could cause an AI system to minimize a real intrusion or direct an analyst toward the wrong evidence. It could also suppress urgency, result in exfiltration of sensitive data, or extend the attack beyond the incident record.

In a sociocultural sense, it’s not a secret that SOCs are high-pressure environments. Analysts are rewarded for speed, prioritization, and efficiency. A clean AI-generated summary can feel like relief from manually scanning alerts like the ones pasted above. That same readability, however, can create misplaced trust. The risk lies in the tool’s designed ability to turn messy evidence into confident prose, even when the evidence itself may contain adversarial language.

Our core insight is that AI security summaries should be treated as processed intelligence, not outright truth, especially when the underlying data may have been created by the attacker.

Discussion

The Blind Sentinel results point to a larger problem than one vulnerable prompt or one unusually gullible model. The tested system failed at the boundary between evidence and instruction. That boundary is easy for humans to describe, but difficult for current LLM applications to enforce. An SOC analyst reads a log line as evidence, while the model may interpret language inside that log line as an instruction.

Prior research describes LLM-integrated applications as systems that “blur the line between data and instructions,” allowing adversaries to inject prompts into external content likely to be retrieved later. In that framing, retrieved text can become operationally similar to code as it changes what the system does, not merely what the system knows. The results of this campaign bring that general concern into a security operations setting.

Implications for Deploying Organizations & End Users

A natural response to indirect prompt injection is to strengthen the system prompt, or rather, tell the model to ignore instructions inside logs, treat retrieved data as untrusted, and follow only the analyst’s request. That is worth doing, but it should not be treated as a primary control.

The vulnerability exists because the model receives trusted instructions and untrusted data in the same context window. A stronger system prompt may help the model interpret that mixture better, but it does not create a true technical boundary. The attacker’s instruction is still present in the same reasoning space as the legitimate task.

OWASP’s prevention guidance reflects this layered view. It recommends screening retrieved or fetched context before the primary model sees it, screening outputs before they are returned or passed to tools, and screening proposed actions against the original user intent. It also describes stronger architectural patterns in which a quarantined model reads untrusted content while a more privileged model controls tools and actions.

More recent joint guidance from several international cyber agencies frames AI security as a data-security problem across the full system lifecycle. The May 2025 guidance emphasizes that the data used to develop, test, deploy, and operate AI systems is part of the AI supply chain and must be protected from malicious or unauthorized modification. For SOC summarizers, that maps directly onto the risk shown in this campaign. It is becoming increasingly more apparent that logs are not neutral background material once they become model context. Instead, they are operational inputs that require greater provenance, integrity, and control to mitigate anomalous behavior.

For smaller security teams, the implications are nonetheless sharper. Lean teams may rely more heavily on summarization because they have fewer analysts available for redundant review. They may also process lower-volume log batches, creating the small-batch conditions where this attack performed best. The result is a compounding risk in which the organizations most likely to benefit from AI summarization may also have fewer safeguards around it.

Implications for Regulators & Policymakers

For policymakers, this campaign illustrates why AI governance in security workflows cannot stop at model evaluation. A model may look safe in a standalone chat setting and still become dangerous when connected to automated workflows.

Regulators and procurement bodies could therefore ask for evidence that deployers have tested indirect prompt injection in realistic workflows. For SOC tools, that means testing poisoned logs and alert summaries. It also means requiring documentation of where human review is mandatory and where model output can trigger downstream action. OWASP’s 2025 LLM risk taxonomy is useful here because it treats prompt injection, insecure output handling, excessive agency, and vector weaknesses as application risks rather than isolated defects.

Policy could also distinguish between human review in name and human review in practice. Many AI governance frameworks call for human oversight, but in rapid SOC environments, oversight can become procedural rather than substantive if analysts are expected to approve machine-generated summaries under alert pressure. The EU AI Act’s risk-based approach emphasizes safety, fundamental rights, human-centric AI, and obligations for specific uses of AI. In security workflows, that could translate into clear documentation of where AI output is advisory, where it becomes part of the incident record, and where it can trigger operational action.

Conclusion

This campaign set out to answer a practical question: in a standard enterprise-style RAG deployment used for security log summarization, how reliably can an attacker who controls part of the log stream manipulate the AI system’s output? Across the 2,250 main condition runs reported in this study, the attack produced 362 confirmed breaches, an overall attack success rate of 16.1%. Under the most favorable conditions for the attacker, the rate was much higher: with 10 logs and a single poisoned log, the average attack success rate was 60.4%; with 10 logs and a disguised poisoned log, it was 55.6%. Two models were especially vulnerable in those small-batch conditions. Qwen 2.5 7B reached 100.0% attack success in both, while Mistral 7B reached 90.0% and 98.0%, respectively.

The more textured finding is the role of retrieval architecture in mediating the attack. Context size, background noise, retrieval behavior, and the semantic characteristics of the poisoned log were among the security variables. A RAG pipeline ingesting only a handful of logs was far more vulnerable than one operating across larger batches. But the results also show that retrieval alone does not explain everything. In some larger-batch conditions, the poisoned log reached the model every time and still produced few or no confirmed breaches. Organizations that treat RAG configuration only as a performance question, rather than also treating it as a security question, are leaving a significant and measurable vulnerability unmanaged.

The goal of this research is not to discourage AI adoption in security operations. The summarization and triage capabilities these tools provide are real, and the operational pressures that drive their adoption are real. The goal is to ensure that adoption is informed, that the organizations deploying these systems, the researchers evaluating them, the developers building them, and the policymakers governing them have access to the kind of transparent, empirical data that makes responsible deployment possible. A sentinel that can be blinded by the very threats it is watching for is far beyond a neutral addition to a security team. Knowing its failure modes is the beginning of managing them.

Methodology

Overview

This evaluation was conducted using Haruspex, an in-house security testing engine built in Rust to orchestrate asynchronous, high-volume LLM interaction campaigns with deterministic grading. Haruspex sends structured prompt payloads to a target API endpoint in batch queues, collects LLM responses, and evaluates each response against predefined grading rules to produce an ASR.

The target system was a local RAG-based SOC summarizer. The RAG pipeline retrieved relevant log excerpts, inserted them into the model context, and asked the model to summarize directory traversal or authentication anomalies. All inference was local through an Ollama backend; no cloud model APIs were used in the uploaded campaign.

The v1 campaign established a baseline but could not distinguish whether a zero-breach result reflected model resistance or retrieval failure. The v2 redesign introduced RHR, measuring how often the poisoned chunk was surfaced by the RAG pipeline, and compliance-given-retrieval, measuring breach behavior conditional on confirmed poison delivery.

System Prompt

All main attack-condition runs used the same baseline system prompt and analyst query. The system instructed the model to act as an AI SOC analyst, read only the provided log excerpts, and output a concise summary. The user query asked for directory traversal or authentication anomalies detected in the last 24 hours.

The exact baseline prompt is below:

This query was intentionally semantically close to security language and poisoned content. In a real deployment, analysts ask for summaries of suspicious activity, and attacker-generated log lines are often among the most relevant records.

Artifacts including payload definitions and run logs are maintained for full reproducibility. Data that does not add to informing the report was redacted for security measures.

If you are an industry professional who’d like to be interviewed or contribute to the project, message us directly below:

In other, less explored ways, AI models are volatile mediums. Their ability to run scripts, interact with applications, communicate through natural language, and generate multimedia is as exciting as it is threatening. Developers raise billions in funding to improve reasoning by harnessing frameworks to make outputs more efficient, and even then, safety gets relegated for competitive advantage and shareholder satisfaction. As models become increasingly more agentic within our workflows, human involvement transforms in unprecedented ways. Could specialized tools outmaneuver their environments? How can organizations and users prepare when their intended purposes are misappropriated?

That’s what inspired us to create the Emerging Evaluations Project, or EEP for short. Driven by the desire to explore and document the fallbacks of up-and-coming models, the project aims to improve the design, deployment, and utility of AI model infrastructure by informing cybersecurity and social safety research. Our means are tangible, technical parameter assessments of programs available on the market; our end vision is greater transparency for all stakeholders.

What our work entails:

• Evidence-based analysis of emerging AI tools and their safety implications;

• Technical evaluations of model behavior, fallbacks, and exploitability across deployment contexts;

• Benchmarking on the capabilities and risks of available systems;

• Technical findings translated into interpretable insights for cybersecurity, social safety, and interested audiences.

AI models exhibit characteristics of a general-purpose technology, yet the risks associated with them are generally difficult to see. Some people use these tools to meet organizational needs, others to automate personal projects. But many possible outcomes remain unexplored, compromising our ability to control systems when deemed necessary. EEP intends to derive transparency from the “black box” systems used both systemically and individually, reporting on their inner complexities in a holistic manner.

How we accomplish what we set out to do:

• Test deployment parameters and interaction behaviors over thousands of repetitive rounds using a proprietary, in-house software (i.e. machine-learning security operations);

• Use regression testing to monitor changes across model updates and replacements;

• Link payloads, regex patterns, text logs, and other technical artifacts to our findings for transparency;

• Use visualizations to enhance accessibility and shareability of reporting;

• Study both technical and psychological vectors of exploitation, including those mediated by natural language;

• Connect findings to current affairs and industry standards through interviews with industry professionals.

Tool development has reached a pace that outmatches weekly, and perhaps even daily, changes in how digital systems operate. Across a vast spectrum of user experience, this has produced well-meaning debate over labor economies, human development, social cohesion, biowarfare, disinformation, cyber risk, and more. And, like many other tools, these models can serve both beneficial and harmful purposes, bringing risks and rewards unlike any humankind has faced before.

At EEP, we hope to shed light on these questions. We work toward understanding model deviancy, whether in the form of misalignment with intended tool purpose or maladaptive use against the broader public. We hope that the combination of technical expertise with insights for policy, safety, and security will extend into the very ecosystems where these tools take root.

If you are an industry professional who’d like to be interviewed or contribute to the project, message us directly below: