[Dev3 ^~^ (shashank03-dev)]

Fixes #3270

Adds an opt-in prefill technique built on PromptSendingAttack. The adapter sends the converted user objective followed by an exact assistant prefix; scoring independently determines attack success.

• Enables request support on Hugging Face, OpenAI Chat, OpenAI Responses, LiteLLM, and AzureML chat targets. Known-model profiles and LiteLLM derivation preserve the adapter default, and custom configuration can disable it.
• Keeps HTTP, Playwright, and Completion targets disabled by default. RoundRobin requires matching child configurations.
• Preserves shared converters, simulated memory roles, retries, scoring, and distinct evaluation hashes. Audio-history and positional-parameter regressions are covered.
• Updates both paired single-turn docs and preserves the web-search and adversarial-prompt support from current main.

Tests: 31,921 passed, 23 skipped; 96.23% overall coverage; 100% changed-line coverage. Python typing, pre-commit, and documentation checks pass. Fake SDK/HTTP tests verify exact objective/prefix order on all four hosted chat adapters, independently of the returned answer.

CI: Fork workflows require maintainer approval before they can run.

[Manohar Paturi (ManoharPaturi)]

This is a great implementation of the prefill attack. Requiring supports_prefill natively in TargetRequirements is the right design decision because attempting to adapt or fall back on targets that drop trailing assistant messages would disguise a target limitation as an attack refusal. Scoping request converters strictly to the prepended user objective preserves the exact prompt tokens intended for continuation, and tagging the prefill as a simulated assistant role keeps memory history truthful. The HuggingFaceChatTarget chat template continuation tests cover the mechanics nicely.

[Dev3 ^~^ (shashank03-dev)]

Manohar Paturi (@ManoharPaturi) Thank you :)

[Roman Lutz (romanlutz)]

Dev3 ^~^ (@shashank03-dev) Sorry, I missed an architectural point in my earlier reviews, especially after asking you to make fixes around the new attack class. I should have caught this before approving. On a closer look, I think prefill should be an attack technique built on PromptSendingAttack, rather than a separate PrefillAttack. I'd like us to make that change before merging.

The capability and target changes are still useful. The part I'd like to change is how we package the behavior. Prefill does not introduce a new conversation algorithm: it prepares a user objective followed by an assistant prefix, then uses the same sending, scoring, retry, and result-handling flow we already have. That is a better fit for the technique/executor split in framework.md.

Here is the direction I suggest:

1. Keep the target capability and native continuation support. Keep CapabilityName.PREFILL, the default-false supports_prefill field, and the Hugging Face chat-template change. The target should still use continue_final_message=True and avoid adding a new generation prompt for a trailing assistant message. Keep OpenAICompletionTarget unsupported unless its adapter can preserve the objective as well as the prefix.

2. Represent prefill as technique configuration. Register a technique using AttackTechniqueFactory(attack_class=PromptSendingAttack, ...), with the prefix supplied as technique-owned seed/configuration. Put the general-purpose technique in pyrit/setup/initializers/techniques/extra.py, rather than adding it to the small default core catalog. Declare native PREFILL, MULTI_TURN, and EDITABLE_HISTORY requirements through TechniqueRequirements and TargetRequirements. Unsupported targets should fail compatibility checks, not silently flatten or drop the prefix.

3. Add only the missing reusable preparation support. This is not quite a configuration-only replacement with today's code. PromptSendingAttack can send a supplied next_message, but SeedGroup.next_message only extracts a final user turn; an assistant-ending seed group is treated as history. Request converters also normally run on the outgoing message. Please use the existing seed/technique composition and add the smallest explicit, reusable opt-in needed for this shape:

   user:      <the current objective, after request conversion>
   assistant: <the exact configured prefix, left open for continuation>
   

   Do not globally reinterpret every assistant-ending seed group as an outgoing request. Other techniques use assistant-ending seeds as completed simulated history before the real request. The prefill behavior needs an explicit contract, while existing seed extraction and ordinary prompt-sending behavior retain their defaults. Technique code should configure preparation, not implement its own send/scoring/retry loop.

4. Preserve converter, memory, and scoring semantics. Request converters should act on the objective, not the prefix. Store the caller-written prefix as simulated_assistant, but send it with the assistant API role. Response converters and scorers should still operate on the generated continuation. Reuse the existing executor lifecycle, retry isolation, labels, and persistence. The technique identity must include the prefix so two different prefixes retain different evaluation hashes. Also declare which incoming seed-group shapes the technique accepts; do not silently discard dataset-provided prompts or history.

5. Replace the dedicated attack API and update the example. Once the shared path works, remove the new PrefillAttack/PrefillAttackParameters class and exports from this PR. Move the relevant coverage to the technique and shared preparation/execution paths. Update both paired single-turn documentation files to show the technique-based usage. There is no need to retain a compatibility wrapper for an API that this unmerged PR has not released.

6. Keep the original regression coverage and add boundary tests. Exercise the public technique/executor path with a recording target or fake provider. Check the exact user/assistant ordering, unchanged prefix tokens, objective-only request conversion, continuation scoring, simulated memory role, distinct persisted evaluation hashes, unsupported-target rejection, and retry isolation. Add controls showing ordinary user-ending requests and existing assistant-ending history still behave as before. Keep the QRCodeConverter/text-only-target rejection test.

There is also still an audio-history regression to address in the shared validation change. With OpenAIChatTarget(audio_response_config=OpenAIChatAudioConfig(voice="alloy")), the first text request can produce a transcript plus an audio_path response. The next text request now rejects that stored assistant audio as unsupported input, before the adapter can omit the audio and send its transcript. Please validate the content the adapter actually sends, or provide a narrow adapter-aware validation hook, without weakening the rejection of genuinely unsupported objective images. Add a two-turn public-send test with a fake SDK: first an audio response, then a text follow-up, and assert that the second payload carries the transcript rather than input_audio.

I realize this changes the direction after the earlier feedback, and I'm sorry for the extra work. The goal is to keep the useful prefill support while avoiding another attack subclass for behavior that can share our existing execution scaffolding. If there is a constraint in the technique path that I have missed, please point it out so we can address that shared gap explicitly.

[Dev3 ^~^ (shashank03-dev)]

Roman Lutz (@romanlutz) Thanks for the detailed guidance. Agreed — I’ll move prefill into a PromptSendingAttack technique, fix the audio-history regression, and add coverage for the edge cases. I’ll update the PR once verified.

[Dev3 ^~^ (shashank03-dev)]

Roman Lutz (@romanlutz) I’ve addressed your feedback and updated the PR. Could you pls review it when you get a chance? Thanks!

[Roman Lutz (romanlutz)]

The technique-based rewrite addresses the architectural feedback, but two regressions remain in the shared code.

[Dev3 ^~^ (shashank03-dev)]

Roman Lutz (@romanlutz) Fixed both: LiteLLM now allows the audio history its adapter omits, and continue_final_assistant_message is keyword-only so the old positional AttackParameters call works again. Added regression tests for each. Could you take another look when you get a chance?

[Roman Lutz (romanlutz)]

Dev3 ^~^ (@shashank03-dev) One more correction to my earlier guidance, sorry. I tried assistant-ending requests against live OpenAI Chat Completions (gpt-5.4) and Responses (gpt-4.1). Both APIs accepted them. The models did not produce the particular suffixes I expected, but that is not enough to conclude that the request is unsupported. We should not confuse capability support with the model doing what the attack wants.

Please adjust the capability policy in this PR:

1. Define supports_prefill as request support. It means the adapter can send the full conversation, including the objective and a caller-supplied trailing assistant prefix. It does not promise that the model will continue the prefix as intended or that the attack will succeed. Please update the field description and documentation to make that distinction clear.

2. Enable it by default on the chat adapters. Keep it enabled on HuggingFaceChatTarget, and enable it on OpenAIChatTarget, OpenAIResponseTarget, LiteLLMChatTarget, and AzureMLChatTarget. Keep the base TargetCapabilities default false, and allow an explicit custom configuration to disable it for a deployment that rejects this request shape.

3. Keep HTTP and Playwright off by default. This includes HTTPTarget, HTTPXAPITarget, PlaywrightTarget, and PlaywrightCopilotTarget. Their behavior depends on the particular integration, so they should not advertise this automatically. Also leave OpenAICompletionTarget disabled for this PR: its current adapter sends only the final message text and would lose the preceding objective. No completion-adapter rewrite is needed here.

4. RoundRobin should support it only when all sub-targets do. Its existing identical-configuration requirement can stay. Cover the all-enabled and all-disabled cases, and make sure a single supporting child cannot make the wrapper advertise support when another child does not.

5. Preserve the policy through configuration and test the actual boundary. Known-model profiles and LiteLLM's capability derivation must not accidentally reset the adapter's prefill default. Explicit custom opt-out must still win. Add coverage for those paths and for the disabled HTTP/Playwright/Completion defaults. Through the public technique/executor path with a fake SDK boundary, assert that both the user objective and exact assistant prefix reach the provider in order. A provider returning a complete answer rather than our desired suffix should not make that request-compatibility test fail; response quality and attack success belong to scoring. Keep the existing converter, memory, retry, and unsupported-image tests, and update both paired documentation files.

This supersedes the part of my earlier comment that treated hosted chat APIs as unsupported by default. The request-level policy should be consistent across these adapters rather than tied to whether a particular probe gets the answer we hoped for.

[Dev3 ^~^ (shashank03-dev)]

Roman Lutz (@romanlutz) Updated the prefill request-support policy as requested, added provider-boundary coverage, and updated both docs. Also resolved the merge conflicts with current main. 31,921 tests pass locally. Could you pls review and approve the CI runs when you get a chance? Thanks!