FIX: Fixing json retry

[rlundeen2]

Description

The bug

JSON retries were not actually getting a fresh, independent response — which means JSON retry wasn't actually really working in many cases. In the adversarial manager, it was retrying via the normalizer, so the history was added, but then retried already containing the responses (which were not removed). Each "retry" was therefore conditioned on poisoned history rather than being a clean, independent attempt, making success non-deterministic (hence the flake).

The fix

A shared, memory-aware helper send_json_with_retry_async (in pyrit/prompt_normalizer/) that makes every attempt independent, and rewiring the two JSON-expecting call sites through it:

• Adversarial conversation manager (_send_and_parse_async)
• Shared LLM scorer (_run_llm_scoring_async) — the normalizer here is injectable (empty None → PromptNormalizer() default) so scoring persists through the same path.

Why we delete a message

The helper records a baseline sequence for the conversation (the max MessagePiece.sequence present before we send). On the first attempt it deletes nothing. On each retry it rolls memory back to that baseline by deleting the failed turn's pieces (everything after the baseline sequence) before resending. This is what actually gives us a new response: the model sees a clean history identical to the first attempt, instead of its own malformed JSON plus a duplicated prompt.

We chose to delete rather than branch onto a new conversation_id so that the conversation identity stays stable for the caller and downstream consumers — only the poisoned turn is removed, everything before the baseline is untouched.

Memory updates

To support the rollback cleanly and keep a record that a retry happened, memory grew two primitives plus a small amount of conversation-scoped state:

• delete_conversation_pieces_after_sequence(conversation_id, sequence) — deletes the failed turn's pieces (returns whether anything was deleted).
• add_conversation_retry(conversation_id, sequence, reason) — records a lightweight ConversationRetry marker (reason=JSON_PARSING) so the retry is auditable even though the failed message pieces are gone. Rather than keeping the poisoned turns around, we keep just this metadata saying "this turn was retried."
• New ConversationRetry / ConversationRetryReason models, and a retries list on the Conversation model (conversation-scoped metadata, not stamped onto every MessagePiece).
• New Alembic migration a1c3e5d7f9b0 adds a retries JSON column to the conversations table.

The retry loop keeps the existing @pyrit_json_retry (tenacity) decorator, so retry logging and RetryCollector attribution (after=log_exception) are preserved.

Tests and Documentation

• Updated test_adversarial_conversation_manager.py, test_crescendo.py, test_red_teaming.py, and test_scorer.py to cover routing through the helper and the memory rollback/retry-marker behavior.
• All affected unit suites pass locally (memory, models, prompt_normalizer, score, executor/attack).
• Pre-commit passes (ruff format/check, ty type-check, async-suffix, alembic immutability, memory-migrations check run directly against the venv). No new JupyText/doc code samples were added.

[adrian-gavrila]

Looks good! One docstring thing that might be worth attention.