Repository navigation
FIX: Annotate audio scorer rationale once instead of duplicating it - #3521
Open
Chen Yufeiyang (feiiiiii5) wants to merge 1 commit into
Open
Chen Yufeiyang (feiiiiii5) wants to merge 1 commit into
Chen Yufeiyang (feiiiiii5) wants to merge 1 commit into
Conversation
The wrapper score copied the transcript scorer's rationale twice: once bare and once under the 'Audio transcript scored:' prefix. The wrapper's job is to attribute the judgment, and model_copy already carries the child's rationale, so keep only the attributed line.
Richard Lundeen (richlundeen)
approved these changes
Oct 25, 2026
Richard Lundeen (richlundeen)
left a comment
Richard Lundeen (richlundeen)
left a comment
Contributor
There was a problem hiding this comment.
Looks good. This removes the duplicate rationale from both audio wrappers and keeps the transcript attribution.
Reviewed with GitHub Copilot.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes a small data-quality bug in the audio scorers' wrapper scores.
AudioTrueFalseScorerandAudioFloatScaleScorerbuild a wrapper score around the transcript scorer's result and set its rationale tof"{rationale}\nAudio transcript scored: {rationale}", so the child's rationale is persisted twice — once bare, once attributed. That rationale is stored with the score and, for example, spliced whole into the adversarial chat's feedback prompt when Crescendo runs withuse_score_as_feedback=True._create_wrapper_scorealready deep-copies the child's judgment; the wrapper's job is to attribute it (docstring: "Copy a child's judgment into a new wrapper-owned result"), and the sibling wrappers (ConversationScorer,FloatScaleThresholdScorer) annotate without repeating the text bare. This keeps only the attributed line.Tests and Documentation
Strengthened the two
test_score_piece_with_transcriptcases to assert the exact rationale. Both fail on main with the bare copy present, e.g.AssertionError: assert 'Test rationale...t transcript.' == 'Audio transc...wav', and pass on this branch. Also ran the fulltests/unit/scoresuite (4533 passed, 115 skipped) and ruff check/format on the changed files. No documentation change needed (behavior now matches the docstring); JupyText not affected by this change and not run.