Skip to content

Reference-chain cache: rebuild improved-path chains instead of erasing them (PROF-16316) - #898

Draft
jbachorik wants to merge 1 commit into
mainfrom
jb/refchain-cache-invalidation
Draft

jbachorik wants to merge 1 commit into
mainfrom
jb/refchain-cache-invalidation

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:
Splits reference-chain deferred cache invalidations into two kinds. EVICT (noise eviction, event filter) keeps today's behavior: the cached chain is erased at dump time. REBUILD (improveChain deeper path, durable-root demotion, root-kind upgrade — the source object is still live and its frontier entry intact) is no longer erased at dump time; the BFS-thread poll rebuilds the chain from the intact frontier instead, overwriting the cache entry on success and keeping the old chain if the rebuild fails.

Motivation:
PROF-16316. improveChain()'s deferred invalidation erased resolved leak-correlated chains that the documented restartSearch() contract says must stay cached across a restart ("a chain resolved by the finishing search stays cached (and keeps being re-emitted on every dump) across the restart, since it describes a sample that is still live"). When the improve fires in the terminal pass of a search (new backing array of a growing holder, deeper path), the drain erases the chains, the terminal branch then zeroes candidate/discovered state so nothing can rebuild them, and the search stays pain-budget-blocked ~11-15 s — every later dump re-emits 0 chains and the correlation is lost for the rest of the window.

Deterministic reproducer in the e2e suite: ThreadLocalLeakReferenceChainTest.shouldCorrelateLeakOnThreadCreatedAfterRecordingStart on JDK 11 fails exactly when the pass-1 CPU pain-budget debt is already repaid when the canary closes (15 instrumented runs: debt ≥1021 ms → 4/4 pass, ≤988 ms → 11/11 fail). Pre-existing: the involved paths are untouched by #889.

Additional Notes:

  • The kind is encoded in-band in the existing bounded pending buffers (sign bit; cache keys are non-negative frontier tags) — no new allocation, no new locks; dump-path drain now partitions in place under the existing pending lock. All pending-buffer reads/writes stay under `_pending_chain_invalidations_lock`.
  • Rebuild is generation-guarded (`source_search_ns`) and replace-only: a rebuild never resurrects an entry a dump has already evicted, and a dead-owner rebuild keeps the old chain (the next dump evicts it if dead).
  • EVICT wins when both kinds are queued for the same tag; duplicate REBUILDs merge.
  • New gtests: `ReferenceChainsBfsTest.ImprovedPathInvalidationKeepsLeakChainAcrossTerminalReset` (fails before, passes after — drives the real `heapReferenceCallback` → `improveChain()` path) and `DumpDrainAppliesEvictAndLeavesRebuildToPoll` (EVICT semantics preserved: noise-evicted chains still erased; EVICT beats REBUILD; failed rebuild keeps the old chain).

How to test the change?:

  • `:ddprof-lib:gtestDebug` green (0 FAILED; referenceChains_ut 189/189 including the two new tests).
  • `ThreadLocalLeakReferenceChainTest.shouldCorrelateLeakOnThreadCreatedAfterRecordingStart` on JDK 11 (11.0.32-tem), debug, quiet Linux x64: 10/10 pass with this fix vs ~2/10 failures before.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the `dd:platform-security-review`
    skill, or file a request via the PSEC review form).
    `bewaire` also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16316

improveChain/reparent/root-kind upgrades queued a deferred invalidation that
drainPendingChainEvents applied as an erase at dump time. If the search then
terminated via the canary, nothing could rebuild the erased chains (candidate
and discovered state are zeroed, search pain-budget blocked ~11-15s), losing
the correlation for the rest of the window - against the restartSearch()
contract that resolved chains stay cached across a restart.

Split deferred invalidations into EVICT (noise eviction, filter: erased at
dump time as before) and REBUILD (path changed, source still live: applied by
the BFS-thread poll, which rebuilds the chain from the intact frontier and
keeps the old chain if the rebuild fails). Kind is encoded in-band in the
existing bounded pending buffers; no new allocation or locks.

Reproducer: ReferenceChainsBfsTest.ImprovedPathInvalidationKeepsLeakChainAcrossTerminalReset
(fails before, passes after). PROF-16316.
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Pipelines

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/java-profiler | gtest-release-arm64 — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2277cb5 | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #40173283212 | Commit: 3b0001f | Duration: 18m 53s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-10-25 07:57:13 UTC

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant