2017-05-26 04:48:44 +08:00
|
|
|
/*
|
|
|
|
|
* SystemMonitor.cpp
|
|
|
|
|
*
|
|
|
|
|
* This source file is part of the FoundationDB open source project
|
|
|
|
|
*
|
2026-01-23 02:49:41 +08:00
|
|
|
* Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
2018-02-22 02:25:11 +08:00
|
|
|
*
|
2017-05-26 04:48:44 +08:00
|
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
|
* you may not use this file except in compliance with the License.
|
|
|
|
|
* You may obtain a copy of the License at
|
2018-02-22 02:25:11 +08:00
|
|
|
*
|
2017-05-26 04:48:44 +08:00
|
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
2018-02-22 02:25:11 +08:00
|
|
|
*
|
2017-05-26 04:48:44 +08:00
|
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
|
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
|
* See the License for the specific language governing permissions and
|
|
|
|
|
* limitations under the License.
|
|
|
|
|
*/
|
|
|
|
|
|
2018-10-20 01:30:13 +08:00
|
|
|
#include "flow/flow.h"
|
|
|
|
|
#include "flow/Platform.h"
|
2026-04-02 13:51:35 +08:00
|
|
|
#include "flow/TDMetric.h"
|
2018-10-20 01:30:13 +08:00
|
|
|
#include "flow/SystemMonitor.h"
|
call-site aware memory tracking (#13344)
* Initial memory tracking design doc draft, plus first round of review comments by me with // TODO annotations
* design/memory-tracker: address first-round review TODOs
Resolves all // TODO annotations from the initial draft:
- live-block table now optional via MEMORY_TRACKING_LIVE_TRACKING knob
- drop the mmap slab pool; std::malloc + in-tracker flag is sufficient
- add MEMORY_TRACKING_FORCE_SAMPLE_BYTES so large allocations are always
captured regardless of the count-rate sampler
- add live block / byte totals to MemoryTrackerSummary
- replace the manual coverage spot-check with a sentinel-function unit
test that introspects the aggregation table directly
- leave ALLOC_INSTRUMENTATION alone; new hooks sit next to (not
replacing) existing conditional ones
- drop SJLJ jargon, trim A4 alternative now that force-sample-large
collapses the byte-rate-vs-count-rate question
* flow: add sampled per-call-site memory tracker
Adds a sampled memory attribution layer (flow/MemoryTracker.{h,cpp})
hooked into the three primary allocation paths — global operator
new/delete, FastAllocator, and ArenaBlock::create — plus a periodic
TraceEvent dump driven from SystemMonitor. Knobs gate sample rate,
force-sample threshold, report cadence, top-N, and capture depth;
prod default is off. See design/memory-tracker.md.
Test fixes uncovered while bringing the unit tests up:
- memTrackerResetForTest now resets the per-thread sample counter and
force-sample threshold. Without this, a test that exercised the
off-switch path left gMemTrackerCounter at INT_MAX, which then
silently suppressed sampling for the remainder of the run.
- Slow-path reseed special-cases inverse==1 to keep the counter at 1.
The general formula 1 + r % (2*inverse) yields counter values 1 or
2 at inverse==1, sampling only ~67% of allocations rather than every
one, which broke a test that asserts exact alloc counts.
- Sentinel functions in MemoryTrackerTest.cpp now route the allocated
pointer through an asm-volatile escape() helper. Clang -O3 was
eliding the new/delete pair (P0593 heap fusion), so the test's
allocations never reached the operator-new override.
* design/memory-tracker: address second-round review
- threshold-based reporting (80 MB default, ~1% of 8 GB target RSS)
replaces fixed top-N
- prod report interval 60 s -> 10 min; sim stays 30 s
- single combined MemoryTrackerAddrCmd event with one addr2line
invocation per dump (positional mapping back to sites), keeping
frame 0 — old design's per-site format_backtrace dropped the
leaf alloc frame
- new R12 "Side-thread coverage" + "Side-thread safety" subsection
documenting the FP-elision crash mode found via joshua repros
(RandomUnitTests / IThreadPool seeds segfaulted in captureStackFP
when walking from FastAllocator<N>::~ThreadData into glibc's
FP-elided pthread shutdown machinery) and the stack-bounds
mitigation via pthread_getattr_np
- R7 wording: live bytes (not cumulative)
- CallSite struct in design overview aligned with implementation;
ForceSampledCount promoted from prose-only to struct + emitted
detail; exemplarFrames sized to MEMORY_TRACKER_MAX_FRAMES (=10)
matching the FRAMES knob's stated 1-10 range
- LIVE_TRACKING=false degraded-mode interaction documented
- stale "slab pool" refs removed (std::malloc was already in code)
and fdbserver.cpp self-contradiction resolved
- R3 / Rollout reconciled (table shows steady-state, step 1 lands
at 0)
- 4-6 frame count flagged as initial estimate, subject to refinement
- file:line citations stripped from path references (line numbers
drift; symbol names are stable)
* flow: threshold-based memory tracker reporting + side-thread safety
Implements the second-round design changes in flow/MemoryTracker.{cpp,h},
flow/Knobs.{cpp,h}, flow/SystemMonitor.cpp.
- MEMORY_TRACKING_TOP_N -> MEMORY_TRACKING_REPORT_BYTES_THRESHOLD
(int64_t, default 80,000,000). MEMORY_TRACKING_REPORT_INTERVAL prod
default 60.0 -> 600.0; sim still 30.0.
- memTrackerDump(int topN) -> memTrackerDump(int64_t bytesThreshold).
Filters by liveBytes (or cumulativeBytes when LIVE_TRACKING=false)
>= threshold; emits MemoryTrackerSite per qualifying site plus one
MemoryTrackerAddrCmd event with a single addr2line invocation
covering every qualifying site's frames in dump order. The Summary
event picks up SitesReported and ReportBytesThreshold details.
- AddrCmd is built directly here (not via platform::format_backtrace,
which deliberately drops index 0 for its single-site use case); the
leaf alloc frame is preserved.
- captureFramesFP gains a per-thread stack-bounds check via
pthread_getattr_np + pthread_attr_getstack, cached in TLS. Without
it, walking the FP chain from FastAllocator<N>::~ThreadData into
glibc's FP-elided pthread shutdown machinery follows an
uninitialized saved-FP slot and dereferences garbage. Fixes the
joshua-found segfaults on RandomUnitTests seeds 3288611985,
3731245491, and 2219741568 (all in the IThreadPool worker-exit
path). See design/memory-tracker.md "Side-thread safety".
* flow/MemoryTracker: stub the FP walker on non-Linux
pthread_getattr_np is glibc-specific and the macOS build broke on
it. Frame-pointer walking on macOS is also unreliable on its own
(system runtime has -fomit-frame-pointer in places we can't
control), so a "loose bounds" workaround would still risk crashes.
FDB is required to compile on macOS but is not run in production
there. Gate initStackBoundsForThread + the real captureFramesFP
on __linux__; provide a return-0 stub on non-Linux. The rest of
the tracker (sample counters, aggregation, dump) still compiles
and runs; per-call-site reports on macOS will just lack stack
attribution.
* flow: clang-format fixup for memory-tracker files
Whitespace-only. Catches up flow/Arena.cpp and flow/MemoryTrackerTest.cpp
with the project's clang-format style; the original implementation
commit (d587f82b) slipped these past the format pre-flight.
* edit for clarity, brevity, and uniform voice
* flow/Arena: fix double-tracking on the >256/huge ArenaBlock paths
ArenaBlock::create's >256 and huge paths go through
allocateAndMaybeKeepalive (`new uint8_t[]`), which fires the global
operator new[] hook in addition to the explicit memTrackerOnAlloc
that fires immediately after. Two sites tracked the same pointer;
on free only the explicit-Arena fingerprint was debited, so the
operator new[] fingerprint accumulated liveBytes monotonically and
LiveBytesTotal/LiveBlocksTotal skewed by +n/+1 per arena alloc/free
pair. Reported as B1 in the PR review.
Fix at the Arena layer (so non-arena allocateAndMaybeKeepalive
callers in serialize.h's PacketBuffer code remain attributed at the
operator-new layer): a MemTrackerSuppress RAII helper held across
the underlying new[]/delete[] in ArenaBlock::create and
ArenaBlock::destroyLeaf.
Adds accounting tests for FastAllocator<32>, Arena small, Arena
medium (the B1 path), and Arena huge. The load-bearing assertion is
"exactly one site has the sentinel's frames AND nonzero bytes" --
fails pre-fix for medium/huge with sites=2. Tests gate on __linux__
since captureFramesFP is a no-op on macOS.
* flow/memory-tracker: address PR review follow-ups
- Knobs.cpp: sim default for MEMORY_TRACKING_REPORT_BYTES_THRESHOLD
drops 80 MB -> 1 MB so sim dumps surface more sites for manual
sanity-checking. Prod unchanged.
- B2: memTrackerForEachSite holds MemTrackerSuppress across the
callback loop so callbacks that allocate (e.g. fprintf failure
dumps) don't re-enter tracking under SAMPLE_INVERSE=1.
- B8: drop the always-zero g_reentrantBailouts and its
SamplesDroppedReentry summary detail. Wiring it would require an
atomic in the inline hot path, which R1 forbids. Design doc
updated.
- B3/B6/B7/B9: explanatory comments only -- frame-strip-count
inlining assumption (B3), unstable sort acceptable under R6 (B6),
shared xorshift seed acceptable in practice (B7),
MEMORY_TRACKING_LIVE_TRACKING is startup-only (B9).
* flow/MemoryTracker: one addr2line per site, drop chunking
Move the addr2line command back onto MemoryTrackerSite as a per-site
AddrCmd detail and remove the MemoryTrackerAddrCmd event entirely.
Each AddrCmd carries exactly that site's stack -- short, well under
the trace-detail truncation cap, ready to paste. Replaces the
consolidated-then-chunked-into-byte-buckets approach, which split
stacks across chunk boundaries and made raw events hard to read.
* add standard Apache 2.0 license headers to new memory-tracker files
The three new files (flow/MemoryTracker.{cpp,h} and
flow/MemoryTrackerTest.cpp) shipped without the project-standard
copyright/license block. Adds the standard 19-line header to each;
file-purpose comments stay below it, switched to // line comments
to keep the license block visually distinct.
AGENTS.md gains a short "Source File Headers" section so the next
contributor doesn't repeat the omission.
* address review comments; reduce cost of enable check; maintain net estimates so users dont have to do it manually
* formatting
* unit test bug fix
* gglass review comments on memory-tracker.md design doc
* design doc updates, and fill in a plan for remaining tests/benchmarks
* delete useless simulation section
* flow/bench: add memory-tracker microbenchmarks; record measured overhead
Add flow/bench/BenchMemoryTracker.cpp (Google Benchmark) measuring the tracker's
per-op cost via three benchmarks -- raw malloc/free baseline, end-to-end
operator new/delete, and isolated memTrackerOnAlloc/OnFree -- each at sample
inverse 0 / 100 / 1. Auto-picked up by the flow_bench CONFIGURE_DEPENDS glob;
run with:
bin/flow_bench --benchmark_filter=memtracker
Replace the placeholder "< 5% delta" targets in the design doc's Microbenchmarks
section with the measured numbers and a projected per-second overhead at an
assumed 100K alloc/sec. Headline: ~1.9 ns/pair disabled, ~5.6 ns/pair at the
production 1% rate (~0.056% of a core at 100K/s, ~1/17th of R0's 1% ceiling);
the every-allocation rows are labeled a buggified worst case, not a default.
Testing: built flow_bench on the dev pod (clean, -Werror) and ran the memtracker
filter six times; results stable to within a few percent across runs.
* design/memory-tracker: describe the coverage test as implemented, not proposed
The "Coverage spot-check via sentinel functions" section described the
already-implemented `coverage` test in proposal tense ("Add an introspection
API:", "The unit test:"), which read as future work. Reword to present tense
referencing the actual `coverage`/`*Accounting` tests and the existing
`memTrackerForEachSite` API. No remaining references to unwritten test cases.
* remove extraneous detail from requirements section
* substantially revise microbenchmark results based on seeing 2M allocations/frees on a CPU-maxed storage server
* add script to drive A/B experiment for sampled memory allocation tracking
* flow/MemoryTracker: move global operator new/delete into a server-only TU
The global operator new/delete replacements that route allocations through
the memory tracker lived in flow/MemoryTracker.cpp, i.e. in the flow static
library. flow is linked into libfdb_c and every client binding, so the
interposition shipped into client artifacts and could interpose the whole
host process's allocator even with sampling off.
Move them into a new fdbserver/GlobalNewDelete.cpp, compiled directly into
the fdbserver executable (which clients never link), mirroring where the
legacy ALLOC_INSTRUMENTATION overrides already lived. This also makes the
replaceable-symbol interposition reliable (guaranteed in the final link)
rather than dependent on static-archive pull-in. The legacy
ALLOC_INSTRUMENTATION overrides move into the same file, selected by
#if defined(ALLOC_INSTRUMENTATION) / #else, so exactly one set of global
operators is ever defined. No CMake change is needed (fdbserver/*.cpp is
globbed).
Also reconcile the design doc with the code: override placement, the Files
section (drop the unnecessary flow/CMakeLists.txt edit), the sim knob-table
values (REPORT_BYTES_THRESHOLD, SAMPLE_INVERSE prod default), the
FORCE_SAMPLE_BYTES "-1 disables" sentinel wording, and drop the stale
"buggify inverse to 1" note -- the every-allocation and sampled/weighted
paths are already pinned deterministically by MemoryTrackerTest.cpp.
Testing: build green (run-ccmk5); fdbserver -r unittests -f /flow/MemoryTracker/
runs all 10 memory-tracker unit tests, 0 failed.
* contrib/mako_ab_memtracker: disable RocksDB direct I/O for tmpfs runs
RocksDB opens its DB with O_DIRECT by default; /mnt/ram (tmpfs), where the
harness puts its data dir, does not support direct I/O, so the storage
engine fails to Open and the cluster never configures (fdbcli "configure
new" hangs, mako never starts). Pass the existing
ROCKSDB_USE_DIRECT_READS / ROCKSDB_USE_DIRECT_IO_FLUSH_COMPACTION knobs (=0)
for the rocksdb arm only -- no new knobs added. redwood is unaffected.
* fdbserver/bench: move memory-tracker microbench to fdbserver_bench
The global operator new/delete override lives in fdbserver/GlobalNewDelete.cpp
(server-only, for client isolation), so flow_bench -- which links only flow --
could not exercise it: bench_memtracker_operator_new hit libc++'s operator new
and its Arg(100)/Arg(1) rows were identical to Arg(0).
Move BenchMemoryTracker.cpp to a new fdbserver/bench/ whose CMake compiles
GlobalNewDelete.cpp into the fdbserver_bench executable (via ADDL_SRCS), so the
real override is a strong definition in the bench link and the operator-new
benchmark measures the actual hooked path. It links only flow, not the fdbserver
dependency graph. Adds a BenchMain.cpp (BENCHMARK_MAIN equivalent) and wires the
subdirectory into fdbserver/CMakeLists.txt.
Testing: fdbserver_bench builds; bench_memtracker_operator_new now shows distinct
off / 1% / every-alloc costs (11.4 / 14.4 / 80.9 ns), confirming the override
fires.
* design/memory-tracker: reconcile with code and slim down
Reconcile the doc with the implementation and trim implementation detail that
duplicated the code and had begun to drift:
- Fix drift found in review: degraded-mode live/peak fields stay 0 (not
"tracking the cumulatives"); the dump thresholds on the estimated fields (fix
the memTrackerDump header comment too); the live-block table is allocated but
empty when live-tracking is off (not "never allocated"); list the test file
and its forceLinkMemoryTrackerTests() wiring.
- Slim ~220 lines: replace the CallSite struct, the full captureStackFP source,
both TraceEvent .detail() schemas, and the file-by-file inventory with prose
that defers exact structs/keys/constants to the code; soften the knob table
to intent (authoritative defaults live in Knobs.cpp).
- Refresh the microbench numbers from fdbserver_bench and annotate the host
(AMD EPYC 9R14, clang -O3); replace the A/B placeholder with the measured
-16.5% (redwood) / -9.9% (rocksdb); note it is a point-in-time snapshot.
- Frame R0 as the target the v1 single-lock design does not yet meet (ships off
pending lock sharding); add a one-line note on table teardown/fork behavior.
* fix clang-tidy error
* mako wrapper scripts: take care to clobber the ramdisk before runs to avoid low-space throttling
* design/memory-tracker: note the two mako A/B harnesses and the off-state result
Point at contrib/mako_ab_memtracker.py (sampling off vs on) and
contrib/mako_ab_binaries.py (vanilla main vs this PR built with tracking off),
and record the latter's measured off-state overhead of -0.53% (redwood) /
-0.10% (rocksdb) on a shared base commit — well within R0's 1% ceiling.
* Add the default code review guidance to AGENTS.md
* address big brother review comments; add some braces and trim some generated comments (hard to believe, but true)
* clang-tidy again
* Final read-through of this PR.
-- Add or enhance a few comments on important items (performance & reliability)
-- Delete some misc agent-written comments, typically exhibiting recency bias e.g. naming bugs identified in code review passes or describing mundane earlier bugs
-- Add braces (InsertBraces style)
* memory-tracker: address round-4 review
Correctness/robustness:
- operator new now runs the installed std::new_handler retry loop, so an
allocation failure (including the tracker's own map growth) reaches FDB's
platform::outOfMemory / FDB_EXIT_NO_MEM path instead of throwing past it.
- Reentrancy guard restored via MemTrackerSuppress RAII on every path (hot path,
dump, reset) so an exception can't permanently disable tracking on a thread.
- Sampling reseed draws from [1, 2N-1] (mean exactly N); the old [1, 2N] biased
the Est* estimate low by ~0.5/N.
Design:
- Drop runtime enable/disable (now an explicit Non-requirement): the sample knob
is read at startup only; park the counter when off. Removes the DISABLED_RESEED
re-park and the on/off reconciliation logic.
- Simulation samples 1-in-10 (was 1-in-2).
Portability (Windows is not build-tested here; lean on existing abstractions):
- Aligned operator new uses platform::aligned_alloc/aligned_free (overflow-guarded)
instead of posix_memalign; guard <pthread.h> under __linux__; add a
force_noinline macro (GNU-only, empty elsewhere); portable volatile-sink
escape() in the test.
Tooling/tests/docs:
- mako A/B scripts: validate the scratch mount (realpath + tmpfs + denylist) before
rm -rf, locate mako_storage_bench.sh via __file__, and black-format.
- Add operatorNewHonorsNewHandler and samplingRate tests; drop enableAfterOff.
- Reconcile the design doc with the code; prune low-value comments.
Testing:
- /flow/MemoryTracker/* unit tests: 11 pass, 0 fail (bin/fdbserver -r unittests).
- Joshua 100k (correctness-8.0.0): 99,995 pass / 1 fail. The single failure is
NativeCdcAssignmentPublication -- a NativeCdc test from main, not this PR
(CommitProxy failed_to_progress -> QuietDatabase DataDistributionActive ->
NativeCdcEndToEnd workload start timed_out). It reproduces bit-identically
(seed 2939264355, unseed 8776) with the tracker built off (sim
sample_inverse=0, confirmed via the MemoryTrackerSummary SampleInverse trace
field), so it is a pre-existing rare CDC/QuietDatabase flake unrelated to this
change.
* memory-tracker: fix GCC IPA-clone breaking frame-attribution tests
Under GCC -O3, IPA constant-propagation cloning (-fipa-cp-clone) specializes the
MemoryTrackerTest sentinels (each called with a constant N) into `.constprop`
clones emitted at a different address than the function symbol. The executed
code -- and thus the captured return addresses -- live in the clone, so the
tests' frameInside(frame, &sentinel) window missed them: fastAlloc32Accounting
aborted (sitesWithSentinelFrames == 0) in GCC CI while clang passed.
Add `noclone` to force_noinline on GCC so each sentinel stays a single body at
the address &fn yields. clang doesn't support noclone (and doesn't clone this
way), so it keeps noinline only; other compilers stay empty.
Testing: /flow/MemoryTracker/* passes 11/11 under both a gcc-toolset-13 build
and a clang build.
* memory-tracker: cheaper disabled alloc hot path (per-thread off flag)
When sampling is off, memTrackerOnAlloc previously still did three TLS accesses
every allocation -- a gInMemTracker load, a gMemTrackerCounter load-decrement-
STORE, and a gForceSampleBytes load. Add a per-thread gMemTrackerOff flag,
checked first, that a thread sets once its slow path observes sampling is off;
the disabled alloc path then short-circuits on a single TLS load + branch (no
counter store, no gForceSampleBytes load).
The flag is per-thread, not global, on purpose: the counter still bootstraps
sampling per thread (first alloc reaches the slow path and reads the knob), so a
single global gate set by an early main-thread allocation before FLOW_KNOBS is
ready would wrongly disable worker threads that bootstrap later. Free keeps the
global g_memTrackerEnabled gate (a free-only thread must see global state to
debit). memTrackerResetForTest clears the flag. Removes the now-dead INT_MAX
counter parking.
The saving is real but small (~2 loads + 1 store per alloc, well under the
mako A/B's run-to-run noise), so the redwood off-state A/B shows no resolvable
change; the win is on principle / at high allocation rates.
Testing: /flow/MemoryTracker/* passes 11/11 (bin/fdbserver -r unittests).
* memory-tracker: add FDB_MEMORY_TRACKER compile-time gate + per-path microbench
Add a compile-time switch, FDB_MEMORY_TRACKER (CMake option, default ON), that
removes the feature entirely when set to 0: the header hooks become no-op inlines,
flow/MemoryTracker.cpp and the MemoryTrackerTest TEST_CASEs are #if'd out (the
forceLink stub stays), and fdbserver/GlobalNewDelete.cpp defines no global
operator new/delete override (libc++'s allocator is used, which still honors the
installed new_handler). This gives operators an escape hatch to zero
always-compiled footprint, and lets the microbench measure present-but-disabled
vs absent cost per allocation path.
Extend fdbserver/bench/BenchMemoryTracker.cpp with plain per-path alloc/free loops
(operator new[] at several sizes, FastAllocator<64/96/256>, Arena medium/huge) so
the same binary built =1 (tracker present, sampling off by default) vs =0 (absent)
isolates each path's unweighted off-state overhead.
Measured (EPYC 9R14 @ 3.7 GHz, ns/op, =1 minus =0):
operator new[] small ~+1.1 ns (~11%); huge ~+2.5 ns
FastAllocator<N> ~0 (within the ~0.6 ns cross-build noise floor)
Arena block ~+3.3 ns (medium) / +4.8 ns (huge)
The per-op-costly paths (operator new, Arena) are the low-volume ones; the
high-volume path (FastAllocator, ~82% of redwood allocs) is ~free, so the weighted
off-state overhead is ~0.1-0.2% of a core -- under the R0 target, though R0 is now
framed as an unproven target with this gate as the escape hatch.
Testing: /flow/MemoryTracker/* passes 11/11 (default build); FDB_MEMORY_TRACKER=OFF
builds clean under -Werror.
* clang-tidy fix
* comment about why fdbserver/bench exists
* address Codex round 5 review comments (MSVC build, startup sequencing, memory allocation failures on tracker-internal bookkeeping, cross-compile, contrib script cleanup)
* add another disclaimer about MSVC/Windows being best effort
* memory-tracker: fail-open ordering + deflake huge-arena unit test
memTrackerSampleAlloc now performs both table insertions (aggregation-map and
live-map nodes) before mutating any per-site or global counter, so if the
tracker's own map growth throws std::bad_alloc the exception unwinds with all
totals in lockstep and the fail-open catch simply drops the sample. This
addresses an adversarial-review finding. Note such a failure does not occur in
practice: FDB "OOM" is an RSS threshold enforced by fdbmonitor (typically
~12-16 GB against an ~8 GB target), not a malloc/operator-new failure. This
feature is for finding leaks and untuned allocations that drive RSS growth,
well short of any allocation failure -- documented in flow/MemoryTracker.cpp and
design/memory-tracker.md so reviewers don't over-index on the bad_alloc path.
arenaHugeAccounting now identifies the huge blocks by their ~100 KB size
signature instead of the frame-pointer sentinel. The huge-Arena path is the
deepest tracked call chain; under a real (non-simulation) network the
best-effort frame-pointer walker is per-allocation nondeterministic there and
cannot reliably attribute the blocks to the test's frame, so the sentinel-frame
assertion flaked (~1-4% of runs). The recorded block size is correct regardless
of which frames were captured, no incidental or foreign-thread allocation comes
near 100 KB, and double-tracking still shows up as 2N. The shallower
fastAlloc32/arenaSmall/arenaMedium/operatorNew accounting tests keep exact
sentinel-frame attribution (reliable on their shorter paths) as the regression
guard.
Testing:
- /flow/MemoryTracker/* via `fdbserver -r unittests`: 500 runs across two seed
spaces (including the sequence that previously flaked), 14/14 test cases pass
every run, 0 failures.
- Builds clean with FDB_MEMORY_TRACKER on and off under -Werror; clang-format
and clang-tidy clean on the changed files.
- Joshua 100k on the parent commit: ensemble
20260728-160205-gglass-0b30fa74f2605807, ended=100000 pass=100000 fail=0.
These changes are a pure counter-update reorder (no simulation-reachable
behavior change -- bad_alloc is not injected in sim), a unit-test-only change,
and documentation, so that result remains representative.
* memory-tracker: note why the fdbserver-local bench binary exists
Add a short comment to BenchMain.cpp explaining that most microbenchmarks
belong in flow/bench and this fdbserver-local benchmark binary exists only for
benchmarks that must link fdbserver-only code, pointing at BenchMemoryTracker.cpp
for the detailed rationale rather than duplicating it.
2026-08-01 01:12:50 +08:00
|
|
|
#include "flow/Knobs.h"
|
|
|
|
|
#include "flow/MemoryTracker.h"
|
2017-05-26 04:48:44 +08:00
|
|
|
|
|
|
|
|
#if defined(ALLOC_INSTRUMENTATION) && defined(__linux__)
|
|
|
|
|
#include <cxxabi.h>
|
|
|
|
|
#endif
|
|
|
|
|
|
2023-01-25 05:04:47 +08:00
|
|
|
#ifdef ADDRESS_SANITIZER
|
|
|
|
|
#include <sanitizer/asan_interface.h>
|
|
|
|
|
#endif
|
|
|
|
|
|
2017-05-26 04:48:44 +08:00
|
|
|
SystemMonitorMachineState machineState;
|
|
|
|
|
|
|
|
|
|
void initializeSystemMonitorMachineState(SystemMonitorMachineState machineState) {
|
|
|
|
|
::machineState = machineState;
|
|
|
|
|
|
|
|
|
|
ASSERT(g_network);
|
|
|
|
|
::machineState.monitorStartTime = now();
|
|
|
|
|
}
|
|
|
|
|
|
2021-10-07 07:16:24 +08:00
|
|
|
double machineStartTime() {
|
|
|
|
|
return ::machineState.monitorStartTime;
|
|
|
|
|
}
|
|
|
|
|
|
2026-02-25 04:58:55 +08:00
|
|
|
void NetworkData::init() {
|
|
|
|
|
bytesSent = Int64Metric::getValueOrDefault("Net2.BytesSent"_sr);
|
|
|
|
|
countPacketsReceived = Int64Metric::getValueOrDefault("Net2.CountPacketsReceived"_sr);
|
|
|
|
|
countPacketsGenerated = Int64Metric::getValueOrDefault("Net2.CountPacketsGenerated"_sr);
|
|
|
|
|
bytesReceived = Int64Metric::getValueOrDefault("Net2.BytesReceived"_sr);
|
|
|
|
|
countWriteProbes = Int64Metric::getValueOrDefault("Net2.CountWriteProbes"_sr);
|
|
|
|
|
countReadProbes = Int64Metric::getValueOrDefault("Net2.CountReadProbes"_sr);
|
|
|
|
|
countReads = Int64Metric::getValueOrDefault("Net2.CountReads"_sr);
|
|
|
|
|
countWouldBlock = Int64Metric::getValueOrDefault("Net2.CountWouldBlock"_sr);
|
|
|
|
|
countWrites = Int64Metric::getValueOrDefault("Net2.CountWrites"_sr);
|
|
|
|
|
countRunLoop = Int64Metric::getValueOrDefault("Net2.CountRunLoop"_sr);
|
|
|
|
|
countCantSleep = Int64Metric::getValueOrDefault("Net2.CountCantSleep"_sr);
|
|
|
|
|
countWontSleep = Int64Metric::getValueOrDefault("Net2.CountWontSleep"_sr);
|
|
|
|
|
countTimers = Int64Metric::getValueOrDefault("Net2.CountTimers"_sr);
|
|
|
|
|
countTasks = Int64Metric::getValueOrDefault("Net2.CountTasks"_sr);
|
|
|
|
|
countYields = Int64Metric::getValueOrDefault("Net2.CountYields"_sr);
|
|
|
|
|
countYieldBigStack = Int64Metric::getValueOrDefault("Net2.CountYieldBigStack"_sr);
|
|
|
|
|
countYieldCalls = Int64Metric::getValueOrDefault("Net2.CountYieldCalls"_sr);
|
|
|
|
|
countASIOEvents = Int64Metric::getValueOrDefault("Net2.CountASIOEvents"_sr);
|
|
|
|
|
countYieldCallsTrue = Int64Metric::getValueOrDefault("Net2.CountYieldCallsTrue"_sr);
|
|
|
|
|
countRunLoopProfilingSignals = Int64Metric::getValueOrDefault("Net2.CountRunLoopProfilingSignals"_sr);
|
|
|
|
|
countConnEstablished = Int64Metric::getValueOrDefault("Net2.CountConnEstablished"_sr);
|
|
|
|
|
countConnClosedWithError = Int64Metric::getValueOrDefault("Net2.CountConnClosedWithError"_sr);
|
|
|
|
|
countConnClosedWithoutError = Int64Metric::getValueOrDefault("Net2.CountConnClosedWithoutError"_sr);
|
|
|
|
|
countTLSPolicyFailures = Int64Metric::getValueOrDefault("Net2.CountTLSPolicyFailures"_sr);
|
|
|
|
|
countLaunchTime = DoubleMetric::getValueOrDefault("Net2.CountLaunchTime"_sr);
|
|
|
|
|
countReactTime = DoubleMetric::getValueOrDefault("Net2.CountReactTime"_sr);
|
|
|
|
|
countFileLogicalWrites = Int64Metric::getValueOrDefault("AsyncFile.CountLogicalWrites"_sr);
|
|
|
|
|
countFileLogicalReads = Int64Metric::getValueOrDefault("AsyncFile.CountLogicalReads"_sr);
|
|
|
|
|
countAIOSubmit = Int64Metric::getValueOrDefault("AsyncFile.CountAIOSubmit"_sr);
|
|
|
|
|
countAIOCollect = Int64Metric::getValueOrDefault("AsyncFile.CountAIOCollect"_sr);
|
|
|
|
|
countFileCacheWrites = Int64Metric::getValueOrDefault("AsyncFile.CountCacheWrites"_sr);
|
|
|
|
|
countFileCacheReads = Int64Metric::getValueOrDefault("AsyncFile.CountCacheReads"_sr);
|
|
|
|
|
countFileCacheWritesBlocked = Int64Metric::getValueOrDefault("AsyncFile.CountCacheWritesBlocked"_sr);
|
|
|
|
|
countFileCacheReadsBlocked = Int64Metric::getValueOrDefault("AsyncFile.CountCacheReadsBlocked"_sr);
|
|
|
|
|
countFileCachePageReadsMerged = Int64Metric::getValueOrDefault("AsyncFile.CountCachePageReadsMerged"_sr);
|
|
|
|
|
countFileCacheFinds = Int64Metric::getValueOrDefault("AsyncFile.CountCacheFinds"_sr);
|
|
|
|
|
countFileCacheReadBytes = Int64Metric::getValueOrDefault("AsyncFile.CountCacheReadBytes"_sr);
|
|
|
|
|
countFilePageCacheHits = Int64Metric::getValueOrDefault("AsyncFile.CountCachePageReadsHit"_sr);
|
|
|
|
|
countFilePageCacheMisses = Int64Metric::getValueOrDefault("AsyncFile.CountCachePageReadsMissed"_sr);
|
|
|
|
|
countFilePageCacheEvictions = Int64Metric::getValueOrDefault("EvictablePageCache.CacheEvictions"_sr);
|
|
|
|
|
}
|
|
|
|
|
|
2017-05-26 04:48:44 +08:00
|
|
|
void systemMonitor() {
|
|
|
|
|
static StatisticsState statState = StatisticsState();
|
2022-02-26 04:54:31 +08:00
|
|
|
#if !DEBUG_DETERMINISM
|
2017-05-26 04:48:44 +08:00
|
|
|
customSystemMonitor("ProcessMetrics", &statState, true);
|
2022-02-26 04:54:31 +08:00
|
|
|
#endif
|
2017-05-26 04:48:44 +08:00
|
|
|
}
|
|
|
|
|
|
2019-02-26 05:45:53 +08:00
|
|
|
SystemStatistics getSystemStatistics() {
|
|
|
|
|
static StatisticsState statState = StatisticsState();
|
2019-02-27 10:04:03 +08:00
|
|
|
const IPAddress ipAddr = machineState.ip.present() ? machineState.ip.get() : IPAddress();
|
2019-02-26 05:45:53 +08:00
|
|
|
return getSystemStatistics(
|
2019-04-08 13:55:19 +08:00
|
|
|
machineState.folder.present() ? machineState.folder.get() : "", &ipAddr, &statState.systemState, false);
|
2019-02-26 05:45:53 +08:00
|
|
|
}
|
|
|
|
|
|
2022-09-13 02:12:26 +08:00
|
|
|
#define TRACEALLOCATOR(size) \
|
|
|
|
|
TraceEvent("MemSample") \
|
|
|
|
|
.detail("Count", FastAllocator<size>::getApproximateMemoryUnused() / size) \
|
|
|
|
|
.detail("TotalSize", FastAllocator<size>::getApproximateMemoryUnused()) \
|
|
|
|
|
.detail("SampleCount", 1) \
|
|
|
|
|
.detail("Hash", "FastAllocatedUnused" #size) \
|
|
|
|
|
.detail("Bt", "na")
|
|
|
|
|
#define DETAILALLOCATORMEMUSAGE(size) \
|
|
|
|
|
detail("TotalMemory" #size, FastAllocator<size>::getTotalMemory()) \
|
|
|
|
|
.detail("ApproximateUnusedMemory" #size, FastAllocator<size>::getApproximateMemoryUnused()) \
|
|
|
|
|
.detail("ActiveThreads" #size, FastAllocator<size>::getActiveThreads())
|
|
|
|
|
|
2022-09-10 02:15:24 +08:00
|
|
|
namespace {
|
|
|
|
|
|
2022-09-13 02:12:26 +08:00
|
|
|
#ifdef __linux__
|
|
|
|
|
// Converts cgroup key, e.g. nr_periods, to NrPeriods
|
2022-09-10 02:15:24 +08:00
|
|
|
std::string capitalizeCgroupKey(const std::string& key) {
|
|
|
|
|
bool wordStart = true;
|
|
|
|
|
std::string result;
|
|
|
|
|
result.reserve(key.size());
|
|
|
|
|
|
|
|
|
|
for (const char ch : key) {
|
|
|
|
|
if (std::isalnum(ch)) {
|
|
|
|
|
if (wordStart) {
|
|
|
|
|
result.push_back(std::toupper(ch));
|
|
|
|
|
wordStart = false;
|
|
|
|
|
} else {
|
|
|
|
|
result.push_back(ch);
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
// Skip non-alnum characters
|
|
|
|
|
wordStart = true;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return result;
|
|
|
|
|
}
|
2022-09-13 02:12:26 +08:00
|
|
|
#endif // __linux__
|
2022-09-10 02:15:24 +08:00
|
|
|
|
|
|
|
|
} // anonymous namespace
|
|
|
|
|
|
2021-06-10 02:37:14 +08:00
|
|
|
SystemStatistics customSystemMonitor(std::string const& eventName, StatisticsState* statState, bool machineMetrics) {
|
2019-02-27 10:04:03 +08:00
|
|
|
const IPAddress ipAddr = machineState.ip.present() ? machineState.ip.get() : IPAddress();
|
|
|
|
|
SystemStatistics currentStats = getSystemStatistics(
|
|
|
|
|
machineState.folder.present() ? machineState.folder.get() : "", &ipAddr, &statState->systemState, true);
|
2017-05-26 04:48:44 +08:00
|
|
|
NetworkData netData;
|
|
|
|
|
netData.init();
|
2020-11-05 09:14:44 +08:00
|
|
|
if (!g_network->isSimulated() && currentStats.initialized) {
|
2017-05-26 04:48:44 +08:00
|
|
|
{
|
2019-06-27 05:03:02 +08:00
|
|
|
TraceEvent(eventName.c_str())
|
2020-10-31 02:20:40 +08:00
|
|
|
.detail("Elapsed", currentStats.elapsed)
|
|
|
|
|
.detail("CPUSeconds", currentStats.processCPUSeconds)
|
|
|
|
|
.detail("MainThreadCPUSeconds", currentStats.mainThreadCPUSeconds)
|
|
|
|
|
.detail("UptimeSeconds", now() - machineState.monitorStartTime)
|
|
|
|
|
.detail("Memory", currentStats.processMemory)
|
|
|
|
|
.detail("ResidentMemory", currentStats.processResidentMemory)
|
|
|
|
|
.detail("UnusedAllocatedMemory", getTotalUnusedAllocatedMemory())
|
|
|
|
|
.detail("MbpsSent",
|
|
|
|
|
((netData.bytesSent - statState->networkState.bytesSent) * 8e-6) / currentStats.elapsed)
|
|
|
|
|
.detail("MbpsReceived",
|
|
|
|
|
((netData.bytesReceived - statState->networkState.bytesReceived) * 8e-6) / currentStats.elapsed)
|
|
|
|
|
.detail("DiskTotalBytes", currentStats.processDiskTotalBytes)
|
|
|
|
|
.detail("DiskFreeBytes", currentStats.processDiskFreeBytes)
|
|
|
|
|
.detail("DiskQueueDepth", currentStats.processDiskQueueDepth)
|
|
|
|
|
.detail("DiskIdleSeconds", currentStats.processDiskIdleSeconds)
|
|
|
|
|
.detail("DiskReads", currentStats.processDiskRead)
|
2022-01-26 07:30:43 +08:00
|
|
|
.detail("DiskReadSeconds", currentStats.processDiskReadSeconds)
|
2020-10-31 02:20:40 +08:00
|
|
|
.detail("DiskWrites", currentStats.processDiskWrite)
|
2022-01-26 07:30:43 +08:00
|
|
|
.detail("DiskWriteSeconds", currentStats.processDiskWriteSeconds)
|
2020-10-31 02:20:40 +08:00
|
|
|
.detail("DiskReadsCount", currentStats.processDiskReadCount)
|
|
|
|
|
.detail("DiskWritesCount", currentStats.processDiskWriteCount)
|
|
|
|
|
.detail("DiskReadSectors", currentStats.processDiskReadSectors)
|
2025-07-16 05:02:10 +08:00
|
|
|
.detail("DiskWriteSectors", currentStats.processDiskWriteSectors)
|
|
|
|
|
.detail("DiskReadBytes", currentStats.processDiskReadBytes)
|
|
|
|
|
.detail("DiskWriteBytes", currentStats.processDiskWriteBytes)
|
|
|
|
|
|
2020-10-31 02:20:40 +08:00
|
|
|
.detail("FileWrites", netData.countFileLogicalWrites - statState->networkState.countFileLogicalWrites)
|
|
|
|
|
.detail("FileReads", netData.countFileLogicalReads - statState->networkState.countFileLogicalReads)
|
|
|
|
|
.detail("CacheReadBytes",
|
|
|
|
|
netData.countFileCacheReadBytes - statState->networkState.countFileCacheReadBytes)
|
|
|
|
|
.detail("CacheFinds", netData.countFileCacheFinds - statState->networkState.countFileCacheFinds)
|
|
|
|
|
.detail("CacheWritesBlocked",
|
|
|
|
|
netData.countFileCacheWritesBlocked - statState->networkState.countFileCacheWritesBlocked)
|
|
|
|
|
.detail("CacheReadsBlocked",
|
|
|
|
|
netData.countFileCacheReadsBlocked - statState->networkState.countFileCacheReadsBlocked)
|
|
|
|
|
.detail("CachePageReadsMerged",
|
|
|
|
|
netData.countFileCachePageReadsMerged - statState->networkState.countFileCachePageReadsMerged)
|
|
|
|
|
.detail("CacheWrites", netData.countFileCacheWrites - statState->networkState.countFileCacheWrites)
|
|
|
|
|
.detail("CacheReads", netData.countFileCacheReads - statState->networkState.countFileCacheReads)
|
|
|
|
|
.detail("CacheHits", netData.countFilePageCacheHits - statState->networkState.countFilePageCacheHits)
|
|
|
|
|
.detail("CacheMisses",
|
|
|
|
|
netData.countFilePageCacheMisses - statState->networkState.countFilePageCacheMisses)
|
|
|
|
|
.detail("CacheEvictions",
|
|
|
|
|
netData.countFilePageCacheEvictions - statState->networkState.countFilePageCacheEvictions)
|
|
|
|
|
.detail("DCID", machineState.dcId)
|
|
|
|
|
.detail("ZoneID", machineState.zoneId)
|
|
|
|
|
.detail("MachineID", machineState.machineId)
|
2022-12-13 05:16:30 +08:00
|
|
|
.detail("Version", machineState.fdbVersion)
|
2020-10-31 02:20:40 +08:00
|
|
|
.detail("AIOSubmitCount", netData.countAIOSubmit - statState->networkState.countAIOSubmit)
|
|
|
|
|
.detail("AIOCollectCount", netData.countAIOCollect - statState->networkState.countAIOCollect)
|
|
|
|
|
.detail("AIOSubmitLag",
|
|
|
|
|
(g_network->networkInfo.metrics.secSquaredSubmit -
|
|
|
|
|
statState->networkMetricsState.secSquaredSubmit) /
|
|
|
|
|
currentStats.elapsed)
|
|
|
|
|
.detail("AIODiskStall",
|
|
|
|
|
(g_network->networkInfo.metrics.secSquaredDiskStall -
|
|
|
|
|
statState->networkMetricsState.secSquaredDiskStall) /
|
|
|
|
|
currentStats.elapsed)
|
|
|
|
|
.detail("CurrentConnections",
|
|
|
|
|
netData.countConnEstablished - netData.countConnClosedWithError -
|
|
|
|
|
netData.countConnClosedWithoutError)
|
|
|
|
|
.detail("ConnectionsEstablished",
|
|
|
|
|
(double)(netData.countConnEstablished - statState->networkState.countConnEstablished) /
|
|
|
|
|
currentStats.elapsed)
|
|
|
|
|
.detail("ConnectionsClosed",
|
|
|
|
|
((netData.countConnClosedWithError - statState->networkState.countConnClosedWithError) +
|
|
|
|
|
(netData.countConnClosedWithoutError - statState->networkState.countConnClosedWithoutError)) /
|
|
|
|
|
currentStats.elapsed)
|
|
|
|
|
.detail("ConnectionErrors",
|
|
|
|
|
(netData.countConnClosedWithError - statState->networkState.countConnClosedWithError) /
|
|
|
|
|
currentStats.elapsed)
|
|
|
|
|
.detail("TLSPolicyFailures",
|
|
|
|
|
(netData.countTLSPolicyFailures - statState->networkState.countTLSPolicyFailures) /
|
|
|
|
|
currentStats.elapsed)
|
2025-07-23 11:41:31 +08:00
|
|
|
|
2020-10-31 02:20:40 +08:00
|
|
|
.trackLatest(eventName);
|
2018-03-07 07:52:03 +08:00
|
|
|
|
|
|
|
|
TraceEvent("MemoryMetrics")
|
2020-10-31 02:20:40 +08:00
|
|
|
.DETAILALLOCATORMEMUSAGE(16)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(32)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(64)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(96)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(128)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(256)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(512)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(1024)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(2048)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(4096)
|
|
|
|
|
.DETAILALLOCATORMEMUSAGE(8192)
|
2022-03-22 04:30:27 +08:00
|
|
|
.DETAILALLOCATORMEMUSAGE(16384)
|
2020-10-31 02:20:40 +08:00
|
|
|
.detail("HugeArenaMemory", g_hugeArenaMemory.load())
|
|
|
|
|
.detail("DCID", machineState.dcId)
|
|
|
|
|
.detail("ZoneID", machineState.zoneId)
|
|
|
|
|
.detail("MachineID", machineState.machineId);
|
2018-03-07 07:52:03 +08:00
|
|
|
|
2021-10-12 06:06:43 +08:00
|
|
|
uint64_t total_memory = 0;
|
|
|
|
|
total_memory += FastAllocator<16>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<32>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<64>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<96>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<128>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<256>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<512>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<1024>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<2048>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<4096>::getTotalMemory();
|
|
|
|
|
total_memory += FastAllocator<8192>::getTotalMemory();
|
2022-03-22 04:30:27 +08:00
|
|
|
total_memory += FastAllocator<16384>::getTotalMemory();
|
2021-10-12 06:06:43 +08:00
|
|
|
|
|
|
|
|
uint64_t unused_memory = 0;
|
|
|
|
|
unused_memory += FastAllocator<16>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<32>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<64>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<96>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<128>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<256>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<512>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<1024>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<2048>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<4096>::getApproximateMemoryUnused();
|
|
|
|
|
unused_memory += FastAllocator<8192>::getApproximateMemoryUnused();
|
2022-03-22 04:30:27 +08:00
|
|
|
unused_memory += FastAllocator<16384>::getApproximateMemoryUnused();
|
2021-10-12 06:06:43 +08:00
|
|
|
|
|
|
|
|
if (total_memory > 0) {
|
|
|
|
|
TraceEvent("FastAllocMemoryUsage")
|
|
|
|
|
.detail("TotalMemory", total_memory)
|
|
|
|
|
.detail("UnusedMemory", unused_memory)
|
|
|
|
|
.detail("Utilization", format("%f%%", (total_memory - unused_memory) * 100.0 / total_memory));
|
|
|
|
|
}
|
|
|
|
|
|
2018-03-07 07:52:03 +08:00
|
|
|
TraceEvent n("NetworkMetrics");
|
2020-10-31 02:20:40 +08:00
|
|
|
n.detail("Elapsed", currentStats.elapsed)
|
|
|
|
|
.detail("CantSleep", netData.countCantSleep - statState->networkState.countCantSleep)
|
|
|
|
|
.detail("WontSleep", netData.countWontSleep - statState->networkState.countWontSleep)
|
|
|
|
|
.detail("Yields", netData.countYields - statState->networkState.countYields)
|
|
|
|
|
.detail("YieldCalls", netData.countYieldCalls - statState->networkState.countYieldCalls)
|
|
|
|
|
.detail("YieldCallsTrue", netData.countYieldCallsTrue - statState->networkState.countYieldCallsTrue)
|
2020-11-17 02:15:23 +08:00
|
|
|
.detail("RunLoopProfilingSignals",
|
|
|
|
|
netData.countRunLoopProfilingSignals - statState->networkState.countRunLoopProfilingSignals)
|
2020-10-31 02:20:40 +08:00
|
|
|
.detail("YieldBigStack", netData.countYieldBigStack - statState->networkState.countYieldBigStack)
|
|
|
|
|
.detail("RunLoopIterations", netData.countRunLoop - statState->networkState.countRunLoop)
|
|
|
|
|
.detail("TimersExecuted", netData.countTimers - statState->networkState.countTimers)
|
|
|
|
|
.detail("TasksExecuted", netData.countTasks - statState->networkState.countTasks)
|
|
|
|
|
.detail("ASIOEventsProcessed", netData.countASIOEvents - statState->networkState.countASIOEvents)
|
|
|
|
|
.detail("ReadCalls", netData.countReads - statState->networkState.countReads)
|
|
|
|
|
.detail("WriteCalls", netData.countWrites - statState->networkState.countWrites)
|
|
|
|
|
.detail("ReadProbes", netData.countReadProbes - statState->networkState.countReadProbes)
|
|
|
|
|
.detail("WriteProbes", netData.countWriteProbes - statState->networkState.countWriteProbes)
|
|
|
|
|
.detail("PacketsRead", netData.countPacketsReceived - statState->networkState.countPacketsReceived)
|
|
|
|
|
.detail("PacketsGenerated",
|
|
|
|
|
netData.countPacketsGenerated - statState->networkState.countPacketsGenerated)
|
|
|
|
|
.detail("WouldBlock", netData.countWouldBlock - statState->networkState.countWouldBlock)
|
|
|
|
|
.detail("LaunchTime", netData.countLaunchTime - statState->networkState.countLaunchTime)
|
|
|
|
|
.detail("ReactTime", netData.countReactTime - statState->networkState.countReactTime)
|
|
|
|
|
.detail("DCID", machineState.dcId)
|
|
|
|
|
.detail("ZoneID", machineState.zoneId)
|
|
|
|
|
.detail("MachineID", machineState.machineId);
|
2018-03-07 07:52:03 +08:00
|
|
|
|
2019-06-27 05:03:02 +08:00
|
|
|
for (int i = 0; i < NetworkMetrics::SLOW_EVENT_BINS; i++) {
|
2020-01-13 08:44:30 +08:00
|
|
|
if (int c = g_network->networkInfo.metrics.countSlowEvents[i] -
|
|
|
|
|
statState->networkMetricsState.countSlowEvents[i]) {
|
2018-06-09 04:57:00 +08:00
|
|
|
n.detail(format("SlowTask%dM", 1 << i).c_str(), c);
|
2019-06-27 05:03:02 +08:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2020-02-08 06:34:24 +08:00
|
|
|
std::map<TaskPriority, double> loggedDurations;
|
|
|
|
|
for (auto& itr : g_network->networkInfo.metrics.activeTrackers) {
|
|
|
|
|
if (itr.second.active) {
|
|
|
|
|
itr.second.duration += now() - itr.second.windowedTimer;
|
|
|
|
|
itr.second.windowedTimer = now();
|
2019-06-27 05:03:02 +08:00
|
|
|
}
|
|
|
|
|
|
2020-02-08 06:34:24 +08:00
|
|
|
if (itr.second.duration / currentStats.elapsed >= FLOW_KNOBS->MIN_LOGGED_PRIORITY_BUSY_FRACTION) {
|
|
|
|
|
loggedDurations[itr.first] = std::min(currentStats.elapsed, itr.second.duration);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
itr.second.duration = 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
for (auto const& itr : loggedDurations) {
|
2020-10-17 04:42:36 +08:00
|
|
|
// PriorityBusyX measures the amount of time spent busy at exactly priority X
|
2020-02-08 06:34:24 +08:00
|
|
|
n.detail(format("PriorityBusy%d", itr.first).c_str(), itr.second);
|
|
|
|
|
}
|
|
|
|
|
|
2020-09-01 00:21:44 +08:00
|
|
|
bool firstTracker = true;
|
2020-02-08 06:34:24 +08:00
|
|
|
for (auto& itr : g_network->networkInfo.metrics.starvationTrackers) {
|
|
|
|
|
if (itr.active) {
|
|
|
|
|
itr.duration += now() - itr.windowedTimer;
|
|
|
|
|
itr.maxDuration = std::max(itr.maxDuration, now() - itr.timer);
|
|
|
|
|
itr.windowedTimer = now();
|
|
|
|
|
}
|
|
|
|
|
|
2020-10-17 04:42:36 +08:00
|
|
|
// PriorityStarvedBelowX: how much of the elapsed time we were running tasks at a priority at or above X
|
|
|
|
|
// PriorityMaxStarvedBelowX: The longest single span of time that you were starved below that priority,
|
|
|
|
|
// which could tell you if you are doing work in bursts.
|
2020-02-08 06:34:24 +08:00
|
|
|
n.detail(format("PriorityStarvedBelow%d", itr.priority).c_str(),
|
|
|
|
|
std::min(currentStats.elapsed, itr.duration));
|
|
|
|
|
n.detail(format("PriorityMaxStarvedBelow%d", itr.priority).c_str(), itr.maxDuration);
|
2019-12-18 01:14:54 +08:00
|
|
|
|
2020-09-01 00:21:44 +08:00
|
|
|
if (firstTracker) {
|
|
|
|
|
g_network->networkInfo.metrics.lastRunLoopBusyness =
|
|
|
|
|
std::min(currentStats.elapsed, itr.duration) / currentStats.elapsed;
|
|
|
|
|
firstTracker = false;
|
|
|
|
|
}
|
|
|
|
|
|
2020-02-08 06:34:24 +08:00
|
|
|
itr.duration = 0;
|
|
|
|
|
itr.maxDuration = 0;
|
2019-06-27 05:03:02 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
n.trackLatest("NetworkMetrics");
|
2017-05-26 04:48:44 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (machineMetrics) {
|
2022-09-10 02:15:24 +08:00
|
|
|
auto traceEvent = TraceEvent("MachineMetrics");
|
|
|
|
|
traceEvent.detail("Elapsed", currentStats.elapsed)
|
2020-10-31 02:20:40 +08:00
|
|
|
.detail("MbpsSent", currentStats.machineMegabitsSent / currentStats.elapsed)
|
|
|
|
|
.detail("MbpsReceived", currentStats.machineMegabitsReceived / currentStats.elapsed)
|
|
|
|
|
.detail("OutSegs", currentStats.machineOutSegs)
|
|
|
|
|
.detail("RetransSegs", currentStats.machineRetransSegs)
|
|
|
|
|
.detail("CPUSeconds", currentStats.machineCPUSeconds)
|
|
|
|
|
.detail("TotalMemory", currentStats.machineTotalRAM)
|
|
|
|
|
.detail("CommittedMemory", currentStats.machineCommittedRAM)
|
|
|
|
|
.detail("AvailableMemory", currentStats.machineAvailableRAM)
|
|
|
|
|
.detail("DCID", machineState.dcId)
|
|
|
|
|
.detail("ZoneID", machineState.zoneId)
|
|
|
|
|
.detail("MachineID", machineState.machineId)
|
2023-01-11 08:37:54 +08:00
|
|
|
.detail("DatahallID", machineState.datahallId)
|
2020-10-31 02:20:40 +08:00
|
|
|
.trackLatest("MachineMetrics");
|
2022-09-13 02:12:26 +08:00
|
|
|
#ifdef __linux__
|
|
|
|
|
for (const auto& [k, v] : linux_os::reportCGroupCpuStat()) {
|
|
|
|
|
traceEvent.detail(capitalizeCgroupKey(k).c_str(), v);
|
|
|
|
|
}
|
|
|
|
|
#endif // __linux__
|
2017-05-26 04:48:44 +08:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#ifdef ALLOC_INSTRUMENTATION
|
|
|
|
|
{
|
|
|
|
|
static double firstTime = 0.0;
|
|
|
|
|
if (firstTime == 0.0)
|
|
|
|
|
firstTime = now();
|
|
|
|
|
if (now() - firstTime > 10 || g_network->isSimulated()) {
|
|
|
|
|
firstTime = now();
|
|
|
|
|
std::vector<std::pair<std::string, const char*>> typeNames;
|
|
|
|
|
for (auto i = allocInstr.begin(); i != allocInstr.end(); ++i) {
|
|
|
|
|
std::string s;
|
|
|
|
|
#ifdef __linux__
|
2020-08-28 06:31:24 +08:00
|
|
|
char* demangled = abi::__cxa_demangle(i->first, nullptr, nullptr, nullptr);
|
2017-05-26 04:48:44 +08:00
|
|
|
if (demangled) {
|
|
|
|
|
s = demangled;
|
2022-09-20 02:35:58 +08:00
|
|
|
if (StringRef(s).startsWith("(anonymous namespace)::"_sr))
|
|
|
|
|
s = s.substr("(anonymous namespace)::"_sr.size());
|
2017-05-26 04:48:44 +08:00
|
|
|
free(demangled);
|
|
|
|
|
} else
|
|
|
|
|
s = i->first;
|
|
|
|
|
#else
|
|
|
|
|
s = i->first;
|
2022-09-20 02:35:58 +08:00
|
|
|
if (StringRef(s).startsWith("class `anonymous namespace'::"_sr))
|
|
|
|
|
s = s.substr("class `anonymous namespace'::"_sr.size());
|
|
|
|
|
else if (StringRef(s).startsWith("class "_sr))
|
|
|
|
|
s = s.substr("class "_sr.size());
|
|
|
|
|
else if (StringRef(s).startsWith("struct "_sr))
|
|
|
|
|
s = s.substr("struct "_sr.size());
|
2017-05-26 04:48:44 +08:00
|
|
|
#endif
|
2021-05-11 07:32:02 +08:00
|
|
|
typeNames.emplace_back(s, i->first);
|
2017-05-26 04:48:44 +08:00
|
|
|
}
|
|
|
|
|
std::sort(typeNames.begin(), typeNames.end());
|
|
|
|
|
for (int i = 0; i < typeNames.size(); i++) {
|
|
|
|
|
const char* n = typeNames[i].second;
|
|
|
|
|
auto& f = allocInstr[n];
|
|
|
|
|
if (f.maxAllocated > 10000)
|
|
|
|
|
TraceEvent("AllocInstrument")
|
|
|
|
|
.detail("CurrentAlloc", f.allocCount - f.deallocCount)
|
|
|
|
|
.detail("Name", typeNames[i].first.c_str());
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
std::unordered_map<uint32_t, BackTraceAccount> traceCounts;
|
|
|
|
|
size_t memSampleSize;
|
|
|
|
|
memSample_entered = true;
|
|
|
|
|
{
|
|
|
|
|
ThreadSpinLockHolder holder(memLock);
|
|
|
|
|
traceCounts = backTraceLookup;
|
|
|
|
|
memSampleSize = memSample.size();
|
|
|
|
|
}
|
|
|
|
|
memSample_entered = false;
|
|
|
|
|
|
|
|
|
|
uint64_t totalSize = 0;
|
|
|
|
|
uint64_t totalCount = 0;
|
|
|
|
|
for (auto i = traceCounts.begin(); i != traceCounts.end(); ++i) {
|
|
|
|
|
std::vector<void*>* frames = i->second.backTrace;
|
2025-09-03 07:55:07 +08:00
|
|
|
std::string backTraceStr = platform::format_backtrace(&(*frames)[0], frames->size());
|
2017-05-26 04:48:44 +08:00
|
|
|
TraceEvent("MemSample")
|
|
|
|
|
.detail("Count", (int64_t)i->second.count)
|
|
|
|
|
.detail("TotalSize", i->second.totalSize)
|
|
|
|
|
.detail("SampleCount", i->second.sampleCount)
|
|
|
|
|
.detail("Hash", format("%lld", i->first))
|
|
|
|
|
.detail("Bt", backTraceStr);
|
|
|
|
|
|
|
|
|
|
totalSize += i->second.totalSize;
|
|
|
|
|
totalCount += i->second.count;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
TraceEvent("MemSampleSummary")
|
|
|
|
|
.detail("InverseByteSampleRatio", SAMPLE_BYTES)
|
|
|
|
|
.detail("MemorySamples", memSampleSize)
|
|
|
|
|
.detail("BackTraces", traceCounts.size())
|
|
|
|
|
.detail("TotalSize", totalSize)
|
|
|
|
|
.detail("TotalCount", totalCount);
|
|
|
|
|
|
|
|
|
|
TraceEvent("MemSample")
|
|
|
|
|
.detail("Count", traceCounts.size())
|
|
|
|
|
.detail("TotalSize", traceCounts.size() * ((int)(sizeof(uint32_t) + sizeof(size_t) + sizeof(size_t))))
|
|
|
|
|
.detail("SampleCount", traceCounts.size())
|
|
|
|
|
.detail("Hash", "backTraces")
|
|
|
|
|
.detail("Bt", "na");
|
|
|
|
|
|
|
|
|
|
TraceEvent("MemSample")
|
|
|
|
|
.detail("Count", memSampleSize)
|
|
|
|
|
.detail("TotalSize", memSampleSize * ((int)(sizeof(void*) + sizeof(uint32_t) + sizeof(size_t))))
|
|
|
|
|
.detail("SampleCount", memSampleSize)
|
|
|
|
|
.detail("Hash", "memSamples")
|
|
|
|
|
.detail("Bt", "na");
|
|
|
|
|
TRACEALLOCATOR(16);
|
|
|
|
|
TRACEALLOCATOR(32);
|
|
|
|
|
TRACEALLOCATOR(64);
|
2019-03-22 04:18:14 +08:00
|
|
|
TRACEALLOCATOR(96);
|
2017-05-26 04:48:44 +08:00
|
|
|
TRACEALLOCATOR(128);
|
|
|
|
|
TRACEALLOCATOR(256);
|
|
|
|
|
TRACEALLOCATOR(512);
|
|
|
|
|
TRACEALLOCATOR(1024);
|
|
|
|
|
TRACEALLOCATOR(2048);
|
|
|
|
|
TRACEALLOCATOR(4096);
|
2019-03-21 04:48:45 +08:00
|
|
|
TRACEALLOCATOR(8192);
|
2017-05-26 04:48:44 +08:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2020-01-13 08:44:30 +08:00
|
|
|
statState->networkMetricsState = g_network->networkInfo.metrics;
|
2017-05-26 04:48:44 +08:00
|
|
|
statState->networkState = netData;
|
call-site aware memory tracking (#13344)
* Initial memory tracking design doc draft, plus first round of review comments by me with // TODO annotations
* design/memory-tracker: address first-round review TODOs
Resolves all // TODO annotations from the initial draft:
- live-block table now optional via MEMORY_TRACKING_LIVE_TRACKING knob
- drop the mmap slab pool; std::malloc + in-tracker flag is sufficient
- add MEMORY_TRACKING_FORCE_SAMPLE_BYTES so large allocations are always
captured regardless of the count-rate sampler
- add live block / byte totals to MemoryTrackerSummary
- replace the manual coverage spot-check with a sentinel-function unit
test that introspects the aggregation table directly
- leave ALLOC_INSTRUMENTATION alone; new hooks sit next to (not
replacing) existing conditional ones
- drop SJLJ jargon, trim A4 alternative now that force-sample-large
collapses the byte-rate-vs-count-rate question
* flow: add sampled per-call-site memory tracker
Adds a sampled memory attribution layer (flow/MemoryTracker.{h,cpp})
hooked into the three primary allocation paths — global operator
new/delete, FastAllocator, and ArenaBlock::create — plus a periodic
TraceEvent dump driven from SystemMonitor. Knobs gate sample rate,
force-sample threshold, report cadence, top-N, and capture depth;
prod default is off. See design/memory-tracker.md.
Test fixes uncovered while bringing the unit tests up:
- memTrackerResetForTest now resets the per-thread sample counter and
force-sample threshold. Without this, a test that exercised the
off-switch path left gMemTrackerCounter at INT_MAX, which then
silently suppressed sampling for the remainder of the run.
- Slow-path reseed special-cases inverse==1 to keep the counter at 1.
The general formula 1 + r % (2*inverse) yields counter values 1 or
2 at inverse==1, sampling only ~67% of allocations rather than every
one, which broke a test that asserts exact alloc counts.
- Sentinel functions in MemoryTrackerTest.cpp now route the allocated
pointer through an asm-volatile escape() helper. Clang -O3 was
eliding the new/delete pair (P0593 heap fusion), so the test's
allocations never reached the operator-new override.
* design/memory-tracker: address second-round review
- threshold-based reporting (80 MB default, ~1% of 8 GB target RSS)
replaces fixed top-N
- prod report interval 60 s -> 10 min; sim stays 30 s
- single combined MemoryTrackerAddrCmd event with one addr2line
invocation per dump (positional mapping back to sites), keeping
frame 0 — old design's per-site format_backtrace dropped the
leaf alloc frame
- new R12 "Side-thread coverage" + "Side-thread safety" subsection
documenting the FP-elision crash mode found via joshua repros
(RandomUnitTests / IThreadPool seeds segfaulted in captureStackFP
when walking from FastAllocator<N>::~ThreadData into glibc's
FP-elided pthread shutdown machinery) and the stack-bounds
mitigation via pthread_getattr_np
- R7 wording: live bytes (not cumulative)
- CallSite struct in design overview aligned with implementation;
ForceSampledCount promoted from prose-only to struct + emitted
detail; exemplarFrames sized to MEMORY_TRACKER_MAX_FRAMES (=10)
matching the FRAMES knob's stated 1-10 range
- LIVE_TRACKING=false degraded-mode interaction documented
- stale "slab pool" refs removed (std::malloc was already in code)
and fdbserver.cpp self-contradiction resolved
- R3 / Rollout reconciled (table shows steady-state, step 1 lands
at 0)
- 4-6 frame count flagged as initial estimate, subject to refinement
- file:line citations stripped from path references (line numbers
drift; symbol names are stable)
* flow: threshold-based memory tracker reporting + side-thread safety
Implements the second-round design changes in flow/MemoryTracker.{cpp,h},
flow/Knobs.{cpp,h}, flow/SystemMonitor.cpp.
- MEMORY_TRACKING_TOP_N -> MEMORY_TRACKING_REPORT_BYTES_THRESHOLD
(int64_t, default 80,000,000). MEMORY_TRACKING_REPORT_INTERVAL prod
default 60.0 -> 600.0; sim still 30.0.
- memTrackerDump(int topN) -> memTrackerDump(int64_t bytesThreshold).
Filters by liveBytes (or cumulativeBytes when LIVE_TRACKING=false)
>= threshold; emits MemoryTrackerSite per qualifying site plus one
MemoryTrackerAddrCmd event with a single addr2line invocation
covering every qualifying site's frames in dump order. The Summary
event picks up SitesReported and ReportBytesThreshold details.
- AddrCmd is built directly here (not via platform::format_backtrace,
which deliberately drops index 0 for its single-site use case); the
leaf alloc frame is preserved.
- captureFramesFP gains a per-thread stack-bounds check via
pthread_getattr_np + pthread_attr_getstack, cached in TLS. Without
it, walking the FP chain from FastAllocator<N>::~ThreadData into
glibc's FP-elided pthread shutdown machinery follows an
uninitialized saved-FP slot and dereferences garbage. Fixes the
joshua-found segfaults on RandomUnitTests seeds 3288611985,
3731245491, and 2219741568 (all in the IThreadPool worker-exit
path). See design/memory-tracker.md "Side-thread safety".
* flow/MemoryTracker: stub the FP walker on non-Linux
pthread_getattr_np is glibc-specific and the macOS build broke on
it. Frame-pointer walking on macOS is also unreliable on its own
(system runtime has -fomit-frame-pointer in places we can't
control), so a "loose bounds" workaround would still risk crashes.
FDB is required to compile on macOS but is not run in production
there. Gate initStackBoundsForThread + the real captureFramesFP
on __linux__; provide a return-0 stub on non-Linux. The rest of
the tracker (sample counters, aggregation, dump) still compiles
and runs; per-call-site reports on macOS will just lack stack
attribution.
* flow: clang-format fixup for memory-tracker files
Whitespace-only. Catches up flow/Arena.cpp and flow/MemoryTrackerTest.cpp
with the project's clang-format style; the original implementation
commit (d587f82b) slipped these past the format pre-flight.
* edit for clarity, brevity, and uniform voice
* flow/Arena: fix double-tracking on the >256/huge ArenaBlock paths
ArenaBlock::create's >256 and huge paths go through
allocateAndMaybeKeepalive (`new uint8_t[]`), which fires the global
operator new[] hook in addition to the explicit memTrackerOnAlloc
that fires immediately after. Two sites tracked the same pointer;
on free only the explicit-Arena fingerprint was debited, so the
operator new[] fingerprint accumulated liveBytes monotonically and
LiveBytesTotal/LiveBlocksTotal skewed by +n/+1 per arena alloc/free
pair. Reported as B1 in the PR review.
Fix at the Arena layer (so non-arena allocateAndMaybeKeepalive
callers in serialize.h's PacketBuffer code remain attributed at the
operator-new layer): a MemTrackerSuppress RAII helper held across
the underlying new[]/delete[] in ArenaBlock::create and
ArenaBlock::destroyLeaf.
Adds accounting tests for FastAllocator<32>, Arena small, Arena
medium (the B1 path), and Arena huge. The load-bearing assertion is
"exactly one site has the sentinel's frames AND nonzero bytes" --
fails pre-fix for medium/huge with sites=2. Tests gate on __linux__
since captureFramesFP is a no-op on macOS.
* flow/memory-tracker: address PR review follow-ups
- Knobs.cpp: sim default for MEMORY_TRACKING_REPORT_BYTES_THRESHOLD
drops 80 MB -> 1 MB so sim dumps surface more sites for manual
sanity-checking. Prod unchanged.
- B2: memTrackerForEachSite holds MemTrackerSuppress across the
callback loop so callbacks that allocate (e.g. fprintf failure
dumps) don't re-enter tracking under SAMPLE_INVERSE=1.
- B8: drop the always-zero g_reentrantBailouts and its
SamplesDroppedReentry summary detail. Wiring it would require an
atomic in the inline hot path, which R1 forbids. Design doc
updated.
- B3/B6/B7/B9: explanatory comments only -- frame-strip-count
inlining assumption (B3), unstable sort acceptable under R6 (B6),
shared xorshift seed acceptable in practice (B7),
MEMORY_TRACKING_LIVE_TRACKING is startup-only (B9).
* flow/MemoryTracker: one addr2line per site, drop chunking
Move the addr2line command back onto MemoryTrackerSite as a per-site
AddrCmd detail and remove the MemoryTrackerAddrCmd event entirely.
Each AddrCmd carries exactly that site's stack -- short, well under
the trace-detail truncation cap, ready to paste. Replaces the
consolidated-then-chunked-into-byte-buckets approach, which split
stacks across chunk boundaries and made raw events hard to read.
* add standard Apache 2.0 license headers to new memory-tracker files
The three new files (flow/MemoryTracker.{cpp,h} and
flow/MemoryTrackerTest.cpp) shipped without the project-standard
copyright/license block. Adds the standard 19-line header to each;
file-purpose comments stay below it, switched to // line comments
to keep the license block visually distinct.
AGENTS.md gains a short "Source File Headers" section so the next
contributor doesn't repeat the omission.
* address review comments; reduce cost of enable check; maintain net estimates so users dont have to do it manually
* formatting
* unit test bug fix
* gglass review comments on memory-tracker.md design doc
* design doc updates, and fill in a plan for remaining tests/benchmarks
* delete useless simulation section
* flow/bench: add memory-tracker microbenchmarks; record measured overhead
Add flow/bench/BenchMemoryTracker.cpp (Google Benchmark) measuring the tracker's
per-op cost via three benchmarks -- raw malloc/free baseline, end-to-end
operator new/delete, and isolated memTrackerOnAlloc/OnFree -- each at sample
inverse 0 / 100 / 1. Auto-picked up by the flow_bench CONFIGURE_DEPENDS glob;
run with:
bin/flow_bench --benchmark_filter=memtracker
Replace the placeholder "< 5% delta" targets in the design doc's Microbenchmarks
section with the measured numbers and a projected per-second overhead at an
assumed 100K alloc/sec. Headline: ~1.9 ns/pair disabled, ~5.6 ns/pair at the
production 1% rate (~0.056% of a core at 100K/s, ~1/17th of R0's 1% ceiling);
the every-allocation rows are labeled a buggified worst case, not a default.
Testing: built flow_bench on the dev pod (clean, -Werror) and ran the memtracker
filter six times; results stable to within a few percent across runs.
* design/memory-tracker: describe the coverage test as implemented, not proposed
The "Coverage spot-check via sentinel functions" section described the
already-implemented `coverage` test in proposal tense ("Add an introspection
API:", "The unit test:"), which read as future work. Reword to present tense
referencing the actual `coverage`/`*Accounting` tests and the existing
`memTrackerForEachSite` API. No remaining references to unwritten test cases.
* remove extraneous detail from requirements section
* substantially revise microbenchmark results based on seeing 2M allocations/frees on a CPU-maxed storage server
* add script to drive A/B experiment for sampled memory allocation tracking
* flow/MemoryTracker: move global operator new/delete into a server-only TU
The global operator new/delete replacements that route allocations through
the memory tracker lived in flow/MemoryTracker.cpp, i.e. in the flow static
library. flow is linked into libfdb_c and every client binding, so the
interposition shipped into client artifacts and could interpose the whole
host process's allocator even with sampling off.
Move them into a new fdbserver/GlobalNewDelete.cpp, compiled directly into
the fdbserver executable (which clients never link), mirroring where the
legacy ALLOC_INSTRUMENTATION overrides already lived. This also makes the
replaceable-symbol interposition reliable (guaranteed in the final link)
rather than dependent on static-archive pull-in. The legacy
ALLOC_INSTRUMENTATION overrides move into the same file, selected by
#if defined(ALLOC_INSTRUMENTATION) / #else, so exactly one set of global
operators is ever defined. No CMake change is needed (fdbserver/*.cpp is
globbed).
Also reconcile the design doc with the code: override placement, the Files
section (drop the unnecessary flow/CMakeLists.txt edit), the sim knob-table
values (REPORT_BYTES_THRESHOLD, SAMPLE_INVERSE prod default), the
FORCE_SAMPLE_BYTES "-1 disables" sentinel wording, and drop the stale
"buggify inverse to 1" note -- the every-allocation and sampled/weighted
paths are already pinned deterministically by MemoryTrackerTest.cpp.
Testing: build green (run-ccmk5); fdbserver -r unittests -f /flow/MemoryTracker/
runs all 10 memory-tracker unit tests, 0 failed.
* contrib/mako_ab_memtracker: disable RocksDB direct I/O for tmpfs runs
RocksDB opens its DB with O_DIRECT by default; /mnt/ram (tmpfs), where the
harness puts its data dir, does not support direct I/O, so the storage
engine fails to Open and the cluster never configures (fdbcli "configure
new" hangs, mako never starts). Pass the existing
ROCKSDB_USE_DIRECT_READS / ROCKSDB_USE_DIRECT_IO_FLUSH_COMPACTION knobs (=0)
for the rocksdb arm only -- no new knobs added. redwood is unaffected.
* fdbserver/bench: move memory-tracker microbench to fdbserver_bench
The global operator new/delete override lives in fdbserver/GlobalNewDelete.cpp
(server-only, for client isolation), so flow_bench -- which links only flow --
could not exercise it: bench_memtracker_operator_new hit libc++'s operator new
and its Arg(100)/Arg(1) rows were identical to Arg(0).
Move BenchMemoryTracker.cpp to a new fdbserver/bench/ whose CMake compiles
GlobalNewDelete.cpp into the fdbserver_bench executable (via ADDL_SRCS), so the
real override is a strong definition in the bench link and the operator-new
benchmark measures the actual hooked path. It links only flow, not the fdbserver
dependency graph. Adds a BenchMain.cpp (BENCHMARK_MAIN equivalent) and wires the
subdirectory into fdbserver/CMakeLists.txt.
Testing: fdbserver_bench builds; bench_memtracker_operator_new now shows distinct
off / 1% / every-alloc costs (11.4 / 14.4 / 80.9 ns), confirming the override
fires.
* design/memory-tracker: reconcile with code and slim down
Reconcile the doc with the implementation and trim implementation detail that
duplicated the code and had begun to drift:
- Fix drift found in review: degraded-mode live/peak fields stay 0 (not
"tracking the cumulatives"); the dump thresholds on the estimated fields (fix
the memTrackerDump header comment too); the live-block table is allocated but
empty when live-tracking is off (not "never allocated"); list the test file
and its forceLinkMemoryTrackerTests() wiring.
- Slim ~220 lines: replace the CallSite struct, the full captureStackFP source,
both TraceEvent .detail() schemas, and the file-by-file inventory with prose
that defers exact structs/keys/constants to the code; soften the knob table
to intent (authoritative defaults live in Knobs.cpp).
- Refresh the microbench numbers from fdbserver_bench and annotate the host
(AMD EPYC 9R14, clang -O3); replace the A/B placeholder with the measured
-16.5% (redwood) / -9.9% (rocksdb); note it is a point-in-time snapshot.
- Frame R0 as the target the v1 single-lock design does not yet meet (ships off
pending lock sharding); add a one-line note on table teardown/fork behavior.
* fix clang-tidy error
* mako wrapper scripts: take care to clobber the ramdisk before runs to avoid low-space throttling
* design/memory-tracker: note the two mako A/B harnesses and the off-state result
Point at contrib/mako_ab_memtracker.py (sampling off vs on) and
contrib/mako_ab_binaries.py (vanilla main vs this PR built with tracking off),
and record the latter's measured off-state overhead of -0.53% (redwood) /
-0.10% (rocksdb) on a shared base commit — well within R0's 1% ceiling.
* Add the default code review guidance to AGENTS.md
* address big brother review comments; add some braces and trim some generated comments (hard to believe, but true)
* clang-tidy again
* Final read-through of this PR.
-- Add or enhance a few comments on important items (performance & reliability)
-- Delete some misc agent-written comments, typically exhibiting recency bias e.g. naming bugs identified in code review passes or describing mundane earlier bugs
-- Add braces (InsertBraces style)
* memory-tracker: address round-4 review
Correctness/robustness:
- operator new now runs the installed std::new_handler retry loop, so an
allocation failure (including the tracker's own map growth) reaches FDB's
platform::outOfMemory / FDB_EXIT_NO_MEM path instead of throwing past it.
- Reentrancy guard restored via MemTrackerSuppress RAII on every path (hot path,
dump, reset) so an exception can't permanently disable tracking on a thread.
- Sampling reseed draws from [1, 2N-1] (mean exactly N); the old [1, 2N] biased
the Est* estimate low by ~0.5/N.
Design:
- Drop runtime enable/disable (now an explicit Non-requirement): the sample knob
is read at startup only; park the counter when off. Removes the DISABLED_RESEED
re-park and the on/off reconciliation logic.
- Simulation samples 1-in-10 (was 1-in-2).
Portability (Windows is not build-tested here; lean on existing abstractions):
- Aligned operator new uses platform::aligned_alloc/aligned_free (overflow-guarded)
instead of posix_memalign; guard <pthread.h> under __linux__; add a
force_noinline macro (GNU-only, empty elsewhere); portable volatile-sink
escape() in the test.
Tooling/tests/docs:
- mako A/B scripts: validate the scratch mount (realpath + tmpfs + denylist) before
rm -rf, locate mako_storage_bench.sh via __file__, and black-format.
- Add operatorNewHonorsNewHandler and samplingRate tests; drop enableAfterOff.
- Reconcile the design doc with the code; prune low-value comments.
Testing:
- /flow/MemoryTracker/* unit tests: 11 pass, 0 fail (bin/fdbserver -r unittests).
- Joshua 100k (correctness-8.0.0): 99,995 pass / 1 fail. The single failure is
NativeCdcAssignmentPublication -- a NativeCdc test from main, not this PR
(CommitProxy failed_to_progress -> QuietDatabase DataDistributionActive ->
NativeCdcEndToEnd workload start timed_out). It reproduces bit-identically
(seed 2939264355, unseed 8776) with the tracker built off (sim
sample_inverse=0, confirmed via the MemoryTrackerSummary SampleInverse trace
field), so it is a pre-existing rare CDC/QuietDatabase flake unrelated to this
change.
* memory-tracker: fix GCC IPA-clone breaking frame-attribution tests
Under GCC -O3, IPA constant-propagation cloning (-fipa-cp-clone) specializes the
MemoryTrackerTest sentinels (each called with a constant N) into `.constprop`
clones emitted at a different address than the function symbol. The executed
code -- and thus the captured return addresses -- live in the clone, so the
tests' frameInside(frame, &sentinel) window missed them: fastAlloc32Accounting
aborted (sitesWithSentinelFrames == 0) in GCC CI while clang passed.
Add `noclone` to force_noinline on GCC so each sentinel stays a single body at
the address &fn yields. clang doesn't support noclone (and doesn't clone this
way), so it keeps noinline only; other compilers stay empty.
Testing: /flow/MemoryTracker/* passes 11/11 under both a gcc-toolset-13 build
and a clang build.
* memory-tracker: cheaper disabled alloc hot path (per-thread off flag)
When sampling is off, memTrackerOnAlloc previously still did three TLS accesses
every allocation -- a gInMemTracker load, a gMemTrackerCounter load-decrement-
STORE, and a gForceSampleBytes load. Add a per-thread gMemTrackerOff flag,
checked first, that a thread sets once its slow path observes sampling is off;
the disabled alloc path then short-circuits on a single TLS load + branch (no
counter store, no gForceSampleBytes load).
The flag is per-thread, not global, on purpose: the counter still bootstraps
sampling per thread (first alloc reaches the slow path and reads the knob), so a
single global gate set by an early main-thread allocation before FLOW_KNOBS is
ready would wrongly disable worker threads that bootstrap later. Free keeps the
global g_memTrackerEnabled gate (a free-only thread must see global state to
debit). memTrackerResetForTest clears the flag. Removes the now-dead INT_MAX
counter parking.
The saving is real but small (~2 loads + 1 store per alloc, well under the
mako A/B's run-to-run noise), so the redwood off-state A/B shows no resolvable
change; the win is on principle / at high allocation rates.
Testing: /flow/MemoryTracker/* passes 11/11 (bin/fdbserver -r unittests).
* memory-tracker: add FDB_MEMORY_TRACKER compile-time gate + per-path microbench
Add a compile-time switch, FDB_MEMORY_TRACKER (CMake option, default ON), that
removes the feature entirely when set to 0: the header hooks become no-op inlines,
flow/MemoryTracker.cpp and the MemoryTrackerTest TEST_CASEs are #if'd out (the
forceLink stub stays), and fdbserver/GlobalNewDelete.cpp defines no global
operator new/delete override (libc++'s allocator is used, which still honors the
installed new_handler). This gives operators an escape hatch to zero
always-compiled footprint, and lets the microbench measure present-but-disabled
vs absent cost per allocation path.
Extend fdbserver/bench/BenchMemoryTracker.cpp with plain per-path alloc/free loops
(operator new[] at several sizes, FastAllocator<64/96/256>, Arena medium/huge) so
the same binary built =1 (tracker present, sampling off by default) vs =0 (absent)
isolates each path's unweighted off-state overhead.
Measured (EPYC 9R14 @ 3.7 GHz, ns/op, =1 minus =0):
operator new[] small ~+1.1 ns (~11%); huge ~+2.5 ns
FastAllocator<N> ~0 (within the ~0.6 ns cross-build noise floor)
Arena block ~+3.3 ns (medium) / +4.8 ns (huge)
The per-op-costly paths (operator new, Arena) are the low-volume ones; the
high-volume path (FastAllocator, ~82% of redwood allocs) is ~free, so the weighted
off-state overhead is ~0.1-0.2% of a core -- under the R0 target, though R0 is now
framed as an unproven target with this gate as the escape hatch.
Testing: /flow/MemoryTracker/* passes 11/11 (default build); FDB_MEMORY_TRACKER=OFF
builds clean under -Werror.
* clang-tidy fix
* comment about why fdbserver/bench exists
* address Codex round 5 review comments (MSVC build, startup sequencing, memory allocation failures on tracker-internal bookkeeping, cross-compile, contrib script cleanup)
* add another disclaimer about MSVC/Windows being best effort
* memory-tracker: fail-open ordering + deflake huge-arena unit test
memTrackerSampleAlloc now performs both table insertions (aggregation-map and
live-map nodes) before mutating any per-site or global counter, so if the
tracker's own map growth throws std::bad_alloc the exception unwinds with all
totals in lockstep and the fail-open catch simply drops the sample. This
addresses an adversarial-review finding. Note such a failure does not occur in
practice: FDB "OOM" is an RSS threshold enforced by fdbmonitor (typically
~12-16 GB against an ~8 GB target), not a malloc/operator-new failure. This
feature is for finding leaks and untuned allocations that drive RSS growth,
well short of any allocation failure -- documented in flow/MemoryTracker.cpp and
design/memory-tracker.md so reviewers don't over-index on the bad_alloc path.
arenaHugeAccounting now identifies the huge blocks by their ~100 KB size
signature instead of the frame-pointer sentinel. The huge-Arena path is the
deepest tracked call chain; under a real (non-simulation) network the
best-effort frame-pointer walker is per-allocation nondeterministic there and
cannot reliably attribute the blocks to the test's frame, so the sentinel-frame
assertion flaked (~1-4% of runs). The recorded block size is correct regardless
of which frames were captured, no incidental or foreign-thread allocation comes
near 100 KB, and double-tracking still shows up as 2N. The shallower
fastAlloc32/arenaSmall/arenaMedium/operatorNew accounting tests keep exact
sentinel-frame attribution (reliable on their shorter paths) as the regression
guard.
Testing:
- /flow/MemoryTracker/* via `fdbserver -r unittests`: 500 runs across two seed
spaces (including the sequence that previously flaked), 14/14 test cases pass
every run, 0 failures.
- Builds clean with FDB_MEMORY_TRACKER on and off under -Werror; clang-format
and clang-tidy clean on the changed files.
- Joshua 100k on the parent commit: ensemble
20260728-160205-gglass-0b30fa74f2605807, ended=100000 pass=100000 fail=0.
These changes are a pure counter-update reorder (no simulation-reachable
behavior change -- bad_alloc is not injected in sim), a unit-test-only change,
and documentation, so that result remains representative.
* memory-tracker: note why the fdbserver-local bench binary exists
Add a short comment to BenchMain.cpp explaining that most microbenchmarks
belong in flow/bench and this fdbserver-local benchmark binary exists only for
benchmarks that must link fdbserver-only code, pointing at BenchMemoryTracker.cpp
for the detailed rationale rather than duplicating it.
2026-08-01 01:12:50 +08:00
|
|
|
|
|
|
|
|
// Periodic dump of the per-call-site memory tracker; cadence from the
|
|
|
|
|
// MEMORY_TRACKING_REPORT_INTERVAL knob (<=0 disables). In simulation the
|
|
|
|
|
// tracker's tables and this static are shared across all simulated
|
|
|
|
|
// processes, so one dump fires per interval cluster-wide and its site
|
|
|
|
|
// numbers blend every process — correct only in a real single-process server.
|
|
|
|
|
if (FLOW_KNOBS && FLOW_KNOBS->MEMORY_TRACKING_REPORT_INTERVAL > 0) {
|
|
|
|
|
static double lastMemTrackerDump = 0;
|
|
|
|
|
if (now() - lastMemTrackerDump >= FLOW_KNOBS->MEMORY_TRACKING_REPORT_INTERVAL) {
|
|
|
|
|
memTrackerDump(FLOW_KNOBS->MEMORY_TRACKING_REPORT_BYTES_THRESHOLD);
|
|
|
|
|
lastMemTrackerDump = now();
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2017-05-26 04:48:44 +08:00
|
|
|
return currentStats;
|
|
|
|
|
}
|
2022-04-07 11:06:24 +08:00
|
|
|
|
|
|
|
|
Future<Void> startMemoryUsageMonitor(uint64_t memLimit) {
|
|
|
|
|
if (memLimit == 0) {
|
|
|
|
|
return Void();
|
|
|
|
|
}
|
|
|
|
|
auto checkMemoryUsage = [=]() {
|
|
|
|
|
if (getResidentMemoryUsage() > memLimit) {
|
2023-01-25 05:04:47 +08:00
|
|
|
#if defined(ADDRESS_SANITIZER) && defined(__linux__)
|
|
|
|
|
__sanitizer_print_memory_profile(/*top percent*/ 100, /*max contexts*/ 10);
|
|
|
|
|
#endif
|
2022-04-07 11:06:24 +08:00
|
|
|
platform::outOfMemory();
|
|
|
|
|
}
|
|
|
|
|
};
|
|
|
|
|
return recurring(checkMemoryUsage, FLOW_KNOBS->MEMORY_USAGE_CHECK_INTERVAL);
|
2023-01-11 08:37:54 +08:00
|
|
|
}
|