* Initial memory tracking design doc draft, plus first round of review comments by me with // TODO annotations
* design/memory-tracker: address first-round review TODOs
Resolves all // TODO annotations from the initial draft:
- live-block table now optional via MEMORY_TRACKING_LIVE_TRACKING knob
- drop the mmap slab pool; std::malloc + in-tracker flag is sufficient
- add MEMORY_TRACKING_FORCE_SAMPLE_BYTES so large allocations are always
captured regardless of the count-rate sampler
- add live block / byte totals to MemoryTrackerSummary
- replace the manual coverage spot-check with a sentinel-function unit
test that introspects the aggregation table directly
- leave ALLOC_INSTRUMENTATION alone; new hooks sit next to (not
replacing) existing conditional ones
- drop SJLJ jargon, trim A4 alternative now that force-sample-large
collapses the byte-rate-vs-count-rate question
* flow: add sampled per-call-site memory tracker
Adds a sampled memory attribution layer (flow/MemoryTracker.{h,cpp})
hooked into the three primary allocation paths — global operator
new/delete, FastAllocator, and ArenaBlock::create — plus a periodic
TraceEvent dump driven from SystemMonitor. Knobs gate sample rate,
force-sample threshold, report cadence, top-N, and capture depth;
prod default is off. See design/memory-tracker.md.
Test fixes uncovered while bringing the unit tests up:
- memTrackerResetForTest now resets the per-thread sample counter and
force-sample threshold. Without this, a test that exercised the
off-switch path left gMemTrackerCounter at INT_MAX, which then
silently suppressed sampling for the remainder of the run.
- Slow-path reseed special-cases inverse==1 to keep the counter at 1.
The general formula 1 + r % (2*inverse) yields counter values 1 or
2 at inverse==1, sampling only ~67% of allocations rather than every
one, which broke a test that asserts exact alloc counts.
- Sentinel functions in MemoryTrackerTest.cpp now route the allocated
pointer through an asm-volatile escape() helper. Clang -O3 was
eliding the new/delete pair (P0593 heap fusion), so the test's
allocations never reached the operator-new override.
* design/memory-tracker: address second-round review
- threshold-based reporting (80 MB default, ~1% of 8 GB target RSS)
replaces fixed top-N
- prod report interval 60 s -> 10 min; sim stays 30 s
- single combined MemoryTrackerAddrCmd event with one addr2line
invocation per dump (positional mapping back to sites), keeping
frame 0 — old design's per-site format_backtrace dropped the
leaf alloc frame
- new R12 "Side-thread coverage" + "Side-thread safety" subsection
documenting the FP-elision crash mode found via joshua repros
(RandomUnitTests / IThreadPool seeds segfaulted in captureStackFP
when walking from FastAllocator<N>::~ThreadData into glibc's
FP-elided pthread shutdown machinery) and the stack-bounds
mitigation via pthread_getattr_np
- R7 wording: live bytes (not cumulative)
- CallSite struct in design overview aligned with implementation;
ForceSampledCount promoted from prose-only to struct + emitted
detail; exemplarFrames sized to MEMORY_TRACKER_MAX_FRAMES (=10)
matching the FRAMES knob's stated 1-10 range
- LIVE_TRACKING=false degraded-mode interaction documented
- stale "slab pool" refs removed (std::malloc was already in code)
and fdbserver.cpp self-contradiction resolved
- R3 / Rollout reconciled (table shows steady-state, step 1 lands
at 0)
- 4-6 frame count flagged as initial estimate, subject to refinement
- file:line citations stripped from path references (line numbers
drift; symbol names are stable)
* flow: threshold-based memory tracker reporting + side-thread safety
Implements the second-round design changes in flow/MemoryTracker.{cpp,h},
flow/Knobs.{cpp,h}, flow/SystemMonitor.cpp.
- MEMORY_TRACKING_TOP_N -> MEMORY_TRACKING_REPORT_BYTES_THRESHOLD
(int64_t, default 80,000,000). MEMORY_TRACKING_REPORT_INTERVAL prod
default 60.0 -> 600.0; sim still 30.0.
- memTrackerDump(int topN) -> memTrackerDump(int64_t bytesThreshold).
Filters by liveBytes (or cumulativeBytes when LIVE_TRACKING=false)
>= threshold; emits MemoryTrackerSite per qualifying site plus one
MemoryTrackerAddrCmd event with a single addr2line invocation
covering every qualifying site's frames in dump order. The Summary
event picks up SitesReported and ReportBytesThreshold details.
- AddrCmd is built directly here (not via platform::format_backtrace,
which deliberately drops index 0 for its single-site use case); the
leaf alloc frame is preserved.
- captureFramesFP gains a per-thread stack-bounds check via
pthread_getattr_np + pthread_attr_getstack, cached in TLS. Without
it, walking the FP chain from FastAllocator<N>::~ThreadData into
glibc's FP-elided pthread shutdown machinery follows an
uninitialized saved-FP slot and dereferences garbage. Fixes the
joshua-found segfaults on RandomUnitTests seeds 3288611985,
3731245491, and 2219741568 (all in the IThreadPool worker-exit
path). See design/memory-tracker.md "Side-thread safety".
* flow/MemoryTracker: stub the FP walker on non-Linux
pthread_getattr_np is glibc-specific and the macOS build broke on
it. Frame-pointer walking on macOS is also unreliable on its own
(system runtime has -fomit-frame-pointer in places we can't
control), so a "loose bounds" workaround would still risk crashes.
FDB is required to compile on macOS but is not run in production
there. Gate initStackBoundsForThread + the real captureFramesFP
on __linux__; provide a return-0 stub on non-Linux. The rest of
the tracker (sample counters, aggregation, dump) still compiles
and runs; per-call-site reports on macOS will just lack stack
attribution.
* flow: clang-format fixup for memory-tracker files
Whitespace-only. Catches up flow/Arena.cpp and flow/MemoryTrackerTest.cpp
with the project's clang-format style; the original implementation
commit (d587f82b) slipped these past the format pre-flight.
* edit for clarity, brevity, and uniform voice
* flow/Arena: fix double-tracking on the >256/huge ArenaBlock paths
ArenaBlock::create's >256 and huge paths go through
allocateAndMaybeKeepalive (`new uint8_t[]`), which fires the global
operator new[] hook in addition to the explicit memTrackerOnAlloc
that fires immediately after. Two sites tracked the same pointer;
on free only the explicit-Arena fingerprint was debited, so the
operator new[] fingerprint accumulated liveBytes monotonically and
LiveBytesTotal/LiveBlocksTotal skewed by +n/+1 per arena alloc/free
pair. Reported as B1 in the PR review.
Fix at the Arena layer (so non-arena allocateAndMaybeKeepalive
callers in serialize.h's PacketBuffer code remain attributed at the
operator-new layer): a MemTrackerSuppress RAII helper held across
the underlying new[]/delete[] in ArenaBlock::create and
ArenaBlock::destroyLeaf.
Adds accounting tests for FastAllocator<32>, Arena small, Arena
medium (the B1 path), and Arena huge. The load-bearing assertion is
"exactly one site has the sentinel's frames AND nonzero bytes" --
fails pre-fix for medium/huge with sites=2. Tests gate on __linux__
since captureFramesFP is a no-op on macOS.
* flow/memory-tracker: address PR review follow-ups
- Knobs.cpp: sim default for MEMORY_TRACKING_REPORT_BYTES_THRESHOLD
drops 80 MB -> 1 MB so sim dumps surface more sites for manual
sanity-checking. Prod unchanged.
- B2: memTrackerForEachSite holds MemTrackerSuppress across the
callback loop so callbacks that allocate (e.g. fprintf failure
dumps) don't re-enter tracking under SAMPLE_INVERSE=1.
- B8: drop the always-zero g_reentrantBailouts and its
SamplesDroppedReentry summary detail. Wiring it would require an
atomic in the inline hot path, which R1 forbids. Design doc
updated.
- B3/B6/B7/B9: explanatory comments only -- frame-strip-count
inlining assumption (B3), unstable sort acceptable under R6 (B6),
shared xorshift seed acceptable in practice (B7),
MEMORY_TRACKING_LIVE_TRACKING is startup-only (B9).
* flow/MemoryTracker: one addr2line per site, drop chunking
Move the addr2line command back onto MemoryTrackerSite as a per-site
AddrCmd detail and remove the MemoryTrackerAddrCmd event entirely.
Each AddrCmd carries exactly that site's stack -- short, well under
the trace-detail truncation cap, ready to paste. Replaces the
consolidated-then-chunked-into-byte-buckets approach, which split
stacks across chunk boundaries and made raw events hard to read.
* add standard Apache 2.0 license headers to new memory-tracker files
The three new files (flow/MemoryTracker.{cpp,h} and
flow/MemoryTrackerTest.cpp) shipped without the project-standard
copyright/license block. Adds the standard 19-line header to each;
file-purpose comments stay below it, switched to // line comments
to keep the license block visually distinct.
AGENTS.md gains a short "Source File Headers" section so the next
contributor doesn't repeat the omission.
* address review comments; reduce cost of enable check; maintain net estimates so users dont have to do it manually
* formatting
* unit test bug fix
* gglass review comments on memory-tracker.md design doc
* design doc updates, and fill in a plan for remaining tests/benchmarks
* delete useless simulation section
* flow/bench: add memory-tracker microbenchmarks; record measured overhead
Add flow/bench/BenchMemoryTracker.cpp (Google Benchmark) measuring the tracker's
per-op cost via three benchmarks -- raw malloc/free baseline, end-to-end
operator new/delete, and isolated memTrackerOnAlloc/OnFree -- each at sample
inverse 0 / 100 / 1. Auto-picked up by the flow_bench CONFIGURE_DEPENDS glob;
run with:
bin/flow_bench --benchmark_filter=memtracker
Replace the placeholder "< 5% delta" targets in the design doc's Microbenchmarks
section with the measured numbers and a projected per-second overhead at an
assumed 100K alloc/sec. Headline: ~1.9 ns/pair disabled, ~5.6 ns/pair at the
production 1% rate (~0.056% of a core at 100K/s, ~1/17th of R0's 1% ceiling);
the every-allocation rows are labeled a buggified worst case, not a default.
Testing: built flow_bench on the dev pod (clean, -Werror) and ran the memtracker
filter six times; results stable to within a few percent across runs.
* design/memory-tracker: describe the coverage test as implemented, not proposed
The "Coverage spot-check via sentinel functions" section described the
already-implemented `coverage` test in proposal tense ("Add an introspection
API:", "The unit test:"), which read as future work. Reword to present tense
referencing the actual `coverage`/`*Accounting` tests and the existing
`memTrackerForEachSite` API. No remaining references to unwritten test cases.
* remove extraneous detail from requirements section
* substantially revise microbenchmark results based on seeing 2M allocations/frees on a CPU-maxed storage server
* add script to drive A/B experiment for sampled memory allocation tracking
* flow/MemoryTracker: move global operator new/delete into a server-only TU
The global operator new/delete replacements that route allocations through
the memory tracker lived in flow/MemoryTracker.cpp, i.e. in the flow static
library. flow is linked into libfdb_c and every client binding, so the
interposition shipped into client artifacts and could interpose the whole
host process's allocator even with sampling off.
Move them into a new fdbserver/GlobalNewDelete.cpp, compiled directly into
the fdbserver executable (which clients never link), mirroring where the
legacy ALLOC_INSTRUMENTATION overrides already lived. This also makes the
replaceable-symbol interposition reliable (guaranteed in the final link)
rather than dependent on static-archive pull-in. The legacy
ALLOC_INSTRUMENTATION overrides move into the same file, selected by
#if defined(ALLOC_INSTRUMENTATION) / #else, so exactly one set of global
operators is ever defined. No CMake change is needed (fdbserver/*.cpp is
globbed).
Also reconcile the design doc with the code: override placement, the Files
section (drop the unnecessary flow/CMakeLists.txt edit), the sim knob-table
values (REPORT_BYTES_THRESHOLD, SAMPLE_INVERSE prod default), the
FORCE_SAMPLE_BYTES "-1 disables" sentinel wording, and drop the stale
"buggify inverse to 1" note -- the every-allocation and sampled/weighted
paths are already pinned deterministically by MemoryTrackerTest.cpp.
Testing: build green (run-ccmk5); fdbserver -r unittests -f /flow/MemoryTracker/
runs all 10 memory-tracker unit tests, 0 failed.
* contrib/mako_ab_memtracker: disable RocksDB direct I/O for tmpfs runs
RocksDB opens its DB with O_DIRECT by default; /mnt/ram (tmpfs), where the
harness puts its data dir, does not support direct I/O, so the storage
engine fails to Open and the cluster never configures (fdbcli "configure
new" hangs, mako never starts). Pass the existing
ROCKSDB_USE_DIRECT_READS / ROCKSDB_USE_DIRECT_IO_FLUSH_COMPACTION knobs (=0)
for the rocksdb arm only -- no new knobs added. redwood is unaffected.
* fdbserver/bench: move memory-tracker microbench to fdbserver_bench
The global operator new/delete override lives in fdbserver/GlobalNewDelete.cpp
(server-only, for client isolation), so flow_bench -- which links only flow --
could not exercise it: bench_memtracker_operator_new hit libc++'s operator new
and its Arg(100)/Arg(1) rows were identical to Arg(0).
Move BenchMemoryTracker.cpp to a new fdbserver/bench/ whose CMake compiles
GlobalNewDelete.cpp into the fdbserver_bench executable (via ADDL_SRCS), so the
real override is a strong definition in the bench link and the operator-new
benchmark measures the actual hooked path. It links only flow, not the fdbserver
dependency graph. Adds a BenchMain.cpp (BENCHMARK_MAIN equivalent) and wires the
subdirectory into fdbserver/CMakeLists.txt.
Testing: fdbserver_bench builds; bench_memtracker_operator_new now shows distinct
off / 1% / every-alloc costs (11.4 / 14.4 / 80.9 ns), confirming the override
fires.
* design/memory-tracker: reconcile with code and slim down
Reconcile the doc with the implementation and trim implementation detail that
duplicated the code and had begun to drift:
- Fix drift found in review: degraded-mode live/peak fields stay 0 (not
"tracking the cumulatives"); the dump thresholds on the estimated fields (fix
the memTrackerDump header comment too); the live-block table is allocated but
empty when live-tracking is off (not "never allocated"); list the test file
and its forceLinkMemoryTrackerTests() wiring.
- Slim ~220 lines: replace the CallSite struct, the full captureStackFP source,
both TraceEvent .detail() schemas, and the file-by-file inventory with prose
that defers exact structs/keys/constants to the code; soften the knob table
to intent (authoritative defaults live in Knobs.cpp).
- Refresh the microbench numbers from fdbserver_bench and annotate the host
(AMD EPYC 9R14, clang -O3); replace the A/B placeholder with the measured
-16.5% (redwood) / -9.9% (rocksdb); note it is a point-in-time snapshot.
- Frame R0 as the target the v1 single-lock design does not yet meet (ships off
pending lock sharding); add a one-line note on table teardown/fork behavior.
* fix clang-tidy error
* mako wrapper scripts: take care to clobber the ramdisk before runs to avoid low-space throttling
* design/memory-tracker: note the two mako A/B harnesses and the off-state result
Point at contrib/mako_ab_memtracker.py (sampling off vs on) and
contrib/mako_ab_binaries.py (vanilla main vs this PR built with tracking off),
and record the latter's measured off-state overhead of -0.53% (redwood) /
-0.10% (rocksdb) on a shared base commit — well within R0's 1% ceiling.
* Add the default code review guidance to AGENTS.md
* address big brother review comments; add some braces and trim some generated comments (hard to believe, but true)
* clang-tidy again
* Final read-through of this PR.
-- Add or enhance a few comments on important items (performance & reliability)
-- Delete some misc agent-written comments, typically exhibiting recency bias e.g. naming bugs identified in code review passes or describing mundane earlier bugs
-- Add braces (InsertBraces style)
* memory-tracker: address round-4 review
Correctness/robustness:
- operator new now runs the installed std::new_handler retry loop, so an
allocation failure (including the tracker's own map growth) reaches FDB's
platform::outOfMemory / FDB_EXIT_NO_MEM path instead of throwing past it.
- Reentrancy guard restored via MemTrackerSuppress RAII on every path (hot path,
dump, reset) so an exception can't permanently disable tracking on a thread.
- Sampling reseed draws from [1, 2N-1] (mean exactly N); the old [1, 2N] biased
the Est* estimate low by ~0.5/N.
Design:
- Drop runtime enable/disable (now an explicit Non-requirement): the sample knob
is read at startup only; park the counter when off. Removes the DISABLED_RESEED
re-park and the on/off reconciliation logic.
- Simulation samples 1-in-10 (was 1-in-2).
Portability (Windows is not build-tested here; lean on existing abstractions):
- Aligned operator new uses platform::aligned_alloc/aligned_free (overflow-guarded)
instead of posix_memalign; guard <pthread.h> under __linux__; add a
force_noinline macro (GNU-only, empty elsewhere); portable volatile-sink
escape() in the test.
Tooling/tests/docs:
- mako A/B scripts: validate the scratch mount (realpath + tmpfs + denylist) before
rm -rf, locate mako_storage_bench.sh via __file__, and black-format.
- Add operatorNewHonorsNewHandler and samplingRate tests; drop enableAfterOff.
- Reconcile the design doc with the code; prune low-value comments.
Testing:
- /flow/MemoryTracker/* unit tests: 11 pass, 0 fail (bin/fdbserver -r unittests).
- Joshua 100k (correctness-8.0.0): 99,995 pass / 1 fail. The single failure is
NativeCdcAssignmentPublication -- a NativeCdc test from main, not this PR
(CommitProxy failed_to_progress -> QuietDatabase DataDistributionActive ->
NativeCdcEndToEnd workload start timed_out). It reproduces bit-identically
(seed 2939264355, unseed 8776) with the tracker built off (sim
sample_inverse=0, confirmed via the MemoryTrackerSummary SampleInverse trace
field), so it is a pre-existing rare CDC/QuietDatabase flake unrelated to this
change.
* memory-tracker: fix GCC IPA-clone breaking frame-attribution tests
Under GCC -O3, IPA constant-propagation cloning (-fipa-cp-clone) specializes the
MemoryTrackerTest sentinels (each called with a constant N) into `.constprop`
clones emitted at a different address than the function symbol. The executed
code -- and thus the captured return addresses -- live in the clone, so the
tests' frameInside(frame, &sentinel) window missed them: fastAlloc32Accounting
aborted (sitesWithSentinelFrames == 0) in GCC CI while clang passed.
Add `noclone` to force_noinline on GCC so each sentinel stays a single body at
the address &fn yields. clang doesn't support noclone (and doesn't clone this
way), so it keeps noinline only; other compilers stay empty.
Testing: /flow/MemoryTracker/* passes 11/11 under both a gcc-toolset-13 build
and a clang build.
* memory-tracker: cheaper disabled alloc hot path (per-thread off flag)
When sampling is off, memTrackerOnAlloc previously still did three TLS accesses
every allocation -- a gInMemTracker load, a gMemTrackerCounter load-decrement-
STORE, and a gForceSampleBytes load. Add a per-thread gMemTrackerOff flag,
checked first, that a thread sets once its slow path observes sampling is off;
the disabled alloc path then short-circuits on a single TLS load + branch (no
counter store, no gForceSampleBytes load).
The flag is per-thread, not global, on purpose: the counter still bootstraps
sampling per thread (first alloc reaches the slow path and reads the knob), so a
single global gate set by an early main-thread allocation before FLOW_KNOBS is
ready would wrongly disable worker threads that bootstrap later. Free keeps the
global g_memTrackerEnabled gate (a free-only thread must see global state to
debit). memTrackerResetForTest clears the flag. Removes the now-dead INT_MAX
counter parking.
The saving is real but small (~2 loads + 1 store per alloc, well under the
mako A/B's run-to-run noise), so the redwood off-state A/B shows no resolvable
change; the win is on principle / at high allocation rates.
Testing: /flow/MemoryTracker/* passes 11/11 (bin/fdbserver -r unittests).
* memory-tracker: add FDB_MEMORY_TRACKER compile-time gate + per-path microbench
Add a compile-time switch, FDB_MEMORY_TRACKER (CMake option, default ON), that
removes the feature entirely when set to 0: the header hooks become no-op inlines,
flow/MemoryTracker.cpp and the MemoryTrackerTest TEST_CASEs are #if'd out (the
forceLink stub stays), and fdbserver/GlobalNewDelete.cpp defines no global
operator new/delete override (libc++'s allocator is used, which still honors the
installed new_handler). This gives operators an escape hatch to zero
always-compiled footprint, and lets the microbench measure present-but-disabled
vs absent cost per allocation path.
Extend fdbserver/bench/BenchMemoryTracker.cpp with plain per-path alloc/free loops
(operator new[] at several sizes, FastAllocator<64/96/256>, Arena medium/huge) so
the same binary built =1 (tracker present, sampling off by default) vs =0 (absent)
isolates each path's unweighted off-state overhead.
Measured (EPYC 9R14 @ 3.7 GHz, ns/op, =1 minus =0):
operator new[] small ~+1.1 ns (~11%); huge ~+2.5 ns
FastAllocator<N> ~0 (within the ~0.6 ns cross-build noise floor)
Arena block ~+3.3 ns (medium) / +4.8 ns (huge)
The per-op-costly paths (operator new, Arena) are the low-volume ones; the
high-volume path (FastAllocator, ~82% of redwood allocs) is ~free, so the weighted
off-state overhead is ~0.1-0.2% of a core -- under the R0 target, though R0 is now
framed as an unproven target with this gate as the escape hatch.
Testing: /flow/MemoryTracker/* passes 11/11 (default build); FDB_MEMORY_TRACKER=OFF
builds clean under -Werror.
* clang-tidy fix
* comment about why fdbserver/bench exists
* address Codex round 5 review comments (MSVC build, startup sequencing, memory allocation failures on tracker-internal bookkeeping, cross-compile, contrib script cleanup)
* add another disclaimer about MSVC/Windows being best effort
* memory-tracker: fail-open ordering + deflake huge-arena unit test
memTrackerSampleAlloc now performs both table insertions (aggregation-map and
live-map nodes) before mutating any per-site or global counter, so if the
tracker's own map growth throws std::bad_alloc the exception unwinds with all
totals in lockstep and the fail-open catch simply drops the sample. This
addresses an adversarial-review finding. Note such a failure does not occur in
practice: FDB "OOM" is an RSS threshold enforced by fdbmonitor (typically
~12-16 GB against an ~8 GB target), not a malloc/operator-new failure. This
feature is for finding leaks and untuned allocations that drive RSS growth,
well short of any allocation failure -- documented in flow/MemoryTracker.cpp and
design/memory-tracker.md so reviewers don't over-index on the bad_alloc path.
arenaHugeAccounting now identifies the huge blocks by their ~100 KB size
signature instead of the frame-pointer sentinel. The huge-Arena path is the
deepest tracked call chain; under a real (non-simulation) network the
best-effort frame-pointer walker is per-allocation nondeterministic there and
cannot reliably attribute the blocks to the test's frame, so the sentinel-frame
assertion flaked (~1-4% of runs). The recorded block size is correct regardless
of which frames were captured, no incidental or foreign-thread allocation comes
near 100 KB, and double-tracking still shows up as 2N. The shallower
fastAlloc32/arenaSmall/arenaMedium/operatorNew accounting tests keep exact
sentinel-frame attribution (reliable on their shorter paths) as the regression
guard.
Testing:
- /flow/MemoryTracker/* via `fdbserver -r unittests`: 500 runs across two seed
spaces (including the sequence that previously flaked), 14/14 test cases pass
every run, 0 failures.
- Builds clean with FDB_MEMORY_TRACKER on and off under -Werror; clang-format
and clang-tidy clean on the changed files.
- Joshua 100k on the parent commit: ensemble
20260728-160205-gglass-0b30fa74f2605807, ended=100000 pass=100000 fail=0.
These changes are a pure counter-update reorder (no simulation-reachable
behavior change -- bad_alloc is not injected in sim), a unit-test-only change,
and documentation, so that result remains representative.
* memory-tracker: note why the fdbserver-local bench binary exists
Add a short comment to BenchMain.cpp explaining that most microbenchmarks
belong in flow/bench and this fdbserver-local benchmark binary exists only for
benchmarks that must link fdbserver-only code, pointing at BenchMemoryTracker.cpp
for the detailed rationale rather than duplicating it.
Refactor ifdefs to make format_backtrace() be exposed for _WIN32, and remove WIN32 ifdefs in non-Platform.cpp code.
In shell commands emitted by format_backtrace, change the addr2line command to specify the full path to the addr2line binary, to avoid people getting hung up by the broken shell alias in /root/.bashrc. As far as I can tell this alias exists to invoke gcc- or clang-specific addr2line binaries, which is easy enough to solve in the .cpp code.
Delete logic that naively appends "debug" related suffixes to the fdbserver binary name. The code doesn't know if it's a debug binary or not, and if it's not, the most obvious solution is to rebuild everything. Copying binaries around manually is not needed and the emitted commands shouldn't encourage this, as far as I can tell. They should just use the given binary name.
Tested the addr2line invocations by inserting some segfaults and running the commands (with minor updates to give the absolute path to the fdbserver binary). Tested on gcc and clang debug builds.
* Specify absolute path to addr2line binaries, to avoid a broken shell alias. Also, move _WIN32 ifdefd code out of general purpose source files and into Platform.cpp
* s/brain dead/not helpful/
* fix comment
* Platform.actor.cpp: reduce size of __unixish__ block to enable format_backtrace to be defined for _WIN32
* format code
* fix i vs j bug in _WIN32 code; obviously not compiled or tested by us
* Refactor getDiskStats() now that it returns about 10 values.
Compute byte stats from sector level stats by assuming sectors are 512 bytes
and using multiplication. Rationale for why sectors can be assumed to be 512
bytes is given in a comment.
Add byte level rate stats to the trace output emitted in customSystemMonitor().
Testing: passed 99995 runs here:
20250715-001140-gglass-90a10b1f1bdd221a compressed=True data_size=41293265 duration=5559400 ended=100000 fail=5 fail_fast=10 max_runs=100000 pass=99995 priority=100 remaining=0 runtime=1:26:20 sanity=False started=100000 stopped=20250715-013800 submitted=20250715-001140 timeout=5400 username=gglass
I couldn't figure out about the 5 failures but I don't think they are related to my change.
* Fix compile error on MacOS relating to readMilliSecs et al now being members of diskStats
---------
Co-authored-by: Gideon Glass <gglass_glass@apple.com>
* Print an asan heap profile on OOM
* Use 32KiB stacks for boost coro
* Print 100%, 10 max contexts for asan OOM
* Lower machineCount to 30 in DataLossRecovery test
* Add asanMachineCount override to control ASAN memory usage
Per #7952, content in /sys/fs/cgroup/cpu,cpuacct/cpu.stat is reported in
MachineMetrics.
If the file does not exist, reports `NoCpuStatFile`.
If the file is not parsable, reports `CpuStatFileParseError`.
Changing `memory` option to limit resident memory instead of virtual memory, in config file and fdbserver/fdbbackup/fdbcli command-line argument. Since `rlimit` doesn't support limiting virtual memory, the current implementation have both of fdbmonitor and the fdbserver/fdbbackup process checking process RSS periodically and kill and restart the process if the limit is exceeded.
Adding a new `memory_vsize` option to limit virtual memory, if backward-compatible behavior is desired.
closes#6671, closes#6672
* Add contrib/debug_determinism
Add an instrumentation-based technique for debugging unseen mismatches. Also guard a few existing sources of nondeterminism that don't affect unseen with the DEBUG_DETERMINISM macro.
Also change the simulated run loop to not run as the only task inside the real run loop, since that was a source of nondeterminism.
Also fix nondeterminism from calling timer_int
* Add StorageMetadataType::currentTime
Basically a deterministic-in-simulation version of timer_int that we can
use instead of timer_int for StorageMetadataType::createdTime
1. Introduce processDiskReadSeconds and processDiskWriteSeconds, which stands for disk read/write times `since the last logging`. They can only be obtained on Linux and macOS, and will be 0 on Windows and FreeBSD;
2. Rename `busyTicks` to `IOMilliSecs`;
3. On FreeBSD, the metrics should be collected among all devices.
This change seems to be incorrect since afaict INetwork::timer isn't
guaranteed to be monotonic. Maybe we can make that guarantee or add an
INetwork::timer_monotonic symbol?
gxglass