83a317ef88
Disable guard pages for valgrind build. Closes #7610 ( #7672 )
2022-07-22 16:51:05 -07:00
03d913a1de
Flow compiling
2022-06-27 17:05:55 -06:00
297d831192
Put guard pages next to fast alloc memory ( #6885 )
...
* Put guard pages next to fast alloc memory
I verified that we can now detect #6753 without creating tons of
threads.
* Use pageSize instead of 4096
* Don't include mmapInternal for windows
2022-04-19 11:22:35 -07:00
a71099471b
Update copyright header dates
2022-03-21 13:36:23 -07:00
68c03a7e32
Jemalloc integration fixes ( #6626 )
...
* Set default for USE_JEMALLOC initially in ConfigureCompiler
Instead of trying to change the value later on. This fixes the valgrind
build, which was previously incorrectly getting jemalloc involved.
* Check aligned_alloc result for null
And OOM if so - don't assert
* Check that we can allocate magazines with no internal fragmentation
We may want to do this so that the jemalloc heap profiler has some
knowledge of FastAlloc
* Populate TestFile field for noSim tests in TestHarness
* Remove handling for nonexistent "ActualRun"
2022-03-17 15:17:27 -07:00
ccc11cbdb2
Apply jemalloc fix, and always build jemalloc ( #6610 )
...
* Apply jemalloc fix, and always build jemalloc
See https://github.com/jemalloc/jemalloc/pull/1924
* Move USE_JEMALLOC to flow/config.h
* Add unit test to validate fix
2022-03-16 13:19:28 -07:00
2bd2ddd779
Fix FastAlloc thread cleanup on OSX ( #6485 )
...
* Use ThreadData constructor and destructor
Instead of pthread api
* Make threadData a function static thread local variable
* Fix linux build
2022-03-04 12:49:40 -08:00
7a9217a392
Add contrib/debug_determinism ( #6389 )
...
* Add contrib/debug_determinism
Add an instrumentation-based technique for debugging unseen mismatches. Also guard a few existing sources of nondeterminism that don't affect unseen with the DEBUG_DETERMINISM macro.
Also change the simulated run loop to not run as the only task inside the real run loop, since that was a source of nondeterminism.
Also fix nondeterminism from calling timer_int
* Add StorageMetadataType::currentTime
Basically a deterministic-in-simulation version of timer_int that we can
use instead of timer_int for StorageMetadataType::createdTime
2022-02-25 12:54:31 -08:00
371a38e6e5
Merge remote-tracking branch 'origin/master' into remove-extra-copies
2021-06-07 10:26:06 -07:00
abf7e4c8dc
Fix signed integer overflow
2021-06-04 10:00:05 -07:00
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
5f64a0942f
Align FastAllocator memory to 4096 for size >= 4096
2021-03-16 00:44:35 +00:00
d09603b6b3
Explain FastAllocator<4096> special case
2021-03-13 00:36:29 +00:00
a76c7b9754
Fix several memory leaks and a thread leak
2021-03-11 23:58:32 +00:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
aaf0a9aa7b
Merge branch 'release-6.3' into merge-release-6.3-into-master
...
# Conflicts:
# build/docker-compose.yaml
# cmake/ConfigureCompiler.cmake
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/IAsyncFile.h
# fdbrpc/IRateControl.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbservice/ServiceBase.cpp
2021-02-08 12:58:34 -08:00
ff0667b28d
winbuild: define BOOST_USE_WINDOWS_H, WIN32_LEAN_AND_MEAN and NOMINMAX to fix various macro expansions issues
...
- NOMINMAX to fix confusion between std::max(..) and max(..) macro redefinition
- BOOST_USE_WINDOWS_H to prevent boost from redefining win32 APIs
- WIN32_LEAN_AND_MEAN to fix include ordering issues with winsock.h
2021-01-30 15:15:02 +01:00
18ac901c40
Align memory for FastAllocator with valgrindPrecise
2020-10-23 02:11:50 +00:00
ab4eb627c2
Replace with environment variable
...
We need to change the behavior of the allocator before main is called,
so a knob won't work.
2020-10-22 16:31:48 +00:00
79163947a6
Add FLOW_KNOBS->VALGRIND_PRECISE
2020-10-22 16:31:48 +00:00
0ac08f6a9b
Replace NULL with nullptr in flow
2020-09-20 11:31:49 -07:00
0c25a953be
Fix compiler error. This cast is safe as it's only used locally so the data layout doesn't matter at all.
2020-09-08 16:05:38 -07:00
440d46bccc
Merge pull request #3523 from Daniel-B-Smith/global-data-races
...
Fix some unprotected accesses to global data
2020-07-29 11:44:11 -07:00
f4c54e5ee6
use std::atomic for simple counters
2020-07-28 17:51:21 +00:00
38cade1146
Merged tag release-6.3.3 into tls-background-handshake.
2020-07-17 23:39:57 -07:00
c6b691d917
Revert "Revert "Fix some unprotected accesses to global data""
...
This reverts commit ac724589cc
2020-07-17 18:02:23 +00:00
ac724589cc
Revert "Fix some unprotected accesses to global data"
...
This reverts commit 6d63aebbbc
2020-07-17 18:02:13 +00:00
6d63aebbbc
Fix some unprotected accesses to global data
2020-07-15 15:57:05 +00:00
b09dddc07e
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/fdbrpc.vcxproj
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
4ce51c4cce
Merge pull request #2768 from atn34/atn34/arena-cpp
...
Check for memory errors in arena-allocated memory
2020-07-06 09:43:28 -07:00
0da4c91ad5
Performance improvements in socket use and buffering which results in TLS network connections using around 3.5x less CPU.
...
PacketBuffers are 16k minimum sizes, using FastAlloc. All calling paths to PacketBuffer creation now pass a size hint if known. P2PNetworkTest improved in many ways, errors go to TraceEvents instead of stdout, error counts are reported in stats, range min/max parameters are combined into "min:max" strings, new options to "stutter" connections by delaying randomly before socket read or write, message size to send is now configured separately for incoming vs outgoing sessions.
2020-06-25 20:44:43 -07:00
95bc24de11
flow: update headers and includes
2020-04-30 18:11:23 +00:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
96187618a0
Fix condition to check whether allocation tracing is enabled
2020-03-16 15:12:50 -07:00
d8cfabe73b
Extend the allocation tracing disabling flag to cover more parts of trace logging as a precaution. Make it possible to disable via knob.
2020-03-16 13:59:31 -07:00
2466749648
Don't disallow allocation tracking when a trace event is open because we now have state trace events. Instead, only block allocation tracking while we are in the middle of allocation tracking already to prevent recursion.
2020-03-12 11:17:49 -07:00
41cf0745a4
Make FastAllocator responsible for marking its internal state addressable
2020-03-03 13:40:54 -08:00
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
31fbf84ac5
Make FastAlloc use crc32c instead of hashlittle2
2020-01-13 18:23:12 -08:00
396b10caca
Add memory profiling for FastAlloc when gperftool is used
...
FastAlloc is the major memory use case in FDB, yet we can't profiling its usage.
This commit replaces FastAlloc memory allocation with malloc so that we may
track its memory usage when gperftool is used.
2019-10-07 19:27:06 -07:00
b5d2234a13
Merge branch 'release-6.1' into merge-release-6.1-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbserver/MoveKeys.actor.cpp
# flow/FastAlloc.h
# versions.target
2019-07-30 16:23:42 -07:00
d981de18e4
Restrict huge arena sampling to the network thread. Revert removal of thread_local definitions.
2019-07-17 16:23:17 -07:00
7d6dcc92d0
I accidentally omitted the important change here.
2019-07-16 15:32:06 -07:00
0bfd474e21
Fix huge arena tracking thread-safety issues. Fixes #1846 .
2019-07-16 15:22:19 -07:00
844dd60202
FDB compiling with intel compiler
2019-06-20 09:29:01 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
a8b9d8e34b
Merge pull request #1336 from tclinken/fast-allocate-ptree-nodes
...
Create 96-byte fast allocator for storage queue PTree nodes
2019-05-17 14:22:46 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
22499666d0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/LogRouter.actor.cpp
# flow/Trace.cpp
# versions.target
2019-05-08 18:19:35 -07:00
aefd68e1e7
The memory tracking trace events can crash if the memory that is being allocated is coming from a trace event itself. Specifically TDMetrics within trace events uses fast allocated memory. Trace events are supposed to be short lived, so this commits adds a global count of outstanding trace events, and disables memory tracking when a trace event exists.
2019-05-06 17:41:32 -07:00
Andrew Noyes