5ea2e69016
Remove a fdbprc header from flow library
...
Flow should be an independent library.
2019-07-03 19:56:38 -07:00
3fb0999e10
revert storage server priority changes
2019-07-02 16:54:47 -07:00
8e1ab6e7db
Merge remote-tracking branch 'upstream/master' into flowlock-api
2019-06-28 17:32:54 -07:00
4cef1d3937
Experimental change of storage write priority
2019-06-28 16:54:22 -07:00
1b939d5208
Merge pull request #1749 from satherton/feature-redwood
...
Update redwood storage engine to latest correctness-passing version
2019-06-28 16:22:06 -07:00
5ccffe3cb3
Merge pull request #1684 from jzhou77/large-packet
...
Better handling for large packets
2019-06-28 16:19:01 -07:00
7f4586ad49
the number of txsTags needs to be tracked separately from the number of transaction logs because of forced recoveries
2019-06-28 12:33:24 -07:00
52efcfd136
fix: properly create the right number for txsTags when changing between different numbers of logs
2019-06-27 15:15:05 -07:00
7efdf492ce
Fixed two silly compilation bugs
2019-06-27 10:29:59 -07:00
a2a5bcc73d
Merge remote-tracking branch 'upstream/master' into flowlock-api
2019-06-26 20:31:09 -07:00
7f23814841
Track run loop busyness and report it in status.
2019-06-26 14:03:02 -07:00
e5d02b7ad4
use _WIN32 to determine platform instead of __WIN32__
2019-06-26 11:55:26 -07:00
bf883d7055
Merge remote-tracking branch 'upstream/master' into flowlock-api
2019-06-25 14:26:50 -07:00
0fe6edc254
Merge pull request #1678 from mpilman/features/external-workload
...
Features/external workload
2019-06-25 13:53:19 -07:00
e6ff23c420
Allow next packet size to be read when receiving data
...
For large packet, allocate sizeof(uint32_t) more bytes for next packet size.
Also add knob MIN_PACKET_BUFFER_FREE_BYTES, which is used to trigger allocation
of a new arena when free bytes are lower than this threshold.
2019-06-25 10:18:56 -07:00
c2c82c89a5
Remove knob DEFAULT_PACKET_BUFFER_BYTES
2019-06-25 10:18:56 -07:00
2363326ecb
Add knobs for various packet buffer sizes
2019-06-25 10:18:56 -07:00
7a500cd37f
A giant translation of TaskFooPriority -> TaskPriority::Foo
...
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
f1f1081202
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-06-24 20:17:49 -07:00
54c2b8d5bf
pseudoLocalities was introduced in the current protocol version, which is 6107
2019-06-24 17:04:52 -07:00
a812b93fbf
Fixed version for PseudoLocalities
2019-06-23 09:46:18 -07:00
37c1df2491
Merge pull request #1705 from bnamasivayam/suspend-process
...
Extend RebootRequest API to include time to suspend the process befor…
2019-06-20 17:36:25 -07:00
df0baa0066
Merge pull request #1720 from mpilman/features/protocol-version
...
Make protocol version a type
2019-06-19 13:46:35 -07:00
2eff2b7e21
First simple test is working (but very buggy)
2019-06-19 13:03:41 -07:00
c8957d93f8
Implementation code complete
2019-06-19 13:03:41 -07:00
68ce9a5e75
ProtocolVersion type - second try
2019-06-18 17:55:27 -07:00
0e5773106c
Platform.cpp is a mess, and I was far too optimistic with that push.
2019-06-18 16:50:08 -07:00
ed118bac17
Fix copy-paste error.
2019-06-18 16:40:17 -07:00
4fa5dc0502
Merge remote-tracking branch 'upstream/master' into cloexec
2019-06-18 16:35:18 -07:00
5da8746803
Move fnctl calls into Platform function, and add traceevent on error.
2019-06-18 16:34:35 -07:00
8576665a90
Revert "Revert "Make protocol version a type""
...
This reverts commit 455bf3b3ec
2019-06-18 14:49:04 -07:00
455bf3b3ec
Revert "Make protocol version a type"
2019-06-18 10:59:17 -07:00
c3aa5819f2
Merge pull request #1417 from mpilman/features/client-buggify
...
Overall framework and first buggify entries
2019-06-18 09:10:11 -07:00
4a14117549
Fixed typo
2019-06-17 09:30:28 -07:00
52da31bfb3
Simplifications and addressed reviews
2019-06-16 11:07:10 -07:00
882fa39e80
fixed typo
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-06-16 09:59:15 -07:00
23fb403216
Fix typo
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-06-16 09:59:15 -07:00
dc9522bd86
Address code-review comments
2019-06-16 09:59:15 -07:00
9fd8dac767
Added license header to new file
2019-06-16 09:59:15 -07:00
da53a92bec
Make protocol version a type
...
This fixes #1214
The basic idea is that ProtocolVersion is now its own type. This
alone is an improvement as it makes many things more typesafe. For
each version, we can now add breaking features (for example Fearless).
After that, there's no need to test against actual (confusing) version
numbers. Instead a developer can simply test
`protocolVersion->hasFearless()` and this will return true iff the
protocolVersion is newer than the newest version that didn't support
fearless.
2019-06-16 09:59:15 -07:00
7483cba26e
Address review comments and added documentation
2019-06-16 09:52:40 -07:00
6ea75713cb
Overall framework and first buggify entries
2019-06-16 09:09:09 -07:00
5eb833759e
Extend RebootRequest API to include time to suspend the process before reboot. This is intended to be used for testing purposes to simulate failures.
2019-06-14 11:35:38 -07:00
6ececa94ce
Merge pull request #1640 from vishesh/task/client-failmon
...
Clients will no longer get failure monitoring info from cluster controller
2019-06-13 17:31:17 -07:00
1e8f7e5b82
Refactor NextFastAllocatedSize to be constexpr function
2019-06-11 15:55:23 -07:00
4c62458172
Make valgrind work on Fedora 30
2019-06-11 10:27:08 -07:00
a8e408e268
run clang-format on changes
2019-06-10 14:10:24 -07:00
0b07dbd05c
Remove a horrible hack aiming at Ubuntu 14.04's libstdc++_pic.
...
Because our supported build environments no longer include Ubuntu 14.04,
and this is causing problems with libc++-based builds.
2019-06-10 13:16:03 -07:00
6fa7081a21
net: Don't make FailureMonitoring requests from client
...
This patch removes the need for clients to continuously contact
cluster coordinator for failure monitoring information. Instead, it
uses the FlowTransport to monitor the statuses of peers and update
FailureMonitor accordingly.
2019-06-09 00:43:38 -07:00
6b4d30c3ae
failmon: Identify client vs server when starting failure monitoring client
2019-06-09 00:43:12 -07:00
526468becb
Fix printable() being incorrect for bytes >127.
...
The use of `auto` hid that TraceableString<T>::begin(value) was a signed
char, and therefore `*iter` was -1, and -1/16 is 0.
Thus, a 0xFF byte was printed as \x0f. Forcing it to an unsigned type
makes it correctly become \xff.
2019-06-05 14:41:31 -07:00
c027792d0f
Make the default seed of the generator returned by deterministicRandom() random. This means that the random number sequence won't be deterministic unless you set a seed.
2019-06-05 11:48:29 -07:00
773f52d0a1
Merge remote-tracking branch 'upstream/master' into cloexec
2019-06-03 15:43:32 -07:00
a8c9432ced
windows build fix : del sys/wait.h in Platform.cpp
2019-05-31 14:59:53 -07:00
362c2bf1e6
improved the cpu efficiency of printable
2019-05-29 14:55:45 -07:00
02882dbf00
Checkpointing progress, RedwoodRecordRef and DeltaTree tests pass but BTree test does not. RedwoodRecordRef::Delta rewritten to actually do prefix compression on key and integer fields. Added related unit tests and benchmarks. Some improvements to DeltaTree and requirements on its T and Delta types to avoid repeated common prefix discovery.
2019-05-29 06:23:32 -07:00
b17ad85497
exec op not supported when log_anti_quorum > 0
2019-05-28 22:07:46 -07:00
bb474dc323
if recovery < fully_recovered then fail the exec
...
Will do more cleanup, pushing it for a test run in CI
2019-05-28 22:07:46 -07:00
925499954b
New status cluster_not_fully_recovered
2019-05-28 22:07:46 -07:00
d68a229772
makefile changes to accommodate boost/process.hpp
2019-05-28 22:07:46 -07:00
b6e037ffbc
Replace fork with boost::process::child
2019-05-28 22:07:46 -07:00
898bed66c1
Allow only whitelisted binary path for exec op
2019-05-28 22:07:46 -07:00
aa79480d69
changes to make fdbfork asynchronous
2019-05-28 22:07:46 -07:00
4bc4c615da
exec op to all tlog, restore change in test &other
...
- exec operation to go to all the TLogs
- minor bug fix in tlog
- restore implementation for the simulator
- restore snap UID to be stored in restartInfo.ini
- test cases added
- indentation and trace file fixes
2019-05-28 22:07:46 -07:00
72dd067173
Trace message changes and fix few FIXMEs
2019-05-28 22:07:46 -07:00
69edefe68b
Snapshot based backup and resotre implementation
2019-05-28 22:07:46 -07:00
ec80ff1e8d
Remove unneeded redefinition of random functions.
2019-05-24 11:55:25 -07:00
fc204c846a
Make IRandom reference counted. Simplify the logic around deterministicRandom() to require less state and branching.
2019-05-23 18:51:59 -07:00
764ca2971c
Merge branch 'master' into thread-safe-random-number-generation
2019-05-23 18:49:49 -07:00
296adada93
Use c++11 thread_local instead of redefining it to platform specific variants.
2019-05-23 18:35:45 -07:00
f417e60264
Merge branch 'merge-release-6.1-into-master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbserver/QuietDatabase.actor.cpp
2019-05-23 09:52:00 -07:00
d29c7e4c9b
Merge branch 'release-6.1' into merge-release-6.1-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/QuietDatabase.actor.cpp
# versions.target
2019-05-23 09:28:45 -07:00
e5381e0612
Fix some new usages of g_random
2019-05-23 09:23:27 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
ebc96a7e0e
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-05-21 23:49:27 -07:00
d3c1f9afbd
Suppress 'TraceEventFieldNotFound' event. Don't suppress simultaneous FDBLibTLSVerifyFailure events (this requires including flow.h and removing the copied version of Reference/ReferenceCounted in FDBLibTLS).
2019-05-20 15:12:28 -07:00
d660ffb654
Merge pull request #1605 from mpilman/remove-traceevent-benchmark
...
remove memory-hungry benchmark
2019-05-17 17:04:45 -07:00
a8b9d8e34b
Merge pull request #1336 from tclinken/fast-allocate-ptree-nodes
...
Create 96-byte fast allocator for storage queue PTree nodes
2019-05-17 14:22:46 -07:00
a0fa81aa1c
remove memory-hungry benchmark
...
This test was used as a benchmark to prove that the
new `IsTraceable` feature improves performance. However
it will cause smaller machines to OOM and it doesn't
help for correctness testing. Therefore this test should
likely be removed
2019-05-17 14:12:44 -07:00
5a8c97480a
Merge pull request #1506 from nikolas-ioannou/feature-pagecache-lru
...
AsyncFileCached: switch from a random to an LRU cache eviction policy
2019-05-17 13:42:21 -07:00
f598d0a614
Merge pull request #1593 from AlvinMooreSr/variant-missing
...
Switched Variant Header
2019-05-17 10:42:21 -07:00
3acaa7343e
Enabled C++17 for all Windows projects
...
Set Visual Studio version to 2017 (first version to support C++17)
2019-05-16 17:44:13 -07:00
94aed513c7
Switched Windows tools within projects to 2017
2019-05-16 15:05:11 -07:00
17a2983943
Switch boost variant to C++17 variant
2019-05-16 13:05:49 -07:00
6389a12931
Switched variant to use boost variant for source consistency
2019-05-16 10:37:33 -07:00
69fb852ee0
Add more CLOEXEC-like things.
...
From missed call sites found during/after code review.
2019-05-14 20:30:58 -10:00
1f02cd30e2
Mark all opened files as close-on-exec.
2019-05-13 16:05:49 -10:00
19165dc873
Mark all network sockets as close-on-exec.
2019-05-13 16:05:49 -10:00
a9b03ec505
Merge pull request #1090 from mpilman/features/protocol-version-flags
...
Features/protocol version flags
2019-05-13 16:33:06 -07:00
60ce790a2b
Consistent use of #if VALGRIND
2019-05-13 12:44:13 -10:00
436645e956
Update flow/ThreadSafeQueue.h
...
Co-Authored-By: Alex Miller <35046903+alexmiller-apple@users.noreply.github.com>
2019-05-13 12:44:13 -10:00
7d38a1e273
Use std::atomics in thread-safe queue.
2019-05-13 12:44:13 -10:00
b601fb2df1
Added flat_buffers source files to Windows project in order to support building via make
2019-05-13 14:22:36 -07:00
5a13915960
implemented std::set serializer for flatbuffers
2019-05-13 14:15:23 -07:00
46e7a0ca56
address reviews and make compile with `-Wunused-variable`
2019-05-13 14:15:23 -07:00
a92233a371
Added TODO to optimize packet serialization
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-05-13 14:15:23 -07:00
b4fa13c33b
Remove obsolete function
2019-05-13 14:15:23 -07:00
96aaa31a6c
Compiling on clang again
2019-05-13 14:15:23 -07:00
0713e06efc
Started to work on Windows
2019-05-13 14:15:23 -07:00
42385c2f81
Fixed issues introduced during rebase
2019-05-13 14:15:23 -07:00
186bff4ae5
removed unnecessary assert and added test
2019-05-13 14:15:23 -07:00
f5fa3a65b4
some more fixes
2019-05-13 14:15:23 -07:00
44db3450ec
Several flatbuffers bug fixes
2019-05-13 14:15:23 -07:00
69fa3d3903
fixed compilation issues after rebase
2019-05-13 14:15:23 -07:00
43a7703a1e
fix addFlag
2019-05-13 14:15:22 -07:00
f0d81c6820
make EnsureTable work with streaming serializer
2019-05-13 14:15:22 -07:00
7affe73774
Added more low-level tests and implemented enums
2019-05-13 14:15:22 -07:00
0a3676111e
Support vector of unions in flatbuffers
2019-05-13 14:15:22 -07:00
642a96807b
Fixed compilation issues after rebase
2019-05-13 14:15:22 -07:00
b06fe39b47
renamed flatbuffers tests for consistency
2019-05-13 14:15:22 -07:00
1df3d8f0c7
disable rlimit when ASAN is used
2019-05-13 14:15:22 -07:00
6afce01744
Implementation complete (not yet working)
2019-05-13 14:15:22 -07:00
1d5572a078
removed unused knob
2019-05-13 14:15:22 -07:00
9eeb48c43d
Allow to turn on object serializer
...
This commit includes functionality to turn on
the object serializer for network communication.
This is done the following way:
- On incoming connections, a process will detect
whether the client supports the object serializer
and will only serialize responses with it, if it does
- On outgoing connections, the command line flag is used
to determine whether the object serializer should be used
to send data.
This way, a cluster can run in mixed mode. To upgrade one
can upgrade one process at a time and set the flag one process
at a time.
This is how this is tested on the simulator:
- The command line flag can take three options: on, off,
and random.
- For off, the object serializer will never we used.
- For on, the object serializer will be always used.
- For random, the simulator will flip a coin for each
process it starts up.
2019-05-13 14:15:22 -07:00
ba83c458a6
types implemented
2019-05-13 14:15:22 -07:00
fe81454ec2
basic functionality for object serializer
...
This commit includes:
- The flatbuffers implementation
- A draft on how it should be used for network messages
- A serializer that can be used independently
What is missing:
- All root objects will need a file identifier
- Many special classes can not be serialized yet as the
corresponding traits are not yet implemented
- Object serialization can not yet be turned on (this will
need a network option)
2019-05-13 14:15:22 -07:00
8bbd28315e
Added Object serializer (does not yet compile)
2019-05-13 14:15:22 -07:00
8e1927a5cb
Rename trace event field to avoid duplicating 'Time' field.
2019-05-13 13:25:01 -07:00
ab4618c3f9
Don't access TraceEvent::eventCounts outside the network thread. Make net2liveness an atomic counter.
2019-05-10 12:35:11 -10:00
acb2fc67ac
Remove unnecessary redefinition of new random functions. Enable rand logging for the generators returned by deterministicRandom().
2019-05-10 14:16:48 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
22499666d0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/LogRouter.actor.cpp
# flow/Trace.cpp
# versions.target
2019-05-08 18:19:35 -07:00
c2827f4fa3
Add page cache hit, miss, and eviction stats to SystemMonitor
2019-05-08 15:41:17 +02:00
a67d47844b
coveragetool: suppress output unless VERBOSE is set
...
When using the CMake build system with tools such as Ninja (instead of
Make), they have very concise output, and the general "UX flow" involves
only outputting warnings that should be fixed or errors that should stop
the build. Furthermore, when building things in CI systems and looking
at build logs, etc, it's often vastly easier to to hunt down errors when
only relevant stuff is output.
With this change, the output from coveragetool.exe is suppressed, which
accounts for a lot of spurious output that isn't necessary when running
these builds.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-07 16:54:05 -10:00
aefd68e1e7
The memory tracking trace events can crash if the memory that is being allocated is coming from a trace event itself. Specifically TDMetrics within trace events uses fast allocated memory. Trace events are supposed to be short lived, so this commits adds a global count of outstanding trace events, and disables memory tracking when a trace event exists.
2019-05-06 17:41:32 -07:00
7a10199a35
flow: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
41ae9cd0d8
Remove ; for namespace declarations
2019-05-06 13:21:28 -07:00
d6170210e7
AsyncFileCached: throw (new) exception, through helper static fn, if cache eviction polity not recognized.
2019-05-06 10:11:46 +02:00
fdb3992990
AsyncFileCached: switch to string for eviction policy knob
2019-05-03 14:04:43 +02:00
b2280e15bf
AsyncFileCached: support for lru cache eviction policy
...
- Added a knob to control page cache eviction policy (0=random default, 1=lru)
- Added page cache hit, miss, and eviction stats
2019-05-02 17:35:30 +02:00
e193cac5ef
Merge remote-tracking branch 'apple/master' into tlog
...
Resolve Conflicts: fdbserver/MasterProxyServer.actor.cpp
2019-05-01 17:18:00 -07:00
2d5043c665
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-30 18:27:04 -07:00
4fa1c008f9
Highly prioritize storageServerRejoin messages on the proxy, so that storage servers can rejoin the cluster even when a proxy is CPU saturated
2019-04-23 20:56:01 -07:00
e0f76edf77
Merge pull request #1471 from AlvinMooreSr/release-6.1-merge
...
Merge Release 6.1 Into Master
2019-04-23 11:08:21 -07:00
c25fac9421
Lower the priority of spilled peeks to below that of spilling.
...
This prevents peeking from starving the TLog of CPU time to spill, and
thus impacting write throughput at prolonged saturation.
2019-04-22 18:39:21 -07:00
8b67da57bb
Fix upgrade test failure
...
Serialize pseudoLocalities if protocol version is larger than 0x0FDB00B061060001LL.
Note this version may need to be changed to "currentProtocolVersion" when merging
into the master, and "currentProtocolVersion" should be incremented.
2019-04-21 10:41:07 -07:00
e7ee61660c
Fix unit test failure due to trace event names
...
Capitalize the first letter.
2019-04-20 10:19:53 -07:00
2bea99591e
Merge branch 'release-6.1' of copy of master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-04-17 15:51:48 -07:00
6888baf1c3
Fix compile error
2019-04-16 15:44:55 -07:00
08e66d1764
Prefer success to cast-to-void
2019-04-16 15:29:03 -07:00
781b6ece77
Fix OPEN_FOR_IDE -Wunused-variable warnings
...
CC #1255 , #1173
2019-04-16 15:28:01 -07:00
baa3e806ef
Address review comments from #1446
2019-04-16 09:48:15 -07:00
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
6c7fd1593f
Avoid potential index-out-of-bounds
2019-04-15 15:20:09 -07:00
dced9232d0
ASSERT_WE_THINK trailing byte is '\0'
2019-04-15 09:43:26 -07:00
369c1a0230
Exclude trailing \0 byte for string literals
...
This will "do the right thing" as long as no one expects something like
char a[] = {'a'};
TraceEvent("Event").detail("a", a);
to trace `{..., "a": "a"}`.
It will trace `{..., "a": ""}`
2019-04-11 10:39:48 -07:00
1135331ce6
Include space as a printable character
2019-04-11 10:01:06 -07:00
cd5c9d91fa
Merge pull request #1443 from etschannen/master
...
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
8e05713a5d
do not log a SevError trace event if we cannot deserialize the connect packet
2019-04-10 17:41:02 -07:00
a9b1b32fde
Handle negative input to base16Char
...
base16Char was being called with (c < 0), causing a failed assertion.
2019-04-09 18:14:09 -07:00
6220a5ce0f
Merge pull request #1370 from jzhou77/fix-unreferenced
...
Remove unused functions
2019-04-09 11:49:45 -07:00
058d028099
Merge pull request #1301 from mpilman/features/cheaper-traces
...
Defer formatting in traces to make them cheaper
2019-04-09 10:11:04 -07:00
af17babb9a
Restore a deleted comment line
2019-04-09 09:38:45 -07:00
e4746888eb
Removed dead code
2019-04-09 09:08:45 -07:00
21c0ba555c
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-08 18:38:42 -07:00
05869a8383
do not log a degraded reset message if the previous reset was more than a week ago
2019-04-07 23:00:58 -07:00
19d9b8fcd8
don’t log MachineLoadDetails from getSystemStatistics()
2019-04-07 22:55:19 -07:00
d7612a4426
Fix OPEN_FOR_IDE build errors
2019-04-05 16:30:42 -07:00
d01cbf3455
Addressed code review comments
2019-04-05 13:12:20 -07:00
4287b1d2a1
resolved minor merge issues
2019-04-05 13:12:19 -07:00
b0877e7dd0
Don't quote spaces in std::strings
2019-04-05 13:12:19 -07:00
1c6c73af6d
Fixed infinite recursion
2019-04-05 13:12:19 -07:00
c6bc23791d
Fixed format error in microbench
2019-04-05 13:12:19 -07:00
ea67b742c7
Implemented Traceable for printable types
2019-04-05 13:12:19 -07:00
c9ee7f54e7
Add compound type to micro-benchmark
2019-04-05 13:12:19 -07:00
8120a33bda
Fix `>`-direction when initializing TraceEvent
2019-04-05 13:12:19 -07:00
32a3aea8bb
Make strings random in tracing micro-benchmark
2019-04-05 13:12:19 -07:00
bb82f8560a
process all volatile ints correctly in traces
2019-04-05 13:12:19 -07:00
9938a49c1e
make primitive test longer
2019-04-05 13:12:19 -07:00
8af0f15a43
primitive performance test for TraceEvent
2019-04-05 13:12:19 -07:00
ae84f5424a
Fix potential segfault when init changes enabled
2019-04-05 13:12:19 -07:00
c008e16c81
Defer formatting in traces to make them cheaper
...
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:
- TraceEvent::detail now takes a c-string instead of std::string for
literals. This prevents unnecessary allocations if the trace is not
going to be printed in the first place (for example for SevDebug).
Before that `detail` expected a `std::string` as key, which mean that
any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
specialized for any type that needs to be printed. The actual
formatting will be deferred to after the `enabled` check. This
provides two benefits: (1) if a TraceEvent is disabled, we don't pay
for the formatting and (2) TraceEvent can trace types that it doesn't
know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
calls, a call to detail will only introduce a if-branch which is much
cheaper than a function call.
2019-04-05 13:12:19 -07:00
390ab9cfed
A process will mark itself as degraded if it continually disconnects from a different process which the failure monitor thinks is healthy
2019-04-04 14:11:12 -07:00
2b75c2e684
Restore removed functions.
...
crc32c.cpp is 3rd party code.
orYield() in genericactors.actor.h might be used in the future code.
2019-04-04 13:24:55 -07:00
39c595223b
Merge branch 'release-6.1'
2019-04-02 22:30:02 -07:00
985e1de467
Update documentation and protocol version for release 6.1.0
2019-04-02 18:39:24 -07:00
47b4b82628
Merge branch 'master' into fix-unreferenced
2019-04-01 14:07:19 -07:00
3f76be8f45
Merge remote-tracking branch 'apple/master' into fix-unreferenced
2019-04-01 14:00:43 -07:00
f7f8ddd894
Fix warnings on unused variables
...
Found by -Wunused-variable flag.
2019-04-01 14:00:20 -07:00
a46620fbee
Merge branch 'release-6.1'
2019-03-30 17:59:28 -07:00
d670b74d69
prevent trace event spam by combining huge arena samples
2019-03-30 13:36:13 -07:00
d882c060bf
Merge commit '5dd6396eed0de0dfea6cf9eecc307995eff5cedc'
2019-03-28 18:00:55 -07:00
aa368c08a2
changed NetworkAddress hash function to use more bytes from the IP address
2019-03-28 17:47:13 -07:00
80ecb12190
change the IPv6 hash function to be more efficient
2019-03-28 14:07:46 -07:00
b6008558d3
renamed BinaryWriter.toStringRef() to .toValue(), because the function now returns a Standalone<StringRef>()
...
eliminated an unnecessary copy from the proxy commit path
eliminated an unnecessary copy from buffered peek cursor
2019-03-28 11:52:50 -07:00
836bb95a7a
Merge pull request #1372 from etschannen/master
...
Merge 6.1 into master
2019-03-27 21:00:49 -07:00
34b9d5e722
Merge pull request #1364 from etschannen/feature-fast-serialize
...
A few performance optimizations
2019-03-27 20:57:25 -07:00
6997075917
changed back to isV6addr instead of isV4addr for compatibility
2019-03-27 19:55:36 -07:00
2fbecac10f
fix: allocation strategy did not take into account the data used by the ArenaBlock
2019-03-27 19:54:58 -07:00
e5a80f2c94
optimized IPaddress
2019-03-27 18:21:13 -07:00
a55f06e082
Remove unused functions
...
Found with -Wunused-function flag.
2019-03-27 15:45:28 -07:00
d5c8b6b083
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
# flow/flow.h
2019-03-27 13:37:15 -07:00
71e2fdafb8
Changes to ratekeeper camel case
2019-03-27 08:24:25 -07:00
9e8237955f
BinaryWriter uses arena allocation
2019-03-26 17:27:33 -07:00
38c6681349
Fix some signed and unsigned mismatch warnings.
2019-03-26 14:54:11 -07:00
ea38b78768
BinaryWriter uses fastAllocator
2019-03-26 12:06:39 -07:00
dbcf1d717c
Added calls to TRACEALLOCATOR(96) and DETAILALLOCATORMEMUSAGE(96)
2019-03-25 13:45:07 -07:00
4a3467296f
Removed Arena96
2019-03-25 13:44:53 -07:00
007abbc45b
Added 96-byte FastAllocator
...
Since storage queue nodes account for a large portion of memory usage,
we can save space by only allocating 96 bytes instead of 128 bytes for
each node.
2019-03-25 13:44:39 -07:00
5e03e178de
Merge pull request #1345 from ajbeamon/support-multiple-client-or-worker-issues
...
Add support for a client or worker having multiple issues.
2019-03-24 17:27:50 -07:00
0ad351751a
fix build failure on Windows from undefined constant and mising parenthesis
2019-03-24 14:03:24 -04:00
24c92a1870
Merge pull request #1352 from etschannen/feature-network-address-list
...
Changed NetworkAddressList to at most two addresses for performance
2019-03-24 10:22:38 -07:00
1fc6937802
changed NetworkAddressList to at most two addresses for performance
2019-03-23 17:54:46 -07:00
09f37cf3d2
Merge pull request #533 from ajbeamon/fix-parent-directory
...
Fixes to parentDirectory() and abspath()
2019-03-22 23:53:46 -07:00
eacde47050
Disable setMemoryQuota for ASAN
2019-03-22 18:47:38 -07:00
382a7bdc5f
Changed behavior regarding ~ and ~user paths to treat them as unresolvable symbolic links.
2019-03-22 16:21:12 -07:00
524a666290
Added back previous handling of ~ in paths, with the improvement that it only treats ~ as special if it is the first character.
2019-03-22 11:44:46 -07:00
4eb5715689
Add support for a client or worker having multiple issues.
2019-03-22 08:29:41 -07:00
c6d96498ca
parentDirectory() is now based on popPath(). Bug fix, abspath() would prepend current working directory even when not resolving symlinks. Added more unit tests. Ported path operation unit tests to fdbmonitor() since the path manipulation function implementations are significantly different. Clarified some comments. The flow project version of abspath() does not allow resolveLinks to be false, for now, because behavior of this on Windows is not well thought out or tested.
2019-03-21 16:56:36 -07:00
bec6f0a37a
Added flags to absPath() and parentDirectory() for whether or not symbolic links should be resolved when possible and whether the final resolved path must actually exist on the filesystem. Updated tests, added more cases.
2019-03-20 22:52:47 -07:00
436c9319ff
Merge pull request #1333 from vishesh/scratch
...
fix: Use '_' instead of ':' in IPv6 tracefile names
2019-03-20 17:00:43 -06:00
c37291a366
fix: Use '_' instead of ':' in IPv6 tracefile names
...
':' is not acceptable on Windows. Reason to choose '_' instead of '.'
is to differentiate b/w IPv4 and IPv6 easily, and also '..1' in
filename looks weirder than '__1', which would happen with shortened
IPv6 addresses. And non-shortened IPv6 addreses are just lots of 0s.
2019-03-20 14:00:33 -07:00
f73aad863b
adding missing logging for 8192 fast allocators
2019-03-20 13:48:45 -07:00
70ac5ffda0
added random logging for huge arenas
2019-03-20 11:20:47 -07:00
c33428b705
do not fail atomicReplace if the files does not exist
2019-03-19 13:38:30 -07:00
10a30fed42
Merge pull request #1316 from senthil-ram/atomicreplace
...
atomicReplace to preserve owner & permission info
2019-03-19 08:21:35 -07:00
ddf8e86730
Merge pull request #1275 from alexmiller-apple/tstlog9
...
Spill-By-Reference TLog Part 5: TLogs That Can Spill Large Amounts Of Data
2019-03-18 17:58:21 -07:00
33a211c834
Merge pull request #1315 from etschannen/feature-memory-logging
...
Occasionally log a backtrace when the fast allocators allocates a new magazine
2019-03-18 16:05:20 -07:00
fe18596c01
Add a comment about ReplaceFile API
2019-03-18 15:24:08 -07:00
c8122edb6b
avoid relying on 128<<10 in knobs.cpp
2019-03-18 12:40:15 -07:00
7711542b55
Update flow/FastAlloc.cpp
...
Co-Authored-By: etschannen <36455792+etschannen@users.noreply.github.com>
2019-03-18 12:34:46 -07:00
c0094830c3
addressing minor format related comments
2019-03-18 11:59:51 -07:00
51a655d9c7
atomicReplace to preserve owner & permission info
...
atomicReplace currently does not preserve the
ownership and permission info, this fix will
preserves the ownership and permission info - it
fails the call if it is not able to do so.
2019-03-18 00:18:05 -07:00
fa5d929106
switch to g_nondeterministic_random to make simulation deterministic
2019-03-17 22:48:43 -07:00
d2eb7578fd
Occasionally log a backtrace when the fast allocators allocates a new magazine to help track down memory leaks
2019-03-17 21:59:56 -07:00
9a4fc25de5
ParentDirectory() in Platform,h no longer uses absPath() so it does not resolve symbolic links, making it suitable for use in fdbmonitor. CleanPath() will return "." instead of empty string to mean a relative path that refers to the current directory. Updated tests, and added new tests for ".".
2019-03-17 15:58:04 -07:00
f88e53e640
Merge branch 'master' of https://github.com/apple/foundationdb into fix-parent-directory
2019-03-16 00:13:09 -07:00
98a295a803
Changed parentDirectory() trace events back to SevError outside of simulation, did the same for cleanPath().
2019-03-16 00:07:38 -07:00
2efb6f4c0d
Added cleanPath() which puts a path in a canonical form without .., ., or duplicate separators without using the filesystem or resolving symbolic links. absPath() redefined to use cleanPath() so it will return the same result for a path without symbolic links regardless of whether or not the path actually exists. Redefined parentDirectory() to use absPath() and error on certain inputs. Added comments describing behavior of these functions, and added a unit test which verbosely tests many inputs to them.
2019-03-15 23:54:33 -07:00
471ac206a3
Add new boost preprocessor definitions to windows build in release mode, still just for flow project to see if it's sufficient.
2019-03-15 21:03:03 -07:00
2c8224f0b7
Add new boost preprocessor definitions to windows build.
2019-03-15 21:03:03 -07:00
7f8475cf27
Removed flow/boost.cpp and the copied boost files it was using, added boost macros to makefile and cmake stuff to hopefully enable building without them.
2019-03-15 21:03:03 -07:00
9f713a6b3d
Testing to see if builds work without the contents of the boost source copies.
2019-03-15 21:03:03 -07:00
5407c87179
Updated two files copied from boost 1.52.0 to their 1.67.0 equivalents since the default boost version to compile against is now 1.67.0 and compilation (on MacOS at least) fails against boost 1.52.0 because some definitions have moved since.
2019-03-15 21:03:03 -07:00
7f5bc2981f
Checksum DiskQueue pages on read, but at a lower priority.
...
If a server has its data spilled, then it's behind the 5s window.
Feeding it data is less important than committing, so we can hide the
extra CPU usage from checksumming the read amplified disk queue pages.
2019-03-15 21:01:19 -07:00
809861ee88
Merge branch 'master' into fix-parent-directory
...
# Conflicts:
# flow/Platform.cpp
2019-03-15 10:51:12 -07:00
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
5a10bf5dfc
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-14 10:35:12 -07:00
c32504f705
io: Add DISABLE_POSIX_KERNEL_AIO knob to use EIO instead of Kernel AIO
...
- Some Linux filesystems don't support O_DIRECT which is required by
Kernel AIO to function properly. Instead of using O_SYNC, EIO is
much better options in terms of performance penalty.
- Some systems may not support AIO at all. Eg. Windows Subsystem for
Linux.
FIXES #842
RELATED #274
2019-03-13 13:39:45 -07:00
e068c478b5
merge master
2019-03-12 18:31:25 -07:00
a7e45cff91
Merge pull request #1176 from jzhou77/ratekeeper
...
Make Ratekeeper a separate role
2019-03-12 15:58:59 -07:00
85c24b0067
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-12 15:20:54 -07:00
5392742902
fixed review comments
2019-03-12 14:38:54 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
bbd5cc2e95
Remove duplicate include due to concurrent update to same file
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-03-11 20:42:35 -07:00
bc4820865e
Fix some Windows issues
...
1. We need to set a Windows target to get rid of warnings when compiling
files that include windows.h
2. By default Boost will try to automatically link against boost_system.
In order to prevent this, we define BOOST_ALL_NO_LIB
3. Somehow, I had to include winioctl.h in Platform.cpp. According to
the documentation from MS, this shouldn't be necessary as windows.h
includes this as well. However, for me it didn't compile otherwise.
2019-03-11 20:42:35 -07:00
880e8643d1
Fix Windows link errors
2019-03-11 17:49:03 -07:00
435e515985
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-11 11:17:40 -07:00
ccab3834e2
merge in master
2019-03-10 23:12:37 -07:00
2ff37f49da
fix: compiler error
2019-03-10 22:56:12 -07:00
Jingyu Zhou