a8e408e268
run clang-format on changes
2019-06-10 14:10:24 -07:00
0b07dbd05c
Remove a horrible hack aiming at Ubuntu 14.04's libstdc++_pic.
...
Because our supported build environments no longer include Ubuntu 14.04,
and this is causing problems with libc++-based builds.
2019-06-10 13:16:03 -07:00
6fa7081a21
net: Don't make FailureMonitoring requests from client
...
This patch removes the need for clients to continuously contact
cluster coordinator for failure monitoring information. Instead, it
uses the FlowTransport to monitor the statuses of peers and update
FailureMonitor accordingly.
2019-06-09 00:43:38 -07:00
6b4d30c3ae
failmon: Identify client vs server when starting failure monitoring client
2019-06-09 00:43:12 -07:00
526468becb
Fix printable() being incorrect for bytes >127.
...
The use of `auto` hid that TraceableString<T>::begin(value) was a signed
char, and therefore `*iter` was -1, and -1/16 is 0.
Thus, a 0xFF byte was printed as \x0f. Forcing it to an unsigned type
makes it correctly become \xff.
2019-06-05 14:41:31 -07:00
c027792d0f
Make the default seed of the generator returned by deterministicRandom() random. This means that the random number sequence won't be deterministic unless you set a seed.
2019-06-05 11:48:29 -07:00
773f52d0a1
Merge remote-tracking branch 'upstream/master' into cloexec
2019-06-03 15:43:32 -07:00
a8c9432ced
windows build fix : del sys/wait.h in Platform.cpp
2019-05-31 14:59:53 -07:00
362c2bf1e6
improved the cpu efficiency of printable
2019-05-29 14:55:45 -07:00
02882dbf00
Checkpointing progress, RedwoodRecordRef and DeltaTree tests pass but BTree test does not. RedwoodRecordRef::Delta rewritten to actually do prefix compression on key and integer fields. Added related unit tests and benchmarks. Some improvements to DeltaTree and requirements on its T and Delta types to avoid repeated common prefix discovery.
2019-05-29 06:23:32 -07:00
b17ad85497
exec op not supported when log_anti_quorum > 0
2019-05-28 22:07:46 -07:00
bb474dc323
if recovery < fully_recovered then fail the exec
...
Will do more cleanup, pushing it for a test run in CI
2019-05-28 22:07:46 -07:00
925499954b
New status cluster_not_fully_recovered
2019-05-28 22:07:46 -07:00
d68a229772
makefile changes to accommodate boost/process.hpp
2019-05-28 22:07:46 -07:00
b6e037ffbc
Replace fork with boost::process::child
2019-05-28 22:07:46 -07:00
898bed66c1
Allow only whitelisted binary path for exec op
2019-05-28 22:07:46 -07:00
aa79480d69
changes to make fdbfork asynchronous
2019-05-28 22:07:46 -07:00
4bc4c615da
exec op to all tlog, restore change in test &other
...
- exec operation to go to all the TLogs
- minor bug fix in tlog
- restore implementation for the simulator
- restore snap UID to be stored in restartInfo.ini
- test cases added
- indentation and trace file fixes
2019-05-28 22:07:46 -07:00
72dd067173
Trace message changes and fix few FIXMEs
2019-05-28 22:07:46 -07:00
69edefe68b
Snapshot based backup and resotre implementation
2019-05-28 22:07:46 -07:00
ec80ff1e8d
Remove unneeded redefinition of random functions.
2019-05-24 11:55:25 -07:00
fc204c846a
Make IRandom reference counted. Simplify the logic around deterministicRandom() to require less state and branching.
2019-05-23 18:51:59 -07:00
764ca2971c
Merge branch 'master' into thread-safe-random-number-generation
2019-05-23 18:49:49 -07:00
296adada93
Use c++11 thread_local instead of redefining it to platform specific variants.
2019-05-23 18:35:45 -07:00
f417e60264
Merge branch 'merge-release-6.1-into-master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbserver/QuietDatabase.actor.cpp
2019-05-23 09:52:00 -07:00
d29c7e4c9b
Merge branch 'release-6.1' into merge-release-6.1-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/QuietDatabase.actor.cpp
# versions.target
2019-05-23 09:28:45 -07:00
e5381e0612
Fix some new usages of g_random
2019-05-23 09:23:27 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
ebc96a7e0e
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-05-21 23:49:27 -07:00
d3c1f9afbd
Suppress 'TraceEventFieldNotFound' event. Don't suppress simultaneous FDBLibTLSVerifyFailure events (this requires including flow.h and removing the copied version of Reference/ReferenceCounted in FDBLibTLS).
2019-05-20 15:12:28 -07:00
d660ffb654
Merge pull request #1605 from mpilman/remove-traceevent-benchmark
...
remove memory-hungry benchmark
2019-05-17 17:04:45 -07:00
a8b9d8e34b
Merge pull request #1336 from tclinken/fast-allocate-ptree-nodes
...
Create 96-byte fast allocator for storage queue PTree nodes
2019-05-17 14:22:46 -07:00
a0fa81aa1c
remove memory-hungry benchmark
...
This test was used as a benchmark to prove that the
new `IsTraceable` feature improves performance. However
it will cause smaller machines to OOM and it doesn't
help for correctness testing. Therefore this test should
likely be removed
2019-05-17 14:12:44 -07:00
5a8c97480a
Merge pull request #1506 from nikolas-ioannou/feature-pagecache-lru
...
AsyncFileCached: switch from a random to an LRU cache eviction policy
2019-05-17 13:42:21 -07:00
f598d0a614
Merge pull request #1593 from AlvinMooreSr/variant-missing
...
Switched Variant Header
2019-05-17 10:42:21 -07:00
3acaa7343e
Enabled C++17 for all Windows projects
...
Set Visual Studio version to 2017 (first version to support C++17)
2019-05-16 17:44:13 -07:00
94aed513c7
Switched Windows tools within projects to 2017
2019-05-16 15:05:11 -07:00
17a2983943
Switch boost variant to C++17 variant
2019-05-16 13:05:49 -07:00
6389a12931
Switched variant to use boost variant for source consistency
2019-05-16 10:37:33 -07:00
69fb852ee0
Add more CLOEXEC-like things.
...
From missed call sites found during/after code review.
2019-05-14 20:30:58 -10:00
1f02cd30e2
Mark all opened files as close-on-exec.
2019-05-13 16:05:49 -10:00
19165dc873
Mark all network sockets as close-on-exec.
2019-05-13 16:05:49 -10:00
a9b03ec505
Merge pull request #1090 from mpilman/features/protocol-version-flags
...
Features/protocol version flags
2019-05-13 16:33:06 -07:00
60ce790a2b
Consistent use of #if VALGRIND
2019-05-13 12:44:13 -10:00
436645e956
Update flow/ThreadSafeQueue.h
...
Co-Authored-By: Alex Miller <35046903+alexmiller-apple@users.noreply.github.com>
2019-05-13 12:44:13 -10:00
7d38a1e273
Use std::atomics in thread-safe queue.
2019-05-13 12:44:13 -10:00
b601fb2df1
Added flat_buffers source files to Windows project in order to support building via make
2019-05-13 14:22:36 -07:00
5a13915960
implemented std::set serializer for flatbuffers
2019-05-13 14:15:23 -07:00
46e7a0ca56
address reviews and make compile with `-Wunused-variable`
2019-05-13 14:15:23 -07:00
a92233a371
Added TODO to optimize packet serialization
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-05-13 14:15:23 -07:00
b4fa13c33b
Remove obsolete function
2019-05-13 14:15:23 -07:00
96aaa31a6c
Compiling on clang again
2019-05-13 14:15:23 -07:00
0713e06efc
Started to work on Windows
2019-05-13 14:15:23 -07:00
42385c2f81
Fixed issues introduced during rebase
2019-05-13 14:15:23 -07:00
186bff4ae5
removed unnecessary assert and added test
2019-05-13 14:15:23 -07:00
f5fa3a65b4
some more fixes
2019-05-13 14:15:23 -07:00
44db3450ec
Several flatbuffers bug fixes
2019-05-13 14:15:23 -07:00
69fa3d3903
fixed compilation issues after rebase
2019-05-13 14:15:23 -07:00
43a7703a1e
fix addFlag
2019-05-13 14:15:22 -07:00
f0d81c6820
make EnsureTable work with streaming serializer
2019-05-13 14:15:22 -07:00
7affe73774
Added more low-level tests and implemented enums
2019-05-13 14:15:22 -07:00
0a3676111e
Support vector of unions in flatbuffers
2019-05-13 14:15:22 -07:00
642a96807b
Fixed compilation issues after rebase
2019-05-13 14:15:22 -07:00
b06fe39b47
renamed flatbuffers tests for consistency
2019-05-13 14:15:22 -07:00
1df3d8f0c7
disable rlimit when ASAN is used
2019-05-13 14:15:22 -07:00
6afce01744
Implementation complete (not yet working)
2019-05-13 14:15:22 -07:00
1d5572a078
removed unused knob
2019-05-13 14:15:22 -07:00
9eeb48c43d
Allow to turn on object serializer
...
This commit includes functionality to turn on
the object serializer for network communication.
This is done the following way:
- On incoming connections, a process will detect
whether the client supports the object serializer
and will only serialize responses with it, if it does
- On outgoing connections, the command line flag is used
to determine whether the object serializer should be used
to send data.
This way, a cluster can run in mixed mode. To upgrade one
can upgrade one process at a time and set the flag one process
at a time.
This is how this is tested on the simulator:
- The command line flag can take three options: on, off,
and random.
- For off, the object serializer will never we used.
- For on, the object serializer will be always used.
- For random, the simulator will flip a coin for each
process it starts up.
2019-05-13 14:15:22 -07:00
ba83c458a6
types implemented
2019-05-13 14:15:22 -07:00
fe81454ec2
basic functionality for object serializer
...
This commit includes:
- The flatbuffers implementation
- A draft on how it should be used for network messages
- A serializer that can be used independently
What is missing:
- All root objects will need a file identifier
- Many special classes can not be serialized yet as the
corresponding traits are not yet implemented
- Object serialization can not yet be turned on (this will
need a network option)
2019-05-13 14:15:22 -07:00
8bbd28315e
Added Object serializer (does not yet compile)
2019-05-13 14:15:22 -07:00
8e1927a5cb
Rename trace event field to avoid duplicating 'Time' field.
2019-05-13 13:25:01 -07:00
ab4618c3f9
Don't access TraceEvent::eventCounts outside the network thread. Make net2liveness an atomic counter.
2019-05-10 12:35:11 -10:00
acb2fc67ac
Remove unnecessary redefinition of new random functions. Enable rand logging for the generators returned by deterministicRandom().
2019-05-10 14:16:48 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
22499666d0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/LogRouter.actor.cpp
# flow/Trace.cpp
# versions.target
2019-05-08 18:19:35 -07:00
c2827f4fa3
Add page cache hit, miss, and eviction stats to SystemMonitor
2019-05-08 15:41:17 +02:00
a67d47844b
coveragetool: suppress output unless VERBOSE is set
...
When using the CMake build system with tools such as Ninja (instead of
Make), they have very concise output, and the general "UX flow" involves
only outputting warnings that should be fixed or errors that should stop
the build. Furthermore, when building things in CI systems and looking
at build logs, etc, it's often vastly easier to to hunt down errors when
only relevant stuff is output.
With this change, the output from coveragetool.exe is suppressed, which
accounts for a lot of spurious output that isn't necessary when running
these builds.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-07 16:54:05 -10:00
aefd68e1e7
The memory tracking trace events can crash if the memory that is being allocated is coming from a trace event itself. Specifically TDMetrics within trace events uses fast allocated memory. Trace events are supposed to be short lived, so this commits adds a global count of outstanding trace events, and disables memory tracking when a trace event exists.
2019-05-06 17:41:32 -07:00
7a10199a35
flow: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
41ae9cd0d8
Remove ; for namespace declarations
2019-05-06 13:21:28 -07:00
d6170210e7
AsyncFileCached: throw (new) exception, through helper static fn, if cache eviction polity not recognized.
2019-05-06 10:11:46 +02:00
fdb3992990
AsyncFileCached: switch to string for eviction policy knob
2019-05-03 14:04:43 +02:00
b2280e15bf
AsyncFileCached: support for lru cache eviction policy
...
- Added a knob to control page cache eviction policy (0=random default, 1=lru)
- Added page cache hit, miss, and eviction stats
2019-05-02 17:35:30 +02:00
e193cac5ef
Merge remote-tracking branch 'apple/master' into tlog
...
Resolve Conflicts: fdbserver/MasterProxyServer.actor.cpp
2019-05-01 17:18:00 -07:00
2d5043c665
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-30 18:27:04 -07:00
4fa1c008f9
Highly prioritize storageServerRejoin messages on the proxy, so that storage servers can rejoin the cluster even when a proxy is CPU saturated
2019-04-23 20:56:01 -07:00
e0f76edf77
Merge pull request #1471 from AlvinMooreSr/release-6.1-merge
...
Merge Release 6.1 Into Master
2019-04-23 11:08:21 -07:00
c25fac9421
Lower the priority of spilled peeks to below that of spilling.
...
This prevents peeking from starving the TLog of CPU time to spill, and
thus impacting write throughput at prolonged saturation.
2019-04-22 18:39:21 -07:00
8b67da57bb
Fix upgrade test failure
...
Serialize pseudoLocalities if protocol version is larger than 0x0FDB00B061060001LL.
Note this version may need to be changed to "currentProtocolVersion" when merging
into the master, and "currentProtocolVersion" should be incremented.
2019-04-21 10:41:07 -07:00
e7ee61660c
Fix unit test failure due to trace event names
...
Capitalize the first letter.
2019-04-20 10:19:53 -07:00
2bea99591e
Merge branch 'release-6.1' of copy of master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-04-17 15:51:48 -07:00
6888baf1c3
Fix compile error
2019-04-16 15:44:55 -07:00
08e66d1764
Prefer success to cast-to-void
2019-04-16 15:29:03 -07:00
781b6ece77
Fix OPEN_FOR_IDE -Wunused-variable warnings
...
CC #1255 , #1173
2019-04-16 15:28:01 -07:00
baa3e806ef
Address review comments from #1446
2019-04-16 09:48:15 -07:00
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
6c7fd1593f
Avoid potential index-out-of-bounds
2019-04-15 15:20:09 -07:00
dced9232d0
ASSERT_WE_THINK trailing byte is '\0'
2019-04-15 09:43:26 -07:00
369c1a0230
Exclude trailing \0 byte for string literals
...
This will "do the right thing" as long as no one expects something like
char a[] = {'a'};
TraceEvent("Event").detail("a", a);
to trace `{..., "a": "a"}`.
It will trace `{..., "a": ""}`
2019-04-11 10:39:48 -07:00
1135331ce6
Include space as a printable character
2019-04-11 10:01:06 -07:00
cd5c9d91fa
Merge pull request #1443 from etschannen/master
...
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
8e05713a5d
do not log a SevError trace event if we cannot deserialize the connect packet
2019-04-10 17:41:02 -07:00
a9b1b32fde
Handle negative input to base16Char
...
base16Char was being called with (c < 0), causing a failed assertion.
2019-04-09 18:14:09 -07:00
6220a5ce0f
Merge pull request #1370 from jzhou77/fix-unreferenced
...
Remove unused functions
2019-04-09 11:49:45 -07:00
058d028099
Merge pull request #1301 from mpilman/features/cheaper-traces
...
Defer formatting in traces to make them cheaper
2019-04-09 10:11:04 -07:00
af17babb9a
Restore a deleted comment line
2019-04-09 09:38:45 -07:00
e4746888eb
Removed dead code
2019-04-09 09:08:45 -07:00
21c0ba555c
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-08 18:38:42 -07:00
05869a8383
do not log a degraded reset message if the previous reset was more than a week ago
2019-04-07 23:00:58 -07:00
19d9b8fcd8
don’t log MachineLoadDetails from getSystemStatistics()
2019-04-07 22:55:19 -07:00
d7612a4426
Fix OPEN_FOR_IDE build errors
2019-04-05 16:30:42 -07:00
d01cbf3455
Addressed code review comments
2019-04-05 13:12:20 -07:00
4287b1d2a1
resolved minor merge issues
2019-04-05 13:12:19 -07:00
b0877e7dd0
Don't quote spaces in std::strings
2019-04-05 13:12:19 -07:00
1c6c73af6d
Fixed infinite recursion
2019-04-05 13:12:19 -07:00
c6bc23791d
Fixed format error in microbench
2019-04-05 13:12:19 -07:00
ea67b742c7
Implemented Traceable for printable types
2019-04-05 13:12:19 -07:00
c9ee7f54e7
Add compound type to micro-benchmark
2019-04-05 13:12:19 -07:00
8120a33bda
Fix `>`-direction when initializing TraceEvent
2019-04-05 13:12:19 -07:00
32a3aea8bb
Make strings random in tracing micro-benchmark
2019-04-05 13:12:19 -07:00
bb82f8560a
process all volatile ints correctly in traces
2019-04-05 13:12:19 -07:00
9938a49c1e
make primitive test longer
2019-04-05 13:12:19 -07:00
8af0f15a43
primitive performance test for TraceEvent
2019-04-05 13:12:19 -07:00
ae84f5424a
Fix potential segfault when init changes enabled
2019-04-05 13:12:19 -07:00
c008e16c81
Defer formatting in traces to make them cheaper
...
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:
- TraceEvent::detail now takes a c-string instead of std::string for
literals. This prevents unnecessary allocations if the trace is not
going to be printed in the first place (for example for SevDebug).
Before that `detail` expected a `std::string` as key, which mean that
any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
specialized for any type that needs to be printed. The actual
formatting will be deferred to after the `enabled` check. This
provides two benefits: (1) if a TraceEvent is disabled, we don't pay
for the formatting and (2) TraceEvent can trace types that it doesn't
know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
calls, a call to detail will only introduce a if-branch which is much
cheaper than a function call.
2019-04-05 13:12:19 -07:00
390ab9cfed
A process will mark itself as degraded if it continually disconnects from a different process which the failure monitor thinks is healthy
2019-04-04 14:11:12 -07:00
2b75c2e684
Restore removed functions.
...
crc32c.cpp is 3rd party code.
orYield() in genericactors.actor.h might be used in the future code.
2019-04-04 13:24:55 -07:00
39c595223b
Merge branch 'release-6.1'
2019-04-02 22:30:02 -07:00
985e1de467
Update documentation and protocol version for release 6.1.0
2019-04-02 18:39:24 -07:00
47b4b82628
Merge branch 'master' into fix-unreferenced
2019-04-01 14:07:19 -07:00
3f76be8f45
Merge remote-tracking branch 'apple/master' into fix-unreferenced
2019-04-01 14:00:43 -07:00
f7f8ddd894
Fix warnings on unused variables
...
Found by -Wunused-variable flag.
2019-04-01 14:00:20 -07:00
a46620fbee
Merge branch 'release-6.1'
2019-03-30 17:59:28 -07:00
d670b74d69
prevent trace event spam by combining huge arena samples
2019-03-30 13:36:13 -07:00
d882c060bf
Merge commit '5dd6396eed0de0dfea6cf9eecc307995eff5cedc'
2019-03-28 18:00:55 -07:00
aa368c08a2
changed NetworkAddress hash function to use more bytes from the IP address
2019-03-28 17:47:13 -07:00
80ecb12190
change the IPv6 hash function to be more efficient
2019-03-28 14:07:46 -07:00
b6008558d3
renamed BinaryWriter.toStringRef() to .toValue(), because the function now returns a Standalone<StringRef>()
...
eliminated an unnecessary copy from the proxy commit path
eliminated an unnecessary copy from buffered peek cursor
2019-03-28 11:52:50 -07:00
836bb95a7a
Merge pull request #1372 from etschannen/master
...
Merge 6.1 into master
2019-03-27 21:00:49 -07:00
34b9d5e722
Merge pull request #1364 from etschannen/feature-fast-serialize
...
A few performance optimizations
2019-03-27 20:57:25 -07:00
6997075917
changed back to isV6addr instead of isV4addr for compatibility
2019-03-27 19:55:36 -07:00
2fbecac10f
fix: allocation strategy did not take into account the data used by the ArenaBlock
2019-03-27 19:54:58 -07:00
e5a80f2c94
optimized IPaddress
2019-03-27 18:21:13 -07:00
a55f06e082
Remove unused functions
...
Found with -Wunused-function flag.
2019-03-27 15:45:28 -07:00
d5c8b6b083
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
# flow/flow.h
2019-03-27 13:37:15 -07:00
71e2fdafb8
Changes to ratekeeper camel case
2019-03-27 08:24:25 -07:00
9e8237955f
BinaryWriter uses arena allocation
2019-03-26 17:27:33 -07:00
38c6681349
Fix some signed and unsigned mismatch warnings.
2019-03-26 14:54:11 -07:00
ea38b78768
BinaryWriter uses fastAllocator
2019-03-26 12:06:39 -07:00
dbcf1d717c
Added calls to TRACEALLOCATOR(96) and DETAILALLOCATORMEMUSAGE(96)
2019-03-25 13:45:07 -07:00
4a3467296f
Removed Arena96
2019-03-25 13:44:53 -07:00
007abbc45b
Added 96-byte FastAllocator
...
Since storage queue nodes account for a large portion of memory usage,
we can save space by only allocating 96 bytes instead of 128 bytes for
each node.
2019-03-25 13:44:39 -07:00
5e03e178de
Merge pull request #1345 from ajbeamon/support-multiple-client-or-worker-issues
...
Add support for a client or worker having multiple issues.
2019-03-24 17:27:50 -07:00
0ad351751a
fix build failure on Windows from undefined constant and mising parenthesis
2019-03-24 14:03:24 -04:00
24c92a1870
Merge pull request #1352 from etschannen/feature-network-address-list
...
Changed NetworkAddressList to at most two addresses for performance
2019-03-24 10:22:38 -07:00
1fc6937802
changed NetworkAddressList to at most two addresses for performance
2019-03-23 17:54:46 -07:00
09f37cf3d2
Merge pull request #533 from ajbeamon/fix-parent-directory
...
Fixes to parentDirectory() and abspath()
2019-03-22 23:53:46 -07:00
eacde47050
Disable setMemoryQuota for ASAN
2019-03-22 18:47:38 -07:00
382a7bdc5f
Changed behavior regarding ~ and ~user paths to treat them as unresolvable symbolic links.
2019-03-22 16:21:12 -07:00
524a666290
Added back previous handling of ~ in paths, with the improvement that it only treats ~ as special if it is the first character.
2019-03-22 11:44:46 -07:00
4eb5715689
Add support for a client or worker having multiple issues.
2019-03-22 08:29:41 -07:00
c6d96498ca
parentDirectory() is now based on popPath(). Bug fix, abspath() would prepend current working directory even when not resolving symlinks. Added more unit tests. Ported path operation unit tests to fdbmonitor() since the path manipulation function implementations are significantly different. Clarified some comments. The flow project version of abspath() does not allow resolveLinks to be false, for now, because behavior of this on Windows is not well thought out or tested.
2019-03-21 16:56:36 -07:00
bec6f0a37a
Added flags to absPath() and parentDirectory() for whether or not symbolic links should be resolved when possible and whether the final resolved path must actually exist on the filesystem. Updated tests, added more cases.
2019-03-20 22:52:47 -07:00
436c9319ff
Merge pull request #1333 from vishesh/scratch
...
fix: Use '_' instead of ':' in IPv6 tracefile names
2019-03-20 17:00:43 -06:00
c37291a366
fix: Use '_' instead of ':' in IPv6 tracefile names
...
':' is not acceptable on Windows. Reason to choose '_' instead of '.'
is to differentiate b/w IPv4 and IPv6 easily, and also '..1' in
filename looks weirder than '__1', which would happen with shortened
IPv6 addresses. And non-shortened IPv6 addreses are just lots of 0s.
2019-03-20 14:00:33 -07:00
f73aad863b
adding missing logging for 8192 fast allocators
2019-03-20 13:48:45 -07:00
70ac5ffda0
added random logging for huge arenas
2019-03-20 11:20:47 -07:00
c33428b705
do not fail atomicReplace if the files does not exist
2019-03-19 13:38:30 -07:00
10a30fed42
Merge pull request #1316 from senthil-ram/atomicreplace
...
atomicReplace to preserve owner & permission info
2019-03-19 08:21:35 -07:00
ddf8e86730
Merge pull request #1275 from alexmiller-apple/tstlog9
...
Spill-By-Reference TLog Part 5: TLogs That Can Spill Large Amounts Of Data
2019-03-18 17:58:21 -07:00
33a211c834
Merge pull request #1315 from etschannen/feature-memory-logging
...
Occasionally log a backtrace when the fast allocators allocates a new magazine
2019-03-18 16:05:20 -07:00
fe18596c01
Add a comment about ReplaceFile API
2019-03-18 15:24:08 -07:00
c8122edb6b
avoid relying on 128<<10 in knobs.cpp
2019-03-18 12:40:15 -07:00
7711542b55
Update flow/FastAlloc.cpp
...
Co-Authored-By: etschannen <36455792+etschannen@users.noreply.github.com>
2019-03-18 12:34:46 -07:00
c0094830c3
addressing minor format related comments
2019-03-18 11:59:51 -07:00
51a655d9c7
atomicReplace to preserve owner & permission info
...
atomicReplace currently does not preserve the
ownership and permission info, this fix will
preserves the ownership and permission info - it
fails the call if it is not able to do so.
2019-03-18 00:18:05 -07:00
fa5d929106
switch to g_nondeterministic_random to make simulation deterministic
2019-03-17 22:48:43 -07:00
d2eb7578fd
Occasionally log a backtrace when the fast allocators allocates a new magazine to help track down memory leaks
2019-03-17 21:59:56 -07:00
9a4fc25de5
ParentDirectory() in Platform,h no longer uses absPath() so it does not resolve symbolic links, making it suitable for use in fdbmonitor. CleanPath() will return "." instead of empty string to mean a relative path that refers to the current directory. Updated tests, and added new tests for ".".
2019-03-17 15:58:04 -07:00
f88e53e640
Merge branch 'master' of https://github.com/apple/foundationdb into fix-parent-directory
2019-03-16 00:13:09 -07:00
98a295a803
Changed parentDirectory() trace events back to SevError outside of simulation, did the same for cleanPath().
2019-03-16 00:07:38 -07:00
2efb6f4c0d
Added cleanPath() which puts a path in a canonical form without .., ., or duplicate separators without using the filesystem or resolving symbolic links. absPath() redefined to use cleanPath() so it will return the same result for a path without symbolic links regardless of whether or not the path actually exists. Redefined parentDirectory() to use absPath() and error on certain inputs. Added comments describing behavior of these functions, and added a unit test which verbosely tests many inputs to them.
2019-03-15 23:54:33 -07:00
471ac206a3
Add new boost preprocessor definitions to windows build in release mode, still just for flow project to see if it's sufficient.
2019-03-15 21:03:03 -07:00
2c8224f0b7
Add new boost preprocessor definitions to windows build.
2019-03-15 21:03:03 -07:00
7f8475cf27
Removed flow/boost.cpp and the copied boost files it was using, added boost macros to makefile and cmake stuff to hopefully enable building without them.
2019-03-15 21:03:03 -07:00
9f713a6b3d
Testing to see if builds work without the contents of the boost source copies.
2019-03-15 21:03:03 -07:00
5407c87179
Updated two files copied from boost 1.52.0 to their 1.67.0 equivalents since the default boost version to compile against is now 1.67.0 and compilation (on MacOS at least) fails against boost 1.52.0 because some definitions have moved since.
2019-03-15 21:03:03 -07:00
7f5bc2981f
Checksum DiskQueue pages on read, but at a lower priority.
...
If a server has its data spilled, then it's behind the 5s window.
Feeding it data is less important than committing, so we can hide the
extra CPU usage from checksumming the read amplified disk queue pages.
2019-03-15 21:01:19 -07:00
809861ee88
Merge branch 'master' into fix-parent-directory
...
# Conflicts:
# flow/Platform.cpp
2019-03-15 10:51:12 -07:00
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
5a10bf5dfc
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-14 10:35:12 -07:00
c32504f705
io: Add DISABLE_POSIX_KERNEL_AIO knob to use EIO instead of Kernel AIO
...
- Some Linux filesystems don't support O_DIRECT which is required by
Kernel AIO to function properly. Instead of using O_SYNC, EIO is
much better options in terms of performance penalty.
- Some systems may not support AIO at all. Eg. Windows Subsystem for
Linux.
FIXES #842
RELATED #274
2019-03-13 13:39:45 -07:00
e068c478b5
merge master
2019-03-12 18:31:25 -07:00
a7e45cff91
Merge pull request #1176 from jzhou77/ratekeeper
...
Make Ratekeeper a separate role
2019-03-12 15:58:59 -07:00
85c24b0067
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-12 15:20:54 -07:00
5392742902
fixed review comments
2019-03-12 14:38:54 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
bbd5cc2e95
Remove duplicate include due to concurrent update to same file
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-03-11 20:42:35 -07:00
bc4820865e
Fix some Windows issues
...
1. We need to set a Windows target to get rid of warnings when compiling
files that include windows.h
2. By default Boost will try to automatically link against boost_system.
In order to prevent this, we define BOOST_ALL_NO_LIB
3. Somehow, I had to include winioctl.h in Platform.cpp. According to
the documentation from MS, this shouldn't be necessary as windows.h
includes this as well. However, for me it didn't compile otherwise.
2019-03-11 20:42:35 -07:00
880e8643d1
Fix Windows link errors
2019-03-11 17:49:03 -07:00
435e515985
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-11 11:17:40 -07:00
ccab3834e2
merge in master
2019-03-10 23:12:37 -07:00
2ff37f49da
fix: compiler error
2019-03-10 22:56:12 -07:00
c6e94293bf
reset a process to not be degraded after 2 days
2019-03-10 22:39:21 -07:00
c304397166
fixed a memory leak when allocating memory > 4K and < 8K
2019-03-10 21:13:00 -07:00
892b649e64
fixed a memory leak when allocating memory > 4K and < 8K
2019-03-10 21:09:47 -07:00
59f598ab51
Try adding a header?
2019-03-08 21:46:32 -08:00
f593562743
use 8KB pages in Arena
2019-03-08 15:03:17 -08:00
426da90493
Add 8192 bit fast allocator
...
StorageServer is getting close to 4KB on older gccs. However,
on some systems (for example Fedora) the storageServer actor
is already larger than 4KB. This results in a linker error.
2019-03-08 15:03:17 -08:00
4891d31cc0
Remove a broken ASSERT.
...
It's now totally valid to have:
permits=3
take(1)
take(2)
take(72)
and the take(72) will only be granted once the first two finish.
2019-03-07 18:31:18 -08:00
3c86643822
Separate Ratekeeper from data distribution.
...
Add a new role for ratekeeper.
Remove StorageServerChanges from data distribution.
Ratekeeper monitors storage servers, which borrows the idea from
DataDistribution.
2019-03-07 13:16:20 -08:00
04880e3d4d
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-06 13:41:16 -08:00
fd34626009
boost: Remove log from Makefile and version check in flow.h
2019-03-05 22:06:12 -08:00
c6a65389ae
Remove noexcept macro and replace with BOOST_NOEXCEPT.
...
BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a
way that is correctly maintained over time.
2019-03-05 22:06:12 -08:00
367bcff700
Error if boost != 1.67.0
2019-03-05 22:06:12 -08:00
af617d68e6
boost 1.52.0 -> 1.67.0 in all vcxproj files
2019-03-05 22:06:12 -08:00
87d0c5bae0
Bug/usability fix: The output URLs of fdbbackup list were meant to be directly usable for backup management operations but they were missing the bucket URL parameter.
2019-03-05 21:14:21 -08:00
82d957e0bb
Merge pull request #1178 from vishesh/task/issue-963-IPv6
...
IPv6 Support
2019-03-05 17:14:16 -08:00
96ee95b9ad
fix: macOS build #963
...
Use the boost representation of IPv6 address internally
and make sure it uses std::array.
2019-03-05 14:03:14 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
e93cd0ff21
Add some checks and comments to IPv6 changes #963
2019-03-04 14:12:45 -08:00
592e224155
net: add/use formatIpPort to format IP:PORT pairs #963
2019-03-04 14:12:45 -08:00
82b2da4b78
net: Add IPAddress::parse() util #963
2019-03-04 14:12:45 -08:00
25daabdc02
net: TraceEvent and toIPVectorString for new IPAddress structure #963
2019-03-04 14:12:45 -08:00
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
1a550712cb
Add serialization support for std::array
2019-03-04 13:33:10 -08:00
75b2546e35
Allow FlowLock::take to succeed if there's no other active requests.
2019-03-04 01:42:38 -08:00
98bb58628f
Upgrade FlowLock from int to int64_t.
2019-03-04 01:42:38 -08:00
a462359af1
Implement g_random->randomSkewedUInt32()
...
This generates an exponential distribution, implemented as being
a "log-uniform" distribution.
2019-03-04 01:42:38 -08:00
39f612d132
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-03-02 17:07:00 -08:00
94385447bc
Status: Get if client configured TLS
...
To understand if all clients have configured TLS,
we check the tlsoption when a client tries to open database.
This is similar to how we track the versions of multi-version clients.
2019-03-01 15:17:01 -08:00
655c9d82c7
Various cleanup from review
2019-03-01 14:06:47 -08:00
13f7f5a073
Bump protocol version
2019-03-01 13:01:10 -08:00
e2bcecb08f
Merge branch 'master' into ratekeeper-batch-priority-limits
2019-02-28 12:52:44 -08:00
eb629d87a5
Add information about batch ratekeeper to status. Make it possible to track latencies in the ReadWrite workload for concurrently run instances separately.
2019-02-28 09:53:16 -08:00
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
82e429f417
Upgrade BinaryReader to support calls ArenaReader has.
2019-02-26 18:00:55 -08:00
abfe057805
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-25 13:47:16 -08:00
78aad255b0
updateProcessStats gets process stats directly
...
Storage servers no longer parse ProcessMetrics trace lines to get their
own cpuUsage and diskUsage statistics.
2019-02-25 13:45:53 -08:00
b8910ba7cd
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.h
# fdbserver/DataDistribution.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-22 14:38:13 -08:00
d008de576e
Merge pull request #1139 from xumengpanda/mengxu/machine-team-upgrade-PR
...
Add background actor to remove redundant teams
2019-02-22 14:22:07 -08:00
6660ccd936
Large rewrite/refactor from the page format outward. Replaced the in-page format, PrefixTree, with more general and templated implementation of the same concept, DeltaTree, which will be incrementally modifiable soon. New implementation for in-page cursors to use a per-page arena and share decoded binary tree nodes between cursors in the same page. New implementation for internal record B+Tree cursor which uses reference counted path hops which makes cloning a cursor very cheap. New implementation of external facing B+Tree cursor which makes efficient use of two internal cursors to find user-visible KV pairs and return them without copying keys and values except in the case of a large sharded value.
2019-02-21 02:46:30 -08:00
171d119d95
another attempt to fix compiler errors
2019-02-20 18:35:08 -08:00
fa96b8dd33
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-20 16:56:16 -08:00
4be90c91d6
Moved THIS and THIS_ADDR to actorcompiler.h
2019-02-19 15:16:59 -08:00
3bd9b9047b
Minor fixes - flow now compiling with intellisense
2019-02-19 15:16:59 -08:00
2c16475906
Added a general copy-constructor for intellisense
...
If a actor returns something of a different type that is
assignable, it will work because the actor compiler will
compile this to an assignment. However, if you compile
without the actor compiler, this will fail.
2019-02-19 15:16:59 -08:00
eb8285f14f
Add mock description for actors for intellisense
2019-02-19 15:16:59 -08:00
9f8c060343
Put assertions in its own scope
...
Some if-else statements in the code don't use
curly-braces, which is problematic if the only
statement used is an assertion. This should fix
that.
2019-02-19 15:16:59 -08:00
801e4c4700
Make when-macros work by accepting varargs
2019-02-19 15:16:59 -08:00
e05b53d755
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-15 20:37:07 -08:00
6d09ac483c
Merge with master
2019-02-15 17:03:40 -08:00
83060c6e56
Merge pull request #1062 from jzhou77/PR
...
Add a new DataDistributor role.
2019-02-15 13:51:27 -08:00
8099a0b665
Merge pull request #1129 from alexmiller-apple/tstlog-1
...
Spill-by-reference TLog Part 1: IDiskQueue::read()
2019-02-15 11:48:14 -08:00
75f692b931
simplify actorcompiler and target to compile coveragetool
2019-02-15 00:01:42 -08:00
5e6577cc82
Final cleanup per review comments
...
Make distributor interface optional in ServerDBInfo and many other small
changes.
2019-02-14 16:37:17 -08:00
578473a974
Various review comments fixes
2019-02-14 16:37:16 -08:00
907446d0ce
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-14 11:37:38 -08:00
b435d51061
Merge branch 'master' into track-server-request-latencies
2019-02-14 08:07:32 -08:00
5481851e82
TeamCollection: Add knobs for team remover
...
Added three knobs to control team remover
bool TR_FLAG_DISABLE_TEAM_REMOVER:
Disable the teamRemover actor
double TR_REMOVE_MACHINE_TEAM_DELAY:
Wait for the specified time before try to remove next machine team
double TR_WAIT_FOR_ALL_MACHINES_HEALTHY_DELAY:
Wait before checking if all machines are healthy
2019-02-13 15:11:56 -08:00
9272a41e5f
Merge pull request #1146 from atn34/fix-actor-warning
...
Fix actor warning for cmake build
2019-02-13 11:01:37 -08:00
1ea58c1e5e
Actually change name of flag
2019-02-13 10:44:58 -08:00
3a38bff8ee
Use DISABLE_ACTOR_WITHOUT_WAIT_WARNING consistently
2019-02-13 10:30:35 -08:00
2570b37e6e
Add function to read pages from RawDiskQueue_TwoFiles
2019-02-12 23:43:10 -08:00
601b229c05
Add a no-op breakpoint_me for easy gdb'ing.
2019-02-12 23:43:10 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
f257cca1bc
Update `Void _ = wait(...)` occurrences
2019-02-12 17:30:30 -08:00
df3454114e
Add TODO
2019-02-12 11:03:08 -08:00
8b0e593f83
Add --disable-actor-without-wait-error flag to actorcompiler
2019-02-12 11:01:17 -08:00
d348846a10
Introduce ErrorMessagePolicy class
...
This encapsulates logic for how to report particular error messages.
So far only used for reporting an actor not containing a wait statement,
as this warning should be suppressed in certain contexts.
We could change other error messages to use this class, but since they
don't have any logic I don't think the extra indirection is appropriate.
2019-02-12 11:01:17 -08:00
6051c8027b
Allow osx tooling to understand c sharp
...
After this change I am able to load foundationdb.sln with vscode and get
features like jump to definition and autocomplete.
IntermediateOutputPath will now use a default value.
2019-02-12 11:00:24 -08:00
b717750c4f
Minor README changes.
2019-02-11 21:40:14 -08:00
685242fbfc
Merge pull request #1092 from atn34/trace-format-network-option
...
Add trace_format network option
2019-02-08 10:52:50 -08:00
06e216f26e
fixed linker error
2019-02-07 15:37:04 -08:00
e45295a1f5
Added support for TLS
2019-02-07 15:37:04 -08:00
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
7a858b902d
Most of FDB compiling on Windows with cmake
2019-02-07 15:37:04 -08:00
d4349293b9
Reworked the way latency counters are tracked. Report the latency bands in separate events from StorageMetrics and ProxyMetrics. Fix a problem when the latency band configuration was changed. Add correctness testing.
2019-02-07 13:39:22 -08:00
0750dc0418
Change store from (Future, T&) to (T&, Future).
...
LHS = RHS, and the name of what's being modified is easier to find.
2019-02-04 18:04:22 -08:00
fa6f43b1f3
Added missing 'static' for actor fwd declarations
...
There is a bug in the code that when an ACTOR forward
declaration is static, the static keyword doesn't get
generated in the forward declaration. This commit fixes
that.
2019-02-01 14:44:47 -08:00
e9ddd94e27
The failure monitor is given a list of all IP addresses associated with a process
...
The connect packet includes the correct remote address
Did a lot of code cleanup
Simulation test mixed TLS and non-TLS listeners on the same process
2019-01-31 18:20:14 -08:00
9087f2948e
Support serialization of std::map objects
2019-01-31 11:24:04 -08:00
0e5db1e7b7
Merge pull request #1096 from mpilman/features/actor-forward-declarations
...
Allow forward declarations to be actors
2019-01-30 13:30:49 -08:00
e055fdab03
Better diagnostics for unrecognized trace format
2019-01-29 13:08:47 -08:00
01730cc461
Allow forward declarations to be actors
...
If an actor is forward declared it is kind of
awkward as the signatures don't fit. For example:
Future<Void> foo(int const& bar);
is the correct forward declaration for this actor:
ACTOR Future<Void> foo(int bar);
This also means that a lot of tooling (like rtags/cquery etc)
doesn't work correctly. It will show errors that the function
calls are ambiguous and "find references"-features typically
don't work.
Allowing to do this in the actor compiler is the first step. The
next will be to refactor all forward declarations.
2019-01-29 10:57:54 -08:00
dc2b740415
Added server_overloaded error and client message
2019-01-25 13:15:19 -08:00
1d7fec3074
Merge commit '048bfc5c368063d9e009513078dab88be0cbd5b0' into task/tls-upgrade-2
...
# Conflicts:
# .gitignore
2019-01-24 17:43:06 -08:00
ec32d3308b
Merge pull request #1086 from mpilman/features/c++-compiler-errors
...
Fixed several minor code issues
2019-01-24 15:24:33 -08:00
79637f07ac
Fixed several minor code issues
...
These will become a problem as soon as we
switch to C++17
2019-01-24 14:43:12 -08:00
4925b61d0e
Bump protocol version.
2019-01-24 13:49:22 -08:00
7fde75c439
Avoid possibly concurrent reassignment of dl_iterate_phdr.
2019-01-24 13:28:45 -08:00
efe06d71ce
Change profilingEnabled to thread_local to eliminate a race.
2019-01-24 13:27:16 -08:00
2198d24ce1
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/ServerDBInfo.h
# fdbserver/Status.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/storageserver.actor.cpp
2019-01-24 11:43:26 -08:00
8e05e95045
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-18 16:18:34 -08:00
7498c2308c
Merge branch 'release-6.0' into track-server-request-latencies
2019-01-16 13:39:01 -08:00
8b3cfa469c
Fix spelling
2019-01-15 19:30:09 +00:00
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
6276d6b4f4
Merge pull request #1026 from etschannen/master
...
Merge 6.0 into master
2019-01-09 16:23:35 -08:00
684a22a52b
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BackupContainer.actor.cpp
# fdbclient/HTTP.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/BackupCorrectness.actor.cpp
# versions.target
2019-01-09 16:14:46 -08:00
Vishesh Yadav