Commit Graph

976 Commits

Author SHA1 Message Date
Jingyu Zhou 3a0514851a Fix a signed and unsigned mismatch 2019-07-31 17:01:23 -07:00
Jingyu Zhou 7592e129d2 Use friend struct for _IncludeVersion 2019-07-31 16:36:08 -07:00
Jingyu Zhou 7775ab7892 Use std::invoke_result_t for type names 2019-07-31 15:45:35 -07:00
Jingyu Zhou c5cc8c5180 Change deprecated std::result_of to std::invoke_result
std::result_of is deprecated in C++17.
2019-07-31 14:18:41 -07:00
Balachandar Namasivayam e8a9931dbe
Merge pull request #1918 from atn34/at-what-cost
Avoid memcpy for small types
2019-07-31 11:39:38 -07:00
Evan Tschannen fcc0f7a681
Merge pull request #1934 from etschannen/feature-bool-knobs
Added a knob to always use CAUSAL_READ_RISKY for GRV
2019-07-30 18:38:20 -07:00
mpilman 058f21b0fc fixed merge conflicts 2019-07-30 18:36:10 -07:00
mpilman aaef83e122 Added first parser functionality for sequences 2019-07-30 18:36:10 -07:00
mpilman 6b8b666ef6 Write object address in probes 2019-07-30 18:36:10 -07:00
mpilman 75e78f106e Added more run_loop probes and added documentation 2019-07-30 18:36:10 -07:00
mpilman 884628bc0a moved actor probes to function 2019-07-30 18:36:10 -07:00
mpilman b7df7f3549 compatability with older cmake versions 2019-07-30 18:36:10 -07:00
mpilman 32d141ad3a Probes with strings 2019-07-30 18:36:10 -07:00
mpilman 9b96d8c166 a step further to uniqueness 2019-07-30 18:36:10 -07:00
mpilman 8c73fa556c Reduced number of macros to 1 2019-07-30 18:36:10 -07:00
mpilman 497c0aa456 fixed typo 2019-07-30 18:36:10 -07:00
mpilman 8eb06f7ab4 Only generate probes where it is supported 2019-07-30 18:36:10 -07:00
mpilman 13e101c441 Added d-trace probes for actors 2019-07-30 18:36:10 -07:00
mpilman b81a4ef6a2 clang+libc++ support on linux + dtrace probes 2019-07-30 18:36:10 -07:00
Evan Tschannen 2d136af2bd bool knobs can now be set with the words “true” or “false” instead of just a number 2019-07-30 18:21:24 -07:00
Evan Tschannen ea1858953b
Merge branch 'master' into merge-release-6.1-into-master 2019-07-30 17:40:15 -07:00
Andrew Noyes d8a11a704e Add gotcha about arena ordering 2019-07-30 17:33:21 -07:00
Andrew Noyes 43e86ff819 Address review comments 2019-07-30 17:33:21 -07:00
Andrew Noyes 8145b45ad5 Update documentation 2019-07-30 17:33:21 -07:00
A.J. Beamon b5d2234a13 Merge branch 'release-6.1' into merge-release-6.1-into-master
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbserver/MoveKeys.actor.cpp
#	flow/FastAlloc.h
#	versions.target
2019-07-30 16:23:42 -07:00
Andrew Noyes d0ecdb4fb1 Replace std::map with sorted std::vector 2019-07-30 16:17:14 -07:00
Evan Tschannen 06fc8cb904
Merge pull request #1919 from etschannen/feature-buffered-popped
Implement popped on bufferedCursor
2019-07-30 15:56:41 -07:00
A.J. Beamon 14648e20f9
Merge pull request #1901 from ajbeamon/data-distribution-receives-bytes-input-rate
Send bytes input rate to data distribution
2019-07-30 15:01:36 -07:00
Evan Tschannen 7a932479dd throw away state if we ever read popped data from the disk queue adapter 2019-07-30 10:14:39 -07:00
Andrew Noyes 9f32edf4df Avoid memcpy for small types
This is undefined behavior, since it's potentially a misaligned access.
But it's _probably_ not worse than the status quo
2019-07-29 17:11:45 -07:00
Andrew Noyes 997da6882a Explain test subtlety 2019-07-29 16:00:14 -07:00
Andrew Noyes 9d32cbcf50 Deserialize Arena after VectorRef in flat_buffers unit test 2019-07-29 15:45:54 -07:00
Evan Tschannen 1a4ca05a04
Merge pull request #1889 from ajbeamon/add-cache-memory-parameter
Add cache_memory parameter to fdbserver
2019-07-26 13:34:24 -07:00
Evan Tschannen 98c3b24036
Merge pull request #1869 from alexmiller-apple/sharded-txs-performance
Raise the priority of TLogRejoin above TLogPeekReply
2019-07-26 13:30:13 -07:00
Andrew Noyes fc7f9e7c69 Fix Void flatbuffers serialization
Before this fix, the encoding of Void happens to 'work' in that this
implementation can both read and write it, but it is not a flatbuffers
message since it contains a invalid offset (the implementation 'works'
because it does not dereference that offset).
2019-07-25 17:30:43 -07:00
A.J. Beamon b91795d288 Send bytes input rate to DD. 2019-07-25 16:27:32 -07:00
A.J. Beamon a92b6cd3d1 Merge branch 'master' into add-cache-memory-parameter
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
2019-07-25 13:41:57 -07:00
sramamoorthy a65c9f92ed get rid of all timeouts and other changes 2019-07-24 15:36:28 -07:00
sramamoorthy a2f2ad96ff code review comments and merge to master changes 2019-07-24 15:36:28 -07:00
sramamoorthy 806a0822f1 timeoutError that takes Future<Error<T>> as input 2019-07-24 15:36:28 -07:00
A.J. Beamon 94be9560ea Add cache_memory parameter to fdbserver to control the size of the (4K) page cache. Change the default slighty from 2000 MiB to 2GiB. 2019-07-23 15:05:21 -07:00
A.J. Beamon 6e078a41a7
Merge pull request #1828 from mpilman/features/lexicographical-ordered-traces
Make trace files lexicographically ordered
2019-07-23 08:31:59 -07:00
A.J. Beamon e98cee016d Fix unsafe usage of now() function from multiple threads in trace logging. 2019-07-22 22:31:38 -07:00
Andrew Noyes 5cef65b6c4
Update flow/FileTraceLogWriter.cpp
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-07-22 16:38:18 -07:00
mpilman de4fe4d640 Remove dead code and revert back to base 10
there can be at most 2^32 files with a distinct index. Therefore
`log10(index) < 10` - this means base 10 is good enough.
2019-07-22 16:20:03 -07:00
mpilman aa93f8411a Merge branch 'features/lexicographical-ordered-traces' of github.com:mpilman/foundationdb into features/lexicographical-ordered-traces 2019-07-22 14:22:51 -07:00
mpilman 001bfaf80c made trace index unsigned 2019-07-22 14:20:09 -07:00
Markus Pilman a48bdc0095
Update flow/FileTraceLogWriter.cpp
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-07-22 14:16:34 -07:00
A.J. Beamon e29a6ea280
Merge pull request #1871 from bnamasivayam/tr-priority-add-client-log
Track the priority of sampled Transaction as part of GetReadVersion e…
2019-07-22 13:04:52 -07:00
mpilman c26c68d6d2 Address review comments 2019-07-22 11:45:05 -07:00
mpilman a66dc937fc Make trace files lexicographically ordered
This resolves #1825

The basic idea is that we prefix every sequence number with its with,
separated by a dot. That way the filename length isn't increased
drastically (unlike using a fixed whidth number with 0-padding)
and the file names will be lexicographically ordered.
2019-07-22 10:10:47 -07:00
Balachandar Namasivayam af267ba053 Track the priority of sampled Transaction as part of GetReadVersion event. 2019-07-19 17:31:49 -07:00
Alex Miller df7f0cffa1 Raise the priority of TLogRejoin above the default work priority.
With sharded txs tags, the master now receives data from transaction
logs at an order of magnitude higher rate.  This is the intentional
desires result of sharding the txs tag.  With a sufficient number of
TLogs, the master will saturate its CPU time handling the peek
responses.

Performance tests revealed some unstable oddities in how long a recovery
would take, which was eventually root caused to a priority inversion
between TLogRejoin requests and TLog peek replies.

Once peek replies saturate the CPU, the master would proceed to ignore
further TLogRejoin messages.  TLogRejoin is what marks a TLog as
available to the failure monitor, which is also what decides between a
ServerPeekCursor and a MergePeekCursor for a SetPeekCursor.  Ignoring
TLogRejoins meant that the sharded txs locality tags for those servers
would be merge peeked over all TLogs.  This is much less efficient than
just peeking one copy of data from the one preferred server.

Depending on the race between TLogPeek replies saturating the CPU and
TLogRejoin requests being submitted, a variable number of tags would be
affected, and thus the performance test would have some variance in its
results.
2019-07-19 16:55:04 -07:00
Evan Tschannen 3045826e3c
Merge pull request #1819 from mpilman/flatbuffers-fixes2
Flatbuffers fixes2
2019-07-19 16:33:50 -07:00
Alex Miller c3a8ae4752
Merge pull request #1791 from fzhjon/fetch-keys-requests-priority
Introduce priority to fetchKeys requests from data distribution
2019-07-19 14:54:51 -07:00
Jingyu Zhou 8e9aaa767e
Merge pull request #1860 from alexmiller-apple/comma-operator
Remove `operator , (vector<T>, T)` as an append operator.
2019-07-19 12:53:32 -07:00
Alex Miller 9f0a9da7c3 Remove `operator , (vector<T>, T)` as an append operator.
Because no one expects the comma overload.
2019-07-18 21:16:08 -07:00
mpilman 1ac2d01b03 Merge remote-tracking branch 'upstream/master' into flatbuffers-fixes2 2019-07-18 09:50:08 -07:00
A.J. Beamon d981de18e4 Restrict huge arena sampling to the network thread. Revert removal of thread_local definitions. 2019-07-17 16:23:17 -07:00
Alvin Moore b9e938972c
Merge pull request #1719 from mpilman/features/static-libc++
statically link libstdc++ on Linux and remove std::variant
2019-07-16 21:00:02 -07:00
Evan Tschannen 5d3e69b6fc
Merge pull request #1820 from fzhjon/load-balance-locality
Introduced a knob that can turn locality on/off
2019-07-16 16:40:43 -07:00
A.J. Beamon 7d6dcc92d0 I accidentally omitted the important change here. 2019-07-16 15:32:06 -07:00
A.J. Beamon 0bfd474e21 Fix huge arena tracking thread-safety issues. Fixes #1846. 2019-07-16 15:22:19 -07:00
mpilman 6a4a129cf5 fixed silly boost visitor bug 2019-07-16 15:10:55 -07:00
mpilman b18666d942 statically link libstdc++ on Linux and remove std::variant
this will hopefully fix #1610
2019-07-16 14:53:16 -07:00
mpilman d5caf0c1b4 Merge branch 'flatbuffers-fixes2' of github.com:mpilman/foundationdb into flatbuffers-fixes2 2019-07-16 14:47:40 -07:00
Andrew Noyes e7e48a40ce Fix a few IncludeVersion bugs 2019-07-16 13:28:29 -07:00
Jon Fu 7d37040725 split the single knob into two for finer-grained control 2019-07-16 12:46:02 -07:00
Alec Grieser a89d09f1a6
Merge pull request #1844 from ajbeamon/configurable-trace-field-length
Make trace event field lengths default knobified and individually configurable.
2019-07-15 15:53:22 -07:00
mpilman 54416f46fd Pass type as param to VectorRef instead of bool 2019-07-15 15:08:49 -07:00
A.J. Beamon 562e41d25d
Apply suggestions from code review
Co-Authored-By: Alec Grieser <alloc@apple.com>
2019-07-15 15:02:55 -07:00
mpilman 707b29cbcc Fixed compilation error 2019-07-15 13:33:22 -07:00
Alex Miller c8e94e601a
Merge pull request #1729 from etschannen/feature-fast-txs-recovery
Improve the recovery speed of the txnStateStore
2019-07-15 13:27:41 -07:00
mpilman 6c6a1ca8f4 Expose serialization context too all traits 2019-07-15 12:58:31 -07:00
mpilman 1ab36cafbc Fixed memory bug 2019-07-15 09:53:26 -07:00
mpilman cb7b941452 expose protocolVersion to serialize function
The main idea here is that we now have a context for
serialization and deserialization. This can be used
to access the protocolVersion.

The context still needs to be exposed to the
deserailization traits
2019-07-12 17:16:54 -07:00
A.J. Beamon d5051b08dd Make trace event field lengths (and total event sizes) default knobified and configurable. Add a transaction option to control the field length of transaction debug logging. Make the program start command line field less likely to be truncated. 2019-07-12 16:12:35 -07:00
mpilman 75d4b612cf Make object serializer versioned 2019-07-12 11:53:14 -07:00
Andrew Noyes 984b603c33 Avoid saving each item twice 2019-07-12 11:20:01 -07:00
Andrew Noyes 11c969511c Update TODOs 2019-07-12 11:17:11 -07:00
mpilman 74f992b2cc fixed wrong assertion 2019-07-11 23:03:31 -07:00
mpilman b68f2d925f Serialize range result to string for speed 2019-07-11 23:03:31 -07:00
Andrew Noyes c63ff98535 Add some documentation todos 2019-07-11 23:03:31 -07:00
Andrew Noyes eb87b1fc61 Remove dead code 2019-07-11 23:03:31 -07:00
Andrew Noyes 137936762d Eliminate unnecessary memcpy 2019-07-11 23:03:31 -07:00
Andrew Noyes 70f0726185 Support PacketBuffer's of arbitrary size 2019-07-11 23:03:31 -07:00
Andrew Noyes 70eac949e6 Zero-initialize memory more precisely
Also fix two alignment bugs
2019-07-11 23:03:31 -07:00
Andrew Noyes a34f8d70d6 Re-use writeToOffsets vector 2019-07-11 23:03:31 -07:00
Andrew Noyes f8584d0df8 Implement new dynamic_size_traits 2019-07-11 23:03:31 -07:00
Andrew Noyes 71119975f5 Documentation WIP 2019-07-11 23:03:31 -07:00
Jon Fu f707d186fe added new priority for fetchkeys requests and adjusted ddmetrics workload to run parallel with mako 2019-07-11 09:56:58 -07:00
A.J. Beamon b4dbc6d7fa Change the way cache hits and misses are tracked to avoid counting blind page writes as misses and count the results of partial page writes. Report cache hit rate in status. 2019-07-10 14:43:20 -07:00
Evan Tschannen d8948c8be1 Merge branch 'master' into feature-fast-txs-recovery
# Conflicts:
#	fdbserver/TagPartitionedLogSystem.actor.cpp
2019-07-10 13:59:52 -07:00
A.J. Beamon 38ae352fc5 Fix a merge issue 2019-07-10 09:46:23 -07:00
A.J. Beamon 69d7c4f79c Merge branch 'master' into track-run-loop-busyness
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	flow/Net2.actor.cpp
#	flow/network.h
2019-07-09 18:39:23 -07:00
Meng Xu cce00bb413
Merge pull request #1808 from ajbeamon/improved-transaction-metrics
Improve TransactionMetrics
2019-07-09 16:46:17 -07:00
Jon Fu 19a91765f6 introduced a knob that can turn locality on/off 2019-07-09 16:16:15 -07:00
Vishesh Yadav 983343978e fdbrpc: ConnectionMonitor should close unreferenced after delay
Potentially for cases, where it goes up to 1 immediately.
2019-07-09 14:24:16 -07:00
Vishesh Yadav 1f9c80f633 fdbrpc: Instead of tracking last sent data, track last sent non-ping data
* This will allow client to continue monitoring peer connections while
connection stays open, so that there is no period of "uncertainity"
without previous no-monitoring approach.

* Use multiplier for incoming connection idle timeout

* Update idle connection timeout values and leaked connection timeout in
simulator.
2019-07-09 14:24:16 -07:00
Vishesh Yadav 867986cdea fdbrpc: Reduced connection monitoring from clients
This patch does two changes to connection monitoring:

1. Connection monitoring at client side will check if the connection
has been stayed idle for some time. If connection is unused for a
while, we close the connection. There is some weirdness involved here
as ping messages are by themselves are connection traffic. We get over
this by making it two-phase process, first being checking idle
reliable traffic, followed by disabling pings and then checking for
idle unreliable traffic.

2. Connection monitoring of clients from server will no longer send
pings to clients. Instead, it keep monitor the received bytes and
close after certain period of inactivity.
2019-07-09 14:24:16 -07:00
Vishesh Yadav 705059dea1 Trace: Add support to print pointers 2019-07-09 14:24:16 -07:00
A.J. Beamon a5a6f8431c Add a random UID to TransactionMetrics in case a client opens multiple connections and also a field to indicate whether the connection is internal. Convert some of the metrics to our Counter object instead of running totals. 2019-07-08 14:01:04 -07:00
Evan Tschannen 15e894c724 Merge in master 2019-07-05 15:49:24 -07:00
Andrew Noyes 6d74af93d3 Use true instead of 1 2019-07-05 14:07:02 -07:00
Andrew Noyes 15c6f2b864 Explain SFINAE for has_serialization_done 2019-07-05 14:07:02 -07:00
Andrew Noyes e2ed56fa56 Convert ownedPtr to unownedPtr for IReplicationPolicy
Remove WriteRawMemory feature

Remove deserialization_done
2019-07-05 14:07:02 -07:00
Andrew Noyes 9894d928a1 Re-use identical vtables 2019-07-05 14:07:02 -07:00
Andrew Noyes 4c5ebd7609 Avoid assert when collecting vtables 2019-07-05 14:07:02 -07:00
Alex Miller 888f4f92e0 Fix errors and TaskPriority more priorities. 2019-07-03 21:03:58 -07:00
Alex Miller ea6898144d Merge remote-tracking branch 'upstream/master' into flowlock-api 2019-07-03 20:44:15 -07:00
Jingyu Zhou 5ea2e69016 Remove a fdbprc header from flow library
Flow should be an independent library.
2019-07-03 19:56:38 -07:00
Evan Tschannen 3fb0999e10 revert storage server priority changes 2019-07-02 16:54:47 -07:00
Alex Miller 8e1ab6e7db Merge remote-tracking branch 'upstream/master' into flowlock-api 2019-06-28 17:32:54 -07:00
Evan Tschannen 4cef1d3937 Experimental change of storage write priority 2019-06-28 16:54:22 -07:00
Evan Tschannen 1b939d5208
Merge pull request #1749 from satherton/feature-redwood
Update redwood storage engine to latest correctness-passing version
2019-06-28 16:22:06 -07:00
Evan Tschannen 5ccffe3cb3
Merge pull request #1684 from jzhou77/large-packet
Better handling for large packets
2019-06-28 16:19:01 -07:00
Evan Tschannen 7f4586ad49 the number of txsTags needs to be tracked separately from the number of transaction logs because of forced recoveries 2019-06-28 12:33:24 -07:00
Evan Tschannen 52efcfd136 fix: properly create the right number for txsTags when changing between different numbers of logs 2019-06-27 15:15:05 -07:00
mpilman 7efdf492ce Fixed two silly compilation bugs 2019-06-27 10:29:59 -07:00
Alex Miller a2a5bcc73d Merge remote-tracking branch 'upstream/master' into flowlock-api 2019-06-26 20:31:09 -07:00
A.J. Beamon 7f23814841 Track run loop busyness and report it in status. 2019-06-26 14:03:02 -07:00
Alec Grieser e5d02b7ad4
use _WIN32 to determine platform instead of __WIN32__ 2019-06-26 11:55:26 -07:00
Alex Miller bf883d7055 Merge remote-tracking branch 'upstream/master' into flowlock-api 2019-06-25 14:26:50 -07:00
Evan Tschannen 0fe6edc254
Merge pull request #1678 from mpilman/features/external-workload
Features/external workload
2019-06-25 13:53:19 -07:00
Jingyu Zhou e6ff23c420 Allow next packet size to be read when receiving data
For large packet, allocate sizeof(uint32_t) more bytes for next packet size.
Also add knob MIN_PACKET_BUFFER_FREE_BYTES, which is used to trigger allocation
of a new arena when free bytes are lower than this threshold.
2019-06-25 10:18:56 -07:00
Jingyu Zhou c2c82c89a5 Remove knob DEFAULT_PACKET_BUFFER_BYTES 2019-06-25 10:18:56 -07:00
Jingyu Zhou 2363326ecb Add knobs for various packet buffer sizes 2019-06-25 10:18:56 -07:00
Alex Miller 7a500cd37f A giant translation of TaskFooPriority -> TaskPriority::Foo
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
Stephen Atherton f1f1081202 Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
# Conflicts:
#	fdbserver/VersionedBTree.actor.cpp
2019-06-24 20:17:49 -07:00
Evan Tschannen 54c2b8d5bf pseudoLocalities was introduced in the current protocol version, which is 6107 2019-06-24 17:04:52 -07:00
mpilman a812b93fbf Fixed version for PseudoLocalities 2019-06-23 09:46:18 -07:00
Evan Tschannen 37c1df2491
Merge pull request #1705 from bnamasivayam/suspend-process
Extend RebootRequest API to include time to suspend the process befor…
2019-06-20 17:36:25 -07:00
Alex Miller df0baa0066
Merge pull request #1720 from mpilman/features/protocol-version
Make protocol version a type
2019-06-19 13:46:35 -07:00
mpilman 2eff2b7e21 First simple test is working (but very buggy) 2019-06-19 13:03:41 -07:00
mpilman c8957d93f8 Implementation code complete 2019-06-19 13:03:41 -07:00
mpilman 68ce9a5e75 ProtocolVersion type - second try 2019-06-18 17:55:27 -07:00
Alex Miller 0e5773106c Platform.cpp is a mess, and I was far too optimistic with that push. 2019-06-18 16:50:08 -07:00
Alex Miller ed118bac17 Fix copy-paste error. 2019-06-18 16:40:17 -07:00
Alex Miller 4fa5dc0502 Merge remote-tracking branch 'upstream/master' into cloexec 2019-06-18 16:35:18 -07:00
Alex Miller 5da8746803 Move fnctl calls into Platform function, and add traceevent on error. 2019-06-18 16:34:35 -07:00
mpilman 8576665a90 Revert "Revert "Make protocol version a type""
This reverts commit 455bf3b3ec.
2019-06-18 14:49:04 -07:00
Alex Miller 455bf3b3ec Revert "Make protocol version a type" 2019-06-18 10:59:17 -07:00
A.J. Beamon c3aa5819f2
Merge pull request #1417 from mpilman/features/client-buggify
Overall framework and first buggify entries
2019-06-18 09:10:11 -07:00
mpilman 4a14117549 Fixed typo 2019-06-17 09:30:28 -07:00
mpilman 52da31bfb3 Simplifications and addressed reviews 2019-06-16 11:07:10 -07:00
Markus Pilman 882fa39e80 fixed typo
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-06-16 09:59:15 -07:00
Markus Pilman 23fb403216 Fix typo
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-06-16 09:59:15 -07:00
mpilman dc9522bd86 Address code-review comments 2019-06-16 09:59:15 -07:00
mpilman 9fd8dac767 Added license header to new file 2019-06-16 09:59:15 -07:00
mpilman da53a92bec Make protocol version a type
This fixes #1214

The basic idea is that ProtocolVersion is now its own type. This
alone is an improvement as it makes many things more typesafe. For
each version, we can now add breaking features (for example Fearless).
After that, there's no need to test against actual (confusing) version
numbers. Instead a developer can simply test
`protocolVersion->hasFearless()` and this will return true iff the
protocolVersion is newer than the newest version that didn't support
fearless.
2019-06-16 09:59:15 -07:00
mpilman 7483cba26e Address review comments and added documentation 2019-06-16 09:52:40 -07:00
mpilman 6ea75713cb Overall framework and first buggify entries 2019-06-16 09:09:09 -07:00
Balachandar Namasivayam 5eb833759e Extend RebootRequest API to include time to suspend the process before reboot. This is intended to be used for testing purposes to simulate failures. 2019-06-14 11:35:38 -07:00
Evan Tschannen 6ececa94ce
Merge pull request #1640 from vishesh/task/client-failmon
Clients will no longer get failure monitoring info from cluster controller
2019-06-13 17:31:17 -07:00
Trevor Clinkenbeard 1e8f7e5b82 Refactor NextFastAllocatedSize to be constexpr function 2019-06-11 15:55:23 -07:00
mpilman 4c62458172 Make valgrind work on Fedora 30 2019-06-11 10:27:08 -07:00
Vishesh Yadav a8e408e268 run clang-format on changes 2019-06-10 14:10:24 -07:00
Alex Miller 0b07dbd05c Remove a horrible hack aiming at Ubuntu 14.04's libstdc++_pic.
Because our supported build environments no longer include Ubuntu 14.04,
and this is causing problems with libc++-based builds.
2019-06-10 13:16:03 -07:00
Vishesh Yadav 6fa7081a21 net: Don't make FailureMonitoring requests from client
This patch removes the need for clients to continuously contact
cluster coordinator for failure monitoring information. Instead, it
uses the FlowTransport to monitor the statuses of peers and update
FailureMonitor accordingly.
2019-06-09 00:43:38 -07:00
Vishesh Yadav 6b4d30c3ae failmon: Identify client vs server when starting failure monitoring client 2019-06-09 00:43:12 -07:00
Alex Miller 526468becb Fix printable() being incorrect for bytes >127.
The use of `auto` hid that TraceableString<T>::begin(value) was a signed
char, and therefore `*iter` was -1, and -1/16 is 0.

Thus, a 0xFF byte was printed as \x0f.  Forcing it to an unsigned type
makes it correctly become \xff.
2019-06-05 14:41:31 -07:00
A.J. Beamon c027792d0f Make the default seed of the generator returned by deterministicRandom() random. This means that the random number sequence won't be deterministic unless you set a seed. 2019-06-05 11:48:29 -07:00
Parallels 773f52d0a1 Merge remote-tracking branch 'upstream/master' into cloexec 2019-06-03 15:43:32 -07:00
sramamoorthy a8c9432ced windows build fix : del sys/wait.h in Platform.cpp 2019-05-31 14:59:53 -07:00
Evan Tschannen 362c2bf1e6 improved the cpu efficiency of printable 2019-05-29 14:55:45 -07:00
Stephen Atherton 02882dbf00 Checkpointing progress, RedwoodRecordRef and DeltaTree tests pass but BTree test does not. RedwoodRecordRef::Delta rewritten to actually do prefix compression on key and integer fields. Added related unit tests and benchmarks. Some improvements to DeltaTree and requirements on its T and Delta types to avoid repeated common prefix discovery. 2019-05-29 06:23:32 -07:00
sramamoorthy b17ad85497 exec op not supported when log_anti_quorum > 0 2019-05-28 22:07:46 -07:00
sramamoorthy bb474dc323 if recovery < fully_recovered then fail the exec
Will do more cleanup, pushing it for a test run in CI
2019-05-28 22:07:46 -07:00
sramamoorthy 925499954b New status cluster_not_fully_recovered 2019-05-28 22:07:46 -07:00
sramamoorthy d68a229772 makefile changes to accommodate boost/process.hpp 2019-05-28 22:07:46 -07:00
sramamoorthy b6e037ffbc Replace fork with boost::process::child 2019-05-28 22:07:46 -07:00
sramamoorthy 898bed66c1 Allow only whitelisted binary path for exec op 2019-05-28 22:07:46 -07:00
sramamoorthy aa79480d69 changes to make fdbfork asynchronous 2019-05-28 22:07:46 -07:00
sramamoorthy 4bc4c615da exec op to all tlog, restore change in test &other
- exec operation to go to all the TLogs
- minor bug fix in tlog
- restore implementation for the simulator
- restore snap UID to be stored in restartInfo.ini
- test cases added
- indentation and trace file fixes
2019-05-28 22:07:46 -07:00
sramamoorthy 72dd067173 Trace message changes and fix few FIXMEs 2019-05-28 22:07:46 -07:00
sramamoorthy 69edefe68b Snapshot based backup and resotre implementation 2019-05-28 22:07:46 -07:00
A.J. Beamon ec80ff1e8d Remove unneeded redefinition of random functions. 2019-05-24 11:55:25 -07:00
A.J. Beamon fc204c846a Make IRandom reference counted. Simplify the logic around deterministicRandom() to require less state and branching. 2019-05-23 18:51:59 -07:00
A.J. Beamon 764ca2971c Merge branch 'master' into thread-safe-random-number-generation 2019-05-23 18:49:49 -07:00
A.J. Beamon 296adada93 Use c++11 thread_local instead of redefining it to platform specific variants. 2019-05-23 18:35:45 -07:00
A.J. Beamon f417e60264 Merge branch 'merge-release-6.1-into-master' into thread-safe-random-number-generation
# Conflicts:
#	fdbserver/QuietDatabase.actor.cpp
2019-05-23 09:52:00 -07:00
A.J. Beamon d29c7e4c9b Merge branch 'release-6.1' into merge-release-6.1-into-master
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/QuietDatabase.actor.cpp
#	versions.target
2019-05-23 09:28:45 -07:00
A.J. Beamon e5381e0612 Fix some new usages of g_random 2019-05-23 09:23:27 -07:00
A.J. Beamon 603721e125 Merge branch 'master' into thread-safe-random-number-generation
# Conflicts:
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/AsyncFileCached.actor.h
#	fdbrpc/genericactors.actor.cpp
#	fdbrpc/sim2.actor.cpp
#	fdbserver/DiskQueue.actor.cpp
#	fdbserver/workloads/BulkSetup.actor.h
#	flow/ActorCollection.actor.cpp
#	flow/Net2.actor.cpp
#	flow/Trace.cpp
#	flow/flow.cpp
2019-05-23 08:35:47 -07:00
Stephen Atherton ebc96a7e0e Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
# Conflicts:
#	fdbserver/VersionedBTree.actor.cpp
2019-05-21 23:49:27 -07:00
A.J. Beamon d3c1f9afbd Suppress 'TraceEventFieldNotFound' event. Don't suppress simultaneous FDBLibTLSVerifyFailure events (this requires including flow.h and removing the copied version of Reference/ReferenceCounted in FDBLibTLS). 2019-05-20 15:12:28 -07:00
Jingyu Zhou d660ffb654
Merge pull request #1605 from mpilman/remove-traceevent-benchmark
remove memory-hungry benchmark
2019-05-17 17:04:45 -07:00
A.J. Beamon a8b9d8e34b
Merge pull request #1336 from tclinken/fast-allocate-ptree-nodes
Create 96-byte fast allocator for storage queue PTree nodes
2019-05-17 14:22:46 -07:00
mpilman a0fa81aa1c remove memory-hungry benchmark
This test was used as a benchmark to prove that the
new `IsTraceable` feature improves performance. However
it will cause smaller machines to OOM and it doesn't
help for correctness testing. Therefore this test should
likely be removed
2019-05-17 14:12:44 -07:00
Steve Atherton 5a8c97480a
Merge pull request #1506 from nikolas-ioannou/feature-pagecache-lru
AsyncFileCached: switch from a random to an LRU cache eviction policy
2019-05-17 13:42:21 -07:00
Alvin Moore f598d0a614
Merge pull request #1593 from AlvinMooreSr/variant-missing
Switched Variant Header
2019-05-17 10:42:21 -07:00
Alvin Moore 3acaa7343e Enabled C++17 for all Windows projects
Set Visual Studio version to 2017 (first version to support C++17)
2019-05-16 17:44:13 -07:00
Alvin Moore 94aed513c7 Switched Windows tools within projects to 2017 2019-05-16 15:05:11 -07:00
Alvin Moore 17a2983943 Switch boost variant to C++17 variant 2019-05-16 13:05:49 -07:00
Alvin Moore 6389a12931 Switched variant to use boost variant for source consistency 2019-05-16 10:37:33 -07:00
Alex Miller 69fb852ee0 Add more CLOEXEC-like things.
From missed call sites found during/after code review.
2019-05-14 20:30:58 -10:00
Alex Miller 1f02cd30e2 Mark all opened files as close-on-exec. 2019-05-13 16:05:49 -10:00
Alex Miller 19165dc873 Mark all network sockets as close-on-exec. 2019-05-13 16:05:49 -10:00
Evan Tschannen a9b03ec505
Merge pull request #1090 from mpilman/features/protocol-version-flags
Features/protocol version flags
2019-05-13 16:33:06 -07:00
A.J. Beamon 60ce790a2b Consistent use of #if VALGRIND 2019-05-13 12:44:13 -10:00