3ff2fa9c2a
Removed uses of outdated cpuTicks and errorCounts
2020-05-13 23:27:44 -07:00
ae2b2700fc
Remove duplicate code from RequestStream::send
2020-05-13 23:24:21 -07:00
cc17484cac
Fixed Windows build
2020-05-13 19:59:35 -07:00
641fd37e71
Remove const_cast from AsyncFileCached::write_impl
2020-05-13 19:59:34 -07:00
4eaf91da14
Remove comma operator from actor fuzz
2020-05-13 19:59:34 -07:00
afe6bf19e2
Fix typos and whitespace
2020-05-11 12:53:19 -07:00
9dc65547ad
Avoid setting failure status of local addresses in main thread
2020-05-11 11:24:19 -07:00
f0f52fb2be
Merge branch 'master' into feature-small-endpoint
...
# Conflicts:
# fdbclient/StorageServerInterface.h
2020-05-08 16:37:35 -07:00
749fcd13b0
Merge branch 'master' into fdb_cache_wo_allocator
2020-05-08 16:23:29 -07:00
62882f5552
Merge pull request #3088 from sears/lb_ub_perf_2
...
unrevert lb_ub_perf pull request
2020-05-07 18:23:17 -07:00
0be453de0a
Merge pull request #3026 from atn34/atn34/tsan
...
Add USE_TSAN and fix some data races
2020-05-07 18:06:18 -07:00
3964fe34b8
Use c11 atomics in libeio
2020-05-07 23:43:50 +00:00
ff992060cd
Merge pull request #3073 from tclinken/fix-open-for-ide-build
...
Fix non-boost-related OPEN_FOR_IDE build errors
2020-05-07 14:47:59 -07:00
574914640a
Merge pull request #2999 from tclinken/more-move-optimizations
...
Generate rvalue reference overloads for actor callback functions
2020-05-07 14:32:27 -07:00
5e1c43f87b
replace double invocations of operator<() with three way compare().
2020-05-07 12:08:29 -07:00
ad900135dd
Simulation did not properly track exclusions of tls processes
2020-05-07 10:53:13 -07:00
51d3aaf4ae
fixed a few rare correctness bugs
2020-05-06 23:24:58 -07:00
8125096e9b
Revert "reduce the number of calls to operator< made by lower_bound and upper…"
2020-05-05 08:17:10 -07:00
dd033736ed
Merge branch 'master' into fdb_cache_subfeature2
2020-05-04 17:29:43 -07:00
4ae356b237
Merge pull request #2882 from sears/lb_ub_perf
...
reduce the number of calls to operator< made by lower_bound and upper…
2020-05-04 15:46:22 -07:00
78d644f673
Merge branch 'master' into bugfixes/cmake-includes
2020-05-04 02:11:46 -07:00
bc19f70455
Move delayedHealthUpdateF declaration
2020-05-01 18:14:51 -07:00
c126422296
fix: firstFree was not reset after using all available space
2020-05-01 17:46:20 -07:00
4d131bdd4a
Merge branch 'master' into feature-small-endpoint
2020-05-01 13:16:15 -07:00
aed2d34bcb
Merge branch 'master' into feature-proxy-load-balance
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# flow/Knobs.cpp
2020-05-01 09:19:39 -07:00
9753013b65
fdbrpc: update cmake & headers
2020-04-30 20:11:43 +00:00
7a8cb3ffcf
add headers
2020-04-30 18:11:23 +00:00
dd90eff171
Added additional assertion to /flow/flow/PromiseStream/move2
2020-04-30 07:55:13 -07:00
fd0ee72293
Merge branch 'master' into feature-small-endpoint
2020-04-29 18:43:10 -07:00
c4dda9195c
Fixed Windows CMake build
2020-04-29 13:22:01 -07:00
c205bc902f
Added SingleCallback<T>::fire(T&&)
2020-04-29 12:28:17 -07:00
dcdda88794
replace double invocations of operator<() with three way compare().
2020-04-29 10:10:30 -07:00
76fb345dd1
Merge branch 'master' into feature-tree-broadcast
...
# Conflicts:
# fdbrpc/FailureMonitor.actor.cpp
2020-04-29 09:51:22 -07:00
2b2e0b9f75
Merge pull request #2932 from vishesh/task/issue-1017-slow-machine-poisoning
...
FailureMonitoring: Mark unstable connections as failed [1/X]
2020-04-28 13:57:49 -07:00
b7f5f3be48
merge in master
2020-04-28 13:11:47 -07:00
2e6990119b
Fix some data races
2020-04-25 00:26:54 +00:00
c87aa33941
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/go/src/fdb/generated.go
# documentation/sphinx/source/api-common.rst.inc
# documentation/sphinx/source/api-ruby.rst
# documentation/sphinx/source/release-notes.rst
# fdbclient/FailureMonitorClient.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-04-23 13:47:53 -07:00
76d4570d5d
Merge pull request #3008 from etschannen/release-6.2
...
Added logging for parallel peeks from TLogs
2020-04-23 10:27:49 -07:00
c30010f5b8
FlowTransport: Increase delay for connecting to failed connections
2020-04-23 09:44:30 -07:00
a835123680
exit fdbserver after a ReceiverError, because a packet which should be delivered will never be received
2020-04-22 23:38:46 -07:00
810bba2067
cleanup calls to FlowTransport::isClient()
2020-04-22 23:36:40 -07:00
3a5315d10c
FlowTransport: Don't immediately mark connections failed
...
In connectionKeeper(), when a connection is failed for
FAILURE_DETECTION_DELAY, then only mark connection as failed.
This is much closer to the original centralized behaviour, and also
adds more confidence on whether the connection is actually failed.
2020-04-22 19:46:04 -07:00
8b004fe8e3
Move stop callbacks to be called after run() in sim2.
2020-04-21 20:22:16 -07:00
2628afe463
Improved PromiseStream move unit tests
2020-04-21 15:18:59 -07:00
885a2e020e
Generate rvalue reference overloads with actor compiler
2020-04-21 15:18:59 -07:00
500a265d20
FlowTransport: Refactor and clear FailureStatus on destroy
2020-04-21 14:11:19 -07:00
92c86a7799
Merge remote-tracking branch 'apple/master' into task/issue-1017-slow-machine-poisoning
2020-04-21 03:36:19 -07:00
2ce539ef6d
Respect flow<->fdbrpc module boundaries.
...
Which fixes a compilation error due to a circular dependency between
flow.a and fdbrpc.a. However, this is now done at the cost of newNet2
users have to remember to add Net2FileSystem::stop() as a callback.
2020-04-20 02:53:07 -07:00
3f3a728bb3
Merge pull request #2970 from xumengpanda/mengxu/fr-range-versions-PR
...
Fix:New backup and restore: Mutations before min range version may not be complete
2020-04-19 09:49:07 -07:00
a442565e13
more work towards shrinking locality
2020-04-18 21:29:38 -07:00
1398e9a82e
Stop background eio threads on Net2::stop().
...
This will stop eio threads for both the client (`fdb_stop_network()`)
and the server. This change is being done more for the former, but I
don't see any harm in doing the latter as well.
2020-04-18 19:40:55 -07:00
99a58f8ee5
fix compiler errors
2020-04-17 17:47:50 -07:00
0ee62badcd
Merge branch 'feature-tree-broadcast' into feature-small-endpoint
2020-04-17 17:16:04 -07:00
ba3e2af473
Merge commit '5288033bcfe40c3ade97c8bf2d04cf31b3f16cb1' into feature-tree-broadcast
2020-04-17 15:17:37 -07:00
992002cf34
FlowTransport: don't start connectionKeeper for local peer
...
getOrOpenPeer() is used for local addresses sometimes, which ends up
starting connectionKeeper() which is unnecessary.
2020-04-16 14:04:32 -07:00
43f19bd463
Don't skip filesystem check on when KAIO is disabled
...
WSL the known Linux system where KAIO is not supported, can run these checks.
2020-04-16 10:56:01 -07:00
8c8f23bff2
Merge remote-tracking branch 'apple/master' into task/issue-1017-slow-machine-poisoning
2020-04-16 00:45:35 -07:00
1901f49b97
Net2FileSystem: Add guards to honor DISABLE_POSIX_KERNEL_AIO
...
- Adds some asserts in KAIO to ensure that when knob is set, we don't
end up using KAIO in any case.
- Fixes a bug where we initialize AsyncFileKAIO on Linux builds even
when KAIO is disabled. This can cause problems in systems such as
Windows Subsystem for Linux where KAIO is not supported.
FIXES #2382
2020-04-15 23:47:37 -07:00
da7d0093ee
Cleanup unused code
2020-04-15 19:48:25 -07:00
023372a226
FailMon: Mark peer failed after retrying
2020-04-15 19:47:24 -07:00
d6c1baa784
FastRestore:Filter out log mutations whose version is smaller than range mutation version
2020-04-15 19:45:03 -07:00
b85dc16c6d
Merge branch 'master' into fdb_cache_subfeature2
2020-04-14 17:07:41 -07:00
f959af8228
Refactor per review comments
2020-04-14 11:30:40 -07:00
2e53c8c5e2
Merge pull request #2915 from tclinken/move-optimizations
...
Avoid unnecessary copies in PromiseStream
2020-04-13 21:15:16 -07:00
dbf6afc78e
more space efficient endpoint map
2020-04-12 23:51:20 -07:00
ff5543b579
working implementation
2020-04-12 22:18:51 -07:00
0c2e8b9462
only serialize a single endpoint for an interface
2020-04-12 16:04:48 -07:00
07cc0a8d74
code cleanup
2020-04-10 17:02:11 -07:00
e8d333733a
Merge branch 'master' into feature-tree-broadcast
2020-04-10 13:51:09 -07:00
ac4654b09e
re-suppress trace event
2020-04-10 13:50:26 -07:00
ce4493f679
many bug fixes
2020-04-10 13:45:16 -07:00
fed5c543d4
Remove leftover TODO code around centralized healthmonitor
2020-04-08 22:56:07 -07:00
13447f439f
fdbrpc: Add a constant to onFailedFor()
...
Since, we mark an address as failed when connection is failed, this
patch adds a contant to compensate the time needed to reconnect and
make sure endpoint is actually down. This contant is equal to
FAILURE_MIN_DELAY which was used by centralized
FailureMonitoringClient earlier removed.
2020-04-08 19:34:40 -07:00
975e6b1d9a
Merge remote-tracking branch 'apple/master' into task/issue-1017-slow-machine-poisoning
...
Removed merge conflict with old build system.
2020-04-08 19:25:13 -07:00
3a01d24970
Pass const ref to a_callback_fire
2020-04-08 14:50:41 -07:00
488c20e58e
Fixed failing "/flow/flow/promisestream callbacks" unit test
2020-04-08 11:24:56 -07:00
ff3e3fcc13
Added /flow/PromiseStream/move unit test
2020-04-08 00:45:56 -07:00
6ab4a57123
Allow RequestStream::send to use move semantics
2020-04-07 11:06:55 -07:00
d4542dbb5a
Delete old build system
2020-04-07 11:03:45 -07:00
36d89db4e7
FlowTransport: Don't always close unused open connection
2020-04-06 23:05:29 -07:00
4214c45218
FlowTransport: Temporarily keep idle connections open for server processes
2020-04-06 21:27:54 -07:00
a51c92854a
Merge branch 'master' into feature-tree-broadcast
...
# Conflicts:
# fdbserver/WorkerInterface.actor.h
# fdbserver/worker.actor.cpp
2020-04-06 21:09:44 -07:00
2a1bd97120
fix compilation errors
2020-04-06 20:58:43 -07:00
8b5780c36c
don't include source and binary dir
...
This forces users to use include paths from the sources root.
So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
2020-04-06 10:13:49 -07:00
477d66b46d
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-05 23:09:36 -07:00
fdc1048f75
Add knob to turn off marking unstable connections
2020-04-03 15:53:00 -07:00
613b8bb169
More TraceEvents and remove 'delayed' used for debugging
2020-04-03 15:53:00 -07:00
1d35f2ff5a
Mark a connection as failed for X seconds if closes too often
2020-04-03 15:53:00 -07:00
d90e168e24
Add HealthMonitoring skeleton code
2020-04-03 15:53:00 -07:00
04f925f770
Format FailureMonitor* files
2020-04-03 15:53:00 -07:00
78f0cddb14
Merge pull request #2684 from mpilman/features/boost70
...
Upgrade to boost 1.72
2020-04-03 09:30:59 -07:00
a0c8946f31
Merge branch 'master' into fdb_cache_subfeature2
2020-04-02 12:27:04 -07:00
ccbbdc4ba4
Unit test:Verify splitMutation by comparing with intersectingRanges result
2020-03-31 12:13:02 -07:00
28cd38913d
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-03-27 13:44:10 -07:00
acaf91ac47
Merge branch 'master' into fdb_cache_subfeature2
2020-03-26 13:33:08 -07:00
a476127f5f
Merge pull request #2802 from xumengpanda/mengxu/debug-master-PR
...
Fix correctness failure on master branch
2020-03-18 16:07:36 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
c197520fa7
Merge pull request #2810 from alexmiller-apple/fdbcli-tlsinfo
...
Add a `tlsinfo` command to fdbcli that prints the certificate chain.
2020-03-16 15:47:32 -07:00
7f559bc712
Cleanup code and apply clang-format
...
Self code review
2020-03-16 15:08:32 -07:00
243c268d9d
Limit the amount of requests the proxy can queue up in memory
2020-03-13 10:17:49 -07:00
0c558efcfe
Add a `tlsinfo` command to fdbcli that prints the certificate chain.
...
This requires the certificate chain to load successfully, otherwise
fdbcli will error out at an earlier point due to Net2 not being able to
configure TLS.
2020-03-13 00:11:53 -07:00
0ef09539a9
addressMap[normalizedAddress]->address may not equal to normalizedAddress
2020-03-12 13:01:25 -07:00
1759d5c8c4
Apply clang-format
2020-03-12 10:18:53 -07:00
a9136f3f72
Add waitForUnreliableExtraStoreReboot to wait for extra store to reboot
2020-03-12 10:18:31 -07:00
bd345f85db
ConsistencyCheck:Fix failue due to address inconsistency between process and worker
...
With TLS, a worker (or process) can have a TLS address and non-TLS address.
When a process is created in simulation, the primary address is TLS by default.
The non-TLS one is the TLS address port plus one.
In a connection between two workers, if their primary addresses do not enable
or disable TLS together, one worker will swap its primary address and secondary address
so that the TLS config of the two endpoints can match.
The swap can make the primary address no longer the TLS one that was created
when the process is created. And the swap only happens for worker instead of
process struct in simulation.
This swap can cause worker->address != process->address.
In checkForExtraDataStores actor, we use worker->address to check if a process
is killable and use the process->address to kill the process. The inconsistency
can cause simulation to kill a protected process that is not killable and leads
to simulation failure.
2020-03-10 21:07:16 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
dbfc0cbcc0
Merge pull request #2781 from alexmiller-apple/certificate-refresh
...
Refresh certificates used for handshaking when they change on disk
2020-03-06 11:12:04 -08:00
effb6d2d49
Add ResolverMetrics trace event
2020-03-05 10:49:21 -08:00
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
976c2fc7a8
Update fdbrpc/FlowTransport.actor.cpp
...
Co-Authored-By: Alex Miller <35046903+alexmiller-apple@users.noreply.github.com>
2020-03-04 16:13:59 -08:00
820957025f
accept connections in batches of 20 to improve performance
2020-03-04 14:24:57 -08:00
c11c24b79d
removed the fdbrpc version of platform.h
2020-02-28 14:56:10 -08:00
6054c05963
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-02-28 12:11:05 -08:00
2586bade68
re-added support for configuration TLS options with environment variables
2020-02-26 15:33:48 -08:00
cd949eca71
Merge branch 'master' into fdb_cache_subfeature2
2020-02-26 11:22:08 -08:00
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
d60268123b
updated comment
2020-02-25 16:00:46 -08:00
6e7d2ff7dd
prevent the proxy from delaying too long based on an incorrect estimate of the compute time
2020-02-25 15:46:13 -08:00
65fbe0d0bc
revert AcceptSocket priority change because of bad performance results
2020-02-21 19:22:14 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
6f1d3ccd35
Merge branch 'release-6.2' into feature-boost-ssl
2020-02-20 20:03:40 -08:00
7056c73f20
fixed a number of problems with findBestPolicySetSimple
2020-02-20 20:00:54 -08:00
dc3826e2fd
fix: tls throttling would re-insert the failure into the map
2020-02-20 18:17:39 -08:00
f04e311a1e
Merge commit 'b46d6e25e24993ab5a5f04091fd3235050b7cd09' into feature-boost-ssl
...
# Conflicts:
# fdbserver/SimulatedCluster.actor.cpp
# flow/Net2.actor.cpp
2020-02-20 17:36:38 -08:00
a50939417b
fix: zoneid is all lower case
2020-02-20 17:26:44 -08:00
f7a37077cc
handshake takes time in simulation
2020-02-20 15:26:56 -08:00
d7c841a28a
Merge pull request #2589 from etschannen/feature-proxy-delay
...
Improve version pipelining on the proxy
2020-02-20 15:23:30 -08:00
8b768e66df
Merge pull request #2694 from dongxinEric/feature/2663/specialize-policy-for-zoneid-in-cc
...
Added a specialized algorithm for PolicyOne and PolicyAcross(,'zoneId…
2020-02-20 14:46:23 -08:00
def8ca6da3
simulation advances timer() separately from now() to better model the real world
2020-02-20 12:10:20 -08:00
298d6cb3d7
Address review comments.
2020-02-20 09:34:01 -08:00
3c4d551647
improve prioritization of connection monitor and listen given that listen is no longer expensive (because handshake is done separately)
2020-02-19 18:50:21 -08:00
761da5a059
code cleanup
2020-02-19 17:59:45 -08:00
a6486766c2
fix: rebooting an unreliable process will make it reliable again, but while unreliable the files for that process could have already been corrupted so simulation will think a process is healthy that is actually corrupted
2020-02-19 15:18:57 -08:00
46d5f5e325
do not trigger the resetPing if we cannot actually remove the peer, because it will cause us to reset the timeout, so repeated calls to removePeer can keep a dead peer from being removed
2020-02-19 15:17:50 -08:00
efc0d7f9d5
Added a specialized algorithm for PolicyOne and PoilcyAcross(,'zoneId',PolicyOne()) to find a set of TLog servers which will be able to fulfill the policy later.
2020-02-19 09:25:57 -08:00
132f5aa9ba
FastRestore:Improve trace name and cosmetic change
2020-02-18 16:41:19 -08:00
31a6ec34b7
Merge branch 'master' into mengxu/fast-restore-agent-PR
2020-02-18 16:17:59 -08:00
9d88356468
Merge pull request #2686 from mpilman/features/avoid-unnecessary-template-instanciations
...
Removed dead code
2020-02-17 14:46:39 -08:00
9144c3e8ca
Merge pull request #2087 from atn34/issue-1226
...
Allow member actors access to private variables
2020-02-17 14:39:31 -08:00
aac94a766b
Removed dead code
2020-02-15 21:56:48 -08:00
c2ccbbadd8
Revert "Several fixes to make FDB compile with clang-cl"
...
This reverts commit 0e1f9efb85
2020-02-14 22:18:30 -08:00
ccf590e193
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-02-14 22:05:51 -08:00
0e1f9efb85
Several fixes to make FDB compile with clang-cl
2020-02-14 22:05:43 -08:00
3a1e878a9b
Upgrade to boost 1.72
2020-02-14 18:10:13 -08:00
1248d2b8b4
Remove USE_OBJECT_SERIALIZER knob
2020-02-12 10:41:52 -08:00
dcbce3593e
fixed TLS in simulation
2020-02-10 14:00:21 -08:00
e71fe44ee3
Merge branch 'master' into features/icc
2020-02-08 21:33:02 -08:00
6b921ac900
Stop building FDBLibTLS and stop linking against libtls.so
...
Which now means OpenSSL and LibreSSL are equally acceptable.
2020-02-06 21:13:58 -08:00
38d8d0d675
fixed simulation
2020-02-06 19:29:31 -08:00
tclinken