49c6bb90ef
Merge pull request #2982 from alexmiller-apple/tls-log-settings
...
Log Net2TLSConfig with paths and settings when using TLS.
2020-04-20 15:46:26 -07:00
75a4f3b7c9
Remove comment about ignoring runOnMainThread errors.
...
If we got an exception, it wouldn't be of type `Error` anyway, so
it seems like things would crash regardless.
2020-04-20 13:19:42 -07:00
da8e47ea25
Merge remote-tracking branch 'upstream/release-6.2' into tls-background-eio-thread
2020-04-20 13:15:05 -07:00
5c399bf725
Move the callbacks into ::run() right before it exits.
...
stopped=true doesn't cause the run loop to immediately exit.
2020-04-20 13:14:19 -07:00
7507f2da81
Merge pull request #2984 from satherton/future-move-t-constructor
...
Added Future<T>(T &&value) constructor to avoid a copy...
2020-04-20 11:47:32 -07:00
ba1b0a1d96
Use std::move() instead of forward.
2020-04-20 11:01:01 -07:00
c28a843251
Merge pull request #2977 from alexmiller-apple/tls-no-atexit
...
Fix clients crashing in TLS code on exit.
2020-04-20 08:40:16 -07:00
022b77e288
Actor compiler will std::move() return expressions that exactly match a state variable.
2020-04-20 04:19:33 -07:00
2ce539ef6d
Respect flow<->fdbrpc module boundaries.
...
Which fixes a compilation error due to a circular dependency between
flow.a and fdbrpc.a. However, this is now done at the cost of newNet2
users have to remember to add Net2FileSystem::stop() as a callback.
2020-04-20 02:53:07 -07:00
7b23c6f640
Future constructor to avoid a copy when Future<T> is initialized from an rvalue reference to T.
2020-04-20 01:52:28 -07:00
21277fdb4f
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2020-04-20 01:47:21 -07:00
cbb6ffb431
Only log OpenSSL error strings for OpenSSL errors.
...
Normal "connection refused" messages would show up with a long verbose
string that doesn't really provide any useful information otherwise.
2020-04-18 20:39:02 -07:00
11eebc4a48
Log Net2TLSConfig with paths and settings when using TLS.
...
There were similar TraceEvents in the FDBLibTLS/LibreSSL TLS
implementaiton that were accidentally dropped in the TLS rewrite.
This makes it so that one does not have to use magic to figure out if a
process was configued with TLS correctly when some of the settings come
from environment variables.
2020-04-18 20:21:10 -07:00
1398e9a82e
Stop background eio threads on Net2::stop().
...
This will stop eio threads for both the client (`fdb_stop_network()`)
and the server. This change is being done more for the former, but I
don't see any harm in doing the latter as well.
2020-04-18 19:40:55 -07:00
94b4f78ea9
Fix clients crashing in TLS code on exit.
...
If client code initiates an FDB operation to a TLS cluster, and then
immediately exits the main thread, then OpenSSL's atexit handler would
potentially run while the network thread is attempting to do TLS
operations, and thus crash.
This commit removes the OpenSSL atexit hander, and instead relies on a
client intentionally ending the network thread to do TLS cleanup. If
the client code exits without stopping the network thread, then we'll
never free OpenSSL data structures, which is the safer thing to do.
2020-04-18 15:48:02 -07:00
cb6389d42d
Prevent main thread from destroying flatbuffers globals
...
We recently witnessed (using tsan) the main thread exiting without first
joining the network thread, and this caused data races and
heap-use-after-free's
Now the lifetime of these globals will be tied to the network thread
itself (and I guess every thread, but the one that actually uses memory
will be owned by the network thread.)
2020-04-17 23:34:28 +00:00
ba3e2af473
Merge commit '5288033bcfe40c3ade97c8bf2d04cf31b3f16cb1' into feature-tree-broadcast
2020-04-17 15:17:37 -07:00
dfec896438
Enforce a throttle limit. Don't count transaction tags on RK if the proxy has updated us in a while.
2020-04-17 11:48:02 -07:00
78d48a0dad
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/Knobs.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/fdbclient.vcxproj
# fdbserver/MasterProxyServer.actor.cpp
2020-04-17 09:23:18 -07:00
8c8f23bff2
Merge remote-tracking branch 'apple/master' into task/issue-1017-slow-machine-poisoning
2020-04-16 00:45:35 -07:00
cb47def22a
Merge pull request #2946 from alexmiller-apple/isystem-flow-lpthread
...
Remove `-isystem flow/-lpthread` from INCLUDES/CXXFLAGS
2020-04-15 11:01:45 -07:00
d8690d31cd
Merge branch 'master' into per-priority-busy-logging
...
# Conflicts:
# flow/Net2.actor.cpp
2020-04-15 08:31:30 -07:00
37e2b0d353
Update flow/network.cpp
2020-04-14 17:12:07 -07:00
a3598a7616
Merge pull request #2738 from ajbeamon/fix-assertion-failure-on-io-error
...
Fix assertion failure in SQLite thread pools on io_error
2020-04-14 16:48:22 -07:00
b1172417f5
Merge branch 'master' into per-priority-busy-logging
...
# Conflicts:
# flow/Knobs.cpp
# flow/Knobs.h
# flow/Net2.actor.cpp
2020-04-14 14:22:12 -07:00
e104a2e3a6
Merge commit 'cf01233f28a2c42908656a39f458a4475c1d44a3' into run-loop-busy-profiler
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/NativeAPI.actor.h
# fdbserver/fdbserver.actor.cpp
# flow/Net2.actor.cpp
2020-04-14 14:02:24 -07:00
cf01233f28
Merge pull request #2549 from ajbeamon/slow-task-and-priority-tracking-improvements
...
Slow task and priority tracker improvements
2020-04-14 13:26:32 -07:00
2e53c8c5e2
Merge pull request #2915 from tclinken/move-optimizations
...
Avoid unnecessary copies in PromiseStream
2020-04-13 21:15:16 -07:00
c851ee4031
Merge pull request #2897 from tclinken/fix-trace-batch-loggroup-and-role
...
Annotate trace batch events before dumping
2020-04-13 11:22:51 -07:00
7dc348d077
Remove `-isystem flow/-lpthread` from INCLUDES/CXXFLAGS
...
This cmake line generated a bogus and nonsensical include path, so as
the entire line isn't necessary, just remove it.
2020-04-13 02:22:25 -07:00
07cc0a8d74
code cleanup
2020-04-10 17:02:11 -07:00
8ef5a04896
Guard all of annotateEvent with mutex
2020-04-10 13:03:15 -07:00
29b2c2f3aa
Add hash to StringRef. Use unordered maps for storing tags. Create some helpful typedefs.
2020-04-10 12:54:59 -07:00
ebeca10bce
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-09 16:55:56 -07:00
01285f3374
Delay annotation of trace batch events created before trace file is opened
2020-04-09 14:09:00 -07:00
8de2bbb2dc
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2020-04-09 09:39:49 -07:00
13447f439f
fdbrpc: Add a constant to onFailedFor()
...
Since, we mark an address as failed when connection is failed, this
patch adds a contant to compensate the time needed to reconnect and
make sure endpoint is actually down. This contant is equal to
FAILURE_MIN_DELAY which was used by centralized
FailureMonitoringClient earlier removed.
2020-04-08 19:34:40 -07:00
975e6b1d9a
Merge remote-tracking branch 'apple/master' into task/issue-1017-slow-machine-poisoning
...
Removed merge conflict with old build system.
2020-04-08 19:25:13 -07:00
af4e0088ba
Merge pull request #2896 from tclinken/atomically-update-dependent-knobs
...
Atomically update dependent knobs
2020-04-08 15:00:49 -07:00
3a01d24970
Pass const ref to a_callback_fire
2020-04-08 14:50:41 -07:00
52860043c9
Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs
2020-04-08 12:26:21 -07:00
872877b221
Added StringRef::copyTo(), a prettier way to memcpy a StringRef somewhere and with a more useful return value.
2020-04-08 03:23:46 -07:00
36da61dd9c
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
2020-04-07 21:12:14 -07:00
d4542dbb5a
Delete old build system
2020-04-07 11:03:45 -07:00
a354d7a037
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2020-04-06 20:29:12 -07:00
e682f6741c
Avoid unnecessary copies in PromiseStream
2020-04-06 17:15:03 -07:00
8b5780c36c
don't include source and binary dir
...
This forces users to use include paths from the sources root.
So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
2020-04-06 10:13:49 -07:00
fdc1048f75
Add knob to turn off marking unstable connections
2020-04-03 15:53:00 -07:00
1d35f2ff5a
Mark a connection as failed for X seconds if closes too often
2020-04-03 15:53:00 -07:00
2336f073ad
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-03 15:24:14 -07:00
760dc68b7f
Merge pull request #2869 from dongxinEric/feature/1689/allow-custome-trace-log-file-identifier
...
Allow the user to provide a custome trace log file identifier that wi…
2020-04-03 14:08:17 -07:00
78f0cddb14
Merge pull request #2684 from mpilman/features/boost70
...
Upgrade to boost 1.72
2020-04-03 09:30:59 -07:00
10fee8fafc
Annotate trace batch events before dumping
2020-04-02 19:34:02 -07:00
bbd2fe62cc
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-04-02 09:21:01 -07:00
6bce67ca75
FastRestore:Apply clang-format
2020-04-01 21:27:54 -07:00
1eea388278
Add unit test for splitMutation test file
...
Comment out debug trace as well.
2020-04-01 21:27:18 -07:00
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
903128a36f
Fix boost error trace event fields to match other trace events by the same name.
2020-04-01 08:18:50 -07:00
c1b31b46ae
Missing include, doesn't cause a compile error but IDE complains when viewing the file.
2020-04-01 02:41:16 -07:00
7847d70e3a
StringRef:toString:Handle empty StringRef
2020-03-31 22:14:05 -07:00
6820167d77
Merge branch 'master' into feature/1689/allow-custome-trace-log-file-identifier
2020-03-31 16:50:46 -07:00
2805111a32
When provided with a custome identifier, use that string instead of the port/PID as the last part of the baseName.
2020-03-31 11:02:02 -07:00
60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
...
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
28cd38913d
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-03-27 13:44:10 -07:00
03e2102a21
Fix macOS build failure.
2020-03-26 11:41:36 -07:00
a0177a9335
Allow the user to provide a custome trace log file identifier that will be used as the prefix of all trace log files created at the client side.
2020-03-26 11:25:05 -07:00
baf0fe956c
Take trace mutex in setLogGroup
2020-03-26 09:55:03 -07:00
7d5ed53215
Allow trace log group to be set after database is created
2020-03-25 13:40:43 -07:00
e0424a52f8
Merge branch 'master' into transaction-tagging
2020-03-25 08:23:11 -07:00
9123cd35ed
Version report_conflicting_keys field
2020-03-24 18:11:15 -07:00
c0bae64105
Use sigaction, _exit if anything fails
2020-03-20 12:50:31 -07:00
26b7e02d4c
Some initial work to support tagging transactions and passing them around.
2020-03-20 11:23:11 -07:00
58a9bfa78b
Merge pull request #2820 from dongxinEric/fix/1977/add-back-trace-event-flush-failure-report
...
Fix/1977/add back trace event flush failure report
2020-03-18 16:11:44 -07:00
bed5d4733a
Fix syntax
2020-03-18 11:00:02 -07:00
0d4f49f02f
Run default signal handler after custom signal handler
2020-03-18 10:54:47 -07:00
d8a9034085
Merge pull request #2741 from ajbeamon/roughness-cleanup
...
Clean up and add comments to Roughness calculation
2020-03-17 15:45:35 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
31a9f0a26c
Fix the segfault
2020-03-17 11:03:46 -07:00
787a5caaca
Merge pull request #2805 from ajbeamon/localized-allocation-trace-depth
...
Don't disallow allocation tracking when a trace event is open
2020-03-16 16:21:09 -07:00
c7a9d184f0
Merge pull request #2790 from tclinken/ignore-create-directory-errors
...
Ignore createDirectory error if directory already exists
2020-03-16 16:13:47 -07:00
c197520fa7
Merge pull request #2810 from alexmiller-apple/fdbcli-tlsinfo
...
Add a `tlsinfo` command to fdbcli that prints the certificate chain.
2020-03-16 15:47:32 -07:00
f1523bd472
Setting the network thread more than once is a no-op
2020-03-16 15:37:06 -07:00
96187618a0
Fix condition to check whether allocation tracing is enabled
2020-03-16 15:12:50 -07:00
ed4d02a3e4
Merge pull request #2812 from etschannen/feature-proxy-mem-limit
...
Limit the amount of requests the proxy can queue up in memory
2020-03-16 14:56:56 -07:00
7769218303
Move an increment after an ASSERT.
2020-03-16 14:11:07 -07:00
d8cfabe73b
Extend the allocation tracing disabling flag to cover more parts of trace logging as a precaution. Make it possible to disable via knob.
2020-03-16 13:59:31 -07:00
89861c661e
Fix the random crash. Use a thread safe 'ThreadReturnPromise' instead of the ThreadFuture.
2020-03-16 13:36:55 -07:00
ee3cde0b0d
Merge pull request #2815 from etschannen/feature-timeout-tlog-create
...
Treat a tlog which takes a long time to create its disk queue as failed
2020-03-16 12:49:33 -07:00
72326fe8af
Fix the build.
2020-03-16 12:46:13 -07:00
db5863145a
Merge remote-tracking branch 'upstream/release-6.2' into fdbcli-tlsinfo
2020-03-16 12:33:50 -07:00
a068d4063f
renamed ProxyGetConsistentReadVersion
2020-03-16 12:11:32 -07:00
77dde00da7
Merge pull request #2818 from ajbeamon/increase-metrics-priority
...
Increase priority of the logging of various metrics trace events
2020-03-16 11:57:37 -07:00
ea98c7a40a
added additional timeout on initPersistentState
2020-03-16 11:38:14 -07:00
5f4373c200
Merge pull request #2811 from alexmiller-apple/tls-failures-status
...
Add TLS Policy Failure count to ProcessMetrics and status json
2020-03-16 11:11:30 -07:00
031b579ede
Increase priority of the logging of various metrics trace events.
2020-03-13 16:20:23 -07:00
a5568b2fc6
Rewrite tlsinfo into --debug-tls, and print out configuration.
2020-03-13 15:46:03 -07:00
243c268d9d
Limit the amount of requests the proxy can queue up in memory
2020-03-13 10:17:49 -07:00
04498cbc0e
Make policy failures be reported as per 1s and not over 5s.
2020-03-13 02:49:06 -07:00
75e2fffe5a
Add a ProcessMetrics.TLSPolicyFailures metric
...
This reports the number of policy failures over the past 5s interval.
It also is step 1 towards getting this information into status json.
2020-03-13 02:24:37 -07:00
0c558efcfe
Add a `tlsinfo` command to fdbcli that prints the certificate chain.
...
This requires the certificate chain to load successfully, otherwise
fdbcli will error out at an earlier point due to Net2 not being able to
configure TLS.
2020-03-13 00:11:53 -07:00
5967ef5eab
Added back the changes that report trace log flush failures and fix the random crash
2020-03-12 14:34:19 -07:00
2466749648
Don't disallow allocation tracking when a trace event is open because we now have state trace events. Instead, only block allocation tracking while we are in the middle of allocation tracking already to prevent recursion.
2020-03-12 11:17:49 -07:00
8cdf918316
Add logging when file identifiers don't match
2020-03-12 11:06:53 -07:00
770ef6e726
Add test
2020-03-10 10:42:57 -07:00
027029cc9b
Remove offending overload?
2020-03-10 10:18:14 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
2017daf7d4
Ignore createDirectory error if directory already exists
2020-03-06 16:48:23 -08:00
dbfc0cbcc0
Merge pull request #2781 from alexmiller-apple/certificate-refresh
...
Refresh certificates used for handshaking when they change on disk
2020-03-06 11:12:04 -08:00
f9969a853c
Merge remote-tracking branch 'origin/certificate-refresh' into certificate-refresh
2020-03-06 11:10:05 -08:00
188d9b8239
Don't swallow actor cancellation in certificate refreshing.
2020-03-06 11:09:17 -08:00
9b760fae2d
Rewrite all Errors into tls_errors if they happen as part of initializing TLS.
2020-03-06 11:06:19 -08:00
1f56bf8933
Fix the build with success()
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-03-06 10:15:04 -08:00
ac52b6b474
Rework a bit of error and exception handling.
...
I went back and dug through all of the "what functions can throw what
types", and made sane decisions about them. boost errors are
aggressively translated into FDB ones, whcih might result in multiple
lines of logging about errors, but this is in infrequently run code, so
it should be fine.
2020-03-06 02:33:16 -08:00
39050308ff
lower accept batch size just to be conservative with the change
2020-03-05 18:17:49 -08:00
1128666840
added additional logging on the log router
2020-03-05 18:17:06 -08:00
ccef3f7d05
Attempt to fix TLS_DISABLED compiles.
2020-03-05 17:32:10 -08:00
2d95a1e64d
Implement certificate refreshing
2020-03-05 17:25:33 -08:00
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
39610d15f8
Revert this change since it somehow introduced a random crash detected on circus
2020-03-04 16:14:38 -08:00
2a877bce9a
Merge pull request #2777 from etschannen/feature-accept-batch
...
Accept connections in batches of 20 to improve performance
2020-03-04 16:14:24 -08:00
c73cae0feb
Merge pull request #2760 from ajbeamon/client-version-fixes
...
Improvements to client version reporting
2020-03-04 15:52:49 -08:00
b3c3f8aa5f
Update flow/genericactors.actor.h
...
Pass by reference
2020-03-04 15:35:51 -08:00
7cbabca124
remove printing to stderr from initTLS because that could cause problems on clients
2020-03-04 15:06:22 -08:00
35a1ac6482
prepare net2 for new versions of boost
2020-03-04 14:26:01 -08:00
da579faf62
add missing task priority
2020-03-04 14:25:30 -08:00
820957025f
accept connections in batches of 20 to improve performance
2020-03-04 14:24:57 -08:00
24bbf5a8f0
Avoid invalid read on invalid Void msg
2020-03-02 12:11:43 -08:00
cdbe3117d7
Fix typo
2020-03-02 12:11:43 -08:00
7119b46eb2
Add unit test
2020-03-02 12:11:43 -08:00
c11c24b79d
removed the fdbrpc version of platform.h
2020-02-28 14:56:10 -08:00
e6d36a0aa5
Fix Makefile build
2020-02-28 13:16:58 -08:00
f29d6c3f67
Move implementation of ArenaBlock members to Arena.cpp
2020-02-28 12:33:57 -08:00
6054c05963
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-02-28 12:11:05 -08:00
d1e1fea42d
Our binaries that act like clients (fdbcli, backup and DR binaries) were reporting an unknown client version. Clients did not react if the list of supported versions changed.
2020-02-28 09:35:21 -08:00
13e72f7b3b
Merge pull request #2605 from dongxinEric/fix/1977/report-inability-to-flush-trace-log
...
Report inability to flush trace logs.
2020-02-27 12:36:55 -08:00
16575ae94d
Address review comments
2020-02-27 11:54:15 -08:00
4ac7b36e44
Added back the mutex holder that was removed accidentally
2020-02-27 10:19:17 -08:00
707fc1ddea
only capture the policy to match prior code
2020-02-26 19:04:49 -08:00
c3299b8ebe
if tls cannot be initialized, throw an error from createDatabase
2020-02-26 18:53:06 -08:00
bf5a95e6df
Merge commit 'dc39bdfbbf94a7f470386f439df08c044d08d90c' into feature-tls-environment-vars
...
# Conflicts:
# flow/Net2.actor.cpp
2020-02-26 18:02:56 -08:00
f035bed870
defer initializing TLS to avoid throwing errors from a constructor and so that errors can be logged to the trace file
2020-02-26 17:50:07 -08:00
4bbac9d996
Change a special case return to -1. Update comments to clarify and correct some things.
2020-02-26 16:39:13 -08:00
f85af10a18
fixed a few problems with tls setup
2020-02-26 16:06:45 -08:00
d1598e7c99
set_verify_peers throws an error instead of returning a value
2020-02-26 16:06:16 -08:00
2586bade68
re-added support for configuration TLS options with environment variables
2020-02-26 15:33:48 -08:00
0f5c999d4b
Better containment of boost errors related to TLS.
2020-02-26 12:26:43 -08:00
087c6fa33d
Merge branch 'master' into feature-redwood
2020-02-26 12:25:04 -08:00
74c929d98d
Fix windows build, again
2020-02-26 10:01:08 -08:00
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
Xin Dong