Commit Graph

186 Commits

Author SHA1 Message Date
Evan Tschannen c11c24b79d removed the fdbrpc version of platform.h 2020-02-28 14:56:10 -08:00
Evan Tschannen 6054c05963 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/fdbserver.actor.cpp
#	versions.target
2020-02-28 12:11:05 -08:00
Evan Tschannen c3299b8ebe if tls cannot be initialized, throw an error from createDatabase 2020-02-26 18:53:06 -08:00
Evan Tschannen bf5a95e6df Merge commit 'dc39bdfbbf94a7f470386f439df08c044d08d90c' into feature-tls-environment-vars
# Conflicts:
#	flow/Net2.actor.cpp
2020-02-26 18:02:56 -08:00
Evan Tschannen d1598e7c99 set_verify_peers throws an error instead of returning a value 2020-02-26 16:06:16 -08:00
Evan Tschannen 2586bade68 re-added support for configuration TLS options with environment variables 2020-02-26 15:33:48 -08:00
A.J. Beamon 0f5c999d4b Better containment of boost errors related to TLS. 2020-02-26 12:26:43 -08:00
Evan Tschannen 96258b9809 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
2020-02-21 19:09:16 -08:00
Alex Miller 927cff3317 Report errors on TLS misconfigurations ... or at least try to. 2020-02-20 16:57:29 -08:00
Evan Tschannen 08c318d28a re-added the connect lock in the fdbcli so that the timeout is not spent before a connection has been initiated (because of the handshake lock) 2020-02-20 10:43:34 -08:00
Evan Tschannen 761da5a059 code cleanup 2020-02-19 17:59:45 -08:00
Alex Miller 88d36af9c7 Fix --tls_password and add better error logging
This refactors all tls settings into a TLSParams object so that we can
set the password before loading any certificates.

It turns out that the FDBLibTLS code did really nice things with error
logging, but I just didn't understand openssl enough before to realize
what pieces I should be copying.
2020-02-19 00:57:05 -08:00
Meng Xu 31a6ec34b7 Merge branch 'master' into mengxu/fast-restore-agent-PR 2020-02-18 16:17:59 -08:00
Markus Pilman e71fe44ee3
Merge branch 'master' into features/icc 2020-02-08 21:33:02 -08:00
Alex Miller e390dbd36c Add a non-FDBLibTLS verify peers framework to new TLS impl 2020-02-06 21:06:52 -08:00
Evan Tschannen 38d8d0d675 fixed simulation 2020-02-06 19:29:31 -08:00
Evan Tschannen 844c8511c4
Merge pull request #2588 from jzhou77/backup-worker
Integrate new backup worker with existing backup command
2020-02-05 14:14:43 -08:00
Evan Tschannen 84853dd1fd switched SSL implementation to use boost ssl 2020-02-04 14:56:40 -08:00
mpilman d09e07f1f5 Merge remote-tracking branch 'upstream/master' into features/icc 2020-02-04 10:26:18 -08:00
Jingyu Zhou 7c10683c77 Backup workers save logs into right containers
The mutation logs of backup workers are saved into "mlogs" directory under the
container directory. The backup worker has been restructured to handle multiple
backups, where each one is stored in a separate backup container.

In the backup worker, mutations pulled from TLogs are buffered in a message
queue. When writing out to different containers, their corresponding mutation
ranges are used to check if a mutation should be written. When a new backup
is submitted by the client, "backupStartedKey" is updated. The worker monitors
this key, updates its internal map of backups, and then next pull from TLog
needs to wait for the readiness of the new backup. This is to ensure when
worker 0 sets the backup is started, all workers have already been logging
mutations for the backup.
2020-02-03 20:27:14 -08:00
Meng Xu 3b57bf1781 Merge branch 'master' into mengxu/fast-restore-agent-PR 2020-02-03 17:23:54 -08:00
Evan Tschannen 4524831456
Merge pull request #2518 from vishesh/task/failmon-remove-server
FailureMonitoring: Server processes no longer need to talk to ClusterController
2020-02-03 17:22:50 -08:00
Meng Xu 9c2046b11b FastRestore:Minic fdbd to monitor coordintors
Before we start a fdb restore process.
2020-02-03 14:48:31 -08:00
Meng Xu 559b95c61a FastRestore:RestoreRole:Mimic how fdbd starts 2020-02-01 10:23:48 -08:00
Meng Xu 8ef56e5cae FastRestore:LeaderElection:Log election progress 2020-01-30 20:03:14 -08:00
Meng Xu ff92401ed5 FastRestore:Add FastRestoreClass and blob option
To simplify test in circus framework, we need a fastrestore class;
To get data from blob in real mode, restore worker should set up
blob credentials in order to call BackupContainer interface to
get all backup files.
2020-01-28 20:25:05 -08:00
Jingyu Zhou 31a1106286 Save mutations to backup files in simulation
This is the first step in the new backup's data pipeline. Verification of file
content is needed in future commits. A clear documentation of file format is a
work in progress.
2020-01-22 19:38:45 -08:00
Vishesh Yadav daef5f011a Merge remote-tracking branch 'apple/master' into task/failmon-remove-server 2020-01-21 13:20:15 -08:00
Evan Tschannen 3f9d9d8b84 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	cmake/FlowCommands.cmake
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/StorageServerInterface.h
#	fdbserver/DataDistributionTracker.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/Knobs.h
#	flow/Platform.cpp
#	versions.target
2020-01-16 18:37:47 -08:00
Evan Tschannen 17e97f24e4
Merge pull request #2526 from etschannen/feature-dd-improvements
Data distribution improvements
2020-01-10 17:53:22 -08:00
Evan Tschannen e4fa4ad0c9 Data distribution will not merge a shard unless it has been low bandwidth for 5 minutes 2020-01-09 17:02:49 -08:00
Vishesh Yadav 6e6cfaff16 Cleanup old Failure Monitoring code 2020-01-07 15:53:32 -08:00
A.J. Beamon 9866d1ce27 Revert change to make g_trace_clock thread_local, instead checking we are on the correct thread when getting the time. 2019-12-06 10:15:49 -08:00
A.J. Beamon ed8d3f163c Rename hgVersion to sourceVersion. 2019-11-15 12:26:51 -08:00
Andrew Noyes de8921b660 Move RestoreWorkerInterface to fdbclient 2019-10-25 10:42:22 -07:00
Andrew Noyes d4de608bb6 Fix OPEN_FOR_IDE build 2019-10-25 10:42:22 -07:00
tclinken bb0ae31002 Removed dead code. 2019-10-18 17:06:48 -07:00
A.J. Beamon 3ba8fd95b5 Add script to parse output from enabling ALLOC_INSTRUMENTATION_STDOUT 2019-10-08 15:50:47 -07:00
Meng Xu e26270b17f Fix compilation errors after merge 6.2 2019-10-02 13:28:24 -07:00
Meng Xu d0147e5e5d Merge branch 'release-6.2' into mengxu/merge-release620-to-master-v3
Resolved Conflicts:
	documentation/sphinx/source/release-notes.rst
	fdbserver/DataDistribution.actor.cpp
	versions.target
2019-10-02 13:22:56 -07:00
Alex Miller 9a64dc7f24
Merge pull request #1792 from ajbeamon/remove-dl-iterate-phdr-hack
Remove dl_iterate_phdr caching used by slow task profiler
2019-09-17 16:51:30 -07:00
Markus Pilman eb29541acb
Update fdbserver/fdbserver.actor.cpp
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-09-13 15:10:24 -07:00
mpilman 8875fe7aac Don't require at leats 2GiB over cache_memory 2019-09-13 13:02:06 -07:00
mpilman 8c1c0296ed Verify that we have enough memory if cache_memory is set 2019-09-13 12:54:37 -07:00
Meng Xu 048c341a7d FastRestore:Bug fix after merge with master
Include RestoreWorkerInterface.h instead of RestoreInterface.h into fdbserver.actor.cpp
Report warning instead of error when unlockDatabase throws error.
2019-09-04 21:14:09 -07:00
Evan Tschannen 24aad14f06 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	documentation/sphinx/source/release-notes.rst
#	versions.target
2019-08-30 17:23:58 -07:00
A.J. Beamon 45373bd04c Attempt to fix some merge madness 2019-08-22 10:17:15 -07:00
A.J. Beamon e465cbef07 Merge branch 'master' into remove-dl-iterate-phdr-hack
# Conflicts:
#	fdbserver/fdbserver.actor.cpp
2019-08-22 10:13:23 -07:00
A.J. Beamon f02799455e Add --loggroup to fdbserver and fdbbackup help text. 2019-08-19 12:59:14 -07:00
Andrew Noyes 2a9eeddf5a Sort roles 2019-08-09 13:10:12 -07:00