Commit Graph

54 Commits

Author SHA1 Message Date
Markus Pilman bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
Markus Pilman 117ee637db Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-15 17:17:47 +01:00
Mohamed Oulmahdi 9e81945301 Enable encryption for Windows 2022-03-11 10:06:07 +01:00
Markus Pilman dc973fb67e Allow List and first test 2022-02-22 11:15:16 +01:00
Andrew Noyes 23de546586 Only build libcoro if COROUTINE_IMPL == libcoro
It doesn't build at all on mac m1
2021-12-03 16:34:35 -08:00
Andrew Noyes b6fd402a3c Add option to use boost or libcoro
By default, use boost everywhere except windows and linux x86 (for
performance reasons)
2021-11-29 13:14:15 -08:00
sfc-gh-tclinkenbeard e918d20f68 Document "averaged" parameter of PerfMetric constructor 2021-08-29 15:38:21 -07:00
Mohamed Oulmahdi c1bb977a93 Set sampling dependencies for Windows only 2021-08-24 10:48:31 +02:00
Mohamed Oulmahdi 2c5510011c Add dependencies between sampling and non-sampling actors projects 2021-08-24 10:48:31 +02:00
Lukas Joswiak 59d535149e Merge branch 'master' into fixes/alp6 2021-07-27 10:07:18 -07:00
Lukas Joswiak e9a1679467 Disable sampling everywhere except fdbserver 2021-07-27 09:53:23 -07:00
sfc-gh-tclinkenbeard ad03a4787a Fix non-TLS build 2021-07-09 21:06:15 -07:00
sfc-gh-tclinkenbeard 81b8292094 Merge remote-tracking branch 'origin' into encrypt-backup-files 2021-06-25 12:21:12 -07:00
Evan Tschannen 1cec223434 cherry pick streaming replies to master 2021-06-19 09:47:13 -07:00
Josh Slocum ce82c9653e Testing Storage Server implementation 2021-05-25 20:28:50 +00:00
sfc-gh-tclinkenbeard a0c49234b2 Merge remote-tracking branch 'origin/master' into encrypt-backup-files 2021-03-19 20:47:53 -07:00
Markus Pilman d13e240496 use libcoroutine for Windows
The Windows CI still doesn't have boost context installed
so we use the old implementation for now
2021-02-17 09:23:41 -07:00
Markus Pilman 534fa45291 Merge branch 'master' of github.com:apple/foundationdb into features/coroutine2 2021-02-01 16:31:16 -07:00
sfc-gh-tclinkenbeard 511ec43e62 Merge remote-tracking branch 'origin/master' into encrypt-backup-files 2021-01-27 11:17:58 -08:00
Markus Pilman d3a85b555a CoroFlow and remove libcoro 2021-01-26 17:23:17 -07:00
sfc-gh-tclinkenbeard 1612c44988 Disable StreamCipher for Windows 2021-01-25 12:36:35 -08:00
sfc-gh-tclinkenbeard 037279c843 Disable StreamCipher if TLS is disabled 2021-01-25 10:42:14 -08:00
sfc-gh-tclinkenbeard 88bc157bd0 Added AsyncFileEncrypted 2021-01-24 22:53:04 -08:00
sfc-gh-tclinkenbeard 61a29ecfc8 Merge remote-tracking branch 'origin/master' into run-minio-joshua 2021-01-01 09:38:36 -04:00
sfc-gh-tclinkenbeard fed1a03950 Added SimExternalConnection (most methods not implemented yet) 2020-12-24 21:20:02 -04:00
Markus Pilman 390a82b6a2
Fix old typo
Co-authored-by: Andrew Noyes <63815641+sfc-gh-anoyes@users.noreply.github.com>
2020-12-09 13:51:50 -07:00
Markus Pilman 9344a5fd0a Fix Coro ucontext bug introduced in #4109 2020-12-09 12:25:31 -07:00
David Youngworth 3192984309 Remove unused zlib library 2020-08-11 17:15:07 -07:00
Meng Xu ef8c1060a2 Merge branch 'master' into mengxu/tmp-merge-6.3 2020-07-13 10:15:56 -07:00
A.J. Beamon b09dddc07e Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
# Conflicts:
#	cmake/ConfigureCompiler.cmake
#	documentation/sphinx/source/downloads.rst
#	fdbrpc/FlowTransport.actor.cpp
#	fdbrpc/fdbrpc.vcxproj
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogSystemPeekCursor.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/Status.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
A.J. Beamon 04d1217941 Track statistics about server-side request latency on each process, to include min, max, mean, and various percentiles. 2020-07-09 16:39:15 -07:00
Jingyu Zhou d972e072b7 Remove unused batcher.actor.h 2020-07-07 21:54:56 -07:00
Dave Cottlehuber 9753013b65 fdbrpc: update cmake & headers 2020-04-30 20:11:43 +00:00
Vishesh Yadav d90e168e24 Add HealthMonitoring skeleton code 2020-04-03 15:53:00 -07:00
Evan Tschannen c11c24b79d removed the fdbrpc version of platform.h 2020-02-28 14:56:10 -08:00
Evan Tschannen 96258b9809 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
2020-02-21 19:09:16 -08:00
Evan Tschannen 761da5a059 code cleanup 2020-02-19 17:59:45 -08:00
Alex Miller da73164eda Move crc32c from fdbrpc to flow
So that we can use it from a piece of flow code without breaking module
boundaries.

Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
2020-01-13 18:19:30 -08:00
Alex Miller 3b9678356e Make FDBLibTLS and thirdparty static libraries.
They're statically linked anyway, and this fixes an issue with CMake
complaining that there are cyclic dependencies that are non-static.
2019-09-30 18:32:24 -07:00
Andrew Noyes a8cdcff0c2 Change --disable-actor-without-wait-warning to --disable-diagnostics
We probably just want to disable all actor diagnostics for the flow test
files.

Also add --generate-probes to the help text
2019-08-16 09:24:57 -07:00
A.J. Beamon 863204a29d Update names in CMakeLists, vcxproj 2019-08-01 08:48:25 -07:00
mpilman 7d247af500 Two minor bug fixes from recent optimizations 2019-07-31 19:14:11 -07:00
mpilman 4c62458172 Make valgrind work on Fedora 30 2019-06-11 10:27:08 -07:00
Andrew Noyes 247f95a6e2 Add -Wunused-variable 2019-04-15 18:13:00 -07:00
Andrew Noyes 6207d724f8 Fix all -Wunused-variable warnings 2019-04-15 18:13:00 -07:00
mpilman 75f692b931 simplify actorcompiler and target to compile coveragetool 2019-02-15 00:01:42 -08:00
Andrew Noyes 3a38bff8ee Use DISABLE_ACTOR_WITHOUT_WAIT_WARNING consistently 2019-02-13 10:30:35 -08:00
Andrew Noyes 874a58cb4f Suppress actor without wait for tests in cmake 2019-02-12 11:01:17 -08:00
mpilman 8a94d80deb fdbservice and fdbrpc now compiling 2019-02-07 15:37:04 -08:00