Commit Graph

397 Commits

Author SHA1 Message Date
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
He Liu dd15489605 rename ssd-rocksdb-experimental as ssd-rocksdb-v1. 2022-03-29 10:53:38 -07:00
sfc-gh-tclinkenbeard 77786f4fc6 Merge remote-tracking branch 'origin/main' into change-data-hall 2022-03-27 12:44:05 -07:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Josh Slocum bf9bf19797 Disabling tenants for BlobGranule tests for now 2022-03-18 10:30:54 -05:00
Josh Slocum 37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
A.J. Beamon ee70837970 Remove some debugging lines, change simulated default cluster frequency, and use a different mechanism to disable tenants for the change feed test. 2022-03-17 12:10:19 -07:00
A.J. Beamon 9e39ebd069 Use the new enum name OPTIONAL_TENANT 2022-03-17 12:10:19 -07:00
A.J. Beamon d0dc756c6d Allow disabling tenant mode in simulation. Fix a few bugs. 2022-03-17 12:10:18 -07:00
A.J. Beamon 81e8c7c362 Various test fixes to work with tenants. 2022-03-17 12:10:18 -07:00
A.J. Beamon 4b521b38cb Update various workloads to specify that they are expecting raw access. A couple other raw access related fixes. Disable tenant tests in backup and DR tests for now. 2022-03-17 12:10:18 -07:00
A.J. Beamon 05495908b8 Implement some tenant tests 2022-03-17 12:10:18 -07:00
Steve Atherton 59762fc784
Merge pull request #6423 from sfc-gh-satherton/redwood-bug-fixes-and-memory-leak
Redwood memory usage and small memory leak fixes
2022-03-15 16:52:28 -07:00
Yao Xiao 09df9d83fd
Enable RocksDB with Valgrind. (#6595) 2022-03-14 21:11:44 -07:00
Josh Slocum 4d909abbd1 re-enabling all storage engine types 2022-03-09 09:21:45 -06:00
Josh Slocum e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
A.J. Beamon 5fa9d3e1b7 Add a tenant parameter to read and commit requests. Store a map of all tenants on commit proxy and storage servers. Add an option to require tenant mode. 2022-03-06 21:54:21 -08:00
Steve Atherton f03c0b8c3c Added ISimulated::restarted for detecting a restarted simulation test. 2022-03-04 17:19:46 -08:00
sfc-gh-tclinkenbeard 59794ffcb2 Add randomlyRenameZoneId test parameter 2022-03-01 00:42:24 -08:00
sfc-gh-tclinkenbeard db8def68db Use std::unique_ptr for ISimulator::extraDB 2022-02-28 13:12:31 -08:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Renxuan Wang 481587a8c6 Turn on hostname logic. 2022-02-22 16:29:59 -08:00
Josh Slocum 38a75a8b89 Merge branch 'main' into blob_integration 2022-02-17 17:47:38 -06:00
sfc-gh-tclinkenbeard 5b8f4d9be1 Move SimpleIni.h to fdbclient 2022-02-07 13:31:04 -08:00
Josh Slocum 4d34ac9545 disabling redwood and memory storage engines for tests for now 2022-02-07 08:15:45 -06:00
Renxuan Wang 334ba66348 Add the support for using hostname in simulation.
This PR does NOT turn the feature on. Also, disable hostname in downgrade restarting tests.
2022-01-21 10:48:33 -08:00
Suraj Gupta 968a4f9f50 Don't rely on database config to be updated. 2021-12-10 14:00:34 -06:00
Suraj Gupta fc3376fe8f Move client knob to database config for blob granules. 2021-12-10 14:00:34 -06:00
Josh Slocum efb21ca6a1 Merge branch 'master' into blob_integration 2021-11-24 13:17:54 -06:00
sfc-gh-tclinkenbeard 766a05d33c Merge remote-tracking branch 'origin/master' into add-format-warning 2021-11-17 12:14:01 -08:00
Renxuan Wang 725d31e264 Fix parameters passes to simulatedMachine(). 2021-11-16 17:50:12 -08:00
Steve Atherton c53f5aa110 Renamed redwood to redwood-1-experimental and file extension to .redwood-v1. 2021-11-16 02:15:22 -08:00
Josh Slocum f140028086 Merge branch 'master' into blob_integration 2021-11-15 15:09:40 -06:00
sfc-gh-tclinkenbeard 62efeb6812 Merge remote-tracking branch 'origin/master' into add-format-warning 2021-11-12 11:50:36 -08:00
Josh Slocum ba53045fcd Fixing BW count for multi-process machines and having BW monitor its own removal since it doesn't get private mutations 2021-11-11 12:33:08 -06:00
sfc-gh-tclinkenbeard 25257f6f87 Enable unused-function warning for clang 2021-11-01 14:18:31 -07:00
sfc-gh-tclinkenbeard 13bb7838aa Enable clang -Wformat warning 2021-10-30 21:07:38 -07:00
Josh Slocum 19076ad4d2 Changed blob manager checkin to by async and to have blob worker speculatively consume change feed ahead of re-snapshot 2021-10-27 16:23:51 -05:00
Josh Slocum 7d25dab96b Re-enabling blob granules for testing 2021-10-26 12:11:37 -05:00
Josh Slocum 0ff8ddc2b6 Merge branch 'master' into blob_full_clean 2021-10-25 13:38:48 -05:00
A.J. Beamon e882eb33fc Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem. 2021-10-22 11:05:18 -07:00
Josh Slocum 912ef76f1c cleanup before merge 2021-10-18 17:11:14 -05:00
Josh Slocum 5f0ec0612a Merge branch 'feature-range-feed' into blob_full 2021-10-13 15:44:35 -05:00
Suraj Gupta 2ec8781224 Merge knobs into one. 2021-10-13 14:00:37 -04:00
Suraj Gupta 5a6a052c55 Add a knob to gate blob-related work. 2021-10-13 09:48:02 -04:00
Neethu Haneesha Bingi 3ea7209013 Simulation changes to randomly wiggle with locality filter and review comments. 2021-09-30 10:00:33 -07:00
Neethu Haneesha Bingi 3e79299898 Locality filter support to perpetual storage wiggler feature. 2021-09-30 10:00:33 -07:00
Suraj Gupta ac14856307 Add BWs to every DC in simulated cluster. 2021-09-27 11:17:37 -04:00
Suraj Gupta 0ca00ce53b Address PR comments.
Add support for more than 1 blob worker in simulated cluster.
We now have at least 2, and up to 7 (randomly).

Also removed the ability for blob workers to onboard onto
unset processes.
2021-09-24 12:17:56 -04:00