Commit Graph

886 Commits

Author SHA1 Message Date
Vaidas Gasiunas 9921a9bd02 Upgrade Tests: Avoid corrupting downloaded files in case of parallel test execution 2022-04-25 15:02:38 +02:00
Vaidas Gasiunas 0a9b2470da Upgrade Tests: Check the hash sum of the dowloaded files, retry the download on errors 2022-04-22 16:37:11 +02:00
Vaidas Gasiunas 9897712307 Merge remote-tracking branch 'apple/main' into vgasiunas-client-tmp-dir 2022-04-22 13:00:09 +02:00
Vaidas Gasiunas f9cb9deb1d Use build-local temp directory for C API tests 2022-04-22 11:57:29 +02:00
Vaidas Gasiunas b4fa9e23f6 Upgrade Tests: Use build local tmp directory for temp client library copies 2022-04-21 19:27:19 +02:00
Josh Slocum d26db64f08 only adding bg_url knob to local cluster config if blob granules enabled 2022-04-21 10:47:36 -05:00
Josh Slocum 3c036300cc Addressing review comments 2022-04-20 14:34:19 -05:00
Josh Slocum 40052c1394 Added Blob Granule API Tester 2022-04-20 10:15:43 -05:00
Josh Slocum 8fa25aa013 Adding blob granule client test target and local cluster setup 2022-04-20 10:15:43 -05:00
Vaidas Gasiunas 3372b3a6eb Upgrade Tests: Check logs for warnings and dump them 2022-04-19 13:37:21 +02:00
Vaidas Gasiunas 79b2ebeb4b Upgrade Tests: Test upgrade from 7.0 to 7.2 2022-04-19 11:53:33 +02:00
A.J. Beamon 19d78cf2a3 When clearing the database between tests, check that clearing the tenant left the entire normal key-space empty. Update the configuration of some tests. Disable a special key-space test that is invoking broken behavior. 2022-04-14 11:39:02 -07:00
Vaidas Gasiunas a03ba928e4 Upgrade Tests: Introduce transaction retry limit; Trace long retry chains; An option to run the test binary with gdb 2022-04-13 16:50:02 +02:00
Vaidas Gasiunas 871cdece0e Merge remote-tracking branch 'apple/main' into vgasiunas-upgrade-test 2022-04-13 14:17:46 +02:00
Vaidas Gasiunas f7636ecf3b Upgrade Tests: Addressing review comments 2022-04-13 14:17:23 +02:00
Ray Jenkins 327c1da0a0 Remove to_7.2.0 tests from CMakeLists.txt 2022-04-12 17:06:18 -07:00
Ray Jenkins f07fa29105 move ConfigureStorageMigrationTestRestarts to to_7.1.0 directory and remove to_7.2.0 2022-04-12 17:06:18 -07:00
Ray Jenkins 3f61f869e6 fix to_7.x tests 2022-04-12 17:06:18 -07:00
Vaidas Gasiunas 9e46128b6f Upgrade Test: Include only currently working tests into the ctest suite 2022-04-12 20:03:29 +02:00
Vaidas Gasiunas 73ed1c6e97 Upgrade Tests: update for 7.2 2022-04-12 14:06:20 +02:00
Vaidas Gasiunas 580010e64c Merge remote-tracking branch 'apple/main' into vgasiunas-upgrade-test 2022-04-12 13:43:36 +02:00
Vaidas Gasiunas ca563466a6
Merge pull request #6401 from sfc-gh-mpilman/features/private-request-streams
Features/private request streams
2022-04-11 18:29:06 +02:00
Ata E Husain Bohra 933e5bbd2e
EncryptKeyProxy server APIs for simulation runs. (#6727)
* EncryptKeyProxy server APIs for simulation runs.

Description

  diff-2: FlowSingleton util class
              Bug fixes
  diff-1: Expected errors returned to the caller

Major changes proposed are:
1. EncryptKeyProxy server APIs:
 1.1. Lookup Cipher details via BaseCipherId
 1.2. Lookup latest Cipher details via encryption domainId.
2. EncyrptKeyProxy implements caches indexed by: baseCipherId &
   encyrptDomainId
3. Periodic task to refresh domainId indexed cache to support
   'limiting cipher lifetime' abilities if supported by
   external KMS solutions.

Testing

EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-11 09:08:42 -07:00
Markus Pilman 16467262f0 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-10 14:12:37 -06:00
Vaidas Gasiunas f91acd323c Merge remote-tracking branch 'apple/main' into vgasiunas-upgrade-test 2022-04-08 20:27:45 +02:00
Vaidas Gasiunas acceebd0b5 Upgrade Test: Fix reporting of errors in logs 2022-04-08 20:27:26 +02:00
Xiaoxi Wang 1eb1cff45b
Merge pull request #6806 from sfc-gh-xwang/fix-conf-restart
Enforce storage migration compatible config in restarting test
2022-04-08 11:15:31 -07:00
Steve Atherton 11a5d14a11
Merge pull request #6108 from sfc-gh-satherton/redwood-header-changes
Redwood page format refactor to support format evolution, forensic analysis and future encryption scheme
2022-04-08 10:59:12 -07:00
Vaidas Gasiunas b544adad12 Merge remote-tracking branch 'apple/main' into vgasiunas-upgrade-test 2022-04-08 15:01:29 +02:00
Xiaoxi Wang 74d81f90e1 enforce storage migration random config 2022-04-08 00:28:05 -07:00
Markus Pilman bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
Vaidas Gasiunas ac48e5000e Upgrade Test: Include upgrade from 7.0 in ctest 2022-04-07 19:32:57 +02:00
Chaoguang Lin f62904187e Disable remote kvs if RocksDB is used 2022-04-06 17:44:20 -07:00
Vaidas Gasiunas ef0ec3d893 Merge remote-tracking branch 'apple/main' into vgasiunas-upgrade-test 2022-04-06 12:01:52 +02:00
Steve Atherton da339ed813 Disable Redwood for upgrades from 7.0 or downgrades to 7.0. 2022-04-03 02:37:37 -07:00
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
He Liu dd15489605 rename ssd-rocksdb-experimental as ssd-rocksdb-v1. 2022-03-29 10:53:38 -07:00
sfc-gh-tclinkenbeard 77786f4fc6 Merge remote-tracking branch 'origin/main' into change-data-hall 2022-03-27 12:44:05 -07:00
Josh Slocum fd6c9544e2 Disabling rocks in blob granule tests 2022-03-24 20:40:16 -05:00
Josh Slocum 3cd1e5599e Cleanup 2022-03-24 17:15:11 -05:00
Josh Slocum 1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
Vaidas Gasiunas 68a9abd3c5 Upgrade Test: Derive API version from the versions tested; Improve handing of error situations 2022-03-24 17:23:17 +01:00
Vaidas Gasiunas c63a1a18dd Upgrade Tests: Check workload progress after each upgrade step 2022-03-24 12:36:22 +01:00
Vaidas Gasiunas c1f4edbfe9 Upgrade Tests: Fix port assignment in local_cluster.py 2022-03-23 09:42:49 +01:00
Vaidas Gasiunas 6f8b410a8a Upgrade Tests: Run API workloads until stop commands; Initial ctest for upgrade 2022-03-22 20:30:06 +01:00
Vaidas Gasiunas 5bcc173044 Merge remote-tracking branch 'apple/main' into vgasiunas-upgrade-test 2022-03-22 12:17:28 +01:00
Vaidas Gasiunas 0f9ebaae37 Upgrade Tests: Scripting download of old libraries and cluster upgrades; Specifying external library directory in C API Tester 2022-03-22 12:14:01 +01:00
Markus Pilman 35f7843d84 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-21 12:33:53 +01:00
Josh Slocum 37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
A.J. Beamon 74487310fa Fix a couple test specification errors 2022-03-17 12:10:19 -07:00