Commit Graph

1220 Commits

Author SHA1 Message Date
Chaoguang Lin 4dbfa01fbf
Add a new robustness workload for testing special keys (#8957)
* Add a new robustness workload for testing special keys

* Fix a few robustness related issues and remove duplicate tests

* Add comments
2022-12-05 14:05:26 -08:00
Vaidas Gasiunas 9826bbce15
Tracking Multi-Version Database Initialization states, failing if client is not compatible to the server (#8868)
* MVC: Keep track of database initialization state; some refactoring of database update for better readability

* Do not create unused temp directory in old binaries downloader

* MVC: Fail transactions with an error code on database initialization errors

* Exit client config tester immediately on errors

* Fail database operations for incompatible clients and database initialization errors

* Some tests for database initialization errors

* local_cluster: reopen fdbmonitor.log in append mode, close it before reopening

* Introduce an option ignore_incompatible_client restoring the behavior of indefinite wait when there is no client matching a cluster version

* Multi-Version API: remove atomics from initializationError

* Update GO bindings: adding option ignore_incompatible_client

* MVC to fail on incompatible clients: addressing review comments

* MVC: An option to fail in case of an incompatible client

* fdb_c_client_config_tester: update help text
2022-12-01 23:28:30 +01:00
sfc-gh-tclinkenbeard c592a26995 Disable log anti quorum for snapshot tests 2022-11-27 13:53:18 -08:00
Vaidas Gasiunas 493ab95464
Define the old FDB release versions used for client testing in one place (#8888)
* Auto-formatting fdb_c_shim_tests.py

* Define the old release versions used for testing in one place; Using PYTHONPATH to find python modules from other project directories

* Remove obsolete upgrade tests; add direct upgrade tests from older versions

* Trigger tests
2022-11-25 13:16:33 +01:00
Nim Wijetunga 97713cadff
Update Encryption Mode DB Config Values (#8839)
* add encryption db config

* address pr comments

* address pr comments

* add comments

* add comment

* modify check

* change condition

* address pr comments

* simplify check

* address pr comments
2022-11-22 16:33:59 -08:00
A.J. Beamon c9b0e0335a
Temporarily disable the default tenant in DifferentClustersameRV (#8895) 2022-11-22 17:35:25 +01:00
Markus Pilman e446394d29
Merge pull request #8898 from sfc-gh-mdvorsky/mdvorsky/avoid-memory-storage-DiskFailureCycle
Avoid memory storage engines in DiskFailureCycle
2022-11-21 13:49:58 -07:00
Marian Dvorsky ccdd644680 Avoid memory storage engines in DiskFailureCycle 2022-11-21 20:19:22 +01:00
Marian Dvorsky f7d1310d0d
Suppress LSAN leak reports from ConnectionReaderActorState (#8889) 2022-11-21 17:58:22 +01:00
Junhyun Shim ce041576f5 Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-17 22:44:31 +01:00
Steve Atherton 8e8c4b4489
Merge pull request #8170 from sfc-gh-sgwydir/ddsketch
Use DDSketch for sample data
2022-11-17 10:38:12 -08:00
Junhyun Shim 7c65ca6648 Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-17 17:48:44 +01:00
Xiaoxi Wang 6b77bcfc35 fix merge conflict upstream/main 2022-11-16 07:34:42 -08:00
Junhyun Shim 632bddd4de Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-16 10:18:34 +01:00
Junhyun Shim bfefbfee8c
Merge pull request #8705 from sfc-gh-jshim/authz-accept-base64-for-jwt-tenant-name
Make token's 'tenants' field base64-encoded (cf. base64url)
2022-11-16 10:17:10 +01:00
Steve Atherton 959f77d03d
Merge pull request #8730 from sfc-gh-xwang/fix/main/restoreStats
Fix the omit StorageWiggler.restoreStats() bug
2022-11-15 22:37:39 -08:00
Xiaoxi Wang 907d7af966 solve merge conflict upstream/main 2022-11-15 14:59:31 -08:00
Junhyun Shim 41ea1678d0 Merge remote-tracking branch 'origin/main' into authz-accept-base64-for-jwt-tenant-name 2022-11-15 22:57:49 +01:00
Jingyu Zhou 37f920a9d2
Merge pull request #8829 from sfc-gh-vgasiunas/vgasiunas-lock-ports-for-tests
Avoid port collisions in ctests
2022-11-15 13:49:51 -08:00
Sam Gwydir 99d4bacf5d Merge remote-tracking branch 'origin/main' into ddsketch 2022-11-15 13:19:42 -08:00
sfc-gh-tclinkenbeard 696c451fa6 Reenable ThroughputQuota.toml test 2022-11-15 12:58:34 -08:00
Vaidas Gasiunas 31b563d9a0 Use gettempdir to determine the system temp directory in local cluster 2022-11-15 19:03:03 +01:00
Vaidas Gasiunas 6086853b46 Use file locking to avoid port collisions in ctests using local cluster 2022-11-15 16:35:08 +01:00
Junhyun Shim bc8f365b04 Merge remote-tracking branch 'origin/main' into authz-accept-base64-for-jwt-tenant-name 2022-11-14 12:51:02 +01:00
Junhyun Shim 7563d59688 Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-14 12:49:07 +01:00
Dennis Zhou 6a86369097
apitester: add better support for blob api commands + tenant abstraction (#8700)
* blob/apitester: rename tenantDebugString() -> debugTenantStr()

* blob/apitester: fix int64_t initialization to -1

* blob/apitester: add missing override for randomOperation()

* blob/cpp: cpp blob tenant operations

* apitester: add tenant to execution context

* apitester: DatabaseOps abstraction

This creates a base abstract class for Database and Tenant operations to
share.

* blob/apitester: add setup phase to verifyRange() prior to operations

This let's us ensure blob storage is setup prior to running apitester
against the cluster.

* apitester: expand execOperation() to take a tenant

Now that db operations can be transparently applied to tenants or
databases through the IDatabaseOps interface, allow execOperation() to
take a tenant accordingly.

* blob/testerapi: blobbify in TesterApiWorkload setup

Before we were blobbifying wholesale through TestRunner. This makes the
change to blobbifying and verifying in setup, so the workload can
properly test database and tenant scope functions.
2022-11-14 10:49:51 +01:00
Nim Wijetunga a1a2839cb3
Revert "Extend Transaction Window during Restore in Simulation" (#8806)
* fix resolver bug

* trigger build
2022-11-12 15:24:30 -08:00
Sam Gwydir 7ea42841a4 Merge remote-tracking branch 'origin/main' into ddsketch 2022-11-12 13:52:57 -08:00
Sam Gwydir 23706c957b Use DDSketch for Sample Data. 2022-11-12 13:45:46 -08:00
Xiaoxi Wang b79268326a Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/dataApi 2022-11-11 08:29:22 -08:00
Ankita Kejriwal 838dfb16c8 Update the simulation testing for storage quota to use/test tenant groups 2022-11-10 19:00:48 -08:00
Vaidas Gasiunas 0160570c2a
Failing client initialization on errors in external clients; Update package version to 7.3.0 (#8719)
* Network setup to fail on the initialization failures of external clients

* MVC: A more intuitive error when failing to load an API function

* Testing initializing FDB client with different configuration options

* Enable strict external client configuration check only for new API versions

* Upgrade FDB package version to 7.3.0; Update upgrade tests
2022-11-10 13:01:41 +01:00
Ankita Kejriwal 105648b888 Merge branch 'main' of github.com:apple/foundationdb into commitproxies 2022-11-09 17:38:30 -08:00
Chaoguang Lin 9a643286cb
Make the hardcoded timeouts as parameters in test files; Also longRunningTest para to disable Simulation timeout restriction (#8765) 2022-11-09 16:57:51 -08:00
Xiaoxi Wang 93daa9ebb7 Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/restoreStats 2022-11-09 16:30:41 -08:00
Nim Wijetunga f1ef9d2671
Fix Bulk Load With Tenants Workload (#8754)
* fix bulk load

* Trigger Build
2022-11-09 13:18:34 -08:00
Xiaoxi Wang 62b88a0772 wait on lock take 2022-11-09 11:34:47 -08:00
Xiaoxi Wang fd425db1cf fix rawStartMovement bugs for merge shard; change the test toml file name 2022-11-08 23:37:58 -08:00
Xiaoxi Wang 4727449ef0 Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/restoreStats 2022-11-08 15:35:15 -08:00
Xiaoxi Wang 317f0e9efe finish workload PerpetualWiggleStatsWorkload 2022-11-08 08:14:21 -08:00
Junhyun Shim 112363ef14 Merge remote-tracking branch 'origin/main' into authz-accept-base64-for-jwt-tenant-name 2022-11-08 13:16:08 +01:00
Junhyun Shim 66e3050062 Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-08 13:15:37 +01:00
Xiaoxi Wang 96cf3f855b add rawMoveShard function 2022-11-07 16:47:14 -08:00
Ankita Kejriwal 3b9431e281 Add testing for rejecting transactions of tenants over storage quota 2022-11-07 16:44:54 -08:00
Xiaoxi Wang fa72f2ef96 fix system access option 2022-11-07 16:13:17 -08:00
Junhyun Shim b64afded69 Parallelize LeakTLogInterface and AuthzSecurity workload 2022-11-08 01:07:26 +01:00
He Liu d28187ca90
Do not set timeout for RocksDB reads in simulation. (#8716)
* Do not set timeout for RocksDB reads in simulation.

* Cleanup.

* Ignore validateStorage test.
2022-11-07 15:59:52 -08:00
He Liu bffa838398
Checkpoint restore with shard (#8667)
* Added SSPhysicalShard.

* Update physicalShards in StorageServer::addShard().

* Handle notAssigned shard.

* fetchKeys() are not stopped during TerminateStorageServer since
physicalShards are not cleared.

* Fixed addingSplitLeft unset shardId.

* Increased the timeout for Rocks reads in simulation.

* Cleanup.

* set SERVE_AUDIT_STORAGE_PARALLELISM to 1.

* Disabled ValidateStorage test.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
2022-11-07 10:02:35 -08:00
Junhyun Shim 1ee1b6ce4d
Merge pull request #8631 from sfc-gh-vgasiunas/vgasiunas-fdbmonitor-in-python
Avoid local cluster tests creating zombie fdbmonitor processes
2022-11-07 14:47:36 +01:00
Vaidas Gasiunas ce3038bfe5
Use a network option for retaining temporary client lib copies instead of a client knob (#8630) 2022-11-07 14:46:17 +01:00