Commit Graph

424 Commits

Author SHA1 Message Date
Andrew Noyes cfd958d8aa
Set FDB_CLUSTER_FILE for command_line_argument_test (#8155)
Previously, you would see the following in command_line_argument_test:

```
Traceback (most recent call last):
  File "/home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/tests/argument_parsing/test_argument_parsing.py", line 113, in <module>
    test_fdbcli(args.build_dir)
  File "/home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/tests/argument_parsing/test_argument_parsing.py", line 79, in test_fdbcli
    check(is_unknown_knob(run_command(command, ["--knob-fake-knob", "foo"])))
  File "/home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/tests/argument_parsing/test_argument_parsing.py", line 30, in check
    assert condition, "Command output:\n" + last_command_output
AssertionError: Command output:
Unable to read cluster file `./fdb.cluster' or `/etc/foundationdb/fdb.cluster' and FDB_CLUSTER_FILE unset: 1515 No cluster file found in current directory or default location
```
2022-09-12 15:35:30 -07:00
sfc-gh-tclinkenbeard 42b3f94afa Add ConfigDBUnitTest.toml 2022-09-11 00:36:18 -07:00
Junhyun Shim 3023096962 Add a knob to allow token-less tenant data access for untrusted clients 2022-09-08 14:53:01 +02:00
Junhyun Shim 6bcfbf421e Merge remote-tracking branch 'origin/main' into authz-general-tls-and-integration-test 2022-08-26 09:20:31 +02:00
Ata E Husain Bohra 00fe4863b6
Implement TenantCacheEntry in-memory cache (#7801)
* Implement TenantCacheEntry in-memory cache

Description

  diff-4: TraceEvent usage improvements 
  diff-3: Address review comments
  diff-2: Add APIs to read counter values, test improvements
  diff-1: Address review comments

Major changes includes:
1. Implements an actor that enables an in-memory caching of
TenantCacheEntry object, allowing the caller to embed custom
information along with TenantCacheEntry.
2. The cache follows read-through cache semantics where the entry
gets loaded from underlying database on a miss.
3. The cache implements a "periodic poller" to refresh known Tenants
by consulting the database. Once a database keyrange-watch feature is
available, cache shall be updated.

Bonus:
Implement a 'recurringAsync' addition to genericActors allowing caller
to schedule a periodic task registering an "actor functor"; the routine
'waits' for the actor unlike existing 'recurring' implementation.

Testing

TenantEntryCache workload
devCorrectnessRun - 100K
2022-08-25 11:42:26 -07:00
Josh Slocum e04e3885b9
Adding BlobRange test (#7868)
* Adding BlobRange test

* refactored blobrange test to use new db functions, and fixed several bugs

* bug fixes for blob manager and verifyBlobRange

* More range unaligned fixes

* cleaning up test and disabling tests that don't work yet for now

* removing overzealous assert in blob manager

* more fixes for overzealous assert

* cleanup and renaming test

* adding chaos to blob ranges test
2022-08-24 11:30:37 -05:00
Junhyun Shim 16f0d53238 Use admin_server as fixture 2022-08-23 15:44:04 +02:00
Junhyun Shim 8397fc3899 Add public key rollover testcase 2022-08-23 14:21:26 +02:00
Junhyun Shim 83a22253a3 Fix multiprocessing and admin server code 2022-08-22 20:55:17 +02:00
Junhyun Shim b72f6e39df Add admin server for proxying management ops 2022-08-22 14:46:47 +02:00
Junhyun Shim 7ac3a92b78 Tidy up test script and add test timeouts 2022-08-19 15:41:12 +02:00
Junhyun Shim 9fc343c442 Use stamp files with authz venv setup
Also fix incorrect bytes-strs usage
2022-08-18 15:43:39 +02:00
Junhyun Shim a312002333 Merge remote-tracking branch 'origin/main' into authz-general-tls-and-integration-test 2022-08-17 17:32:00 +02:00
Junhyun Shim bcb9bc7e62 Set up authz integration test environment using Python venv
Add Authz-related support to {tmp|local}_cluster.py
Also fix public key set load/refresh mechanism
2022-08-17 14:57:05 +02:00
Junhyun Shim a8894232d1 Python virtual env for integration-testing token-based authz 2022-08-16 13:21:44 +02:00
Marian Dvorsky 35a3a33d1c
Resolve TSAN-related issues in ctests (#7864) 2022-08-12 19:23:32 +02:00
He Liu a2dc4e20e9
Store rocksdb::DBOptions and rocksdb::ColumnFamilyOptions to (#7766)
* Store rocksdb::DBOptions and rocksdb::ColumnFamilyOptions to
SharedRocksDBState.

* Make KeyValueStoreRocksDBTest.toml `UNIT`.

* Added TEST_STORAGE_ENGINE_TYPE

* Added readOptions.
2022-08-08 13:38:02 -07:00
Markus Pilman 828d15907e
Merge pull request #7798 from sfc-gh-mdvorsky/mdvorsky/unittests_in_joshua
Add unit tests to the correctness package
2022-08-08 11:23:22 -06:00
Marian Dvorsky e88947cdf6 Remove non-existing unittest 2022-08-05 21:57:22 +02:00
A.J. Beamon ff23d5994e
Merge pull request #7729 from sfc-gh-ajbeamon/feature-metacluster
Metacluster
2022-08-04 15:29:44 -07:00
Josh Slocum b1ff8b8340
Merge pull request #7753 from sfc-gh-jslocum/cf_op_chaos
Add Chaos to Change Feed Operations test
2022-08-02 09:04:38 -05:00
A.J. Beamon 1b693a588a Merge branch 'main' into feature-metacluster 2022-08-01 14:43:14 -07:00
Josh Slocum 83806b4a31 disable movekeys granule test and shift other granule tests around 2022-08-01 14:48:38 -05:00
Andrew Noyes 1558edce8c Python_EXECUTABLE to Python3_EXECUTABLE 2022-07-29 14:57:29 -07:00
A.J. Beamon 8b7b6d1d4c Various cleanup; change some test parameters; add a test for metacluster management operations 2022-07-29 09:24:06 -07:00
A.J. Beamon e8e4f3ad3a Merge branch 'main' into feature-metacluster
# Conflicts:
#	fdbclient/include/fdbclient/Tenant.h
2022-07-28 16:53:29 -07:00
Junhyun Shim c6342a6e5b
Merge branch 'main' into features/authz 2022-07-27 20:51:32 +02:00
A.J. Beamon 7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
Josh Slocum 2190ceb003 Adding chaos to ChangeFeedOperations 2022-07-27 08:34:11 -05:00
Junhyun Shim e2a3fedfc7
Merge branch 'main' into features/authz 2022-07-27 00:08:57 +02:00
Josh Slocum 0609aa57d1 Merge branch 'main' into improved_cf_testing 2022-07-26 16:44:13 -05:00
A.J. Beamon 41c1bef3bf Merge branch 'feature-tenant-groups' into feature-metacluster 2022-07-22 14:26:02 -07:00
He Liu 7a8be255cd
Ss shard management (#7340)
* Storage server shard management with physical shards.

* Cleanup.

* Resolved comments.

* Added `UnlimintedCommitBytes`.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-22 09:30:44 -07:00
Josh Slocum 3f8f44dd16 Merge branch 'main' into improved_cf_testing 2022-07-20 08:19:52 -05:00
A.J. Beamon 3ceb2a0639 Merge branch 'main' into feature-metacluster 2022-07-19 13:04:54 -07:00
Trevor Clinkenbeard 1eec6f993d
Merge pull request #7572 from sfc-gh-akejriwal/tenantquota
Introduce storage quotas per tenant
2022-07-18 20:00:22 -07:00
A.J. Beamon 860d3843cc Merge remote-tracking branch 'origin/feature-tenant-groups' into feature-metacluster 2022-07-16 19:33:26 -07:00
Ata E Husain Bohra 24b2de8de8 BlobFile Encryption and compression support
Description

Testing
2022-07-14 17:04:14 -07:00
Ankita Kejriwal bb05321d24 Introduce storage quotas per tenant.
This change adds:
* ability to store the mapping from tenants to quota in the system keyspace,
* a setter and getter function
* a new workload to test this functionality

FDBCORE-2437
2022-07-14 16:35:12 -07:00
A.J. Beamon cb499fa5db Merge branch 'main' into feature-metacluster 2022-07-13 15:28:34 -07:00
Josh Slocum 142ed7713e Initial version of ChangeFeedOperations workload 2022-07-13 11:16:53 -05:00
Markus Pilman 2edbcf2c65
Merge pull request #44 from apple/main
Merge main
2022-07-12 07:51:22 -06:00
Josh Slocum 9831be866f Adding RandomMoveKeys blob granule cycle test and moving lower value fast/ tests to rare/ 2022-07-11 15:49:55 -05:00
He Liu bc5bfaffda
Shard based move (#6981)
* Shard based move.

* Clean up.

* Clear results on retry in getInitialDataDistribution.

* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-07 20:49:16 -07:00
A.J. Beamon aea4d802c6 Merge branch 'main' into feature-metacluster 2022-06-30 15:08:09 -07:00
sfc-gh-tclinkenbeard 72f5335460 Remove TPCC workloads 2022-06-29 22:31:56 -07:00
A.J. Beamon e1a93988ef Merge branch 'main' into feature-metacluster 2022-06-28 14:58:07 -07:00
Markus Pilman 9d16260822 Merge remote-tracking branch 'origin/main' into features/authz 2022-06-28 13:13:51 -06:00
sfc-gh-tclinkenbeard 086e4bff06 Merge remote-tracking branch 'origin/main' into global-tag-throttling3 2022-06-28 10:18:13 -07:00
He Liu 61d621a4c4
Fix read iterator pool race (#7455) 2022-06-27 14:51:26 -07:00