Commit Graph

1090 Commits

Author SHA1 Message Date
Andrew Noyes 27e0a16cdb Remove testClass from restart tests
Old binaries don't understand the testClass field, so we can't have it
in test files old binaries might read. In the future we can have it for
7.2+
2022-10-05 15:49:24 -07:00
A.J. Beamon 53593c77ac When updating watches after a database switch, lookup the tenant again to get the ID on the new cluster. Update the switching test to set up tenants properly. 2022-10-03 09:24:13 -07:00
Ata E Husain Bohra 03f1d13be3
Enable encryption authentication configurability (#8312)
* Enable encryption authentication configurability

Description

 diff-1: Remove memcpy due to auth-token computation
         Address review comments

Patch proposes major changes:
1. Enable FDB to choose encryption authentication as a configurable
parameter. Fix issues choosing ENCRYPT_HEADER_AUTH_TOKEN_NONE mode.
2. Introduce AES_CMAC as supported encryption authentication scheme.

Patch allows cluster to govern: if encryption authentication needs to
enabled, if yes, then choose from two supported schemes:
1. HMAC_SHA_256
2. AES_256_CMAC

Testing

devRunCorrectness - 100K
BlobCipher unittests
EncryptionOps.toml
BlobGranuleCorrectness/BlobGranuleCorrectnessClean
2022-09-29 16:18:55 -07:00
Nim Wijetunga 232e23c87e
Snapshot Backup Encryption (#8095)
* inital commit

* add encryption support

* refactor passing encryption keys

* disable encryption codepath

* pass enable encryption cli arg for fdbbackup

* change encryption key passing

* only enable encryption for backup if tenant mode is required

* revert server knobs

* ignore json trace files

* --amend

* integrate ekp

* add comments

* comments

* comments

* revert knobs

* fix bug

* modify encrypted backup logic

* use common prefix

* check boundaries between begin and end key

* return system encrypt domain

* modify

* add tenant entry cache to backup

* slight revert

* format

* Trigger Build

* format

* use tenant name to fetch keys

* Trigger Build

* Trigger Build

* merge

* fix getting ekp

* fix bug

* address pr comments

* Trigger Build

* Trigger Build

* remove faulty check
2022-09-29 14:45:47 -07:00
Xiaoxi Wang 3578832c69 merge upstream/main 2022-09-28 14:01:50 -07:00
A.J. Beamon c8bb15e8ee
Merge pull request #8216 from sfc-gh-ajbeamon/backup-support-for-tenants
Include tenant and metacluster metadata in backups
2022-09-28 12:00:24 -07:00
A.J. Beamon 3353103d9d Fix filtering of potential backup mutations in commit proxy and backup worker; add code probe to ensure we are testing default backup sharing and add some tests to hit it 2022-09-27 15:25:30 -07:00
Xiaoxi Wang e476e814b2 merge upstream/main 2022-09-27 10:15:26 -07:00
sfc-gh-tclinkenbeard 7fc5c196c4 Make read and write quotas fungible 2022-09-25 21:00:11 -07:00
Xiaoxi Wang 050bf22e1e add comments to test toml and moveKeys to non-const method 2022-09-22 13:11:53 -07:00
A.J. Beamon fda0d7223d Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants. 2022-09-22 10:00:13 -07:00
Junhyun Shim 25d551d2f7 Disable authz integration test for sanitizer builds
Language bindings do not work well with sanitizer instrumentation
2022-09-22 18:30:42 +02:00
Chaoguang Lin 9628561235
Add DataDistributionMetrics workload into correctness packages, (#8237)
which makes the code probe hit in nightly tests.
2022-09-20 15:33:15 -07:00
Xiaoxi Wang 1f1a66be39 merge upstream/main 2022-09-14 12:32:28 -07:00
Xiaoxi Wang 46eadd53cf finish getInitialDataDistribution test verification 2022-09-13 22:43:15 -07:00
Lukas Joswiak 9fd22546ea Add workload to test coordinator changes with dynamic knobs 2022-09-13 16:53:54 -07:00
Lukas Joswiak 74ac617a34 Add support for changing coordinators to the configuration database
Configuration database data lives on the coordinators. When a change
coordinators command is issued, the data must be sent to the new
coordinators to keep the database consistent.
2022-09-13 16:53:54 -07:00
Xiaoxi Wang eaff46cd27 IDDTxnProcessorApiWorkload setup - Get Initial DD in real cluster 2022-09-13 11:04:23 -07:00
Dennis Zhou d6d69379a8 ApiVersion: tenant specific guards 2022-09-12 17:08:56 -07:00
Andrew Noyes cfd958d8aa
Set FDB_CLUSTER_FILE for command_line_argument_test (#8155)
Previously, you would see the following in command_line_argument_test:

```
Traceback (most recent call last):
  File "/home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/tests/argument_parsing/test_argument_parsing.py", line 113, in <module>
    test_fdbcli(args.build_dir)
  File "/home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/tests/argument_parsing/test_argument_parsing.py", line 79, in test_fdbcli
    check(is_unknown_knob(run_command(command, ["--knob-fake-knob", "foo"])))
  File "/home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/tests/argument_parsing/test_argument_parsing.py", line 30, in check
    assert condition, "Command output:\n" + last_command_output
AssertionError: Command output:
Unable to read cluster file `./fdb.cluster' or `/etc/foundationdb/fdb.cluster' and FDB_CLUSTER_FILE unset: 1515 No cluster file found in current directory or default location
```
2022-09-12 15:35:30 -07:00
Markus Pilman 59ce49913a
Merge pull request #8146 from sfc-gh-tclinkenbeard/improve-code-coverage
Increase the number of unit tests run in `RandomUnitTests.toml`
2022-09-12 15:10:47 -06:00
Jingyu Zhou f530ffdac0
Merge pull request #8148 from sfc-gh-tclinkenbeard/improve-restart-code-coverage
Fix configuration section of ConfigureTestRestart-2.toml test file
2022-09-12 09:31:18 -07:00
sfc-gh-tclinkenbeard 42b3f94afa Add ConfigDBUnitTest.toml 2022-09-11 00:36:18 -07:00
sfc-gh-tclinkenbeard 924c198a5b Run 10 unit tests within RandomUnitTests.toml 2022-09-11 00:36:18 -07:00
sfc-gh-tclinkenbeard 8d988bb1fb Fix configuration section of ConfigureTestRestart-2.toml test file 2022-09-11 00:18:39 -07:00
Vaidas Gasiunas 81fff640bd
Testing with invalid cluster files, fixing update from changed cluster file (#8126)
* ApiTester: test with invalid cluster files

* More asserts in monitorProxies

* ApiTester: Test tampering the cluster file

* Fix update of connection string from the cluster file to use the new connection string only if it valid

* ApiTester: add linker dependency on std++fs

* upgrade_test: no-cleanup-on-error option

* ApiTester: use atomic operations to change and access the transaction handle
2022-09-10 09:23:00 +02:00
Jon Fu 75a096a5e5 Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-special-key-space 2022-09-09 10:12:19 -07:00
Junhyun Shim 3023096962 Add a knob to allow token-less tenant data access for untrusted clients 2022-09-08 14:53:01 +02:00
Jon Fu da7ce5231c Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-special-key-space 2022-09-07 13:30:08 -07:00
Jon Fu dbb6357371 add conflict range tests and change tenant prefix code to work with RYW 2022-09-06 16:55:57 -07:00
He Liu 0bbce98da2
Disable shard aware (#8072)
* Removed STORAGE_SERVER_SHARD_AWARE knob.

* Fixed PhysicalShardMove test.

Co-authored-by: He Liu <heliu@apple.com>
2022-09-02 09:07:34 -07:00
Josh Slocum 269d54b7ad
One additional case for new error, and dropping retry limit for upgrade test back down (#8041) 2022-08-30 11:01:18 -05:00
Josh Slocum 825a58880e
Reworked multi-version client readBlobGranules to not get stuck on client version changes (#8017)
* Reworked multi-version client readBlobGranules to not get stuck on version changes

* Addressing review comments
2022-08-30 09:16:09 +02:00
Junhyun Shim ac6889286c
Merge pull request #7731 from sfc-gh-jshim/authz-general-tls-and-integration-test
Authz general tls and integration test
2022-08-30 02:19:11 +02:00
Josh Slocum d0a4a16bd6
scaling back ChangeFeedOperationsMove to avoid TraceTooManyLines (#7958) 2022-08-29 15:30:22 -05:00
Sreenath Bodagala f257b6c2d0
- Consider logging a trace event in "DatabaseContext::getLatestCommitVersions" (#8008)
only if the version vector doesn't have an entry for a tag.
- Address a bug in "run_custom_cluster.sh" (Note: code change provided
by Dan).
2022-08-29 09:22:34 -07:00
Junhyun Shim f7ef5edab4 Merge remote-tracking branch 'origin/main' into authz-general-tls-and-integration-test 2022-08-29 16:35:57 +02:00
Junhyun Shim 6f7e2bfb34 Simplify public keyset update tests
Also add test cases for correctness in handling broken/empty/deleted key files
2022-08-29 16:17:02 +02:00
Markus Pilman 98039f6ca1 Merge remote-tracking branch 'origin/main' into features/new-test-harness 2022-08-26 13:52:25 -06:00
Junhyun Shim 6bcfbf421e Merge remote-tracking branch 'origin/main' into authz-general-tls-and-integration-test 2022-08-26 09:20:31 +02:00
Ata E Husain Bohra 00fe4863b6
Implement TenantCacheEntry in-memory cache (#7801)
* Implement TenantCacheEntry in-memory cache

Description

  diff-4: TraceEvent usage improvements 
  diff-3: Address review comments
  diff-2: Add APIs to read counter values, test improvements
  diff-1: Address review comments

Major changes includes:
1. Implements an actor that enables an in-memory caching of
TenantCacheEntry object, allowing the caller to embed custom
information along with TenantCacheEntry.
2. The cache follows read-through cache semantics where the entry
gets loaded from underlying database on a miss.
3. The cache implements a "periodic poller" to refresh known Tenants
by consulting the database. Once a database keyrange-watch feature is
available, cache shall be updated.

Bonus:
Implement a 'recurringAsync' addition to genericActors allowing caller
to schedule a periodic task registering an "actor functor"; the routine
'waits' for the actor unlike existing 'recurring' implementation.

Testing

TenantEntryCache workload
devCorrectnessRun - 100K
2022-08-25 11:42:26 -07:00
Josh Slocum e04e3885b9
Adding BlobRange test (#7868)
* Adding BlobRange test

* refactored blobrange test to use new db functions, and fixed several bugs

* bug fixes for blob manager and verifyBlobRange

* More range unaligned fixes

* cleaning up test and disabling tests that don't work yet for now

* removing overzealous assert in blob manager

* more fixes for overzealous assert

* cleanup and renaming test

* adding chaos to blob ranges test
2022-08-24 11:30:37 -05:00
Junhyun Shim cc5105278d Harden public keyset rollover testcase against possible flakiness 2022-08-24 16:17:08 +02:00
Junhyun Shim 041e030e92 Add testcases bad_token and system_and_special_key_range_disallowed 2022-08-24 15:40:01 +02:00
Josh Slocum cfc5ab550e
Update BlobGranules API tests (#7947)
* blob granule fixes from api tests

* Adding BlobGranuleErrors workload

* bug fixes and debugging improvements for blob granule api tests

* cleanup and refactoring of bg api tests

* better memory management for loadAndMaterialize
2022-08-23 17:47:02 -05:00
Junhyun Shim 94866d54d0 Merge remote-tracking branch 'origin/main' into authz-general-tls-and-integration-test 2022-08-23 18:07:49 +02:00
Junhyun Shim 16f0d53238 Use admin_server as fixture 2022-08-23 15:44:04 +02:00
Junhyun Shim 8397fc3899 Add public key rollover testcase 2022-08-23 14:21:26 +02:00
Junhyun Shim 83a22253a3 Fix multiprocessing and admin server code 2022-08-22 20:55:17 +02:00
Dan Lambright a8dd6b65ec
Support HA in loopback cluster test script (#7953)
* update tag client cache on getKeyLocation

* Support HA for custom cluster test script
2022-08-22 14:33:36 -04:00