Description
FDB native encryption data at-rest supports two type of cipher-keys
in-memory caching:
1. Revocable keys - with a definite expiry (future timestamp)
2. Non-revocable keys - with or without expiry timestamp and/or
refreshAt timestamp.
Patch update BlobCipherKey in-memory cache to respect EKP/KMS
supplied 'refreshAt' and 'expireAt' timestamp. GetLatestCipher
validates `cipher key freshness' as well as GetCipherKey checks
for 'cipher key liveness' before replying details to the caller.
Patch also optimizes the BlobCipher module logging by taking
following measures:
1. BLOB_CIPHER_DEBUG macro to guard spammy log messages needed
mostly for debugging failures.
2. Minimize log volume by logging cipherKey details for any new
key added to the cache, key-refreshes are not logged.
3. Categorize logs into: debug, info and warn on per-usecase basis
Testing
devRunCorrectness - 100K
EncryptOps.toml - 100K
* Test more C API in upgrade tests
* Fix saving results in blob granule API correctness test
* Enable multitenancy in upgrade test with API version >=720
* Disable the upgrade test with blob granule workload
* Granule purge cannot delete history entry for fully deleting granule until all children are completely done splitting
* Several purging fixes related to granule history
* Fixed typo in refactor
* fixing memory model for purgeRange
* formatting
* weakening granule purge test for now
* cleanup
* First version of force purging granules
* fixing issue in BW range assignment reporting
* Fixing incorrect assert with force purging
* Error handling when checking force purged state
* fixed force purging and recover/reassign range races and check
* Handling force purge + boundary change race
* more places to check for force purged status
* fixed manager restart in the middle of force purge bug
* fixing same-BM purge and assignment races in all cases
* weakening orphaned granule history check a bit because of difficult to solve races
* fixing txn options on retry
* loading force purged ranges at start to avoid resuming a merge that is being force purged
* cleanup
* Enabling purging in granule tests, and adding check for leaked change feeds in force purge
* formatting
* missed parameter in merge conflicts
* Fixing leaked change feed race with merge and force purge
* adding change feed cleanup when new blob manager recovers in-progress merge that raced with force purge
* added forcepurge fdbcli command
* Change to old url format azure://<endpoint>/<account>/<container>/
* Remove unnecessary dependency
* Add cmake_minimum_required for azure backup cmake file
* Add the comments for new supported urls
* Add the host name resolution logic
* format files
* refactor part of the code
* Add documentation about Azure backup testing
* Add more comments, update the doc
* Store rocksdb::DBOptions and rocksdb::ColumnFamilyOptions to
SharedRocksDBState.
* Make KeyValueStoreRocksDBTest.toml `UNIT`.
* Added TEST_STORAGE_ENGINE_TYPE
* Added readOptions.
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli
* Auto format python files used in upgrade tests
* Add a test for upgrading to a future FDB version
* Emphasize that the options for using future protocol version are intended for test purposes only
* Make the global variable for current protocol version visible only locally
* Refactirng to avoid using currentProtocolVersion() in static intialization
* Update go bindings
* better check for granule-ification
* Handling blob granule initial split too large
* Re-evaluating split size if too large, even if read doesn't get transaction_too_old
* reworked to have blob worker propose split key
* New GranuleStatusReply to avoid seqno issue stream side effects
* Handling retries on reevaluateInitialSplit properly
* Waiting for stream to be initialized
* Checking reevaluate split for additional split points beyond proposed
* Fixing more races in reevaluate initial split
* properly handling cleaning up old change feed after split re-evaluate
* fixing granule conversion bug with hard boundaries
* fixing clear and merge check race with cycle test
* refactor missed knob check for clearAndMerge
* Fixing formatting
* review comments and improving large range conversion
* fixing typo
* more formatting
* Using knownBlobRanges for blob granule ranges whether tenants are enabled or not
* Effectively disabled blob granule tests when tenants enabled to fix ctest
* blob: read TenantMap during recovery
Future functionality in the blob subsystem will rely on the tenant data
being loaded. This fixes this issue by loading the tenant data before
completing recovery such that continued actions on existing blob
granules will have access to the tenant data.
Example scenario with failover, splits are restarted before loading the
tenant data:
BM - BlobManager
epoch 3: epoch 4:
BM record intent to split.
Epoch fails.
BM recovery begins.
BM fails to persist split.
BM recovery finishes.
BM.checkBlobWorkerList()
maybeSplitRange().
BM.monitorClientRanges().
loads tenant data.
bin/fdbserver -r simulation -f tests/slow/BlobGranuleCorrectness.toml \
-s 223570924 -b on --crash --trace_format json
* blob: add tuple key truncation for blob granule alignment
FDB has a backup system available using the blob manager and blob
granule subsystem. If we want to audit the data in the blobs, it's a lot
easier if we can align them to something meaningful.
When a blob granule is being split, we ask the storage metrics system
for split points as it holds approximate data distribution metrics.
These keys are then processed to determine if they are a tuple and
should be truncated according to the new knob,
BG_KEY_TUPLE_TRUNCATE_OFFSET.
Here we keep all aligned keys together in the same granule even if it is
larger than the allowed granule size. The following commit will address
this by adding merge boundaries.
* blob: minor clean ups in merging code
1. Rename mergeNow -> seen. This is more inline with clocksweep naming
and removes the confusion between mergeNow and canMergeNow.
2. Make clearMergeCandidate() reset to MergeCandidateCannotMerge to make
a clear distinction what we're accomplishing.
3. Rename canMergeNow() -> mergeEligble().
* blob: add explicit (hard) boundaries
Blob ranges can be specified either through explicit ranges or at the
tenant level. Right now this is managed implicitly. This commit aims to
make it a little more explicit.
Blobification begins in monitorClientRanges() which parses either the
explicit blob ranges or the tenant map. As we do this and add new
ranges, let's explicitly track what is a hard boundary and what isn't.
When blob merging occurs, we respect this boundary. When a hard boundary
is encountered, we submit the found eligible ranges and start looking
for a new range beginning with this hard boundary.
* blob: create BlobGranuleSplitPoints struct
This is a setup for the following commit. Our goal here is to provide a
structure for split points to be passed around. The need is for us to be
able to carry uncommitted state until it is committed and we can apply
these mutations to the in-memory data structures.
* blob: implement soft boundaries
An earlier commit establishes the need to create data boundaries within
a tenant. The reality is we may encounter a set of keys that degnerate
to the same key prefix. We'll need to be able to split those across
granules, but we want to ensure we merge the split granules together
before merging with other granules.
This adds to the BlobGranuleSplitPoints state of new
BlobGranuleMergeBoundary items. BlobGranuleMergeBoundary contains state
saying if it is a left or right boundary. This information is used to,
like hard boundaries, force merging of like granules first.
We read the BlobGranuleMergeBoundary map into memory at recovery.
* Disabled tests for ShardedRocks.
Cleaned up ShardedRocks TraceEvent.
Added assertion in ShardManager::validate().
* Added test trace.
* Make sure TraceEvent contains `ShardedRocks`.
Co-authored-by: He Liu <heliu@apple.com>
* Storage server shard management with physical shards.
* Cleanup.
* Resolved comments.
* Added `UnlimintedCommitBytes`.
Co-authored-by: He Liu <heliu@apple.com>
This change adds:
* ability to store the mapping from tenants to quota in the system keyspace,
* a setter and getter function
* a new workload to test this functionality
FDBCORE-2437
Previously to get around the issue that EKP is not present when restart test switching encryption from on to off and read encrypted data, EKP was made to start in simulation regardless of encryption knob. This PR revert that change, and instead force restart test not to change encryption knob, by passing previous encryption knob through restartInfo.ini file. Also since we don't allow downgrading an encrypted cluster to previous version, disable encryption in downgrade tests.
Also adding an assert to allow reading encrypted mutations only if encryption knob is on. We may reconsider allowing switching encryption on/off for existing cluster, but for now we don't allow it.
* Adding sources of the Implib.so project
* Run C unit tests and API tests with the shim library
* Reuse compilation of C test binaries with and without shim library
* Resolve client library path from an environment variable
* Refactoring: Reusable module for downloading FDB binaries
* Testing client shim library with current version and last release version
* Tests for specifying client library over an environment variable
* Enable C shim library tests on ARM
* Restore the original path for including fdb_api.hpp
* Shard based move.
* Clean up.
* Clear results on retry in getInitialDataDistribution.
* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
* Log failed connection attempts in monitorProxies
* Update coordinator list from the cluster file after failing to connect to all coordinators
* Wiggle and upgrade test with legacy version monitoring; updating tests to use 7.1.9
* Update coordinator list from the cluster file: addressing review comments
* Update coordinator list from the cluster file: addressing review comments
* Wait on future for all setAndPersistConnectionString calls
* Add simulation test for 1 data hall + 1 machine failure case.
* Disable BUGGIFY for DEGRADED_RESET_INTERVAL.
A simulation test discovered a situation where machines attempting to connect
to a dead coordinator (with a well-known endpoint) were getting themselves
marked degraded. This flapping of the degraded state prevented recovery from
completing, as it started over any time it noticed that tlogs on degraded
hosts could be relocated to non-degraded ones.
bin/fdbserver -r simulation -f tests/rare/CycleWithDeadHall.toml -b on -s 276841956
* REST KmsConnector implementation
Description
diff-1: Address review comments.
Add utility interface to Platform namespace to
create and operate on tmpfile
diff-2: Address review comments
Link Boost::filesystem to CMake build process
Major changes includes:
1. Implement REST based KmsConnector implementation.
2. Salient features of the connector:
2.1. Two required configuration are:
a. Discovery KMS URLs - enable KMS discovery on bootstrap
b. Endpoint path configuration to construct URI to fetch/refresh
encryption keys
c. Configuration to provide "validationTokens" to connect with
external KMS. Patch implements file-based token validation scheme.
2.2. On startup, RESTKmsConnector discovers KMS Urls and caches
them in-memory. Extracts "validationTokens" based on input config.
2.3. Expose endpoints to allow fetch/refresh of encryption keys.
2.4. Defines JSON format to interact with external KMS - request &
response payload format.
3. Extend Platform namespace with an interface to create and operate on
tmp files.
4. Update Platform 'readFileBytes' and 'writeFileBytes' to leverage
fstream supported implementation.
NOTE: KMS URLs fetched after initial discovery will be persisted using
DynamicKnobs. It is TODO at the moment and shall be completed
once DynamicKnobs is feature complete
Testing
Unit test to validation following:
1. Parsing on "validation tokens" logic.
2. Construction and parsing of REST JSON request and response strings.
Junhyun Shim