Adding the following metrics:
* BlobCipherKeyCache hit/miss
* EKP: KMS requests latencies
* For each component that using encryption, they now need to pass a UsageType enum to the encryption helper methods (GetEncryptCipherKeys/GetLatestEncryptCipherKey/encrypt/decrypt) and those methods will help to log get cipher key latency samples and encryption/decryption cpu times accordingly.
* blob granule fixes from api tests
* Adding BlobGranuleErrors workload
* bug fixes and debugging improvements for blob granule api tests
* cleanup and refactoring of bg api tests
* better memory management for loadAndMaterialize
* Cleaned up BlobGranule TODO + FIXMEs and addressed some
* popping feed at correct version
* blob worker taking over a granule will pop from where previous worker left off
* addressed fixme of blob worker not re-snapshotting from old change feed
* formatting
* more change feed popped fixes after pop updates
* Getting rid of change feed parallelism lock since it can cause deadlocks in fetching, and relying on full fetch lock
* New blob worker metric and fixing old one
* server-side popped checking still doesn't work because of pops at non-mutation versions
* format
Description
FDB native encryption data at-rest supports two type of cipher-keys
in-memory caching:
1. Revocable keys - with a definite expiry (future timestamp)
2. Non-revocable keys - with or without expiry timestamp and/or
refreshAt timestamp.
Patch update BlobCipherKey in-memory cache to respect EKP/KMS
supplied 'refreshAt' and 'expireAt' timestamp. GetLatestCipher
validates `cipher key freshness' as well as GetCipherKey checks
for 'cipher key liveness' before replying details to the caller.
Patch also optimizes the BlobCipher module logging by taking
following measures:
1. BLOB_CIPHER_DEBUG macro to guard spammy log messages needed
mostly for debugging failures.
2. Minimize log volume by logging cipherKey details for any new
key added to the cache, key-refreshes are not logged.
3. Categorize logs into: debug, info and warn on per-usecase basis
Testing
devRunCorrectness - 100K
EncryptOps.toml - 100K
* Encrypt BlobGranule delta files
Description
diff-1: Address review comments
Major changes proposed by the patch are:
1. Refactor code to allow caching of 'encryption key ctx' as part of
BlobFilePointerRef. The refactoring allows snapshot and/or delta files
to store their own file encryption context.
2. Enable BlobGranule delta file encryption/decryption semantics.
Testing
BlobGranuleCorrrectness
BlobGranuleCorrectnessClean
BlobGranuleFileUnitTestToml
Description
Testing
Patch addresses an issue where loop generating random buffer
for the test might run in an infinite loop if KeySpace gets
exhausted but the 'targetBuffer' bytes aren't generated.
Description
Testing
Few fixes
- Update BlobGranuleCipherKeysMeta to use 'std::string' to
persist 'Initialization Vector'
- Update WriteSnapshot to persist 'BlobGranuleCipherKeysMeta' as part
of BlobFileIndex
Description
Testing
A.J. Beamon