* REST KmsConnector implementation
Description
diff-1: Address review comments.
Add utility interface to Platform namespace to
create and operate on tmpfile
diff-2: Address review comments
Link Boost::filesystem to CMake build process
Major changes includes:
1. Implement REST based KmsConnector implementation.
2. Salient features of the connector:
2.1. Two required configuration are:
a. Discovery KMS URLs - enable KMS discovery on bootstrap
b. Endpoint path configuration to construct URI to fetch/refresh
encryption keys
c. Configuration to provide "validationTokens" to connect with
external KMS. Patch implements file-based token validation scheme.
2.2. On startup, RESTKmsConnector discovers KMS Urls and caches
them in-memory. Extracts "validationTokens" based on input config.
2.3. Expose endpoints to allow fetch/refresh of encryption keys.
2.4. Defines JSON format to interact with external KMS - request &
response payload format.
3. Extend Platform namespace with an interface to create and operate on
tmp files.
4. Update Platform 'readFileBytes' and 'writeFileBytes' to leverage
fstream supported implementation.
NOTE: KMS URLs fetched after initial discovery will be persisted using
DynamicKnobs. It is TODO at the moment and shall be completed
once DynamicKnobs is feature complete
Testing
Unit test to validation following:
1. Parsing on "validation tokens" logic.
2. Construction and parsing of REST JSON request and response strings.
Description
Major changes proposed:
1. This reverts commit f38b2e8209.
2. Also add fix for Valgrind failure due to unintialized variables.
3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache
isn't as expected
Testing
Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20)
EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430)
EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
* Update EKPServer LookupKeysByKeyIds API
Description
Major changes include:
1. Update EKPServer LookupKeysByKeyIds API to include
'encryptDomainId' as input along with 'baseCipherId'
2. Unify EKPServer response definition to return
std::vector<EKPBaseCipherDetails>
3. Update SimKmsProxy interfaces as per above mentioned
changes.
Testing
1. Update EncryptKeyProxyTest simulation test.
2. Update SimEncyrptKeyProxy unit test
* Update 'salt' details for EncryptHeader AuthToken details
Description
Major changes:
1. Add 'salt' to BlobCipherEncryptHeader::cipherHeaderDetails.
2. During decryption it is possible that BlobKeyCacheId doesn't
contain required baseCipherDetails. Add API to KeyCache to
allowing re-populating of CipherDetails with a given 'salt'
3. Update BaseCipherKeyIdCache indexing using {BaseCipherKeyId, salt}
tuple. FDB processes leverage BlobCipherKeyCache to implement
in-memory caching of cipherKeys, given EncryptKeyProxy supplies
BaseCipher details, each encryption participant service would
generate its derived key by using different 'salt'. Further,
it is possible to cache multiple {baseCipherKeyId, salt} tuples;
for instance: CP encrypted mutations being deciphered by
StorageServer etc.
Testing
1. Update EncyrptionOps simulation test to simulate KeyCache miss
2. Update BlobCipher unit tests to validate above mentioned changes
* Don't test requests that don't initialize properly
Some request objects don't initialize their members
properly when being constructed using the default
constructor. This makes valgrind unhappy. Don't test
these endpoints for now.
* fixed code formatting
Encoding methods used:
- Tag localities: Run length encoding
- Tag ids: Compact representation
- Commit versions: delta encoding.
If "n" is the number of entries in the version vector, with the tags
spread over "m" data centers, these techniques will reduce the number
of bytes to represent the version vector from "(11 * n)" bytes to
"(3 * m + 2 * n)" / "(3 * m + 3 * n)" bytes (depending on the max tag
id value, and ignoring some constants) in the best case.
* EncryptKeyProxy server APIs for simulation runs.
Description
diff-2: FlowSingleton util class
Bug fixes
diff-1: Expected errors returned to the caller
Major changes proposed are:
1. EncryptKeyProxy server APIs:
1.1. Lookup Cipher details via BaseCipherId
1.2. Lookup latest Cipher details via encryption domainId.
2. EncyrptKeyProxy implements caches indexed by: baseCipherId &
encyrptDomainId
3. Periodic task to refresh domainId indexed cache to support
'limiting cipher lifetime' abilities if supported by
external KMS solutions.
Testing
EncyrptKeyProxyTest workload to validate the newly added code.
* Fixing leaked stream with explicit notify failed before destructor
* better logic to prevent races in change feed fetching
* Found new race that makes assert incorrect
* handle server overloaded in initial read from fdb
* Handling more blob error types in granule retry
* Fixing rollback metadata problem, added better debugging
* Fixing version race when fetching change feed metadata
* Better racing split request handling
* fixing assert
* Handle change feed popped check in the blob worker
* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)
* more merge conflict issues
* Change feed destroy fixes
* Fixing change feed destroy and move race
* Check error condition in BG file req
* Using relative endpoints for blob worker interface
* Fixing bug in previous fix
* More destroy and move race fixes
* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read
* Bug fix (#6796)
* fix: do not use a RYW transaction for a versionstamp because of randomize API version
* fix: if the initialSnapshotVersion was pruned, granule history was incorrect
* added a way to compress null bytes in printable()
* Fixing durability issue with moving and destroying change feeds
* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot
* More destroy and move races
* Fixing change feed destroy and pop races
* Renaming bg prune to purge, and adding a C api and unit test for it
* more cleanup
* review comments
* Observability for granule purging
* better handling for change feed not registered
* Fixed purging bugs (#6815)
* fix: do not use a RYW transaction for a versionstamp because of randomize API version
* fix: if the initialSnapshotVersion was pruned, granule history was incorrect
* added a way to compress null bytes in printable()
* fixed a few purging bugs
Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
* Initialize cluster version at wall-clock time
Previously, new clusters would begin at version 0. After this change,
clusters will initialize at a version matching wall-clock time. Instead
of using the Unix epoch (or Windows epoch), FDB clusters will use a new
epoch, defaulting to January 1, 2010, 01:00:00+00:00. In the future,
this base epoch will be modifiable through fdbcli, allowing
administrators to advance the cluster version.
Basing the version off of time allows different FDB clusters to share
data without running into version issues.
* Send version epoch to master
* Cleanup
* Update fdbserver/storageserver.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Jump directly to expected version if possible
* Fix initial version issue on storage servers
* Add random recovery offset to start version in simulation
* Type fixes
* Disable reference time by default
Enable on a cluster using the fdbcli command `versionepoch add 0`.
* Use correct recoveryTransactionVersion when recovering
* Allow version epoch to be adjusted forwards (to decrease the version)
* Set version epoch in simulation
* Add quiet database check to ensure small version offset
* Fix initial version issue on storage servers
* Disable reference time by default
Enable on a cluster using the fdbcli command `versionepoch add 0`.
* Add fdbcli command to read/write version epoch
* Cause recovery when version epoch is set
* Handle optional version epoch key
* Add ability to clear the version epoch
This causes version advancement to revert to the old methodology whereas
versions attempt to advance by about a million versions per second,
instead of trying to match the clock.
* Update transaction access
* Modify version epoch to use microseconds instead of seconds
* Modify fdbcli version target API
Move commands from `versionepoch` to `targetversion` top level command.
* Add fdbcli tests for
* Temporarily disable targetversion cli tests
* Fix version epoch fetch issue
* Fix Arena issue
* Reduce max version jump in simulation to 1,000,000
* Rework fdbcli API
It now requires two commands to fully switch a cluster to using the
version epoch. First, enable the version epoch with `versionepoch
enable` or `versionepoch set <versionepoch>`. At this point, versions
will be given out at a faster or slower rate in an attempt to reach the
expected version. Then, run `versionepoch commit` to perform a one time
jump to the expected version. This is essentially irreversible.
* Temporarily disable old targetversion tests
* Cleanup
* Move version epoch buggify to sequencer
This will cause some issues with the QuietDatabase check for the version
offset - namely, it won't do anything, since the version epoch is not
being written to the txnStateStore in simulation. This will get fixed in
the future.
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
Ata E Husain Bohra