Currently, GlobalConfig is a singleton that means for each process there is only
one GlobalConfig object. This is bug from clients perspective as a client can
keep connections to several databases. This patch tracks GlobalConfig for each
database using an unordered_map in flowGlobals.
We discovered this bug while testing multi-version client, where the client got
stuck. This was lucky, as normally it'd just write down config to the wrong
database.
* REST KmsConnector implementation
Description
diff-1: Address review comments.
Add utility interface to Platform namespace to
create and operate on tmpfile
diff-2: Address review comments
Link Boost::filesystem to CMake build process
Major changes includes:
1. Implement REST based KmsConnector implementation.
2. Salient features of the connector:
2.1. Two required configuration are:
a. Discovery KMS URLs - enable KMS discovery on bootstrap
b. Endpoint path configuration to construct URI to fetch/refresh
encryption keys
c. Configuration to provide "validationTokens" to connect with
external KMS. Patch implements file-based token validation scheme.
2.2. On startup, RESTKmsConnector discovers KMS Urls and caches
them in-memory. Extracts "validationTokens" based on input config.
2.3. Expose endpoints to allow fetch/refresh of encryption keys.
2.4. Defines JSON format to interact with external KMS - request &
response payload format.
3. Extend Platform namespace with an interface to create and operate on
tmp files.
4. Update Platform 'readFileBytes' and 'writeFileBytes' to leverage
fstream supported implementation.
NOTE: KMS URLs fetched after initial discovery will be persisted using
DynamicKnobs. It is TODO at the moment and shall be completed
once DynamicKnobs is feature complete
Testing
Unit test to validation following:
1. Parsing on "validation tokens" logic.
2. Construction and parsing of REST JSON request and response strings.
* re-use same arena when construct mapped key
ConstructMappedKey seems to be a hotspot, try eliminate unnecessary
operations by
* re-using the same Tuple
* preprocess the formatTuple to get a list of Tuples and strings
* throw mapper_bad_range_decriptor when range query is not the last element
* SimKmsConnector fix domain id being unsigned
* SimKmsConnector fix returning cipher id 0 as latest key, which is invalid
* SimKmsConnector fix keys initialized as c-style strings with incorrect length and uninitialized bytes
* SimKmsConnector fix returning different keys for the same id after restart
* BlobCipher change APIs to return null reference when key not found
* BlobCipher insertCipherKey to return the inserted key
* Clean up constructMappedKey method
* fix comments and bugs
* Fix search index
* Fix bug, if escaped is true after first round, it would skip the second round
Markus Pilman