wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/tests/fast
History
Ata E Husain Bohra 33ae398268
REST KmsConnector implementation (#6994) 
* REST KmsConnector implementation

Description
  diff-1: Address review comments.
          Add utility interface to Platform namespace to
          create and operate on tmpfile
 diff-2: Address review comments
         Link Boost::filesystem to CMake build process

Major changes includes:
1. Implement REST based KmsConnector implementation.
2. Salient features of the connector:
 2.1. Two required configuration are:
   a. Discovery KMS URLs - enable KMS discovery on bootstrap
   b. Endpoint path configuration to construct URI to fetch/refresh
      encryption keys
   c. Configuration to provide "validationTokens" to connect with
      external KMS. Patch implements file-based token validation scheme.
 2.2. On startup, RESTKmsConnector discovers KMS Urls and caches
      them in-memory. Extracts "validationTokens" based on input config.
 2.3. Expose endpoints to allow fetch/refresh of encryption keys.
 2.4. Defines JSON format to interact with external KMS - request &
      response payload format.
3. Extend Platform namespace with an interface to create and operate on
   tmp files.
4. Update Platform 'readFileBytes' and 'writeFileBytes' to leverage
   fstream supported implementation.

NOTE: KMS URLs fetched after initial discovery will be persisted using
      DynamicKnobs. It is TODO at the moment and shall be completed
      once DynamicKnobs is feature complete

Testing

Unit test to validation following:
1. Parsing on "validation tokens" logic.
2. Construction and parsing of REST JSON request and response strings.
 		2022-05-07 13:18:35 -07:00
..
AtomicBackupCorrectness.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
AtomicBackupToDBCorrectness.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
AtomicOps.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
AtomicOpsApiCorrectness.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
BackupAzureBlobCorrectness.toml Update blob backup test files 2020-12-24 21:20:05 -04:00
BackupCorrectness.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
BackupCorrectnessClean.toml Renamed prefixesAllowed again for clarity. 2021-04-20 11:54:40 -07:00
BackupS3BlobCorrectness.toml Fix Azure blob storage backup test 2020-12-24 21:20:05 -04:00
BackupToDBCorrectness.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
BackupToDBCorrectnessClean.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
BlobGranuleVerifyAtomicOps.toml Disabling rocks in blob granule tests 2022-03-24 20:40:16 -05:00
BlobGranuleVerifyCycle.toml Disabling rocks in blob granule tests 2022-03-24 20:40:16 -05:00
BlobGranuleVerifySmall.toml Disabling rocks in blob granule tests 2022-03-24 20:40:16 -05:00
BlobGranuleVerifySmallClean.toml Disabling rocks in blob granule tests 2022-03-24 20:40:16 -05:00
CacheTest.toml Significantly lower the expected rate in the cache test 2022-03-16 15:26:43 -07:00
ChangeFeeds.toml Fix a couple test specification errors 2022-03-17 12:10:19 -07:00
CloggedSideband.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
ConfigureLocked.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
ConstrainedRandomSelector.toml Make sure only uppercase characters follow underscore in test titles 2020-11-08 14:30:55 -08:00
CycleAndLock.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
CycleTest.toml Addressed review comments, added test 2022-03-15 16:57:26 +01:00
DataLossRecovery.toml When clearing the database between tests, check that clearing the tenant left the entire normal key-space empty. Update the configuration of some tests. Disable a special key-space test that is invoking broken behavior. 2022-04-14 11:39:02 -07:00
EncryptKeyProxyTest.toml EncryptKeyProxy server APIs for simulation runs. (#6727) 2022-04-11 09:08:42 -07:00
EncryptionOps.toml Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) 2022-01-31 19:52:44 -06:00
FuzzApiCorrectness.toml Lazily assign the default tenant in simulated tests so that we can enforce that raw access isn't used on tenant transactions. Update various tests to avoid using raw access or system key access after they have performed other operations on the transaction. 2022-03-17 12:10:19 -07:00
FuzzApiCorrectnessClean.toml Lazily assign the default tenant in simulated tests so that we can enforce that raw access isn't used on tenant transactions. Update various tests to avoid using raw access or system key access after they have performed other operations on the transaction. 2022-03-17 12:10:19 -07:00
GetMappedRange.toml GetMappedRange support serializable & check RYW & continuation (#6181) 2022-03-10 10:05:44 -08:00
IncrementTest.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
IncrementalBackup.toml change test spec 2020-10-12 15:28:21 -04:00
InventoryTestAlmostReadOnly.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
InventoryTestSomeWrites.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
KillRegionCycle.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
LocalRatekeeper.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
LongStackWriteDuringRead.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
LowLatency.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
LowLatencySingleClog.toml Increase max latency for LowLatencySingleClog test 2021-07-19 22:34:27 -07:00
MemoryLifetime.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
MoveKeysCycle.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
MutationLogReaderCorrectness.toml Check in the prototype of MutationLogReader. 2021-08-18 14:59:50 -07:00
PhysicalShardMove.toml When clearing the database between tests, check that clearing the tenant left the entire normal key-space empty. Update the configuration of some tests. Disable a special key-space test that is invoking broken behavior. 2022-04-14 11:39:02 -07:00
PrivateEndpoints.toml Addressed review comments, added test 2022-03-15 16:57:26 +01:00
ProtocolVersion.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
RESTKmsConnectorUnit.toml REST KmsConnector implementation (#6994) 2022-05-07 13:18:35 -07:00
RESTUtilsUnit.toml REST KmsConnector implementation (#6994) 2022-05-07 13:18:35 -07:00
RandomSelector.toml Make sure only uppercase characters follow underscore in test titles 2020-11-08 14:30:55 -08:00
RandomUnitTests.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
ReadHotDetectionCorrectness.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
ReportConflictingKeys.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
SelectorCorrectness.toml Make sure only uppercase characters follow underscore in test titles 2020-11-08 14:30:55 -08:00
Sideband.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
SidebandSingle.toml Clone sideband workload to check consistency of cache 2021-12-02 14:17:59 -05:00
SidebandWithStatus.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
SimpleAtomicAdd.toml fix spacing and newlines at ends of files 2020-09-28 16:32:20 -04:00
SpecialKeySpaceCorrectness.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
StreamingRangeRead.toml rename test file 2021-09-20 13:58:05 -07:00
SwizzledRollbackSideband.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
SystemRebootTestCycle.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
TaskBucketCorrectness.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
TimeKeeperCorrectness.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
TxnStateStoreCycleTest.toml When clearing the database between tests, check that clearing the tenant left the entire normal key-space empty. Update the configuration of some tests. Disable a special key-space test that is invoking broken behavior. 2022-04-14 11:39:02 -07:00
UDP.toml minor bug fixes 2020-08-06 16:56:38 -06:00
Unreadable.toml Rewrite most .txt tests into (pretty) .toml files. 2020-07-12 14:47:40 -07:00
VersionStamp.toml fixed conflicts with tests 2020-08-31 21:18:49 -07:00
Watches.toml resolve pr comments 2021-02-09 17:56:30 -05:00
WriteDuringRead.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00
WriteDuringReadClean.toml allow simulation properties to be overwritten 2021-05-20 13:48:41 -06:00