wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/tests
History
Ata E Husain Bohra 33ae398268
REST KmsConnector implementation (#6994) 
* REST KmsConnector implementation

Description
  diff-1: Address review comments.
          Add utility interface to Platform namespace to
          create and operate on tmpfile
 diff-2: Address review comments
         Link Boost::filesystem to CMake build process

Major changes includes:
1. Implement REST based KmsConnector implementation.
2. Salient features of the connector:
 2.1. Two required configuration are:
   a. Discovery KMS URLs - enable KMS discovery on bootstrap
   b. Endpoint path configuration to construct URI to fetch/refresh
      encryption keys
   c. Configuration to provide "validationTokens" to connect with
      external KMS. Patch implements file-based token validation scheme.
 2.2. On startup, RESTKmsConnector discovers KMS Urls and caches
      them in-memory. Extracts "validationTokens" based on input config.
 2.3. Expose endpoints to allow fetch/refresh of encryption keys.
 2.4. Defines JSON format to interact with external KMS - request &
      response payload format.
3. Extend Platform namespace with an interface to create and operate on
   tmp files.
4. Update Platform 'readFileBytes' and 'writeFileBytes' to leverage
   fstream supported implementation.

NOTE: KMS URLs fetched after initial discovery will be persisted using
      DynamicKnobs. It is TODO at the moment and shall be completed
      once DynamicKnobs is feature complete

Testing

Unit test to validation following:
1. Parsing on "validation tokens" logic.
2. Construction and parsing of REST JSON request and response strings.
 		2022-05-07 13:18:35 -07:00
..
TestRunner Upgrade Tests: Create the destination directory before copying a client library from the local repository 2022-04-29 16:41:03 +02:00
argument_parsing tests/ black code reformat 2022-04-27 17:01:20 +02:00
fast REST KmsConnector implementation (#6994) 2022-05-07 13:18:35 -07:00
loopback_cluster loopback_cluster: Find `fdbcli` from `$BUILD` path 2021-04-06 13:44:33 +00:00
noSim Implement single deletion in RocksDB. 2022-01-04 09:59:40 -08:00
python_tests tests/ black code reformat 2022-04-27 17:01:20 +02:00
rare Split ConfigIncrement workload 2022-02-09 14:18:40 -08:00
restarting Increase lower bound for snapshot restart tests to 7.1.0 2022-04-26 11:48:46 -07:00
slow When clearing the database between tests, check that clearing the tenant left the entire normal key-space empty. Update the configuration of some tests. Disable a special key-space test that is invoking broken behavior. 2022-04-14 11:39:02 -07:00
status Rename more places from proxy to commit proxy 2020-09-15 22:29:49 -07:00
AsyncFileCorrectness.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
AsyncFileMix.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
AsyncFileRead.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
AsyncFileReadRandom.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
AsyncFileWrite.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
BGServerCommonUnit.toml Cleanup 2022-03-24 17:15:11 -05:00
BackupContainers.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
BandwidthThrottle.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
BigInsert.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
BlobGranuleFileUnit.toml Cleanup 2022-03-24 17:15:11 -05:00
BlobManagerUnit.toml Cleanup 2022-03-24 17:15:11 -05:00
CMakeLists.txt REST KmsConnector implementation (#6994) 2022-05-07 13:18:35 -07:00
CTestCustom.ctest.cmake disable simulation in ctest by default 2021-07-27 11:17:55 -06:00
ClusterControllerTests.txt Add updateWorkerHealth interface in cluster controller 2021-06-24 19:42:28 -07:00
ConsistencyCheck.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
DDMetricsExclude.txt Make sure only uppercase characters follow underscore in test titles 2020-11-08 14:30:55 -08:00
DataDistributionMetrics.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
DiskDurability.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
FileSystem.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
Happy.txt Initial repository commit 2017-05-25 13:48:44 -07:00
IThreadPool.txt Added unit test. 2021-08-09 16:21:43 -07:00
IncrementalDelete.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
KVStoreMemTest.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
KVStoreReadMostly.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
KVStoreTest.txt Renamed redwood to redwood-1-experimental and file extension to .redwood-v1. 2021-11-16 02:15:22 -08:00
KVStoreTestRead.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
KVStoreTestWrite.txt Initial repository commit 2017-05-25 13:48:44 -07:00
KVStoreValueSize.txt Initial repository commit 2017-05-25 13:48:44 -07:00
LayerStatusMerge.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
Mako.txt Add a simple workload, ReportConflictingKeysWorkload, to test correctness of the API and performance overhead added to the resovler. 2019-12-06 16:21:03 -08:00
ParallelRestoreApiCorrectnessAtomicRestore.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
ParallelRestoreOldBackupApiCorrectnessAtomicRestore.toml test:Mute ParallelRestoreOldBackupApiCorrectnessAtomicRestore 2020-10-15 15:44:29 -07:00
PerfUnitTests.toml added file to run perf unit tests 2021-11-08 15:54:18 -07:00
PopulateTPCC.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
PureNetwork.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
RRW2500.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
RandomRangeRead.txt Minor Redwood comparison optimizations 2021-04-23 18:49:43 +00:00
RandomRead.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
RandomReadWrite.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
ReadAbsent.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
ReadAfterWrite.txt Add a ReadAfterWrite workload, to measure TLog->SS propagation delay. 2020-07-01 02:17:43 -07:00
ReadHalfAbsent.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
RedwoodCorrectness.txt Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection. 2021-04-05 00:03:15 -07:00
RedwoodCorrectnessBTree.txt Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection. 2021-04-05 00:03:15 -07:00
RedwoodCorrectnessPager.txt Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection. 2021-04-05 00:03:15 -07:00
RedwoodCorrectnessUnits.txt Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection. 2021-04-05 00:03:15 -07:00
RedwoodPerfPrefixCompression.txt Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection. 2021-04-05 00:03:15 -07:00
RedwoodPerfRandomRangeScans.txt Minor Redwood comparison optimizations 2021-04-23 18:49:43 +00:00
RedwoodPerfSequentialInsert.txt Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection. 2021-04-05 00:03:15 -07:00
RedwoodPerfSet.txt Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection. 2021-04-05 00:03:15 -07:00
RedwoodPerfTests.txt Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection. 2021-04-05 00:03:15 -07:00
RocksDBTest.txt rename ssd-rocksdb-experimental as ssd-rocksdb-v1. 2022-03-29 10:53:38 -07:00
S3BlobStore.txt Disambiguate between S3BlobStore and other blob stores 2020-10-29 20:42:23 -07:00
SampleNoSimAttrition.txt fixed indentation issues 2019-10-24 13:21:28 -07:00
SimpleExternalTest.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
SlowTask.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
SpecificUnitTest.txt fix test changes 2022-03-15 17:28:36 +01:00
StorageMetricsSampleTests.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
StorageServerInterface.txt Testing Storage Server implementation 2021-05-25 20:28:50 +00:00
StreamingWrite.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
SystemData.txt Testing Storage Server implementation 2021-05-25 20:28:50 +00:00
TPCC.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
ThreadSafety.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
TraceEventMetrics.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
WorkerTests.txt revert unintended changes 2021-08-09 16:29:57 -07:00
default.txt Initial repository commit 2017-05-25 13:48:44 -07:00
errors.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
fail.txt Initial repository commit 2017-05-25 13:48:44 -07:00
killall.txt Make the testspec more restrictive in terms of what can be set where. 2020-07-06 02:03:30 -07:00
latency.txt Initial repository commit 2017-05-25 13:48:44 -07:00
performance-fs.txt Initial repository commit 2017-05-25 13:48:44 -07:00
performance.txt Initial repository commit 2017-05-25 13:48:44 -07:00
ping.TXT Initial repository commit 2017-05-25 13:48:44 -07:00
pingServers.TXT Initial repository commit 2017-05-25 13:48:44 -07:00
pt.TXT Initial repository commit 2017-05-25 13:48:44 -07:00
randomSelector.txt Make sure only uppercase characters follow underscore in test titles 2020-11-08 14:30:55 -08:00
s3VersionHeaders.txt Support AWS v4 header for s3 backup and restore 2022-02-07 17:53:05 -08:00
selectorCorrectness.txt Make sure only uppercase characters follow underscore in test titles 2020-11-08 14:30:55 -08:00