wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbserver/workloads
History
Ata E Husain Bohra 33ae398268
REST KmsConnector implementation (#6994) 
* REST KmsConnector implementation

Description
  diff-1: Address review comments.
          Add utility interface to Platform namespace to
          create and operate on tmpfile
 diff-2: Address review comments
         Link Boost::filesystem to CMake build process

Major changes includes:
1. Implement REST based KmsConnector implementation.
2. Salient features of the connector:
 2.1. Two required configuration are:
   a. Discovery KMS URLs - enable KMS discovery on bootstrap
   b. Endpoint path configuration to construct URI to fetch/refresh
      encryption keys
   c. Configuration to provide "validationTokens" to connect with
      external KMS. Patch implements file-based token validation scheme.
 2.2. On startup, RESTKmsConnector discovers KMS Urls and caches
      them in-memory. Extracts "validationTokens" based on input config.
 2.3. Expose endpoints to allow fetch/refresh of encryption keys.
 2.4. Defines JSON format to interact with external KMS - request &
      response payload format.
3. Extend Platform namespace with an interface to create and operate on
   tmp files.
4. Update Platform 'readFileBytes' and 'writeFileBytes' to leverage
   fstream supported implementation.

NOTE: KMS URLs fetched after initial discovery will be persisted using
      DynamicKnobs. It is TODO at the moment and shall be completed
      once DynamicKnobs is feature complete

Testing

Unit test to validation following:
1. Parsing on "validation tokens" logic.
2. Construction and parsing of REST JSON request and response strings.
 		2022-05-07 13:18:35 -07:00
..
ApiCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ApiWorkload.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ApiWorkload.h OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
AsyncFile.actor.h Update copyright header dates 2022-03-21 13:36:23 -07:00
AsyncFile.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
AsyncFileCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
AsyncFileRead.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
AsyncFileWrite.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
AtomicOps.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
AtomicOpsApiCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
AtomicRestore.actor.cpp Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
AtomicSwitchover.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
BackgroundSelectors.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
BackupAndParallelRestoreCorrectness.actor.cpp Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
BackupCorrectness.actor.cpp Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
BackupToBlob.actor.cpp Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
BackupToDBAbort.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
BackupToDBCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
BackupToDBUpgrade.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
BlobGranuleCorrectnessWorkload.actor.cpp Added basic blob granule consistency check 2022-04-04 11:38:42 -05:00
BlobGranuleVerifier.actor.cpp Blob integration (#6808) 2022-04-08 14:15:25 -07:00
BlobStoreWorkload.h Update copyright header dates 2022-03-21 13:36:23 -07:00
BulkLoad.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
BulkSetup.actor.h Update copyright header dates 2022-03-21 13:36:23 -07:00
Cache.actor.cpp Add missing copyright headers 2022-03-16 11:25:02 -07:00
ChangeConfig.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ChangeFeeds.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ClearSingleRange.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ClientTransactionProfileCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ClientWorkload.actor.cpp Address review comments 2022-04-11 09:17:10 -06:00
ClogSingleConnection.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
CommitBugCheck.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ConfigIncrement.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ConfigureDatabase.actor.cpp fix configure workload typo 2022-04-08 11:40:03 -07:00
ConflictRange.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ConsistencyCheck.actor.cpp OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
CpuProfiler.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Cycle.actor.cpp OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
DDBalance.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
DDMetrics.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
DDMetricsExclude.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
DataDistributionMetrics.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
DataLossRecovery.actor.cpp Fix spurious test failure 2022-04-26 09:40:43 -07:00
DifferentClustersSameRV.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
DiskDurability.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
DiskDurabilityTest.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
DiskFailureInjection.actor.cpp Remove getMinReplicasRemaining PromiseStream. 2022-04-07 23:23:23 -07:00
DummyWorkload.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
EncryptKeyProxyTest.actor.cpp Update EKPServer LookupKeysByKeyIds API (#6898) 2022-04-21 14:56:09 -07:00
EncryptionOps.actor.cpp Remove ENCRYPTION_ENABLED macro 2022-05-02 22:26:31 -07:00
ExternalWorkload.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
FastTriggeredWatches.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
FileSystem.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Fuzz.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
FuzzApiCorrectness.actor.cpp Better accounting for tenant prefixes when working with key size limits (#6973) 2022-04-27 14:38:59 -07:00
GetMappedRange.actor.cpp Fix GetMappedRange test when rangeResult has more 2022-04-21 10:16:27 -07:00
GetRangeStream.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
HealthMetricsApi.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
HighContentionPrefixAllocatorWorkload.actor.cpp Fix formatting 2022-01-31 14:43:04 -08:00
Increment.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
IncrementalBackup.actor.cpp Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
IndexScan.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Inventory.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
KVStoreTest.actor.cpp Add sharded rocksdb type (#6862) 2022-04-21 22:53:14 -04:00
KillRegion.actor.cpp Move resolutionBalancing() back to master 2022-03-23 09:57:31 -07:00
LocalRatekeeper.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
LockDatabase.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
LockDatabaseFrequently.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
LogMetrics.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
LowLatency.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
MachineAttrition.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Mako.actor.cpp Add missing copyright headers 2022-03-16 11:25:02 -07:00
MemoryKeyValueStore.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
MemoryKeyValueStore.h Update copyright header dates 2022-03-21 13:36:23 -07:00
MemoryLifetime.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
MetricLogging.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
MiniCycle.actor.cpp OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
MutationLogReaderCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ParallelRestore.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Performance.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
PhysicalShardMove.actor.cpp When clearing the database between tests, check that clearing the tenant left the entire normal key-space empty. Update the configuration of some tests. Disable a special key-space test that is invoking broken behavior. 2022-04-14 11:39:02 -07:00
Ping.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
PopulateTPCC.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
PrivateEndpoints.actor.cpp Don't test requests that don't initialize properly (#6880) 2022-04-18 10:44:56 -07:00
ProtocolVersion.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
PubSubMultiples.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
QueuePush.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
RYWDisable.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
RYWPerformance.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
RandomClogging.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
RandomMoveKeys.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
RandomSelector.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ReadAfterWrite.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ReadHotDetection.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
ReadWrite.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
RemoveServersSafely.actor.cpp Check in the new Hostname logic. (#6926) 2022-04-27 21:54:13 -07:00
ReportConflictingKeys.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
RestoreBackup.actor.cpp Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
RestoreFromBlob.actor.cpp Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Rollback.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
RyowCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
SaveAndKill.actor.cpp Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
SelectorCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Serializability.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Sideband.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
SidebandSingle.actor.cpp Fix two more cases of error(e, true) 2022-03-04 12:44:52 -08:00
SimpleAtomicAdd.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
SlowTaskWorkload.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
SnapTest.actor.cpp Add missing copyright headers 2022-03-16 11:25:02 -07:00
SpecialKeySpaceCorrectness.actor.cpp Check in the new Hostname logic. (#6926) 2022-04-27 21:54:13 -07:00
StatusWorkload.actor.cpp add getStoreTypeStr method 2022-04-26 13:35:15 -07:00
Storefront.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
StreamingRangeRead.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
StreamingRead.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
SubmitBackup.actor.cpp Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
SuspendProcesses.actor.cpp Add missing copyright headers 2022-03-16 11:25:02 -07:00
TPCC.actor.cpp fix bug in tpcc workload 2022-05-02 09:28:23 -07:00
TPCCWorkload.h Update copyright header dates 2022-03-21 13:36:23 -07:00
TagThrottleApi.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
TargetedKill.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
TaskBucketCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
TenantManagement.actor.cpp The tenant deletion test now deletes multiple tenants concurrently rather than serially. Fix some variable shadowing in the delete test. 2022-04-15 13:17:19 -07:00
ThreadSafety.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Throttling.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Throughput.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
TimeKeeperCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
TriggerRecovery.actor.cpp Add missing copyright headers 2022-03-16 11:25:02 -07:00
UDPWorkload.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
UnitPerf.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
UnitTests.actor.cpp REST KmsConnector implementation (#6994) 2022-05-07 13:18:35 -07:00
Unreadable.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
VersionStamp.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
WatchAndWait.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
Watches.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
WatchesSameKeyCorrectness.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
WorkerErrors.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
WriteBandwidth.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
WriteDuringRead.actor.cpp Fix for WriteDuringRead workload key sizes with useSystemKeys=true but writing to normal key space 2022-04-29 11:33:54 -05:00
WriteTagThrottling.actor.cpp Update copyright header dates 2022-03-21 13:36:23 -07:00
workloads.actor.h Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-10 14:12:37 -06:00