wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbserver/workloads/EncryptionOps.actor.cpp

384 lines
16 KiB
C++
Raw Normal View History

Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
/*
* EncryptionOps.actor.cpp
*
* This source file is part of the FoundationDB open source project
*
Update copyright header dates
2022-03-22 04:36:23 +08:00
* Copyright 2013-2022 Apple Inc. and the FoundationDB project authors
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "fdbclient/DatabaseContext.h"
#include "fdbclient/NativeAPI.actor.h"
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
#include "flow/EncryptUtils.h"
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
#include "flow/IRandom.h"
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
#include "flow/BlobCipher.h"
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
#include "fdbserver/workloads/workloads.actor.h"
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
#include "flow/ITrace.h"
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
#include "flow/Trace.h"
#include <chrono>
#include <cstring>
#include <memory>
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
#include <random>
#include "flow/actorcompiler.h" // This must be the last #include.
#if ENCRYPTION_ENABLED
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
#define MEGA_BYTES (1024 * 1024)
#define NANO_SECOND (1000 * 1000 * 1000)
struct WorkloadMetrics {
double totalEncryptTimeNS;
double totalDecryptTimeNS;
double totalKeyDerivationTimeNS;
int64_t totalBytes;
void reset() {
totalEncryptTimeNS = 0;
totalDecryptTimeNS = 0;
totalKeyDerivationTimeNS = 0;
totalBytes = 0;
}
WorkloadMetrics() { reset(); }
double computeEncryptThroughputMBPS() {
// convert bytes -> MBs & nano-seonds -> seconds
return (totalBytes * NANO_SECOND) / (totalEncryptTimeNS * MEGA_BYTES);
}
double computeDecryptThroughputMBPS() {
// convert bytes -> MBs & nano-seonds -> seconds
return (totalBytes * NANO_SECOND) / (totalDecryptTimeNS * MEGA_BYTES);
}
void updateKeyDerivationTime(double val) { totalKeyDerivationTimeNS += val; }
void updateEncryptionTime(double val) { totalEncryptTimeNS += val; }
void updateDecryptionTime(double val) { totalDecryptTimeNS += val; }
void updateBytes(int64_t val) { totalBytes += val; }
void recordMetrics(const std::string& mode, const int numIterations) {
TraceEvent("EncryptionOpsWorkload")
.detail("Mode", mode)
.detail("EncryptTimeMS", totalEncryptTimeNS / 1000)
.detail("DecryptTimeMS", totalDecryptTimeNS / 1000)
.detail("EncryptMBPS", computeEncryptThroughputMBPS())
.detail("DecryptMBPS", computeDecryptThroughputMBPS())
.detail("KeyDerivationTimeMS", totalKeyDerivationTimeNS / 1000)
.detail("TotalBytes", totalBytes)
.detail("AvgCommitSize", totalBytes / numIterations);
}
};
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
// Workload generator for encryption/decryption operations.
// 1. For every client run, it generate unique random encryptionDomainId range and simulate encryption of
// either fixed size or variable size payload.
// 2. For each encryption run, it would interact with BlobCipherKeyCache to fetch the desired encryption key,
// which then is used for encrypting the plaintext payload.
// 3. Encryption operation generates 'encryption header', it is leveraged to decrypt the ciphertext obtained from
// step#2 (simulate real-world scenario)
//
// Correctness validations:
// -----------------------
// Correctness invariants are validated at various steps:
// 1. Encryption key correctness: as part of performing decryption, BlobCipherKeyCache lookup is done to procure
// desired encrytion key based on: {encryptionDomainId, baseCipherId}; the obtained key is validated against
// the encryption key used for encrypting the data.
// 2. After encryption, generated 'encryption header' fields are validated, encrypted buffer size and contents are
// validated.
// 3. After decryption, the obtained deciphertext is validated against the orginal plaintext payload.
//
// Performance metrics:
// -------------------
// The workload generator profiles below operations across the iterations and logs the details at the end, they are:
// 1. Time spent in encryption key fetch (and derivation) operations.
// 2. Time spent encrypting the buffer (doesn't incude key lookup time); also records the throughput in MB/sec.
// 3. Time spent decrypting the buffer (doesn't incude key lookup time); also records the throughput in MB/sec.
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
struct EncryptionOpsWorkload : TestWorkload {
int mode;
int64_t numIterations;
int pageSize;
int maxBufSize;
std::unique_ptr<uint8_t[]> buff;
Arena arena;
std::unique_ptr<WorkloadMetrics> metrics;
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
EncryptCipherDomainId minDomainId;
EncryptCipherDomainId maxDomainId;
EncryptCipherBaseKeyId minBaseCipherId;
EncryptCipherBaseKeyId headerBaseCipherId;
Revert "Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)" (#6922) Description Major changes proposed: 1. This reverts commit f38b2e82098f432bc4437f823c5c1dafd52e1319. 2. Also add fix for Valgrind failure due to unintialized variables. 3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache isn't as expected Testing Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20) EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430) EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
2022-04-22 09:57:56 +08:00
EncryptCipherRandomSalt headerRandomSalt;
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
EncryptionOpsWorkload(WorkloadContext const& wcx) : TestWorkload(wcx) {
mode = getOption(options, LiteralStringRef("fixedSize"), 1);
numIterations = getOption(options, LiteralStringRef("numIterations"), 10);
pageSize = getOption(options, LiteralStringRef("pageSize"), 4096);
maxBufSize = getOption(options, LiteralStringRef("maxBufSize"), 512 * 1024);
buff = std::make_unique<uint8_t[]>(maxBufSize);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
// assign unique encryptionDomainId range per workload clients
minDomainId = wcx.clientId * 100 + mode * 30 + 1;
maxDomainId = deterministicRandom()->randomInt(minDomainId, minDomainId + 10) + 5;
minBaseCipherId = 100;
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
headerBaseCipherId = wcx.clientId * 100 + 1;
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
metrics = std::make_unique<WorkloadMetrics>();
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
TraceEvent("EncryptionOpsWorkload")
.detail("Mode", getModeStr())
.detail("MinDomainId", minDomainId)
.detail("MaxDomainId", maxDomainId);
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
}
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
~EncryptionOpsWorkload() { TraceEvent("EncryptionOpsWorkload_Done").log(); }
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
bool isFixedSizePayload() { return mode == 1; }
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
std::string getModeStr() const {
if (mode == 1) {
return "FixedSize";
} else if (mode == 0) {
return "VariableSize";
}
// no other mode supported
throw internal_error();
}
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
void generateRandomBaseCipher(const int maxLen, uint8_t* buff, int* retLen) {
memset(buff, 0, maxLen);
*retLen = deterministicRandom()->randomInt(maxLen / 2, maxLen);
generateRandomData(buff, *retLen);
}
void setupCipherEssentials() {
EncryptKeyProxy server APIs for simulation runs. (#6727) * EncryptKeyProxy server APIs for simulation runs. Description diff-2: FlowSingleton util class Bug fixes diff-1: Expected errors returned to the caller Major changes proposed are: 1. EncryptKeyProxy server APIs: 1.1. Lookup Cipher details via BaseCipherId 1.2. Lookup latest Cipher details via encryption domainId. 2. EncyrptKeyProxy implements caches indexed by: baseCipherId & encyrptDomainId 3. Periodic task to refresh domainId indexed cache to support 'limiting cipher lifetime' abilities if supported by external KMS solutions. Testing EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-12 00:08:42 +08:00
Reference<BlobCipherKeyCache> cipherKeyCache = BlobCipherKeyCache::getInstance();
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
TraceEvent("SetupCipherEssentials_Start").detail("MinDomainId", minDomainId).detail("MaxDomainId", maxDomainId);
uint8_t buff[AES_256_KEY_LENGTH];
std::vector<Reference<BlobCipherKey>> cipherKeys;
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
int cipherLen = 0;
for (EncryptCipherDomainId id = minDomainId; id <= maxDomainId; id++) {
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
generateRandomBaseCipher(AES_256_KEY_LENGTH, &buff[0], &cipherLen);
EncryptKeyProxy server APIs for simulation runs. (#6727) * EncryptKeyProxy server APIs for simulation runs. Description diff-2: FlowSingleton util class Bug fixes diff-1: Expected errors returned to the caller Major changes proposed are: 1. EncryptKeyProxy server APIs: 1.1. Lookup Cipher details via BaseCipherId 1.2. Lookup latest Cipher details via encryption domainId. 2. EncyrptKeyProxy implements caches indexed by: baseCipherId & encyrptDomainId 3. Periodic task to refresh domainId indexed cache to support 'limiting cipher lifetime' abilities if supported by external KMS solutions. Testing EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-12 00:08:42 +08:00
cipherKeyCache->insertCipherKey(id, minBaseCipherId, buff, cipherLen);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
ASSERT(cipherLen > 0 && cipherLen <= AES_256_KEY_LENGTH);
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
EncryptKeyProxy server APIs for simulation runs. (#6727) * EncryptKeyProxy server APIs for simulation runs. Description diff-2: FlowSingleton util class Bug fixes diff-1: Expected errors returned to the caller Major changes proposed are: 1. EncryptKeyProxy server APIs: 1.1. Lookup Cipher details via BaseCipherId 1.2. Lookup latest Cipher details via encryption domainId. 2. EncyrptKeyProxy implements caches indexed by: baseCipherId & encyrptDomainId 3. Periodic task to refresh domainId indexed cache to support 'limiting cipher lifetime' abilities if supported by external KMS solutions. Testing EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-12 00:08:42 +08:00
cipherKeys = cipherKeyCache->getAllCiphers(id);
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
ASSERT_EQ(cipherKeys.size(), 1);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
}
Revert "Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)" (#6922) Description Major changes proposed: 1. This reverts commit f38b2e82098f432bc4437f823c5c1dafd52e1319. 2. Also add fix for Valgrind failure due to unintialized variables. 3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache isn't as expected Testing Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20) EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430) EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
2022-04-22 09:57:56 +08:00
// insert the Encrypt Header cipherKey; record cipherDetails as getLatestCipher() may not work with multiple
// test clients
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
generateRandomBaseCipher(AES_256_KEY_LENGTH, &buff[0], &cipherLen);
Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902) This reverts commit a38318a6acc1ef71ebcc5ea4af6bb2ea0edf0887.
2022-04-22 00:04:40 +08:00
cipherKeyCache->insertCipherKey(ENCRYPT_HEADER_DOMAIN_ID, headerBaseCipherId, buff, cipherLen);
Revert "Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)" (#6922) Description Major changes proposed: 1. This reverts commit f38b2e82098f432bc4437f823c5c1dafd52e1319. 2. Also add fix for Valgrind failure due to unintialized variables. 3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache isn't as expected Testing Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20) EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430) EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
2022-04-22 09:57:56 +08:00
Reference<BlobCipherKey> latestCipher = cipherKeyCache->getLatestCipherKey(ENCRYPT_HEADER_DOMAIN_ID);
ASSERT_EQ(latestCipher->getBaseCipherId(), headerBaseCipherId);
ASSERT_EQ(memcmp(latestCipher->rawBaseCipher(), buff, cipherLen), 0);
headerRandomSalt = latestCipher->getSalt();
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
Revert "Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)" (#6922) Description Major changes proposed: 1. This reverts commit f38b2e82098f432bc4437f823c5c1dafd52e1319. 2. Also add fix for Valgrind failure due to unintialized variables. 3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache isn't as expected Testing Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20) EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430) EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
2022-04-22 09:57:56 +08:00
TraceEvent("SetupCipherEssentials_Done")
.detail("MinDomainId", minDomainId)
.detail("MaxDomainId", maxDomainId)
.detail("HeaderBaseCipherId", headerBaseCipherId)
.detail("HeaderRandomSalt", headerRandomSalt);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
}
void resetCipherEssentials() {
EncryptKeyProxy server APIs for simulation runs. (#6727) * EncryptKeyProxy server APIs for simulation runs. Description diff-2: FlowSingleton util class Bug fixes diff-1: Expected errors returned to the caller Major changes proposed are: 1. EncryptKeyProxy server APIs: 1.1. Lookup Cipher details via BaseCipherId 1.2. Lookup latest Cipher details via encryption domainId. 2. EncyrptKeyProxy implements caches indexed by: baseCipherId & encyrptDomainId 3. Periodic task to refresh domainId indexed cache to support 'limiting cipher lifetime' abilities if supported by external KMS solutions. Testing EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-12 00:08:42 +08:00
Reference<BlobCipherKeyCache> cipherKeyCache = BlobCipherKeyCache::getInstance();
cipherKeyCache->cleanup();
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
TraceEvent("ResetCipherEssentials_Done").log();
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
}
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
void updateLatestBaseCipher(const EncryptCipherDomainId encryptDomainId,
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
uint8_t* baseCipher,
int* baseCipherLen,
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
EncryptCipherBaseKeyId* nextBaseCipherId) {
EncryptKeyProxy server APIs for simulation runs. (#6727) * EncryptKeyProxy server APIs for simulation runs. Description diff-2: FlowSingleton util class Bug fixes diff-1: Expected errors returned to the caller Major changes proposed are: 1. EncryptKeyProxy server APIs: 1.1. Lookup Cipher details via BaseCipherId 1.2. Lookup latest Cipher details via encryption domainId. 2. EncyrptKeyProxy implements caches indexed by: baseCipherId & encyrptDomainId 3. Periodic task to refresh domainId indexed cache to support 'limiting cipher lifetime' abilities if supported by external KMS solutions. Testing EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-12 00:08:42 +08:00
Reference<BlobCipherKeyCache> cipherKeyCache = BlobCipherKeyCache::getInstance();
Reference<BlobCipherKey> cipherKey = cipherKeyCache->getLatestCipherKey(encryptDomainId);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
*nextBaseCipherId = cipherKey->getBaseCipherId() + 1;
generateRandomBaseCipher(AES_256_KEY_LENGTH, baseCipher, baseCipherLen);
ASSERT(*baseCipherLen > 0 && *baseCipherLen <= AES_256_KEY_LENGTH);
TraceEvent("UpdateBaseCipher").detail("DomainId", encryptDomainId).detail("BaseCipherId", *nextBaseCipherId);
}
Revert "Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)" (#6922) Description Major changes proposed: 1. This reverts commit f38b2e82098f432bc4437f823c5c1dafd52e1319. 2. Also add fix for Valgrind failure due to unintialized variables. 3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache isn't as expected Testing Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20) EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430) EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
2022-04-22 09:57:56 +08:00
Reference<BlobCipherKey> getEncryptionKey(const EncryptCipherDomainId& domainId,
const EncryptCipherBaseKeyId& baseCipherId,
const EncryptCipherRandomSalt& salt) {
const bool simCacheMiss = deterministicRandom()->randomInt(1, 100) < 15;
Reference<BlobCipherKeyCache> cipherKeyCache = BlobCipherKeyCache::getInstance();
Reference<BlobCipherKey> cipherKey = cipherKeyCache->getCipherKey(domainId, baseCipherId, salt);
if (simCacheMiss) {
TraceEvent("SimKeyCacheMiss").detail("EncyrptDomainId", domainId).detail("BaseCipherId", baseCipherId);
// simulate KeyCache miss that may happen during decryption; insert a CipherKey with known 'salt'
cipherKeyCache->insertCipherKey(domainId,
baseCipherId,
cipherKey->rawBaseCipher(),
cipherKey->getBaseCipherLen(),
cipherKey->getSalt());
// Ensure the update was a NOP
Reference<BlobCipherKey> cKey = cipherKeyCache->getCipherKey(domainId, baseCipherId, salt);
ASSERT(cKey->isEqual(cipherKey));
}
return cipherKey;
}
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
Reference<EncryptBuf> doEncryption(Reference<BlobCipherKey> textCipherKey,
Reference<BlobCipherKey> headerCipherKey,
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
uint8_t* payload,
int len,
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
const EncryptAuthTokenMode authMode,
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
BlobCipherEncryptHeader* header) {
uint8_t iv[AES_256_IV_LENGTH];
generateRandomData(&iv[0], AES_256_IV_LENGTH);
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
EncryptBlobCipherAes265Ctr encryptor(textCipherKey, headerCipherKey, &iv[0], AES_256_IV_LENGTH, authMode);
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
auto start = std::chrono::high_resolution_clock::now();
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
Reference<EncryptBuf> encrypted = encryptor.encrypt(payload, len, header, arena);
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
auto end = std::chrono::high_resolution_clock::now();
// validate encrypted buffer size and contents (not matching with plaintext)
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
ASSERT_EQ(encrypted->getLogicalSize(), len);
ASSERT_NE(memcmp(encrypted->begin(), payload, len), 0);
ASSERT_EQ(header->flags.headerVersion, EncryptBlobCipherAes265Ctr::ENCRYPT_HEADER_VERSION);
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
metrics->updateEncryptionTime(std::chrono::duration<double, std::nano>(end - start).count());
return encrypted;
}
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
void doDecryption(Reference<EncryptBuf> encrypted,
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
int len,
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
const BlobCipherEncryptHeader& header,
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
uint8_t* originalPayload,
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
Reference<BlobCipherKey> orgCipherKey) {
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
ASSERT_EQ(header.flags.headerVersion, EncryptBlobCipherAes265Ctr::ENCRYPT_HEADER_VERSION);
ASSERT_EQ(header.flags.encryptMode, ENCRYPT_CIPHER_MODE_AES_256_CTR);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
Revert "Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)" (#6922) Description Major changes proposed: 1. This reverts commit f38b2e82098f432bc4437f823c5c1dafd52e1319. 2. Also add fix for Valgrind failure due to unintialized variables. 3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache isn't as expected Testing Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20) EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430) EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
2022-04-22 09:57:56 +08:00
Reference<BlobCipherKey> cipherKey = getEncryptionKey(header.cipherTextDetails.encryptDomainId,
header.cipherTextDetails.baseCipherId,
header.cipherTextDetails.salt);
Reference<BlobCipherKey> headerCipherKey = getEncryptionKey(header.cipherHeaderDetails.encryptDomainId,
header.cipherHeaderDetails.baseCipherId,
header.cipherHeaderDetails.salt);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
ASSERT(cipherKey.isValid());
ASSERT(cipherKey->isEqual(orgCipherKey));
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
DecryptBlobCipherAes256Ctr decryptor(cipherKey, headerCipherKey, &header.cipherTextDetails.iv[0]);
const bool validateHeaderAuthToken = deterministicRandom()->randomInt(0, 100) < 65;
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
auto start = std::chrono::high_resolution_clock::now();
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
if (validateHeaderAuthToken) {
decryptor.verifyHeaderAuthToken(header, arena);
}
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
Reference<EncryptBuf> decrypted = decryptor.decrypt(encrypted->begin(), len, header, arena);
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
auto end = std::chrono::high_resolution_clock::now();
// validate decrypted buffer size and contents (matching with original plaintext)
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
ASSERT_EQ(decrypted->getLogicalSize(), len);
ASSERT_EQ(memcmp(decrypted->begin(), originalPayload, len), 0);
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
metrics->updateDecryptionTime(std::chrono::duration<double, std::nano>(end - start).count());
}
Future<Void> setup(Database const& ctx) override { return Void(); }
std::string description() const override { return "EncryptionOps"; }
Future<Void> start(Database const& cx) override {
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
uint8_t baseCipher[AES_256_KEY_LENGTH];
int baseCipherLen = 0;
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
EncryptCipherBaseKeyId nextBaseCipherId;
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
// Setup encryptDomainIds and corresponding baseCipher details
setupCipherEssentials();
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
for (int i = 0; i < numIterations; i++) {
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
bool updateBaseCipher = deterministicRandom()->randomInt(1, 100) < 5;
// Step-1: Encryption key derivation, caching the cipher for later use
EncryptKeyProxy server APIs for simulation runs. (#6727) * EncryptKeyProxy server APIs for simulation runs. Description diff-2: FlowSingleton util class Bug fixes diff-1: Expected errors returned to the caller Major changes proposed are: 1. EncryptKeyProxy server APIs: 1.1. Lookup Cipher details via BaseCipherId 1.2. Lookup latest Cipher details via encryption domainId. 2. EncyrptKeyProxy implements caches indexed by: baseCipherId & encyrptDomainId 3. Periodic task to refresh domainId indexed cache to support 'limiting cipher lifetime' abilities if supported by external KMS solutions. Testing EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-12 00:08:42 +08:00
Reference<BlobCipherKeyCache> cipherKeyCache = BlobCipherKeyCache::getInstance();
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
// randomly select a domainId
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
const EncryptCipherDomainId encryptDomainId = deterministicRandom()->randomInt(minDomainId, maxDomainId);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
ASSERT(encryptDomainId >= minDomainId && encryptDomainId <= maxDomainId);
if (updateBaseCipher) {
// simulate baseCipherId getting refreshed/updated
updateLatestBaseCipher(encryptDomainId, &baseCipher[0], &baseCipherLen, &nextBaseCipherId);
EncryptKeyProxy server APIs for simulation runs. (#6727) * EncryptKeyProxy server APIs for simulation runs. Description diff-2: FlowSingleton util class Bug fixes diff-1: Expected errors returned to the caller Major changes proposed are: 1. EncryptKeyProxy server APIs: 1.1. Lookup Cipher details via BaseCipherId 1.2. Lookup latest Cipher details via encryption domainId. 2. EncyrptKeyProxy implements caches indexed by: baseCipherId & encyrptDomainId 3. Periodic task to refresh domainId indexed cache to support 'limiting cipher lifetime' abilities if supported by external KMS solutions. Testing EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-12 00:08:42 +08:00
cipherKeyCache->insertCipherKey(encryptDomainId, nextBaseCipherId, &baseCipher[0], baseCipherLen);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
}
auto start = std::chrono::high_resolution_clock::now();
EncryptKeyProxy server APIs for simulation runs. (#6727) * EncryptKeyProxy server APIs for simulation runs. Description diff-2: FlowSingleton util class Bug fixes diff-1: Expected errors returned to the caller Major changes proposed are: 1. EncryptKeyProxy server APIs: 1.1. Lookup Cipher details via BaseCipherId 1.2. Lookup latest Cipher details via encryption domainId. 2. EncyrptKeyProxy implements caches indexed by: baseCipherId & encyrptDomainId 3. Periodic task to refresh domainId indexed cache to support 'limiting cipher lifetime' abilities if supported by external KMS solutions. Testing EncyrptKeyProxyTest workload to validate the newly added code.
2022-04-12 00:08:42 +08:00
Reference<BlobCipherKey> cipherKey = cipherKeyCache->getLatestCipherKey(encryptDomainId);
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
// Each client working with their own version of encryptHeaderCipherKey, avoid using getLatest()
Reference<BlobCipherKey> headerCipherKey =
Revert "Revert "Update 'salt' details for EncryptHeader AuthToken details (#6881)" (#6902)" (#6922) Description Major changes proposed: 1. This reverts commit f38b2e82098f432bc4437f823c5c1dafd52e1319. 2. Also add fix for Valgrind failure due to unintialized variables. 3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache isn't as expected Testing Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20) EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430) EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
2022-04-22 09:57:56 +08:00
cipherKeyCache->getCipherKey(ENCRYPT_HEADER_DOMAIN_ID, headerBaseCipherId, headerRandomSalt);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
auto end = std::chrono::high_resolution_clock::now();
metrics->updateKeyDerivationTime(std::chrono::duration<double, std::nano>(end - start).count());
// Validate sanity of "getLatestCipher", especially when baseCipher gets updated
if (updateBaseCipher) {
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
ASSERT_EQ(cipherKey->getBaseCipherId(), nextBaseCipherId);
ASSERT_EQ(cipherKey->getBaseCipherLen(), baseCipherLen);
ASSERT_EQ(memcmp(cipherKey->rawBaseCipher(), baseCipher, baseCipherLen), 0);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
}
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
int dataLen = isFixedSizePayload() ? pageSize : deterministicRandom()->randomInt(100, maxBufSize);
generateRandomData(buff.get(), dataLen);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
// Encrypt the payload - generates BlobCipherEncryptHeader to assist decryption later
BlobCipherEncryptHeader header;
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
const EncryptAuthTokenMode authMode = deterministicRandom()->randomInt(0, 100) < 50
? ENCRYPT_HEADER_AUTH_TOKEN_MODE_SINGLE
: ENCRYPT_HEADER_AUTH_TOKEN_MODE_MULTI;
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
try {
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
Reference<EncryptBuf> encrypted =
doEncryption(cipherKey, headerCipherKey, buff.get(), dataLen, authMode, &header);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
// Decrypt the payload - parses the BlobCipherEncryptHeader, fetch corresponding cipherKey and
// decrypt
doDecryption(encrypted, dataLen, header, buff.get(), cipherKey);
} catch (Error& e) {
TraceEvent("Failed")
.detail("DomainId", encryptDomainId)
Encryption header authentication tokens (#6750) * Encryption header authentication tokens Description diff-1: Allow NONE AuthTokenMode operations Address review comments Major changes proposed are: 1.Encryption header support two modes of generation 'authentication tokens': a) SingleAuthTokenMode: the scheme generates single crypto-secure auth token to protect {cipherText + header} payload. Scheme is geared towards optimizing cost due to crypto-secure auth-token generation, however, on decryption client needs to be read 'header' + 'encrypted-buffer' to validate the 'auth-token'. The scheme is ideal for usecases where payload represented by the encryptionHeader is not large and it is desirable to minimize CPU/latency penalty due to crypto-secure ops, such as: CommitProxies encrypted inline transactions, StorageServer encrypting pages etc. b) MultiAuthTokenMode: Scheme generates separate authTokens for 'encrypted buffer' & 'encryption-header'. The scheme is ideal where payload represented by encryptionHeader is large enough such that it is desirable to optimize cost of upfront reading full 'encrypted buffer', compared to reading only encryptionHeader and ensuring its sanity; for instance: backup-files 2. Leverage full crypto-secure digest as 'authentication token' Testing Update EncryptionOps simulation test Update BlobCipher unit test 20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44 20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
2022-04-09 02:32:05 +08:00
.detail("BaseCipherId", cipherKey->getBaseCipherId())
.detail("AuthMode", authMode);
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
throw;
}
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
metrics->updateBytes(dataLen);
}
Introduce BlobCipher interface and cipher caching interface (#6391) * Introduce BlobCipher interface and cipher caching interface diff-3: Update the code to avoid deriving encryption key periodically. Implement EncyrptBuf interface to limit memcpys. Improve both unit test and simulation to better code coverage. diff-2: Add specific error code for OpenSSL AES call failures diff-1: Update encryption scheme to AES-256-CTR. Minor updates to Header to capture more information. Major changes proposed are: 1. Introduce encyrption header format. 2. Introduce a BlobCipher cipher key representation encoding following information: baseCipher details, derived encryption cipher details, creationTime and random salt. 3. Introduce interface to support block cipher encrytion and decrytion operations. Encyrption populates encryption header allowing client to persist them on-disk, this header is then read allowing decryption on reads. 4. Introduce interface to allow in-memory caching of cipher keys. The cache allowing mapping of "encryption domain" -> "base cipher id" -> "derived cipher keys" (3D hash map). This cache interface will be used by FDB processes participating in encryption to cache recently used ciphers (performance optimization). Testing: 1. Unit test to validate caching interface. 2. Update EncryptionOps simulation test to validate block cipher operations.
2022-03-24 22:31:49 +08:00
// Cleanup cipherKeys
resetCipherEssentials();
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314) * Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor Major changes proposed are: 1. Refactor StreamCipher code to enable instantiation of multiple encryption keys. However, code still retains a globalEncryption key semantics used in Backup file encryption usecase. 2. Enhance StreamCipher to provide HMAC signature digest generation. Further, the class implements HMAC encryption key derivation function. 3. Upgrade StreamCipher to use AES 256 GCM mode from currently supported AES 128 GCM mode. Note: The code changes the encryption key size, however, the feature is NOT currently in use, hence, should be OK. 3. Add EncryptionOps validation and benchmark toml supported workload, it does the following: a. Allow user to configure encrypt-decrypt of a fixed size buffer or variable size buffer [100, 512K] b. Allow user to configure number of interactions of the runs, in each iteration: generate random data, derive an encryption key using HMAC SHA256 method, encrypt data and then decrypt data. It collects following metrics: i) time taken to derive encryption key. ii) time taken to encrypt the buffer. iii) time taken to decrypt the buffer. iv) total bytes encrypted and/or decrypted c. Along with stats it basic basic validations on the encrypted and decrypted buffer d. On completion for test, records the above mentioned metrics in trace files.
2022-02-01 09:52:44 +08:00
return Void();
}
Future<bool> check(Database const& cx) override { return true; }
void getMetrics(std::vector<PerfMetric>& m) override { metrics->recordMetrics(getModeStr(), numIterations); }
};
WorkloadFactory<EncryptionOpsWorkload> EncryptionOpsWorkloadFactory("EncryptionOps");
#endif // ENCRYPTION_ENABLED