Commit Graph

8060 Commits

Author SHA1 Message Date
Jingyu Zhou f361f5a96c Rename isEmptyMessage to messagesWritten
The variable name means exactly the opposite.
2022-02-03 14:55:44 -08:00
Josh Slocum b324cf4bc4 Fixing TSS buggify in restarting tests 2022-02-03 06:46:28 -06:00
Ata E Husain Bohra 591ef57857
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314)
* Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor

Major changes proposed are:
1. Refactor StreamCipher code to enable instantiation of
   multiple encryption keys. However, code still retains
   a globalEncryption key semantics used in Backup file
   encryption usecase.
2. Enhance StreamCipher to provide HMAC signature digest
   generation. Further, the class implements HMAC encryption
   key derivation function.
3. Upgrade StreamCipher to use AES 256 GCM mode from currently
   supported AES 128 GCM mode.
   Note: The code changes the encryption key size, however, the
         feature is NOT currently in use, hence, should be OK.
3. Add EncryptionOps validation and benchmark toml supported
   workload, it does the following:
   a. Allow user to configure encrypt-decrypt of a fixed size
      buffer or variable size buffer [100, 512K]
   b. Allow user to configure number of interactions of the runs,
      in each iteration: generate random data, derive an encryption
      key using HMAC SHA256 method, encrypt data and
      then decrypt data. It collects following metrics:
    i) time taken to derive encryption key.
    ii) time taken to encrypt the buffer.
    iii) time taken to decrypt the buffer.
    iv) total bytes encrypted and/or decrypted
   c. Along with stats it basic basic validations on the encrypted
      and decrypted buffer
   d. On completion for test, records the above mentioned metrics
      in trace files.
2022-01-31 19:52:44 -06:00
A.J. Beamon 8c00ba6716
Merge pull request #6324 from sfc-gh-ajbeamon/high-contention-allocator
Rename high contention allocator implementation in fdbclient
2022-01-31 15:51:03 -08:00
A.J. Beamon 7195d4dadd Fix formatting 2022-01-31 14:43:04 -08:00
A.J. Beamon 6affc58e97 Rename high contention allocator implementation in fdbclient 2022-01-31 14:25:38 -08:00
Markus Pilman 60cbce72c2
Merge pull request #6317 from sfc-gh-ajbeamon/high-contention-allocator
Added a generic high contention allocator implementation to fdbclient
2022-01-31 14:50:51 -07:00
Trevor Clinkenbeard e6b02532a6
Merge pull request #6192 from sfc-gh-tclinkenbeard/improve-commitbatcher-performance
Make `VectorRefPreserializer` move constructor noexcept
2022-01-31 09:52:51 -08:00
A.J. Beamon 027fe80594 Added a generic high contention allocator implementation to fdbclient. This is an adapted version of the flow bindings HCA implementation. 2022-01-28 15:34:30 -08:00
A.J. Beamon 213ddc9ac3
Merge pull request #6315 from sfc-gh-ajbeamon/fix-init-order
Fix constructor initialization order warning
2022-01-28 12:46:55 -08:00
Renxuan Wang f9f3735f73 Add resolveHostnamesBlocking() in ConnectionString and IClusterConnectionRecord.
Also, combine IClusterConnectionRecord::getConnectionString() and IClusterConnectionRecord::getMutableConnectionString() to IClusterConnectionRecord::getConnectionString(), and rename setConnectionString() to setAndPersistConnectionString().
2022-01-28 12:20:41 -08:00
gaozengqi 23ea6e2c7e Fix typo 2022-01-28 10:40:33 -08:00
gaozengqi f1bd0f16da Run clang-format 2022-01-28 10:40:33 -08:00
gaozengqi a0c0342e76 Fix usage 2022-01-28 10:40:33 -08:00
gaozengqi 0e6dbf434b Fix use allKeys.end 2022-01-28 10:40:33 -08:00
gaozengqi 1419d1aab7 Fix short hand 2022-01-28 10:40:33 -08:00
gaozengqi 697d075e20 Add role 'kvfiledump' to dump key-values from storage file 2022-01-28 10:40:33 -08:00
A.J. Beamon 0803abedaa Adjust the name order of constructor variable initialization to match declaration order. 2022-01-28 10:27:13 -08:00
Evan Tschannen bb082344e2
Merge pull request #6101 from sfc-gh-ahusain/ahusain-encryptServerRole
Add new FDB EncryptKeyProxy role
2022-01-26 14:13:56 -08:00
Andrew Noyes c967e6246b
Fix uninitialized memory in storage server (#6282) 2022-01-26 13:24:35 -08:00
Ata E Husain Bohra f3c3ab06f1 Add new FDB EncryptKeyProxy role
diff-1: Address review comments

Major changes includes:

1. Add a new FDB role responsible- EncyrptKeyProxy. The role is
   responsible to expose APIs to fetch encyrption keys interacting
   with external Encryption KeyManager interface.
2. The process is a FDB singleton process following similar recruitment
   rules as other singleton processes in the system.
3. Code to recruit the worker process; given the encryption keys are
   needed during recovery (decode TLog records), for now the process
   is co-located in same datacenter as ClusterController.
4. Skeleton process actor code; more functionality will be added in
   subsequent PRs.

NOTE: The code is protected under a SERVER_KNOB with the default
      value as 'false' for now.:%s
2022-01-25 23:12:49 -08:00
Ata E Husain Bohra 87ee4cf958 Add new FDB EncryptKeyProxy role
Major changes includes:

1. Add a new FDB role responsible- EncyrptKeyProxy. The role is
   responsible to expose APIs to fetch encyrption keys interacting
   with external Encryption KeyManager interface.
2. The process is a FDB singleton process following similar recruitment
   rules as other singleton processes in the system.
3. Code to recruit the worker process; given the encryption keys are
   needed during recovery (decode TLog records), for now the process
   is co-located in same datacenter as ClusterController.
4. Skeleton process actor code; more functionality will be added in
   subsequent PRs.

NOTE: The code is protected under a SERVER_KNOB with the default
      value as 'false' for now.
2022-01-25 17:38:27 -08:00
Yao Xiao c605226a56
Add error handling in RocksDB KVS. (#6277) 2022-01-25 13:59:25 -08:00
He Liu 5e0b1fd13f Resolved comments. 2022-01-21 16:58:42 -08:00
He Liu 0421562ce9 Resolved comments. 2022-01-21 16:58:42 -08:00
He Liu b7388a714b Added get, scan, and commit counters. 2022-01-21 16:58:42 -08:00
He Liu da8453eb5d Added systemClearRange. 2022-01-21 16:58:42 -08:00
He Liu f14c832031 Added storage engine metrics in SS. 2022-01-21 16:58:42 -08:00
Renxuan Wang 334ba66348 Add the support for using hostname in simulation.
This PR does NOT turn the feature on. Also, disable hostname in downgrade restarting tests.
2022-01-21 10:48:33 -08:00
neethuhaneesha c0c0f101c5
Merge pull request #6235 from neethuhaneesha/writeRateLimiter
Rocksdb write rate limiter.
2022-01-19 12:22:07 -08:00
Ata E Husain Bohra 703364d146
Update cluster recovery documentation (#6255)
Patch updates code documentation to reflect the recent code
refactoring where ClusterController process drives recovery
instead of sequencer/master process.
2022-01-18 13:54:00 -08:00
Neethu Haneesha Bingi 162bce7a58 Rocksdb write rate limiter. 2022-01-18 13:23:00 -08:00
neethuhaneesha 034b934ecd
Merge pull request #6204 from neethuhaneesha/reuseIterators
Rocksdb read range iterator pool to reuse iterators.
2022-01-18 12:59:52 -08:00
Neethu Haneesha Bingi ef4038fe8d Rocksdb read range iterator pool to reuse iterators. 2022-01-18 02:05:21 -08:00
A.J. Beamon 07e5319477
Merge pull request #6165 from sfc-gh-ajbeamon/native-api-refactor
Refactor Native API Transactions
2022-01-13 14:53:19 -08:00
A.J. Beamon 17415168b6 Make the use provisional proxies parameter be of type UseProvisionalProxies everywhere. 2022-01-13 12:41:20 -08:00
Trevor Clinkenbeard de39293b8d
Merge pull request #6206 from sfc-gh-tclinkenbeard/testworkload-smart-pointers
Use smart pointers to manage `TestWorkload` objects
2022-01-11 23:33:48 -08:00
sfc-gh-tclinkenbeard 05b94fed03 Add virtual destructor for IWorkloadFactory 2022-01-11 17:00:43 -08:00
A.J. Beamon b44ebe0c65 Fix typo in trace event name 2022-01-11 13:22:00 -08:00
Jingyu Zhou db436fb494 Remove unneeded Arena in Requests/Replies
If the Request/Reply doesn't have *Ref types, we typically don't need to have
an Arena.
2022-01-10 10:26:02 -08:00
sfc-gh-tclinkenbeard ac8979d624 Use smart pointers to manage TestWorkload objects 2022-01-06 14:31:50 -08:00
Ata E Husain Bohra 936bf5336a
Revert "Revert "Refactor: ClusterController driving cluster-recovery state machine" (#6191)
* Revert "Revert "Refactor: ClusterController driving cluster-recovery state machine""

Major changes includes:
1. Re-revert Sequencer refactor commits listed below (in listed order):
1.a. This reverts commit bb17e194d9.
1.b. This reverts commit d174bb2e06.
1.c. This reverts commit 30b05b469c.

2. Update Status.actor to track ClusterController interface to track
   recovery status.
3. Introduce a ServerKnob to define "cluster recovery trace event"
   prefix; for now keeping it as "Master", however, it should allow
   smooth transition to "Cluster" prefix as it seems more appropriate.
2022-01-06 12:15:51 -08:00
sfc-gh-tclinkenbeard 54eb3b2e91 Disable static assertion in commitBatcher 2022-01-06 12:14:13 -08:00
Andrew Noyes a3f37df94a
Merge pull request #6175 from sfc-gh-anoyes/anoyes/delete-non-virtual-destructor
Enable -Wdelete-non-virtual-dtor for clang build
2022-01-05 15:41:59 -08:00
A.J. Beamon 59503a397e Fix duplicate trace field. 2022-01-05 11:30:21 -08:00
A.J. Beamon 138932c12f Fixed a memory bug in the watch map. A few other formatting tweaks, etc. 2022-01-05 11:26:43 -08:00
He Liu 1c2c2783ea Implement single deletion in RocksDB.
If the target deletion range contains a single key, convert it into a
single deletion, to avoid extra cost in RocksDB due to range deletion.
2022-01-04 09:59:40 -08:00
sfc-gh-tclinkenbeard 648609d764 Make VectorRefPreserializer move constructor noexcept 2021-12-30 22:46:16 -08:00
Aaron Molitor 30b05b469c Revert "Refactor: ClusterController driving cluster-recovery state machine"
This reverts commit dfe9d184ff.
2021-12-24 11:25:51 -08:00
Aaron Molitor d174bb2e06 Revert "Refactor: ClusterController driving cluster-recovery state machine"
This reverts commit abd2959702.
2021-12-24 11:25:51 -08:00