* Add server-side latency metrics for Resolver requests.
* Add separate resolver latency metrics for queue wait and compute time.
* Add histogram for queue depth observed on resolver (during metrics interval).
* Fix tlog latency measurement to use timer() instead of now().
* [EAR]: Remove usage of EncryptDomainName for Encryption at-rest operations
Description
diff-1: Address review comments
EncryptDomainName is an auxillary information, given EAR encryption domain
matches with Tenants, EncryptDomainName maps to TenantName in the current
code. However, this mapping adds EAR depedency has multiple drawbacks:
1. In some scenarios obtaning consistent mapping of TenantId <-> TenantName
is difficult to maintain. For instance: StorageServer (SS) TLog mutation
pop loop, it is possible that same commit batch contains: TenantMap update
mutation as well as a Tenant user mutation. SS would parse TenantMap update
mutation (FDB System Keyspace encryption domain), process the mutation, but,
doesn't apply it to the process local TenantMap. SS then attempts to process,
Tenant user mutation and fails to decrypt the mutation given TenantMetadaMap
isn't updated yet.
2. FDB codebase uses EncryptDomainId matching TenantId, TenantName is used as
an auxillary information source and feels better to be handled by an
external KMS.
Major changes include:
1. EAR to remove TenantName dependency across all participating processes
such as: CommitProxy, Redwood, BlobGranule and Backup agent.
2. Update EKP and KmsConnector APIs to avoid relying on "domainName"
information being passed around to external KMS EAR endpoints.
Testing
devRunCorrectness - 100K
EncryptKeyProxyTest - 100K
EncryptionOps Test - 100K
Changes:
1. Change `isEncryptionOpSupported` to not check against `clientDBInfo.isEncryptionEnabled`, but instead against ENABLE_ENCRYPTION server knob. The problem with clientDBInfo is before its being broadcast to the workers, its content is uninitialized, during which some data (e.g. item 2) is not getting encrypted when they should.
2. Fix CommitProxy not encrypting metadata mutations which are recovered from txnStateStore
3. Fix KeyValueStoreMemory (thus TxnStateStore) partial transaction coming from recovery is not encrypted
4. new CODE_PROBE for the above fixes
5. Logging changes
* proof of concept
* use code-probe instead of test
* code probe working on gcc
* code probe implemented
* renamed TestProbe to CodeProbe
* fixed refactoring typo
* support filtered output
* print probes at end of simulation
* fix missed probes print
* fix deduplication
* Fix refactoring issues
* revert bad refactor
* make sure file paths are relative
* fix more wrong refactor changes
The erased bytes from recent state transactions was accidently set to 0 and
cause anyPopped to be always false. Thus, when totalStateBytes will never
decrease and can cause high latency between "Resolver.resolveBatch.Before" and
"Resolver.resolveBatch.AfterQueueSizeCheck", in hundreds of milliseconds. This
will cause high server side commit latency.
Adding encryption support for TxnStateStore. It is done by supporting encryption. for KeyValueStoreMemory. The encryption is currently done on operation level when the operations are being write to the underlying log file. See inline comment for the encrypted data format.
This PR depends on #7252. It is part of the effort to support TLog encryption #6942.
Nim Wijetunga