* REST KmsConnector implementation
Description
diff-1: Address review comments.
Add utility interface to Platform namespace to
create and operate on tmpfile
diff-2: Address review comments
Link Boost::filesystem to CMake build process
Major changes includes:
1. Implement REST based KmsConnector implementation.
2. Salient features of the connector:
2.1. Two required configuration are:
a. Discovery KMS URLs - enable KMS discovery on bootstrap
b. Endpoint path configuration to construct URI to fetch/refresh
encryption keys
c. Configuration to provide "validationTokens" to connect with
external KMS. Patch implements file-based token validation scheme.
2.2. On startup, RESTKmsConnector discovers KMS Urls and caches
them in-memory. Extracts "validationTokens" based on input config.
2.3. Expose endpoints to allow fetch/refresh of encryption keys.
2.4. Defines JSON format to interact with external KMS - request &
response payload format.
3. Extend Platform namespace with an interface to create and operate on
tmp files.
4. Update Platform 'readFileBytes' and 'writeFileBytes' to leverage
fstream supported implementation.
NOTE: KMS URLs fetched after initial discovery will be persisted using
DynamicKnobs. It is TODO at the moment and shall be completed
once DynamicKnobs is feature complete
Testing
Unit test to validation following:
1. Parsing on "validation tokens" logic.
2. Construction and parsing of REST JSON request and response strings.
* SimKmsConnector fix domain id being unsigned
* SimKmsConnector fix returning cipher id 0 as latest key, which is invalid
* SimKmsConnector fix keys initialized as c-style strings with incorrect length and uninitialized bytes
* SimKmsConnector fix returning different keys for the same id after restart
* BlobCipher change APIs to return null reference when key not found
* BlobCipher insertCipherKey to return the inserted key
Description
Major changes proposed:
1. This reverts commit f38b2e8209.
2. Also add fix for Valgrind failure due to unintialized variables.
3. Improve checks to catch is cipherKey details cached in BlobCipherKeyCache
isn't as expected
Testing
Overall correctness: 10K (20220421-193911-ahusain-foundationdb-a730e5cb38541e20)
EncyrptionOps correctness: 100K (20220421-194315-ahusain-foundationdb-29c598a8b9420430)
EncryptionOps Valgrind: 100 (20220421-194434-ahusain-foundationdb-7fc5f98eddc0921a)
* Add tryResolveHostnames() in connection string.
* Add missing hostname to related interfaces.
* Do not pass RequestStream into *GetReplyFromHostname() functions.
Because we are using new RequestStream for each request anyways. Also, the passed in pointer could be nullptr, which results in seg faults.
* Add dynamic hostname resolve and reconnect intervals.
* Address comments.
* Put guard pages next to fast alloc memory
I verified that we can now detect #6753 without creating tons of
threads.
* Use pageSize instead of 4096
* Don't include mmapInternal for windows
* Update 'salt' details for EncryptHeader AuthToken details
Description
Major changes:
1. Add 'salt' to BlobCipherEncryptHeader::cipherHeaderDetails.
2. During decryption it is possible that BlobKeyCacheId doesn't
contain required baseCipherDetails. Add API to KeyCache to
allowing re-populating of CipherDetails with a given 'salt'
3. Update BaseCipherKeyIdCache indexing using {BaseCipherKeyId, salt}
tuple. FDB processes leverage BlobCipherKeyCache to implement
in-memory caching of cipherKeys, given EncryptKeyProxy supplies
BaseCipher details, each encryption participant service would
generate its derived key by using different 'salt'. Further,
it is possible to cache multiple {baseCipherKeyId, salt} tuples;
for instance: CP encrypted mutations being deciphered by
StorageServer etc.
Testing
1. Update EncyrptionOps simulation test to simulate KeyCache miss
2. Update BlobCipher unit tests to validate above mentioned changes
* Re-throw operation_cancelled
There's a few places in fdbcli where we don't rethrow operation
cancelled but wait on a future. It's very unusual that you don't want to
rethrow operation_cancelled.
* Update ASSERT
It's possible to get error_code_broken_promise here if the network has
already shutdown.
* EncryptKeyProxy server APIs for simulation runs.
Description
diff-2: FlowSingleton util class
Bug fixes
diff-1: Expected errors returned to the caller
Major changes proposed are:
1. EncryptKeyProxy server APIs:
1.1. Lookup Cipher details via BaseCipherId
1.2. Lookup latest Cipher details via encryption domainId.
2. EncyrptKeyProxy implements caches indexed by: baseCipherId &
encyrptDomainId
3. Periodic task to refresh domainId indexed cache to support
'limiting cipher lifetime' abilities if supported by
external KMS solutions.
Testing
EncyrptKeyProxyTest workload to validate the newly added code.
* Fixing leaked stream with explicit notify failed before destructor
* better logic to prevent races in change feed fetching
* Found new race that makes assert incorrect
* handle server overloaded in initial read from fdb
* Handling more blob error types in granule retry
* Fixing rollback metadata problem, added better debugging
* Fixing version race when fetching change feed metadata
* Better racing split request handling
* fixing assert
* Handle change feed popped check in the blob worker
* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)
* more merge conflict issues
* Change feed destroy fixes
* Fixing change feed destroy and move race
* Check error condition in BG file req
* Using relative endpoints for blob worker interface
* Fixing bug in previous fix
* More destroy and move race fixes
* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read
* Bug fix (#6796)
* fix: do not use a RYW transaction for a versionstamp because of randomize API version
* fix: if the initialSnapshotVersion was pruned, granule history was incorrect
* added a way to compress null bytes in printable()
* Fixing durability issue with moving and destroying change feeds
* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot
* More destroy and move races
* Fixing change feed destroy and pop races
* Renaming bg prune to purge, and adding a C api and unit test for it
* more cleanup
* review comments
* Observability for granule purging
* better handling for change feed not registered
* Fixed purging bugs (#6815)
* fix: do not use a RYW transaction for a versionstamp because of randomize API version
* fix: if the initialSnapshotVersion was pruned, granule history was incorrect
* added a way to compress null bytes in printable()
* fixed a few purging bugs
Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
* Encryption header authentication tokens
Description
diff-1: Allow NONE AuthTokenMode operations
Address review comments
Major changes proposed are:
1.Encryption header support two modes of generation 'authentication tokens':
a) SingleAuthTokenMode: the scheme generates single crypto-secure auth
token to protect {cipherText + header} payload. Scheme is geared towards
optimizing cost due to crypto-secure auth-token generation, however,
on decryption client needs to be read 'header' + 'encrypted-buffer'
to validate the 'auth-token'. The scheme is ideal for usecases where
payload represented by the encryptionHeader is not large and it is
desirable to minimize CPU/latency penalty due to crypto-secure ops,
such as: CommitProxies encrypted inline transactions,
StorageServer encrypting pages etc.
b) MultiAuthTokenMode: Scheme generates separate authTokens for
'encrypted buffer' & 'encryption-header'. The scheme is ideal where
payload represented by encryptionHeader is large enough such that it
is desirable to optimize cost of upfront reading full 'encrypted buffer',
compared to reading only encryptionHeader and ensuring its sanity;
for instance: backup-files
2. Leverage full crypto-secure digest as 'authentication token'
Testing
Update EncryptionOps simulation test
Update BlobCipher unit test
20220408-182229-ahusain-foundationdb-7fd2e4b19328cd44
20220408-175754-ahusain-foundationdb-5352e37e1dcabfc8
Changing `memory` option to limit resident memory instead of virtual memory, in config file and fdbserver/fdbbackup/fdbcli command-line argument. Since `rlimit` doesn't support limiting virtual memory, the current implementation have both of fdbmonitor and the fdbserver/fdbbackup process checking process RSS periodically and kill and restart the process if the limit is exceeded.
Adding a new `memory_vsize` option to limit virtual memory, if backward-compatible behavior is desired.
closes#6671, closes#6672
* Allocate at least sizeof(ArenaBlock) for an ArenaBlock
* Fix message pack unit test
Previously we were using only the 4 least significan bits as the length
of a message pack string, but it should be 5 according to https://github.com/msgpack/msgpack/blob/master/spec.md#str-format-family
* Change DNS cache to use std::map.
Revert commit 90c259d84e, because if we use unordered_map, toString() can be inconsistent.
* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.
* Move Hostname to its own files.
Also, add resolve-related variables and functions in Hostname.
* OTEL Span Implementation.
* Addi trace logging, refactor constructors, unit tests.
* Unit tests for creating OTELSpans
* refactor flag names
* Additional comments.
* Formatting.
* Add back Arena.h include
* cleanup header includes
* Remove include cstddef.
* Remove memory include.
* Remove trailing commas on enums.
* Enum formatting.
* Changing SpanStatus enum from ERROR to ERR to see if it is clashing with Windows.h.
* Move OTELEvents to SmallVectorRef<KeyValueRef>.
* Clean up unused includes.
* Unit tests
* Const reference arguments for OTEL constructors and additional addAttribute
unit tests. Adding return of OTELSpan reference on addAttribute.
* Formatting.
* Begin messagepack encoding tests.
* Formatting.
* MessagePack encoding unit tests.
* Formatting.
* Remove swapBinary.
* remove ambiguous helper methods
* Formatting fixes
* Fix ambiguous calls in AddEvents unit tests.
* Include AddAttributes unit test.
* descope windows for UDP encoding test
* Move ifndef WIN32 around MPEncoding unit test.
* Fix AddEvents Attributes size assertion.
* Formatting.
* Enable AddLinks unit test.
* Full MP encoding testing.
* Fix for encoding longer strings with MessagePack and unit test.
* Remove unnecessary header includes and serialize_string_ref function.
* Fix typos
* Update flow/Tracing.actor.cpp
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
* Update flow/Tracing.actor.cpp
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
* Use ASSERT_WE_THINK and add logging.
We don't want people creating incredibly large traces, so we are only
supporting a subset of MessagePack collection and string sizes. Assert
and log when we hit these unsupported sizes.
* Remove TODOs no longer applicable.
* Refactor OTELEvent to OTELEventRef.
* Remove unnecessary public declaration in struct.
* fix OTELEventRef attribute size assertion
* Formatting
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
* initial structure for remote IKVS server
* moved struct to .h file, added new files to CMakeList
* happy path implementation, connection error when testing
* saved minor local change
* changed tracing to debug
* fixed onClosed and getError being called before init is finished
* fix spawn process bug, now use absolute path
* added server knob to set ikvs process port number
* added server knob for remote/local kv store
* implement simulator remote process spawning
* fixed bug for simulator timeout
* commit all changes
* removed print lines in trace
* added FlowProcess implementation by Markus
* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child
* temporary fix for process factory throwing segfault on create
* specify public address in command
* change remote kv store knob to false for jenkins build
* made port 0 open random unused port
* change remote store knob to true for benchmark
* set listening port to randomly opened port
* added print lines for jenkins run open kv store timeout debug
* removed most tracing and print lines
* removed tutorial changes
* update handleIOErrors error handling to handle remote-ikvs cases
* Push all debugging changes
* A version where worker bug exists
* A version where restarting tests fail
* Use both the name and the port to determine the child process
* Remove unnecessary update on local address
* Disable remote-kvs for DiskFailureCycle test
* A version where restarting stuck
* A version where most restarting tests green
* Reset connection with child process explicitly
* Remove change on unnecessary files
* Unify flags from _ to -
* fix merging unexpected changes
* fix trac.error to .errorUnsuppressed
* Add license header
* Remove unnecessary header in FlowProcess.actor.cpp
* Fix Windows build
* Fix Windows build, add missing ;
* Fix a stupid bug caused by code dropped by code merging
* Disable remote kvs by default
* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune
* serialization change on readrange
* Update traces
* Refactor the RemoteIKVS interface
* Format files
* Update sim2 interface to not clog connections between parent and child processes in simulation
* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled
* Add comments, format files
* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections
* Commit the IConnection interface change, forgot in previous commit
* Fix the issue that onClosed request is cancelled by ActorCollection
* Enable the remote kv store knob
* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process
* Fix the bug where one process starts storage server more than once
* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally
* Remove unreachable code path and add comments
* Clang format the code
* Fix a simple wait error
* Clang format after merging the main branch
* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false
* Disable remote kvs for PhysicalShardMove which is for RocksDB
* Cleanup #include orders, remove debugging traces
* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build
Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
* Combine vector_like_traits::{insert,reserve}
and explain semantics better. This should make it more clear what
implementers need to do when implementing the vector_like_traits
concept.
* Update std::unordered_set vector_like_traits impl
Junhyun Shim