Description
find_package was used to find and link `zlib` library needed to enable
boost::gzip compression filter. However, the code adds dynamic linkage
of zlib shared object with generated binaries (fdbserver for instance).
For now disable the ZLIB find code to effectively disable GZIP compression
support.
Testing
* flow: add ApiVersion to replace hard coding api version
Instead of hard coding api value, let's rely on feature versions akin to
ProtocolVersion.
* ApiVersion: remove use of -1 for latest and use LATEST_VERSION
* Don't build fdb c shim with ubsan
This avoids duplicate symbols when linking. It doesn't really make sense
to assemble files with -fsanitize=undefined anyway, since it won't
insert instrumentation.
* Consolidate boost_asan with boost_target
* Fix ASAN build
Description
-diff-2: Fix Mac build issues
-diff-1: Address review comments
Patch addresses the issue where ASAN build failed after introducing
BlobGranule compression.
Testing
ASAN build
* Attempt to fix windows build
The windows build is complaining that some symbols in flow are
duplicate, which seems fair since we're currently compiling all the flow
sources and _also_ linking to flow for flowlinktest. This change makes
it so that we only link flow, and don't compile the flow source files
again.
* Remove source files from fdbclient and fdbrpc link tests
Currently, we have code in different folders like `flow/` and `fdbrpc/`
that should remain isolated. For example, `flow/` files should not
include functionality from any other modules. `fdbrpc/` files should
only be able to include functionality from itself and from `flow/`.
However, when creating a shared library, the linker doesn't complain
about undefined symbols -- this only happens when creating an
executable. Thus, for example, it is possible to forward declare an
`fdbclient` function in an `fdbrpc` file and then use it, and nothing
will break (when it should, because this is illegal).
This change adds dummy executables for a few modules (`flow`, `fdbrpc`,
`fdbclient`) that will cause a linker error if there are included
symbols which the linker can't resolve.
* Add JWT support to TokenSign
* Encapsulate OpenSSL public/private key type
Type-safe passing around of keys without having to DER/PEM-serialize
(OpenSSL doesn't have distinct types for public and private key)
* Apply Clang format
* Add verify benchmark for JWT and FlatBuffers token
* Unit test base64url::{encode, decode}
* Make all payload fields optional
Let user code validate non-signature fields
* Make all payload fields optional
Completely defer field check to user code
* Move rapidjson from fdbclient to contrib
* Make fdbrpc's rapidjson linkage private
Currently only sources include them.
* Modify rapidjson path in apiversioner.py
* Algorithm::Unknown > Algorithm::UNKNOWN
* Enable debugId tracing for encryption requests
Description
diff-1: Minor fixes, address review comment
Proposed changes include:
1. Update EncryptKeyProxy API to embded Optional<UID> for debugging
request execution.
2. Encryption participant FDB processes can set 'debugId' enabling
tracing requests within FDB cluster processes and beyond.
3. The 'debugId' if available is embedded as part of 'request_json_payload'
by RESTKmsConnector, enabling tracing request between FDB <--> KMS.
4. Fix EncryptKeyProxyTest which got broken due to recent changes.
Testing
Updated following test:
1. EncryptKeyProxy simulation test.
2. RESTKmsConnector simulation test.
Description
Testing
* Change DNS cache to use std::map.
Revert commit 90c259d84e, because if we use unordered_map, toString() can be inconsistent.
* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.
* Move Hostname to its own files.
Also, add resolve-related variables and functions in Hostname.
* Introduce BlobCipher interface and cipher caching interface
diff-3: Update the code to avoid deriving encryption key periodically.
Implement EncyrptBuf interface to limit memcpys.
Improve both unit test and simulation to better code coverage.
diff-2: Add specific error code for OpenSSL AES call failures
diff-1: Update encryption scheme to AES-256-CTR. Minor
updates to Header to capture more information.
Major changes proposed are:
1. Introduce encyrption header format.
2. Introduce a BlobCipher cipher key representation encoding
following information: baseCipher details, derived encryption cipher
details, creationTime and random salt.
3. Introduce interface to support block cipher encrytion and decrytion
operations. Encyrption populates encryption header allowing client to
persist them on-disk, this header is then read allowing decryption
on reads.
4. Introduce interface to allow in-memory caching of cipher keys. The
cache allowing mapping of "encryption domain" -> "base cipher id" ->
"derived cipher keys" (3D hash map). This cache interface will be used
by FDB processes participating in encryption to cache recently used
ciphers (performance optimization).
Testing:
1. Unit test to validate caching interface.
2. Update EncryptionOps simulation test to validate block cipher
operations.
Hui Liu