Description
find_package was used to find and link `zlib` library needed to enable
boost::gzip compression filter. However, the code adds dynamic linkage
of zlib shared object with generated binaries (fdbserver for instance).
For now disable the ZLIB find code to effectively disable GZIP compression
support.
Testing
* Introduce "default encryption domain"
Description
In current FDB native encryption data at-rest implementation,
an entity getting encrypted (mutation, KV and/or file) is categorized
into one of following encryption domains:
1. Tenant domain, where, Encryption domain == Tenant boundaries
2. FDB system keyspace - FDB metadata encryption domain
3. FDB Encryption Header domain - used to generate digest for
plaintext EncryptionHeader.
The scheme doesn't support encryption if an entity can't be categorized
into any of above mentioned encryption domains, for instance, non-tenant
mutations are NOT supported.
Patch extend the encryption support for mutations for which corresponding
Tenant information can't be obtained (Key length shorter than TenantPrefix)
and/or mutations do not belong to any valid Tenant
(FDB management cluster data) by mapping such mutations to a
"default encryption domain".
TODO
CommitProxy driven TLog encryption implementation requires every transaction
mutation to contain 1 KV, not crossing Tenant-boundaries. Only exception to
this rule is ClearRange mutations. For now ClearRange mutations are mapped
to 'default encryption domain', in subsequent patch appropriate handling
for ClearRange mutations shall be proposed.
Testing
devRunCorrectness - 100k
Adding the following metrics:
* BlobCipherKeyCache hit/miss
* EKP: KMS requests latencies
* For each component that using encryption, they now need to pass a UsageType enum to the encryption helper methods (GetEncryptCipherKeys/GetLatestEncryptCipherKey/encrypt/decrypt) and those methods will help to log get cipher key latency samples and encryption/decryption cpu times accordingly.
* Assert that arena's appear last in serializer calls
* Fix all occurrences of Arena's not appearing last in serializer call
* Work around issue from Standalone inheriting from Arena privately
* Attempt to fix windows build
Use fb_ prefix instead of detail namespace to scope implementation
details in headers
* Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (no need to guard)
* Enable the relaxed special key access in transactions that need to use special key-space APIs introduced in 7.2
Valgrind is complaining about use of uninitialized memory in
absl::GetStackTrace, and the cases where it complains the backtraces are
incomplete. Note: this means that jemalloc heap profiling no longer
works out of the box. Advanced users who want to enable jemalloc heap
profiling will now have to revert this change and build from source.
* flow: add ApiVersion to replace hard coding api version
Instead of hard coding api value, let's rely on feature versions akin to
ProtocolVersion.
* ApiVersion: remove use of -1 for latest and use LATEST_VERSION
* Move arena members to the end of serializer calls
See
https://github.com/apple/foundationdb/tree/main/flow#flatbuffersobjectserializer
for why this is necessary.
* Fix a heap-use-after-free
Previously memory owned by
EncryptKeyProxyData::baseCipherDomainIdKeyIdCache was borrowed by a call
to EncryptKeyProxyData::insertIntoBaseDomainIdCache where it was
invalidated and then used. Now
EncryptKeyProxyData::insertIntoBaseDomainIdCache takes shared ownership
by taking a Standalone.
And also rename some types to end in Ref to follow the flow conventions
described here: https://github.com/apple/foundationdb/tree/main/flow#arenas
* Make envvar access in envvar-knob parsing cross-platform
Variable `environ' is defined for Linux,
does not exist in Mac, and is deprecated in Windows
* Prevent env knob strings from being copied at each iteration
* Add headers for {Get|Free}EnvironmentStrings()
* Adjust for different memory layout of environ vs GetEnvironmentStrings()
GetEnvironmentStrings returns a single byte string
with adjacent key=value strings delimited with null bytes,
whereas environ is an array of key=value strings.
* Fix missing return value for Windows build
* Have just one return statement
* Implement TenantCacheEntry in-memory cache
Description
diff-4: TraceEvent usage improvements
diff-3: Address review comments
diff-2: Add APIs to read counter values, test improvements
diff-1: Address review comments
Major changes includes:
1. Implements an actor that enables an in-memory caching of
TenantCacheEntry object, allowing the caller to embed custom
information along with TenantCacheEntry.
2. The cache follows read-through cache semantics where the entry
gets loaded from underlying database on a miss.
3. The cache implements a "periodic poller" to refresh known Tenants
by consulting the database. Once a database keyrange-watch feature is
available, cache shall be updated.
Bonus:
Implement a 'recurringAsync' addition to genericActors allowing caller
to schedule a periodic task registering an "actor functor"; the routine
'waits' for the actor unlike existing 'recurring' implementation.
Testing
TenantEntryCache workload
devCorrectnessRun - 100K
* Update network address in trace logs; Add system monitor for flowprocess
* Create a new trace file with the correct process address for flowprocess
* Remove unused debugging traces
* Add a new error lock_file_failure; Change please_reboot_remote_kv_store to please_reboot_kv_store; Add the code to only reboot the kv store but not the worker; Remove some unnecessay traces
* Add error handling for file_not_found in handleIOErrors
* Format worker.actor.cpp file
Steve Atherton