* EaR: Update KMS URL refresh policy and fix bugs
Description
RESTKmsConnector implements discovery and refresh semantics i.e.
on bootstrap it discovers KMS Urls and periodically refresh the
URLs (handle server upgrade scenario). The current implementation
caches the URLs in a min-heap, as part of serving a request, actor
pops out elements from min-heap and attempts connecting to the server,
on failure, the URL is temporarily stored in a stack, at the end of
the request processing, the stack is merged back into the heap.
The code doesn't work as expected if there are multiple requests
consumes the heap causing following issues:
1. Min-heap would retain old URLs replaced by latest refresh (stack merge)
2. URL discovery file is read more than expected as multiple requests can
empty heap, causing the code to read URLs from the file.
Patch proposes following policy to cache and maintain URLs priority:
1. Unresponsiveness penalty: KMS flaky connection or overload can cause
requests to timeout or fail; each such instance updates unresponsiveness
penalty of associated URL context. Further, the penalty is time bound and
deteriorate with time.
2. Cached URLs are sorted once a failure is encountered, priority followed
is:
2.1. Unresponsiveness penalty server(s) least preferred
2.2. Server(s) with high total-failures less preferred
2.3. Server(s) with high total-malformed response less preferred.
3. Updates RESTClient to throw 'retryable' error up to the client such as:
'connection_failed' and/or 'timeout'
4. Extend RESTUrl to support IPv6 format.
Testing
RESTUnit - 100K (new test added for coverage)
devRunCorrectness
* EaR: reduce metrics logging
BlobCipherMetrics used to break down by usage types (whehter it is for tlog, redwood, backup, etc), and these counters will be printed to trace log even when encryption is not enabled, or the specific usage is not happening on a node (e.g. a node with only stateless roles will also print blob cipher counters for redwood). We are reducing the BlobCipherMetrics loggings by:
1. Default to not breakdown the metrics by usage type, and the behavior is controlled by the knob `ENCRYPT_KEY_CACHE_ENABLE_DETAIL_LOGGING`
2. When the detail breakdown is enabled, the counters are lazily initialize
3. Even if the counters are initialized, they will not be logged if the count is 0 (so like if a node was recruited as tlog but then drops the tlog role later on, the tlog counter inside BlobCipherMetrics will not be logged anymore).
* buggify BlobCipherMetrics detail logging knob
* format
* EaR: Add test case to validate decryption with invalid key
Description
Extend BlobCipher unit test to provide coverage for the scenario
where buffer got encrypted with a EncryptionKey K, however,
decryption for some reason got attempted with K'.
Testing
EncryptionUnit.toml - 100K
* EaR: Add test case to validate decryption with invalid key
Description
Address review comments
Testing
* added operational metrics and some polish
* moving consistency scan enablement in simulation tests to main tester workflow
* more stats and throttling polish
Make a local copy of the promise before calling `send` in case the
promise gets destroyed as a result of fulfilling it.
This issue was previously fixed for sending errors to the `result`
promise, but it was never fixed when fulfilling the promise. The issue
manifested as an invalid generation returned when running a `set`
against the configuration database immediately followed by a `get` with
a new transaction object.
* Return const references in PTree accessors
Many usages do not require copying the reference (and incurring the
ref-counting overhead)
* Remove unnecessary refcounting for rotating ptree
* adding consistency scan clear stats and testing in simulation
* Adding test that intentionally injects corruption in consistency scan requests and ensures the scan finds it
* cleanup
* adding assert false to disabled code
* list audits
* cancel audits and corresponding tests
* make audit storage dblock aware
* increase audit retry since we are able to cancel
* fix updateAuditState and fdb github ci
* fmt
* fix fdbcli audit_storage and fix CI issue
* fix fdb cli
* address comments
* fmt
* Added location_metadata fdbcli to query shard locations, assignements, numbers etc.
* Added `listshards` to get some random physical/non-physical shards.
* Resolved comments.
* [fdbserver] workaround the FRT type layout issue to get Swfit getVersion working
* MasterData.actor.h: fix comment typo
* masterserver.swift: some tweaks
* masterserver.swift: remove getVersion function, use the method
* masterserver.swift: print replied version to output for tracing
* [swift] add radar links for C++ interop issues found in getVersion bringup
* Update fdbserver.actor.cpp
* Migrate MasterData closer to full reference type
This removes the workaround for the FRT type layout issue, and gets us closer to making MasterData a full reference type
* [interop] require a new toolchain (>= Oct 19th) to build
* [Swift] fix computation of toAdd for getVersion Swift implementation
* add Swift to FDBClient and add async `atLeast` to NotifiedVersion
* fix
* use new atLeast API in master server
* =build fixup link dependencies in swift fdbclient
* clocks
* +clock implement Clock using Flow's notion of time
* [interop] workaround the immortal retain/release issue
* [swift] add script to get latest centos toolchain
* always install swift hooks; not only in "test" mode
* simulator - first thing running WIP
* cleanups
* more cleanup
* working snapshot
* remove sim debug printlns
* added convenience for whenAtLeast
* try Alex's workaround
* annotate nonnull
* cleanup clock a little bit
* fix missing impls after rebase
* Undo the swift_lookup_Map_UID_CommitProxyVersionReplies workaround
No longer needed - the issue was retain/release
* [flow][swift] add Swift version of BUGGIFY
* [swiftication] add CounterValue type to provide value semantics for Counter types on the Swift side
* remove extraneous requestingProxyUID local
* masterserver: initial Swift state prototype
* [interop] make the Swiftied getVersion work
* masterserver - remove the C++ implementation (it can't be supported as state is now missing)
* Remove unnecessary SWIFT_CXX_REF_IMMORTAL annotations from Flow types
* Remove C++ implementation of CommitProxyVersionReplies - it's in Swift now
* [swift interop] remove more SWIFT_CXX_REF_IMMORTAL
* [swift interop] add SWIFT_CXX_IMMORTAL_SINGLETON_TYPE annotation for semanticly meaningful immortal uses
* rename SWIFT_CXX_REF_IMMORTAL -> UNSAFE_SWIFT_CXX_IMMORTAL_REF
* Move master server waitForPrev to swift
* =build fix linking swift in all modules
* =build single link option
* =cmake avoid manual math, just get "last" element from list
* implement Streams support (#18)
* [interop] update to new toolchain #6
* [interop] remove C++ vtable linking workarounds
* [interop] make MasterData proper reference counted SWIFT_CXX_REF_MASTERDATA
* [interop] use Swift array to pass UIDs to registerLastCommitProxyVersionReplies
* [interop] expose MasterServer actor to C++ without wrapper struct
* [interop] we no longer need expose on methods 🥳
* [interop] initial prototype of storing CheckedContinuation on the C++ side
* Example of invoking a synchronous swift function from a C++ unit test. (#21)
* move all "tests" we have in Swift, and priority support into real modules (#24)
* Make set continuation functions inline
* Split flow_swift into flow_swift and flow_swift_future to break circular dependency
* rename SwiftContinuationCallbackStruct to FlowCallbackForSwiftContinuation
* Future interop: use a method in a class template for continuation set call
* Revert "Merge pull request #22 from FoundationDB/cpp-continuation" (#30)
* Basic Swift Guide (#29)
Co-authored-by: Alex Lorenz <arphaman@gmail.com>
* Revert "Revert "Merge pull request #22 from FoundationDB/cpp-continuation" (#30)"
This reverts commit c025fe6258.
* Restore the C++ continuation, but it seems waitValue is broken for CInt somehow now
* disable broken tests - waitValue not accessible
* Streams can be async iterated over (#27)
Co-authored-by: Alex Lorenz <arphaman@gmail.com>
* remove work in progress things (#35)
* remove some not used (yet) code
* remove expose func for CInt, it's a primitive so we always have witness info (#37)
* +masterdata implement provideVersions in Swift (#36)
* serveLiveCommittedVersion in Swift (#38)
* Port updateLiveCommittedVersion to swift (#33)
Co-authored-by: Konrad `ktoso` Malawski <konrad_malawski@apple.com>
* Implement updateRecoveryData in Swift (#39)
Co-authored-by: Alex Lorenz <arphaman@gmail.com>
* Simplify flow_swift to avoid multiple targets and generate separate CheckedContinuation header
* Uncomment test which was blocked on extensions not being picked up (#31)
* [interop] Use a separate target for Swift-to-C++ header generation
* reduce boilerplate in future and stream support (#41)
* [interop] require interop v8 - that will fix linker issue (https://github.com/apple/swift/issues/62448)
* [interop] fix swift_stream_support.h Swift include
* [interop] bump up requirement to version 9
* [interop] Generalize the Flow.Optional -> Swift.Optional conversion using generics
* [WIP] masterServer func in Swift (#45)
* [interop] Try conforms_to with a SWIFT_CONFORMS_TO macro for Optional conformance (#49)
* [interop] include FlowOptionalProtocol source file when generating Flow_CheckedContinuation.h
This header generation step depends on the import of the C++ Flow module, which requires the presence of FlowOptionalProtocol
* conform Future to FlowFutureOps
* some notes
* move to value() so we can use discardable result for Flow.Void
* make calling into Swift async funcs nicer by returning Flow Futures
* [interop] hide initial use of FlowCheckedContinuation in flow.h to break dependency cycle
* [fdbserver] fix an EncryptionOpsUtils.h modularization issue (showed up with modularized libc++)
* Pass GCC toolchain using CMAKE_Swift_COMPILE_EXTERNAL_TOOLCHAIN to Swift's clang importer
* [interop] drop the no longer needed libstdc++ include directories
* [cmake] add a configuration check to ensure Swift can import C++ standard library
* [swift] include msgpack from msgpack_DIR
* [interop] make sure the FDB module maps have 'export' directive
* add import 'flow_swift' to swift_fdbserver_cxx_swift_value_conformance.swift
This is needed for CONFORMS_TO to work in imported modules
* make sure the Swift -> C++ manually bridged function signature matches generated signature
* [interop][workaround] force back use of @expose attribute before _Concurrency issue is fixed
* [interop] make getResolutionBalancer return a pointer to allow Swift to use it
We should revert back to a reference once compiler allows references again
* [interop] add a workaround for 'pop' being marked as unsafe in Swift
* masterserver.swift: MasterData returns the Swift actor pointer in an unsafe manner
* Add a 'getCopy' method to AsyncVar to make it more Swift friendly
* [interop] bump up the toolchain requirement
* Revert "[interop][workaround] force back use of @expose attribute before _Concurrency issue is fixed"
This reverts commit b01b271a76.
* [interop] add FIXME comments highlighting new issue workarounds
* [interop] adopt the new C++ interoperability compiler flag
* [interop] generate swift compile commands
* Do not deduplicate Swift compilation commands
* [interop] generate swift compile commands
* Do not deduplicate Swift compilation commands
* flow actorcompiler.h: add a SWIFT_ACTOR empty macro definition
This is needed to make the actor files parsable by clangd
* [cmake] add missing dependencies
* experimental cross compile
* [cmake] fix triple in cross-compiled cmake flags
* [interop] update to interop toolchain version 16
* [x-compile] add flags for cross-compiling boost
* cleanup x-compile cmake changes
* [cmake] fix typo in CMAKE_Swift_COMPILER_EXTERNAL_TOOLCHAIN config variable
* [interop] pass MasterDataActor from Swift to C++ and back to Swift
* [fdbserver] Swift->C++ header generation for FDBServer should use same module cache path
* Update swift_get_latest_toolchain.sh to fetch 5.9 toochains
* set HAVE_FLAG_SEARCH_PATHS_FIRST for cross compilation
* Resolve conflicts in net2/sim2/actors, can't build yet
* undo SWIFT_ACTOR changes, not necessary for merge
* guard c++ compiler flags with is_cxx_compile
* Update flow/actorcompiler/ActorParser.cs
Co-authored-by: Evan Wilde <etceterawilde@gmail.com>
* update the boost dependency
* Include boost directory from the container for Swift
* conform flow's Optional to FlowOptionalProtocol again
* Guard entire RocksDBLogForwarder.h with SSD_ROCKSDB_EXPERIMENTAL to avoid failing on missing rocksdb APIs
* remove extraneous merge marker
* [swift] update swift_test_streams.swifto to use vars in more places
* Add header guard to flow/include/flow/ThreadSafeQueue.h to fix moduralization issue
* Update net and sim impls
* [cmake] use prebuilt libc++ boost only when we're actually using libc++
* [fdbserver] Swift->C++ header generation for FDBServer should use same module cache path
* fixups after merge
* remove CustomStringConvertible conformance that would not be used
* remove self-caused deprecation warnings in future_support
* handle newly added task priority
* reformatting
* future: make value() not mutating
* remove FIXME, not needed anymore
* future: clarify why as functions
* Support TraceEvent in Swift
* Enable TraceEvent using a class wrapper in Swift
* prearing WITH_SWIFT flag
* wip disabled failing Go stuff
* cleanup WITH_SWIFT_FLAG and reenable Go
* wip disabled failing Go stuff
* move setting flag before printing it
* Add SWIFT_IDE_SETUP and cleanup guides and build a bit
* Revert "Wipe packet buffers that held serialized WipedString (#10018)"
This reverts commit e2df6e3302.
* [Swift] Compile workaround in KeyBackedRangeMap; default init is incorrect
* [interop] do not add FlowFutureOps conformance when building flow clang module for Flow checked continuation header pre-generation
* make sure to show -DUSE_LIBCXX=OFF in readme
* readme updates
* do not print to stderr
* Update Swift and C++ code to build with latest Swift 5.9 toolchain now that we no longer support universal references and bridge the methods that take in a constant reference template parameter correctly
* Fix SERVER_KNOBS and enable use them for masterserver
* Bump to C++20, Swift is now able to handle it as well
* Put waitForPrev behind FLOW_WITH_SWIFT knob
* Forward declare updateLiveCommittedVersion
* Remove unused code
* fix wrong condition set for updateLiveCommittedVersion
* Revert "Revert "Wipe packet buffers that held serialized WipedString (#10018)""
This reverts commit 5ad8dce052.
* Enable go-bindings in cmake
* Revert "Revert "Wipe packet buffers that held serialized WipedString (#10018)""
This reverts commit 5ad8dce052.
* USE_SWIFT flag so we "build without swift" until ready to by default
* uncomment a few tests which were disabled during USE_SWIFT enablement
* the option is WITH_SWIFT, not USE
* formatting
* Fix masterserver compile error
* Fix some build errors.
How did it not merge cleanly? :/
* remove initializer list from constructor
* Expect Swift toolchain only if WITH_SWIFT is enabled
* Don't require Flow_CheckedContinuation when Swift is disabled
* Don't compile FlowCheckedContinuation when WITH_SWIFT=OFF
* No-op Swift macros
* More compile guards
* fix typo
* Run clang-format
* Guard swift/bridging include in fdbrpc
* Remove printf to pass the test
* Remove some more printf to avoid potential issues
TODO: Need to be TraceEvents instead
* Remove __has_feature(nullability) as its only used in Swift
* Don't use __FILENAME__
* Don't call generate_module_map outside WITH_SWIFT
* Add some more cmake stuff under WITH_SWIFT guard
* Some more guards
* Bring back TLSTest.cpp
* clang-format
* fix comment formatting
* Remove unused command line arg
* fix cmake formatting in some files
* Address some review comments
* fix clang-format error
---------
Co-authored-by: Alex Lorenz <arphaman@gmail.com>
Co-authored-by: Russell Sears <russell_sears@apple.com>
Co-authored-by: Evan Wilde <etceterawilde@gmail.com>
Co-authored-by: Alex Lorenz <aleksei_lorenz@apple.com>
Co-authored-by: Vishesh Yadav <vishesh_yadav@apple.com>
Co-authored-by: Vishesh Yadav <vishesh3y@gmail.com>
Description
Optimize logging emitted from GetEncryptCipherKey module,
especially the one more useful for debugging and not very useful
in the production
Testing
SwizzledRollbackSideBand - randomSeed (276500218)
devRunCorrectness - 100k
When buggify is enabled, it's possible the version map has 5 entries, which is
larger than BACKUP_MAP_KEY_LOWER_LIMIT, causing the range task to be delayed
infinitely: the BackupRangeTaskFunc::_execute() skips the execution and
schedules the task to be added back in BackupRangeTaskFunc::_finish().
Reproduction:
Seed: -f ./tests/slow/SharedDefaultBackupCorrectness.toml -s 3202874095 -b on
-f ./tests/slow/VersionStampBackupToDB.toml -s 1190111003 -b on
Commit: 6e5773dd5 at release-7.3
Build: clang
* Make CodeProbeImpl::_hitCount atomic
* Structure access to TraceLog::logTraceEventMetrics so that it is written before a trace log is opened and only read from one thread after it is opened.
* Fix condition in assert
* Rename TraceLog::log to logMetrics and move initialization of trace log metrics into TraceLog::open
---------
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
The blob worker needs more time to catchup, about 388s in the failed simulation
test.
Reproduction:
seed: -f ./tests/slow/BlobGranuleVerifyLargeClean.toml -s 4068151139 -b on
commit: 3bdd71cb0 at release-7.3 branch
build: gcc
* Test watch cleanup on cancel
* Fix clearing the database in Java integration tests
* Always cancel the futures wrapped by MVC abortable futures
* More tests for watch cleanup
* Fix clear database database in some Java integration tests
* improve audit throughput
* if ssshard fails do audit due to ssi failure, then global retry is required
* fix a trace event name
* fix budget release in doAudit
* avoid throttling in general simultion tests
* fix doAuditOnStorageServer throw error
* avoid starting a task that has been complete
* when ddaudit ssshard failed, check if ssi is removed, if yes, silently exit
* fix trace detail name of AuditUtilStorageServerRemovedEnd evenrt
* redo schedule in doAuditOnStorageServer
* schedule does not wait doAudit
* remove TESTING_AUDIT_STORAGE_THROTTLING
* ssaudit stops proceeding if ddauditstate is not in running phase
* make tester audit storage only happen when simulation, and randomly set CONCURRENT_AUDIT_TASK_COUNT_MAX
* Remove duplicate getRange() for DB handles and update existing GetRange to accept DB handles.
* Initial progress checkpoint on new ConsistencyScan role.
* Updated TODOs, finished most if not all state updates.
* placeholder
* Add more TODOs, documentation and comment improvements.
* Checkpoint round state to avoid advancing progress if commit fails.
* Bug fix, check is supposed to be for overlap, not lack of overlap.
* Added more TODO's and added faked read results / exceptions and faked DB size retrieval to prove the consistencyScanCore logic works.
* Update JSON schemas and command help.
* Add comment about lifetime stats reset.
* More TODO comments and some renames for clarity, some bug fixes.
* properly stopping consistency scan in simulation so that it doesn't run forever and cause quiet database to fail
* removing trailing comma from consistency_scan json schema
* Making CC inconsistency not an error if it's intentional tss corruption
* consistency scan actually reads storage locations
* added check that consistency scan actually completes a round in simulation, fixed bug and added debugging around consistency scan getting stuck
* made consistency scan properly fetch database size
* refactoring data check to be used in both consistency scan and consistency check
* checking that consistency scan always completes at least one round and doesn't get stuck
* cleanup
* fixing ide build
* consistencyscan fdbcli command wasn't actually changing db state
* consistencyscan fdbcli command always said enabled even when it wasn't
---------
Co-authored-by: Steve Atherton <steve.atherton@snowflake.com>
* EaR: REST based Simulated KMS Vault request hanlder interface
Description
diff-1: Address review comments
Improve unit test case coverage
diff-2: Extend RESTKmsConnectorUtil to generate HTTP::Header
EaR simulation testing is currently driven using SimKmsConnector
interface, it exposes endpoints directly invoked by EKP to fetch
encryption keys. Approach avoids testing RESTKms communication
path. Recently FDB codebase got extended by adding HTTPServer
interface, which was a gap prohibiting end-to-end testing of
EaR code.
Patch proposes following changes:
1. Refactor RESTKmsConnector to move common code and definitions
to RESTKmsConnectorUtil namespace
2. Introduce RESTSimKmsVault accepting HTTP format requests and
providing appropriate HTTP response.
Testing
RESTUnit 100K + 5k valgrind
devRunCorrectness 100K
Testing
* when trigger doAuditOnStorageServer, check remainingBudgetForAuditTasks
* add trace event of audit progress
* address comments
* code clean up
* make dispatch and schedule audit be more clear
* make dispatch and schedule audit be more clear 2
* make dispatch and schedule audit be more clear 3
* address comments
* Add networkoption to disable non-TLS connections
* add disable plaintext connection to fdbserver
* python doc
* Formatting
* Add tls disable plaintext connection to client api test
* review
* fix negative test
* formatting
* add TLS support to c client config tests
Adds support for TLS in the client and server separately
* add tests for disable_plaintext_connections
Test TLS and Plaintext Clusters and Clients
* Fix documentation
* Rename option to indicate it is client-only
* clearer formatting
* default to allowing plaintext connections
* add SetTLSDisablePlaintextConnection to go bindings
* clean up old audit metadata
* change comments
* fix audit cleanup rule as PR description claim and reduce timeout of auditStorageCorrectness in tester
* address comment
* clear audit metadata should not throw error
* cleanup progress metadata by type
* control number of AuditStatistic events
* carefully persist new audit state
* add unit tests and fix issues
* cleanup
* allow audit concurrent run for different types and fix some bug in auditutl
* fix ci issue and nits
* Added `get_audit_status checkmigration` to print out the number of data shards and `physical shards`, so that we know the progress of migration to `shard_encode_location_metadata`
* Fixed print format.
* Addressed comments.
* fixing bugs with tenant_mode required on external clients and changing test to find them
* Update fdbcli/BlobKeyCommand.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
---------
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* buggified max_shards_on_large_teams, and had the consistency scan verify the proper number of shards have been overreplicated
* fix: when restarting the data distributor, do no allow more than max_shards_on_large_teams shards to be marked as healthy
* cleanup audit progress metadata and tester directly issue audit requests to DD instead of CC
* address comments and fix test dd issue request but dd not present
Jay Zhuang