Commit Graph

740 Commits

Author SHA1 Message Date
Sepeth 3854dbfe4d
Upgrade fmt from 8.1.1 to 11.0.2 (#11601)
And, added via cmake FetchContent, and removed contrib/fmt-8.1.1
2024-09-10 14:42:43 -07:00
Vishesh Yadav a84026cca5
[testing] Automatically discover unit-test and register as ctest (#11612)
* [testing] Automatically discover unit-test and register as ctest

This patch adds `collect_unit_tests()` to CMake which searches over
the codebase and finds all the unit-tests written using Flow's TEST_CASE
macro and register as ctest.

The test then can be then run using ctest command or directly via Test
Explorer in VSCode.

* Update CMakeLists.txt

* Check failed tests

* Update TestDirectory.py to create more unique directory

* Put the feature behind flag
2024-09-04 11:31:49 -07:00
Xiaoge Su 4c94acbd53 fixup! 2024-08-16 09:46:06 -07:00
Xiaoge Su 8197348f95 Add flag for custom jemalloc build 2024-08-16 09:45:55 -07:00
Syed Paymaan Raza ae39e18b7d
Allow exporting rocksdb compile commands (#11560) 2024-08-06 12:46:39 -07:00
Vishesh Yadav 6cd5ad2ffe Add code back 2024-07-10 18:52:14 -07:00
Vishesh Yadav 591efa1d1b Remove swift 2024-06-29 11:11:48 -07:00
Jingyu Zhou 8b99af2057 Remove old test harness binaries from correctness packages
Since we are using TestHarness2 now.
2024-05-22 08:46:07 -07:00
Jingyu Zhou cc31013545
Merge pull request #11333 from xis19/main
Fix the python package issues and go test failure
2024-04-29 17:30:43 -07:00
Xiaoge Su 2091f8dae7 fixup! Fix not found issue caused by abuse of Python3_EXECUTABLE variable 2024-04-29 14:15:25 -07:00
Yao Xiao 99910100a5 versoin upgrade 2024-04-26 14:53:01 -05:00
Xiaoge Su 38adabf8df fixup! Fix the tests related to python 2024-04-22 18:44:32 -07:00
Xiaoge Su 010d070de0 Modernize the Python3 package build configuration 2024-04-22 18:44:32 -07:00
Aaron Molitor 38bb833d41 updates to add fdb-kubernetes-monitor to the standard build flow 2024-04-12 08:31:48 -05:00
Yao Xiao 4d027a05e9
Upgrade to RocksDB 8.10.0. (#11174) 2024-02-07 12:41:34 -08:00
Dimitris Apostolou a88114c222
Fix typos 2024-02-07 01:16:00 +02:00
mavenraven 8fea64db80 Makes Jinja2 a required dependency of the build. 2024-01-27 17:19:33 -05:00
Jingyu Zhou e2fc5f3f1c
Merge pull request #10935 from drTr0jan/msgpack
GetMsgpack: add 4+ versions of upstream
2024-01-17 13:02:00 -08:00
Markus Pilman e07b3e35ca Added C++ Coroutine support to Flow 2023-11-14 10:10:11 +01:00
Yao Xiao 33a29ddd85
Upgrade RocksDB version and disable CF range deletion optimization. (#11045)
* Upgrade RocksDB version and disable CF range deletion optimization.wq

* Disable iterator.
2023-11-02 17:25:11 -07:00
Jingyu Zhou a083d81737 Fix MacOS compiling errors 2023-10-23 12:32:05 -07:00
Boris Korzun c35a23d3f6 GetMsgpack: add 4+ versions of upstream 2023-09-26 18:16:57 +03:00
Xiaoge Su 91ec1fdf10 Provide actor call backtrace
See design/AcAC.md
2023-09-19 20:58:33 -07:00
Xiaoge Su b343c37246 Bring Boost Valgrind flags back
The issue caused Boost build failure is found and fixed at #10733. Here
the flag BOOST_USE_VALGRIND is reintroduced back.
2023-08-07 13:13:23 -07:00
Evan Wilde f66d45ca7c
Shadow Cxx flags for Boost build
Creating a shadow copy of the C++ and linker flags for the boost build
because configure_file cannot expand generator expressions.
`file(GENERATE...` can in theory, but CMake runs into memory corruption
when it tries.
2023-08-04 09:36:01 -07:00
Jingyu Zhou ae7fe844d2
Merge pull request #10689 from xis19/rocksdbflag
Add a flag for enabling/disabling RocksDB storage engine in simulations
2023-08-01 10:50:09 -07:00
Xiaoge Su a52cc2ba2e Fix the ConfigureCompiler.cmake
1. The original lines are problematic -- <> are not paired.
2. Fixing the original line by pairing <> would cause build failure if
   valgrind is turned on.
3. git grep BOOST_USE_VALGRIND returned only one single usage, which is
   in the cmake file.
4. Removing the lines caused build success.
2023-07-27 23:53:23 -07:00
Xiaoge Su 0036bab8a4 Rename SSD_ROCKSDB_EXPERIMENTAL to WITH_ROCKSDB 2023-07-27 14:04:10 -07:00
Xiaoge Su e655feb07a Remove WITH_ROCKSDB_EXPERIMENTAL, enforce using enum for StorageEngine type in simulation 2023-07-27 10:31:48 -07:00
Xiaoge Su f1a92047db
Report to stdout if DTrace is enabled (#10474) 2023-07-25 13:59:08 -07:00
Alex Lorenz 448ea9d507
[swift] fix Swift specific build regressions (#10598)
* Always set WITH_SWIFT clang preprocessor option when building Swift code

* [swift][cmake] ensure that -gdwarf-4 is only passed for C++ files

* [swift] migrate to SWIFT_CONFORMS_TO_PROTOCOL from swift/bridging header correctly

* [swift] add rapidjson header search dir for fdbserver_swift

* [swift] fix clang diagnostics in Swift

* [swift] formatting fixes for CI

* Guard SWIFT_CONFORMS_TO_PROTOCOL in WITH_SWIFT block

* add additional WITH_SWIFT guard
2023-07-17 17:54:53 -05:00
Konrad `ktoso` Malawski c26aa0b2a3
Introduce initial Swift support in fdbserver (#10156)
* [fdbserver] workaround the FRT type layout issue to get Swfit getVersion working

* MasterData.actor.h: fix comment typo

* masterserver.swift: some tweaks

* masterserver.swift: remove getVersion function, use the method

* masterserver.swift: print replied version to output for tracing

* [swift] add radar links for C++ interop issues found in getVersion bringup

* Update fdbserver.actor.cpp

* Migrate MasterData closer to full reference type

This removes the workaround for the FRT type layout issue, and gets us closer to making MasterData a full reference type

* [interop] require a new toolchain (>= Oct 19th) to build

* [Swift] fix computation of toAdd for getVersion Swift implementation

* add Swift to FDBClient and add async `atLeast` to NotifiedVersion

* fix

* use new atLeast API in master server

* =build fixup link dependencies in swift fdbclient

* clocks

* +clock implement Clock using Flow's notion of time

* [interop] workaround the immortal retain/release issue

* [swift] add script to get latest centos toolchain

* always install swift hooks; not only in "test" mode

* simulator - first thing running WIP

* cleanups

* more cleanup

* working snapshot

* remove sim debug printlns

* added convenience for whenAtLeast

* try Alex's workaround

* annotate nonnull

* cleanup clock a little bit

* fix missing impls after rebase

* Undo the swift_lookup_Map_UID_CommitProxyVersionReplies workaround

No longer needed - the issue was retain/release

* [flow][swift] add Swift version of BUGGIFY

* [swiftication] add CounterValue type to provide value semantics for Counter types on the Swift side

* remove extraneous requestingProxyUID local

* masterserver: initial Swift state prototype

* [interop] make the Swiftied getVersion work

* masterserver - remove the C++ implementation (it can't be supported as state is now missing)

* Remove unnecessary SWIFT_CXX_REF_IMMORTAL annotations from Flow types

* Remove C++ implementation of CommitProxyVersionReplies - it's in Swift now

* [swift interop] remove more SWIFT_CXX_REF_IMMORTAL

* [swift interop] add SWIFT_CXX_IMMORTAL_SINGLETON_TYPE annotation for semanticly meaningful immortal uses

* rename SWIFT_CXX_REF_IMMORTAL -> UNSAFE_SWIFT_CXX_IMMORTAL_REF

* Move master server waitForPrev to swift

* =build fix linking swift in all modules

* =build single link option

* =cmake avoid manual math, just get "last" element from list

* implement Streams support (#18)

* [interop] update to new toolchain #6

* [interop] remove C++ vtable linking workarounds

* [interop] make MasterData proper reference counted SWIFT_CXX_REF_MASTERDATA

* [interop] use Swift array to pass UIDs to registerLastCommitProxyVersionReplies

* [interop] expose MasterServer actor to C++ without wrapper struct

* [interop] we no longer need expose on methods 🥳

* [interop] initial prototype of storing CheckedContinuation on the C++ side

* Example of invoking a synchronous swift function from a C++ unit test. (#21)

* move all "tests" we have in Swift, and priority support into real modules (#24)

* Make set continuation functions inline

* Split flow_swift into flow_swift and flow_swift_future to break circular dependency

* rename SwiftContinuationCallbackStruct to FlowCallbackForSwiftContinuation

* Future interop: use a method in a class template for continuation set call

* Revert "Merge pull request #22 from FoundationDB/cpp-continuation" (#30)

* Basic Swift Guide (#29)

Co-authored-by: Alex Lorenz <arphaman@gmail.com>

* Revert "Revert "Merge pull request #22 from FoundationDB/cpp-continuation" (#30)"

This reverts commit c025fe6258.

* Restore the C++ continuation, but it seems waitValue is broken for CInt somehow now

* disable broken tests - waitValue not accessible

* Streams can be async iterated over (#27)

Co-authored-by: Alex Lorenz <arphaman@gmail.com>

* remove work in progress things (#35)

* remove some not used (yet) code

* remove expose func for CInt, it's a primitive so we always have witness info (#37)

* +masterdata implement provideVersions in Swift (#36)

* serveLiveCommittedVersion in Swift (#38)

* Port updateLiveCommittedVersion to swift (#33)

Co-authored-by: Konrad `ktoso` Malawski <konrad_malawski@apple.com>

* Implement updateRecoveryData in Swift (#39)

Co-authored-by: Alex Lorenz <arphaman@gmail.com>

* Simplify flow_swift to avoid multiple targets and generate separate CheckedContinuation header

* Uncomment test which was blocked on extensions not being picked up (#31)

* [interop] Use a separate target for Swift-to-C++ header generation

* reduce boilerplate in future and stream support (#41)

* [interop] require interop v8 - that will fix linker issue (https://github.com/apple/swift/issues/62448)

* [interop] fix swift_stream_support.h Swift include

* [interop] bump up requirement to version 9

* [interop] Generalize the Flow.Optional -> Swift.Optional conversion using generics

* [WIP] masterServer func in Swift (#45)

* [interop] Try conforms_to with a SWIFT_CONFORMS_TO macro for Optional conformance (#49)

* [interop] include FlowOptionalProtocol source file when generating Flow_CheckedContinuation.h

This header generation step depends on the import of the C++ Flow module, which requires the presence of FlowOptionalProtocol

* conform Future to FlowFutureOps

* some notes

* move to value() so we can use discardable result for Flow.Void

* make calling into Swift async funcs nicer by returning Flow Futures

* [interop] hide initial use of FlowCheckedContinuation in flow.h to break dependency cycle

* [fdbserver] fix an EncryptionOpsUtils.h modularization issue (showed up with modularized libc++)

* Pass GCC toolchain using CMAKE_Swift_COMPILE_EXTERNAL_TOOLCHAIN to Swift's clang importer

* [interop] drop the no longer needed libstdc++ include directories

* [cmake] add a configuration check to ensure Swift can import C++ standard library

* [swift] include msgpack from msgpack_DIR

* [interop] make sure the FDB module maps have 'export' directive

* add import 'flow_swift' to swift_fdbserver_cxx_swift_value_conformance.swift

This is needed for CONFORMS_TO to work in imported modules

* make sure the Swift -> C++ manually bridged function signature matches generated signature

* [interop][workaround] force back use of @expose attribute before _Concurrency issue is fixed

* [interop] make getResolutionBalancer return a pointer to allow Swift to use it

We should revert back to a reference once compiler allows references again

* [interop] add a workaround for 'pop' being marked as unsafe in Swift

* masterserver.swift: MasterData returns the Swift actor pointer in an unsafe manner

* Add a 'getCopy' method to AsyncVar to make it more Swift friendly

* [interop] bump up the toolchain requirement

* Revert "[interop][workaround] force back use of @expose attribute before _Concurrency issue is fixed"

This reverts commit b01b271a76.

* [interop] add FIXME comments highlighting new issue workarounds

* [interop] adopt the new C++ interoperability compiler flag

* [interop] generate swift compile commands

* Do not deduplicate Swift compilation commands

* [interop] generate swift compile commands

* Do not deduplicate Swift compilation commands

* flow actorcompiler.h: add a SWIFT_ACTOR empty macro definition

This is needed to make the actor files parsable by clangd

* [cmake] add missing dependencies

* experimental cross compile

* [cmake] fix triple in cross-compiled cmake flags

* [interop] update to interop toolchain version 16

* [x-compile] add flags for cross-compiling boost

* cleanup x-compile cmake changes

* [cmake] fix typo in CMAKE_Swift_COMPILER_EXTERNAL_TOOLCHAIN config variable

* [interop] pass MasterDataActor from Swift to C++ and back to Swift

* [fdbserver] Swift->C++ header generation for FDBServer should use same module cache path

* Update swift_get_latest_toolchain.sh to fetch 5.9 toochains

* set HAVE_FLAG_SEARCH_PATHS_FIRST for cross compilation

* Resolve conflicts in net2/sim2/actors, can't build yet

* undo SWIFT_ACTOR changes, not necessary for merge

* guard c++ compiler flags with is_cxx_compile

* Update flow/actorcompiler/ActorParser.cs

Co-authored-by: Evan Wilde <etceterawilde@gmail.com>

* update the boost dependency

* Include boost directory from the container for Swift

* conform flow's Optional to FlowOptionalProtocol again

* Guard entire RocksDBLogForwarder.h with SSD_ROCKSDB_EXPERIMENTAL to avoid failing on missing rocksdb APIs

* remove extraneous merge marker

* [swift] update swift_test_streams.swifto to use vars in more places

* Add header guard to flow/include/flow/ThreadSafeQueue.h to fix moduralization issue

* Update net and sim impls

* [cmake] use prebuilt libc++ boost only when we're actually using libc++

* [fdbserver] Swift->C++ header generation for FDBServer should use same module cache path

* fixups after merge

* remove CustomStringConvertible conformance that would not be used

* remove self-caused deprecation warnings in future_support

* handle newly added task priority

* reformatting

* future: make value() not mutating

* remove FIXME, not needed anymore

* future: clarify why as functions

* Support TraceEvent in Swift

* Enable TraceEvent using a class wrapper in Swift

* prearing WITH_SWIFT flag

* wip disabled failing Go stuff

* cleanup WITH_SWIFT_FLAG and reenable Go

* wip disabled failing Go stuff

* move setting flag before printing it

* Add SWIFT_IDE_SETUP and cleanup guides and build a bit

* Revert "Wipe packet buffers that held serialized WipedString (#10018)"

This reverts commit e2df6e3302.

* [Swift] Compile workaround in KeyBackedRangeMap; default init is incorrect

* [interop] do not add FlowFutureOps conformance when building flow clang module for Flow checked continuation header pre-generation

* make sure to show  -DUSE_LIBCXX=OFF in readme

* readme updates

* do not print to stderr

* Update Swift and C++ code to build with latest Swift 5.9 toolchain now that we no longer support universal references and bridge the methods that take in a constant reference template parameter correctly

* Fix SERVER_KNOBS and enable use them for masterserver

* Bump to C++20, Swift is now able to handle it as well

* Put waitForPrev behind FLOW_WITH_SWIFT knob

* Forward declare updateLiveCommittedVersion

* Remove unused code

* fix wrong condition set for updateLiveCommittedVersion

* Revert "Revert "Wipe packet buffers that held serialized WipedString (#10018)""

This reverts commit 5ad8dce052.

* Enable go-bindings in cmake

* Revert "Revert "Wipe packet buffers that held serialized WipedString (#10018)""

This reverts commit 5ad8dce052.

* USE_SWIFT flag so we "build without swift" until ready to by default

* uncomment a few tests which were disabled during USE_SWIFT enablement

* the option is WITH_SWIFT, not USE

* formatting

* Fix masterserver compile error

* Fix some build errors.

How did it not merge cleanly? :/

* remove initializer list from constructor

* Expect Swift toolchain only if WITH_SWIFT is enabled

* Don't require Flow_CheckedContinuation when Swift is disabled

* Don't compile FlowCheckedContinuation when WITH_SWIFT=OFF

* No-op Swift macros

* More compile guards

* fix typo

* Run clang-format

* Guard swift/bridging include in fdbrpc

* Remove printf to pass the test

* Remove some more printf to avoid potential issues

TODO: Need to be TraceEvents instead

* Remove __has_feature(nullability) as its only used in Swift

* Don't use __FILENAME__

* Don't call generate_module_map outside WITH_SWIFT

* Add some more cmake stuff under WITH_SWIFT guard

* Some more guards

* Bring back TLSTest.cpp

* clang-format

* fix comment formatting

* Remove unused command line arg

* fix cmake formatting in some files

* Address some review comments

* fix clang-format error

---------

Co-authored-by: Alex Lorenz <arphaman@gmail.com>
Co-authored-by: Russell Sears <russell_sears@apple.com>
Co-authored-by: Evan Wilde <etceterawilde@gmail.com>
Co-authored-by: Alex Lorenz <aleksei_lorenz@apple.com>
Co-authored-by: Vishesh Yadav <vishesh_yadav@apple.com>
Co-authored-by: Vishesh Yadav <vishesh3y@gmail.com>
2023-06-02 16:09:28 -05:00
Vaidas Gasiunas 8158ce49fd
Change compiler options order to avoid -gdwarf overriding -g1 (#10377) 2023-05-31 10:51:08 +02:00
Aaron Molitor f94eada7c0 add cmake option to include RocksDB Tools with the Rocks DB compile 2023-05-30 09:59:37 -05:00
He Liu 4c75cee2e3
Upgrade to RocksDB 8.1.1 (#10039)
* Upgrade to RocksDB 8.1.1

* Revert an ASSERT on the previous rocksdb bug. The test was a guard to show the failure.
2023-04-27 12:32:04 -07:00
neethuhaneesha 44fba4e84c
RocksDB 7.10.2 version upgrade (#9828) 2023-04-03 10:01:22 -07:00
Xiaoge Su 011de97959 fixup! Launch ccache in CMake in a proper way
The old use of RULE_LAUNCH_COMPILE is discouraged, see

https://cmake.org/cmake/help/latest/prop_gbl/RULE_LAUNCH_COMPILE.html

The proper way is to use CMAKE_<LANG>_COMPILER_LAUNCHER
2023-03-23 20:17:18 -07:00
Xiaoge Su 3a34ee7a53 Add comments for this change in CMake 2023-03-23 20:17:18 -07:00
Xiaoge Su d618a7777c Statically link the OpenSSL library 2023-03-23 20:17:18 -07:00
Xiaoge Su 88eeb5a526 Remove WolfSSL support in FoundationDB 2023-03-23 20:17:18 -07:00
Xiaoge Su 0f93031083 Do not use virtual environment if system has built-in Sphinx 2023-03-22 20:41:28 -07:00
Xiaoge Su 21095bd036 fixup! Fix the version issue 2023-03-14 15:03:49 -07:00
Xiaoge Su c4a5cf0e8e Enforce the jemalloc version
With this patch, jemalloc version must be 5.3.0; otherwise error is
thrown.
2023-03-10 23:37:15 -08:00
Xiaoge Su 4373f111fb Let FDB uses Findjemalloc.cmake 2023-03-08 13:09:13 -08:00
Xiaoge Su 873b800ba8 Add Findjemalloc.cmake 2023-03-08 13:09:13 -08:00
Markus Pilman 5bebb5b4aa
Merge pull request #9492 from sfc-gh-vgasiunas/vgasiunas-api-version-defs
Centralize definition of API Version for Java, Python and C API
2023-02-28 12:04:02 -07:00
Russell Sears d2af157189 Add support for using pre-packaged jemalloc, doctest_proj, and fix python support for old fdbserver binaries 2023-02-27 15:38:58 -06:00
Russell Sears bcc05b1058 Improve support for prebuilt boost 2023-02-27 15:38:58 -06:00
Vaidas Gasiunas ddb183e18b Fixing bindingtester 2023-02-27 16:46:01 +01:00
Xiaoge Su 282f681d13 fixup! Address comments 2023-02-22 18:11:07 -08:00