Commit Graph

660 Commits

Author SHA1 Message Date
Andrew Noyes e363030a35 Re-enable aws sdk + arm
This is fixed since f13c8525e3
2022-10-07 10:54:38 -07:00
Xiaoge Su 8868201ff9 fixup! Restrict the flags to CLang only 2022-10-05 12:52:39 -07:00
Xiaoge Su c3e84f4e8f Allow build FoundationDB with zstd compression library
With this patch, a new option, FLOW_USE_ZSTD, is introduced. If turned
ON, flow will compile with zstd compression library.
2022-10-05 12:52:39 -07:00
Andrew Noyes 7a81628c94 Quote aws sdk cxx flags
Cmake 3.24 seems to require this
2022-09-28 11:43:21 -07:00
Andrew Noyes f13c8525e3 Make sure loader can find libc++
Otherwise things like try_run don't work, since you can't run an
executable built with -stdlib=libc++ successfully. This fixes the TSAN
build.
2022-09-27 12:39:58 -07:00
Xiaoge Su 0f8820e03d
Fix RocksDB link issue (#8296)
* Fix the RocksDB compile issue with clang

By default, RocksDB is using its own compile/link flags, no matter how
FDB flags are. This led to the issue that if FDB decides to use
clang/ldd/libc++, RocksDB will pick up the compiler/linker but still use
libstdc++, which is incompatible to libc++, causing Symobl Missing error
during the link stage.

With this patch, if FDB uses libc++, then the information is stored in
CMAKE_CXX_FLAGS and being forwarded to RocksDB. RocksDB will then use
libc++ and compatible with FDB.

* fixup! update the indentation
2022-09-26 09:19:21 -07:00
Jingyu Zhou d43df45eff
Merge pull request #8284 from sfc-gh-anoyes/anoyes/disable-aws-backup-aarch64-for-now
Limit BUILD_AWS_BACKUP to x86_64 for now
2022-09-23 09:35:11 -07:00
Ata E Husain Bohra 52169d2b8e
Enable ZSTD compression support (#8014)
* Enable ZSTD compression filter

Description

  diff-4: Randomize Knob Compression filter selection
  diff-3: Minor refactoring
  diff-2: Limit ZSTD availability to CLANG compiler
  diff-1: Add ZSTD compression option to BlobGranule tests

Major changes includes:
1. Update FDB CMake to download, install and build Boost with
ZSTD compatibility
2. Update CompressionUtils to enable boost::iostreams::zstd
compression filter

Testing

CompressionUtilsUnit.toml
BlobGranuleCorrectness/BlobGranuleCorrectnessClean
devRunCorrectness - 100K (in-progress)
2022-09-22 14:31:49 -07:00
Andrew Noyes 129edebb91 Limit BUILD_AWS_BACKUP to x86_64 for now
With aarch64, in the current docker image, linking curl statically doesn't work yet.

This is the diagnostic:

```
fdbclient/awssdk-build/curl/lib/strerror.c:32:6: error: #error "strerror_r MUST be either POSIX, glibc or vxworks-style"
2022-09-22 16:05:30   #    error "strerror_r MUST be either POSIX, glibc or vxworks-style"
2022-09-22 16:05:30        ^~~~~
```

The root cause is that curl's cmake feature detection logic gets
confused since it can't build binaries that execute successfully with
the link flags it wants:

```
$ cc test.c -lssl
$ ./a.out
./a.out: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
```
2022-09-22 09:35:38 -07:00
Andrew Noyes e7c171737e
Swap the order of curl and zlib for awssdk (#8268)
According to
https://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_node/ld_3.html,
dependencies should appear after. Since curl depends on zlib, put zlib
after curl.
2022-09-21 14:20:36 -07:00
Andrew Noyes 65ec5ef37f
Statically link libcurl for the aws sdk (#8226) 2022-09-20 11:23:18 -07:00
Vaidas Gasiunas 118d693783
Reduce the size of debug symbols in FDB binaries (#8244)
* Compress debug symbols; generate less debug symbols by default

* Generate full debug symbols for CMAKE_BUILD_TYPE=Debug builds
2022-09-20 19:07:44 +02:00
1inker cc1b0edd44
Patch for commit #24b2de8de8: set Boost_iostreams_FOUND variable before check it (#8203) 2022-09-19 09:30:27 -07:00
Sam Gwydir d78b996ac0 fix logic; print message 2022-09-15 14:17:31 -07:00
Sam Gwydir f1d25aea64 Linking atomic explicitly is not needed 2022-09-15 13:53:40 -07:00
Russell Sears a4fd0c0a9e
Fix clang build in current dev container (#8121)
* Fix clang build in current dev container

* Fix RocksDB build w/ clang

* Fix cmake warning about download timestamp semantics

* Disable RocksDB build on windows.
2022-09-12 12:03:40 -05:00
1inker db888245ed
Pass compilers choice to subprojects (#8135) 2022-09-09 11:53:48 -07:00
Vaidas Gasiunas 70f4d4078c
Increase timeout for client tests in sanitizer builds (#8002) 2022-08-29 15:35:04 +02:00
Markus Pilman 98039f6ca1 Merge remote-tracking branch 'origin/main' into features/new-test-harness 2022-08-26 13:52:25 -06:00
Marian Dvorsky a666a1ca79
Upgrade awssdk to v1.9.331 (#7982) 2022-08-24 20:37:48 +02:00
Markus Pilman 3545708b91 addressed review comments 2022-08-22 21:06:11 -06:00
Marian Dvorsky d251fef5e9
Use -fno-sanitize=function with UBSAN as a work around (#7956) 2022-08-22 19:18:48 +02:00
Markus Pilman 0971ee5f7a enable new test harness for correctness builds 2022-08-16 11:31:37 -06:00
Mohamed Oulmahdi 7824de5ff4 Fix Windows build broken by #7682 2022-08-15 05:02:46 +02:00
Marian Dvorsky 35a3a33d1c
Resolve TSAN-related issues in ctests (#7864) 2022-08-12 19:23:32 +02:00
Sagar Vemuri 11db333b7b Add tenant mode option to fdbclient tests 2022-08-11 08:11:55 -07:00
Markus Pilman 86ee76ec50
Merge pull request #7816 from sfc-gh-mdvorsky/mdvorsky/coverage
Flush gcov coverage upon SIGTERM
2022-08-08 12:03:22 -06:00
Markus Pilman 828d15907e
Merge pull request #7798 from sfc-gh-mdvorsky/mdvorsky/unittests_in_joshua
Add unit tests to the correctness package
2022-08-08 11:23:22 -06:00
Marian Dvorsky 722f66beb7 Flush gcov coverage upon SIGTERM 2022-08-08 17:11:20 +02:00
Marian Dvorsky d332e13eff Add unit tests to the correctness package 2022-08-05 20:28:53 +02:00
Andrew Noyes 1558edce8c Python_EXECUTABLE to Python3_EXECUTABLE 2022-07-29 14:57:29 -07:00
Andrew Noyes db348598a1 Update Python target name to Python3 2022-07-29 14:57:29 -07:00
Andrew Noyes 52e9990d32 Python_Interpreter_FOUND -> Python3_Interpreter_FOUND 2022-07-29 14:57:29 -07:00
Andrew Noyes efc3045834 Disallow python2 for cmake scripts
The fdb c shim build requires python3. Let's not allow python2 to get
used there.

Otherwise we can see something like this:
```
FAILED: bindings/c/libfdb_c.so.init.c bindings/c/libfdb_c.so.tramp.S
cd /home/anoyes/workspace/foundationdb/build/bindings/c && /usr/bin/python2.7 /home/anoyes/workspace/foundationdb/contrib/Implib.so/implib-gen.py --target aarch64 --outdir /home/anoyes/workspace/foundationdb/build/bindings/c --dlopen-callback=fdb_shim_dlopen_callback /home/anoyes/workspace/foundationdb/build/lib/libfdb_c.so
  File "/home/anoyes/workspace/foundationdb/contrib/Implib.so/implib-gen.py", line 27
    sys.stderr.write(f'{me}: warning: {msg}\n')
```
2022-07-29 14:57:29 -07:00
Junhyun Shim 2a172e1586 Fix WinGDI name collision by defining macro NOGDI
TenantState::ERROR collides with WinGDI's #define ERROR 0
2022-07-25 09:11:34 +02:00
Andrew Noyes 45c8a4e09d
Build fixes/improvements for sanitizers (#7657)
* Don't build fdb c shim with ubsan

This avoids duplicate symbols when linking. It doesn't really make sense
to assemble files with -fsanitize=undefined anyway, since it won't
insert instrumentation.

* Consolidate boost_asan with boost_target
2022-07-21 12:43:38 -07:00
Ata E Husain Bohra 9a3e88df46
Fix ASAN build (#7629)
* Fix ASAN build

Description
  -diff-2: Fix Mac build issues
  -diff-1: Address review comments

Patch addresses the issue where ASAN build failed after introducing
BlobGranule compression.

Testing

ASAN build
2022-07-20 14:14:46 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Ata E Husain Bohra 24b2de8de8 BlobFile Encryption and compression support
Description

Testing
2022-07-14 17:04:14 -07:00
Markus Pilman fa9e623fc1
Merge pull request #7459 from sfc-gh-ljoswiak/features/module-link-tests
Add test executables to catch missing symbols in modules
2022-07-11 17:31:11 -06:00
Andrei Gorneanu 3a0444b569 Merge remote-tracking branch 'upstream/main' into add_intel_compiler_support 2022-07-07 17:30:09 +02:00
Lukas Joswiak 618f8455c4 Add test executables to catch missing symbols
Currently, we have code in different folders like `flow/` and `fdbrpc/`
that should remain isolated. For example, `flow/` files should not
include functionality from any other modules. `fdbrpc/` files should
only be able to include functionality from itself and from `flow/`.
However, when creating a shared library, the linker doesn't complain
about undefined symbols -- this only happens when creating an
executable. Thus, for example, it is possible to forward declare an
`fdbclient` function in an `fdbrpc` file and then use it, and nothing
will break (when it should, because this is illegal).

This change adds dummy executables for a few modules (`flow`, `fdbrpc`,
`fdbclient`) that will cause a linker error if there are included
symbols which the linker can't resolve.
2022-07-06 14:49:33 -07:00
Steven Li d4273c4dda revise prerelease pkg format 2022-06-30 18:04:38 -05:00
Mohamed Oulmahdi 6fcbf5784c Formatting 2022-06-30 00:10:47 +02:00
Mohamed Oulmahdi 7af9b2fae3 Enhance TOML dependencies 2022-06-29 23:45:28 +02:00
Markus Pilman 41181cda26 fdbserver compiling (but linking still fails) 2022-06-27 18:54:58 -06:00
Markus Pilman d31fb30f57 fdbclient compiling 2022-06-27 18:31:05 -06:00
Markus Pilman 88af6b403f fdbrpc compiling 2022-06-27 18:11:55 -06:00
Markus Pilman 03d913a1de Flow compiling 2022-06-27 17:05:55 -06:00
goandrei 06004e8b25 Add Intel compiler support 2022-06-26 14:02:07 +00:00