Commit Graph

695 Commits

Author SHA1 Message Date
Markus Pilman 5bebb5b4aa
Merge pull request #9492 from sfc-gh-vgasiunas/vgasiunas-api-version-defs
Centralize definition of API Version for Java, Python and C API
2023-02-28 12:04:02 -07:00
Russell Sears d2af157189 Add support for using pre-packaged jemalloc, doctest_proj, and fix python support for old fdbserver binaries 2023-02-27 15:38:58 -06:00
Russell Sears bcc05b1058 Improve support for prebuilt boost 2023-02-27 15:38:58 -06:00
Vaidas Gasiunas ddb183e18b Fixing bindingtester 2023-02-27 16:46:01 +01:00
Xiaoge Su 282f681d13 fixup! Address comments 2023-02-22 18:11:07 -08:00
Xiaoge Su 242962aed3 Provide a tool that allows downloading logs when simulation RocksDB failures
A script, rocksdb_logtool.py, is available to upload/download generated
XML/JSON log files when test harness 2 detects that the test is failed,
and the script detects that the test is using RocksDB.

To upload, no actions is needed, using the regular

```
joshua start --tarball correctness.tgz
```

The build system will automatically pack the rocksdb_logtool.py into the
tarball and test harness 2 will call the script if it thinks the test is
failed.

To download, simply provide the ensemble id and test uid, e.g.

```
python3 rocksdb_logtool.py download --ensemble-id
20230222-204240-xiaogesu-cb6ea277a898f134 --test-uid
ab6fb792-088f-49d6-92d2-43bc4fb81668
```

Note the test UID can be retrieved by

```
joshua tail ensemble_id
```

output, it is in the field `TestUID` in <Test> element of test harness 2
generated XML.

For convenience, it is possible to do a

```
python3 rocksdb_logtool.py list --ensemble-id ensemble-id
```

to generate all possible download commands for failed tests. However,
the list subcommand will *NOT* verify if the test failure is coming from
RocksDB, i.e. other test failues may be included and it is the caller's
responsibility to verify. If the test is not RocksDB related, the
download would fail as nothing is uploaded.
2023-02-22 13:15:57 -08:00
Vaidas Gasiunas afc25035f8
Client status report API in Java and python bindings (#9366)
* get_client_status: add to Java bindings

* Add get_client_status in python binding

* Make python unit tests available as ctest

* Fix file name in the copyright header

* Fix library path name for mac

* get_client_status: Minor changes in Java & Python bindings addressing review comments

* Rename fdb_python_unit_tests to unit_tests
2023-02-15 18:00:53 +01:00
Xiaoge Su 006a59b26f fixup! Fix the CMake build failure 2023-01-27 00:03:47 -08:00
Xiaoge Su 9a0fd7f999 Generate bindingtester2 tarball for test
bindingtester2-${FDB_VERSION} is generated in packages/ directory.
It is possible to run the test by

j start --tarball bindingtester2-7.3.0.tar.gz --max-runs 1000

and submit the test to joshua system. Joshua will run the tests in
parallel and tests will not impact each other. Yet it will take long
time for each run.
2023-01-23 17:56:36 -08:00
neethuhaneesha c4ccb80261
add some rocksdb compile options that can be passed in at build time (#9141) 2023-01-13 14:03:51 -08:00
Mohamed Oulmahdi 6472282e07 Fix Windows build broken by #8904 2023-01-03 12:45:59 -06:00
A.J. Beamon 7ba1438d3f Fix unquoted variable in cmake that cause certain builds to fail 2022-12-21 16:20:20 -08:00
A.J. Beamon 722e5dbc65 Use dwarf-4 in clang builds because the results with dwarf-5 do not currently play well with GNU Binutils 2022-12-20 11:10:49 -08:00
Junhyun Shim 77e523ce91 Merge remote-tracking branch 'origin/main' into mako-authz-token-generation 2022-12-15 17:02:19 -08:00
Vaidas Gasiunas 585996a88b
Do not compress debug sections in release builds (#9005) 2022-12-12 16:19:57 +01:00
Junhyun Shim 48f387d4bf Merge remote-tracking branch 'origin/main' into mako-authz-token-generation 2022-12-07 09:06:39 +01:00
Junhyun Shim 36b4569a08 Merge remote-tracking branch 'origin/main' into mako-authz-token-generation 2022-12-07 08:49:20 +01:00
Daniel Luan 0afad44d4e disable ambiguous-reversed-operator warning 2022-12-06 14:19:06 -08:00
Daniel Luan 4be05e5a5d Upgrade C++ Standard to 20 2022-12-06 14:19:06 -08:00
Junhyun Shim 0c204586d7 fork a pipe-IPC server subprocess from main 2022-12-06 14:04:30 +01:00
A.J. Beamon ed9acd550e Properly generate operations with the _TENANT suffix.
Add atomic op functions to the tenant object in Python.
Fix a Python tester bug for tenant operations.
Fix a build error with the binding tester package.
2022-11-30 15:30:10 -08:00
Vaidas Gasiunas 493ab95464
Define the old FDB release versions used for client testing in one place (#8888)
* Auto-formatting fdb_c_shim_tests.py

* Define the old release versions used for testing in one place; Using PYTHONPATH to find python modules from other project directories

* Remove obsolete upgrade tests; add direct upgrade tests from older versions

* Trigger tests
2022-11-25 13:16:33 +01:00
Junhyun Shim b28cb708a2 RESOURCE_LOCK venv setup testcase
Concurrent venv-setup and PIP upgrade breaks test
2022-11-22 21:24:47 +01:00
Junhyun Shim 72b4dc9852 Make client tests run in venv 2022-11-22 18:20:28 +01:00
neethuhaneesha d865e77f06
RocksDB 7.7.3 version upgrade (#8858) 2022-11-17 15:39:22 -08:00
Jingyu Zhou a121205d22
Merge pull request #8504 from sfc-gh-anoyes/anoyes/go-hint
Add a hint for the GO path
2022-11-07 09:31:31 -08:00
Markus Pilman 989731f7f4 Fix AWS SDK build and removed check for old build system 2022-10-26 11:48:10 -06:00
Andrew Noyes 666075cdf2 Add a hint for the GO path
Before this, cmake would not find the go binary in the docker image.
After this, cmake finds it.
2022-10-24 09:26:15 -07:00
Chaoguang Lin 78e2871a79
Add a long running test (#8505)
* Add a long running test

* Make new scripts executable and update the test file
2022-10-21 11:11:11 -07:00
Jingyu Zhou d40f1555bb
Merge pull request #8481 from sfc-gh-satherton/fix-boost-rebuild-every-time
Fix cmake config so boost does not rebuild every time
2022-10-16 13:12:30 -07:00
Steve Atherton a452e6ee82 Build bug fix, the location of Boost config.hpp was wrong so ninja decides it is missing and rebuilds boost on every build. 2022-10-15 23:40:48 -07:00
Chaoguang Lin 2ba98f3a60
Add a cmake option ENABLE_LONG_RUNNING_TESTS (#8364)
* Add a cmake option ENABLE_LONG_RUNNING_TESTS to generate separate correctness package for long running tests

* Remove the unnessary long_running_tests directory

* Use the default bash files
2022-10-11 15:28:07 -07:00
Markus Pilman b295a43e1d
Merge pull request #8453 from sfc-gh-anoyes/anoyes/ld-library-path-static-libcxx
We need try_run to work even when linking libc++ statically
2022-10-11 15:07:04 -06:00
Andrew Noyes 5411b687a6 We need try_run to work even when linking libc++ statically 2022-10-11 09:21:55 -07:00
Hui Liu a4c73a5f0a add benchmark for zstd 2022-10-11 08:23:55 -07:00
Andrew Noyes e363030a35 Re-enable aws sdk + arm
This is fixed since f13c8525e3
2022-10-07 10:54:38 -07:00
Xiaoge Su 8868201ff9 fixup! Restrict the flags to CLang only 2022-10-05 12:52:39 -07:00
Xiaoge Su c3e84f4e8f Allow build FoundationDB with zstd compression library
With this patch, a new option, FLOW_USE_ZSTD, is introduced. If turned
ON, flow will compile with zstd compression library.
2022-10-05 12:52:39 -07:00
Andrew Noyes 7a81628c94 Quote aws sdk cxx flags
Cmake 3.24 seems to require this
2022-09-28 11:43:21 -07:00
Andrew Noyes f13c8525e3 Make sure loader can find libc++
Otherwise things like try_run don't work, since you can't run an
executable built with -stdlib=libc++ successfully. This fixes the TSAN
build.
2022-09-27 12:39:58 -07:00
Xiaoge Su 0f8820e03d
Fix RocksDB link issue (#8296)
* Fix the RocksDB compile issue with clang

By default, RocksDB is using its own compile/link flags, no matter how
FDB flags are. This led to the issue that if FDB decides to use
clang/ldd/libc++, RocksDB will pick up the compiler/linker but still use
libstdc++, which is incompatible to libc++, causing Symobl Missing error
during the link stage.

With this patch, if FDB uses libc++, then the information is stored in
CMAKE_CXX_FLAGS and being forwarded to RocksDB. RocksDB will then use
libc++ and compatible with FDB.

* fixup! update the indentation
2022-09-26 09:19:21 -07:00
Jingyu Zhou d43df45eff
Merge pull request #8284 from sfc-gh-anoyes/anoyes/disable-aws-backup-aarch64-for-now
Limit BUILD_AWS_BACKUP to x86_64 for now
2022-09-23 09:35:11 -07:00
Ata E Husain Bohra 52169d2b8e
Enable ZSTD compression support (#8014)
* Enable ZSTD compression filter

Description

  diff-4: Randomize Knob Compression filter selection
  diff-3: Minor refactoring
  diff-2: Limit ZSTD availability to CLANG compiler
  diff-1: Add ZSTD compression option to BlobGranule tests

Major changes includes:
1. Update FDB CMake to download, install and build Boost with
ZSTD compatibility
2. Update CompressionUtils to enable boost::iostreams::zstd
compression filter

Testing

CompressionUtilsUnit.toml
BlobGranuleCorrectness/BlobGranuleCorrectnessClean
devRunCorrectness - 100K (in-progress)
2022-09-22 14:31:49 -07:00
Andrew Noyes 129edebb91 Limit BUILD_AWS_BACKUP to x86_64 for now
With aarch64, in the current docker image, linking curl statically doesn't work yet.

This is the diagnostic:

```
fdbclient/awssdk-build/curl/lib/strerror.c:32:6: error: #error "strerror_r MUST be either POSIX, glibc or vxworks-style"
2022-09-22 16:05:30   #    error "strerror_r MUST be either POSIX, glibc or vxworks-style"
2022-09-22 16:05:30        ^~~~~
```

The root cause is that curl's cmake feature detection logic gets
confused since it can't build binaries that execute successfully with
the link flags it wants:

```
$ cc test.c -lssl
$ ./a.out
./a.out: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
```
2022-09-22 09:35:38 -07:00
Andrew Noyes e7c171737e
Swap the order of curl and zlib for awssdk (#8268)
According to
https://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_node/ld_3.html,
dependencies should appear after. Since curl depends on zlib, put zlib
after curl.
2022-09-21 14:20:36 -07:00
Andrew Noyes 65ec5ef37f
Statically link libcurl for the aws sdk (#8226) 2022-09-20 11:23:18 -07:00
Vaidas Gasiunas 118d693783
Reduce the size of debug symbols in FDB binaries (#8244)
* Compress debug symbols; generate less debug symbols by default

* Generate full debug symbols for CMAKE_BUILD_TYPE=Debug builds
2022-09-20 19:07:44 +02:00
1inker cc1b0edd44
Patch for commit #24b2de8de8: set Boost_iostreams_FOUND variable before check it (#8203) 2022-09-19 09:30:27 -07:00
Sam Gwydir d78b996ac0 fix logic; print message 2022-09-15 14:17:31 -07:00
Sam Gwydir f1d25aea64 Linking atomic explicitly is not needed 2022-09-15 13:53:40 -07:00