Commit Graph

206 Commits

Author SHA1 Message Date
Bala Namasivayam 03382c6267 Fix ErrorCommitInfo event parser 2022-08-15 13:22:17 -07:00
Markus Pilman 072c10ed44
Merge pull request #7823 from sfc-gh-anoyes/anoyes/improve-determinism
Improve determinism (based on JOSHUA_SEED)
2022-08-15 14:09:49 -06:00
Marian Dvorsky 35a3a33d1c
Resolve TSAN-related issues in ctests (#7864) 2022-08-12 19:23:32 +02:00
Marian Dvorsky 8a02b666d0
Make test_fdb_pkgs faster and fix test_execstack_permissions_libfdb_c (#7832) 2022-08-09 21:58:38 +02:00
Andrew Noyes 2002ebf6ef Track resident memory usage in test harness
We switched to limiting based on resident memory usage, so track that
instead.
2022-08-08 10:05:39 -07:00
Andrew Noyes 2e9f5f138d Sort test files before picking one based on JOSHUA_SEED
Previously it was dependent on whatever order files were listed in a
directory, which is suboptimal.
2022-08-08 10:05:39 -07:00
A.J. Beamon b42cb48dd4 Report the unit tests being run in test harness 2022-08-07 07:37:29 -07:00
Andrew Noyes da1ffebcb0
Improve test harness logging when there are no trace files (#7785)
* Log OldBinary even if there are no trace files

DeterminismCheck and OldBinary attributes don't actually depend on
information in the ProgramStart event, so we can add them
unconditionally.

* Add JoshuaSeed attribute to Test element in test harness

* Add NoTraceFilesFound event in test harness

There's already something similar: NoTraceFileGenerated. It appears that
the original author only wants to log that if the process exited 0
though. I'm not sure what the reason for that is so I think it's safer
to add a new event. This will make it more clear if say an old binary is
corrupt.
2022-08-03 17:14:33 -07:00
Bala Namasivayam 996484191b Fix protocol version 2022-08-03 11:16:15 -07:00
Bala Namasivayam bf3009d6c9 Add missing CommitInfo fields to the transaction profiling analyzer 2022-08-02 17:13:03 -07:00
Vaidas Gasiunas d99d0370b1
C shim library: API for setting client library path; additional tests (#7702)
* Adding sources of the Implib.so project

* Run C unit tests and API tests with the shim library

* Reuse compilation of C test binaries with and without shim library

* Resolve client library path from an environment variable

* Refactoring: Reusable module for downloading FDB binaries

* Testing client shim library with current version and last release version

* Tests for specifying client library over an environment variable

* Enable C shim library tests on ARM

* Restore the original path for including fdb_api.hpp

* Improve cmake dependencies on doctest

* Remove unnecessary Implib.so files

* Shim library: API to specify path to the library, additional tests

* Adding shim library to Linux distribution packages

* Update link dependencies of fdb_c_shim_lib_tester

* Add dependencies on Implib.so sources

* Abort on fatal errors of shim library

* fix typo
2022-07-29 11:45:45 +02:00
Markus Pilman 92b1d1f647 Don't fail TestHarness if comment doesn't exist 2022-07-28 11:03:09 -06:00
Markus Pilman ef46f72d0c Merge remote-tracking branch 'origin/main' into features/code-coverage-test-harness 2022-07-27 10:41:18 -06:00
A.J. Beamon dec6dbfbfb
Merge pull request #7549 from sfc-gh-ajbeamon/feature-tenant-groups
Add support for tenant groups
2022-07-27 07:56:27 -07:00
Andrew Noyes de331ba080
Fix incorrect deserialization of FdbClientLogEvents::Event (#7707)
We're trying to interpret an Optional<TenantName> here, but the
python truthiness of self.get_bytes(1) is _always_ true, since it's
non-empty. We need to look at the _contents_ of that byte instead.
2022-07-26 13:10:31 -07:00
A.J. Beamon e81ed7ce51 Include encode.h in decode.h to get access to the BUFFERSIZE macro 2022-07-26 09:29:22 -07:00
Renxuan Wang dc9599f2e9
Get network interfaces ready for https proxy. (#7556)
* Move HTTP from fdbclient/ to fdbrpc/.

* Move md5 and libb64 to contrib/.

* Get network interfaces ready for https proxy.

* Rebase
2022-07-25 17:08:32 -07:00
Kevin Hoxha 058276493f
ddsketch_utility: Add utility scripts for ddsketch (#7602)
* Adds ddsketch_calc.py which implements a class for DDSketch related calculations
* Adds ddsketch_conversion.py for quickly computing a bucket index to a value or vice-versa
* Adds ddsketch_compare.py to compute how similar two ddsketch distributions are
* Adds export_graph.py to graph the ddsketch distribution outputted from mako

The arguments for ddsketch_conversion.py are:
-b, --bucket: the bucket index that we need to calculate the value from (optional)
-v, --value: the value that we need to calculate the bucket index from (optional)
-e, --error_guarantee: the error guarantee for ddsketch (optional, default is 0.005)

The arguments for ddsketch_compare.py are:
--file1: Path to first ddsketch json
--file2: Path to second ddsketch json
--txn1: The transaction type for the first file
--txn2: The transaction type for the second file
--op: The operation name (ex: GRV, GET ...)

The arguments for export_graph.py:
--file: path to ddsketch distribution
--txn, -t: Transaction type from file
--title: title for graph (optional, otherwise "Title" is used)
--savefig: Path to save the image plot (optional)
--op: Which operation to plot
2022-07-25 10:29:33 -07:00
Xiaoge Su 8ea3ef5639 fixup! Let clang accepts crc32_wrapper.c 2022-07-23 16:38:10 -07:00
Markus Pilman 3327ab395b Change test harness to include comments 2022-07-19 15:41:53 -06:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Marian Dvorsky ce4dc0fb0f
Update ctest_to_joshua to work with new ctests (#7584)
* Package inferred directories to the resulting tarball in ctest_to_joshua (except for the root source and build directories). This allows the API tests (which specify the workloads as a directory containing the workload specification files) to now work.
* Add bin/mkcert which is used by some tests.
* Rewrite fdb_c_shim_tests.py to specify its dependencies on command line.
2022-07-14 18:34:48 +02:00
Markus Pilman fa9e623fc1
Merge pull request #7459 from sfc-gh-ljoswiak/features/module-link-tests
Add test executables to catch missing symbols in modules
2022-07-11 17:31:11 -06:00
Vaidas Gasiunas 0a2e5c830c
C shim library build clean up (#7560)
* Remove unnecessary Implib.so files

* Improve cmake dependencies on doctest
2022-07-11 17:57:28 +02:00
Vaidas Gasiunas 1e8feb9cb8
Generate a shim library for the FDB C API (#7506)
* Adding sources of the Implib.so project

* Run C unit tests and API tests with the shim library

* Reuse compilation of C test binaries with and without shim library

* Resolve client library path from an environment variable

* Refactoring: Reusable module for downloading FDB binaries

* Testing client shim library with current version and last release version

* Tests for specifying client library over an environment variable

* Enable C shim library tests on ARM

* Restore the original path for including fdb_api.hpp
2022-07-08 16:28:35 +02:00
Andrei Gorneanu 35f693b238 Disable ffast-math for Intel compiler 2022-07-07 17:36:26 +02:00
Lukas Joswiak 618f8455c4 Add test executables to catch missing symbols
Currently, we have code in different folders like `flow/` and `fdbrpc/`
that should remain isolated. For example, `flow/` files should not
include functionality from any other modules. `fdbrpc/` files should
only be able to include functionality from itself and from `flow/`.
However, when creating a shared library, the linker doesn't complain
about undefined symbols -- this only happens when creating an
executable. Thus, for example, it is possible to forward declare an
`fdbclient` function in an `fdbrpc` file and then use it, and nothing
will break (when it should, because this is illegal).

This change adds dummy executables for a few modules (`flow`, `fdbrpc`,
`fdbclient`) that will cause a linker error if there are included
symbols which the linker can't resolve.
2022-07-06 14:49:33 -07:00
Mohamed Oulmahdi 958b1faf81
Merge pull request #7498 from apple/7477
7477 - Fix Windows build
2022-06-30 09:54:37 +02:00
Markus Pilman 8123070c2a move rapidxml to its own target 2022-06-29 16:10:14 -06:00
Mohamed Oulmahdi a5cf6f38f3 Add missing Windows crc32 headers 2022-06-29 23:44:33 +02:00
Markus Pilman 20e7371036 some Linux fixes 2022-06-28 08:51:13 -06:00
Markus Pilman 4ac236eeba fdbcli compiling 2022-06-27 19:20:18 -06:00
Markus Pilman 41181cda26 fdbserver compiling (but linking still fails) 2022-06-27 18:54:58 -06:00
Markus Pilman 03d913a1de Flow compiling 2022-06-27 17:05:55 -06:00
Ata E Husain Bohra 9396b691b7
Generate GNU compatible build-id for mockkms golang binary (#7389)
* Generate GNU compatible build-id for mockkms golang binary

Description

 diff-1: Fix compilation issue

Generate GNU compatible build-id for mockkms golang binary
Leverage "cgo" to generate build-id

Testing

Debian package build, verified the GNU build-id
2022-06-15 10:43:46 -07:00
Junhyun Shim 3e79735b2f
Authz JWT support (#7279)
* Add JWT support to TokenSign

* Encapsulate OpenSSL public/private key type

Type-safe passing around of keys without having to DER/PEM-serialize
(OpenSSL doesn't have distinct types for public and private key)

* Apply Clang format

* Add verify benchmark for JWT and FlatBuffers token

* Unit test base64url::{encode, decode}

* Make all payload fields optional

Let user code validate non-signature fields

* Make all payload fields optional

Completely defer field check to user code

* Move rapidjson from fdbclient to contrib

* Make fdbrpc's rapidjson linkage private

Currently only sources include them.

* Modify rapidjson path in apiversioner.py

* Algorithm::Unknown > Algorithm::UNKNOWN
2022-06-02 13:22:50 +02:00
Markus Pilman 452315ee78
Build mockkms and add mockkms test (#7153) 2022-05-12 15:07:02 -07:00
Andrew Noyes 17140a2645 Always run valgrind on the binary under test (if valgrind is enabled) 2022-05-10 14:46:22 -07:00
Bala Namasivayam 57f5ca2522 Fix deserialization of tenant field in 7.1 2022-05-09 19:12:12 -07:00
Andrew Noyes 9f8e2a4517
Test compiling an fdb_c app in pkg_tester (#6940)
* Use execstack to test executable stack permissions

The output of readelf wasn't stable

* Tee output to stdout in pkg_tester "run" function

* Test compiling an app that includes fdb_c.h

Featuring both cmake and pkg_tester, since we intend to support those in
our packages.

* Fix packaging bug

The libdir in the pkg-config file in the packages was wrong
2022-04-26 11:54:52 -07:00
Aaron Molitor c440365779 update version to 7.2.0 -- pr comment protocol version 2022-04-11 23:23:27 -05:00
Aaron Molitor 8db8545db2 update version to 7.2.0 -- protocol version changes 2022-04-11 23:23:27 -05:00
A.J. Beamon ca653c77ee
Tenant binding tester support (#6642)
* Add binding tester support for tenants

* Configure tenant_mode=optional_experimental in the binding tester

* Use the special key-space to delete tenants in between binding tester runs.

* Separate tenant tester spec into its own file.
2022-03-23 13:57:45 -07:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
Josh Slocum 37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
Andrew Noyes 68c03a7e32
Jemalloc integration fixes (#6626)
* Set default for USE_JEMALLOC initially in ConfigureCompiler

Instead of trying to change the value later on. This fixes the valgrind
build, which was previously incorrectly getting jemalloc involved.

* Check aligned_alloc result for null

And OOM if so - don't assert

* Check that we can allocate magazines with no internal fragmentation

We may want to do this so that the jemalloc heap profiler has some
knowledge of FastAlloc

* Populate TestFile field for noSim tests in TestHarness

* Remove handling for nonexistent "ActualRun"
2022-03-17 15:17:27 -07:00
A.J. Beamon 00277140ec Non-simulated test harness runs don't check the unseed. 2022-03-17 11:09:14 -07:00
Steve Atherton 59762fc784
Merge pull request #6423 from sfc-gh-satherton/redwood-bug-fixes-and-memory-leak
Redwood memory usage and small memory leak fixes
2022-03-15 16:52:28 -07:00
A.J. Beamon ecccfd0868 Add cache invalidation to tenant cache. Send tenant ID along with tenant name in requests to validate that the tenant hasn't changed. Fix a few bugs. 2022-03-15 09:23:30 -07:00
Josh Slocum e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00