Commit Graph

1720 Commits

Author SHA1 Message Date
Jon Fu 9bfc6ee59e Merge branch 'main' of github.com:apple/foundationdb into jfu-mako-active-tenants 2022-06-16 10:32:20 -07:00
Jon Fu 2b9c8ca874 introduce concept of active versus total tenants in mako 2022-06-15 15:30:02 -07:00
Jon Fu 06c8f9068e fix arg parse ordering 2022-06-14 16:45:58 -07:00
Jon Fu 184c266bdf adjust rows calculation after parsing all arguments but before validation 2022-06-14 16:34:39 -07:00
Jon Fu 6999ebc86f When provided tenants, divide number of rows by number of tenants. Adjust population and range reads to account for this scenario 2022-06-14 16:22:07 -07:00
Hao Fu 9cee4c94e7
Safely remove fdb_transaction_get_range_and_flat_map (#7314) 2022-06-13 19:05:00 -07:00
Andrew Noyes 2a8d8a1932
Fix more heap overflows (#7360)
From calling strlen on a not necessarily null-terminated buffer.
2022-06-13 13:13:05 -07:00
sfc-gh-tclinkenbeard 13efda66fd Fix -Wunused-but-set-variable warning in DDSketchBase::percentile 2022-06-13 10:18:31 -07:00
Junhyun Shim 631a59a65e
Merge pull request #7299 from sfc-gh-mdvorsky/mdvorsky/remove_tester_api_wrapper
Remove TesterApiWrapper, replace its uses with fdb_api.hpp
2022-06-09 10:42:05 +02:00
Robert Barabas 8606923da2
Arm64 related build fixes (#7319)
* Add missing include

* Fix open call on arm64

* Bump up doctest to 2.4.8
2022-06-08 11:20:27 -07:00
Andrew Noyes 07f49392ac
Avoid using structured bindings in doctest assertions (#7335)
* Avoid using structured bindings in doctest assertions

clang doesn't allow this with the latest releases of doctest

This will unblock #7319

* Add constructor to MappedKV
2022-06-08 09:36:18 -07:00
Andrew Noyes e0eedc6a37
Fix asan message detection (#7338)
* Fix asan message detection

* Fix heap-buffer-overflow

Call to strlen on a not necessarily null terminated string

* Fix gcc build
2022-06-08 09:35:30 -07:00
Marian Dvorsky 21b041317c Address review comments 2022-06-08 16:45:29 +02:00
Ray Jenkins 5a4082bfe8
Add DistributedTracerClient enum to mako. (#7297) 2022-06-07 14:07:08 -05:00
A.J. Beamon c97a1168cb Fix formatting issue 2022-06-07 08:51:10 -07:00
A.J. Beamon 0431635674
Merge branch 'main' into jfu-mako-tenant-support 2022-06-07 08:43:56 -07:00
Kevin Hoxha e579038018
Add DDSketch to mako (#7167)
* Add logic for DDSketch in mako

* Return double from percentile() and fix crash in deserialize()

* make sure to serialize and print result from mergeSketchReport()

* clean up comments

* move ddsketch into its own file

* remove LatencySampleBin and add DDSketch to ThreadStatistics

* Update DDSketch implementation

* remove assertions that cause circular references

* add DDSketchMako as a subsclass from DDSketch

* Merge branch 'ddsketch_mako' of github.com:sfc-gh-khoxha/foundationdb into ddsketch_mako

* Revert "Merge branch 'ddsketch_mako' of github.com:sfc-gh-khoxha/foundationdb into ddsketch_mako"

This reverts commit cc29a68aef.

* add ddsketch mako class and rename export flag

* remove redundant decimal roundings

* print max/min/avg from ddsketch

* remove latency sample bin completly

* Make ThreadStatistics dump latency to a file and read from file in printReport()

* make sure to add latency data from file to final stats

* change mergeSketchReport to use new ThreadStatistics serialization (1)

* use C-style string arrays in Arguments instead of std::string

* remove unused header

* only serialize non-empty sketches

* fix CentOS build error

* Update report file count properly

* avoid deserializing empty sketches

* fix segmentation fault when getting file name for export_sketch_path

* make sure to properly add file to report_files list

* fix printing bugs when running in report mode

* fix incorrect insertion of report files

* don't use range based loop for char array

* don't reset args.num_report_files

* Update the usage info for new options

* switch to using std::vector for sketches instead of std::array

* make sure to use true/false instead of 1/0 for booleans

* remove op_name if not being used

* remove fp code in dumpThreadSamples

* replace lambda with function in printReport

* merge and print stats in seperate functions

* make sure to exit after printing report

* address review feedback

* make defaultMin, defaultMax static and move setBucketSize to protected

* switch to reverse iterators when moving backwards along bucket array
2022-06-06 18:19:31 -07:00
Clement Pang 043bc411ee
Make KeyArrayResult constructor public. (#7308) 2022-06-06 13:15:33 -07:00
Marian Dvorsky f887e3e364 Address review comments 2022-06-06 16:29:42 +02:00
Hao Fu 96d3951205
Fix versionstamp in Tester (#7312) 2022-06-03 08:56:01 -07:00
Marian Dvorsky d41ec69b23 Remove TesterApiWrapper, replace its uses with fdb_api.hpp 2022-06-03 11:55:33 +02:00
Hao Fu e7fa8e9f6f
Add versionstamp support in tuple (#7293)
Tuple in C++ needs to support Versionstamp.
2022-06-02 17:44:10 -07:00
Junhyun Shim 3e79735b2f
Authz JWT support (#7279)
* Add JWT support to TokenSign

* Encapsulate OpenSSL public/private key type

Type-safe passing around of keys without having to DER/PEM-serialize
(OpenSSL doesn't have distinct types for public and private key)

* Apply Clang format

* Add verify benchmark for JWT and FlatBuffers token

* Unit test base64url::{encode, decode}

* Make all payload fields optional

Let user code validate non-signature fields

* Make all payload fields optional

Completely defer field check to user code

* Move rapidjson from fdbclient to contrib

* Make fdbrpc's rapidjson linkage private

Currently only sources include them.

* Modify rapidjson path in apiversioner.py

* Algorithm::Unknown > Algorithm::UNKNOWN
2022-06-02 13:22:50 +02:00
Marian Dvorsky 138618eef9
Include symbols from fdb_c_internal.h to C bindings client library for Apple platform (#7292) 2022-06-01 14:04:53 -07:00
A.J. Beamon 526d8b217e Fix formatting issues 2022-05-31 11:37:52 -07:00
A.J. Beamon ee3508c2f8 Address review comments. Primarily, this changes Tenant creation to be done via a method call on Database. 2022-05-31 11:19:02 -07:00
Junhyun Shim f76d2454f5
Correctly wait on onError() future
Currently Mako worker thread waits on the original future that produced the error. Not the future returned by onError. Fix this.
2022-05-27 23:36:33 +02:00
Jon Fu 3bf4a8d521 move transaction creation to outside outer populate loop 2022-05-27 15:59:57 -04:00
Jon Fu 55972ca07e address code review comments 2022-05-27 10:57:57 -04:00
Jon Fu 8a7a8d0d59 delete tenants in memory when workload is done running 2022-05-26 17:14:41 -04:00
Jon Fu 8180414b4a remove printf debugging statements 2022-05-26 16:51:31 -04:00
Jon Fu c6ac9cf2a5 Merge branch 'main' of github.com:apple/foundationdb into jfu-mako-tenant-support 2022-05-26 16:29:52 -04:00
Jon Fu ddda238c01 fix issues with tenant creation/deletion and temporary tenants and strings 2022-05-26 16:29:16 -04:00
Jon Fu 01fa56630a WIP commit attempting to solve some tenant open and deletion issues 2022-05-26 14:55:59 -04:00
Jon Fu c24970915f fix off-by-one error 2022-05-25 12:16:15 -04:00
Jon Fu f25450ddd8 keep tenants in memory when running workload and introduce tenants to async run 2022-05-25 11:23:58 -04:00
Lukas Joswiak 18f80256b0 Add versionepoch fdbcli tests
Also removes the old targetversion fdbcli tests, as this command has
been removed (at least for now).
2022-05-23 11:45:18 -07:00
Junhyun Shim 02b2f97e99
mTLS test helpers (#7218)
* Add TLS option to (local_cluster|tmp_cluster).py

* Add TLS-enabled C API test
2022-05-23 12:47:51 +02:00
Kao Makino a9d62ea031
Merge pull request #7169 from sfc-gh-khoxha/main
Use correct transaction latency in async mako
2022-05-20 15:49:39 -07:00
Chaoguang Lin 5e8c3130e6 Extend the reboot interface to accept comma-delimited string;
Only fetch requested worker interfaces in rebootWorkerActor
Add killall test to multiprocess_fdbcli_tests
Make the rebootWorker only sends requests if all the requested workers are valid
Add comments; Organize print texts; Fix the bug clear the map every time run kill/suspend/expensive_data_check
2022-05-20 12:06:33 -07:00
Jon Fu 8ae951079f remove extra space for CI formatting 2022-05-20 13:28:26 -04:00
Jon Fu 612b94efa3 remove tenant_main arg and use CharsRef in error message format 2022-05-20 13:16:49 -04:00
Jon Fu b92b3b21b0 Refactor Tenant class in fdb_api header to match conventions of existing classes and move tenant creation into populate function 2022-05-20 13:03:59 -04:00
Hao Fu fc8d04b19d
Make java binding work for prefetch match index modes. (#7186)
* Make java binding work for prefetch match index modes.

It adds boundaryAndExist in return value, so that caller can see
whether a boundary index is orphan or not.

* Make Java code work with index match modes

* Add test for boundaryAndExist in java binding test
2022-05-20 09:10:02 -07:00
Jon Fu cdca68e26a only use tenant management prefix and default tenants to 0 in mako 2022-05-18 17:26:05 -04:00
Jon Fu 5a1c7f5c08 attempt to create fdb_api.cpp to separate static declaration 2022-05-18 17:16:30 -04:00
Jon Fu cc6620e9bb fix CI errors 2022-05-18 16:03:44 -04:00
Hao Fu dcacb30688
Add missing secondary queries tests for index prefetch (#7185)
* Add missing secondary queries tests for index prefetch

This change adds test for missing secondary queries for index prefetch,
in this case, MATCHED_ONLY mode would NOT return KV and UNMATCHED_ONLY
mode would return KV.

* remove default value for params
2022-05-18 12:13:19 -07:00
Jon Fu 17c8b4b2a0 Initial mako changes introducing Tenant API and replacing transaction resets 2022-05-18 14:48:40 -04:00
Hao Fu f00606fd71
Enable MATCHED and UNMATCHED mode for index prefetch (#7162)
* Enable MATCHED and UNMATCHED mode for index prefetch

MATCHED mode returns index entries whose secondary KVs are present,
UNMATCHED mode returns index entries whose secondary KVs are absent.

Note that the conflict read range of this txn is set in 2 steps:
* Set the conflict range for primary query according to request
* Set the conflict ranges for secondary queries according to responses.

As a result, conflicts of different match_index mode are taken care of.

* Fix c binding
2022-05-18 09:16:28 -07:00
Vaidas Gasiunas 5205b565ab
Upgrade Tests: Testing downgrade from 7.2 to 7.1 (#7179) 2022-05-18 11:59:29 +02:00
Ray Jenkins 7594f5c0f9
Mako add distributed tracing client (#7180)
* Add support for specifying distributed client tracer in Mako.

Introduces a new command line option --distributed_tracer_client. Defaults to disabled, user may specify
disabled, log_file, network_lossy, or sim_end.

Testing locally and verified with distributed tracing receiver. Note: We use a simple integer here
to denote tracer type rather directly using the TracerType enum in flow/Tracing.h. Including Tracing.h pulls in
FDBTypes.h and results in conflicts with several classes, including Database.
2022-05-17 15:23:14 -05:00
Vaidas Gasiunas 9966e3f7f7 Api Tester: print workload progress statistics in regular time intervals 2022-05-17 17:35:01 +02:00
Kevin Hoxha 4e6265537a Add correct transaction latency in ResumableStateForRunWorkload::onTransactionSuccess() 2022-05-16 14:06:55 -07:00
Alec Grieser 940ca2c6b8
Merge pull request #7166 from imperatorx/keyarrayresult-add-getter
Update KeyArrayResult.java
2022-05-16 14:03:24 -07:00
Ray Jenkins 3edc3431fe Update go bindings. 2022-05-16 14:48:40 -05:00
imperatorx 5e01c171be
Update KeyArrayResult.java
Add missing getter
2022-05-16 20:02:36 +02:00
Clement Pang 89c5901228 make MappedKeyValue constructor public 2022-05-16 09:49:50 -07:00
Vaidas Gasiunas 194d5d99e3
Merge pull request #7157 from sfc-gh-vgasiunas/vgasiunas-fix-upgrade-test
Restoring checks after each step in upgrade test
2022-05-16 09:12:22 +02:00
Hao Fu 853e6a346b
Optimization: support removing index conditionally (#7116) 2022-05-13 10:10:43 -07:00
Vaidas Gasiunas 7c9a213127 Upgrade Tests: removing failing downgrade test to 7.1 2022-05-13 11:47:57 +02:00
Markus Pilman 452315ee78
Build mockkms and add mockkms test (#7153) 2022-05-12 15:07:02 -07:00
Vaidas Gasiunas 1992898323 Add a regression test to upgrade from 7.1 to 7.2 and downgrade back 2022-05-12 15:34:03 +02:00
Vaidas Gasiunas c84e0b0d7a
Merge pull request #7085 from sfc-gh-vgasiunas/vgasiunas-cluster-wiggle-test
Cluster wiggle test
2022-05-11 13:06:33 +02:00
Markus Pilman 524365083d
Merge pull request #7106 from sfc-gh-ahusain/ahusain-fdb-mock-kms
FDB native MockKMS REST server implementation
2022-05-10 09:10:54 -07:00
Markus Pilman e1ab0f5c4d
Merge pull request #7086 from sfc-gh-vgasiunas/vgasiunas-upgrade-tests-71
Upgrade Tests: Including 7.1 into the upgrade paths
2022-05-10 09:09:55 -07:00
Hao Fu 968c2cad43
Reduce workload in CycleMultiClientIntegrationTest (#7100) 2022-05-09 11:48:45 -07:00
Ata E Husain Bohra d6e5549226 FDB native MockKMS REST server implementation - Golang
Description

Major changes include:
1. FDB native MockKMS REST server implementation - Golang based.
2. Implements "getEncryptionKeys" endpoint utilized by
   FDB RESTKmsConnector module.
3. Ability to inject faults to induce errors at various points
   during query execution

NextSteps:

Need to integrate MockKMS to FDB build system.

Testing

1. Implements mockkms_test.go module providing extensive test coverage
   for newly added code.
2. Postman based local testing.
2022-05-07 20:53:13 -07:00
A.J. Beamon 68a46ca14d
Merge pull request #6852 from sfc-gh-jfu/jfu-list-tenants
Add listTenants commands to python and java bindings
2022-05-06 08:37:17 -07:00
Vaidas Gasiunas 27c0113305 Upgrade Tests: Including 7.1 into the upgrade paths 2022-05-06 15:21:57 +02:00
Vaidas Gasiunas fab8f35683 Cluster wiggle test 2022-05-06 15:00:40 +02:00
Vaidas Gasiunas 254da4d796
Merge pull request #7065 from sfc-gh-jslocum/arm_bg_fix
Fix for arm BG tests
2022-05-05 11:18:44 +02:00
Jon Fu c001d55c24 push tenant_list result as a packed tuple to stay consistent with other operations 2022-05-04 16:33:46 -04:00
Josh Slocum a7fdb42db8 Fix for arm BG tests 2022-05-04 14:54:16 -05:00
Jon Fu b5556c57f9 adjust instruction ordering in tenant_list api for binding tester 2022-05-04 13:59:19 -04:00
Jon Fu 9798e5fc83 Merge branch 'main' of github.com:apple/foundationdb into jfu-list-tenants 2022-05-04 12:49:43 -04:00
Jon Fu 96a35264b4 return tenant_list as one operation and validate with assertions 2022-05-04 12:48:48 -04:00
Hao Fu 97eb12381b
implement equals and hashCode in MappedKeyValue (#7041) 2022-05-03 12:24:26 -07:00
Hao Fu fa2e85f1d3
Add comment about getMappedRange parameters (#7044) 2022-05-02 15:17:14 -07:00
Rajiv Ranganath cf6e39af79 docs: add `GET_RANGE_SPLIT_POINTS`
Add `GET_RANGE_SPLIT_POINTS` instruction documentation.
2022-05-02 13:31:20 -07:00
Ray Jenkins dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Josh Slocum 57e1b487f1 Fixing ASAN alloc-dealloc-mismatch 2022-05-02 12:56:05 -05:00
Jon Fu ff216c2f57 add tenant list metadata to binding tester 2022-05-02 13:42:11 -04:00
Jon Fu d953b961b7 Merge branch 'main' of github.com:apple/foundationdb into jfu-list-tenants 2022-04-29 13:16:54 -04:00
Jon Fu 2afaf55a48 fixed some binding tests and split stack operation between key and value of tenant list 2022-04-29 13:16:04 -04:00
Josh Slocum 14e7738455 review comments 2022-04-27 16:08:00 -05:00
Josh Slocum 92da9d12d4 Improvements to fix 2022-04-27 16:08:00 -05:00
Josh Slocum 1bd72dc99e Allowing for blob granules to initialize at start of TesterBlobGranuleCorrectness 2022-04-27 16:08:00 -05:00
Vaidas Gasiunas 6f841446a5 API Tests: Enable client logs. Dump out the logs in case of an error 2022-04-27 10:02:20 -04:00
Vaidas Gasiunas d021b404ea ApiTester: Fix error code handing in case of a timeout 2022-04-27 10:02:20 -04:00
Junhyun Shim 1ee7702779
Merge pull request #6489 from sfc-gh-jshim/mako-cpp-async
Refactor Mako for C++ and add asynchronous execution support
2022-04-27 10:49:21 +02:00
Junhyun Shim 2f9506f5cd Introduce hard cap to latency sampling & apply clang format 2022-04-26 11:19:48 +02:00
Junhyun Shim 81211989f1 Disable new sample collection after first bad_alloc
- Add more info to --async_xacts description
- Clean up unused macro in fdb_api.hpp
2022-04-26 10:54:39 +02:00
Jon Fu a5e52c9450 Merge branch 'main' of github.com:apple/foundationdb into jfu-list-tenants 2022-04-25 14:56:20 -04:00
Ray Jenkins 1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. (#6855)" (#6941)
This reverts commit 5df3bac110.
2022-04-25 09:29:56 -05:00
Junhyun Shim 81f9279db2 Fix GCC error about memset-ing atomic vars 2022-04-25 15:09:29 +02:00
Junhyun Shim d431a9f11b Wrap platform-specific PATH_MAX header in another 2022-04-25 13:53:05 +02:00
Junhyun Shim c4ac1ab9d0 Fix SGETRANGE 2022-04-25 13:24:56 +02:00
Junhyun Shim cfdff223ff Merge remote-tracking branch 'origin/main' into mako-cpp-async 2022-04-25 13:21:14 +02:00
Junhyun Shim 3536cab730 Apply clang format 2022-04-25 10:48:11 +02:00