Commit Graph

954 Commits

Author SHA1 Message Date
Xiaoge Su b2e50721e4 fixup! Fix the regexp warning in generate_asm.py 2024-04-22 18:44:32 -07:00
Boris Korzun f7a443aea4 Add libfmt 10+ support (#11140) 2024-02-13 23:09:58 +03:00
Dimitris Apostolou a88114c222
Fix typos 2024-02-07 01:16:00 +02:00
Dan Lambright 015167c17e
Throttle commits against hot shards (#10970)
* throttle hot shards

* expire throttled shards over time

* add backoff

* Parallelize messaging from RK to CP

* Obtain shards from a single SS

* handle expired transactions

* bump transaction_throttled_hot_shard

* Change SevError to SevWarn for CannotMonitorHotShardForSS

* Add log per request
2023-10-31 12:01:34 -04:00
Boris Korzun ac30cb0783 Add preinstalled doctest support 2023-10-06 13:33:30 +03:00
Boris Korzun e04f0467a0 Disable explicit std::filesystem linking on CLang 2023-10-05 17:23:26 +03:00
Jingyu Zhou 4b00018a33
Fix Implib.so imports when 'amd64' architecture is using (#10618)
Co-authored-by: Alexey Marchenko <santana705736@gmail.com>
2023-07-17 12:04:56 +02:00
Zhe Wu 5c8a163c72
Update main branch to 7.4 (#10459)
* Update main branch to 7.4

* Update API version to 740

* Makes fdb_c_client_config_tests.py passing after API version update

* Remove from_7.3.0_until_7.4.0 and add from_7.3.0

* Update tests in fdb_c_client_config_tests.py
2023-06-15 10:19:39 +02:00
Andrew Noyes 4d2f038b60 Remove unnecessary duplicate declaration of fdb_tenant_list_blobbified_ranges 2023-05-18 16:20:18 -07:00
Sam Gwydir 6c16875c34
Add networkoption to disable non-TLS connections (#9984)
* Add networkoption to disable non-TLS connections

* add disable plaintext connection to fdbserver

* python doc

* Formatting

* Add tls disable plaintext connection to client api test

* review

* fix negative test

* formatting

* add TLS support to c client config tests

Adds support for TLS in the client and server separately

* add tests for disable_plaintext_connections

Test TLS and Plaintext Clusters and Clients

* Fix documentation

* Rename option to indicate it is client-only

* clearer formatting

* default to allowing plaintext connections

* add SetTLSDisablePlaintextConnection to go bindings
2023-05-13 00:14:11 +02:00
Zhe Wu e3490a1af8 Update API version to 730 2023-05-10 11:26:46 -07:00
Zhe Wu a6d6c70aad
Merge pull request #10103 from halfprice/zhewu/update-main-to-7.4
Bring main branch to 7.3
2023-05-10 09:53:27 -07:00
Zhe Wu 761cdbc019 Bring main to 7.3 2023-05-09 21:14:16 -07:00
A.J. Beamon 686c391281
Merge pull request #10093 from sfc-gh-ajbeamon/apply-flake8
Fix several issues found by flake8
2023-05-09 09:05:14 -07:00
Josh Slocum b5a9997217
adding DBPerTXN blob granule tests (#10095)
* adding DBPerTXN blob granule tests

* fixing comment
2023-05-08 15:43:29 +02:00
A.J. Beamon fef2967d62 Fix several issues found by flake8. Ignore E402, which detects when we have import statements not at the top of the file, since we have many files that use this pattern and are not easy to fix. 2023-05-02 10:06:40 -07:00
A.J. Beamon 182dc93ebd Apply black format to most Python files, excluding a few cases where we have Python 2 files and a few files written externally. Add external files as exclusions to the precommit checks. 2023-04-28 11:46:41 -07:00
hao fu 29161b2fda Revert matchIndex feature
It is not protocol compatible, revert it to avoid deployment issue.
Will have a new PR to have the feature if moving forward.
2023-04-17 09:39:45 -07:00
Ata E Husain Bohra fe0a4df06a
EaR: Implement Key Check Value semantics (#9936)
* EaR: Implement Key Check Value semantics

Description

Key Check Value (KCV) is a checksum of cryptographic encryption key
used to validate encryption keys's integrity. FDB Encryption at-rest
relies on external KMS to supply encryption keys.

Patch proposes following major changes:
1. Implement Sha256 based KCV implementation to protect against
'baseCipher' corruption in two possible scenarios:
 a) potential corruption external to FDB
 b) potential corruption within FDB processes.
2. Scheme persists computed KCV token in block encryption header,
which then gets validated as part of header validation during
decryption.
3. FDB Encryption key derivation uses HMAC_SHA256 digest generation
scheme, which allows max 64 bytes of 'cipher buffer', patch add
required check to ensure 'baseCipher' length are within bounds.
OpenSSL HMAC underlying call ignores extra length if supplied, however,
it weakens the security guarantees, hence, disallowed.

Testing

devRunCorrectness - multiple 500K runs
Valgrind & Asan - BlobCipherUnit, RESTKMSUnit, BlobGranuleCorrectness*,
EncryptionOps, EncryptKeyProxyTest
2023-04-12 14:29:31 -07:00
Vaidas Gasiunas 894f555e95
Test loop profiler using API Tester (#7174)
* Api Tester: Specify knobs in the toml file; Test loop profiler

* Gracefully stop the loop profiler thread

* Protect loop profiler thread by mutex

* Create loop  profiler thread only if is not stopped
2023-03-30 23:00:23 +02:00
Vaidas Gasiunas 7870479b66 Improve stability of test_external_client_not_matching_cluster_version_ignore 2023-03-20 17:30:35 -07:00
A.J. Beamon 33d501d36a Avoid potential access of empty optionals when printing errors in the api tester 2023-03-14 10:41:43 -07:00
Marian Dvorsky 90048d1e92
Enable fdb_install of fdb_c_shim for sanitizer builds (#9646) 2023-03-10 14:33:59 +01:00
Josh Slocum e1b620135b Merge branch 'main' into bg_latency_fixes 2023-03-06 09:23:11 -06:00
Junhyun Shim a2a29af56e
Temporarily disable client trace check for older version (#9578)
* Disable client trace check test on older version

Older version doesn't guarantee trace flush upon network::stop()
Comment it out for the time being

* Black-reformat authz and client config tester scripts
2023-03-06 15:36:28 +01:00
Josh Slocum 526134f598 review comments 2023-03-03 13:52:59 -06:00
Vaidas Gasiunas a0e1b4b369 Fixing exported symbols extraction from C headers for Apple build 2023-03-03 14:04:35 +01:00
Vaidas Gasiunas e95ff58b95 Reformat symbolify.py 2023-03-03 14:00:59 +01:00
Josh Slocum fdcb6aed5f add timestamp to bg verbose tester logs 2023-03-02 09:34:40 -06:00
Markus Pilman 5bebb5b4aa
Merge pull request #9492 from sfc-gh-vgasiunas/vgasiunas-api-version-defs
Centralize definition of API Version for Java, Python and C API
2023-02-28 12:04:02 -07:00
Russell Sears d2af157189 Add support for using pre-packaged jemalloc, doctest_proj, and fix python support for old fdbserver binaries 2023-02-27 15:38:58 -06:00
Vaidas Gasiunas 8ddda99036 Fix name of FDB_USE_LATEST_BINDINGS_API_VERSION 2023-02-27 16:53:35 +01:00
Vaidas Gasiunas ba726fac87 Replace hardcoded API version checks for 720 and 730 2023-02-27 16:18:01 +01:00
Vaidas Gasiunas 1b6ced1a09 Define latest C API version in one place 2023-02-27 15:13:56 +01:00
Josh Slocum 958f3b531b
Plumbing blob worker mapping through commit proxy like storage server (#9401)
* Plumbing blob worker mapping through commit proxy like storage server mapping

* review comments

* formatting
2023-02-21 13:21:44 -06:00
sfc-gh-tclinkenbeard 398079db3a Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-20 17:54:06 -08:00
Josh Slocum bfb3ffc509
added c and java apis for granule flush (#9412) 2023-02-20 10:28:11 -06:00
sfc-gh-tclinkenbeard 1aef6cb5f7 Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-17 20:41:59 -08:00
Josh Slocum 6c2fb13173
adding wait parameter to blobbify api (#9360)
* adding wait parameter to blobbify api

* formatting

* fixing comment style

* fixing bug and adding debugging

* adding blob ranges unit test

* testing both blobbify cases in cancel

* formatting

* switch to explicit blocking api instead of boolean flag

* remove comments

* format
2023-02-17 12:20:53 -06:00
Josh Slocum c26831ec04
adding version metadata to blob granule file pointers (#9392) 2023-02-16 17:11:11 -06:00
Josh Slocum eefc889389
Add tenant and encryption support to new bg file apis (#9315)
* Add tenant and encryption support to new bg file apis

* formatting

* fixing comment style for linter
2023-02-15 11:48:40 -06:00
Junhyun Shim 380a5aa8e8
Merge pull request #9335 from sfc-gh-jshim/fix-mako-token-reassign-after-reset
Mako: fix token not being reassigned after every reset
2023-02-09 22:14:39 +01:00
Josh Slocum 81c984e48a
adding encryption at rest support to local cluster and api tester (#9325)
* adding encryption at rest support to local cluster and api tester

* adding encrypted variant of bg tenant test and renaming ear
2023-02-09 10:23:17 -06:00
Junhyun Shim 357cec17d8 Mako: fix createNewTransation return value
Should return token instead of tenant name
2023-02-09 16:00:48 +01:00
Junhyun Shim 499a7de718 Mako: add missing token reassignment after transaction reset
Also remove redundant calls to tx.reset() in step functions after OP_COMMIT
(The step function caller is already resetting the object after OP_COMMIT is run successfully)
2023-02-09 13:35:21 +01:00
sfc-gh-tclinkenbeard 09ad864eb5 Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-08 11:25:14 -08:00
Junhyun Shim d7d0b658a3
Merge pull request #9314 from sfc-gh-jshim/authz-tenant-name-to-tenant-id
Make Authz use tenant ids instead of tenant names
2023-02-08 20:13:29 +01:00
Josh Slocum ebbd221fe7
formatting files (#9322)
* formatting files

* another file
2023-02-08 13:01:24 -06:00
Junhyun Shim d32d3dd085 Make token option NOT survive transaction hard resets 2023-02-08 18:12:22 +01:00
Vaidas Gasiunas e1ec0d1ae2
Fix UBSAN build for C shim library (#9311) 2023-02-07 20:17:07 +01:00