Commit Graph

26498 Commits

Author SHA1 Message Date
Zhe Wu 3056702f2b A version that pass 100K simulation 2023-07-26 11:12:46 -07:00
Zhe Wu dcfbb55ad5 Resolve rebase conflict 2023-07-17 11:18:43 -07:00
Zhe Wu 16b3c643d7 Testing changes 2023-07-17 11:14:24 -07:00
Zhe Wu 00cdf09966 Cluster controller monitors recovery stats and trigger recovery if current recovery contains degraded servers 2023-07-17 11:14:22 -07:00
Zhe Wu 9670ed1cd8 Make TLog explicitely monitor connectivity issue during [start version, recover version] recovery 2023-07-17 11:12:28 -07:00
Zhe Wu 458d210584 Add an option in worker to force detect connectivity issue to other transaction workers in primary region 2023-07-17 11:12:28 -07:00
Nim Wijetunga 01acd8d3c6
blob granule inplace encryption (#10619) 2023-07-17 10:44:11 -07:00
Jingyu Zhou 4b00018a33
Fix Implib.so imports when 'amd64' architecture is using (#10618)
Co-authored-by: Alexey Marchenko <santana705736@gmail.com>
2023-07-17 12:04:56 +02:00
Zhe Wang 522c9d4f0f
Add new implementation of audit storage for user data (#10613)
* remainingBudgetForAuditTasks should be managed within audit

* fix CI

* add audit storage test for various ranges

* clean DD

* new auditStorageUserDataQ

* fix assert fail in startTrackShardAssignment

* fix assert fail in ssaudit

* address comments

* replace assert with audit_cancel in ss audits

* add audit check progress tool

* add observability to audit progress and fix audit bugs

* fix audit progress issues and add sim test for audit progress and add trace event for the audit progress and add fdbcli to track the audit progress

* remove old audit storage on SS

* check audit progress when auditCore completes
2023-07-16 09:56:26 -07:00
Nim Wijetunga 7f2260bbd2
Add Encryption Related Latency Metrics (#10596)
* add ss and cp latency metrics

* make changes
2023-07-14 11:30:16 -07:00
Hui Liu 609d43a384
Find next full keys for split points (#10616) 2023-07-14 10:25:54 -07:00
Josh Slocum 8ce1796a7a
Stable http ports (#10604)
* Ensuring HTTP ports are stable for the same servers, and adding a test http server that ensures no other users accidentally talk to it

* fixing warning with werror

* more werror fixes
2023-07-13 14:06:52 -04:00
Yanqin Jin 9c51fa082e
Merge pull request #10592 from sfc-gh-yajin/validate-no-data-outside-tenants
If tenant mode is REQUIRED, then we should verify that in the normal key space, no data exists outside tenants' prefixes. This applies to data clusters (also known as partition clusters) in a metacluster and standalone clusters with tenants.
For the management cluster of a metacluster, we should verify that no data exists outside the prefix ranges specified by tenant/ and metacluster/ in the normal key space.

Test plan:
devRunCorrectnessFiltered +Metacluster* +Tenant* --max-runs 100000

20230702-052847-yajin-082705d269588494. 0 Failure

devRunCorrectness --max-runs 100000

20230702-134219-yajin-e9cce7bd165e70a9. 1 Failure, unrelated to this change
2023-07-12 09:05:39 -07:00
Jingyu Zhou 6e43d0e759
Merge pull request #10609 from xis19/local_cluster
fixup! Set up fdbcli path when running a local cluster
2023-07-11 16:12:13 -07:00
Xiaoge Su 252cf224e4 fixup! Set up fdbcli path when running a local cluster 2023-07-11 13:13:39 -07:00
Trevor Clinkenbeard c54e1101c3
Merge pull request #10586 from sfc-gh-tclinkenbeard/main-gtt-forget-old-ss
Add `ExpectStableThroughput` simulation test
2023-07-10 14:16:05 -07:00
A.J. Beamon c24709d630
Merge pull request #10552 from sfc-gh-ajbeamon/automatic-idempotency-fix
Automatic idempotency test flagged improper cleanup
2023-07-10 10:17:33 -07:00
Josh Slocum 0e6216e926
fix per-scan-loop throttling to properly throtte in the loop (#10502) 2023-07-10 13:02:12 -04:00
Josh Slocum 8ec439f023
fixing apparent source of nondeterminism in bgfiles (#10605) 2023-07-10 12:59:30 -04:00
sfc-gh-tclinkenbeard 2673a727ac Merge remote-tracking branch 'origin/main' into main-gtt-forget-old-ss 2023-07-09 12:57:23 -07:00
Zhe Wu 8f33bb2e8a
Merge pull request #10602 from halfprice/zhewu/sim-disconnection
Simulate connection failure between specific pairs of machines
2023-07-07 14:24:59 -07:00
Zhe Wu d8eaf28bfd Simulate connection failure in simulation 2023-07-07 11:34:12 -07:00
Yanqin Jin 2c8b682310 Fix build issues 2023-07-05 15:49:29 -07:00
Yi Wu ddf1dd7cd7
Merge pull request #10589 from sfc-gh-yiwu/fix_inject
fix worker server reporting IoTimeoutError as injected fault
2023-07-05 15:23:22 -07:00
Yi Wu b16c7518b4 fix worker server reporting IoTimeoutError as injected fault 2023-07-05 13:30:56 -07:00
Jon Fu d9e61d3d98
Fix bug in FDB MultiVersionTransaction.actor.cpp (#10576)
* throw error in tenantUpdater if result is an error

* avoid throwing error in thread future lambda
2023-07-05 15:23:38 -04:00
Yao Xiao ab72951034
Add knob for manifest file size and log rocksdb status. (#10567) 2023-07-05 10:40:43 -07:00
Yanqin Jin 13fac35f53 SNOW-791059 Verify no data outside tenants in REQUIRED mode (#489)
If tenant mode is REQUIRED, then we should verify that in the normal key space, no data exists outside
tenants' prefixes. This applies to data clusters (also known as partition clusters) in a metacluster and standalone clusters
with tenants.
For the management cluster of a metacluster, we should verify that no data exists outside the prefix ranges specified by `tenant/` and `metacluster/` in the normal key space.

Test plan:
devRunCorrectnessFiltered +Metacluster* +Tenant* --max-runs 100000

20230702-052847-yajin-082705d269588494. 0 Failure
devRunCorrectness --max-runs 100000

20230702-134219-yajin-e9cce7bd165e70a9. 1 Failure, unrelated to this change
2023-07-05 10:33:49 -07:00
sfc-gh-tclinkenbeard 7151dccd34 Fix typo in ExpectStableThroughputWorkload::disableFailureInjectionWorkloads 2023-07-04 14:22:22 -07:00
Yanqin Jin 032f49bc1a
Merge pull request #9181 from sfc-gh-yajin/update-doc-building_cluster
Update documentation about building a cluster

Test plan:

cd foundationdb/build/ && cmake -G Ninja .. && ninja docpreview
The above ninja docpreview will indicate a port number, e.g. 14244.

Open browser and go to: http://127.0.0.1:14244/building-cluster.html?highlight=build
2023-07-02 22:34:41 -07:00
Yanqin Jin 09d44d9d9f Merge remote-tracking branch 'origin/main' into update-doc-building_cluster 2023-07-02 16:43:55 -07:00
A.J. Beamon 696a5b3674 Iterate decodedKeys by const& 2023-06-30 18:26:23 -07:00
Ata E Husain Bohra 7779c908b3
EaR: Remove usage of ENABLE_CONFIGURABLE_ENCRYPTION knob (#10570)
Description

Given Configurable encryption has been checked in and being tested via
simulation for more than a month and also to avoid penalty of accessing
KNOBS in inline commit path, patch retires the KNOB and make
ConfigurationEncryption default EaR mode for FDB.

BlobCipher still supports the old format header and encryption semantics,
will remove the dead code as a followup PR.

Testing

devRunCorrectness - 100K
2023-06-30 17:48:09 -07:00
A.J. Beamon 76b73de506 Disable automatic idempotency most of the time, and when it is disabled check that cleaning is working 2023-06-30 16:09:11 -07:00
sfc-gh-tclinkenbeard 6fbd6ef35c Apply clang-format to fdbserver/workloads/ThroughputQuota.actor.cpp 2023-06-30 13:20:47 -07:00
sfc-gh-tclinkenbeard b7bd6fc602 Cleanup stale throughput statistics 2023-06-30 13:04:56 -07:00
Hui Liu aed406e947
Show manifest dump info in status json even when no flushing (#10584) 2023-06-30 13:04:43 -07:00
sfc-gh-tclinkenbeard ba74cd25f3 Add ExpectStableThroughput test 2023-06-29 22:19:39 -07:00
sfc-gh-tclinkenbeard d33d0ece55 GlobalTagThrottler should decay throughput from missing storage servers 2023-06-29 20:53:09 -07:00
Hui Liu 9655cbf220
Fix ubsan error BlobGranuleVerifyBalanceClean.toml (#10581) 2023-06-29 16:46:42 -07:00
He Liu a6bd2699d2
cleanups, no logic changes. (#10566) 2023-06-29 13:20:06 -07:00
Trevor Clinkenbeard 4ec3eac0e8
Merge pull request #10572 from sfc-gh-tclinkenbeard/main-decouple-token-bucket-knobs
Decouple token bucket knobs for different types of throttlers
2023-06-28 12:29:31 -07:00
Hui Liu 66a7acd960
Fix blob restore stuck issue (#10574) 2023-06-28 10:23:11 -07:00
sfc-gh-tclinkenbeard 36542c139d Update test parameters for /GrvProxyTagThrottler/LimitedIdleBudget test
This is to match the updated default value of
TAG_THROTTLE_MAX_EMPTY_QUEUE_BUDGET.
2023-06-28 09:24:32 -07:00
sfc-gh-tclinkenbeard 4f711b15cd Fix comments for proxy knobs 2023-06-28 01:27:35 -07:00
sfc-gh-tclinkenbeard 86fae8b7b5 Increase default value of TAG_THROTTLE_MAX_EMPTY_QUEUE_BUDGET 2023-06-28 01:27:28 -07:00
sfc-gh-tclinkenbeard 6e86f94cb9 Decouple token bucket knobs for different types of throttlers 2023-06-28 01:27:19 -07:00
Jingyu Zhou cf3b5d8c57
Merge pull request #10568 from jzhou77/main 2023-06-27 17:25:51 -07:00
Jingyu Zhou bbf9632964 Fix joshua_logtool.py to allow . in ensemble ID
e.g., 20230627-102446-nightly-release-7.3-clang-45609535-729-20a8c
2023-06-27 15:09:47 -07:00
Jingyu Zhou 178cddbffd
Merge pull request #10563 from sfc-gh-etschannen/fix-durable-change-feeds
cancel durable change feed actors in DatabaseContext destructor
2023-06-27 13:39:20 -07:00