Commit Graph

26732 Commits

Author SHA1 Message Date
Boris Korzun 4419cd6bec Implib.so: Fix issue with trailing newline in c++filt output 2023-09-20 15:25:42 +03:00
Jingyu Zhou aa1d005cc5
Merge pull request #10907 from jzhou77/main
ClusterController watches changes to storage metadata
2023-09-19 12:21:35 -07:00
Jingyu Zhou f72eab118d Skip redwood for clearInflightCommits unit test
Redwood complains DB is invalid for this unit test:

Assertion keyProvider.isValid() || db.isValid() failed @ /root/src/foundationdb/fdbserver/VersionedBTree.actor.cpp 8029:
2023-09-18 14:34:36 -07:00
Jingyu Zhou 307491d68e Use getRange for server metadatas
To reduce read load on SSes that serve the reads.
2023-09-18 13:32:53 -07:00
Zhe Wu 8706098916
Merge pull request #10904 from halfprice/zhewu/txn-window
Create MAX_WRITE_TRANSACTION_LIFE_VERSIONS client knob
2023-09-15 15:51:13 -07:00
Jingyu Zhou 12fe500633 ClusterController watches changes to storage metadata
To retrieve storage metadata for every status json request is very expensive
for clusters with a large number of storage servers. So I change the logic so
that ClusterController actively monitors changes to storage metadata, and only
retrieves them when there is a change.
2023-09-15 14:19:04 -07:00
Zhe Wu aea57f6da4 Create MAX_WRITE_TRANSACTION_LIFE_VERSIONS client knob 2023-09-14 14:01:43 -07:00
Zhe Wang 1f6215808b
fix multiple requests from DDDoAudit (#10902) 2023-09-14 11:04:19 -07:00
Zhe Wang 29a2f63f8d
Fix SSShard Audit (#10896)
* fix ssshard

* address comments

* fmt
2023-09-13 21:15:12 -07:00
Jingyu Zhou ad3bda993d
Merge pull request #10901 from jzhou77/main
Add a test case for storage engine
2023-09-13 20:24:44 -07:00
Jingyu Zhou 977851fa39 Fix sharded rocks failure by adding a shard 2023-09-13 15:53:27 -07:00
Yao Xiao d90a5fa742
Disable read aware DD. (#10900) 2023-09-13 15:50:58 -07:00
Jingyu Zhou 42df53b3bc Add a test case for storage engine
This test validates that in-flight commit to the storage engine is properly
handled. As found in https://github.com/apple/foundationdb/pull/10714, an
engine could misses in-flight data and cause data corruptions.

The test case is modeled after the above corruption: insert data, then clear
the data in the next commit to the storage engine, and finally varify that the
data is cleared.
2023-09-13 15:25:25 -07:00
Jingyu Zhou 5b4e1ae5f2
Merge pull request #10897 from jzhou77/main
Disable sharded rocks for more sim tests
2023-09-13 12:25:57 -07:00
Jingyu Zhou d568832a8f Disable sharded rocks for more sim tests
Saw ExternalTimeout test failures for these.
2023-09-13 10:58:13 -07:00
A.J. Beamon 908d84c893
Merge pull request #10518 from oleg68/bugfix/clang-17
Fixed compiling foundationdb with the clang 17 compiler
2023-09-13 09:59:52 -07:00
Oleg Samarin b9b4e1ebe4 Fixed formatting 2023-09-13 18:56:25 +03:00
Oleg Samarin 3c75b5d7de
Apply suggestions from code review
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-09-13 17:18:06 +03:00
Jingyu Zhou 6ff6f98fab
Merge pull request #10894 from jzhou77/main
Fix an assertion failure in tests
2023-09-12 15:29:40 -07:00
Zhe Wang 2a40fb4135
fix audit state serializer (#10889) 2023-09-12 14:48:04 -07:00
Jingyu Zhou bed58e38f2 Fix an assertion failure in tests
The buggified value 120 triggers assertion failure, so restore it to the default.

Reproduction seed: -f ./tests/rare/ClogTlog.toml -s 537536800 -b on
commit: 163774cfd at release-7.1
2023-09-12 14:06:30 -07:00
Yao Xiao 55f2eaec2d
Release notes for 7.3.17. (#10874)
* Release notes for 7.3.17.

* Update documentation/sphinx/source/release-notes/release-notes-730.rst

Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>

* Update documentation/sphinx/source/release-notes/release-notes-730.rst

Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>

* Update documentation/sphinx/source/release-notes/release-notes-730.rst

Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>

* Update documentation/sphinx/source/release-notes/release-notes-730.rst

Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>

* Update release-notes-730.rst

* Update release-notes-730.rst

---------

Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2023-09-11 13:11:23 -07:00
Jingyu Zhou cac3038bbb
Merge pull request #10888 from jzhou77/main
Disable sharded rocks for more sim tests
2023-09-11 13:09:33 -07:00
Jingyu Zhou b2d724f4a5 Disable sharded rocks for more sim tests
Saw ExternalTimeout test failures for these.
2023-09-11 10:57:54 -07:00
Jingyu Zhou d37ed04f6c
Merge pull request #10885 from hfu94/proxyM
Add proxy to backup agent via global var
2023-09-09 11:13:22 -07:00
neethuhaneesha cb3dac21a9
Release notes for 7.1.38/7.1.39 (#10883) 2023-09-09 10:23:43 -07:00
Hao Fu 6d9c53f8c4 Add proxy to backup agent via global var
backup agent itself does not have proxy info.
This changes adds the proxy via a global var.
2023-09-08 10:27:01 -07:00
Zhe Wu dc48f1b325
Merge pull request #10876 from halfprice/zhewu/make-sure-end-is-not-persist
Make sure that storage and tlog are always set to a valid type
2023-09-07 10:58:35 -07:00
Jingyu Zhou d47e0eaaaa
Merge pull request #10877 from neethuhaneesha/wigglelocality
Locality check on perpetualStorageWiggleIDPrefix when DD restarts
2023-09-07 10:09:08 -07:00
Hui Liu b2ee1fb6c4
Tune blob restore default knob (#10871) 2023-09-07 09:36:51 -07:00
Steve Atherton 02359370b0
Merge pull request #10878 from sfc-gh-satherton/backup-agent-layer-status-yield
Add yields to backup agents to avoid slow tasks
2023-09-06 23:30:53 -07:00
Steve Atherton 596ab8d83d Yield between parsing backup layer status JSON documents and between merging them into the combined backup layer doc to avoid SlowTasks when there are many agents running. 2023-09-06 19:24:15 -07:00
neethuhaneesha 9bafef4fd2 Locality check on perpetualStorageWiggleIDPrefix when DD restarts 2023-09-06 16:00:32 -07:00
Zhe Wu 9e5488dd3d Make sure that storage and tlog are always set to a valid type 2023-09-06 14:58:42 -07:00
dependabot[bot] e81cf16676
Bump gopkg.in/yaml.v3 in /fdbkubernetesmonitor (#10850)
Bumps gopkg.in/yaml.v3 from 3.0.0-20210107192922-496545a6307b to 3.0.0.

---
updated-dependencies:
- dependency-name: gopkg.in/yaml.v3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-06 12:17:11 +02:00
Hui Liu 4d2a7d507d
Add a new blob restore state to fix a race after data copy (#10854) 2023-09-05 14:04:35 -07:00
Zhe Wu ae11557fc2
Merge pull request #10866 from halfprice/zhewu/clear-wiggle-storage-engine
Fix status json schemas perpetual_storage_wiggle_engine
2023-09-05 12:23:00 -07:00
Zhe Wu cb1e792169 Fix status json schemas perpetual_storage_wiggle_engine 2023-09-05 11:10:53 -07:00
Zhe Wu e2f5c50a7b
Merge pull request #10828 from halfprice/zhewu/clear-wiggle-storage-engine
Add option to set perpetual_storage_wiggle_engine to none
2023-09-05 11:06:21 -07:00
Hui Liu 00d3062728
Initialize apply mutations map for restore to version (#10857) 2023-09-05 10:03:35 -07:00
Jingyu Zhou 3c5a38b120
Merge pull request #10859 from sfc-gh-ljoswiak/fixes/database-context-status
Avoid creating a DatabaseContext on every status json call
2023-09-02 10:25:47 -07:00
Jingyu Zhou 52209ef551
Merge pull request #10714 from neethuhaneesha/singlekeydelete
Storing the inprocess-commit deleted keysets until the commit is complete
2023-09-01 20:54:49 -07:00
Lukas Joswiak b3e58761d3 Avoid creating a DatabaseContext on every status json call
DatabaseContext currently leaks memory by creating `Counter`s with
unique IDs on construction. Each status json call creates a new
`DatabaseContext` object, causing a memory leak over time.
2023-09-01 19:55:17 -07:00
A.J. Beamon ead6c37e4a
Merge pull request #10662 from sfc-gh-tclinkenbeard/main-fix-grv-queue-leak
Fix GRV queue leak
2023-09-01 09:45:49 -07:00
Lukas Joswiak bfb1c51299 Add `clearknob` fdbcli command
The `clearknob` command clears the value that a knob has been set to in
the configuration database. Note that this does not mean the knob value
itself gets cleared - only the value in the configuration database is
cleared. The value of the knob will revert to whatever is hardcoded in
the corresponding `*Knobs.cpp` file.

Sample `fdbcli` session:

```
Welcome to the fdbcli. For help, type `help'.
fdb> getknob min_trace_severity
`min_trace_severity' is not found
fdb> setknob min_trace_severity 20
Please set a description for the change. Description must be non-empty
description: test
Committed (2)
fdb> getknob min_trace_severity
`min_trace_severity' is `20'
fdb> clearknob min_trace_severity
Please set a description for the change. Description must be non-empty
description: clear
Committed (4)
fdb> getknob min_trace_severity
`min_trace_severity' is not found
```

Transactions are also supported with the new `clearknob` command:

```
Welcome to the fdbcli. For help, type `help'.
fdb> begin
Transaction started
fdb> setknob min_trace_severity 20
fdb> clearknob min_trace_severity
fdb> commit
Please set a description for the change. Description must be non-empty.
description: test
Committed (16)
fdb> getknob min_trace_severity
`min_trace_severity' is not found
```
2023-08-31 17:36:05 -07:00
Adonis Ling 46c837830d
Fix missing jemalloc header errors (#10849) 2023-08-31 13:34:06 -05:00
Egor Zhdan 3428d4678c
[swift] Remove deprecated Swift attr spelling: `import_as_ref` (#10825)
This spelling is deprecated and will be removed in the future version of Swift:
```
__attribute__((swift_attr("import_as_ref")))
```

The new correct spelling is:
```
__attribute__((swift_attr("import_reference")))
```
2023-08-31 13:23:23 -05:00
neethuhaneesha 08cc6518bb Deleting rocksdb keysSet after the commit completes. 2023-08-30 15:25:23 -07:00
neethuhaneesha 68d5708e02 Storing the inprocess-commit deleted keysets until the commit is complete. (#10672) 2023-08-30 15:25:23 -07:00
Ankita Kejriwal 2e6aed58b4
Update Storage Quota Documentation (#10839) 2023-08-30 13:40:01 -07:00