2fa80e7912
Address review comments
2021-05-19 22:04:43 +00:00
3066e856c9
Expose "bounce impact" and Storage Server "version catch-up rate" metrics
...
Changes:
storageserver.actor.cpp: Use counters to capture (a) how fast a storage
server is catching up in versions and (b) the version fetch frequency.
Status.actor.cpp: Report the captured counter metrics as part of storage
metrics.
2021-05-19 16:08:32 +00:00
fcc6efd3b1
Add .cluster.configuration status json field
2021-05-18 10:47:16 -07:00
d8cad8efca
Report bounce impact info as part of cluster JSON object.
2021-05-13 16:36:57 +00:00
160293bd54
Report bounce impact in fdbcli status
...
Changes:
Schemas.cpp: Extend the JSON schema to report whether the cluster is
bounceable and if not, report the reason for why it is not bounceable.
Status.actor.cpp: Extend recoveryStateStatusFetcher() to populate the
bounce related field(s).
mr-status-json-schemas.rst.inc: Update the schema to reflect the change
made in Schemas.cpp.
release-notes-700.rst: Add a note about the new status fields in "Status"
section.
2021-05-13 14:28:06 +00:00
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
336a9bff66
Provide "time since last full recovery" in fdbcli status
...
Changes:
Schemas.cpp: Extend the JSON schema to include a new field that reports
the number of seconds since last full recovery.
Status.actor.cpp: Extend recoveryStateStatusFetcher() to populate the
new field that has been added to Schemas.cpp.
mr-status-json-schemas.rst.inc: Update the schema to reflect the change
made in Schemas.cpp.
2021-05-05 19:43:44 +00:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
f7e28c50d4
Merge pull request #4735 from sbodagala/master
...
Expose CommitBatchingWindowSize metric to fdbcli status
2021-04-30 15:52:29 -04:00
f151df3203
Expose CommitBatchingWindowSize metric to fdbcli status
...
Changes:
Schemas.cpp:
- Extend JSON schema to include aggregated information about
CommitBatchingWindowSize samples.
Status.actor.cpp:
- Extend getStorageServersAndMetrics() to gather metrics about
CommitBatchingWindowSize.
- Extend CommitProxy AddRole() to populate the status-JSON object
with the metrics about CommitBatchingWindowSize.
2021-04-29 22:11:09 +00:00
715c98572c
bit more documentation
2021-04-21 10:48:35 -04:00
a95e845200
document changes
2021-04-06 14:56:58 -04:00
7faca702d2
Fix bug in writes to json objects.
2021-04-06 13:05:09 -04:00
cabf192f57
Respond to review comments 3/23
2021-04-06 13:05:09 -04:00
48a475366c
Log latency metrics for batch GRV requests
2021-04-06 13:05:09 -04:00
c3aff4340f
Address merge conflicts
2021-03-26 11:36:02 -07:00
bff7b9dcae
Merge pull request #4454 from vishesh/task/tlog-rf-old-tlog
...
status: Ignore LogSets with no tLogs when computing FT
2021-03-22 11:18:00 -07:00
27060cf4a9
Populate min_replicas_remaining field for all regions during cluster status computation. This provides more information about the number of available replicas currently in the database.
2021-03-17 15:45:12 -07:00
d13ac84180
Update fdbserver/Status.actor.cpp
...
Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2021-03-10 10:31:24 -08:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
8f5934330a
status: Ignore LogSets with no tLogs when computing FT
...
We can have LogSets where there are no tLogs but some LogRouters. Its
the way recruiting is implemented for old LogRouters in
TagPartitionedLogSystem, where it adds an empty LogSet for missing
locality.
2021-03-09 09:50:57 -08:00
79cec09255
Apply clang-tidy's performance-inefficient-vector-operation fix
...
I ran this command in my build directory after compiling with
OPEN_FOR_IDE. It took a few small tweaks to get it to compile, which is
outside the scope of this commit.
$ python run-clang-tidy.py -j $(nproc) -checks='-*,performance-inefficient-vector-operation' -fix
2021-03-04 03:58:25 +00:00
971b1aa1e6
Add txnRejectedForQueuedTooLong in ProxyStats
...
Added transactions.rejected_for_queued_too_long for bookkeeping the
number of transactions rejected by commit proxy because its queuing
time exceeds MVCC window.
2021-02-17 21:40:46 -08:00
fb9632297e
Add txnRejectedForQueuedTooLong in ProxyStats
2021-02-12 13:04:58 -08:00
aaf0a9aa7b
Merge branch 'release-6.3' into merge-release-6.3-into-master
...
# Conflicts:
# build/docker-compose.yaml
# cmake/ConfigureCompiler.cmake
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/IAsyncFile.h
# fdbrpc/IRateControl.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbservice/ServiceBase.cpp
2021-02-08 12:58:34 -08:00
67e783acf8
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/CompileBoost.cmake
# cmake/FDBComponents.cmake
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/storageserver.actor.cpp
# flow/Knobs.h
# flow/network.h
2021-02-08 09:20:28 -08:00
164ef39254
more fixes
2021-02-03 16:58:36 -08:00
ed7ec2b728
another attempt to fix compile errors
2021-02-03 16:53:08 -08:00
89651e2461
fix compile error
2021-02-03 16:38:13 -08:00
b2ffdf47f0
added low priority reads to status
2021-02-03 13:24:34 -08:00
ff7d306b09
Merge branch 'release-6.3' into anoyes/merge-6.3-to-master
...
Include conflict markers for now. Will resolve.
2021-01-15 18:04:09 +00:00
31cc888562
Make fault_tolerance_without_losing data consistent with 6.2 in HA.
...
Don't consider satellites for now. This is a HACK which needs to be fixed
soon, but for now need this to keep the monitoring sane.
2021-01-05 13:53:48 -08:00
a1809f7d86
doc: Fault Tolerance and Region Configuration
2020-12-22 16:25:17 -08:00
c77d9e4abe
merge conflicts
2020-12-02 21:53:19 +00:00
d64cf8b9e3
Merge branch 6.3 into master
2020-11-17 11:22:45 -08:00
c50e997f60
Make status tests deterministic
...
This change seems to be incorrect since afaict INetwork::timer isn't
guaranteed to be monotonic. Maybe we can make that guarantee or add an
INetwork::timer_monotonic symbol?
2020-11-05 17:05:34 +00:00
cf4c8e375f
Merge remote-tracking branch 'origin/release-6.3' into merge
2020-10-29 22:15:41 -07:00
92a5178b4a
Merge branch 'release-6.3' into release-6-2-2020-10-23
2020-10-23 12:24:48 -07:00
055add9682
conflicts
2020-10-23 06:33:00 +00:00
6a6ea56596
Restore line that stores the data lag seconds of a storage server. This value is used to add a data lag message to status.
2020-10-20 10:12:00 -07:00
944f30484a
Merge pull request #3759 from dongxinEric/misc/3739/expose-time-since-last-recovery
...
This resolves issue #3739 by exposing time since last full recovery.
2020-10-19 09:03:31 -07:00
41843f07e6
add simulator support for different process versions and ProtocolVersion test
2020-10-12 18:19:31 +00:00
268ba0bddc
Merge remote-tracking branch 'origin/release-6.3' into merge-6.3
2020-10-01 14:14:06 -06:00
b1180f8eb4
fixed naming and comments
2020-09-30 20:35:09 -07:00
b1570c740f
extraTlogEligileZones should consider the database available both during a failover and also if the cluster cannot recruit tlogs in the remote region
2020-09-30 18:10:04 -07:00
8c729ca8e6
only add additional fault tolerance for availability if automatic failover is enabled
2020-09-30 18:04:23 -07:00
9f61039858
more fixes
2020-09-30 16:52:58 -07:00
d7454ac7da
fixed compile error
2020-09-30 16:49:36 -07:00
fe5c30e778
fault tolerance was not being properly increased when usable regions was 2 and satellites are configured.
2020-09-30 16:41:00 -07:00
480fc82779
Resolve review comments
2020-09-25 16:58:54 -07:00
a96d6f85c5
Removed redundant field number_of_old_generations_of_tlogs from status json
2020-09-24 09:44:51 -07:00
77048c3d0f
Handle possbile timeout when getting a read version. Updated documentation of the status json format
2020-09-23 13:50:59 -07:00
50f681cd32
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-09-23 10:54:49 -07:00
cf69f455a9
Merge pull request #3785 from apple/release-6.3
...
Merge Release 6.3 to master
2020-09-17 14:43:56 -07:00
4df0f60729
Instead of using fully_recovered, use accepting_commits as a singal of DB turned available. Also add the number of old generations into status
2020-09-17 09:55:25 -07:00
cc5bc16bd8
Rename more places from proxy to commit proxy
2020-09-15 22:29:49 -07:00
3c7bd3549a
Fix compile errors
2020-09-11 14:23:27 -07:00
f2f3351560
Only report if the field FullyRecoveredAtVersion exists.
2020-09-11 13:44:17 -07:00
2619e4d3df
Use version clock to mitigate network clock skew.
2020-09-11 13:39:16 -07:00
224f23b0f8
Rely on MasterRecoveryState message since we only care about the current generation.
2020-09-11 11:45:02 -07:00
35bef73a1c
Rename proxy to commit proxy
2020-09-10 17:44:15 -07:00
1867ee1f5f
Change cli output format
2020-09-09 22:34:36 -07:00
1155d015c9
fetch current log generation as well
2020-09-09 11:54:58 -07:00
62dd1f7234
Merge pull request #3696 from sfc-gh-xwang/tag-report
...
report busiest write tag of each storage server
2020-09-08 15:21:14 -07:00
2935d3d4f6
change workload; solve some comments
2020-09-08 21:47:49 +00:00
4363dd0f25
This resolves issue #3739 by exposing time since last full recovery.
2020-09-08 14:26:01 -07:00
23e1ff694c
Report missing old tlogs in recovery between accepting commits and storage recovered
2020-09-08 13:35:42 -07:00
6c3d919295
Fix status fetcher for GrvProxyStats
2020-09-08 11:11:45 -07:00
ecf2c0109c
more concise status json
2020-09-04 18:40:45 +00:00
5b5087c566
format
2020-09-04 16:34:05 +00:00
7660fb3beb
report busiest tags in status json
2020-09-04 16:33:59 +00:00
63b3612ad5
Merge master branch and resolve conflicts
2020-08-24 16:42:31 -07:00
1f134d1534
format
2020-08-21 05:06:13 +00:00
9398a78a3a
add busy-read count and busy-write count to status json
2020-08-21 04:50:56 +00:00
bc6e42c634
add status json report for recommended throttled tags
2020-08-19 19:22:16 +00:00
d6a23a4d6b
Resolve comments to make GRV proxy a separate process class
2020-08-06 00:01:57 -07:00
df6b676ccb
Fix status bug and backup minKnownCommittedVersion bug
2020-07-24 00:49:16 -07:00
ff4bae5cd3
Fix status test
2020-07-23 12:04:02 -07:00
229ab0d5f1
Fix some conflicts and remote debugging trace events
2020-07-22 23:35:46 -07:00
525f10e30c
Merge master branch
2020-07-22 16:08:49 -07:00
302cf5c45f
Remove debug trace events
2020-07-22 12:20:22 -07:00
5b06d69d25
Pass watches test
2020-07-15 00:37:41 -07:00
b09dddc07e
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/fdbrpc.vcxproj
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
8befb0829d
Merge pull request #3481 from ajbeamon/fix-dc-timeout-message
...
Add missing messages to schema and rename one to match later versions
2020-07-10 10:30:21 -07:00
b51beead53
The backport of a change in later versions didn't include some updates to the schema and a change to the name of one of the messages.
2020-07-09 16:58:13 -07:00
04d1217941
Track statistics about server-side request latency on each process, to include min, max, mean, and various percentiles.
2020-07-09 16:39:15 -07:00
e10704fd76
Cherry-pick region related status changes from 6.3
2020-06-09 14:56:21 -07:00
d128252e90
Merge release-6.3 into master
2020-05-22 09:25:32 -07:00
87350e1bf7
Merge pull request #3174 from ajbeamon/process-available-memory-balancing
...
Balance available memory based on the limits set for each process.
2020-05-20 14:20:11 -07:00
d636194d0d
Remove deprecated fields in status: worst_version_lag_storage_server and limiting_version_lag_storage_server
2020-05-19 13:12:10 -07:00
b49eb0f67a
Balance available memory based on the limits set for each process. Don't report more available memory than the limit.
2020-05-14 15:49:59 -07:00
bc0873adf0
Update tag throttle count status fields
2020-05-12 15:50:08 -07:00
e0526e0095
Add busiest read tags to storage server status
2020-05-12 15:49:40 -07:00
aed97a9f20
Merge branch 'master' into transaction-tagging
2020-05-07 14:52:22 -07:00
ff992060cd
Merge pull request #3073 from tclinken/fix-open-for-ide-build
...
Fix non-boost-related OPEN_FOR_IDE build errors
2020-05-07 14:47:59 -07:00
36454bb3b8
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbclient/NativeAPI.actor.cpp
2020-05-04 10:23:25 -07:00
943e8e7e84
More fixes for OPEN_FOR_IDE build
2020-05-01 23:04:12 -07:00
17815fb6bf
Merge pull request #3037 from ajbeamon/status-busy-use-new-field
...
Use the updated field name in status when fetching process busyness info
2020-05-01 23:02:54 -07:00
6ada5359b8
Merge branch 'master' into transaction-tagging
2020-04-29 14:27:21 -07:00
054d6bca65
Use the updated field name in status when fetching process busyness info
2020-04-27 11:38:54 -07:00
33efb9ec97
code cleanup based on review comments
2020-04-17 15:05:01 -07:00
1476057996
properly cache serialization of serverDBInfo
2020-04-11 19:30:05 -07:00
07cc0a8d74
code cleanup
2020-04-10 17:02:11 -07:00
e8d333733a
Merge branch 'master' into feature-tree-broadcast
2020-04-10 13:51:09 -07:00
ce4493f679
many bug fixes
2020-04-10 13:45:16 -07:00
36da61dd9c
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
2020-04-07 21:12:14 -07:00
2309e9f156
Consistently use timeout instead of timedout in status messages.
2020-04-07 08:43:23 -07:00
a51c92854a
Merge branch 'master' into feature-tree-broadcast
...
# Conflicts:
# fdbserver/WorkerInterface.actor.h
# fdbserver/worker.actor.cpp
2020-04-06 21:09:44 -07:00
477d66b46d
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-05 23:09:36 -07:00
2336f073ad
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-03 15:24:14 -07:00
9a451bbf1a
Address review comments
2020-04-03 10:49:40 -07:00
dfe5ae3f4c
Update fdbserver/Status.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-04-03 10:25:49 -07:00
eaae9397e5
Address review comments
2020-04-02 11:04:58 -07:00
5f710bde6a
Apply suggestions from code review
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-04-02 09:40:13 -07:00
e755583c07
Address review comments.
2020-04-01 15:13:04 -07:00
484393e879
Update fdbserver/Status.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-03-31 09:42:42 -07:00
012d41548e
Address review comments
2020-03-30 13:55:59 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
ed4d02a3e4
Merge pull request #2812 from etschannen/feature-proxy-mem-limit
...
Limit the amount of requests the proxy can queue up in memory
2020-03-16 14:56:56 -07:00
fe19f30999
Merge pull request #2813 from etschannen/feature-satellite-usable-regions
...
do not recruit satellite tlogs when usable regions=1
2020-03-16 11:54:42 -07:00
f2defc3a3a
Merge pull request #2814 from etschannen/feature-delay-recovery
...
Prevent coordinated state from filling up with too many old generations
2020-03-16 11:45:17 -07:00
e5d53c863b
report in status the number of active generations
2020-03-16 10:29:17 -07:00
04b752b40a
Added additional logging related to memory errors (including in status)
2020-03-13 18:31:22 -07:00
4640edf5d6
do not recruit satellite tlogs when usable regions=1
2020-03-13 10:24:52 -07:00
d86a601b84
Add cluster.processes.id.network.tls_policy.hz to status.
...
This allows monitoring of TLS policy failures, but one has to go scrape
for TLSPolicyFailure trace events to figure out why they're happening.
2020-03-13 02:46:10 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
6296465e07
Make the DD priority associated with populating a remote region lower than machine failures
2020-03-04 14:07:32 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
78cb1071dc
Status should use the full list of proxies
2020-02-07 15:44:02 -08:00
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
f597aa7e18
WIP : deployable/stable version since Nov 3. Start rebase to master branch
2020-01-15 13:49:45 -08:00
3325980c03
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/DataDistribution.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/WorkerInterface.actor.h
# fdbserver/worker.actor.cpp
# versions.target
2019-10-24 17:38:15 -07:00
35ac0071a8
fixed a compiler error
2019-10-22 17:06:54 -07:00
2d74288d16
Added a comment to clarify why cleanup work is done in status
2019-10-22 16:33:44 -07:00
3478652d06
Apply suggestions from code review
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-10-22 13:32:09 -07:00
d5c2147c0c
Update fdbserver/Status.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-10-22 13:27:52 -07:00
2caad04d9c
Keys in the destUIDLookupPrefix can be cleaned up automatically if they do not have an associated entry in the logRangesRange keyspace
2019-10-22 11:58:40 -07:00
688940b685
merge 6.2 into master
2019-10-21 11:43:46 -07:00
42b7acf7b7
Merge pull request #2202 from etschannen/feature-share-mutations
...
Backup and DR would not share mutations if started on different versions of FDB
2019-10-16 20:28:39 -07:00
587cbefe7f
duplicate mutation stream checker did not have a timeout
...
duplicate mutation stream did not work properly when multiple ranges exist with the same begin key
2019-10-16 20:17:09 -07:00
5be773f145
Update fdbserver/Status.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-10-16 16:35:24 -07:00
2facfc090b
Update fdbserver/Status.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-10-16 16:35:12 -07:00
552eb44bf8
Merge pull request #2230 from ajbeamon/fix-fault-tolerance-reporting-with-remote-regions
...
Fix: status would fail to account for remote regions when...
2019-10-16 14:51:48 -07:00
8b09cd16b2
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-share-mutations
2019-10-16 14:50:37 -07:00
86bcb84b45
Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards
2019-10-11 17:50:43 -07:00
a6da9d3df5
Fix: status would fail to account for remote regions when computing fault tolerance in the presence of a failure on the primary.
2019-10-10 10:36:35 -07:00
628b4e0220
added a warning if multiple log ranges exist for the same range
2019-10-02 17:06:19 -07:00
d0147e5e5d
Merge branch 'release-6.2' into mengxu/merge-release620-to-master-v3
...
Resolved Conflicts:
documentation/sphinx/source/release-notes.rst
fdbserver/DataDistribution.actor.cpp
versions.target
2019-10-02 13:22:56 -07:00
045175bd0e
added tracking for the size of the system keyspace
2019-09-27 22:39:19 -07:00
b495cc697b
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-09-13 09:25:08 -07:00
Sreenath Bodagala