wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbclient/Schemas.cpp

1032 lines
31 KiB
C++
Raw Normal View History

Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
/*
* Schemas.cpp
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2013-2018 Apple Inc. and the FoundationDB project authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbclient/Schemas.h"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
update json schemas
2021-06-18 04:20:39 +08:00
// NOTE: also change mr-status-json-schemas.rst.inc
fixed issues from review, added a safeguard to prevent configuring a cluster to an invalid configuration
2018-09-05 13:16:35 +08:00
const KeyRef JSONSchemas::statusSchema = LiteralStringRef(R"statusSchema(
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
{
"cluster":{
"layers":{
"_valid":true,
"_error":"some error description"
},
"processes":{
"$map":{
"version":"3.0.0",
"machine_id":"0ccb4e0feddb5583010f6b77d9d10ece",
"locality":{
"$map":"value"
},
"class_source":{
"$enum":[
"command_line",
"configure_auto",
"set_class"
]
},
"class_type":{
"$enum":[
"unset",
"storage",
"transaction",
"resolution",
use the stateless process class instead of master or resolution in simulation because it is the recommended process class, and the others are not deterministic when recruited in a constrained process situation
2021-04-27 00:49:26 +08:00
"stateless",
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
"commit_proxy",
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
"grv_proxy",
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"master",
Merging with Master.
2019-11-13 05:01:29 +08:00
"test",
"storage_cache"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
]
},
added status fields detailing which processes are degraded, and also the total number of degraded processes
2019-03-11 13:58:15 +08:00
"degraded":true,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"roles":[
{
"query_queue_max":0,
Track the local ratekeeper rate as a percentage This value is reported in status for each storage server.
2019-06-16 06:02:43 +08:00
"local_rate":0,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"input_bytes":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"stored_bytes":12341234,
"kvstore_used_bytes":12341234,
"kvstore_available_bytes":12341234,
"kvstore_free_bytes":12341234,
"kvstore_total_bytes":12341234,
WIP : deployable/stable version since Nov 3. Start rebase to master branch
2019-01-10 10:03:54 +08:00
"kvstore_total_size":12341234,
"kvstore_total_nodes":12341234,
"kvstore_inline_keys":12341234,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"durable_bytes":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"queue_disk_used_bytes":12341234,
"queue_disk_available_bytes":12341234,
"queue_disk_free_bytes":12341234,
"queue_disk_total_bytes":12341234,
"role":{
"$enum":[
"master",
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
"commit_proxy",
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
"grv_proxy",
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"log",
"storage",
"resolver",
Fix SchemaMismatch error Add data_distributor and ratekeeper roles to schema.
2019-03-28 00:54:01 +08:00
"cluster_controller",
"data_distributor",
added additional logging on the logs and log routers
2019-05-03 07:16:25 +08:00
"ratekeeper",
Merging with Master.
2019-11-13 05:01:29 +08:00
"storage_cache",
Add 'coordinator' to the list of roles that a process can have in status.
2019-08-16 05:42:49 +08:00
"router",
"coordinator"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
]
},
"data_version":12341234,
Add new metrics for bytes queried, keys queried, mutation bytes, mutations, and durable lag to the storage role in status.
2018-09-28 05:33:21 +08:00
"durable_version":12341234,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"data_lag": {
"seconds":5.0,
"versions":12341234
},
Rename durable_lag to durability_lag
2018-10-02 00:58:49 +08:00
"durability_lag": {
Add new metrics for bytes queried, keys queried, mutation bytes, mutations, and durable lag to the storage role in status.
2018-09-28 05:33:21 +08:00
"seconds":5.0,
"versions":12341234
},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"id":"eb84471d68c12d1d26f692a50000003f",
Lot metrics about read requests as well as completed reads.
2018-09-28 06:32:39 +08:00
"total_queries":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"finished_queries":{
"hz":0.0,
"counter":0,
"roughness":0.0
Add new metrics for bytes queried, keys queried, mutation bytes, mutations, and durable lag to the storage role in status.
2018-09-28 05:33:21 +08:00
},
added low priority reads to status
2021-02-04 05:24:34 +08:00
"low_priority_queries":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
Add new metrics for bytes queried, keys queried, mutation bytes, mutations, and durable lag to the storage role in status.
2018-09-28 05:33:21 +08:00
"bytes_queried":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"keys_queried":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"mutation_bytes":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"mutations":{
"hz":0.0,
"counter":0,
"roughness":0.0
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
},
Expose "bounce impact" and Storage Server "version catch-up rate" metrics Changes: Schemas.cpp: Extend the JSON schema to report the new metrics that have been added. mr-status-json-schemas.rst.inc: Update the schema to reflect the changes made to the JSON schema. release-notes-700.rst: Add a note about the new metrics in "Status" section.
2021-05-20 03:54:49 +08:00
"fetched_versions":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
Address review comments
2021-05-20 06:04:43 +08:00
"fetches_from_logs":{
Expose "bounce impact" and Storage Server "version catch-up rate" metrics Changes: Schemas.cpp: Extend the JSON schema to report the new metrics that have been added. mr-status-json-schemas.rst.inc: Update the schema to reflect the changes made to the JSON schema. release-notes-700.rst: Add a note about the new metrics in "Status" section.
2021-05-20 03:54:49 +08:00
"hz":0.0,
"counter":0,
"roughness":0.0
},
Track statistics about server-side request latency on each process, to include min, max, mean, and various percentiles.
2020-07-10 07:39:15 +08:00
"grv_latency_statistics":{
Fix make build, fix GRV schema
2020-07-10 07:50:08 +08:00
"default":{
"count":0,
"min":0.0,
"max":0.0,
"median":0.0,
"mean":0.0,
"p25":0.0,
"p90":0.0,
"p95":0.0,
"p99":0.0,
"p99.9":0.0
Respond to review comments 3/23
2021-03-25 01:41:13 +08:00
},
"batch":{
Log latency metrics for batch GRV requests
2021-03-22 22:39:06 +08:00
"count":0,
"min":0.0,
"max":0.0,
"median":0.0,
"mean":0.0,
"p25":0.0,
"p90":0.0,
"p95":0.0,
"p99":0.0,
"p99.9":0.0
}
Respond to review comments 3/23
2021-03-25 01:41:13 +08:00
},
Track statistics about server-side request latency on each process, to include min, max, mean, and various percentiles.
2020-07-10 07:39:15 +08:00
"read_latency_statistics":{
"count":0,
"min":0.0,
"max":0.0,
"median":0.0,
"mean":0.0,
"p25":0.0,
"p90":0.0,
"p95":0.0,
"p99":0.0,
"p99.9":0.0
},
"commit_latency_statistics":{
"count":0,
"min":0.0,
"max":0.0,
"median":0.0,
"mean":0.0,
"p25":0.0,
"p90":0.0,
"p95":0.0,
"p99":0.0,
"p99.9":0.0
},
Expose CommitBatchingWindowSize metric to fdbcli status Changes: Schemas.cpp: - Extend JSON schema to include aggregated information about CommitBatchingWindowSize samples. Status.actor.cpp: - Extend getStorageServersAndMetrics() to gather metrics about CommitBatchingWindowSize. - Extend CommitProxy AddRole() to populate the status-JSON object with the metrics about CommitBatchingWindowSize.
2021-04-30 06:11:09 +08:00
"commit_batching_window_size":{
"count":0,
"min":0.0,
"max":0.0,
"median":0.0,
"mean":0.0,
"p25":0.0,
"p90":0.0,
"p95":0.0,
"p99":0.0,
"p99.9":0.0
},
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
"grv_latency_bands":{
Reworked the way latency counters are tracked. Report the latency bands in separate events from StorageMetrics and ProxyMetrics. Fix a problem when the latency band configuration was changed. Add correctness testing.
2019-02-08 05:39:22 +08:00
"$map": 1
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
},
"read_latency_bands":{
Reworked the way latency counters are tracked. Report the latency bands in separate events from StorageMetrics and ProxyMetrics. Fix a problem when the latency band configuration was changed. Add correctness testing.
2019-02-08 05:39:22 +08:00
"$map": 1
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
},
"commit_latency_bands":{
Reworked the way latency counters are tracked. Report the latency bands in separate events from StorageMetrics and ProxyMetrics. Fix a problem when the latency band configuration was changed. Add correctness testing.
2019-02-08 05:39:22 +08:00
"$map": 1
Add busiest read tags to storage server status
2020-05-13 06:49:40 +08:00
},
"busiest_read_tag":{
"tag": "",
"fractional_cost": 0.0,
update Schemas.cpp
2020-09-03 09:05:48 +08:00
"estimated_cost":{
"hz": 0.0
}
},
"busiest_write_tag":{
"tag": "",
"fractional_cost": 0.0,
Add busiest read tags to storage server status
2020-05-13 06:49:40 +08:00
"estimated_cost":{
"hz": 0.0
Revert "Revert "Initial RocksDB""
2020-06-16 00:45:36 +08:00
}
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}
}
],
"command_line":"-r simulation",
"memory":{
"available_bytes":0,
"limit_bytes":0,
"unused_allocated_memory":0,
"used_bytes":0
},
"messages":[
{
"time":12345.12312,
"type":"x",
"name":{
"$enum":[
"file_open_error",
"incorrect_cluster_file_contents",
Added back the changes that report trace log flush failures and fix the random crash
2020-03-13 05:34:19 +08:00
"trace_log_file_write_error",
"trace_log_could_not_create_file",
"trace_log_writer_thread_unresponsive",
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"process_error",
"io_error",
"io_timeout",
"platform_error",
"storage_server_lagging",
"(other FDB error messages)"
]
},
"raw_log_message":"<stuff/>",
"description":"abc"
}
],
"fault_domain":"0ccb4e0fdbdb5583010f6b77d9d10ece",
updated status with information about ongoing maintenance clear the maintenance zone if a different storage server is detected failed
2019-04-03 05:15:51 +08:00
"under_maintenance":true,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"excluded":false,
"address":"1.2.3.4:1234",
"disk":{
"free_bytes":3451233456234,
"reads":{
"hz":0.0,
"counter":0,
"sectors":0
},
"busy":0.0,
"writes":{
"hz":0.0,
"counter":0,
"sectors":0
},
"total_bytes":123412341234
},
"uptime_seconds":1234.2345,
"cpu":{
"usage_cores":0.0
},
"network":{
"current_connections":0,
"connections_established":{
"hz":0.0
},
"connections_closed":{
"hz":0.0
},
"connection_errors":{
"hz":0.0
},
"megabits_sent":{
"hz":0.0
},
"megabits_received":{
"hz":0.0
Add cluster.processes.id.network.tls_policy.hz to status. This allows monitoring of TLS policy failures, but one has to go scrape for TLSPolicyFailure trace events to figure out why they're happening.
2020-03-13 17:40:45 +08:00
},
Remove comma, and add schema change to documentation
2020-03-14 05:51:56 +08:00
"tls_policy_failures":{
"hz":0.0
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}
Track run loop busyness and report it in status.
2019-06-27 05:03:02 +08:00
},
"run_loop_busy":0.2
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}
},
fetch current log generation as well
2020-09-10 02:54:58 +08:00
"logs":[
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
{
fetch current log generation as well
2020-09-10 02:54:58 +08:00
"log_interfaces":[
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
{
"id":"7f8d623d0cb9966e",
"healthy":true,
"address":"1.2.3.4:1234"
}
],
fetch current log generation as well
2020-09-10 02:54:58 +08:00
"epoch":1,
"current":false,
"begin_version":23,
"end_version":112315141,
"possibly_losing_data":true,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"log_replication_factor":3,
"log_write_anti_quorum":0,
"log_fault_tolerance":2,
"remote_log_replication_factor":3,
"remote_log_fault_tolerance":2,
"satellite_log_replication_factor":3,
"satellite_log_write_anti_quorum":0,
"satellite_log_fault_tolerance":2
}
],
"fault_tolerance":{
Rename max_machine_failures_without_losing_X to max_zone_failures_without_losing_X in status.
2019-07-31 05:02:31 +08:00
"max_zone_failures_without_losing_availability":0,
"max_zone_failures_without_losing_data":0
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
},
"qos":{
"worst_queue_bytes_log_server":460,
Update status schema for correctness. Send the count of batch transactions started back to ratekeeper so that it can be logged with other ratekeeper metrics.
2019-03-01 04:00:58 +08:00
"batch_performance_limited_by":{
"reason_server_id":"7f8d623d0cb9966e",
"reason_id":0,
"name":{
"$enum":[
"workload",
"storage_server_write_queue_size",
"storage_server_write_bandwidth_mvcc",
"storage_server_readable_behind",
"log_server_mvcc_write_bandwidth",
"log_server_write_queue",
"storage_server_min_free_space",
"storage_server_min_free_space_ratio",
"log_server_min_free_space",
knobified the durability lag rate controls
2019-07-12 09:34:19 +08:00
"log_server_min_free_space_ratio",
Ratekeeper to throttle tpsLimit to 1 if it is not able to fetch storage server list for some configurable amount of time.
2019-07-18 05:47:08 +08:00
"storage_server_durability_lag",
"storage_server_list_fetch_failed"
Update status schema for correctness. Send the count of batch transactions started back to ratekeeper so that it can be logged with other ratekeeper metrics.
2019-03-01 04:00:58 +08:00
]
},
"description":"The database is not being saturated by the workload."
},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"performance_limited_by":{
"reason_server_id":"7f8d623d0cb9966e",
Lot metrics about read requests as well as completed reads.
2018-09-28 06:32:39 +08:00
"reason_id":0,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"name":{
"$enum":[
"workload",
"storage_server_write_queue_size",
"storage_server_write_bandwidth_mvcc",
"storage_server_readable_behind",
"log_server_mvcc_write_bandwidth",
"log_server_write_queue",
"storage_server_min_free_space",
"storage_server_min_free_space_ratio",
"log_server_min_free_space",
knobified the durability lag rate controls
2019-07-12 09:34:19 +08:00
"log_server_min_free_space_ratio",
Ratekeeper to throttle tpsLimit to 1 if it is not able to fetch storage server list for some configurable amount of time.
2019-07-18 05:47:08 +08:00
"storage_server_durability_lag",
"storage_server_list_fetch_failed"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
]
},
"description":"The database is not being saturated by the workload."
},
Update status schema for correctness. Send the count of batch transactions started back to ratekeeper so that it can be logged with other ratekeeper metrics.
2019-03-01 04:00:58 +08:00
"batch_transactions_per_second_limit":0,
Lot metrics about read requests as well as completed reads.
2018-09-28 06:32:39 +08:00
"transactions_per_second_limit":0,
Update status schema for correctness. Send the count of batch transactions started back to ratekeeper so that it can be logged with other ratekeeper metrics.
2019-03-01 04:00:58 +08:00
"batch_released_transactions_per_second":0,
Lot metrics about read requests as well as completed reads.
2018-09-28 06:32:39 +08:00
"released_transactions_per_second":0,
Fix typo
2020-05-13 09:38:20 +08:00
"throttled_tags":{
add throttle objects into Schemas.cpp
2020-08-25 07:30:39 +08:00
"auto" : {
"busy_read" : 0,
"busy_write" : 0,
more concise status json
2020-09-05 02:40:45 +08:00
"count" : 0,
change workload; solve some comments
2020-09-09 01:28:47 +08:00
"recommended_only": 0
Add busiest read tags to storage server status
2020-05-13 06:49:40 +08:00
},
add throttle objects into Schemas.cpp
2020-08-25 07:30:39 +08:00
"manual" : {
"count" : 0
Add busiest read tags to storage server status
2020-05-13 06:49:40 +08:00
}
},
Lot metrics about read requests as well as completed reads.
2018-09-28 06:32:39 +08:00
"limiting_queue_bytes_storage_server":0,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"worst_queue_bytes_storage_server":0,
Report lag with the usual "seconds" and "versions" fields. Rename and deprecate the qos.*version_lag_storage_server fields.
2019-08-16 04:42:39 +08:00
"limiting_data_lag_storage_server":{
"versions":0,
"seconds":0.0
},
"worst_data_lag_storage_server":{
"versions":0,
"seconds":0.0
},
"limiting_durability_lag_storage_server":{
"versions":0,
"seconds":0.0
},
"worst_durability_lag_storage_server":{
"versions":0,
"seconds":0.0
}
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
},
"incompatible_connections":[
],
Rename datacenter_version_difference to datacenter_lag and include both seconds and versions.
2019-07-06 05:36:18 +08:00
"datacenter_lag": {
"seconds" : 1.0,
"versions" : 1000000
},
Testing Storage Server implementation
2021-03-06 03:28:15 +08:00
"active_tss_count":0,
added status fields detailing which processes are degraded, and also the total number of degraded processes
2019-03-11 13:58:15 +08:00
"degraded_processes":0,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"database_available":true,
Added database_lock_state to statusSchema
2019-09-07 05:26:25 +08:00
"database_lock_state": {
"locked": true,
s/lockUID/lock_uid in status
2019-09-07 12:58:35 +08:00
"lock_uid": "00000000000000000000000000000000"
Added database_lock_state to statusSchema
2019-09-07 05:26:25 +08:00
},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"generation":2,
"latency_probe":{
"read_seconds":7,
Lot metrics about read requests as well as completed reads.
2018-09-28 06:32:39 +08:00
"immediate_priority_transaction_start_seconds":0.0,
"batch_priority_transaction_start_seconds":0.0,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"transaction_start_seconds":0.0,
"commit_seconds":0.02
},
"clients":{
"count":1,
"supported_versions":[
{
"client_version":"3.0.0",
"connected_clients":[
{
"address":"127.0.0.1:9898",
added status information on connected clients
2019-07-26 08:15:31 +08:00
"log_group":"default"
}
],
"max_protocol_clients":[
{
"address":"127.0.0.1:9898",
"log_group":"default"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}
],
"count" : 1,
added status information on connected clients
2019-07-26 08:15:31 +08:00
"max_protocol_count" : 1,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"protocol_version" : "fdb00a400050001",
"source_version" : "9430e1127b4991cbc5ab2b17f41cfffa5de07e9d"
}
]
},
Change the way cache hits and misses are tracked to avoid counting blind page writes as misses and count the results of partial page writes. Report cache hit rate in status.
2019-07-11 05:43:20 +08:00
"page_cache":{
"log_hit_rate":0.5,
"storage_hit_rate":0.5
},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"messages":[
{
"reasons":[
{
"description":"Blah."
}
],
"unreachable_processes":[
{
"address":"1.2.3.4:1234"
}
],
"name":{
"$enum":[
"unreachable_master_worker",
Status: Add schema for the new data distributor role
2019-02-23 02:05:08 +08:00
"unreachable_dataDistributor_worker",
Fix status message for ratekeeper
2019-02-23 07:04:38 +08:00
"unreachable_ratekeeper_worker",
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"unreadable_configuration",
"full_replication_timeout",
"client_issues",
"unreachable_processes",
"immediate_priority_transaction_start_probe_timeout",
"batch_priority_transaction_start_probe_timeout",
"transaction_start_probe_timeout",
"read_probe_timeout",
"commit_probe_timeout",
"storage_servers_error",
"status_incomplete",
"layer_status_incomplete",
Add the new messages to Schema.
2019-07-16 04:47:27 +08:00
"database_availability_timeout",
"consistencycheck_suspendkey_fetch_timeout",
updated status schema
2019-10-17 13:37:57 +08:00
"consistencycheck_disabled",
"duplicate_mutation_streams",
Fix the missing schema field which caused a lot noise in nightly
2020-04-07 01:18:58 +08:00
"duplicate_mutation_fetch_timeout",
"primary_dc_missing",
Consistently use timeout instead of timedout in status messages.
2020-04-07 23:43:23 +08:00
"fetch_primary_dc_timeout"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
]
},
"issues":[
{
"name":{
"$enum":[
Added back the changes that report trace log flush failures and fix the random crash
2020-03-13 05:34:19 +08:00
"incorrect_cluster_file_contents",
"trace_log_file_write_error",
"trace_log_could_not_create_file",
"trace_log_writer_thread_unresponsive"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
]
},
"description":"Cluster file contents do not match current cluster connection string. Verify cluster file is writable and has not been overwritten externally."
}
],
"description":"abc"
}
],
WIP : deployable/stable version since Nov 3. Start rebase to master branch
2019-01-10 10:03:54 +08:00
)statusSchema"
Addressed review comments. Fix the bug where issues on a worker may be wrongly cleared by subsequent GetDBinfo request.
2020-02-06 02:05:48 +08:00
R"statusSchema(
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"recovery_state":{
Resolve review comments
2020-09-26 07:58:54 +08:00
"seconds_since_last_recovered":1,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"required_resolvers":1,
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
"required_commit_proxies":1,
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
"required_grv_proxies":1,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"name":{
"$enum":[
"reading_coordinated_state",
"locking_coordinated_state",
"locking_old_transaction_servers",
"reading_transaction_system_state",
"configuration_missing",
"configuration_never_created",
"configuration_invalid",
"recruiting_transaction_servers",
"initializing_transaction_servers",
"recovery_transaction",
"writing_coordinated_state",
"accepting_commits",
"all_logs_recruited",
"storage_recovered",
"fully_recovered"
]
},
"required_logs":3,
"missing_logs":"7f8d623d0cb9966e",
update status schema
2020-03-17 02:00:51 +08:00
"active_generations":1,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"description":"Recovery complete."
},
"workload":{
"operations":{
"writes":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"reads":{
"hz":0.0,
"counter":0,
"roughness":0.0
Lot metrics about read requests as well as completed reads.
2018-09-28 06:32:39 +08:00
},
"read_requests":{
"hz":0.0,
"counter":0,
"roughness":0.0
Added additional logging related to memory errors (including in status)
2020-03-14 09:31:22 +08:00
},
added low priority reads to status
2021-02-04 05:24:34 +08:00
"low_priority_reads":{
Lot metrics about read requests as well as completed reads.
2018-09-28 06:32:39 +08:00
"hz":0.0,
"counter":0,
"roughness":0.0
Added additional logging related to memory errors (including in status)
2020-03-14 09:31:22 +08:00
},
"location_requests":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"memory_errors":{
"hz":0.0,
"counter":0,
"roughness":0.0
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}
},
"bytes":{
"written":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"read":{
"hz":0.0,
"counter":0,
"roughness":0.0
}
},
"keys":{
"read":{
"hz":0.0,
"counter":0,
"roughness":0.0
}
},
"transactions":{
"started":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
Add information about transaction starts at different priorities to status.
2019-07-12 04:54:44 +08:00
"started_immediate_priority":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"started_default_priority":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
"started_batch_priority":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"conflicted":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
Add txnRejectedForQueuedTooLong in ProxyStats
2021-02-08 06:56:34 +08:00
"rejected_for_queued_too_long":{
"hz":0.0,
"counter":0,
"roughness":0.0
},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"committed":{
"hz":0.0,
"counter":0,
"roughness":0.0
}
}
},
"cluster_controller_timestamp":1415650089,
"protocol_version":"fdb00a400050001",
"connection_string":"a:a@127.0.0.1:4000",
"full_replication":true,
updated status with information about ongoing maintenance clear the maintenance zone if a different storage server is detected failed
2019-04-03 05:15:51 +08:00
"maintenance_zone":"0ccb4e0fdbdb5583010f6b77d9d10ece",
"maintenance_seconds_remaining":1.0,
do not duplicate data distribution disabled fields in status fixed a few bugs related to the existing data distribution disabled fields in status
2019-08-30 09:41:34 +08:00
"data_distribution_disabled_for_ss_failures":true,
"data_distribution_disabled_for_rebalance":true,
"data_distribution_disabled":true,
Address review comments.
2020-04-02 06:13:04 +08:00
"active_primary_dc":"pv",
Report bounce impact info as part of cluster JSON object.
2021-05-14 00:36:57 +08:00
"bounce_impact":{
"can_clean_bounce":true,
"reason":""
},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"configuration":{
"log_anti_quorum":0,
"log_replicas":2,
"log_replication_policy":"(zoneid^3x1)",
"redundancy_mode":{
"$enum":[
"single",
"double",
"triple",
"three_datacenter",
"three_datacenter_fallback",
added three_data_hall_fallback configuration
2019-04-08 13:58:18 +08:00
"three_data_hall",
"three_data_hall_fallback"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
]},
"regions":[{
"datacenters":[{
"id":"mr",
"priority":1,
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-15 09:30:15 +08:00
"satellite":1,
"satellite_logs":2
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}],
"satellite_redundancy_mode":{
"$enum":[
"one_satellite_single",
"one_satellite_double",
"one_satellite_triple",
"two_satellite_safe",
"two_satellite_fast"
]},
"satellite_log_replicas":1,
"satellite_usable_dcs":1,
"satellite_anti_quorum":0,
"satellite_log_policy":"(zoneid^3x1)",
"satellite_logs":2
}],
"remote_redundancy_mode":{
"$enum":[
"remote_default",
"remote_single",
"remote_double",
"remote_triple",
"remote_three_data_hall"
]},
"remote_log_replicas":3,
"remote_logs":5,
"log_routers":10,
"usable_regions":1,
"repopulate_anti_quorum":1,
"storage_replicas":1,
"resolvers":1,
"storage_replication_policy":"(zoneid^3x1)",
"logs":2,
Implement log_version. This mega-commit introduces a new configuration setting, `log_version`, that controls the TLog implementations and features that are available within FDB, so that users can opt in to new features if they're willing to sacrifice backwards compatibility.
2019-02-23 04:15:23 +08:00
"log_version":2,
Set log_spill in SimulationConfig. Which also revealed that it needed to be added to the schema.
2019-02-20 14:02:44 +08:00
"log_engine":1,
"log_spill":1,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"storage_engine":{
"$enum":[
"ssd",
"ssd-1",
"ssd-2",
Renamed redwood to redwood-experimental. UnitTest names can now be hidden using # as the first character so that random correctness tests will not run them. Excluded redwood tests from correctness testing. Reverted default storage engine to ssd.
2018-10-06 05:43:54 +08:00
"ssd-redwood-experimental",
Revert "Revert "Initial RocksDB""
2020-06-16 00:45:36 +08:00
"ssd-rocksdb-experimental",
configure memory now selects the ssd engine for transaction log spilling. Transaction log spilling is only used when the transaction logs cannot keep all of the unpopped mutations it has in memory. If we are only using this data structure because we do not have enough memory, it is much less safe to use the memory storage engine for this purpose.
2019-03-17 13:48:24 +08:00
"memory",
changed the old memory configuration to memory-1
2019-03-19 06:10:04 +08:00
"memory-1",
WIP : deployable/stable version since Nov 3. Start rebase to master branch
2019-01-10 10:03:54 +08:00
"memory-2",
change memory storage engine name from memory-radixtree to memory-radixtree-beta
2020-01-10 05:58:06 +08:00
"memory-radixtree-beta"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
]},
Testing Storage Server implementation
2021-03-06 03:28:15 +08:00
"tss_count":1,
"tss_storage_engine":{
"$enum":[
"ssd",
"ssd-1",
"ssd-2",
"ssd-redwood-experimental",
"ssd-rocksdb-experimental",
"memory",
"memory-1",
"memory-2",
"memory-radixtree-beta"
]},
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"coordinators_count":1,
"excluded_servers":[
{
Status json schema update, includelocalities back for consistency check, review comments.
2021-06-17 15:27:26 +08:00
"address":"10.0.4.1",
exclude to work with any locality data match.
2021-06-24 04:55:17 +08:00
"locality":"locality_processid:e9816ca4a89ff64ddb7ba2a5ec10b75b"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}
],
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
"auto_commit_proxies":3,
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
"auto_grv_proxies":1,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"auto_resolvers":1,
"auto_logs":3,
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
"commit_proxies":5,
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
"grv_proxies":1,
Add proxies back to schema
2021-06-09 02:12:33 +08:00
"proxies":6,
fix Schema check error
2021-05-20 07:52:16 +08:00
"backup_worker_enabled":1,
"perpetual_storage_wiggle":0
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
},
"data":{
"least_operating_space_bytes_log_server":0,
"average_partition_size_bytes":0,
"state":{
"healthy":true,
"min_replicas_remaining":0,
"name":{
"$enum":[
"initializing",
"missing_data",
"healing",
Status:healthy: Add optimizing_team_collections Change removing_redundant_teams status name to optimizing_team_collections. The new name is more general and can be applied in the future when we switch storage engines.
2019-03-07 07:05:21 +08:00
"optimizing_team_collections",
updated documentation for 6.2.18
2020-03-07 03:16:10 +08:00
"healthy_populating_region",
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"healthy_repartitioning",
"healthy_removing_server",
"healthy_rebalancing",
"healthy"
]
},
"description":""
},
"least_operating_space_ratio_storage_server":0.1,
"max_machine_failures_without_losing_availability":0,
"total_disk_used_bytes":0,
"total_kv_size_bytes":0,
added tracking for the size of the system keyspace
2019-09-28 13:39:19 +08:00
"system_kv_size_bytes":0,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"partitions_count":2,
"moving_data":{
"total_written_bytes":0,
"in_flight_bytes":0,
"in_queue_bytes":0,
"highest_priority":0
},
"team_trackers":[
{
"primary":true,
"in_flight_bytes":0,
"unhealthy_servers":0,
"state":{
"healthy":true,
"min_replicas_remaining":0,
"name":{
"$enum":[
"initializing",
"missing_data",
"healing",
Status:healthy: Add optimizing_team_collections Change removing_redundant_teams status name to optimizing_team_collections. The new name is more general and can be applied in the future when we switch storage engines.
2019-03-07 07:05:21 +08:00
"optimizing_team_collections",
updated documentation for 6.2.18
2020-03-07 03:16:10 +08:00
"healthy_populating_region",
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"healthy_repartitioning",
"healthy_removing_server",
"healthy_rebalancing",
"healthy"
]
},
"description":""
}
}
],
Testing Storage Server implementation
2021-03-06 03:28:15 +08:00
"least_operating_space_bytes_storage_server":0,
"max_machine_failures_without_losing_data":0
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
},
"machines":{
"$map":{
"network":{
"megabits_sent":{
"hz":0.0
},
"megabits_received":{
"hz":0.0
},
"tcp_segments_retransmitted":{
"hz":0.0
}
},
"memory":{
"free_bytes":0,
"committed_bytes":0,
"total_bytes":0
},
"contributing_workers":4,
"datacenter_id":"6344abf1813eb05b",
"excluded":false,
"address":"1.2.3.4",
"machine_id":"6344abf1813eb05b",
"locality":{
"$map":"value"
},
"cpu":{
"logical_core_utilization":0.4
}
}
}
},
"client":{
"coordinators":{
"coordinators":[
{
"reachable":true,
switch to test newer incompatible version. Fix PR comments. Modify schema
2020-10-10 04:22:40 +08:00
"address":"127.0.0.1:4701",
change protocol version to hex encoded string in status json. Move constant from flow transport header to cpp
2020-10-27 03:35:38 +08:00
"protocol": "0fdb00b070010001"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}
],
"quorum_reachable":true
},
"database_status":{
"available":true,
"healthy":true
},
"messages":[
{
"name":{
"$enum":[
"inconsistent_cluster_file",
"unreachable_cluster_controller",
"no_cluster_controller",
"status_incomplete_client",
"status_incomplete_coordinators",
"status_incomplete_error",
"status_incomplete_timeout",
"status_incomplete_cluster",
"quorum_not_reachable"
]
},
"description":"The cluster file is not up to date."
}
],
"timestamp":1415650089,
"cluster_file":{
"path":"/etc/foundationdb/fdb.cluster",
"up_to_date":true
}
}
})statusSchema");
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
const KeyRef JSONSchemas::clusterConfigurationSchema = LiteralStringRef(R"configSchema(
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
{
"create":{
"$enum":[
"new"
]},
fixed issues from review, added a safeguard to prevent configuring a cluster to an invalid configuration
2018-09-05 13:16:35 +08:00
"log_anti_quorum":0,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"log_replicas":2,
"log_replication_policy":"(zoneid^3x1)",
"redundancy_mode":{
"$enum":[
"single",
"double",
"triple",
"three_datacenter",
"three_datacenter_fallback",
added three_data_hall_fallback configuration
2019-04-08 13:58:18 +08:00
"three_data_hall",
"three_data_hall_fallback"
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
]},
"regions":[{
fixed issues from review, added a safeguard to prevent configuring a cluster to an invalid configuration
2018-09-05 13:16:35 +08:00
"datacenters":[{
"id":"mr",
"priority":1,
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-15 09:30:15 +08:00
"satellite":1,
"satellite_logs":2
fixed issues from review, added a safeguard to prevent configuring a cluster to an invalid configuration
2018-09-05 13:16:35 +08:00
}],
"satellite_redundancy_mode":{
"$enum":[
"one_satellite_single",
"one_satellite_double",
"one_satellite_triple",
"two_satellite_safe",
"two_satellite_fast"
]},
"satellite_log_replicas":1,
"satellite_usable_dcs":1,
"satellite_anti_quorum":0,
"satellite_log_policy":"(zoneid^3x1)",
"satellite_logs":2
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
}],
"remote_redundancy_mode":{
"$enum":[
"remote_default",
"remote_single",
"remote_double",
"remote_triple",
"remote_three_data_hall"
]},
"remote_log_replicas":3,
"remote_logs":5,
"log_routers":10,
"usable_regions":1,
"repopulate_anti_quorum":1,
"storage_replicas":1,
"resolvers":1,
"storage_replication_policy":"(zoneid^3x1)",
"logs":2,
"storage_engine":{
"$enum":[
"ssd",
"ssd-1",
"ssd-2",
"memory"
]},
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
"auto_commit_proxies":3,
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
"auto_grv_proxies":1,
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
"auto_resolvers":1,
"auto_logs":3,
fileconfigure fails with Assertion false failed
2020-09-26 01:07:55 +08:00
"commit_proxies":5,
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
"grv_proxies":1
Added the ability to configure a cluster from a JSON file
2018-08-17 08:34:59 +08:00
})configSchema");
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
const KeyRef JSONSchemas::latencyBandConfigurationSchema = LiteralStringRef(R"configSchema(
{
"get_read_version":{
"bands":[
0.0
]
},
"read":{
"bands":[
0.0
],
"max_key_selector_offset":0,
"max_read_bytes":0
},
"commit":{
"bands":[
0.0
],
"max_commit_bytes":0
}
})configSchema");
Add schemas, and check dataDistributionStatsSchema
2020-06-30 08:14:42 +08:00
const KeyRef JSONSchemas::dataDistributionStatsSchema = LiteralStringRef(R"""(
{
Use snake_case to match status json convention
2020-07-01 00:58:55 +08:00
"shard_bytes": 1947000
Add schemas, and check dataDistributionStatsSchema
2020-06-30 08:14:42 +08:00
}
)""");
const KeyRef JSONSchemas::logHealthSchema = LiteralStringRef(R"""(
{
Use snake_case to match status json convention
2020-07-01 00:58:55 +08:00
"log_queue": 156
Add schemas, and check dataDistributionStatsSchema
2020-06-30 08:14:42 +08:00
}
)""");
const KeyRef JSONSchemas::storageHealthSchema = LiteralStringRef(R"""(
{
Use snake_case to match status json convention
2020-07-01 00:58:55 +08:00
"cpu_usage": 3.28629447047675,
"disk_usage": 0.19997897369207954,
"storage_durability_lag": 5050809,
"storage_queue": 2030
Add schemas, and check dataDistributionStatsSchema
2020-06-30 08:14:42 +08:00
}
)""");
const KeyRef JSONSchemas::aggregateHealthSchema = LiteralStringRef(R"""(
{
Use snake_case to match status json convention
2020-07-01 00:58:55 +08:00
"batch_limited": false,
Revert "Revert "Add limiting health metrics"" This reverts commit 209ebcc5950fe246ee0a6b995dcbe9ecc2688c68.
2020-11-14 09:24:57 +08:00
"limiting_storage_durability_lag": 5050809,
"limiting_storage_queue": 2030,
Use snake_case to match status json convention
2020-07-01 00:58:55 +08:00
"tps_limit": 457082.8105811302,
"worst_storage_durability_lag": 5050809,
"worst_storage_queue": 2030,
"worst_log_queue": 156
Add schemas, and check dataDistributionStatsSchema
2020-06-30 08:14:42 +08:00
}
)""");
Add json schema for the management api error message in special key space
2020-07-28 08:37:19 +08:00
const KeyRef JSONSchemas::managementApiErrorSchema = LiteralStringRef(R"""(
{
Change json string to snake_case
2020-07-29 02:42:03 +08:00
"retriable": false,
"command": "exclude",
"message": "The reason of the error"
Add json schema for the management api error message in special key space
2020-07-28 08:37:19 +08:00
}
)""");