wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbclient/Knobs.h

241 lines
7.7 KiB
C
Raw Normal View History

Initial repository commit
2017-05-26 04:48:44 +08:00
/*
* Knobs.h
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2013-2018 Apple Inc. and the FoundationDB project authors
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* http://www.apache.org/licenses/LICENSE-2.0
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef FDBCLIENT_KNOBS_H
#define FDBCLIENT_KNOBS_H
#pragma once
#include "flow/Knobs.h"
#include "flow/flow.h"
class ClientKnobs : public Knobs {
public:
int TOO_MANY; // FIXME: this should really be split up so we can control these more specifically
double SYSTEM_MONITOR_INTERVAL;
address pr comments
2021-03-18 02:16:18 +08:00
double NETWORK_BUSYNESS_MONITOR_INTERVAL; // The interval in which we should update the network busyness metric
Initial repository commit
2017-05-26 04:48:44 +08:00
double FAILURE_MAX_DELAY;
double FAILURE_MIN_DELAY;
double FAILURE_TIMEOUT_DELAY;
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-02 05:04:32 +08:00
double CLIENT_FAILURE_TIMEOUT_DELAY;
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
double FAILURE_EMERGENCY_DELAY;
double FAILURE_MAX_GENERATIONS;
delay recoveries after 70 outstanding generations, and stop recoveries after 100 outstanding generations to prevent a death spiral from filling up the coordinated state
2020-03-14 01:28:32 +08:00
double RECOVERY_DELAY_START_GENERATION;
double RECOVERY_DELAY_SECONDS_PER_GENERATION;
double MAX_GENERATIONS;
double MAX_GENERATIONS_OVERRIDE;
Fix: simulation could still stall writes for 10 seconds even when speedUpSimulation was on Fix: disable connection failures in simulation when there are too many generations outstanding
2021-04-29 03:41:48 +08:00
double MAX_GENERATIONS_SIM;
Initial repository commit
2017-05-26 04:48:44 +08:00
removed verbose trace messages
2019-07-25 06:07:41 +08:00
double COORDINATOR_RECONNECTION_DELAY;
added status information on connected clients
2019-07-26 08:15:31 +08:00
int CLIENT_EXAMPLE_AMOUNT;
double MAX_CLIENT_STATUS_AGE;
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
int MAX_COMMIT_PROXY_CONNECTIONS;
Pass watches test
2020-07-15 15:37:41 +08:00
int MAX_GRV_PROXY_CONNECTIONS;
Clients which fetch status should not disconnect from the coordinators and cluster controller between each retrieval
2020-01-23 07:41:22 +08:00
double STATUS_IDLE_TIMEOUT;
removed verbose trace messages
2019-07-25 06:07:41 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
// wrong_shard_server sometimes comes from the only nonfailed server, so we need to avoid a fast spin
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
double WRONG_SHARD_SERVER_DELAY; // SOMEDAY: This delay can limit performance of retrieving data when the cache is
// mostly wrong (e.g. dumping the database after a test)
Initial repository commit
2017-05-26 04:48:44 +08:00
double FUTURE_VERSION_RETRY_DELAY;
int REPLY_BYTE_LIMIT;
double DEFAULT_BACKOFF;
double DEFAULT_MAX_BACKOFF;
double BACKOFF_GROWTH_RATE;
Proxy going OOM when applying high volumes of writes to a proxy, particular in a sudden fashion before ratekeeper can control the workload. Address this issue by proactively monitoring the memory used by commit batches and dropping requests if a certain memory limit is exceeded.
2018-06-02 06:21:40 +08:00
double RESOURCE_CONSTRAINED_MAX_BACKOFF;
account for the overhead of tags on a message when batching transactions into a version increased the default location cache size
2018-11-03 03:52:34 +08:00
int PROXY_COMMIT_OVERHEAD_BYTES;
use median shard size"
2020-08-05 11:57:25 +08:00
double SHARD_STAT_SMOOTH_AMOUNT;
int INIT_MID_SHARD_BYTES;
Initial repository commit
2017-05-26 04:48:44 +08:00
Add transaction size option
2019-06-19 22:40:54 +08:00
int TRANSACTION_SIZE_LIMIT;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t KEY_SIZE_LIMIT;
int64_t SYSTEM_KEY_SIZE_LIMIT;
int64_t VALUE_SIZE_LIMIT;
int64_t SPLIT_KEY_SIZE_LIMIT;
implemented the \xff/metadataVersion key, which can be used by layers to help them cheaply cache metadata and know when their cache is invalid
2019-03-01 09:45:00 +08:00
int METADATA_VERSION_CACHE_SIZE;
Initial repository commit
2017-05-26 04:48:44 +08:00
int MAX_BATCH_SIZE;
double GRV_BATCH_TIMEOUT;
fixed a performance regression related to broadcasting a read version to too many transactions simultaneously
2019-03-23 07:05:20 +08:00
int BROADCAST_BATCH_SIZE;
Limit length of delays in timeout, repeating them as necessary.
2019-11-14 04:49:07 +08:00
double TRANSACTION_TIMEOUT_DELAY_INTERVAL;
Initial repository commit
2017-05-26 04:48:44 +08:00
// When locationCache in DatabaseContext gets to be this size, items will be evicted
int LOCATION_CACHE_EVICTION_SIZE;
int LOCATION_CACHE_EVICTION_SIZE_SIM;
int GET_RANGE_SHARD_LIMIT;
int WARM_RANGE_SHARD_LIMIT;
int STORAGE_METRICS_SHARD_LIMIT;
Do not merge more than 100 shards together to avoid creating untrackable shards
2020-01-16 01:33:27 +08:00
int SHARD_COUNT_LIMIT;
Initial repository commit
2017-05-26 04:48:44 +08:00
double STORAGE_METRICS_UNFAIR_SPLIT_LIMIT;
double STORAGE_METRICS_TOO_MANY_SHARDS_DELAY;
Client lazily fetches health metrics from proxies
2019-03-05 06:16:39 +08:00
double AGGREGATE_HEALTH_METRICS_MAX_STALENESS;
double DETAILED_HEALTH_METRICS_MAX_STALENESS;
use median shard size"
2020-08-05 11:57:25 +08:00
double MID_SHARD_SIZE_MAX_STALENESS;
Merge release-6.3 into master
2020-05-23 00:25:32 +08:00
bool TAG_ENCODE_KEY_SERVERS;
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// KeyRangeMap
Initial repository commit
2017-05-26 04:48:44 +08:00
int KRM_GET_RANGE_LIMIT;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
int KRM_GET_RANGE_LIMIT_BYTES; // This must be sufficiently larger than KEY_SIZE_LIMIT to ensure that at least two
// entries will be returned from an attempt to read a key range map
Initial repository commit
2017-05-26 04:48:44 +08:00
int DEFAULT_MAX_OUTSTANDING_WATCHES;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
int ABSOLUTE_MAX_WATCHES; // The client cannot set the max outstanding watches higher than this
Initial repository commit
2017-05-26 04:48:44 +08:00
double WATCH_POLLING_TIME;
made a large number of fixes to make fearless DR correctness clean.
2017-10-20 06:36:32 +08:00
double NO_RECENT_UPDATES_DURATION;
double FAST_WATCH_TIMEOUT;
double WATCH_TIMEOUT;
Initial repository commit
2017-05-26 04:48:44 +08:00
double IS_ACCEPTABLE_DELAY;
// Core
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
int64_t CORE_VERSIONSPERSECOND; // This is defined within the server but used for knobs based on server value
Initial repository commit
2017-05-26 04:48:44 +08:00
int LOG_RANGE_BLOCK_SIZE;
int MUTATION_BLOCK_SIZE;
// Taskbucket
added logging to TaskBucket
2019-11-13 11:15:56 +08:00
double TASKBUCKET_LOGGING_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
int TASKBUCKET_MAX_PRIORITY;
double TASKBUCKET_CHECK_TIMEOUT_CHANCE;
double TASKBUCKET_TIMEOUT_JITTER_OFFSET;
double TASKBUCKET_TIMEOUT_JITTER_RANGE;
double TASKBUCKET_CHECK_ACTIVE_DELAY;
int TASKBUCKET_CHECK_ACTIVE_AMOUNT;
int TASKBUCKET_TIMEOUT_VERSIONS;
int TASKBUCKET_MAX_TASK_KEYS;
// Backup
Add write buffering to BackupContainerLocalDirectory::BackupFile to greatly reduce the number of IAsyncFile::write() calls made when writing backup data. The buffer size is controlled by a knob.
2021-01-09 23:03:47 +08:00
int BACKUP_LOCAL_FILE_WRITE_BLOCK;
If backup expiration determines that force is required but the force parameter is not set, it will no longer throw an error unless the backup contains data from prior to the expire_before_version.
2018-03-09 03:27:15 +08:00
int BACKUP_CONCURRENT_DELETES;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BACKUP_SIMULATED_LIMIT_BYTES;
int BACKUP_GET_RANGE_LIMIT_BYTES;
int BACKUP_LOCK_BYTES;
double BACKUP_RANGE_TIMEOUT;
double BACKUP_RANGE_MINWAIT;
Added SnapshotDispatch task for writing snapshots in random order over a specified period of time and adapting speed to a growing or shrinking database. TaskBucket now supports scheduling tasks. TaskFuture now correctly recognizes multiple tasks in its callback space. TaskBucket extendTimeout() now supports specifying the new timeout version. Submitting a backup now requires a snapshot duration.
2017-12-14 17:44:38 +08:00
int BACKUP_SNAPSHOT_DISPATCH_INTERVAL_SEC;
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 16:49:08 +08:00
int BACKUP_DEFAULT_SNAPSHOT_INTERVAL_SEC;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BACKUP_SHARD_TASK_LIMIT;
double BACKUP_AGGREGATE_POLL_RATE;
double BACKUP_AGGREGATE_POLL_RATE_UPDATE_INTERVAL;
int BACKUP_LOG_WRITE_BATCH_MAX_SIZE;
int BACKUP_LOG_ATOMIC_OPS_SIZE;
int BACKUP_MAX_LOG_RANGES;
int BACKUP_SIM_COPY_LOG_RANGES;
int BACKUP_OPERATION_COST_OVERHEAD;
int BACKUP_VERSION_DELAY;
int BACKUP_MAP_KEY_LOWER_LIMIT;
int BACKUP_MAP_KEY_UPPER_LIMIT;
int BACKUP_COPY_TASKS;
int BACKUP_BLOCK_SIZE;
Increased CopyLogRange performance
2020-06-16 18:59:47 +08:00
int COPY_LOG_BLOCK_SIZE;
int COPY_LOG_BLOCKS_PER_TASK;
int COPY_LOG_PREFETCH_BLOCKS;
Speed up sync between Primary and DR
2020-07-03 16:36:20 +08:00
int COPY_LOG_READ_AHEAD_BYTES;
Increased CopyLogRange performance
2020-06-16 18:59:47 +08:00
double COPY_LOG_TASK_DURATION_NANOS;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BACKUP_TASKS_PER_AGENT;
Really fix the stuck of backup
2020-03-28 08:12:50 +08:00
int BACKUP_POLL_PROGRESS_SECONDS;
int64_t VERSIONS_PER_SECOND; // Copy of SERVER_KNOBS, as we can't link with it
Initial repository commit
2017-05-26 04:48:44 +08:00
int SIM_BACKUP_TASKS_PER_AGENT;
int BACKUP_RANGEFILE_BLOCK_SIZE;
int BACKUP_LOGFILE_BLOCK_SIZE;
Added new knob for how many tasks to add per transaction in backup dispatch, instead of using the value for restore which has much lower overhead per task.
2018-01-17 02:45:21 +08:00
int BACKUP_DISPATCH_ADDTASK_SIZE;
Initial repository commit
2017-05-26 04:48:44 +08:00
int RESTORE_DISPATCH_ADDTASK_SIZE;
int RESTORE_DISPATCH_BATCH_SIZE;
int RESTORE_WRITE_TX_SIZE;
int APPLY_MAX_LOCK_BYTES;
int APPLY_MIN_LOCK_BYTES;
int APPLY_BLOCK_SIZE;
double APPLY_MAX_DECAY_RATE;
double APPLY_MAX_INCREASE_FACTOR;
double BACKUP_ERROR_DELAY;
double BACKUP_STATUS_DELAY;
double BACKUP_STATUS_JITTER;
optimized log range clearing to clear everything for each possible hash (256 clears) if that would be more efficient than one clear per second that has elapsed aborting a DR without the —cleanup flag will still attempt to cleanup for 30 seconds before giving up added a cleanup command to fdbbackup which can remove mutations from orphaned DRs which were stopped without the —cleanup flag
2019-09-28 09:32:27 +08:00
double MIN_CLEANUP_SECONDS;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Configuration
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
int32_t DEFAULT_AUTO_COMMIT_PROXIES;
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
int32_t DEFAULT_AUTO_GRV_PROXIES;
Change description of GRV proxy and Commit proxy in docs and add backward compatibility support for configuring proxies in fdbcli
2020-09-26 01:49:56 +08:00
int32_t DEFAULT_COMMIT_GRV_PROXIES_RATIO;
int32_t DEFAULT_MAX_GRV_PROXIES;
Initial repository commit
2017-05-26 04:48:44 +08:00
int32_t DEFAULT_AUTO_RESOLVERS;
int32_t DEFAULT_AUTO_LOGS;
// Client Status Info
double CSI_SAMPLING_PROBABILITY;
int64_t CSI_SIZE_LIMIT;
double CSI_STATUS_DELAY;
int HTTP_SEND_SIZE;
int HTTP_READ_SIZE;
int HTTP_VERBOSE_LEVEL;
Added optional support for sending a unique id per request in a header for logging/tracking purposes.
2019-01-09 06:48:47 +08:00
std::string HTTP_REQUEST_ID_HEADER;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BLOBSTORE_CONNECT_TRIES;
int BLOBSTORE_CONNECT_TIMEOUT;
Added DNS resolution. Interface is INetworkConnections::resolveTCPEndpoint() to resolve, or for convenience INetworkConnections::connect(host, service) will resolve host and service (port number or service name like http) and connect to one of the addresses at random. BlobStoreEndpoint now only accepts hostnames and an optional service, so this update is not compatible with the previous URL formats having many IP addresses.
2017-10-16 12:51:11 +08:00
int BLOBSTORE_MAX_CONNECTION_LIFE;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BLOBSTORE_REQUEST_TRIES;
Renamed knob and URL parameter for request timeout to request timeout min to more accurately reflect what they do.
2020-07-20 08:30:58 +08:00
int BLOBSTORE_REQUEST_TIMEOUT_MIN;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BLOBSTORE_REQUESTS_PER_SECOND;
Added operation-specific rate controls to blob store interface.
2018-06-21 11:34:34 +08:00
int BLOBSTORE_LIST_REQUESTS_PER_SECOND;
int BLOBSTORE_WRITE_REQUESTS_PER_SECOND;
int BLOBSTORE_READ_REQUESTS_PER_SECOND;
int BLOBSTORE_DELETE_REQUESTS_PER_SECOND;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BLOBSTORE_CONCURRENT_REQUESTS;
int BLOBSTORE_MULTIPART_MAX_PART_SIZE;
int BLOBSTORE_MULTIPART_MIN_PART_SIZE;
int BLOBSTORE_CONCURRENT_UPLOADS;
Bug fixes. Fdbbackup delete was broken. Blobstore backup container deletion would do too much listing before deletions began due to list operations queueing up ahead of and starving the delete operations. Created new knob and blob endpoint limit for concurrent list operations to fix this. Increased blob request timeout default because some requests were taking longer. Crash fixes in blobstore doRequest() which wasn't checking that response object is valid before using it in error conditions. Filesystem-like backup container class (covering blobstore and local dirs) now ignores unrecognized filenames for describe() and expire() operations.
2018-01-06 15:06:39 +08:00
int BLOBSTORE_CONCURRENT_LISTS;
Improved error handling and memory usage in AsyncFileBlobStoreWrite. Writes will now fail if any upload has already failed, rather than buffering unboundedly until sync() is called to complete the file. There is also a configurable limit on how many uploads can be pending before writes will stall waiting for one to finish.
2017-10-18 20:51:30 +08:00
int BLOBSTORE_CONCURRENT_WRITES_PER_FILE;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BLOBSTORE_CONCURRENT_READS_PER_FILE;
int BLOBSTORE_READ_BLOCK_SIZE;
int BLOBSTORE_READ_AHEAD_BLOCKS;
int BLOBSTORE_READ_CACHE_BLOCKS_PER_FILE;
int BLOBSTORE_MAX_SEND_BYTES_PER_SECOND;
int BLOBSTORE_MAX_RECV_BYTES_PER_SECOND;
Dynamically rate limit consistency check.
2019-02-08 08:08:39 +08:00
int CONSISTENCY_CHECK_RATE_LIMIT_MAX;
int CONSISTENCY_CHECK_ONE_ROUND_TARGET_COMPLETION_TIME;
re-added the connect lock in the fdbcli so that the timeout is not spent before a connection has been initiated (because of the handshake lock)
2020-02-21 02:43:34 +08:00
DD:Add trace for detailed relocate shard info
2020-03-01 05:45:00 +08:00
// fdbcli
re-added the connect lock in the fdbcli so that the timeout is not spent before a connection has been initiated (because of the handshake lock)
2020-02-21 02:43:34 +08:00
int CLI_CONNECT_PARALLELISM;
double CLI_CONNECT_TIMEOUT;
DD:Add trace for detailed relocate shard info
2020-03-01 05:45:00 +08:00
Now limit the file identifier to a). has a length no longer than the value defined in knobs; b). can only contain [A-Za-z0-9_] chars.
2020-04-01 07:42:00 +08:00
// trace
int TRACE_LOG_FILE_IDENTIFIER_MAX_LENGTH;
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-10 07:55:56 +08:00
// transaction tags
int MAX_TRANSACTION_TAG_LENGTH;
int MAX_TAGS_PER_TRANSACTION;
sample on cost instead of bytes
2020-08-11 06:29:59 +08:00
int COMMIT_SAMPLE_COST; // The expectation of sampling is every COMMIT_SAMPLE_COST sample once
Knob adjustment
2020-08-18 04:01:19 +08:00
int WRITE_COST_BYTE_FACTOR;
add INCOMPLETE_SHARD_PLUS knob
2020-08-04 11:18:05 +08:00
int INCOMPLETE_SHARD_PLUS; // The size of (possible) incomplete shard when estimate clear range
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-10 07:55:56 +08:00
double READ_TAG_SAMPLE_RATE; // Communicated to clients from cluster
Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients. A few refactorings.
2020-04-17 05:43:22 +08:00
double TAG_THROTTLE_SMOOTHING_WINDOW;
Throttled tags on the client periodically check in with the cluster to update their rates if they get heavily throttled.
2020-04-22 02:56:58 +08:00
double TAG_THROTTLE_RECHECK_INTERVAL;
Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters.
2020-04-24 11:50:40 +08:00
double TAG_THROTTLE_EXPIRATION_INTERVAL;
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-10 07:55:56 +08:00
Atomically update dependent knobs
2020-04-02 04:59:06 +08:00
ClientKnobs();
void initialize(bool randomize = false);
Initial repository commit
2017-05-26 04:48:44 +08:00
};
Simplified global knobs
2020-11-22 04:55:03 +08:00
extern std::unique_ptr<ClientKnobs> globalClientKnobs;
Initial repository commit
2017-05-26 04:48:44 +08:00
extern ClientKnobs const* CLIENT_KNOBS;
#endif