wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbserver/Knobs.h

593 lines
22 KiB
C
Raw Normal View History

Initial repository commit
2017-05-26 04:48:44 +08:00
/*
* Knobs.h
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2013-2018 Apple Inc. and the FoundationDB project authors
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* http://www.apache.org/licenses/LICENSE-2.0
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef FDBSERVER_KNOBS_H
#define FDBSERVER_KNOBS_H
#pragma once
#include "flow/Knobs.h"
#include "fdbrpc/fdbrpc.h"
#include "fdbclient/Knobs.h"
// Disk queue
static const int _PAGE_SIZE = 4096;
class ServerKnobs : public Knobs {
public:
added a buffered cursor, which efficiently merges lots of peek cursors
2018-07-13 03:09:48 +08:00
// Versions
int64_t VERSIONS_PER_SECOND;
int64_t MAX_VERSIONS_IN_FLIGHT;
int64_t MAX_VERSIONS_IN_FLIGHT_FORCED;
int64_t MAX_READ_TRANSACTION_LIFE_VERSIONS;
int64_t MAX_WRITE_TRANSACTION_LIFE_VERSIONS;
double MAX_COMMIT_BATCH_INTERVAL; // Each master proxy generates a CommitTransactionBatchRequest at least this often, so that versions always advance smoothly
Initial repository commit
2017-05-26 04:48:44 +08:00
// TLogs
double TLOG_TIMEOUT; // tlog OR master proxy failure - master's reaction time
double RECOVERY_TLOG_SMART_QUORUM_DELAY; // smaller might be better for bug amplification
double TLOG_STORAGE_MIN_UPDATE_INTERVAL;
double BUGGIFY_TLOG_STORAGE_MIN_UPDATE_INTERVAL;
double UNFLUSHED_DATA_RATIO;
int DESIRED_TOTAL_BYTES;
The storage server would block the main thread when processing a single version with a large amount of data
2018-10-19 04:37:31 +08:00
int DESIRED_UPDATE_BYTES;
double UPDATE_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
int MAXIMUM_PEEK_BYTES;
int APPLY_MUTATION_BYTES;
int RECOVERY_DATA_BYTE_LIMIT;
int BUGGIFY_RECOVERY_DATA_LIMIT;
double LONG_TLOG_COMMIT_TIME;
int64_t LARGE_TLOG_COMMIT_BYTES;
double BUGGIFY_RECOVER_MEMORY_LIMIT;
double BUGGIFY_WORKER_REMOVED_MAX_LAG;
int64_t UPDATE_STORAGE_BYTE_LIMIT;
Batch more when spilling data.
2019-03-04 04:57:35 +08:00
int64_t REFERENCE_SPILL_UPDATE_STORAGE_BYTE_LIMIT;
Initial repository commit
2017-05-26 04:48:44 +08:00
double TLOG_PEEK_DELAY;
int LEGACY_TLOG_UPGRADE_ENTRIES_PER_VERSION;
int VERSION_MESSAGES_OVERHEAD_FACTOR_1024THS; // Multiplicative factor to bound total space used to store a version message (measured in 1/1024ths, e.g. a value of 2048 yields a factor of 2).
The tlog uses the tags embedded in the message instead of a separate vector of locations optimized remote tlog committing to avoid re-serializing the message
2018-03-17 07:47:05 +08:00
int64_t VERSION_MESSAGES_ENTRY_BYTES_WITH_OVERHEAD;
Initial repository commit
2017-05-26 04:48:44 +08:00
double TLOG_MESSAGE_BLOCK_OVERHEAD_FACTOR;
int64_t TLOG_MESSAGE_BLOCK_BYTES;
int64_t MAX_MESSAGE_SIZE;
int LOG_SYSTEM_PUSHED_DATA_BLOCK_SIZE;
double PEEK_TRACKER_EXPIRATION_TIME;
int PARALLEL_GET_MORE_REQUESTS;
multi cursors start a get more for the first 10 cursors to hide latency
2018-10-04 04:57:45 +08:00
int MULTI_CURSOR_PRE_FETCH_LIMIT;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t MAX_QUEUE_COMMIT_BYTES;
Knobified a few variables
2019-11-05 12:21:38 +08:00
int DESIRED_OUTSTANDING_MESSAGES;
double DESIRED_GET_MORE_DELAY;
tlogs serve reads to log routers at a low priority, to prevent them from using all their resources catching up a remote dc that has been down for a long time increase the amount of memory ratekeeper budgets for tlogs so that there is a gap after the spill threshold to prevent temporarily overshooting the budget
2018-08-05 01:31:30 +08:00
int CONCURRENT_LOG_ROUTER_READS;
Log Routers will prefer to peek from satellite logs. Formerly, they would prefer to peek from the primary's logs. Testing of a failed region rejoining the cluster revealed that this becomes quite a strain on the primary logs when extremely large volumes of peek requests are coming from the Log Routers. It happens that we have satellites that contain the same mutations with Log Router tags, that have no other peeking load, so we can prefer to use the satellite to peek rather than the primary to distribute load across TLogs better. Unfortunately, this revealed a latent bug in how tagged mutations in the KnownCommittedVersion->RecoveryVersion gap were copied across generations when the number of log router tags were decreased. Satellite TLogs would be assigned log router tags using the team-building based logic in getPushLocations(), whereas TLogs would internally re-index tags according to tag.id%logRouterTags. This mismatch would mean that we could have: Log0 -2:0 ----- -2:0 Log 0 Log1 -2:1 \ >--- -2:1,-2:0 (-2:2 mod 2 becomes -2:0) Log 1 Log2 -2:2 / And now we have data that's tagged as -2:0 on a TLog that's not the preferred location for -2:0, and therefore a BestLocationOnly cursor would miss the mutations. This was never noticed before, as we never used a satellite as a preferred location to peek from. Merge cursors always peek from all locations, and thus a peek for -2:0 that needed data from the satellites would have gone to both TLogs and merged the results. We now take this mod-based re-indexing into account when assigning which TLogs need to recover which tags from the previous generation, to make sure that tag.id%logRouterTags always results in the assigned TLog being the preferred location. Unfortunately, previously existing will potentially have existing satellites with log router tags indexed incorrectly, so this transition needs to be gated on a `log_version` transition. Old LogSets will have an old LogVersion, and we won't prefer the sattelite for peeking. Log Sets post-6.2 (opt-in) or post-6.3 (default) will be indexed correctly, and therefore we can safely offload peeking onto the satellites.
2019-07-09 13:25:01 +08:00
int LOG_ROUTER_PEEK_FROM_SATELLITES_PREFERRED; // 0==peek from primary, non-zero==peek from satellites
fix: if the disk queue adapter peek hangs for a while, switch to a peek from a different locality
2018-10-04 04:58:55 +08:00
double DISK_QUEUE_ADAPTER_MIN_SWITCH_TIME;
double DISK_QUEUE_ADAPTER_MAX_SWITCH_TIME;
Implement hard limiting of memory used to serve peek requests.
2019-03-04 04:57:38 +08:00
int64_t TLOG_SPILL_REFERENCE_MAX_PEEK_MEMORY_BYTES;
Add a knob to limit amount of data read from sqlite for one PeekRequest. This prevents peeking from degrading over time if there are a very large number of SpilledData entries for one particular tag.
2019-05-02 08:40:31 +08:00
int64_t TLOG_SPILL_REFERENCE_MAX_BATCHES_PER_PEEK;
Add a knob to limit the number of commits indexed per key. Theoretically, we could spill 20MB of 22B mutations for one key, which would generate a very long value being stored in SQLite, and very inefficiently read back. This stops that from being a problem, at the cost of some extra write calls.
2019-05-04 03:48:28 +08:00
int64_t TLOG_SPILL_REFERENCE_MAX_BYTES_PER_BATCH;
Allow the disk queue to shrink if it has unneeded slack space.
2019-03-04 04:57:44 +08:00
int64_t DISK_QUEUE_FILE_EXTENSION_BYTES; // When we grow the disk queue, by how many bytes should it grow?
int64_t DISK_QUEUE_FILE_SHRINK_BYTES; // When we shrink the disk queue, by how many bytes should it shrink?
Rename DISK_QUEUE_MAX_TRUNCATE_EXTENTS -> ..._BYTES So as to not make filesystem assumptions. This knob did technically appear in (only the) 6.1.5 release, but this feature was broken 6.1.5, so thus impossible to use anyway.
2019-05-11 12:26:22 +08:00
int DISK_QUEUE_MAX_TRUNCATE_BYTES; // A truncate larger than this will cause the file to be replaced instead.
when a tlog queue commit takes longer than 5 seconds, its process is marked as degraded
2019-03-09 00:46:34 +08:00
double TLOG_DEGRADED_DURATION;
Merging with Master.
2019-11-13 05:01:29 +08:00
int64_t MAX_CACHE_VERSIONS;
fix: after a forced recovery it is possible to not have logs from all generations, so only wait at most a second for getting a popped txs version
2019-08-07 07:32:28 +08:00
double TXS_POPPED_MAX_DELAY;
treat a tlog which takes a long time to create its disk queue as failed
2020-03-14 01:31:59 +08:00
double TLOG_MAX_CREATE_DURATION;
added logging for parallel peeks on TLogs
2020-04-23 05:24:45 +08:00
int PEEK_LOGGING_AMOUNT;
double PEEK_LOGGING_DELAY;
reset WAN network connections every 5 minutes is responses take more than 500ms
2020-07-10 13:50:47 +08:00
double PEEK_RESET_INTERVAL;
double PEEK_MAX_LATENCY;
bool PEEK_COUNT_SMALL_MESSAGES;
double PEEK_STATS_INTERVAL;
double PEEK_STATS_SLOW_AMOUNT;
double PEEK_STATS_SLOW_RATIO;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Data distribution queue
double HEALTH_POLL_TIME;
double BEST_TEAM_STUCK_DELAY;
Address review comments. 50K correctness with no failures.
2019-07-31 11:20:02 +08:00
double BG_REBALANCE_POLLING_INTERVAL;
double BG_REBALANCE_SWITCH_CHECK_INTERVAL;
Initial repository commit
2017-05-26 04:48:44 +08:00
double DD_QUEUE_LOGGING_INTERVAL;
double RELOCATION_PARALLELISM_PER_SOURCE_SERVER;
int DD_QUEUE_MAX_KEY_SERVERS;
int DD_REBALANCE_PARALLELISM;
int DD_REBALANCE_RESET_AMOUNT;
double BG_DD_MAX_WAIT;
double BG_DD_MIN_WAIT;
double BG_DD_INCREASE_RATE;
double BG_DD_DECREASE_RATE;
double BG_DD_SATURATION_DELAY;
double INFLIGHT_PENALTY_HEALTHY;
TeamRemover: Add status to show redundant team removing Distinguish the removal of unhealthy team and redundant team. Change status report to include redundant team removal report.
2019-02-22 06:16:44 +08:00
double INFLIGHT_PENALTY_REDUNDANT;
Initial repository commit
2017-05-26 04:48:44 +08:00
double INFLIGHT_PENALTY_UNHEALTHY;
double INFLIGHT_PENALTY_ONE_LEFT;
added a knob which reverts the new queue behavior
2020-03-05 08:23:49 +08:00
bool USE_OLD_NEEDED_SERVERS;
FastRestore:Replace FastRestoreOpConfig with Knobs And randomize value for the rest of knobs
2020-01-24 22:58:33 +08:00
Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards
2019-10-12 08:50:43 +08:00
// Higher priorities are executed first
// Priority/100 is the "priority group"/"superpriority". Priority inversion
// is possible within but not between priority groups; fewer priority groups
// mean better worst case time bounds
// Maximum allowable priority is 999.
int PRIORITY_RECOVER_MOVE;
int PRIORITY_REBALANCE_UNDERUTILIZED_TEAM;
int PRIORITY_REBALANCE_OVERUTILIZED_TEAM;
int PRIORITY_TEAM_HEALTHY;
int PRIORITY_TEAM_CONTAINS_UNDESIRED_SERVER;
int PRIORITY_TEAM_REDUNDANT;
merge priority did not need to be raised, because we no longer merge shards until they are untrackable max_commit_updates was too large, and could cause proxies to run out of memory
2020-01-18 06:24:58 +08:00
int PRIORITY_MERGE_SHARD;
Make the DD priority associated with populating a remote region lower than machine failures
2020-03-05 06:07:32 +08:00
int PRIORITY_POPULATE_REGION;
Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards
2019-10-12 08:50:43 +08:00
int PRIORITY_TEAM_UNHEALTHY;
int PRIORITY_TEAM_2_LEFT;
int PRIORITY_TEAM_1_LEFT;
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-22 04:31:35 +08:00
int PRIORITY_TEAM_FAILED; // Priority when a server in the team is excluded as failed
Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards
2019-10-12 08:50:43 +08:00
int PRIORITY_TEAM_0_LEFT;
int PRIORITY_SPLIT_SHARD;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Data distribution
double RETRY_RELOCATESHARD_DELAY;
double DATA_DISTRIBUTION_FAILURE_REACTION_TIME;
the keyservers shards are always a fixed large size
2017-10-28 02:52:11 +08:00
int MIN_SHARD_BYTES, SHARD_BYTES_RATIO, SHARD_BYTES_PER_SQRT_BYTES, MAX_SHARD_BYTES, KEY_SERVER_SHARD_BYTES;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t SHARD_MAX_BYTES_PER_KSEC, // Shards with more than this bandwidth will be split immediately
SHARD_MIN_BYTES_PER_KSEC, // Shards with more than this bandwidth will not be merged
SHARD_SPLIT_BYTES_PER_KSEC; // When splitting a shard, it is split into pieces with less than this bandwidth
Instead of using absolute value as the max bytesReadPerKSec threshold, use a pre-defined read traffic to byte size ratio to decide that value dynamically based on the actual size of the shard.
2019-12-07 08:35:26 +08:00
double SHARD_MAX_READ_DENSITY_RATIO;
Added the actual read hot detection algorithm and logging machanism. - When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm - The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio - Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-13 07:30:36 +08:00
int64_t SHARD_READ_HOT_BANDWITH_MIN_PER_KSECONDS;
Address review comments
2019-09-13 01:17:12 +08:00
double SHARD_MAX_BYTES_READ_PER_KSEC_JITTER;
Initial repository commit
2017-05-26 04:48:44 +08:00
double STORAGE_METRIC_TIMEOUT;
double METRIC_DELAY;
double ALL_DATA_REMOVED_DELAY;
double INITIAL_FAILURE_REACTION_DELAY;
fix: give time to detect failed servers before building teams
2018-06-11 11:21:39 +08:00
double CHECK_TEAM_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
double LOG_ON_COMPLETION_DELAY;
int BEST_TEAM_MAX_TEAM_TRIES;
int BEST_TEAM_OPTION_COUNT;
int BEST_OF_AMT;
double SERVER_LIST_DELAY;
double RECRUITMENT_IDLE_DELAY;
double STORAGE_RECRUITMENT_DELAY;
double DATA_DISTRIBUTION_LOGGING_INTERVAL;
double DD_ENABLED_CHECK_DELAY;
fix: simulation could buggify a delay such that data distribution incorrectly thinks the queue is not processing unhealthy relocations
2019-02-19 06:57:07 +08:00
double DD_STALL_CHECK_DELAY;
Data distribution will not merge a shard unless it has been low bandwidth for 5 minutes
2020-01-10 09:02:49 +08:00
double DD_LOW_BANDWIDTH_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
double DD_MERGE_COALESCE_DELAY;
double STORAGE_METRICS_POLLING_DELAY;
double STORAGE_METRICS_RANDOM_DELAY;
a high free_space_ratio_cutoff is not needed anymore because avoid teams with low disk space is no longer the responsibility of getLoadBytes()
2020-02-26 07:47:10 +08:00
double AVAILABLE_SPACE_RATIO_CUTOFF;
Initial repository commit
2017-05-26 04:48:44 +08:00
int DESIRED_TEAMS_PER_SERVER;
fix: do not attempt data movement to an unhealthy destination team allow building more teams than desired if all teams are unhealthy bestTeamStuck is an error in simulation again
2018-07-08 07:51:16 +08:00
int MAX_TEAMS_PER_SERVER;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t DD_SHARD_SIZE_GRANULARITY;
int64_t DD_SHARD_SIZE_GRANULARITY_SIM;
int DD_MOVE_KEYS_PARALLELISM;
Added a flow lock to prevent too many source server fetches from happening at the same time and running the data distributor out of memory
2020-07-10 01:38:19 +08:00
int DD_FETCH_SOURCE_PARALLELISM;
Initial repository commit
2017-05-26 04:48:44 +08:00
int DD_MERGE_LIMIT;
double DD_SHARD_METRICS_TIMEOUT;
int64_t DD_LOCATION_CACHE_SIZE;
double MOVEKEYS_LOCK_POLLING_DELAY;
double DEBOUNCE_RECRUITING_DELAY;
addressed review comments
2019-07-31 08:04:41 +08:00
int REBALANCE_MAX_RETRIES;
int DD_OVERLAP_PENALTY;
Code Review Changes
2019-09-25 01:04:56 +08:00
int DD_EXCLUDE_MIN_REPLICAS;
DD:DD_VALIDATE_LOCALITY:Guard the checking of locality validity
2019-09-14 04:18:33 +08:00
bool DD_VALIDATE_LOCALITY;
DD:Handle worker with incorrect locality When a worker has incorrect locality, the worker will be excluded from storage recruitment. When the worker has its locality corrected by system operators, the worker will be reincluded for storage recruitment.
2019-09-15 02:21:51 +08:00
int DD_CHECK_INVALID_LOCALITY_DELAY;
DD:Add trace for detailed relocate shard info
2020-03-01 05:45:00 +08:00
bool DD_ENABLE_VERBOSE_TRACING;
Addressed simple review comments
2020-04-01 09:25:11 +08:00
int64_t DD_SS_FAILURE_VERSIONLAG; // Allowed SS version lag from the current read version before marking it as failed.
Version difference is now the diff between TLog versions and SS version.
2020-04-04 10:04:43 +08:00
int64_t DD_SS_ALLOWED_VERSIONLAG; // SS will be marked as healthy if it's version lag goes below this value.
double DD_SS_STUCK_TIME_LIMIT; // If a storage server is not getting new versions for this amount of time, then it becomes undesired.
control tag encoding of keyServersValue using a knob
2020-05-14 04:29:13 +08:00
TeamCollection: Add knobs for team remover Added three knobs to control team remover bool TR_FLAG_DISABLE_TEAM_REMOVER: Disable the teamRemover actor double TR_REMOVE_MACHINE_TEAM_DELAY: Wait for the specified time before try to remove next machine team double TR_WAIT_FOR_ALL_MACHINES_HEALTHY_DELAY: Wait before checking if all machines are healthy
2019-02-14 07:11:53 +08:00
// TeamRemover to remove redundant teams
ServerTeamRemover:Bug fix and clang-format
2019-07-09 08:04:27 +08:00
bool TR_FLAG_DISABLE_MACHINE_TEAM_REMOVER; // disable the machineTeamRemover actor
TeamCollection: Add knobs for team remover Added three knobs to control team remover bool TR_FLAG_DISABLE_TEAM_REMOVER: Disable the teamRemover actor double TR_REMOVE_MACHINE_TEAM_DELAY: Wait for the specified time before try to remove next machine team double TR_WAIT_FOR_ALL_MACHINES_HEALTHY_DELAY: Wait before checking if all machines are healthy
2019-02-14 07:11:53 +08:00
double TR_REMOVE_MACHINE_TEAM_DELAY; // wait for the specified time before try to remove next machine team
MachineTeamRemover:Change to remove MT with most teams Change to remove machine team with most machine teams, using the same logic as the serverTeamRemover. The featue is guarded by TR_FLAG_REMOVE_MT_WITH_MOST_TEAMS knob.
2019-07-16 02:48:42 +08:00
bool TR_FLAG_REMOVE_MT_WITH_MOST_TEAMS; // guard to select which machineTeamRemover logic to use
Initial repository commit
2017-05-26 04:48:44 +08:00
ServerTeamRemover:Bug fix and clang-format
2019-07-09 08:04:27 +08:00
bool TR_FLAG_DISABLE_SERVER_TEAM_REMOVER; // disable the serverTeamRemover actor
Add serverTeamRemover to remove redundant server teams
2019-07-03 06:58:31 +08:00
double TR_REMOVE_SERVER_TEAM_DELAY; // wait for the specified time before try to remove next server team
ServerTeamRemover runs after machineTeamRemover finishes If serverTeamRemover removes a team before machineTeamRemover brings the machine team number down to the desired number, DD may create a new team (due to teams removed by serverTeamRemover), which may be removed later by machineTeamRemover. This causes unnnecessary extra data movement.
2019-07-20 07:48:50 +08:00
double TR_REMOVE_SERVER_TEAM_EXTRA_DELAY; // serverTeamRemover waits for the delay and check DD healthyness again to ensure it runs after machineTeamRemover
Initial repository commit
2017-05-26 04:48:44 +08:00
StorageEngineSwitch:Stop removeWrongStoreType actor if no SS has wrong storeType
2019-08-17 07:11:28 +08:00
// Remove wrong storage engines
double DD_REMOVE_STORE_ENGINE_DELAY; // wait for the specified time before remove the next batch
StorageEngineSwitch:Graceful switch When fdbcli change storeType for storage engines, we switch the store type of storage servers one by one gracefully. This avoids recruiting multiple storage servers on the same process, which can cause OOM error.
2019-08-13 01:08:12 +08:00
Add a failure timeout knob for data distributor. Set default time to 1.0s.
2019-01-19 07:43:43 +08:00
double DD_FAILURE_TIME;
Fix another build team bug The buildTeam() can create teams with undesired storage servers, which are considered unhealthy. As a result, the data movement can become stuck. Fix this by adding an ACTOR monitorHealthyTeams that builds team every one second whenever there is no healthy teams. Clean up storageServerTracker() interface.
2019-02-13 06:02:21 +08:00
double DD_ZERO_HEALTHY_TEAM_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
VersionedBTree now uses PrefixTree based pages (with bugs). This required significant changes to both classes because the interface and semantics for building, seeking in, and iterating through pages is very different from the previous trivial approach which was based on serialized vectors. PrefixTree node format rewritten to support optional values without increasing overhead for common node scenarios. PrefixTree::Cursor rewritten to reuse path prefix memory instead of allocating new memory on each movement which is then 'leaked' until destruction. PrefixTree::Cursor movement modified to work better with VersionedBTree::InternalCursor, which was also heavily modified. Added knobs related to key arrangement in PrefixTree nodes. Added StringRef::toHexString() as an alternative to printable() to make reading raw PrefixTree data easier. PrefixTree performance is temporarily worse with this update and VersionedBtree fails its unit test.
2018-06-08 18:32:34 +08:00
// Redwood Storage Engine
int PREFIX_TREE_IMMEDIATE_KEY_SIZE_LIMIT;
int PREFIX_TREE_IMMEDIATE_KEY_SIZE_MIN;
Initial repository commit
2017-05-26 04:48:44 +08:00
// KeyValueStore SQLITE
int CLEAR_BUFFER_SIZE;
double READ_VALUE_TIME_ESTIMATE;
double READ_RANGE_TIME_ESTIMATE;
double SET_TIME_ESTIMATE;
double CLEAR_TIME_ESTIMATE;
double COMMIT_TIME_ESTIMATE;
int CHECK_FREE_PAGE_AMOUNT;
double DISK_METRIC_LOGGING_INTERVAL;
int64_t SOFT_HEAP_LIMIT;
int SQLITE_PAGE_SCAN_ERROR_LIMIT;
int SQLITE_BTREE_PAGE_USABLE;
int SQLITE_BTREE_CELL_MAX_LOCAL;
int SQLITE_BTREE_CELL_MIN_LOCAL;
int SQLITE_FRAGMENT_PRIMARY_PAGE_USABLE;
int SQLITE_FRAGMENT_OVERFLOW_PAGE_USABLE;
double SQLITE_FRAGMENT_MIN_SAVINGS;
Implemented the chunk size and db size hint fileControl options in our SQLite VFS implementation. KeyValueStoreSQLite now sets file chunk size based on a new knob, SQLITE_CHUNK_SIZE_PAGES.
2019-04-23 19:50:58 +08:00
int SQLITE_CHUNK_SIZE_PAGES;
Increased the SQLITE_CHUNK_SIZE to 100MB (left at 4MB for simulation)
2019-04-27 04:53:56 +08:00
int SQLITE_CHUNK_SIZE_PAGES_SIM;
Initial repository commit
2017-05-26 04:48:44 +08:00
// KeyValueStoreSqlite spring cleaning
Increase the frequency that lazy deletes are run. Add more parameters for better control over the spring cleaning process.
2019-04-24 05:01:51 +08:00
double SPRING_CLEANING_NO_ACTION_INTERVAL;
double SPRING_CLEANING_LAZY_DELETE_INTERVAL;
double SPRING_CLEANING_VACUUM_INTERVAL;
double SPRING_CLEANING_LAZY_DELETE_TIME_ESTIMATE;
double SPRING_CLEANING_VACUUM_TIME_ESTIMATE;
Initial repository commit
2017-05-26 04:48:44 +08:00
double SPRING_CLEANING_VACUUMS_PER_LAZY_DELETE_PAGE;
int SPRING_CLEANING_MIN_LAZY_DELETE_PAGES;
int SPRING_CLEANING_MAX_LAZY_DELETE_PAGES;
int SPRING_CLEANING_LAZY_DELETE_BATCH_SIZE;
int SPRING_CLEANING_MIN_VACUUM_PAGES;
int SPRING_CLEANING_MAX_VACUUM_PAGES;
knobify replace contents bytes
2018-05-02 10:43:35 +08:00
// KeyValueStoreMemory
int64_t REPLACE_CONTENTS_BYTES;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Leader election
fix: fixed a memory leak where leaderInfo notifications are not cleared out
2018-07-07 08:40:29 +08:00
int MAX_NOTIFICATIONS;
slowly send notifications to clients to clear the list of dead clients
2018-08-09 08:29:32 +08:00
int MIN_NOTIFICATIONS;
double NOTIFICATION_FULL_CLEAR_TIME;
Initial repository commit
2017-05-26 04:48:44 +08:00
double CANDIDATE_MIN_DELAY;
double CANDIDATE_MAX_DELAY;
double CANDIDATE_GROWTH_RATE;
double POLLING_FREQUENCY;
double HEARTBEAT_FREQUENCY;
// Master Proxy
double START_TRANSACTION_BATCH_INTERVAL_MIN;
double START_TRANSACTION_BATCH_INTERVAL_MAX;
double START_TRANSACTION_BATCH_INTERVAL_LATENCY_FRACTION;
double START_TRANSACTION_BATCH_INTERVAL_SMOOTHER_ALPHA;
double START_TRANSACTION_BATCH_QUEUE_CHECK_INTERVAL;
double START_TRANSACTION_MAX_TRANSACTIONS_TO_START;
increased the GRV client batch size, similarly increased the proxy limits related to the number of transactions started in a batch
2019-03-17 07:18:58 +08:00
int START_TRANSACTION_MAX_REQUESTS_TO_START;
Change the algorithm for the proxy handing out read versions to improve performance and increase responsiveness to changes in workload.
2020-02-25 01:52:31 +08:00
double START_TRANSACTION_RATE_WINDOW;
double START_TRANSACTION_MAX_EMPTY_QUEUE_BUDGET;
Limit the amount of requests the proxy can queue up in memory
2020-03-14 01:17:49 +08:00
int START_TRANSACTION_MAX_QUEUE_SIZE;
int KEY_LOCATION_MAX_QUEUE_SIZE;
Initial repository commit
2017-05-26 04:48:44 +08:00
double COMMIT_TRANSACTION_BATCH_INTERVAL_FROM_IDLE;
double COMMIT_TRANSACTION_BATCH_INTERVAL_MIN;
double COMMIT_TRANSACTION_BATCH_INTERVAL_MAX;
double COMMIT_TRANSACTION_BATCH_INTERVAL_LATENCY_FRACTION;
double COMMIT_TRANSACTION_BATCH_INTERVAL_SMOOTHER_ALPHA;
int COMMIT_TRANSACTION_BATCH_COUNT_MAX;
int COMMIT_TRANSACTION_BATCH_BYTES_MIN;
int COMMIT_TRANSACTION_BATCH_BYTES_MAX;
double COMMIT_TRANSACTION_BATCH_BYTES_SCALE_BASE;
double COMMIT_TRANSACTION_BATCH_BYTES_SCALE_POWER;
Proxy going OOM when applying high volumes of writes to a proxy, particular in a sudden fashion before ratekeeper can control the workload. Address this issue by proactively monitoring the memory used by commit batches and dropping requests if a certain memory limit is exceeded.
2018-06-02 06:21:40 +08:00
int64_t COMMIT_BATCHES_MEM_BYTES_HARD_LIMIT;
double COMMIT_BATCHES_MEM_FRACTION_OF_TOTAL;
double COMMIT_BATCHES_MEM_TO_TOTAL_MEM_SCALE_FACTOR;
Initial repository commit
2017-05-26 04:48:44 +08:00
double RESOLVER_COALESCE_TIME;
int BUGGIFIED_ROW_LIMIT;
double PROXY_SPIN_DELAY;
added a workload which tests killing an entire region, and recovering from the failure with data loss. fix: we cannot pop the txs tag from remote logs until they have a full copy of the txnStateStore fix: we have to modify all of history, we cannot stop after finding a local remote
2018-09-18 09:32:39 +08:00
double UPDATE_REMOTE_LOG_VERSION_INTERVAL;
int MAX_TXS_POP_VERSION_HISTORY;
only skip confirm epoch live if CAUSAL_READ_RISKY is enabled time checked on the proxy should be less than the time waited by the master to account for clock speed differences setting REQUIRED_MIN_RECOVERY_DURATION and ENFORCED_MIN_RECOVERY_DURATION to 0 will go back to the old behavior
2019-07-13 08:58:16 +08:00
double MIN_CONFIRM_INTERVAL;
double ENFORCED_MIN_RECOVERY_DURATION;
double REQUIRED_MIN_RECOVERY_DURATION;
Added a knob to always use CAUSAL_READ_RISKY for GRV
2019-07-31 09:21:46 +08:00
bool ALWAYS_CAUSAL_READ_RISKY;
limit the number of committed version updates in progress simultaneously to prevent running out of memory
2019-10-22 07:01:45 +08:00
int MAX_COMMIT_UPDATES;
more accurate calculation on the amount of time that proxy should wait before getting a version from the master
2020-01-27 11:47:12 +08:00
double MIN_PROXY_COMPUTE;
prevent the proxy from delaying too long based on an incorrect estimate of the compute time
2020-02-26 07:46:13 +08:00
double MAX_PROXY_COMPUTE;
more accurate calculation on the amount of time that proxy should wait before getting a version from the master
2020-01-27 11:47:12 +08:00
int PROXY_COMPUTE_BUCKETS;
double PROXY_COMPUTE_GROWTH_RATE;
code cleanup
2020-04-11 08:02:11 +08:00
int TXN_STATE_SEND_AMOUNT;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Master Server
double COMMIT_SLEEP_TIME;
double MIN_BALANCE_TIME;
int64_t MIN_BALANCE_DIFFERENCE;
double SECONDS_BEFORE_NO_FAILURE_DELAY;
int64_t MAX_TXS_SEND_MEMORY;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
int64_t MAX_RECOVERY_VERSIONS;
double MAX_RECOVERY_TIME;
fix: restarting the provisional proxy causes all tlog peeks to restart, so if tlog peeks take longer than 1 second this could end in an infinite loop
2019-03-23 05:13:58 +08:00
double PROVISIONAL_START_DELAY;
double PROVISIONAL_DELAY_GROWTH;
double PROVISIONAL_MAX_DELAY;
Add delay for master to recruit backup workers This delay is to ensure old epoch's backup workers can save their progress in the database. Otherwise, the new master could attempts to recruit backup workers for the old epoch on version ranges that have already been popped. As a result, the logs will lose data.
2020-03-06 03:34:37 +08:00
double SECONDS_BEFORE_RECRUIT_BACKUP_WORKER;
wait for the correct cluster controller interface before starting master recovery
2020-07-21 02:29:37 +08:00
double CC_INTERFACE_TIMEOUT;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Resolver
tuned resolver balancing to keep the resolvers within 5MB per second of each other
2018-11-12 15:42:45 +08:00
int64_t KEY_BYTES_PER_SAMPLE;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t SAMPLE_OFFSET_PER_KEY;
double SAMPLE_EXPIRATION_TIME;
double SAMPLE_POLL_TIME;
int64_t RESOLVER_STATE_MEMORY_LIMIT;
Add backup role, class, and worker skeleton
2019-04-25 06:12:37 +08:00
// Backup Worker
Add backup worker recruitment in master.
2019-05-15 00:55:39 +08:00
double BACKUP_TIMEOUT; // master's reaction time for backup failure
Add an option to get minKnownCommittedVersion from Proxies The backup worker needs to use this version for popping when running in a NOOP mode. This option is added to GetReadVersionRequest and proxies will send back minKnownCommittedVersion if the option is set. Also add a couple of knobs for backup workers.
2020-01-08 06:15:29 +08:00
double BACKUP_NOOP_POP_DELAY;
int BACKUP_FILE_BLOCK_BYTES;
Limit memory usage of backup workers
2020-05-15 03:05:16 +08:00
int64_t BACKUP_LOCK_BYTES;
BackupWorker:Buggify upload delay Add questions to code as well.
2020-03-19 10:04:43 +08:00
double BACKUP_UPLOAD_DELAY;
Add backup role, class, and worker skeleton
2019-04-25 06:12:37 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
//Cluster Controller
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
double CLUSTER_CONTROLLER_LOGGING_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
double MASTER_FAILURE_REACTION_TIME;
double MASTER_FAILURE_SLOPE_DURING_RECOVERY;
int WORKER_COORDINATION_PING_DELAY;
double SIM_SHUTDOWN_TIMEOUT;
double SHUTDOWN_TIMEOUT;
double MASTER_SPIN_DELAY;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
double CC_CHANGE_DELAY;
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
double CC_CLASS_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
double WAIT_FOR_GOOD_RECRUITMENT_DELAY;
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
double WAIT_FOR_GOOD_REMOTE_RECRUITMENT_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
double ATTEMPT_RECRUITMENT_DELAY;
Per review comments Add a knob for starting distributor delay. Move distributor failed variable to a local loop.
2019-02-01 02:51:25 +08:00
double WAIT_FOR_DISTRIBUTOR_JOIN_DELAY;
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
double WAIT_FOR_RATEKEEPER_JOIN_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
double WORKER_FAILURE_TIME;
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
double CHECK_OUTSTANDING_INTERVAL;
Initial repository commit
2017-05-26 04:48:44 +08:00
double INCOMPATIBLE_PEERS_LOGGING_INTERVAL;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
double VERSION_LAG_METRIC_INTERVAL;
int64_t MAX_VERSION_DIFFERENCE;
knobify force recovery recovery check delay
2019-02-20 08:05:20 +08:00
double FORCE_RECOVERY_CHECK_DELAY;
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
double RATEKEEPER_FAILURE_TIME;
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
double REPLACE_INTERFACE_DELAY;
double REPLACE_INTERFACE_CHECK_DELAY;
added status information on connected clients
2019-07-26 08:15:31 +08:00
double COORDINATOR_REGISTER_INTERVAL;
fix: lastRequestTime was not updated fix: COORDINATOR_REGISTER_INTERVAL was not set fixed review comments
2019-07-27 04:23:56 +08:00
double CLIENT_REGISTER_INTERVAL;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Knobs used to select the best policy (via monte carlo)
int POLICY_RATING_TESTS; // number of tests per policy (in order to compare)
int POLICY_GENERATIONS; // number of policies to generate
int EXPECTED_MASTER_FITNESS;
int EXPECTED_TLOG_FITNESS;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
int EXPECTED_LOG_ROUTER_FITNESS;
Initial repository commit
2017-05-26 04:48:44 +08:00
int EXPECTED_PROXY_FITNESS;
int EXPECTED_RESOLVER_FITNESS;
double RECRUITMENT_TIMEOUT;
code cleanup
2020-04-11 08:02:11 +08:00
int DBINFO_SEND_AMOUNT;
double DBINFO_BATCH_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
//Move Keys
double SHARD_READY_DELAY;
double SERVER_READY_QUORUM_INTERVAL;
double SERVER_READY_QUORUM_TIMEOUT;
double REMOVE_RETRY_DELAY;
int MOVE_KEYS_KRM_LIMIT;
int MOVE_KEYS_KRM_LIMIT_BYTES; //This must be sufficiently larger than CLIENT_KNOBS->KEY_SIZE_LIMIT (fdbclient/Knobs.h) to ensure that at least two entries will be returned from an attempt to read a key range map
int MAX_SKIP_TAGS;
fix: start move keys will only move destination servers to become source servers if less than destination servers are healthy and the total number of sources is less than 2x the number of destinations
2018-09-01 03:43:14 +08:00
double MAX_ADDED_SOURCES_MULTIPLIER;
Initial repository commit
2017-05-26 04:48:44 +08:00
//FdbServer
double MIN_REBOOT_TIME;
double MAX_REBOOT_TIME;
Add a string knob to hold the Log directory, and write profiles to it. This is the combination of two small changes. 1. Add support for a string knob type. 2. Change profiles to be written to the log directory instead of the working directory. We have three options of where to write files: the working directory, the data directory, and the log directory. The working directory may be set to a non-writable location, and likely contains the fdb binaries. Allowing these files to be overwritten would likely not be a wise idea. The data directory hosts our sqlite b-trees. It would also be very unfortunate if these were ever overwritten by an unfortunate profile name. The log directory contains logs. Out of the three, these matter the least if they disappear or become corrupted. Thus, we write to the log directory.
2017-10-13 08:49:41 +08:00
std::string LOG_DIRECTORY;
Proxy going OOM when applying high volumes of writes to a proxy, particular in a sudden fashion before ratekeeper can control the workload. Address this issue by proactively monitoring the memory used by commit batches and dropping requests if a certain memory limit is exceeded.
2018-06-02 06:21:40 +08:00
int64_t SERVER_MEM_LIMIT;
Initial repository commit
2017-05-26 04:48:44 +08:00
//Ratekeeper
double SMOOTHING_AMOUNT;
double SLOW_SMOOTHING_AMOUNT;
double METRIC_UPDATE_RATE;
Track health metrics in Ratekeeper and send these metrics to proxies in GetRateInfoReply messages
2019-02-01 04:56:58 +08:00
double DETAILED_METRIC_UPDATE_RATE;
Initial repository commit
2017-05-26 04:48:44 +08:00
double LAST_LIMITED_RATIO;
Clamp infinite limit in ratekeeper
2020-01-15 07:45:24 +08:00
double RATEKEEPER_DEFAULT_LIMIT;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t TARGET_BYTES_PER_STORAGE_SERVER;
Initial implementation of adding separate limits for batch priority in ratekeeper
2019-02-28 02:31:56 +08:00
int64_t SPRING_BYTES_STORAGE_SERVER;
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-25 02:31:16 +08:00
int64_t AUTO_TAG_THROTTLE_STORAGE_QUEUE_BYTES;
Initial implementation of adding separate limits for batch priority in ratekeeper
2019-02-28 02:31:56 +08:00
int64_t TARGET_BYTES_PER_STORAGE_SERVER_BATCH;
int64_t SPRING_BYTES_STORAGE_SERVER_BATCH;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t TARGET_BYTES_PER_TLOG;
Initial implementation of adding separate limits for batch priority in ratekeeper
2019-02-28 02:31:56 +08:00
int64_t SPRING_BYTES_TLOG;
int64_t TARGET_BYTES_PER_TLOG_BATCH;
int64_t SPRING_BYTES_TLOG_BATCH;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t TLOG_SPILL_THRESHOLD;
added a hard memory limit to the TLog to prevent it from running out of memory. Because remote logs are not ratekeeper controlled this is their only protection
2018-06-19 08:22:40 +08:00
int64_t TLOG_HARD_LIMIT_BYTES;
added a separate knob for tlog’s recoverMemoryLimit
2018-08-22 12:11:23 +08:00
int64_t TLOG_RECOVER_MEMORY_LIMIT;
ignorePopDeadline to have high limit in simulator - ignorePopDeadline to have highier limit in simulator to accommdate for the buggify delays and make snapshot succeed. - introduce a new knob for auto resetting the disabling of tlog pop
2019-05-14 06:43:27 +08:00
double TLOG_IGNORE_POP_AUTO_ENABLE_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
Enforce a throttle limit. Don't count transaction tags on RK if the proxy has updated us in a while.
2020-04-18 02:48:02 +08:00
int64_t MAX_MANUAL_THROTTLED_TRANSACTION_TAGS;
int64_t MAX_AUTO_THROTTLED_TRANSACTION_TAGS;
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-25 02:31:16 +08:00
double MIN_TAG_COST;
double AUTO_THROTTLE_TARGET_TAG_BUSYNESS;
Initial support for ramping load back up. Fix some logging. Update auto-throttles less frequently.
2020-04-29 06:50:45 +08:00
double AUTO_THROTTLE_RAMP_TAG_BUSYNESS;
double AUTO_TAG_THROTTLE_RAMP_UP_TIME;
Reorganization of throttle storage in ratekeeper to support various auto-throttling related actions
2020-04-29 05:30:37 +08:00
double AUTO_TAG_THROTTLE_DURATION;
Various bug fixes
2020-04-23 03:28:51 +08:00
double TAG_THROTTLE_PUSH_INTERVAL;
Reorganization of throttle storage in ratekeeper to support various auto-throttling related actions
2020-04-29 05:30:37 +08:00
double AUTO_TAG_THROTTLE_START_AGGREGATION_TIME;
Initial support for ramping load back up. Fix some logging. Update auto-throttles less frequently.
2020-04-29 06:50:45 +08:00
double AUTO_TAG_THROTTLE_UPDATE_FREQUENCY;
Fix various bugs and make sure to cleanup throttles from the database when they expire
2020-05-02 12:36:28 +08:00
double TAG_THROTTLE_EXPIRED_CLEANUP_INTERVAL;
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-04 06:24:14 +08:00
bool AUTO_TAG_THROTTLING_ENABLED;
Initial repository commit
2017-05-26 04:48:44 +08:00
double MAX_TRANSACTIONS_PER_BYTE;
Once available space ratio falls below 0.3 avoid moving data to teams with less free space than the median team
2020-02-22 07:14:32 +08:00
int64_t MIN_AVAILABLE_SPACE;
double MIN_AVAILABLE_SPACE_RATIO;
double TARGET_AVAILABLE_SPACE_RATIO;
double AVAILABLE_SPACE_UPDATE_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
double MAX_TL_SS_VERSION_DIFFERENCE; // spring starts at half this value
Initial implementation of adding separate limits for batch priority in ratekeeper
2019-02-28 02:31:56 +08:00
double MAX_TL_SS_VERSION_DIFFERENCE_BATCH;
Initial repository commit
2017-05-26 04:48:44 +08:00
int MAX_MACHINES_FALLING_BEHIND;
knobified the durability lag rate controls
2019-07-12 09:34:19 +08:00
int MAX_TPS_HISTORY_SAMPLES;
int NEEDED_TPS_HISTORY_SAMPLES;
int64_t TARGET_DURABILITY_LAG_VERSIONS;
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-25 02:31:16 +08:00
int64_t AUTO_TAG_THROTTLE_DURABILITY_LAG_VERSIONS;
knobified the durability lag rate controls
2019-07-12 09:34:19 +08:00
int64_t TARGET_DURABILITY_LAG_VERSIONS_BATCH;
avoid going to unlimited immediately after going below the durabilityLagTargetVersion
2019-07-13 04:40:18 +08:00
int64_t DURABILITY_LAG_UNLIMITED_THRESHOLD;
knobified the durability lag rate controls
2019-07-12 09:34:19 +08:00
double INITIAL_DURABILITY_LAG_MULTIPLIER;
double DURABILITY_LAG_REDUCTION_RATE;
avoid going to unlimited immediately after going below the durabilityLagTargetVersion
2019-07-13 04:40:18 +08:00
double DURABILITY_LAG_INCREASE_RATE;
Added metrics for read hot key detection
2019-08-30 05:44:16 +08:00
Ratekeeper to throttle tpsLimit to 1 if it is not able to fetch storage server list for some configurable amount of time.
2019-07-18 05:47:08 +08:00
double STORAGE_SERVER_LIST_FETCH_TIMEOUT;
Added metrics for read hot key detection
2019-08-30 05:44:16 +08:00
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-25 02:31:16 +08:00
// disk snapshot
double SNAP_CREATE_MAX_TIMEOUT;
Initial repository commit
2017-05-26 04:48:44 +08:00
//Storage Metrics
double STORAGE_METRICS_AVERAGE_INTERVAL;
double STORAGE_METRICS_AVERAGE_INTERVAL_PER_KSECONDS;
double SPLIT_JITTER_AMOUNT;
int64_t IOPS_UNITS_PER_SAMPLE;
int64_t BANDWIDTH_UNITS_PER_SAMPLE;
Added metrics for read hot key detection
2019-08-30 05:44:16 +08:00
int64_t BYTES_READ_UNITS_PER_SAMPLE;
Done the plumbing work at the SSI side.
2019-12-20 08:02:42 +08:00
int64_t READ_HOT_SUB_RANGE_CHUNK_SIZE;
Defined a minimum read cost (a penalty) for empty read or read size smaller than it. Fixed several review comments.
2019-10-31 01:04:19 +08:00
int64_t EMPTY_READ_PENALTY;
Code clean up
2019-11-22 05:39:19 +08:00
bool READ_SAMPLING_ENABLED;
Initial repository commit
2017-05-26 04:48:44 +08:00
//Storage Server
double STORAGE_LOGGING_DELAY;
double STORAGE_SERVER_POLL_METRICS_DELAY;
double FUTURE_VERSION_DELAY;
int STORAGE_LIMIT_BYTES;
int BUGGIFY_LIMIT_BYTES;
int FETCH_BLOCK_BYTES;
int FETCH_KEYS_PARALLELISM_BYTES;
removed extra parameter from getRange, added knob to guard new changes, and adjusted style/formatting in several places
2019-07-09 02:52:25 +08:00
int FETCH_KEYS_LOWER_PRIORITY;
Initial repository commit
2017-05-26 04:48:44 +08:00
int BUGGIFY_BLOCK_BYTES;
int64_t STORAGE_HARD_LIMIT_BYTES;
Prototype of local ratekeeper
2019-03-02 05:14:18 +08:00
int64_t STORAGE_DURABILITY_LAG_HARD_MAX;
restored the STORAGE_DURABILITY_LAG_SOFT_MAX knob and made the rk target slightly smaller than the soft limit, to avoid inaccuracies in ratekeeper control causing behavior changes on the storage servers
2019-06-26 09:22:44 +08:00
int64_t STORAGE_DURABILITY_LAG_SOFT_MAX;
Avoid jumping from rejecting 0% of requests directly to 20% of requests
2019-06-26 09:29:08 +08:00
double STORAGE_DURABILITY_LAG_REJECT_THRESHOLD;
fix: we cannot reject 100% of requests, because a storage server which is stuck needs to get a future version error to trigger an all alternatives failed message from load balance so that clients will re-grab storage server interfaces from the proxy
2019-07-06 06:46:16 +08:00
double STORAGE_DURABILITY_LAG_MIN_RATE;
Initial repository commit
2017-05-26 04:48:44 +08:00
int STORAGE_COMMIT_BYTES;
double STORAGE_COMMIT_INTERVAL;
double UPDATE_SHARD_VERSION_INTERVAL;
int BYTE_SAMPLING_FACTOR;
int BYTE_SAMPLING_OVERHEAD;
int MAX_STORAGE_SERVER_WATCH_BYTES;
int MAX_BYTE_SAMPLE_CLEAR_MAP_SIZE;
double LONG_BYTE_SAMPLE_RECOVERY_DELAY;
byte sample recovery did not use limits for its range reads, leading to slow tasks
2019-01-05 02:32:31 +08:00
int BYTE_SAMPLE_LOAD_PARALLELISM;
double BYTE_SAMPLE_LOAD_DELAY;
Prevent the byte sample recovery from interfering with storage server recovery
2019-06-14 06:55:25 +08:00
double BYTE_SAMPLE_START_DELAY;
Storage server periodically calculates cpuUsage and diskUsage metrics. These metrics (as well as all other metrics necessary for health metrics calculation) are sent in the StorageQueuingMetricsReply message.
2019-02-01 03:49:23 +08:00
double UPDATE_STORAGE_PROCESS_STATS_INTERVAL;
fix: clients would waste time attempting to read from a remote region when it was in the process of catching up
2020-01-11 04:23:59 +08:00
double BEHIND_CHECK_DELAY;
int BEHIND_CHECK_COUNT;
int64_t BEHIND_CHECK_VERSIONS;
occasionally throw wrong_shard_server when waitMetrics times out so that the waitMetrics request can get the correct number of shards if two shards have been merged or split and the same storage server owns all the chunks
2020-01-16 05:22:18 +08:00
double WAIT_METRICS_WRONG_SHARD_CHANCE;
Some initial work to support tagging transactions and passing them around.
2020-03-21 02:23:11 +08:00
int64_t MIN_TAG_PAGES_READ_RATE;
Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients. A few refactorings.
2020-04-17 05:43:22 +08:00
double READ_TAG_MEASUREMENT_INTERVAL;
Adjust and knobify cost function for ops on the storage server
2020-04-23 05:39:32 +08:00
int64_t OPERATION_COST_BYTE_FACTOR;
Add a knob which controls writing prefix compressed kvs mem snapshots. Which will be set to on by default in 7.0
2020-05-13 08:01:52 +08:00
bool PREFIX_COMPRESS_KVS_MEM_SNAPSHOTS;
Initial repository commit
2017-05-26 04:48:44 +08:00
//Wait Failure
int MAX_OUTSTANDING_WAIT_FAILURE_REQUESTS;
double WAIT_FAILURE_DELAY_LIMIT;
//Worker
double WORKER_LOGGING_INTERVAL;
Dump heap profiler when memory usage is high Set the threshold of dump to 2GB.
2019-04-06 05:45:58 +08:00
double HEAP_PROFILER_INTERVAL;
A process will mark itself as degraded if it continually disconnects from a different process which the failure monitor thinks is healthy
2019-04-05 05:11:12 +08:00
double DEGRADED_RESET_INTERVAL;
double DEGRADED_WARNING_LIMIT;
do not log a degraded reset message if the previous reset was more than a week ago
2019-04-08 14:00:58 +08:00
double DEGRADED_WARNING_RESET_DELAY;
Now the inability to flush trace logs will be reported to both 'stderr' and also the status json object. - Since the first flush failure, if the accumulated consecutive failure count exceeds the value defined in knobs, it will trigger the current worker process to report this issue via the 'GetServerDBInfo' interface of the cluster controler - A successful flush will reset the accumulated counter. Notice that the current solution does not take the time into consideration. The assumption is that flush failures tend to only happen in a clustered manner. The intermittent, but short, periods of flush failures are not considered as a problem since the memory pressure built by them should be negligible.
2020-01-27 13:11:15 +08:00
int64_t TRACE_LOG_FLUSH_FAILURE_CHECK_INTERVAL_SECONDS;
Added the ability to ping a trace log writer thread and the monitoring in worker.actor.cpp. The current solution is simple a loose check. We can change this to be accurate check by using 'pthread_kill(writer_thread, 0)'
2020-01-29 02:47:33 +08:00
double TRACE_LOG_PING_TIMEOUT_SECONDS;
Delay cluster controller candidacy for all worst fit processes, not just storage servers.
2020-06-11 00:59:56 +08:00
double MIN_DELAY_CC_WORST_FIT_CANDIDACY_SECONDS; // Listen for a leader for N seconds, and if not heard, then try to become the leader.
double MAX_DELAY_CC_WORST_FIT_CANDIDACY_SECONDS;
knobified parameter
2020-04-12 11:54:17 +08:00
double DBINFO_FAILED_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Test harness
double WORKER_POLL_DELAY;
// Coordination
double COORDINATED_STATE_ONCONFLICT_POLL_INTERVAL;
// Buggification
double BUGGIFIED_EVENTUAL_CONSISTENCY;
bool BUGGIFY_ALL_COORDINATION;
// Status
double STATUS_MIN_TIME_BETWEEN_REQUESTS;
Throttle status requests with MAX_STATUS_REQUESTS_PER_SECOND knob, whenever status batching is used.
2019-01-29 07:37:30 +08:00
double MAX_STATUS_REQUESTS_PER_SECOND;
Initial repository commit
2017-05-26 04:48:44 +08:00
int CONFIGURATION_ROWS_TO_FETCH;
added a warning if multiple log ranges exist for the same range
2019-10-03 08:06:19 +08:00
bool DISABLE_DUPLICATE_LOG_WARNING;
Initial repository commit
2017-05-26 04:48:44 +08:00
Merge branch 'master' into feature-redwood # Conflicts: # fdbserver/fdbserver.vcxproj # fdbserver/fdbserver.vcxproj.filters
2017-06-10 05:56:41 +08:00
// IPager
int PAGER_RESERVED_PAGES;
// IndirectShadowPager
int FREE_PAGE_VACUUM_THRESHOLD;
int VACUUM_QUEUE_SIZE;
int VACUUM_BYTES_PER_SECOND;
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
// Timekeeper
int64_t TIME_KEEPER_DELAY;
int64_t TIME_KEEPER_MAX_ENTRIES;
FastRestore:Refactor code 1) Use the runRYWTransaction for simple DB access 2) Replace some printf with TraceEvent 3) Remove printf not used in debugging 4) Avoid wait inside the condition in loop-choose-when for the core routine of restore worker, loader and applier. 5) Rename Restore.actor.cpp to RestoreWorker.actor.cpp since the file only has functionalities related to restore worker. Passed correctness test
2019-06-01 02:09:31 +08:00
// Fast Restore
int64_t FASTRESTORE_FAILURE_TIMEOUT;
int64_t FASTRESTORE_HEARTBEAT_INTERVAL;
FastRestore:Sampling:Resolve review comments
2019-12-05 09:11:40 +08:00
double FASTRESTORE_SAMPLING_PERCENT;
FastRestore:Replace FastRestoreOpConfig with Knobs And randomize value for the rest of knobs
2020-01-24 22:58:33 +08:00
int64_t FASTRESTORE_NUM_LOADERS;
int64_t FASTRESTORE_NUM_APPLIERS;
FastRestore:Resolve review comments
2020-02-19 05:55:00 +08:00
// FASTRESTORE_TXN_BATCH_MAX_BYTES is target txn size used by appliers to apply mutations
FastRestore:Replace FastRestoreOpConfig with Knobs And randomize value for the rest of knobs
2020-01-24 22:58:33 +08:00
double FASTRESTORE_TXN_BATCH_MAX_BYTES;
// FASTRESTORE_VERSIONBATCH_MAX_BYTES is the maximum data size in each version batch
double FASTRESTORE_VERSIONBATCH_MAX_BYTES;
FastRestore:Introduce FASTRESTORE_VB_PARALLELISM for controlling the number of concurrently running version batches.
2020-01-29 02:16:44 +08:00
// FASTRESTORE_VB_PARALLELISM is the number of concurrently running version batches
int64_t FASTRESTORE_VB_PARALLELISM;
FastRestore:monitorFinishedVersion for measuring perf quickly
2020-02-06 06:25:26 +08:00
int64_t FASTRESTORE_VB_MONITOR_DELAY; // How quickly monitor finished version batch
FastRestore:Add trace for each phase on master
2020-02-09 14:10:57 +08:00
int64_t FASTRESTORE_VB_LAUNCH_DELAY;
FastRestore:Loader:Add metrics counter
2020-02-10 11:19:32 +08:00
int64_t FASTRESTORE_ROLE_LOGGING_DELAY;
FastRestore:Applier:Add metrics counter and proc counter
2020-02-11 07:06:03 +08:00
int64_t FASTRESTORE_UPDATE_PROCESS_STATS_INTERVAL; // How quickly to update process metrics for restore
FastRestore:AtomicOp:Intro weighted size for atomicOp atomicOp has an amplified performance overhead to the cluster, for example, an ADD operation can be small, but SS has to load the value to do the operation and the value can be large.
2020-02-12 03:44:53 +08:00
int64_t FASTRESTORE_ATOMICOP_WEIGHT; // workload amplication factor for atomic op
FastRestore:Applier:Precompute mutations and apply in parallel Precompute mutations received by an applier; Only apply the final result to the destination DB; Execute multiple txns in parallel to apply final results to the destination DB.
2020-02-13 13:45:29 +08:00
int64_t FASTRESTORE_APPLYING_PARALLELISM; // number of outstanding txns writing to dest. DB
FastRestore:Minor revision for code review
2020-02-20 03:52:24 +08:00
int64_t FASTRESTORE_MONITOR_LEADER_DELAY;
FastRestore:Add unit name to threshold knob name
2020-03-03 02:52:44 +08:00
int64_t FASTRESTORE_STRAGGLER_THRESHOLD_SECONDS;
FastRestore:Use more concise code as suggested in review
2020-03-02 12:39:56 +08:00
bool FASTRESTORE_TRACK_REQUEST_LATENCY; // true to track reply latency of each request in a request batch
FastRestore:Switch to new sendBatchRequests that tracks performance and straggler
2020-02-22 07:39:13 +08:00
bool FASTRESTORE_TRACK_LOADER_SEND_REQUESTS; // track requests of load send mutations to appliers?
FastRestore:Introduce OOM protection An actor is schedulable to run if the current worker has enough resourc, i.e., the worker's memory usage is below the threshold; Exception: If the actor is working on the current version batch, we have to schedule the actor to run to avoid dead-lock. Future: When we release the actors that are blocked by memory usage, we should release them in increasing order of their version batch.
2020-02-27 05:43:30 +08:00
int64_t FASTRESTORE_MEMORY_THRESHOLD_MB_SOFT; // threshold when pipelined actors should be delayed
int64_t FASTRESTORE_WAIT_FOR_MEMORY_LATENCY;
FastRestore:Master:Report unavailable role periodically Ping all restore roles and report unavailable ones.
2020-02-27 07:45:21 +08:00
int64_t FASTRESTORE_HEARTBEAT_DELAY; // interval for master to ping loaders and appliers
int64_t FASTRESTORE_HEARTBEAT_MAX_DELAY; // master claim a node is down if no heart beat from the node for this delay
FastRestore:Applier:Limit fetch keys number in a txn in getAndComputeStagingKeys
2020-02-29 08:00:47 +08:00
int64_t FASTRESTORE_APPLIER_FETCH_KEYS_SIZE; // number of keys to fetch in a txn on applier
FastRestore:Loader:Send mutations at different versions in the same message to appliers This increases the bandwidth sent from loaders to appliers.
2020-04-12 13:31:55 +08:00
int64_t FASTRESTORE_LOADER_SEND_MUTATION_MSG_BYTES; // desired size of mutation message sent from loader to appliers
FastRestore:Use knob to guard the expensive way to get range versions
2020-04-18 01:02:53 +08:00
bool FASTRESTORE_GET_RANGE_VERSIONS_EXPENSIVE; // parse each range file to get (range, version) it has?
FastRestore:Add knob to control parallelism in waiting requests
2020-04-20 11:50:51 +08:00
int64_t FASTRESTORE_REQBATCH_PARALLEL; // number of requests to wait on for getBatchReplies()
FastRestore:Logging for getReplyBatches
2020-05-05 11:12:59 +08:00
bool FASTRESTORE_REQBATCH_LOG; // verbose log information for getReplyBatches
FastRestoreApplier:Trace clear range op when it has too many for debug
2020-05-06 00:28:45 +08:00
int FASTRESTORE_TXN_CLEAR_MAX; // threshold to start tracking each clear op in a txn
Resovle review comments
2020-05-08 06:06:59 +08:00
int FASTRESTORE_TXN_RETRY_MAX; // threshold to start output error on too many retries
Rename knob to FASTRESTORE_TXN_EXTRA_DELAY
2020-06-30 12:16:30 +08:00
double FASTRESTORE_TXN_EXTRA_DELAY; // extra delay to avoid overwhelming fdb
ClientKnobs: Add CHECK_CONNECTED_COORDINATOR_NUM_DELAY
2019-03-14 07:54:56 +08:00
Implemented concurrent read limit in IKeyValueStore interface for Redwood. Added knobs for Redwood page size, concurrent read limit, and page fill factor. Changed commitSubtree() recursion back to use a vector and waitForAll() because it seems to be lower overhead than ActorCollection.
2020-05-11 07:13:22 +08:00
int REDWOOD_DEFAULT_PAGE_SIZE; // Page size for new Redwood files
int REDWOOD_KVSTORE_CONCURRENT_READS; // Max number of simultaneous point or range reads in progress.
Added flowlock around reads done on the write path so that those reads cannot starve reads to support reads done by the storage server.
2020-06-10 08:00:21 +08:00
int REDWOOD_COMMIT_CONCURRENT_READS; // Max number of concurrent reads done to support commit operations
Implemented concurrent read limit in IKeyValueStore interface for Redwood. Added knobs for Redwood page size, concurrent read limit, and page fill factor. Changed commitSubtree() recursion back to use a vector and waitForAll() because it seems to be lower overhead than ActorCollection.
2020-05-11 07:13:22 +08:00
double REDWOOD_PAGE_REBUILD_FILL_FACTOR; // When rebuilding pages, start a new page after this capacity
Lazy delete cycles now run more often and not just when the btree is having mutations actively merged into it. Specifically, it is launched upon recovery, then stopped and relaunched after every commit. This will make better use of I/O in between calls to commit(). Also added knobs to control lazy delete cycle parallelism and work limits.
2020-05-13 17:27:03 +08:00
int REDWOOD_LAZY_CLEAR_BATCH_SIZE_PAGES; // Number of pages to try to pop from the lazy delete queue and process at once
int REDWOOD_LAZY_CLEAR_MIN_PAGES; // Minimum number of pages to free before ending a lazy clear cycle, unless the queue is empty
int REDWOOD_LAZY_CLEAR_MAX_PAGES; // Maximum number of pages to free before ending a lazy clear cycle, unless the queue is empty
Pager remap remover now accumulates a configurable version interval of page updates behind the oldest retained version which are used to coalesce updates to the same original page ID to reduce write amplification for many workloads.
2020-07-20 19:08:33 +08:00
int64_t REDWOOD_REMAP_CLEANUP_WINDOW; // Remap remover lag interval in which to coalesce page writes
double REDWOOD_REMAP_CLEANUP_LAG; // Maximum allowed remap remover lag behind the cleanup window as a multiple of the window size
More refactor of Redwood metrics. Added per-level BTree stats. Added Pager total physical disk reads/writes, and uncacheable read hit/miss counts. Added logging to TraceEvents.
2020-05-16 17:51:57 +08:00
double REDWOOD_LOGGING_INTERVAL;
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3 # Conflicts: # cmake/ConfigureCompiler.cmake # documentation/sphinx/source/downloads.rst # fdbrpc/FlowTransport.actor.cpp # fdbrpc/fdbrpc.vcxproj # fdbserver/DataDistributionQueue.actor.cpp # fdbserver/Knobs.cpp # fdbserver/Knobs.h # fdbserver/LogSystemPeekCursor.actor.cpp # fdbserver/MasterProxyServer.actor.cpp # fdbserver/Status.actor.cpp # fdbserver/storageserver.actor.cpp # flow/flow.vcxproj
2020-07-11 06:06:34 +08:00
Track statistics about server-side request latency on each process, to include min, max, mean, and various percentiles.
2020-07-10 07:39:15 +08:00
// Server request latency measurement
int LATENCY_SAMPLE_SIZE;
double LATENCY_METRICS_LOGGING_INTERVAL;
Implemented concurrent read limit in IKeyValueStore interface for Redwood. Added knobs for Redwood page size, concurrent read limit, and page fill factor. Changed commitSubtree() recursion back to use a vector and waitForAll() because it seems to be lower overhead than ActorCollection.
2020-05-11 07:13:22 +08:00
Atomically update dependent knobs
2020-04-02 04:59:06 +08:00
ServerKnobs();
void initialize(bool randomize = false, ClientKnobs* clientKnobs = NULL, bool isSimulated = false);
Initial repository commit
2017-05-26 04:48:44 +08:00
};
extern ServerKnobs const* SERVER_KNOBS;
#endif