wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/flow/Knobs.h

224 lines
6.4 KiB
C
Raw Normal View History

Initial repository commit
2017-05-26 04:48:44 +08:00
/*
* Knobs.h
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2013-2018 Apple Inc. and the FoundationDB project authors
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* http://www.apache.org/licenses/LICENSE-2.0
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef FLOW_KNOBS_H
#define FLOW_KNOBS_H
#pragma once
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "flow/Platform.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include <map>
#include <string>
#include <stdint.h>
class Knobs {
public:
bool setKnob( std::string const& name, std::string const& value ); // Returns true if the knob name is known, false if it is unknown
void trace();
protected:
void initKnob( double& knob, double value, std::string const& name );
void initKnob( int64_t& knob, int64_t value, std::string const& name );
void initKnob( int& knob, int value, std::string const& name );
Add a string knob to hold the Log directory, and write profiles to it. This is the combination of two small changes. 1. Add support for a string knob type. 2. Change profiles to be written to the log directory instead of the working directory. We have three options of where to write files: the working directory, the data directory, and the log directory. The working directory may be set to a non-writable location, and likely contains the fdb binaries. Allowing these files to be overwritten would likely not be a wise idea. The data directory hosts our sqlite b-trees. It would also be very unfortunate if these were ever overwritten by an unfortunate profile name. The log directory contains logs. Out of the three, these matter the least if they disappear or become corrupted. Thus, we write to the log directory.
2017-10-13 08:49:41 +08:00
void initKnob( std::string& knob, const std::string& value, const std::string& name );
bool knobs can now be set with the words “true” or “false” instead of just a number
2019-07-31 09:21:24 +08:00
void initKnob( bool& knob, bool value, std::string const& name );
Initial repository commit
2017-05-26 04:48:44 +08:00
std::map<std::string, double*> double_knobs;
std::map<std::string, int64_t*> int64_knobs;
std::map<std::string, int*> int_knobs;
Add a string knob to hold the Log directory, and write profiles to it. This is the combination of two small changes. 1. Add support for a string knob type. 2. Change profiles to be written to the log directory instead of the working directory. We have three options of where to write files: the working directory, the data directory, and the log directory. The working directory may be set to a non-writable location, and likely contains the fdb binaries. Allowing these files to be overwritten would likely not be a wise idea. The data directory hosts our sqlite b-trees. It would also be very unfortunate if these were ever overwritten by an unfortunate profile name. The log directory contains logs. Out of the three, these matter the least if they disappear or become corrupted. Thus, we write to the log directory.
2017-10-13 08:49:41 +08:00
std::map<std::string, std::string*> string_knobs;
bool knobs can now be set with the words “true” or “false” instead of just a number
2019-07-31 09:21:24 +08:00
std::map<std::string, bool*> bool_knobs;
Initial repository commit
2017-05-26 04:48:44 +08:00
};
class FlowKnobs : public Knobs {
public:
int AUTOMATIC_TRACE_DUMP;
double PREVENT_FAST_SPIN_DELAY;
double CACHE_REFRESH_INTERVAL_WHEN_ALL_ALTERNATIVES_FAILED;
double DELAY_JITTER_OFFSET;
double DELAY_JITTER_RANGE;
double BUSY_WAIT_THRESHOLD;
double CLIENT_REQUEST_INTERVAL;
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-02 05:04:32 +08:00
double SERVER_REQUEST_INTERVAL;
Initial repository commit
2017-05-26 04:48:44 +08:00
int DISABLE_ASSERTS;
double QUEUE_MODEL_SMOOTHING_AMOUNT;
int RANDOMSEED_RETRY_LIMIT;
avoid relying on 128<<10 in knobs.cpp
2019-03-19 03:40:15 +08:00
double FAST_ALLOC_LOGGING_BYTES;
added random logging for huge arenas
2019-03-21 02:20:47 +08:00
double HUGE_ARENA_LOGGING_BYTES;
prevent trace event spam by combining huge arena samples
2019-03-31 04:36:13 +08:00
double HUGE_ARENA_LOGGING_INTERVAL;
Initial repository commit
2017-05-26 04:48:44 +08:00
//slow task profiling
double SLOWTASK_PROFILING_INTERVAL;
double SLOWTASK_PROFILING_MAX_LOG_INTERVAL;
double SLOWTASK_PROFILING_LOG_BACKOFF;
//connectionMonitor
double CONNECTION_MONITOR_LOOP_TIME;
double CONNECTION_MONITOR_TIMEOUT;
fdbrpc: Reduced connection monitoring from clients This patch does two changes to connection monitoring: 1. Connection monitoring at client side will check if the connection has been stayed idle for some time. If connection is unused for a while, we close the connection. There is some weirdness involved here as ping messages are by themselves are connection traffic. We get over this by making it two-phase process, first being checking idle reliable traffic, followed by disabling pings and then checking for idle unreliable traffic. 2. Connection monitoring of clients from server will no longer send pings to clients. Instead, it keep monitor the received bytes and close after certain period of inactivity.
2019-06-28 15:39:51 +08:00
double CONNECTION_MONITOR_IDLE_TIMEOUT;
fdbrpc: Instead of tracking last sent data, track last sent non-ping data * This will allow client to continue monitoring peer connections while connection stays open, so that there is no period of "uncertainity" without previous no-monitoring approach. * Use multiplier for incoming connection idle timeout * Update idle connection timeout values and leaked connection timeout in simulator.
2019-06-29 04:43:22 +08:00
double CONNECTION_MONITOR_INCOMING_IDLE_MULTIPLIER;
fdbrpc: ConnectionMonitor should close unreferenced after delay Potentially for cases, where it goes up to 1 immediately.
2019-07-09 10:13:53 +08:00
double CONNECTION_MONITOR_UNREFERENCED_CLOSE_DELAY;
Initial repository commit
2017-05-26 04:48:44 +08:00
//FlowTransport
double CONNECTION_REJECTED_MESSAGE_DELAY;
double CONNECTION_ID_TIMEOUT;
double CONNECTION_CLEANUP_DELAY;
double INITIAL_RECONNECTION_TIME;
double MAX_RECONNECTION_TIME;
double RECONNECTION_TIME_GROWTH_RATE;
double RECONNECTION_RESET_TIME;
Make sure we still accept some connections even if we are CPU bound by high priority work
2018-08-11 08:47:21 +08:00
double CONNECTION_ACCEPT_DELAY;
Remove --object-serializer flag from executables
2019-08-07 00:25:40 +08:00
int USE_OBJECT_SERIALIZER;
Initial repository commit
2017-05-26 04:48:44 +08:00
Merge pull request #628 from alexmiller-apple/reloadcertificates Reload certificates if changed. This is a cherry-pick of #628 back to release-6.0
2018-07-27 08:21:23 +08:00
int TLS_CERT_REFRESH_DELAY_SECONDS;
Establishing TLS connection through the handshake process is expensive and the fdbserver process can get easily saturated with doing repeated TLS handshakes with only a few hundreds of clients have bad certificate. Hence throttle the number of handshakes done on the server per client ip if it has a bad certificate.
2020-01-11 08:19:41 +08:00
double TLS_SERVER_CONNECTION_THROTTLE_TIMEOUT;
throttle client TLS errors longer than server errors so that when both happen simultaneously the server throttling will be disabled when the client makes its next attempt
2020-01-13 14:12:18 +08:00
double TLS_CLIENT_CONNECTION_THROTTLE_TIMEOUT;
added knobs
2020-01-22 10:58:34 +08:00
int TLS_SERVER_CONNECTION_THROTTLE_ATTEMPTS;
int TLS_CLIENT_CONNECTION_THROTTLE_ATTEMPTS;
Merge pull request #628 from alexmiller-apple/reloadcertificates Reload certificates if changed. This is a cherry-pick of #628 back to release-6.0
2018-07-27 08:21:23 +08:00
Add some tuning knobs to networktestclient; also, measure latency directly.
2020-02-11 07:03:35 +08:00
int NETWORK_TEST_CLIENT_COUNT;
Make the networktest payload size a knob so that it can be changed.
2019-08-22 09:37:50 +08:00
int NETWORK_TEST_REPLY_SIZE;
Add some tuning knobs to networktestclient; also, measure latency directly.
2020-02-11 07:03:35 +08:00
int NETWORK_TEST_REQUEST_COUNT;
int NETWORK_TEST_REQUEST_SIZE;
bool NETWORK_TEST_SCRIPT_MODE;
Establishing TLS connection through the handshake process is expensive and the fdbserver process can get easily saturated with doing repeated TLS handshakes with only a few hundreds of clients have bad certificate. Hence throttle the number of handshakes done on the server per client ip if it has a bad certificate.
2020-01-11 08:19:41 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
//AsyncFileCached
int64_t PAGE_CACHE_4K;
int64_t PAGE_CACHE_64K;
int64_t SIM_PAGE_CACHE_4K;
int64_t SIM_PAGE_CACHE_64K;
int64_t BUGGIFY_SIM_PAGE_CACHE_4K;
int64_t BUGGIFY_SIM_PAGE_CACHE_64K;
AsyncFileCached: switch to string for eviction policy knob
2019-05-03 20:04:43 +08:00
std::string CACHE_EVICTION_POLICY; // for now, "random", "lru", are supported
Initial repository commit
2017-05-26 04:48:44 +08:00
int MAX_EVICT_ATTEMPTS;
Port truncate optimization from Snowflake PR in order to make quick changes for a patch release.
2018-10-06 06:05:26 +08:00
double PAGE_CACHE_TRUNCATE_LOOKUP_FRACTION;
A process will mark itself as degraded if it continually disconnects from a different process which the failure monitor thinks is healthy
2019-04-05 05:11:12 +08:00
double TOO_MANY_CONNECTIONS_CLOSED_RESET_DELAY;
int TOO_MANY_CONNECTIONS_CLOSED_TIMEOUT;
fdbrpc: Add warning when peer is unavailable for long time
2020-01-09 05:31:58 +08:00
int PEER_UNAVAILABLE_FOR_LONG_TIME_TIMEOUT;
Initial repository commit
2017-05-26 04:48:44 +08:00
Introduced knob to set eio parallelism
2019-10-16 01:22:18 +08:00
//AsyncFileEIO
int EIO_MAX_PARALLELISM;
Don't use O_DIRECT in EIO by default
2019-10-25 02:39:55 +08:00
int EIO_USE_ODIRECT;
Introduced knob to set eio parallelism
2019-10-16 01:22:18 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
//AsyncFileKAIO
int MAX_OUTSTANDING;
int MIN_SUBMIT;
Merge branch 'release-4.6' into release-5.0 # Conflicts: # fdbrpc/AsyncFileKAIO.actor.h # fdbrpc/Net2FileSystem.cpp # fdbrpc/sim2.actor.cpp
2017-06-16 17:14:19 +08:00
int PAGE_WRITE_CHECKSUM_HISTORY;
io: Add DISABLE_POSIX_KERNEL_AIO knob to use EIO instead of Kernel AIO - Some Linux filesystems don't support O_DIRECT which is required by Kernel AIO to function properly. Instead of using O_SYNC, EIO is much better options in terms of performance penalty. - Some systems may not support AIO at all. Eg. Windows Subsystem for Linux. FIXES #842 RELATED #274
2019-03-13 05:04:18 +08:00
int DISABLE_POSIX_KERNEL_AIO;
Initial repository commit
2017-05-26 04:48:44 +08:00
//AsyncFileNonDurable
double MAX_PRIOR_MODIFICATION_DELAY;
//GenericActors
double BUGGIFY_FLOW_LOCK_RELEASE_DELAY;
//IAsyncFile
int64_t INCREMENTAL_DELETE_TRUNCATE_AMOUNT;
double INCREMENTAL_DELETE_INTERVAL;
//Net2
double MIN_COALESCE_DELAY;
double MAX_COALESCE_DELAY;
double SLOW_LOOP_CUTOFF;
double SLOW_LOOP_SAMPLING_RATE;
int64_t TSC_YIELD_TIME;
int64_t REACTOR_FLAGS;
//Network
int64_t PACKET_LIMIT;
int64_t PACKET_WARNING; // 2MB packet warning quietly allows for 1MB system messages
double TIME_OFFSET_LOGGING_INTERVAL;
Add knobs for various packet buffer sizes
2019-06-12 08:52:25 +08:00
int MAX_PACKET_SEND_BYTES;
int MIN_PACKET_BUFFER_BYTES;
Allow next packet size to be read when receiving data For large packet, allocate sizeof(uint32_t) more bytes for next packet size. Also add knob MIN_PACKET_BUFFER_FREE_BYTES, which is used to trigger allocation of a new arena when free bytes are lower than this threshold.
2019-06-26 00:52:05 +08:00
int MIN_PACKET_BUFFER_FREE_BYTES;
Add FLOW_TCP_NODELAY and FLOW_TCP_QUICKACK (Linux-only) knobs
2019-08-24 02:07:46 +08:00
int FLOW_TCP_NODELAY;
int FLOW_TCP_QUICKACK;
Initial repository commit
2017-05-26 04:48:44 +08:00
//Sim2
//FIMXE: more parameters could be factored out
double MIN_OPEN_TIME;
double MAX_OPEN_TIME;
int64_t SIM_DISK_IOPS;
int64_t SIM_DISK_BANDWIDTH;
double MIN_NETWORK_LATENCY;
double FAST_NETWORK_LATENCY;
double SLOW_NETWORK_LATENCY;
double MAX_CLOGGING_LATENCY;
double MAX_BUGGIFIED_DELAY;
simulation can sometimes randomly hang or throw connection_failed, instead of always doing one or the other
2019-11-22 08:24:18 +08:00
int SIM_CONNECT_ERROR_MODE;
Initial repository commit
2017-05-26 04:48:44 +08:00
//Tracefiles
int ZERO_LENGTH_FILE_PAD;
double TRACE_FLUSH_INTERVAL;
double TRACE_RETRY_OPEN_INTERVAL;
int MIN_TRACE_SEVERITY;
int MAX_TRACE_SUPPRESSIONS;
Rework the formatter a bit and extract out the writing logic so that it could be pluggable.
2018-06-01 04:27:35 +08:00
int TRACE_SYNC_ENABLED;
Throttle Spammy logs. Three knobs are added. Trace Events are sampled and cached with an expiration set. Every TraceEvent above SevDebug is checked against this cache to see if it exceeded a set threshold. If yes, then throttle the TraceEvent. If a TraceEvent is throttled, a warning msg is logged.
2017-10-03 09:43:11 +08:00
int TRACE_EVENT_METRIC_UNITS_PER_SAMPLE;
Address review comments
2018-06-27 05:37:21 +08:00
int TRACE_EVENT_THROTTLER_SAMPLE_EXPIRY;
Throttle Spammy logs. Three knobs are added. Trace Events are sampled and cached with an expiration set. Every TraceEvent above SevDebug is checked against this cache to see if it exceeded a set threshold. If yes, then throttle the TraceEvent. If a TraceEvent is throttled, a warning msg is logged.
2017-10-03 09:43:11 +08:00
int TRACE_EVENT_THROTTLER_MSG_LIMIT;
Make trace event field lengths (and total event sizes) default knobified and configurable. Add a transaction option to control the field length of transaction debug logging. Make the program start command line field less likely to be truncated.
2019-07-13 07:12:35 +08:00
int MAX_TRACE_FIELD_LENGTH;
int MAX_TRACE_EVENT_LENGTH;
Initial repository commit
2017-05-26 04:48:44 +08:00
//TDMetrics
int64_t MAX_METRIC_SIZE;
int64_t MAX_METRIC_LEVEL;
Add a string knob to hold the Log directory, and write profiles to it. This is the combination of two small changes. 1. Add support for a string knob type. 2. Change profiles to be written to the log directory instead of the working directory. We have three options of where to write files: the working directory, the data directory, and the log directory. The working directory may be set to a non-writable location, and likely contains the fdb binaries. Allowing these files to be overwritten would likely not be a wise idea. The data directory hosts our sqlite b-trees. It would also be very unfortunate if these were ever overwritten by an unfortunate profile name. The log directory contains logs. Out of the three, these matter the least if they disappear or become corrupted. Thus, we write to the log directory.
2017-10-13 08:49:41 +08:00
double METRIC_LEVEL_DIVISOR;
Initial repository commit
2017-05-26 04:48:44 +08:00
int METRIC_LIMIT_START_QUEUE_SIZE;
int METRIC_LIMIT_RESPONSE_FACTOR;
int MAX_METRICS;
//Load Balancing
split the single knob into two for finer-grained control
2019-07-17 03:46:02 +08:00
int LOAD_BALANCE_ZONE_ID_LOCALITY_ENABLED;
int LOAD_BALANCE_DC_ID_LOCALITY_ENABLED;
Initial repository commit
2017-05-26 04:48:44 +08:00
double LOAD_BALANCE_MAX_BACKOFF;
double LOAD_BALANCE_START_BACKOFF;
double LOAD_BALANCE_BACKOFF_RATE;
int64_t MAX_LAGGING_REQUESTS_OUTSTANDING;
double INSTANT_SECOND_REQUEST_MULTIPLIER;
double BASE_SECOND_REQUEST_TIME;
double SECOND_REQUEST_MULTIPLIER_GROWTH;
double SECOND_REQUEST_MULTIPLIER_DECAY;
double SECOND_REQUEST_BUDGET_GROWTH;
double SECOND_REQUEST_MAX_BUDGET;
double ALTERNATIVES_FAILURE_RESET_TIME;
double ALTERNATIVES_FAILURE_MIN_DELAY;
double ALTERNATIVES_FAILURE_DELAY_RATIO;
The delay for all_alteratives_failed can scale all the way up to 30.0 at a much slow time ratio
2019-08-10 03:35:19 +08:00
double ALTERNATIVES_FAILURE_MAX_DELAY;
double ALTERNATIVES_FAILURE_SLOW_DELAY_RATIO;
double ALTERNATIVES_FAILURE_SLOW_MAX_DELAY;
allow one requests every second to skip there all_alteratives_failed delay, because if a client has a timeout longer than the delay we will never invalidate the key servers cache
2019-08-10 04:03:40 +08:00
double ALTERNATIVES_FAILURE_SKIP_DELAY;
made a large number of fixes to make fearless DR correctness clean.
2017-10-20 06:36:32 +08:00
double FUTURE_VERSION_INITIAL_BACKOFF;
double FUTURE_VERSION_MAX_BACKOFF;
double FUTURE_VERSION_BACKOFF_GROWTH;
load balance will send reads to remote servers if more than one alternative is failed or overloaded
2019-08-20 04:59:49 +08:00
int LOAD_BALANCE_MAX_BAD_OPTIONS;
bool LOAD_BALANCE_PENALTY_IS_BAD;
Initial repository commit
2017-05-26 04:48:44 +08:00
FlowKnobs(bool randomize = false, bool isSimulated = false);
};
extern FlowKnobs const* FLOW_KNOBS;
#endif