Add documentation for the tag throttling feature
This commit is contained in:
A.J. Beamon
parent
8a36cdd814
commit
530da59d62
|
|
@ -541,6 +541,11 @@ In the present version of FoundationDB, disaster recovery (DR) is implemented vi
|
|||
|
||||
Recovery takes place by reversing the asynchronous replication, so the data in the destination cluster is streamed back to a source cluster. For further information, see the :ref:`overview of backups <backup-introduction>` and the :ref:`fdbdr tool <fdbdr-intro>` that performs asynchronous replication.
|
||||
|
||||
Managing traffic
|
||||
================
|
||||
|
||||
If clients of the database make use of the :doc:`transaction tagging feature <transaction-tagging>`, then the number of transactions allowed to start for different tags can be controlled using the :ref:`throttle command <cli-throttle>` in ``fdbcli``.
|
||||
|
||||
.. _administration-other-administrative-concerns:
|
||||
|
||||
Other administrative concerns
|
||||
|
|
|
|||
|
|
@ -46,6 +46,8 @@ FoundationDB may return the following error codes from API functions. If you nee
|
|||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| future_released | 1102| Future has been released |
|
||||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| tag_throttled | 1213| Transaction tag is being throttled |
|
||||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| platform_error | 1500| Platform error |
|
||||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| large_alloc_failed | 1501| Large block allocation failed |
|
||||
|
|
@ -128,6 +130,12 @@ FoundationDB may return the following error codes from API functions. If you nee
|
|||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| unsupported_operation | 2108| Operation is not supported |
|
||||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| too_many_tags | 2109| Too many tags set on transaction |
|
||||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| tag_too_long | 2110| Tag set on transaction is too long |
|
||||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| too_many_tag_throttles | 2111| Too many tag throttles have been created |
|
||||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| api_version_unset | 2200| API version is not set |
|
||||
+-----------------------------------------------+-----+--------------------------------------------------------------------------------+
|
||||
| api_version_already_set | 2201| API version may be set only once |
|
||||
|
|
|
|||
|
|
@ -18,6 +18,8 @@ FoundationDB supports language bindings for application development using the or
|
|||
|
||||
* :doc:`api-general` contains information on FoundationDB clients applicable across all language bindings.
|
||||
|
||||
* :doc:`transaction-tagging` contains information about using transaction tags in your client code to enable targeted transaction throttling.
|
||||
|
||||
* :doc:`api-version-upgrade-guide` contains information about upgrading client code to a new API version.
|
||||
|
||||
* :doc:`known-limitations` describes both long-term design limitations of FoundationDB and short-term limitations applicable to the current version.
|
||||
|
|
@ -35,5 +37,6 @@ FoundationDB supports language bindings for application development using the or
|
|||
data-modeling
|
||||
client-testing
|
||||
api-general
|
||||
transaction-tagging
|
||||
known-limitations
|
||||
api-version-upgrade-guide
|
||||
|
|
|
|||
|
|
@ -301,6 +301,82 @@ status json
|
|||
|
||||
``status json`` will provide the cluster status in its JSON format. For a detailed description of this format, see :doc:`mr-status`.
|
||||
|
||||
.. _cli-throttle:
|
||||
|
||||
throttle
|
||||
--------
|
||||
|
||||
The throttle command is used to inspect and modify the list of throttled transaction tags in the cluster. For more information, see :doc:`transaction-tagging`. The throttle command has the following subcommands:
|
||||
|
||||
on
|
||||
^^
|
||||
|
||||
``throttle on tag <TAG> [RATE] [DURATION] [PRIORITY]``
|
||||
|
||||
Enables throttling for the specified transaction tag.
|
||||
|
||||
``TAG`` - the tag being throttled. This argument is required.
|
||||
|
||||
``RATE`` - the number of transactions that may be started per second. Defaults to 0.
|
||||
|
||||
``DURATION`` - the duration that the throttle should remain in effect, which must include a time suffix (``s`` - seconds, ``m`` - minutes, ``h`` - hours, ``d`` - days). Defaults to ``1h``.
|
||||
|
||||
``PRIORITY`` - the maximum priority that the throttle will apply to. Choices are ``default``, ``batch``, and ``immediate``. Defaults to ``default``.
|
||||
|
||||
off
|
||||
^^^
|
||||
|
||||
``throttle off [all|auto|manual] [tag <TAG>] [PRIORITY]``
|
||||
|
||||
Disables throttling for all transaction tag throttles that match the specified filtering criteria. At least one filter must be specified, and filters can be given in any order.
|
||||
|
||||
``all`` - affects all throttles (auto and manual).
|
||||
|
||||
``auto`` - only throttles automatically created by the cluster will be affected.
|
||||
|
||||
``manual`` - only throttles created manually will be affected (this is the default).
|
||||
|
||||
``tag`` - only the specified tag will be affected.
|
||||
|
||||
``PRIORITY`` - the priority of the throttle to disable. Choices are ``default``, ``batch``, and ``immediate``. Defaults to ``default``.
|
||||
|
||||
For example, to disable all throttles, run::
|
||||
|
||||
> throttle off all
|
||||
|
||||
To disable all manually created batch priority throttles, run::
|
||||
|
||||
> throttle off batch
|
||||
|
||||
To disable auto throttles at batch priority on the tag ``foo``, run::
|
||||
|
||||
> throttle off auto tag foo batch
|
||||
|
||||
enable
|
||||
^^^^^^
|
||||
|
||||
``throttle enable auto``
|
||||
|
||||
Enables cluster auto-throttling for busy transaction tags.
|
||||
|
||||
disable
|
||||
^^^^^^^
|
||||
|
||||
``throttle disable auto``
|
||||
|
||||
Disables cluster auto-throttling for busy transaction tags. This does not disable any currently active throttles. To do so, run the following command after disabling auto-throttling::
|
||||
|
||||
> throttle off auto
|
||||
|
||||
list
|
||||
^^^^
|
||||
|
||||
``throttle list [LIMIT]``
|
||||
|
||||
Prints a list of currently active transaction tag throttles.
|
||||
|
||||
``LIMIT`` - The number of throttles to print. Defaults to 100.
|
||||
|
||||
unlock
|
||||
------
|
||||
|
||||
|
|
|
|||
|
|
@ -24,6 +24,8 @@ Ready to operate an externally accessible FoundationDB cluster? You'll find what
|
|||
|
||||
* :doc:`platforms` describes issues on particular platforms that affect the operation of FoundationDB.
|
||||
|
||||
* :doc:`transaction-tagging` gives an overview of transaction tagging, including details about throttling particular tags.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
:titlesonly:
|
||||
|
|
@ -38,3 +40,4 @@ Ready to operate an externally accessible FoundationDB cluster? You'll find what
|
|||
backups
|
||||
disk-snapshot-backup
|
||||
platforms
|
||||
transaction-tagging
|
||||
|
|
|
|||
|
|
@ -0,0 +1,74 @@
|
|||
.. _transaction-tagging:
|
||||
|
||||
###################
|
||||
Transaction Tagging
|
||||
###################
|
||||
|
||||
FoundationDB provides the ability to add arbitrary byte-string tags to transactions. The cluster can be configured to limit the rate of transactions with certain tags, either automatically in response to tags that are very busy, or manually using the :ref:`throttle command <cli-throttle>` in ``fdbcli``.
|
||||
|
||||
This document describes how to configure and observe the behavior of transaction tagging and throttling.
|
||||
|
||||
Adding tags to transactions
|
||||
===========================
|
||||
|
||||
Tags are added to transaction by using transaction options. Each transaction can include up to five tags, and each tag must not exceed 16 characters. There are two options that can be used to add tags:
|
||||
|
||||
* ``TAG`` - Adds a tag to the transaction. This tag will not be used for auto-throttling and is not included with read requests. Tags set in this way can only be throttled manually.
|
||||
* ``AUTO_THROTTLE_TAG`` - Adds a tag to the transaction that can be both automatically and manually throttled. To support busy tag detection, these tags may be sent as part of read requests.
|
||||
|
||||
See the documentation for your particular language binding for details about setting this option.
|
||||
|
||||
.. note:: If setting hierarchical tags, it is recommended that you not use auto-throttle tags at multiple levels of the hierarchy. Otherwise, the cluster will favor throttling those tags set at higher levels, as they will include more transactions.
|
||||
|
||||
.. note:: Tags must be included as part of all get read version requests, and a sample of read requests will include auto-throttled tags. Additionally, each tag imposes additional costs with those requests. It is therefore recommended that you not use excessive numbers or lengths of transaction tags.
|
||||
|
||||
Tag throttling overview
|
||||
=======================
|
||||
|
||||
The cluster has the ability to enforce a transaction rate for particular tags. This rate can be set manually via the :ref:`throttle command <cli-throttle>` in ``fdbcli`` or automatically when a storage server gets busy and has a lot of traffic from a single tag.
|
||||
|
||||
Tag throttles have the following attributes:
|
||||
|
||||
* tag - the tag being throttled.
|
||||
* rate - the desired number of transactions to start per second for this tag.
|
||||
* priority - the maximum priority to throttle. All lower priorities will also have this throttle applied.
|
||||
* auto/manual - whether this tag was throttled automatically or manually.
|
||||
* expiration - the time that the throttle will expire.
|
||||
|
||||
It is possible for multiple throttles to apply to the same transaction. For example, the transaction could use a tag that is both automatically and manually throttled, there may be throttles set at multiple priorities greater than or equal to the priority of the transaction, or it may set multiple tags. In these cases, the lowest limit of any throttle affecting the transaction will be used.
|
||||
|
||||
The limit set by the throttles is not a hard limit, meaning that it may be possible for clients to start more transactions in aggregate than allowed by the limit during a given window. The cluster should react to this condition by reducing the number of transactions that can be started on each client going forward.
|
||||
|
||||
``tag_throttled`` error
|
||||
=======================
|
||||
|
||||
When a transaction tag is throttled, this information will be communicated to the client as part of the get read version request, which will then fail with a ``tag_throttled`` error. This error is retryable by ``on_error`` and the transaction retry loops provided in the language bindings. Subsequent requests from the same client that exceed the limit will be throttled without sending a request to the cluster at all, except for periodic requests intended to poll for updates to the throttles.
|
||||
|
||||
|
||||
Automatic transaction tag throttling
|
||||
====================================
|
||||
|
||||
When using the ``AUTO_THROTTLE_TAG`` transaction option, the cluster will monitor read activity for the chosen tags and may choose to reduce a tag's transaction rate limit if a storage server gets busy enough and has a sufficient portion of its read traffic coming from that one tag.
|
||||
|
||||
When a tag is auto-throttled, the default priority transaction rate will be decreased to reduce the percentage of traffic attributable to that tag to a reasonable amount of total traffic on the affected storage server(s), and batch priority transactions for that tag will be stopped completely.
|
||||
|
||||
Auto-throttles are created with a duration of a few minutes, at the end of which the cluster will try to gradually lift the limit. If the cluster detects that it needs to continue throttling the tag, then the duration of the throttle will be extended.
|
||||
|
||||
Manual transaction tag throttling
|
||||
=================================
|
||||
|
||||
In addition to automatically throttling transaction tags, cluster operators have the ability to manually throttle and unthrottle tags using the :ref:`throttle command <cli-throttle>` in ``fdbcli``. Manual tag throttles are created in addition to automatic ones, meaning that the same tag can be throttled internally by the cluster and manually by the operator, in which case the lower limit applies. All transaction tags can be manually throttled.
|
||||
|
||||
There can be at most 40 tags throttled manually at one time.
|
||||
|
||||
Enabling and disabling
|
||||
----------------------
|
||||
|
||||
By default, automatic transaction tag throttling is enabled for any tags that are set using the ``AUTO_THROTTLE_TAG`` option. This feature can be enabled and disabled using ``fdbcli``::
|
||||
|
||||
> throttle <enable|disable> auto
|
||||
|
||||
Viewing active throttles
|
||||
========================
|
||||
|
||||
A list of active throttles and their parameters can be viewed using the :ref:`throttle list <cli-throttle>` command in ``fdbcli``.
|
||||
|
|
@ -69,7 +69,7 @@ ERROR( serialization_failed, 1044, "Failed to deserialize an object" )
|
|||
ERROR( connection_unreferenced, 1048, "No peer references for connection" )
|
||||
ERROR( connection_idle, 1049, "Connection closed after idle timeout" )
|
||||
ERROR( disk_adapter_reset, 1050, "The disk queue adpater reset" )
|
||||
ERROR(batch_transaction_throttled, 1051, "Batch GRV request rate limit exceeded")
|
||||
ERROR( batch_transaction_throttled, 1051, "Batch GRV request rate limit exceeded")
|
||||
|
||||
ERROR( broken_promise, 1100, "Broken promise" )
|
||||
ERROR( operation_cancelled, 1101, "Asynchronous operation cancelled" )
|
||||
|
|
|
|||
Loading…
Reference in New Issue