wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #3670 from sfc-gh-ngoyal/experimental 

Documentation on transaction profiling and analyzing.
This commit is contained in:
Markus Pilman 2020-11-17 10:09:37 -07:00 committed by GitHub
parent 748d0944ba 2dbf4efc85
commit 930afc7130
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 51 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
documentation/sphinx/source/client-design.rst
View File

@ -22,6 +22,8 @@ FoundationDB supports language bindings for application development using the or
* :doc:`api-version-upgrade-guide` contains information about upgrading client code to a new API version.
* :doc:`transaction-profiler-analyzer` contains information about enabling transaction profiling and analyzing.
* :doc:`known-limitations` describes both long-term design limitations of FoundationDB and short-term limitations applicable to the current version.
.. toctree::
@ -38,4 +40,5 @@ FoundationDB supports language bindings for application development using the or
api-general
transaction-tagging
known-limitations
transaction-profiler-analyzer
api-version-upgrade-guide

2
documentation/sphinx/source/command-line-interface.rst
View File

@ -313,7 +313,7 @@ client
``profile client <get|set>``
Reads or sets parameters of client transaction sampling. Use ``get`` to list the current parameters, and ``set <RATE|default> <SIZE|default>`` to set them. ``RATE`` is the fraction of transactions to be sampled, and ``SIZE`` is the amount (in bytes) of sampled data to store in the database.
Reads or sets parameters of client transaction sampling. Use ``get`` to list the current parameters, and ``set <RATE|default> <SIZE|default>`` to set them. ``RATE`` is the fraction of transactions to be sampled, and ``SIZE`` is the amount (in bytes) of sampled data to store in the database. For more information, see :doc:`transaction-profiler-analyzer`.
list
^^^^

47
documentation/sphinx/source/transaction-profiler-analyzer.rst Normal file
View File

@ -0,0 +1,47 @@
.. _transaction-profiler-analyzer:
###################################
Transaction profiling and analyzing
###################################
FoundationDB natively implements transaction profiling and analyzing. There are two ways to enable transaction profiling in FoundationDB. One is globally through the database, via ``fdbcli`` command which sets keys in the database and the clients pick it up.
``e.g. fdbcli> profile client set 0.01 100MB`` profiles 1% of transactions and maintains 100MB worth of history in the database.
Second way is through client side knobs ``CSI_SAMPLING_PROBABILITY`` and ``CSI_SIZE_LIMIT`` which have to be set at every client that you want to profile. Enabling transaction profiling through the database setting has higher precedence and overrides any client knob settings.
There are only two inputs for transaction profiling i.e. sampling rate and the size limit.
The transactions are sampled at the specified rate and all the events for that sampled transaction are recorded. Then at 30 second interval, the data for all the sampled transactions during that interval is flushed to the database. The sampled data is written into special key space ``“\xff\x02/fdbClientInfo/ - \xff\x02/fdbClientInfo0”``
The second part of transaction profiling involves deleting old sampled data to restrict the size. Retention is purely based on the input size limit. If the size of all the recorded data exceeds the input limit, then the old ones get deleted. But the limit is a soft limit, you could go over the limit temporarily.
There are many ways that this data can be exposed for analysis. One can imagine building a client that reads the data from the database and streams it to external tools such as Wavefront.
One such tool that’s available as part of open source FDB is a python script called ``transaction_profiling_analyzer.py`` that's available here on `GitHUb <https://github.com/apple/foundationdb/blob/master/contrib/transaction_profiling_analyzer/transaction_profiling_analyzer.py>`_. It reads the sampled data from the database and outputs it in a user friendly format. Currently it’s most useful in identifying hot key-ranges (for both reading and writing).
Prerequisites
=============
* ``python3``
* ``fdb python bindings`` - If you don't have the Python bindings installed, you can append $BUILDDIR/bindings/python to the PYTHONPATH environment variable, then you should be able to import fdb
Additional packages
===================
* ``dateparser`` - for human date parsing
* ``sortedcontainers`` - for estimating key range read/write density
Sample usage
============
* ``$python3 transaction_profiling_analyzer.py --help`` - Shows the help message and exits
* ``python3 transaction_profiling_analyzer.py -C fdb.cluster --start-time "17:00 2020/07/07 PDT" --end-time "17:50 2020/07/07 PDT"`` - Analyzes and prints full information between a start and end time frame
Using filters:
==============
* ``python3 ~/transaction_profiling_analyzer.py -C fdb.cluster --filter-get --start-time "17:00 2020/07/07 PDT" --end-time "17:50 2020/07/07 PDT"`` - Analyzes and prints information about gets between a start and end time frame
* ``python3 ~/transaction_profiling_analyzer.py -C fdb.cluster --filter-get --start-time "17:00 2020/07/07 PDT" --end-time "17:50 2020/07/07 PDT" --top-requests 5`` - Analyzes and prints information about top 5 keys for gets between a start and end time frame