Merge pull request #2452 from ajbeamon/document-network-thread-and-client-logging

Add some documentation for the client network thread and client logging.

documentation/sphinx/source/administration.rst

@@ -508,7 +508,7 @@ If ``kill_on_configuration_change`` parameter is unset or set to ``true`` in fou
 Managing trace files
 ====================
 
- By default, trace files are output to:
+By default, trace files are output to:
 
 * ``/var/log/foundationdb/`` on Linux
 * ``/usr/local/foundationdb/logs/`` on macOS

documentation/sphinx/source/api-general.rst

@@ -74,7 +74,7 @@ Multi-version client
 
 The FoundationDB client library supports connecting to clusters with server processes running at a different version than the client. To do so, it must have access to a version of the client compatible with the cluster, which it loads and then proxies API calls through.
 
-To make use of the multi-version client capabilities, you must set at least one of two network options (``EXTERNAL_CLIENT_LIBRARY`` and/or ``EXTERNAL_CLIENT_DIRECTORY``) that specify the location of these additional client libraries. The client library will start a new network thread for each external client and attempt to make connections to the cluster over all of them (as well as the local library). When making API calls, the library will use the version that is able to communicate with the cluster (choosing an arbitrary one if there are multiple compatible clients), and it will also automatically switch to the appropriate client should the cluster's protocol version change.
+To make use of the multi-version client capabilities, you must set at least one of two network options (``EXTERNAL_CLIENT_LIBRARY`` and/or ``EXTERNAL_CLIENT_DIRECTORY``) that specify the location of these additional client libraries. The client library will start a new :ref:`network thread <client-network-thread>` for each external client and attempt to make connections to the cluster over all of them (as well as the local library). When making API calls, the library will use the version that is able to communicate with the cluster (choosing an arbitrary one if there are multiple compatible clients), and it will also automatically switch to the appropriate client should the cluster's protocol version change. Each loaded library will generate its own :ref:`trace files <client-trace-logging-multi-version-client>`, if enabled.
 
 The multi-version client API adds a new ``cluster_version_changed`` error that is used to cancel operations performed by the client when it switches to using a different version of the client library. This error is a retryable one, meaning that all standard retry loops will automatically rerun a transaction that fails with ``cluster_version_changed``. 
 
@@ -104,3 +104,36 @@ If you want to set the same option multiple times (e.g. to add multiple client l
 	FDB_NETWORK_OPTION_EXTERNAL_CLIENT_DIRECTORY = /path/to/dir1:/path/to/dir2:...
 
 Network options specified using environment variables are set at the end of the call to set the API version. They will be applied in the order of the corresponding option codes assigned to the options. If there is an error reading the appropriate environment variables or if the option cannot be set with the specified value, then the call to set the API version may return an error. In that case, you may attempt to set the API version again, but you must do so with the same version as the attempt that failed.
+
+.. _client-network-thread:
+
+Client network thread
+=====================
+
+FoundationDB clients start a thread that is used to serialize operations and communicate with the cluster. This thread is commonly referred to as the network thread, and most operations performed by a client (such as reads, writes, and commits) will take place on this thread. It is important that a client application does not block on this thread, such as by issuing a blocking call in a callback from a FoundationDB operation. Some client language bindings (e.g. Java) will protect you from this risk by notifying the application that an operation has completed from a different thread automatically.
+
+A client process can have *only one* network thread for the entire lifetime of that process. While it's possible to stop the network thread, it is not possible to restart it.
+
+.. _client-network-thread-performance:
+
+Performance
+-----------
+
+Because it can have only a single network thread, a client process may become limited in its throughput by the amount of work that can be performed on that thread. A client with a saturated network thread may begin to experience increased latencies while it struggles to keep up. If a client application needs to support higher throughput than a single network thread can provide, then more network threads can be started by running additional client processes.
+
+If you suspect that a client process's workload may be saturating the network thread, this can be confirmed by checking whether the network thread is running with high CPU usage. In the :ref:`client trace logs <client-trace-logging>`, the ``ProcessMetrics`` trace event has a field for ``MainThreadCPUSeconds`` that indicates the number of seconds out of ``Elapsed`` that the network thread was busy. You can also attempt to identify a busy thread from any tool that reports the CPU activity of threads in your process.
+
+.. _client-trace-logging:
+
+Client trace logging
+====================
+
+By default, clients do not generate trace logs. In order to enable them, the ``TRACE_ENABLE`` network option must be set on your client. See the documentation for your language binding for instructions on how to do this.
+
+.. _client-trace-logging-multi-version-client:
+
+Multi-version client API
+------------------------
+
+When trace logs are enabled while using the :ref:`multi-version client API <multi-version-client-api>`, separate files will be generated for the primary client library and each of the loaded external clients. The ``ClientStart`` event can be used to identify the version of the client that generated a particular log file. Most of the relevant logs will be found in the version that matches the cluster's version, but for some types of issues with startup or upgrades (particularly those related to the multi-version client itself), relevant logs can be found in the logs for the primary client as well.
+

documentation/sphinx/source/developer-guide.rst

@@ -430,7 +430,7 @@ By default, FoundationDB supports read-your-writes, meaning that reads reflect t
 
 Another approach to programming with futures in FoundationDB is to set a callback function to be invoked asynchronously when the future is ready.
 
-.. note:: Be very careful when mixing callbacks with explicit or implicit blocking. Blocking in a callback on a non-ready future will cause a deadlock. Blocking on anything else or performing CPU intensive tasks will block the FoundationDB client thread and therefore all database access from that client.
+.. note:: Be very careful when mixing callbacks with explicit or implicit blocking. Blocking in a callback on a non-ready future will cause a deadlock. Blocking on anything else or performing CPU intensive tasks will block the FoundationDB :ref:`client thread <client-network-thread>` and therefore all database access from that client.
 
 For further details, see the :doc:`API reference <api-reference>` documentation for your language.