From b51503d4201025605ae4bba6e4f25287756e820b Mon Sep 17 00:00:00 2001 From: Balachandar Namasivayam Date: Thu, 15 Mar 2018 18:08:58 -0700 Subject: [PATCH 01/26] Update configuration.rst with locality info --- documentation/sphinx/source/configuration.rst | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/documentation/sphinx/source/configuration.rst b/documentation/sphinx/source/configuration.rst index f5d289d603..7e0cb281eb 100644 --- a/documentation/sphinx/source/configuration.rst +++ b/documentation/sphinx/source/configuration.rst @@ -234,15 +234,19 @@ Contains settings applicable to all processes (e.g. fdbserver, backup_agent). Th logdir = /var/log/foundationdb # logsize = 10MiB # maxlogssize = 100MiB - # machine_id = - # datacenter_id = # class = # memory = 8GiB # storage_memory = 1GiB + # locality_machineid = + # locality_zoneid = + # locality_data_hall = + # locality_dcid = + # io_trust_seconds = 20 Contains default parameters for all fdbserver processes on this machine. These same options can be overridden for individual processes in their respective ``[fdbserver.]`` sections. In this section, the ID of the individual fdbserver can be substituted by using the ``$ID`` variable in the value. For example, ``public_address = auto:$ID`` makes each fdbserver listen on a port equal to its ID. .. note:: |multiplicative-suffixes| +.. note:: In general locality id's are used to specify the location of processes which in turn is used to determine fault and replication domains. * ``command``: The location of the ``fdbserver`` binary. * ``public_address``: The publicly visible IP:Port of the process. If ``auto``, the address will be the one used to communicate with the coordination servers. @@ -251,11 +255,14 @@ Contains default parameters for all fdbserver processes on this machine. These s * ``logdir``: A writable directory (by root or by the user set in the [fdbmonitor] section) where FoundationDB will store log files. * ``logsize``: Roll over to a new log file after the current log file reaches the specified size. The default value is 10MiB. * ``maxlogssize``: Delete the oldest log file when the total size of all log files exceeds the specified size. If set to 0B, old log files will not be deleted. The default value is 100MiB. -* ``machine_id``: Machine identifier key. Processes that share a key are considered non-unique for the purposes of data replication. By default, processes on a machine determine a unique key to share. This does not generally need to be set. The ID can be up to 16 hexadecimal digits. -* ``datacenter_id``: Data center identifier key. All processes physically located in a data center should share the id. If unset, defaults to a special "default" data center. If you are depending on data center based replication this must be set on all processes. The ID can be up to 16 hexadecimal digits. * ``class``: Machine class specifying the roles that will be taken in the cluster. Valid options are ``storage``, ``transaction``, ``resolution``. See :ref:`configuration-large-cluster-performance` for machine class recommendations in large clusters. * ``memory``: Maximum memory used by the process. The default value is 8GiB. When specified without a unit, MiB is assumed. This parameter does not change the memory allocation of the program. Rather, it sets a hard limit beyond which the process will kill itself and be restarted. The default value of 8GiB is double the intended memory usage in the default configuration (providing an emergency buffer to deal with memory leaks or similar problems). It is *not* recommended to decrease the value of this parameter below its default value. It may be *increased* if you wish to allocate a very large amount of storage engine memory or cache. In particular, when the ``storage_memory`` parameter is increased, the ``memory`` parameter should be increased by an equal amount. * ``storage_memory``: Maximum memory used for data storage. This paramenter is used *only* with memory storage engine, not the ssd storage engine. The default value is 1GiB. When specified without a unit, MB is assumed. Clusters will be restricted to using this amount of memory per process for purposes of data storage. Memory overhead associated with storing the data is counted against this total. If you increase the ``storage_memory``, you should also increase the ``memory`` parameter by the same amount. +* ``locality_machineid``: Machine identifier key. All processes on a machine should share a unique id. By default, processes on a machine determine a unique id to share. This does not generally need to be set. +* ``locality_zoneid``: Zone identifier key. Processes that share a zone id are considered non-unique for the purposes of data replication. If unset, defaults to machine id. +* ``locality_dcid``: Data center identifier key. All processes physically located in a data center should share the id. No default value. If you are depending on data center based replication this must be set on all processes. +* ``locality_data_hall``: Data hall identifier key. All processes physically located in a data hall should share the id. No default value. If you are depending on data hall based replication this must be set on all processes. +* ``io_trust_seconds``: Time in seconds that a read or write operation is allowed to take before timing out with an error. If an operation times out, all future operations on that file will fail with an error as well. Only has an effect when using AsyncFileKAIO in Linux. If unset, defaults to 0 which means timeout is disabled. ``[fdbserver.]`` section(s) --------------------------------- From b52cae3d6a6b0056c0285dab7b0f55f58d49a375 Mon Sep 17 00:00:00 2001 From: Alvin Moore Date: Mon, 19 Mar 2018 13:27:09 -0700 Subject: [PATCH 02/26] Modified the name of the documentation artifact to vary in release and snapshot build --- Makefile | 8 ++++---- documentation/sphinx/Makefile | 10 ++-------- 2 files changed, 6 insertions(+), 12 deletions(-) diff --git a/Makefile b/Makefile index 060bdbf74e..d679dc7888 100644 --- a/Makefile +++ b/Makefile @@ -180,13 +180,13 @@ docpreview: javadoc godoc docpreview_clean: CLEAN_TARGETS= $(MAKE) -C documentation docpreview_clean -packages/foundationdb-docs-$(VERSION)-SNAPSHOT.tar.gz: FORCE javadoc godoc +packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz: FORCE javadoc godoc TARGETS= $(MAKE) -C documentation docpackage @mkdir -p packages - @rm -f packages/foundationdb-docs-SNAPSHOT.tar.gz - @cp documentation/sphinx/.dist/foundationdb-docs-$(VERSION)-SNAPSHOT.tar.gz packages/foundationdb-docs-$(VERSION)-SNAPSHOT.tar.gz + @rm -f packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz + @cp documentation/sphinx/.dist/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz -docpackage: packages/foundationdb-docs-$(VERSION)-SNAPSHOT.tar.gz +docpackage: packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz FORCE: diff --git a/documentation/sphinx/Makefile b/documentation/sphinx/Makefile index eeccec56ce..88adeefccb 100644 --- a/documentation/sphinx/Makefile +++ b/documentation/sphinx/Makefile @@ -4,18 +4,11 @@ PROJECT_NAME := foundationdb-docs #VERSION := $(shell cat version) -ifeq ($(RELEASE_BUILD),true) -RELEASE := $(VERSION) -else -RELEASE := $(VERSION)-SNAPSHOT -endif - GIT_HEAD_REF := $(shell git rev-parse --short HEAD) GIT_BRANCH := $(shell git symbolic-ref --short HEAD) GIT_REPO_URL := $(shell git config --get remote.origin.url) # You can set these variables from the command line. -#VERSIONOPTS := -D version=$(VERSION) -D release=$(RELEASE) SPHINXOPTS := -c . PAPER = ROOTDIR := $(CURDIR) @@ -110,4 +103,5 @@ livehtml: html # removed html prerequisite because it is previously explictly invoked package: mkdir -p $(DISTDIR) - cd $(BUILDDIR)/html && tar czf $(DISTDIR)/$(PROJECT_NAME)-$(RELEASE).tar.gz . + rm -f $(DISTDIR)/$(PROJECT_NAME)-$(VERSION)-$(PKGRELEASE).tar.gz + cd $(BUILDDIR)/html && tar czf $(DISTDIR)/$(PROJECT_NAME)-$(VERSION)-$(PKGRELEASE).tar.gz . From feec2d3367db0f89a908b23a5f802e6cda59b6dd Mon Sep 17 00:00:00 2001 From: swetha mundla Date: Mon, 19 Mar 2018 13:45:26 -0700 Subject: [PATCH 03/26] admin changes --- .../sphinx/source/administration.rst | 240 ++++++++++++------ 1 file changed, 164 insertions(+), 76 deletions(-) diff --git a/documentation/sphinx/source/administration.rst b/documentation/sphinx/source/administration.rst index d94f29c063..8e0e298cbd 100644 --- a/documentation/sphinx/source/administration.rst +++ b/documentation/sphinx/source/administration.rst @@ -24,7 +24,7 @@ Starting and stopping After installation, FoundationDB is set to start automatically. You can manually start and stop the database with the commands shown below. -These commands start and stop the master ``fdbmonitor`` process, which in turn starts ``fdbserver`` and ``backup-agent`` processes. See :ref:`administration_fdbmonitor` for details. +These commands start and stop the master ``fdbmonitor`` process, which in turn starts ``fdbserver`` , ``backup-agent`` and ``backup-agent`` processes. See :ref:`administration_fdbmonitor` for details. Linux ----- @@ -225,44 +225,46 @@ Use the ``status`` command of ``fdbcli`` to determine if the cluster is up and r The database is available. Welcome to the fdbcli. For help, type `help'. - fdb> status +fdb> status -The ``status`` command displays general information about the FoundationDB cluster:: +Configuration: + Redundancy mode - triple + Storage engine - ssd-2 + Coordinators - 5 + Desired Proxies - 5 + Desired Logs - 8 - Configuration: - Redundancy mode - triple - Storage engine - ssd - Coordinators - 3 +Cluster: + FoundationDB processes - 272 + Machines - 16 + Memory availability - 14.5 GB per process on machine with least available + Retransmissions rate - 20 Hz + Fault Tolerance - 2 machines + Server time - 03/19/18 08:51:52 - Cluster: - FoundationDB processes - 3 - Machines - 3 - Memory availability - 4.1 GB per process on machine with least available - Fault Tolerance - 0 machines - Server time - Thu Mar 15 14:41:34 2018 +Data: + Replication health - Healthy + Moving data - 0.000 GB + Sum of key-value sizes - 3.298 TB + Disk space used - 15.243 TB - Data: - Replication health - Healthy - Moving data - 0.000 GB - Sum of key-value sizes - 8 MB - Disk space used - 103 MB +Operating space: + Storage server - 1656.2 GB free on most full server + Log server - 1794.7 GB free on most full server - Operating space: - Storage server - 1.0 GB free on most full server - Log server - 1.0 GB free on most full server +Workload: + Read rate - 55990 Hz + Write rate - 14946 Hz + Transactions started - 6321 Hz + Transactions committed - 1132 Hz + Conflict rate - 0 Hz - Workload: - Read rate - 2 Hz - Write rate - 0 Hz - Transactions started - 2 Hz - Transactions committed - 0 Hz - Conflict rate - 0 Hz +Backup and DR: + Running backups - 1 + Running DRs - 1 as primary - Backup and DR: - Running backups - 0 - Running DRs - 0 - - Client time: Thu Mar 15 14:41:34 2018 +Client time: 03/19/18 08:51:51 + The summary fields are interpreted as follows: @@ -270,15 +272,18 @@ The summary fields are interpreted as follows: Redundancy mode The currently configured redundancy mode (see the section :ref:`configuration-choosing-redundancy-mode`) Storage engine The currently configured storage engine (see the section :ref:`configuration-configuring-storage-subsystem`) Coordinators The number of FoundationDB coordination servers +Desired Proxies Number of proxies desired.If replication mode is 3 then default number of proxies is 3 +Desired Logs Number of logs desired.If replication mode is 3 then default number of logs is 3 FoundationDB processes Number of FoundationDB processes participating in the cluster Machines Number of physical machines running at least one FoundationDB process that is participating in the cluster Memory availability RAM per process on machine with least available (see details below) +Retransmissions rate Ratio of retransmitted packets to the total number of packets. Fault tolerance Maximum number of machines that can fail without losing data or availability (number for losing data will be reported separately if lower) Server time Timestamp from the server Replication health A qualitative estimate of the health of data replication Moving data Amount of data currently in movement between machines Sum of key-value sizes Estimated total size of keys and values stored (not including any overhead or replication) -Disk space used Sum of space used across all nodes in the cluster +Disk space used Overall disk space used by the cluster Storage server Free space for storage on the server with least available. For ``ssd`` storage engine, includes only disk; for ``memory`` storage engine, includes both RAM and disk. Log server Free space for log server on the server with least available. Read rate The current number of reads per second @@ -286,8 +291,8 @@ Write rate The current number of writes per second Transaction started The current number of transactions started per second Transaction committed The current number of transactions committed per second Conflict rate The current number of conflicts per second -Running backups Number of backups currently running. Different backups could be backing up to different prefixes and/or to different targets. -Running DRs Number of DRs currently running. Different DRs could be streaming different prefixes and/or to different DR clusters. +Running backups Determines if backup is running for this cluster.If it displays 1 then it is running backup and if it displays 0 then it is not running backup +Running DRs Determines if the cluster is primary or secondary cluster.If it displays '1 as primary' then it is primary cluster and if it displays '1 as secondary' then it is secondary cluster ====================== ========================================================================================================== The "Memory availability" is a conservative estimate of the minimal RAM available to any ``fdbserver`` process across all machines in the cluster. This value is calculated in two steps. Memory available per process is first calculated *for each machine* by taking: @@ -318,58 +323,139 @@ The ``status`` command can provide detailed statistics about the cluster and the Welcome to the fdbcli. For help, type `help'. fdb> status details - Configuration: - Redundancy mode - triple - Storage engine - ssd - Coordinators - 3 - Cluster: - FoundationDB processes - 3 - Machines - 3 - Memory availability - 4.1 GB per process on machine with least available - Fault Tolerance - 0 machines - Server time - Thu Mar 15 14:41:34 2018 +Configuration: + Redundancy mode - triple + Storage engine - ssd-2 + Coordinators - 5 - Data: - Replication health - Healthy - Moving data - 0.000 GB - Sum of key-value sizes - 8 MB - Disk space used - 103 MB +Cluster: + FoundationDB processes - 85 + Machines - 5 + Memory availability - 7.4 GB per process on machine with least available + Retransmissions rate - 5 Hz + Fault Tolerance - 2 machines + Server time - 03/19/18 08:59:37 - Operating space: - Storage server - 1.0 GB free on most full server - Log server - 1.0 GB free on most full server +Data: + Replication health - Healthy + Moving data - 0.000 GB + Sum of key-value sizes - 87.068 GB + Disk space used - 327.819 GB - Workload: - Read rate - 2 Hz - Write rate - 0 Hz - Transactions started - 2 Hz - Transactions committed - 0 Hz - Conflict rate - 0 Hz +Operating space: + Storage server - 888.2 GB free on most full server + Log server - 897.3 GB free on most full server - Backup and DR: - Running backups - 0 - Running DRs - 0 +Workload: + Read rate - 117 Hz + Write rate - 0 Hz + Transactions started - 43 Hz + Transactions committed - 1 Hz + Conflict rate - 0 Hz - Process performance details: - 10.0.4.1:4500 ( 3% cpu; 2% machine; 0.004 Gbps; 0% disk; 2.5 GB / 4.1 GB RAM ) - 10.0.4.2:4500 ( 1% cpu; 2% machine; 0.004 Gbps; 0% disk; 2.5 GB / 4.1 GB RAM ) - 10.0.4.3:4500 ( 1% cpu; 2% machine; 0.004 Gbps; 0% disk; 2.5 GB / 4.1 GB RAM ) +Process performance details: + 10.214.189.44:4689 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) + 10.214.189.44:4690 ( 1% cpu; 2% machine; 0.010 Gbps; 3% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.189.44:4691 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.189.44:4692 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.189.44:4693 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.189.44:4694 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.189.44:4695 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.189.44:4696 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.189.44:4697 ( 2% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.189.44:4698 ( 2% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.189.44:4699 ( 1% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.189.44:4700 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.189.44:4701 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.189.44:4702 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.189.44:4703 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.214.189.44:4704 ( 12% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.214.189.44:4705 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.3 GB / 7.4 GB RAM ) + 10.214.190.100:4689 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) + 10.214.190.100:4690 ( 15% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4691 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4692 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4693 ( 2% cpu; 3% machine; 0.124 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4694 ( 18% cpu; 3% machine; 0.124 Gbps; 18% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4695 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4696 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4697 ( 2% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4698 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4699 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4700 ( 2% cpu; 3% machine; 0.124 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4701 ( 2% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.190.100:4702 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.190.100:4703 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.214.190.100:4704 ( 11% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.214.190.100:4705 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.6 GB / 7.4 GB RAM ) + 10.214.193.108:4689 ( 14% cpu; 3% machine; 0.284 Gbps; 26% disk IO; 3.0 GB / 7.4 GB RAM ) + 10.214.193.108:4690 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) + 10.214.193.108:4691 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) + 10.214.193.108:4692 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.193.108:4693 ( 7% cpu; 3% machine; 0.284 Gbps; 12% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.193.108:4694 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.193.108:4695 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.193.108:4696 ( 2% cpu; 3% machine; 0.284 Gbps; 26% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.193.108:4697 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.193.108:4698 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.193.108:4699 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.193.108:4700 ( 2% cpu; 3% machine; 0.284 Gbps; 12% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.193.108:4701 ( 2% cpu; 3% machine; 0.284 Gbps; 3% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.193.108:4702 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.193.108:4703 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) + 10.214.193.108:4704 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) + 10.214.193.108:4705 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) + 10.214.202.60:4689 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) + 10.214.202.60:4690 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4691 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4692 ( 2% cpu; 4% machine; 0.065 Gbps; 16% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4693 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.202.60:4694 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4695 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4696 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4697 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4698 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4699 ( 24% cpu; 4% machine; 0.065 Gbps; 15% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4700 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) + 10.214.202.60:4701 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.202.60:4702 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.202.60:4703 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.214.202.60:4704 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.214.202.60:4705 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.6 GB / 7.4 GB RAM ) + 10.214.203.88:4689 ( 6% cpu; 2% machine; 0.076 Gbps; 7% disk IO; 3.2 GB / 7.4 GB RAM ) + 10.214.203.88:4690 ( 2% cpu; 2% machine; 0.076 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4691 ( 1% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4692 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4693 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.203.88:4694 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.203.88:4695 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4696 ( 2% cpu; 2% machine; 0.076 Gbps; 6% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4697 ( 31% cpu; 2% machine; 0.076 Gbps; 8% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.203.88:4698 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4699 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.214.203.88:4700 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4701 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4702 ( 0% cpu; 2% machine; 0.076 Gbps; 3% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.214.203.88:4703 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.214.203.88:4704 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.214.203.88:4705 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.6 GB / 7.4 GB RAM ) - Coordination servers: - 10.0.4.1:4500 - 10.0.4.2:4500 - 10.0.4.3:4500 - - Client time: Thu Mar 15 14:41:34 2018 +Coordination servers: + 10.214.189.44:4689 (reachable) + 10.214.190.100:4689 (reachable) + 10.214.193.108:4689 (reachable) + 10.214.202.60:4689 (reachable) + 10.214.203.88:4689 (reachable) +Client time: 03/19/18 08:59:37 Several details about individual FoundationDB processes are displayed in a list format in parenthesis after the IP address and port: ======= ========================================================================= cpu CPU utilization of the individual process machine CPU utilization of the machine the process is running on (over all cores) Gbps Total input + output network traffic, in Gbps -disk Percentage busy time of the disk subsystem on which the data resides +disk IO Percentage busy time of the disk subsystem on which the data resides REXMIT! Displayed only if there have been more than 10 TCP segments retransmitted in last 5s RAM Total physical memory used by process / memory available per process ======= ========================================================================= @@ -393,6 +479,8 @@ To make configuring, starting, stopping, and restarting ``fdbserver`` processes During normal operation, ``fdbmonitor`` is transparent, and you interact with it only by modifying the configuration in :ref:`foundationdb.conf ` and perhaps occasionally by :ref:`starting and stopping ` it manually. If some problem prevents an ``fdbserver`` or ``backup-agent`` process from starting or causes it to stop unexpectedly, ``fdbmonitor`` will log errors to the system log. +If kill_on_configuration_change parameter is unset or set to `true` in foundationdb.conf then fdbmonitor will restart on changes automatically.If this parameter is set to `false` it will not restart on changes. + .. _administration-managing-trace-files: Managing trace files @@ -441,7 +529,7 @@ Using the ``memory`` storage engine, both memory and disk space need to be consi Running out of storage space ---------------------------- -FoundationDB is aware of the free storage space on each node. It attempts to load all nodes equally so that no node runs out of space before the others. The database attempts to gracefully stop writes as storage space decreases to 100 MB, refusing to start new transactions with priorities other than ``SYSTEM_IMMEDIATE``. This lower bound on free space leaves space to allow you to use ``SYSTEM_IMMEDIATE`` transactions to remove data. +FoundationDB is aware of the free storage space on each node. It attempts to distribute data equally on all the nodes so that no node runs out of space before the others. The database attempts to gracefully stop writes as storage space decreases to 100 MB, refusing to start new transactions with priorities other than ``SYSTEM_IMMEDIATE``. This lower bound on free space leaves space to allow you to use ``SYSTEM_IMMEDIATE`` transactions to remove data. The measure of free space depends on the storage engine. For the memory storage engine, which is the default after installation, total space is limited to the lesser of the ``storage_memory`` configuration parameter (1 GB in the default configuration) or a fraction of the free disk space. @@ -450,12 +538,12 @@ If the disk is rapidly filled by other programs, trace files, etc., FoundationDB Virtual machines ---------------- -Processes running in different VMs on a single machine will appear to FoundationDB as being hardware isolated. FoundationDB takes pains to assure that data replication is protected from hardware-correlated failures. If FoundationDB is run in multiple VMs on a single machine this protection will be subverted. An administrator can inform FoundationDB of this hardware sharing, however, by specifying a machine ID using the ``machine_id`` parameter in :ref:`foundationdb.conf `. All processes on VMs that share hardware should specify the same ``machine_id``. +Processes running in different VMs on a single machine will appear to FoundationDB as being hardware isolated. FoundationDB takes pains to assure that data replication is protected from hardware-correlated failures. If FoundationDB is run in multiple VMs on a single machine this protection will be subverted. An administrator can inform FoundationDB of this hardware sharing, however, by specifying a machine ID using the ``locality_machineid`` parameter in :ref:`foundationdb.conf `. All processes on VMs that share hardware should specify the same ``locality_machineid``. Datacenters ------------ -FoundationDB is datacenter aware and supports operation across datacenters. In a multiple-datacenter configuration, it is recommended that you set the :ref:`redundancy mode ` to ``three_datacenter`` and that you set the ``datacenter_id`` parameter for all FoundationDB processes in :ref:`foundationdb.conf `. +FoundationDB is datacenter aware and supports operation across datacenters. In a multiple-datacenter configuration, it is recommended that you set the :ref:`redundancy mode ` to ``three_datacenter`` and that you set the ``locality_dcid`` parameter for all FoundationDB processes in :ref:`foundationdb.conf `. If you specify the ``-a`` option to any FoundationDB process in your cluster, you should specify it to all such processes. Processes which do not have a specified datacenter ID on the command line are considered part of a default "unset" datacenter. FoundationDB will incorrectly believe that these processes are failure-isolated from other datacenters, which can reduce performance and fault tolerance. From 03fc931e86f9cb7b265427317cee4ddeba0abf45 Mon Sep 17 00:00:00 2001 From: Alec Grieser Date: Mon, 19 Mar 2018 14:06:17 -0700 Subject: [PATCH 04/26] use JAVA_HOME to find location of JVM for headers for JNI build --- bindings/java/local.mk | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/bindings/java/local.mk b/bindings/java/local.mk index b3184614a2..52b1608a19 100644 --- a/bindings/java/local.mk +++ b/bindings/java/local.mk @@ -38,13 +38,14 @@ else endif ifeq ($(PLATFORM),linux) - fdb_java_CFLAGS += -I/usr/lib/jvm/java-8-openjdk-amd64/include -I/usr/lib/jvm/java-8-openjdk-amd64/include/linux + JAVA_HOME ?= /usr/lib/jvm/java-8-openjdk-amd64 + fdb_java_CFLAGS += -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux fdb_java_LDFLAGS += -static-libgcc java_ARCH := amd64 else ifeq ($(PLATFORM),osx) - # FIXME: Surely there is a better way to grab the JNI headers on any version of macOS. - fdb_java_CFLAGS += -I/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers + JAVA_HOME ?= $(shell /usr/libexec/java_home || /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home) + fdb_java_CFLAGS += -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/darwin java_ARCH := x86_64 endif From 42fe4bbaed57265bb6f21f20a67184fb916be04a Mon Sep 17 00:00:00 2001 From: swetha mundla Date: Mon, 19 Mar 2018 14:26:36 -0700 Subject: [PATCH 05/26] administration guide changes --- .../sphinx/source/administration.rst | 194 +++++++++--------- 1 file changed, 97 insertions(+), 97 deletions(-) diff --git a/documentation/sphinx/source/administration.rst b/documentation/sphinx/source/administration.rst index 8e0e298cbd..f134cb3b5e 100644 --- a/documentation/sphinx/source/administration.rst +++ b/documentation/sphinx/source/administration.rst @@ -24,7 +24,7 @@ Starting and stopping After installation, FoundationDB is set to start automatically. You can manually start and stop the database with the commands shown below. -These commands start and stop the master ``fdbmonitor`` process, which in turn starts ``fdbserver`` , ``backup-agent`` and ``backup-agent`` processes. See :ref:`administration_fdbmonitor` for details. +These commands start and stop the master ``fdbmonitor`` process, which in turn starts ``fdbserver`` and ``backup-agent`` processes. See :ref:`administration_fdbmonitor` for details. Linux ----- @@ -272,8 +272,8 @@ The summary fields are interpreted as follows: Redundancy mode The currently configured redundancy mode (see the section :ref:`configuration-choosing-redundancy-mode`) Storage engine The currently configured storage engine (see the section :ref:`configuration-configuring-storage-subsystem`) Coordinators The number of FoundationDB coordination servers -Desired Proxies Number of proxies desired.If replication mode is 3 then default number of proxies is 3 -Desired Logs Number of logs desired.If replication mode is 3 then default number of logs is 3 +Desired Proxies Number of proxies desired. If replication mode is 3 then default number of proxies is 3 +Desired Logs Number of logs desired. If replication mode is 3 then default number of logs is 3 FoundationDB processes Number of FoundationDB processes participating in the cluster Machines Number of physical machines running at least one FoundationDB process that is participating in the cluster Memory availability RAM per process on machine with least available (see details below) @@ -291,8 +291,8 @@ Write rate The current number of writes per second Transaction started The current number of transactions started per second Transaction committed The current number of transactions committed per second Conflict rate The current number of conflicts per second -Running backups Determines if backup is running for this cluster.If it displays 1 then it is running backup and if it displays 0 then it is not running backup -Running DRs Determines if the cluster is primary or secondary cluster.If it displays '1 as primary' then it is primary cluster and if it displays '1 as secondary' then it is secondary cluster +Running backups Number of backups currently running. Different backups could be backing up to different prefixes and/or to different targets. +Running DRs Number of DRs currently running. Different DRs could be streaming different prefixes and/or to different DR clusters. ====================== ========================================================================================================== The "Memory availability" is a conservative estimate of the minimal RAM available to any ``fdbserver`` process across all machines in the cluster. This value is calculated in two steps. Memory available per process is first calculated *for each machine* by taking: @@ -355,98 +355,98 @@ Workload: Conflict rate - 0 Hz Process performance details: - 10.214.189.44:4689 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) - 10.214.189.44:4690 ( 1% cpu; 2% machine; 0.010 Gbps; 3% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.189.44:4691 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.189.44:4692 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.189.44:4693 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.189.44:4694 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.189.44:4695 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.189.44:4696 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.189.44:4697 ( 2% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.189.44:4698 ( 2% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.189.44:4699 ( 1% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.189.44:4700 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.189.44:4701 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.189.44:4702 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.189.44:4703 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) - 10.214.189.44:4704 ( 12% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) - 10.214.189.44:4705 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.3 GB / 7.4 GB RAM ) - 10.214.190.100:4689 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) - 10.214.190.100:4690 ( 15% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4691 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4692 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4693 ( 2% cpu; 3% machine; 0.124 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4694 ( 18% cpu; 3% machine; 0.124 Gbps; 18% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4695 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4696 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4697 ( 2% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4698 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4699 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4700 ( 2% cpu; 3% machine; 0.124 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4701 ( 2% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.190.100:4702 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.190.100:4703 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) - 10.214.190.100:4704 ( 11% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) - 10.214.190.100:4705 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.6 GB / 7.4 GB RAM ) - 10.214.193.108:4689 ( 14% cpu; 3% machine; 0.284 Gbps; 26% disk IO; 3.0 GB / 7.4 GB RAM ) - 10.214.193.108:4690 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) - 10.214.193.108:4691 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) - 10.214.193.108:4692 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.193.108:4693 ( 7% cpu; 3% machine; 0.284 Gbps; 12% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.193.108:4694 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.193.108:4695 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.193.108:4696 ( 2% cpu; 3% machine; 0.284 Gbps; 26% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.193.108:4697 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.193.108:4698 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.193.108:4699 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.193.108:4700 ( 2% cpu; 3% machine; 0.284 Gbps; 12% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.193.108:4701 ( 2% cpu; 3% machine; 0.284 Gbps; 3% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.193.108:4702 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.193.108:4703 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) - 10.214.193.108:4704 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) - 10.214.193.108:4705 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) - 10.214.202.60:4689 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) - 10.214.202.60:4690 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4691 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4692 ( 2% cpu; 4% machine; 0.065 Gbps; 16% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4693 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.202.60:4694 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4695 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4696 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4697 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4698 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4699 ( 24% cpu; 4% machine; 0.065 Gbps; 15% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4700 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) - 10.214.202.60:4701 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.202.60:4702 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.202.60:4703 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.2 GB / 7.4 GB RAM ) - 10.214.202.60:4704 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.2 GB / 7.4 GB RAM ) - 10.214.202.60:4705 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.6 GB / 7.4 GB RAM ) - 10.214.203.88:4689 ( 6% cpu; 2% machine; 0.076 Gbps; 7% disk IO; 3.2 GB / 7.4 GB RAM ) - 10.214.203.88:4690 ( 2% cpu; 2% machine; 0.076 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4691 ( 1% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4692 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4693 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.203.88:4694 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.203.88:4695 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4696 ( 2% cpu; 2% machine; 0.076 Gbps; 6% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4697 ( 31% cpu; 2% machine; 0.076 Gbps; 8% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.203.88:4698 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4699 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) - 10.214.203.88:4700 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4701 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4702 ( 0% cpu; 2% machine; 0.076 Gbps; 3% disk IO; 2.6 GB / 7.4 GB RAM ) - 10.214.203.88:4703 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) - 10.214.203.88:4704 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) - 10.214.203.88:4705 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.6 GB / 7.4 GB RAM ) + 10.0.4.1:4500 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) + 10.0.4.1:4501 ( 1% cpu; 2% machine; 0.010 Gbps; 3% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.1:4502 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.1:4503 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.1:4504 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.1:4505 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.1:4506 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.1:4507 ( 2% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.1:4508 ( 2% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.1:4509 ( 2% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.1:4510 ( 1% cpu; 2% machine; 0.010 Gbps; 1% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.1:4511 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.1:4512 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.1:4513 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.1:4514 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.0.4.1:4515 ( 12% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.0.4.1:4516 ( 0% cpu; 2% machine; 0.010 Gbps; 0% disk IO; 0.3 GB / 7.4 GB RAM ) + 10.0.4.2:4500 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) + 10.0.4.2:4501 ( 15% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4502 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4503 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4504 ( 2% cpu; 3% machine; 0.124 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4505 ( 18% cpu; 3% machine; 0.124 Gbps; 18% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4506 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4507 ( 2% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4508 ( 2% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4509 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4510 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4511 ( 2% cpu; 3% machine; 0.124 Gbps; 1% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4512 ( 2% cpu; 3% machine; 0.124 Gbps; 19% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.2:4513 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.2:4514 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.0.4.2:4515 ( 11% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.0.4.2:4516 ( 0% cpu; 3% machine; 0.124 Gbps; 0% disk IO; 0.6 GB / 7.4 GB RAM ) + 10.0.4.3:4500 ( 14% cpu; 3% machine; 0.284 Gbps; 26% disk IO; 3.0 GB / 7.4 GB RAM ) + 10.0.4.3:4501 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) + 10.0.4.3:4502 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) + 10.0.4.3:4503 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.3:4504 ( 7% cpu; 3% machine; 0.284 Gbps; 12% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.3:4505 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.3:4506 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.3:4507 ( 2% cpu; 3% machine; 0.284 Gbps; 26% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.3:4508 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.3:4509 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.3:4510 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.3:4511 ( 2% cpu; 3% machine; 0.284 Gbps; 12% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.3:4512 ( 2% cpu; 3% machine; 0.284 Gbps; 3% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.3:4513 ( 2% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.3:4514 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) + 10.0.4.3:4515 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) + 10.0.4.3:4516 ( 0% cpu; 3% machine; 0.284 Gbps; 0% disk IO; 0.1 GB / 7.4 GB RAM ) + 10.0.4.4:4500 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 3.2 GB / 7.4 GB RAM ) + 10.0.4.4:4501 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4502 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4503 ( 2% cpu; 4% machine; 0.065 Gbps; 16% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4504 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.4:4505 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4506 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4507 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4508 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4509 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4510 ( 24% cpu; 4% machine; 0.065 Gbps; 15% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4511 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.8 GB / 7.4 GB RAM ) + 10.0.4.4:4512 ( 2% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.4:4513 ( 0% cpu; 4% machine; 0.065 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.4:4514 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.0.4.4:4515 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.0.4.4:4516 ( 0% cpu; 4% machine; 0.065 Gbps; 1% disk IO; 0.6 GB / 7.4 GB RAM ) + 10.0.4.5:4500 ( 6% cpu; 2% machine; 0.076 Gbps; 7% disk IO; 3.2 GB / 7.4 GB RAM ) + 10.0.4.5:4501 ( 2% cpu; 2% machine; 0.076 Gbps; 19% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4502 ( 1% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4503 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4504 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.5:4505 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.5:4506 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4507 ( 2% cpu; 2% machine; 0.076 Gbps; 6% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4508 ( 31% cpu; 2% machine; 0.076 Gbps; 8% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.5:4509 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4510 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.7 GB / 7.4 GB RAM ) + 10.0.4.5:4511 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4512 ( 2% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4513 ( 0% cpu; 2% machine; 0.076 Gbps; 3% disk IO; 2.6 GB / 7.4 GB RAM ) + 10.0.4.5:4514 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.0.4.5:4515 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.2 GB / 7.4 GB RAM ) + 10.0.4.5:4516 ( 0% cpu; 2% machine; 0.076 Gbps; 0% disk IO; 0.6 GB / 7.4 GB RAM ) Coordination servers: - 10.214.189.44:4689 (reachable) - 10.214.190.100:4689 (reachable) - 10.214.193.108:4689 (reachable) - 10.214.202.60:4689 (reachable) - 10.214.203.88:4689 (reachable) + 10.0.4.1:4500 (reachable) + 10.0.4.2:4500 (reachable) + 10.0.4.3:4500 (reachable) + 10.0.4.4:4500 (reachable) + 10.0.4.5:4500 (reachable) Client time: 03/19/18 08:59:37 Several details about individual FoundationDB processes are displayed in a list format in parenthesis after the IP address and port: @@ -479,7 +479,7 @@ To make configuring, starting, stopping, and restarting ``fdbserver`` processes During normal operation, ``fdbmonitor`` is transparent, and you interact with it only by modifying the configuration in :ref:`foundationdb.conf ` and perhaps occasionally by :ref:`starting and stopping ` it manually. If some problem prevents an ``fdbserver`` or ``backup-agent`` process from starting or causes it to stop unexpectedly, ``fdbmonitor`` will log errors to the system log. -If kill_on_configuration_change parameter is unset or set to `true` in foundationdb.conf then fdbmonitor will restart on changes automatically.If this parameter is set to `false` it will not restart on changes. +If kill_on_configuration_change parameter is unset or set to `true` in foundationdb.conf then fdbmonitor will restart on changes automatically. If this parameter is set to `false` it will not restart on changes. .. _administration-managing-trace-files: @@ -529,7 +529,7 @@ Using the ``memory`` storage engine, both memory and disk space need to be consi Running out of storage space ---------------------------- -FoundationDB is aware of the free storage space on each node. It attempts to distribute data equally on all the nodes so that no node runs out of space before the others. The database attempts to gracefully stop writes as storage space decreases to 100 MB, refusing to start new transactions with priorities other than ``SYSTEM_IMMEDIATE``. This lower bound on free space leaves space to allow you to use ``SYSTEM_IMMEDIATE`` transactions to remove data. +FoundationDB is aware of the free storage space on each node. It attempts to distribute data equally on all the nodes so that no node runs out of space before the others. The database attempts to gracefully stop writes as storage space decreases to 100 MB, refusing to start new transactions with priorities other than ``SYSTEM_IMMEDIATE``. This lower bound on free space leaves space to allow you to use ``SYSTEM_IMMEDIATE`` transactions to remove data. The measure of free space depends on the storage engine. For the memory storage engine, which is the default after installation, total space is limited to the lesser of the ``storage_memory`` configuration parameter (1 GB in the default configuration) or a fraction of the free disk space. From 15a151fbd0c452908599a952905458c501f7546c Mon Sep 17 00:00:00 2001 From: Balachandar Namasivayam Date: Mon, 19 Mar 2018 14:36:17 -0700 Subject: [PATCH 06/26] Update configuration.rst on setting process class. --- documentation/sphinx/source/configuration.rst | 29 ++++++++----------- 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/documentation/sphinx/source/configuration.rst b/documentation/sphinx/source/configuration.rst index 7e0cb281eb..08904f6632 100644 --- a/documentation/sphinx/source/configuration.rst +++ b/documentation/sphinx/source/configuration.rst @@ -10,7 +10,7 @@ This document contains *reference* information for configuring a new FoundationD .. note:: In FoundationDB, a "cluster" refers to one or more FoundationDB processes spread across one or more physical machines that together host a FoundationDB database. -To plan an externally accessible cluster, you need to understand some basic aspects of the system. You can start by reviewing the :ref:`system requirements `, then how to :ref:`choose ` and :ref:`change coordination servers `. Next, you should look at the :ref:`configuration file `, which controls most other aspects of the system. Then, you should understand how to :ref:`choose a redundancy mode ` and :ref:`configure the storage subsystem `. Finally, there are some configurations you can adjust to improve performance if your :ref:`cluster is large `. +To plan an externally accessible cluster, you need to understand some basic aspects of the system. You can start by reviewing the :ref:`system requirements `, then how to :ref:`choose ` and :ref:`change coordination servers `. Next, you should look at the :ref:`configuration file `, which controls most other aspects of the system. Then, you should understand how to :ref:`choose a redundancy mode ` and :ref:`configure the storage subsystem `. Finally, there are some guidelines for setting :ref:`process class configurations `. .. _system-requirements: @@ -255,7 +255,7 @@ Contains default parameters for all fdbserver processes on this machine. These s * ``logdir``: A writable directory (by root or by the user set in the [fdbmonitor] section) where FoundationDB will store log files. * ``logsize``: Roll over to a new log file after the current log file reaches the specified size. The default value is 10MiB. * ``maxlogssize``: Delete the oldest log file when the total size of all log files exceeds the specified size. If set to 0B, old log files will not be deleted. The default value is 100MiB. -* ``class``: Machine class specifying the roles that will be taken in the cluster. Valid options are ``storage``, ``transaction``, ``resolution``. See :ref:`configuration-large-cluster-performance` for machine class recommendations in large clusters. +* ``class``: Process class specifying the roles that will be taken in the cluster. Recommended options are ``storage``, ``transaction``, ``stateless``. See :ref:`guidelines-process-class-config` for process class config recommendations. * ``memory``: Maximum memory used by the process. The default value is 8GiB. When specified without a unit, MiB is assumed. This parameter does not change the memory allocation of the program. Rather, it sets a hard limit beyond which the process will kill itself and be restarted. The default value of 8GiB is double the intended memory usage in the default configuration (providing an emergency buffer to deal with memory leaks or similar problems). It is *not* recommended to decrease the value of this parameter below its default value. It may be *increased* if you wish to allocate a very large amount of storage engine memory or cache. In particular, when the ``storage_memory`` parameter is increased, the ``memory`` parameter should be increased by an equal amount. * ``storage_memory``: Maximum memory used for data storage. This paramenter is used *only* with memory storage engine, not the ssd storage engine. The default value is 1GiB. When specified without a unit, MB is assumed. Clusters will be restricted to using this amount of memory per process for purposes of data storage. Memory overhead associated with storing the data is counted against this total. If you increase the ``storage_memory``, you should also increase the ``memory`` parameter by the same amount. * ``locality_machineid``: Machine identifier key. All processes on a machine should share a unique id. By default, processes on a machine determine a unique id to share. This does not generally need to be set. @@ -514,27 +514,22 @@ When creating a partition for use with FoundationDB using the standard Linux fdi For an SSD with a single partition, the partition should typically begin at sector 2048 (512 byte sectors yields 1024 KiB alignment). -.. _configuration-large-cluster-performance: +.. _guidelines-process-class-config: -Large cluster performance -========================= +Guidelines for setting process class +==================================== -.. note:: For small-to-medium clusters (32 processes or fewer), FoundationDB's default behavior generally provides the best performance, and you should ignore this section. Further configuration is recommended only for large clusters (> 32 processes) or if you have special latency requirements. +In a FoundationDB cluster, each of the ``fdbserver`` processes perform different tasks. Each process is recruited to do a particular task based on its process ``class``. For example, processes with ``class=storage`` are given preference to be recruited for doing storage server tasks, ``class=transaction`` are for log server processes and ``class=stateless`` are for stateless processes like proxies, resolvers, etc., -In a FoundationDB cluster, each of the ``fdbserver`` processes perform different tasks. FoundationDB automatically assigns each machine in the cluster a ``class`` that specifies the tasks it will perform. For large clusters, FoundationDB also provides the ability to tune cluster performance by manually assigning the ``class`` of some machines. +The recommended minimum number of ``class=transaction`` (log server) processes is 8 (active) + 2 (standby) and the recommended minimum number for ``class=stateless`` processes is 4 (proxy) + 1 (resolver) + 1 (Cluster Controller) + 1 (Master) + 2 (standby). It is better to spread the transaction and stateless processes across as many machines as possible. -To assign machine classes manually, set the ``class=transaction`` parameter in :ref:`foundationdb.conf ` on all processes on selected machines. The ratio of total processes to ``class``-specified processes should be about 8:1. For example, if you have 64 processes on 16 machines, you would set ``class=transaction`` for 8 processes on 2 machines. +``fdbcli`` is used to set the desired number of processes of a particular process type. To do so, you would issue the ``fdbcli`` commands:: -For large clusters with high write workloads (greater than 100,000 writes/second), you can increase performance by increasing the number of proxies, resolvers, and log servers. These are set using ``fdbcli`` in equal (1:1:1) proportions among the processes on machines set to ``class=transaction``. + fdb> configure proxies=5 + fdb> configure logs=8 -For example, if you have 384 processes on 96 machines, and a workload greater than 100,000 writes per second, you would set ``class=transaction`` for 48 processes on 12 machines. Of the latter, you would set 16 processes on 4 machines each as *proxies*, *resolvers*, and *log servers*. To do so, you would issue the ``fdbcli`` commands:: +.. note:: In the present release, the default value for proxies and log servers is 3 and for resolvers is 1. You should not set the value of a process type to less than its default. - fdb> configure proxies=16 - fdb> configure resolvers=16 - fdb> configure logs=16 - -.. note:: In the present release, the default value for proxies and log servers is 3 and for resolvers is 1. The ratios discussed above are guidelines; regardless of them, you should not set the value of a process type to less than its default. For example, on clusters ranging from 36 to 60 processes with high write workloads, you may choose to increase the number of resolvers to 2. In this case, you would nevertheless leave the number of proxies and log servers at their default values of 3. - -.. warning:: The conflict-resolution algorithm used by FoundationDB is conservative: it guarantees that no conflicting transactions will be committed, but it may fail to commit some transactions that theoretically could have been. The effects of this conservatism may increase as you increase the number of proxies. It is therefore important to employ the recommended techniques for :ref:`minimizing conflicts ` when increasing the number of proxies. +.. warning:: The conflict-resolution algorithm used by FoundationDB is conservative: it guarantees that no conflicting transactions will be committed, but it may fail to commit some transactions that theoretically could have been. The effects of this conservatism may increase as you increase the number of resolvers. It is therefore important to employ the recommended techniques for :ref:`minimizing conflicts ` when increasing the number of resolvers. You can contact us on the `community forums `_ if you are interested in more details or if you are benchmarking or performance-tuning on large clusters. Also see our `performance benchmarks `_ for a baseline of how a well-configured cluster should perform. From 28bd3ec68f9f107024e740ad367816e980b7df18 Mon Sep 17 00:00:00 2001 From: "A.J. Beamon" Date: Mon, 19 Mar 2018 14:44:32 -0700 Subject: [PATCH 07/26] Update download page links, add Windows package, remove Node packages, and some rewording. Fix a comment in the Windows installer. --- documentation/sphinx/source/downloads.rst | 66 ++++++++++++++++------- packaging/msi/FDBInstaller.wxs | 2 +- 2 files changed, 48 insertions(+), 20 deletions(-) diff --git a/documentation/sphinx/source/downloads.rst b/documentation/sphinx/source/downloads.rst index 6c250a6bfc..c7c0f21267 100644 --- a/documentation/sphinx/source/downloads.rst +++ b/documentation/sphinx/source/downloads.rst @@ -5,12 +5,43 @@ Downloads Client & Server Packages ======================== -FoundationDB packages are available on Artifactory for the following operating systems: +macOS +----- -* `macOS `_. Supported on macOS >= 10.7. Installs client and (optionally) server. -* `Ubuntu `_. The server depends on the client. Supported on 64-bit Ubuntu >= 12.04, but beware of the Linux kernel bug in Ubuntu 12.x. -* `RHEL/CentOS EL6 `_. The server depends on the client. Supported on 64-bit RHEL/CentOS (6.x). -* `RHEL/CentOS EL7 `_. The server depends on the client. Supported on 64-bit RHEL/CentOS (7.x). +The macOS installation package is supported on macOS 10.7+. It includes the client and (optionally) the server. + +* `FoundationDB-5.1.0.pkg `_ + +Ubuntu +------ + +The Ubuntu packages are supported on 64-bit Ubuntu 12.04+, but beware of the Linux kernel bug in Ubuntu 12.x. + +* `foundationdb-clients-5.1.0-1_amd64.deb `_ +* `foundationdb-server-5.1.0-1_amd64.deb `_ (depends on the clients package) + +RHEL/CentOS EL6 +--------------- + +The RHEL/CentOS EL6 packages are supported on 64-bit RHEL/CentOS 6.x. + +* `foundationdb-clients-5.1.0-1.el6.x86_64.rpm `_ +* `foundationdb-server-5.1.0-1.el6.x86_64.rpm `_ (depends on the clients package) + +RHEL/CentOS EL7 +--------------- + +The RHEL/CentOS EL7 packages are supported on 64-bit RHEL/CentOS 7.x. + +* `foundationdb-clients-5.1.0-1.el7.x86_64.rpm `_ +* `foundationdb-server-5.1.0-1.el7.x86_64.rpm `_ (depends on the clients package) + +Windows +------- + +The Windows installer is supported on 64-bit Windows XP and later. It includes the client and (optionally) the server. + +* `foundationdb-5.1.0-x64.msi `_ API Language Bindings ===================== @@ -23,27 +54,24 @@ FoundationDB's C bindings are installed with the FoundationDB client binaries. Y Python 2.7 - 3.4 ---------------- -The FoundationDB Python API is installed as part of your FoundationDB installation. +On macOS and Windows, the FoundationDB Python API bindings are installed as part of your FoundationDB installation. -If you need to use the FoundationDB Python API from other Python installations or paths, download the `package `_. +If you need to use the FoundationDB Python API from other Python installations or paths, download the Python package: -Ruby 1.9.3/3.0.2 ----------------- +* `foundationdb-5.1.0.tar.gz `_ -Download the `gem `_. - -Java JRE 1.8+ -------------- - -Download the `jar and javadoc.jar `_. - -Node 0.8.x/0.10.x +Ruby 1.9.3/2.0.0+ ----------------- -Download the `node package `_. +* `fdb-5.1.0.gem `_ +Java 8+ +------- + +* `fdb-java-5.1.0.jar `_ +* `fdb-java-5.1.0-javadoc.jar `_ Go 1.1+ ------- -The FoundationDB Go package is available on `github `_ +The FoundationDB Go package is available on `GitHub `_. diff --git a/packaging/msi/FDBInstaller.wxs b/packaging/msi/FDBInstaller.wxs index e7cb747d3c..68bfbd2eb9 100644 --- a/packaging/msi/FDBInstaller.wxs +++ b/packaging/msi/FDBInstaller.wxs @@ -297,7 +297,7 @@ - + From ac322249230ee79109e241666b2d485c3d55e09e Mon Sep 17 00:00:00 2001 From: "A.J. Beamon" Date: Mon, 19 Mar 2018 14:49:22 -0700 Subject: [PATCH 08/26] Fix link. --- documentation/sphinx/source/performance.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/sphinx/source/performance.rst b/documentation/sphinx/source/performance.rst index 7e55053900..39ee0cbb24 100644 --- a/documentation/sphinx/source/performance.rst +++ b/documentation/sphinx/source/performance.rst @@ -105,4 +105,4 @@ A lot of things affect the simple first-order model of performance you see here. * To keep up with the performance modeled above, your disk subsystem will need to do a little over 1 IOPS per write, and about 1 IOPS per (uncached) read. * Network performance tuning at the operating system level can be very important for both latency and throughput, especially in larger clusters. * Running DR and/or backup requires applying each mutation multiple times and then reading those mutations from the database. Using either feature will reduce throughput. -* See our :ref:`known-limitations ` for other considerations which may affect performance. +* See our :ref:`known limitations ` for other considerations which may affect performance. From 828e08a345f6f2d5f998a96501cbbe903cdd24a5 Mon Sep 17 00:00:00 2001 From: John Brownlee Date: Mon, 19 Mar 2018 14:55:28 -0700 Subject: [PATCH 09/26] Add generated.go as a dependency for the godocs. --- bindings/go/include.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bindings/go/include.mk b/bindings/go/include.mk index 40b343b192..403115c8ec 100644 --- a/bindings/go/include.mk +++ b/bindings/go/include.mk @@ -98,7 +98,7 @@ $(GO_DEST)/fdb/generated.go: fdb_go_path fdb_go_fmt_check lib/libfdb_c.$(DLEXT) @echo "Building $@" @go run bindings/go/src/_util/translate_fdb_options.go < fdbclient/vexillographer/fdb.options > $@ -godoc: fdb_go_path $(GO_SRC) +godoc: fdb_go_path $(GO_SRC) $(GO_DEST)/fdb/generated.go @echo "Generating Go Documentation" @rm -rf $(GODOC_DIR)/godoc @mkdir -p $(GODOC_DIR)/godoc From af738e0f4556623514c7b80d5827f33e5396c523 Mon Sep 17 00:00:00 2001 From: "A.J. Beamon" Date: Mon, 19 Mar 2018 15:47:56 -0700 Subject: [PATCH 10/26] Fix Python3 directory layer bug where str() was used instead of bytes(). Fix bug in binding tester that caused directory tests to not compare results, which resulted in the directory layer bug going undetected. --- bindings/bindingtester/tests/directory.py | 4 ++-- bindings/python/fdb/directory_impl.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/bindings/bindingtester/tests/directory.py b/bindings/bindingtester/tests/directory.py index e84db7980c..861ca93972 100644 --- a/bindings/bindingtester/tests/directory.py +++ b/bindings/bindingtester/tests/directory.py @@ -340,9 +340,9 @@ class DirectoryTest(Test): # errors += directory_util.check_for_duplicate_prefixes(db, self.prefix_log) return errors - def get_result_specfications(self): + def get_result_specifications(self): return [ - ResultSpecification(self.stack, key_start_index=1, ordering_index=1), + ResultSpecification(self.stack_subspace, key_start_index=1, ordering_index=1), ResultSpecification(self.directory_log, ordering_index=0), ResultSpecification(self.subspace_log, ordering_index=0) ] diff --git a/bindings/python/fdb/directory_impl.py b/bindings/python/fdb/directory_impl.py index 2a1292c861..f8c4f8d89d 100755 --- a/bindings/python/fdb/directory_impl.py +++ b/bindings/python/fdb/directory_impl.py @@ -75,7 +75,7 @@ class HighContentionAllocator (object): count = tr.snapshot[self.counters[start]] if count != None: - count = struct.unpack(" Date: Mon, 19 Mar 2018 17:18:30 -0700 Subject: [PATCH 11/26] Update docs for extra storage for DR/backup and upgrades --- documentation/sphinx/source/administration.rst | 17 +++++++++++------ documentation/sphinx/source/index.rst | 2 ++ 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/documentation/sphinx/source/administration.rst b/documentation/sphinx/source/administration.rst index 2055503e72..5469c61b09 100644 --- a/documentation/sphinx/source/administration.rst +++ b/documentation/sphinx/source/administration.rst @@ -436,6 +436,8 @@ Using the ``memory`` storage engine, both memory and disk space need to be consi * Disk space usage is about 8x the original data size. The memory storage engine interleaves a snapshot on disk with a transaction log, with the resulting snapshot 2x the data size. A snapshot can't be dropped from its log until the next snapshot is completely written, so 2 snapshots must be kept at 4x the data size. The two-file durable queue can't overwrite data in one file until all the data in the other file has been dropped, resulting in 8x the data size. Finally, it should be noted that disk space is not reclaimed when key-value pairs are cleared. +For either storage engine, there is possible additional overhead when running backup or DR. In usual operation, the overhead is negligible but if backup is unable to write or a secondary cluster is unavailable, mutation logs will build up until copying can resume, occupying space in your cluster. + Running out of storage space ---------------------------- @@ -455,7 +457,7 @@ Datacenters FoundationDB is datacenter aware and supports operation across datacenters. In a multiple-datacenter configuration, it is recommended that you set the :ref:`redundancy mode ` to ``three_datacenter`` and that you set the ``datacenter_id`` parameter for all FoundationDB processes in :ref:`foundationdb.conf `. -If you specify the ``-a`` option to any FoundationDB process in your cluster, you should specify it to all such processes. Processes which do not have a specified datacenter ID on the command line are considered part of a default "unset" datacenter. FoundationDB will incorrectly believe that these processes are failure-isolated from other datacenters, which can reduce performance and fault tolerance. +If you specify the ``--datacenter_id`` option to any FoundationDB process in your cluster, you should specify it to all such processes. Processes which do not have a specified datacenter ID on the command line are considered part of a default "unset" datacenter. FoundationDB will incorrectly believe that these processes are failure-isolated from other datacenters, which can reduce performance and fault tolerance. .. _administration-removing: @@ -494,13 +496,16 @@ To upgrade a FoundationDB cluster, you must install the updated version of Found .. warning:: |development-use-only-warning| +Install updated client binaries +------------------------------- + +Apart from patch version upgrades, you should install the new client binary on all your clients to ensure they can reconnect after the upgrade. See :ref:`multi-version-client-api` for more information. Running ``status json`` will show you which versions clients are connecting with so you can verify before upgrading that clients are correctly configured. + Stage the packages ------------------ Go to :doc:`downloads` and select Ubuntu or RHEL/CentOS, as appropriate for your system. Download both the client and server packages and copy them to each machine in your cluster. -.. warning:: |upgrade-client-server-warning| - Perform the upgrade ------------------- @@ -525,10 +530,10 @@ Test the database Test the database to verify that it is operating normally by running ``fdbcli`` and :ref:`reviewing the cluster status `. -Restart your application clients --------------------------------- +Remove old client library versions +---------------------------------- -Stop and restart all application clients to reload the upgraded FoundationDB dynamic libraries. +You can now remove old client library versions from your clients. This is only to stop creating unnecessary connections. .. _version-specific-upgrading: diff --git a/documentation/sphinx/source/index.rst b/documentation/sphinx/source/index.rst index a388b4b924..af11c6dd9e 100644 --- a/documentation/sphinx/source/index.rst +++ b/documentation/sphinx/source/index.rst @@ -36,6 +36,8 @@ The latest changes are detailed in :doc:`release-notes`. The documentation has t * :doc:`tutorials` provide simple examples of client design using FoundationDB. +* :doc:`administration` contains documentation on administering FoundationDB. + .. toctree:: :maxdepth: 1 :titlesonly: From 4c8395fba78474724626bf2a26982d35da3d0591 Mon Sep 17 00:00:00 2001 From: Alvin Moore Date: Mon, 19 Mar 2018 20:12:49 -0700 Subject: [PATCH 12/26] Removed outdated information messages --- build/scver.mk | 2 -- documentation/sphinx/Makefile | 1 - 2 files changed, 3 deletions(-) diff --git a/build/scver.mk b/build/scver.mk index afc5e6dc6c..bdae8befb7 100644 --- a/build/scver.mk +++ b/build/scver.mk @@ -147,10 +147,8 @@ info: @echo "Package: $(PACKAGE_NAME)" @echo "Version ID: $(VERSION_ID)" @echo "Package ID: $(PKGRELEASE)" - @echo "Source Control: $(SOURCE_CONTROL)" @echo "SC Branch: $(SCBRANCH)" @echo "Git Dir: $(GITPRESENT)" - @echo "Mercurial Dir: $(HGPRESENT)" @echo "Make Dir: $(MAKEDIR)" @echo "Foundation Dir: $(FDBDIR)" @echo "Fdb Dir Base: $(FDBDIRBASE)" diff --git a/documentation/sphinx/Makefile b/documentation/sphinx/Makefile index 88adeefccb..eaa5937f0d 100644 --- a/documentation/sphinx/Makefile +++ b/documentation/sphinx/Makefile @@ -2,7 +2,6 @@ # # local vars: PROJECT_NAME := foundationdb-docs -#VERSION := $(shell cat version) GIT_HEAD_REF := $(shell git rev-parse --short HEAD) GIT_BRANCH := $(shell git symbolic-ref --short HEAD) From 8a2aaf1268872fc6c3018488a0415e741c077ade Mon Sep 17 00:00:00 2001 From: Alec Grieser Date: Tue, 20 Mar 2018 09:17:16 -0700 Subject: [PATCH 13/26] remove somewhat questionable fallback if /user/libexec/java_home is not present --- bindings/java/local.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bindings/java/local.mk b/bindings/java/local.mk index 52b1608a19..baa8430975 100644 --- a/bindings/java/local.mk +++ b/bindings/java/local.mk @@ -44,7 +44,7 @@ ifeq ($(PLATFORM),linux) java_ARCH := amd64 else ifeq ($(PLATFORM),osx) - JAVA_HOME ?= $(shell /usr/libexec/java_home || /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home) + JAVA_HOME ?= $(shell /usr/libexec/java_home) fdb_java_CFLAGS += -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/darwin java_ARCH := x86_64 From 9ed7861f8b18ee8e8349db3f56e4d352cfceecf0 Mon Sep 17 00:00:00 2001 From: Alec Grieser Date: Tue, 20 Mar 2018 11:31:49 -0700 Subject: [PATCH 14/26] make the go build dependencies more sane --- bindings/go/include.mk | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/bindings/go/include.mk b/bindings/go/include.mk index 403115c8ec..d56669b084 100644 --- a/bindings/go/include.mk +++ b/bindings/go/include.mk @@ -25,7 +25,7 @@ GOPATH := $(CURDIR)/bindings/go/build GO_IMPORT_PATH := github.com/apple/foundationdb/bindings/go/src GO_DEST := $(GOPATH)/src/$(GO_IMPORT_PATH) -.PHONY: fdb_go fdb_go_path fdb_go_tester fdb_go_tester_clean godoc godoc_clean +.PHONY: fdb_go fdb_go_path fdb_go_fmt fdb_go_fmt_check fdb_go_tester fdb_go_tester_clean godoc godoc_clean # We only override if the environment didn't set it (this is used by # the fdbwebsite documentation build process) @@ -49,7 +49,7 @@ GO_PACKAGE_OBJECTS := $(addprefix $(GO_PACKAGE_OUTDIR)/,$(GO_PACKAGES:=.a)) GO_SRC := $(shell find $(CURDIR)/bindings/go/src -name '*.go') -fdb_go: $(GO_PACKAGE_OBJECTS) $(GO_SRC) +fdb_go: $(GO_PACKAGE_OBJECTS) $(GO_SRC) fdb_go_fmt_check fdb_go_fmt: $(GO_SRC) @echo "Formatting fdb_go" @@ -59,10 +59,13 @@ fdb_go_fmt_check: $(GO_SRC) @echo "Checking fdb_go" @bash -c 'fmtoutstr=$$(gofmt -l $(GO_SRC)) ; if [[ -n "$${fmtoutstr}" ]] ; then echo "Detected go formatting violations for the following files:" ; echo "$${fmtoutstr}" ; echo "Try running: make fdb_go_fmt"; exit 1 ; fi' -fdb_go_path: $(GO_SRC) +$(GO_DEST)/.stamp: $(GO_SRC) @echo "Creating fdb_go_path" @mkdir -p $(GO_DEST) @cp -r bindings/go/src/* $(GO_DEST) + @touch $(GO_DEST)/.stamp + +fdb_go_path: $(GO_DEST)/.stamp fdb_go_clean: @echo "Cleaning fdb_go" @@ -74,27 +77,27 @@ fdb_go_tester_clean: @echo "Cleaning fdb_go_tester" @rm -rf $(GOPATH)/bin -$(GOPATH)/bin/_stacktester: fdb_go_path fdb_go_fmt_check $(GO_SRC) $(GO_PACKAGE_OBJECTS) $(GO_DEST)/fdb/generated.go +$(GOPATH)/bin/_stacktester: $(GO_DEST)/.stamp $(GO_SRC) $(GO_PACKAGE_OBJECTS) $(GO_DEST)/fdb/generated.go @echo "Compiling $(basename $(notdir $@))" @go install $(GO_IMPORT_PATH)/_stacktester -$(GO_PACKAGE_OUTDIR)/fdb/tuple.a: fdb_go_path fdb_go_fmt_check $(GO_SRC) $(GO_PACKAGE_OUTDIR)/fdb.a $(GO_DEST)/fdb/generated.go +$(GO_PACKAGE_OUTDIR)/fdb/tuple.a: $(GO_DEST)/.stamp $(GO_SRC) $(GO_PACKAGE_OUTDIR)/fdb.a $(GO_DEST)/fdb/generated.go @echo "Compiling fdb/tuple" @go install $(GO_IMPORT_PATH)/fdb/tuple -$(GO_PACKAGE_OUTDIR)/fdb/subspace.a: fdb_go_path fdb_go_fmt_check $(GO_SRC) $(GO_PACKAGE_OUTDIR)/fdb.a $(GO_PACKAGE_OUTDIR)/fdb/tuple.a $(GO_DEST)/fdb/generated.go +$(GO_PACKAGE_OUTDIR)/fdb/subspace.a: $(GO_DEST)/.stamp $(GO_SRC) $(GO_PACKAGE_OUTDIR)/fdb.a $(GO_PACKAGE_OUTDIR)/fdb/tuple.a $(GO_DEST)/fdb/generated.go @echo "Compiling fdb/subspace" @go install $(GO_IMPORT_PATH)/fdb/subspace -$(GO_PACKAGE_OUTDIR)/fdb/directory.a: fdb_go_path fdb_go_fmt_check $(GO_SRC) $(GO_PACKAGE_OUTDIR)/fdb.a $(GO_PACKAGE_OUTDIR)/fdb/tuple.a $(GO_PACKAGE_OUTDIR)/fdb/subspace.a $(GO_DEST)/fdb/generated.go +$(GO_PACKAGE_OUTDIR)/fdb/directory.a: $(GO_DEST)/.stamp $(GO_SRC) $(GO_PACKAGE_OUTDIR)/fdb.a $(GO_PACKAGE_OUTDIR)/fdb/tuple.a $(GO_PACKAGE_OUTDIR)/fdb/subspace.a $(GO_DEST)/fdb/generated.go @echo "Compiling fdb/directory" @go install $(GO_IMPORT_PATH)/fdb/directory -$(GO_PACKAGE_OUTDIR)/fdb.a: fdb_go_path fdb_go_fmt_check $(GO_SRC) $(GO_DEST)/fdb/generated.go +$(GO_PACKAGE_OUTDIR)/fdb.a: $(GO_DEST)/.stamp lib/libfdb_c.$(DLEXT) $(GO_SRC) $(GO_DEST)/fdb/generated.go @echo "Compiling fdb" @go install $(GO_IMPORT_PATH)/fdb -$(GO_DEST)/fdb/generated.go: fdb_go_path fdb_go_fmt_check lib/libfdb_c.$(DLEXT) bindings/go/src/_util/translate_fdb_options.go fdbclient/vexillographer/fdb.options +$(GO_DEST)/fdb/generated.go: $(GO_DEST)/.stamp bindings/go/src/_util/translate_fdb_options.go fdbclient/vexillographer/fdb.options @echo "Building $@" @go run bindings/go/src/_util/translate_fdb_options.go < fdbclient/vexillographer/fdb.options > $@ From c88d3b9e1119219053acb25ed93702bf412af8d9 Mon Sep 17 00:00:00 2001 From: Richard Low Date: Tue, 20 Mar 2018 13:45:24 -0700 Subject: [PATCH 15/26] Add comment to bounce clients after install new binary --- documentation/sphinx/source/administration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/sphinx/source/administration.rst b/documentation/sphinx/source/administration.rst index 5469c61b09..5ed1939ec5 100644 --- a/documentation/sphinx/source/administration.rst +++ b/documentation/sphinx/source/administration.rst @@ -499,7 +499,7 @@ To upgrade a FoundationDB cluster, you must install the updated version of Found Install updated client binaries ------------------------------- -Apart from patch version upgrades, you should install the new client binary on all your clients to ensure they can reconnect after the upgrade. See :ref:`multi-version-client-api` for more information. Running ``status json`` will show you which versions clients are connecting with so you can verify before upgrading that clients are correctly configured. +Apart from patch version upgrades, you should install the new client binary on all your clients and restart them to ensure they can reconnect after the upgrade. See :ref:`multi-version-client-api` for more information. Running ``status json`` will show you which versions clients are connecting with so you can verify before upgrading that clients are correctly configured. Stage the packages ------------------ From 7d93c69d4e529888fde9ce3feca6a72326de986d Mon Sep 17 00:00:00 2001 From: swetha mundla Date: Tue, 20 Mar 2018 14:14:54 -0700 Subject: [PATCH 16/26] administration guide changes --- documentation/sphinx/source/administration.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/documentation/sphinx/source/administration.rst b/documentation/sphinx/source/administration.rst index f134cb3b5e..ea5ec98882 100644 --- a/documentation/sphinx/source/administration.rst +++ b/documentation/sphinx/source/administration.rst @@ -277,7 +277,7 @@ Desired Logs Number of logs desired. If replication mode is 3 then de FoundationDB processes Number of FoundationDB processes participating in the cluster Machines Number of physical machines running at least one FoundationDB process that is participating in the cluster Memory availability RAM per process on machine with least available (see details below) -Retransmissions rate Ratio of retransmitted packets to the total number of packets. +Retransmissions rate Ratio of retransmitted packets to the total number of packets. Fault tolerance Maximum number of machines that can fail without losing data or availability (number for losing data will be reported separately if lower) Server time Timestamp from the server Replication health A qualitative estimate of the health of data replication @@ -291,8 +291,8 @@ Write rate The current number of writes per second Transaction started The current number of transactions started per second Transaction committed The current number of transactions committed per second Conflict rate The current number of conflicts per second -Running backups Number of backups currently running. Different backups could be backing up to different prefixes and/or to different targets. -Running DRs Number of DRs currently running. Different DRs could be streaming different prefixes and/or to different DR clusters. +Running backups Number of backups currently running. Different backups could be backing up to different prefixes and/or to different targets. +Running DRs Number of DRs currently running. Different DRs could be streaming different prefixes and/or to different DR clusters. ====================== ========================================================================================================== The "Memory availability" is a conservative estimate of the minimal RAM available to any ``fdbserver`` process across all machines in the cluster. This value is calculated in two steps. Memory available per process is first calculated *for each machine* by taking: From e9e86745168f83bca4f9a3a49c3e6e41a04e09d5 Mon Sep 17 00:00:00 2001 From: Yichi Chiang Date: Tue, 20 Mar 2018 15:31:17 -0700 Subject: [PATCH 17/26] Fix java dead links and anchors --- .../java/src/main/com/apple/foundationdb/Database.java | 4 ++-- .../src/main/com/apple/foundationdb/KeySelector.java | 4 ++-- .../src/main/com/apple/foundationdb/Transaction.java | 10 +++++----- .../apple/foundationdb/directory/DirectoryLayer.java | 2 +- .../foundationdb/directory/DirectoryPartition.java | 2 +- .../com/apple/foundationdb/directory/package-info.java | 2 +- .../src/main/com/apple/foundationdb/package-info.java | 2 +- .../main/com/apple/foundationdb/subspace/Subspace.java | 2 +- .../com/apple/foundationdb/subspace/package-info.java | 2 +- .../com/apple/foundationdb/tuple/package-info.java | 2 +- bindings/java/src/main/overview.html.in | 2 +- 11 files changed, 17 insertions(+), 17 deletions(-) diff --git a/bindings/java/src/main/com/apple/foundationdb/Database.java b/bindings/java/src/main/com/apple/foundationdb/Database.java index c697eea0bf..57754ecd51 100644 --- a/bindings/java/src/main/com/apple/foundationdb/Database.java +++ b/bindings/java/src/main/com/apple/foundationdb/Database.java @@ -147,7 +147,7 @@ public interface Database extends AutoCloseable, TransactionContext { * be unable to determine whether a transaction succeeded. In these cases, your * transaction may be executed twice. For more information about how to reason * about these situations see - * the FounationDB Developer Guide * * @param retryable the block of logic to execute in a {@link Transaction} against @@ -183,7 +183,7 @@ public interface Database extends AutoCloseable, TransactionContext { * be unable to determine whether a transaction succeeded. In these cases, your * transaction may be executed twice. For more information about how to reason * about these situations see - * the FounationDB Developer Guide
*
* Any errors encountered executing {@code retryable}, or received from the diff --git a/bindings/java/src/main/com/apple/foundationdb/KeySelector.java b/bindings/java/src/main/com/apple/foundationdb/KeySelector.java index 3cba7ec523..c7d74f1c14 100644 --- a/bindings/java/src/main/com/apple/foundationdb/KeySelector.java +++ b/bindings/java/src/main/com/apple/foundationdb/KeySelector.java @@ -32,7 +32,7 @@ import com.apple.foundationdb.tuple.ByteArrayUtil; * {@link Transaction#getRange(KeySelector, KeySelector) getRange()}.
*
* For more about how key selectors work in practice, see - * the KeySelector documentation. + * the KeySelector documentation. *
*
* Generally one of the following static methods should be used to construct a {@code KeySelector}: @@ -119,7 +119,7 @@ public class KeySelector { * poor choice for iterating through a large range. (Instead, use the keys * returned from a range query operation * themselves to create a new beginning {@code KeySelector}.) For more information see - * the KeySelector documentation. + * the KeySelector documentation. * * @param offset the number of keys to offset the {@code KeySelector}. This number can be * negative. diff --git a/bindings/java/src/main/com/apple/foundationdb/Transaction.java b/bindings/java/src/main/com/apple/foundationdb/Transaction.java index 8d917c78f4..7cf3cdc9ea 100644 --- a/bindings/java/src/main/com/apple/foundationdb/Transaction.java +++ b/bindings/java/src/main/com/apple/foundationdb/Transaction.java @@ -34,12 +34,12 @@ import com.apple.foundationdb.tuple.Tuple; * the underlying database if and when the transaction is committed. Read operations do see the * effects of previous write operations on the same transaction. Committing a transaction usually * succeeds in the absence of - * conflicts.
+ * conflicts.
*
* Transactions group operations into a unit with the properties of atomicity, isolation, and * durability. Transactions also provide the ability to maintain an application's invariants or * integrity constraints, supporting the property of consistency. Together these properties are - * known as ACID.
+ * known as acid.
*
* Transactions are also causally consistent: once a transaction has been successfully committed, * all subsequently created transactions will see the modifications made by it. @@ -49,7 +49,7 @@ import com.apple.foundationdb.tuple.Tuple; *
* Keys and values in FoundationDB are byte arrays. To encode other data types, see the * {@link Tuple Tuple API} and - * tuple layer documentation.
+ * tuple layer documentation.
*
* When used as a {@link TransactionContext}, the methods {@code run()} and * {@code runAsync()} on a {@code Transaction} will simply attempt the operations @@ -79,11 +79,11 @@ public interface Transaction extends AutoCloseable, ReadTransaction, Transaction /** * Return special-purpose, read-only view of the database. Reads done through this interface are known as "snapshot reads". * Snapshot reads selectively relax FoundationDB's isolation property, reducing - * Transaction conflicts + * Transaction conflicts * but making reasoning about concurrency harder.
*
* For more information about how to use snapshot reads correctly, see - * Using snapshot reads. + * Using snapshot reads. */ ReadTransaction snapshot(); diff --git a/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryLayer.java b/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryLayer.java index b0f6b696ad..add1f42c06 100644 --- a/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryLayer.java +++ b/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryLayer.java @@ -55,7 +55,7 @@ import com.apple.foundationdb.tuple.Tuple; * to manage its subspaces. * * For general guidance on directory usage, see the discussion in the - * Developer Guide. + * Developer Guide. *

*

* Directories are identified by hierarchical paths analogous to the paths diff --git a/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java b/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java index 229dcb8cd2..8dc9eb3c6f 100644 --- a/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java +++ b/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java @@ -38,7 +38,7 @@ import com.apple.foundationdb.tuple.Tuple; *

* * For general guidance on partition usage, see - * The Developer Guide. + * The Developer Guide. */ class DirectoryPartition extends DirectorySubspace { diff --git a/bindings/java/src/main/com/apple/foundationdb/directory/package-info.java b/bindings/java/src/main/com/apple/foundationdb/directory/package-info.java index 568ecd5a68..206189cd0e 100644 --- a/bindings/java/src/main/com/apple/foundationdb/directory/package-info.java +++ b/bindings/java/src/main/com/apple/foundationdb/directory/package-info.java @@ -31,7 +31,7 @@ * used for the corresponding subspace. In effect, directories provide * a level of indirection for access to subspaces.
*
- * See general + * See general * directory documentation for information about how directories work and * interact with other parts of the built-in keyspace management features. */ diff --git a/bindings/java/src/main/com/apple/foundationdb/package-info.java b/bindings/java/src/main/com/apple/foundationdb/package-info.java index 0cef5cace3..953390b019 100644 --- a/bindings/java/src/main/com/apple/foundationdb/package-info.java +++ b/bindings/java/src/main/com/apple/foundationdb/package-info.java @@ -29,7 +29,7 @@ * assure that {@link com.apple.foundationdb.Transaction#commit()} has returned successfully * before itself returning. If you are not able to use these functions for some reason * please closely read and understand the other - * developer + * developer * documentation on FoundationDB transactions. */ package com.apple.foundationdb; diff --git a/bindings/java/src/main/com/apple/foundationdb/subspace/Subspace.java b/bindings/java/src/main/com/apple/foundationdb/subspace/Subspace.java index 328dd915ed..4de24eb907 100644 --- a/bindings/java/src/main/com/apple/foundationdb/subspace/Subspace.java +++ b/bindings/java/src/main/com/apple/foundationdb/subspace/Subspace.java @@ -38,7 +38,7 @@ import com.apple.foundationdb.tuple.Versionstamp; * *

* For general guidance on subspace usage, see the discussion in - * Developer Guide. + * Developer Guide. *

* *

diff --git a/bindings/java/src/main/com/apple/foundationdb/subspace/package-info.java b/bindings/java/src/main/com/apple/foundationdb/subspace/package-info.java index 2379d99d32..ee499123bb 100644 --- a/bindings/java/src/main/com/apple/foundationdb/subspace/package-info.java +++ b/bindings/java/src/main/com/apple/foundationdb/subspace/package-info.java @@ -26,7 +26,7 @@ * from the result. As a best practice, API clients should use at least one * subspace for application data.
*
- * See general + * See general * subspace documentation for information about how subspaces work and * interact with other parts of the built-in keyspace management features. */ diff --git a/bindings/java/src/main/com/apple/foundationdb/tuple/package-info.java b/bindings/java/src/main/com/apple/foundationdb/tuple/package-info.java index 2adb752d6a..1025e793ee 100644 --- a/bindings/java/src/main/com/apple/foundationdb/tuple/package-info.java +++ b/bindings/java/src/main/com/apple/foundationdb/tuple/package-info.java @@ -22,7 +22,7 @@ * Provides a set of utilities for serializing and deserializing typed data * for use in FoundationDB. When packed together into a {@link com.apple.foundationdb.tuple.Tuple} * this data is suitable for use as an index or organizational structure within FoundationDB - * keyspace. See general Tuple + * keyspace. See general Tuple * documentation for information about how Tuples sort and can be used to efficiently * model data. */ diff --git a/bindings/java/src/main/overview.html.in b/bindings/java/src/main/overview.html.in index 5127b460c1..47860d27d1 100644 --- a/bindings/java/src/main/overview.html.in +++ b/bindings/java/src/main/overview.html.in @@ -69,7 +69,7 @@ cross-language support for storing and retrieving typed data from the binary data that FoundationDB supports. And, just as importantly, data packed into {@code Tuple}s and used as keys sort in predictable and useful ways. See the {@link com.apple.foundationdb.tuple Tuple class documentation} for information about use in Java -and general Tuple documentation +and general Tuple documentation for information about how Tuples sort and can be used to efficiently model data.

FoundationDB {@link com.apple.foundationdb.directory Directory API}

From 8af451dae5014649f6f9e0c1b7be44209777e959 Mon Sep 17 00:00:00 2001 From: Alvin Moore Date: Tue, 20 Mar 2018 15:43:14 -0700 Subject: [PATCH 18/26] Added clean targets Merged upstream into branch --- Makefile | 1 + bindings/c/local.mk | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/Makefile b/Makefile index d679dc7888..05e29dc644 100644 --- a/Makefile +++ b/Makefile @@ -150,6 +150,7 @@ clean: $(CLEAN_TARGETS) docpreview_clean @rm -rf $(DEPSDIR) @rm -rf lib/libstdc++.a @rm -rf bin/coverage.*.xml + @rm -f packages/cie-foundationdb-backup_$(VERSION)-$(PKGRELEASE).tar.gz packages/cie-foundationdb-client_$(VERSION)-$(PKGRELEASE).tar.gz packages/cie-foundationdb-python_$(VERSION)-$(PKGRELEASE).tar.gz packages/cie-foundationdb-server_$(VERSION)-$(PKGRELEASE).tar.gz targets: @echo "Available targets:" diff --git a/bindings/c/local.mk b/bindings/c/local.mk index 0b0301180c..7bab3d9919 100644 --- a/bindings/c/local.mk +++ b/bindings/c/local.mk @@ -26,6 +26,8 @@ fdb_c_LIBS := lib/libfdbclient.a lib/libfdbrpc.a lib/libflow.a fdb_c_tests_LIBS := -Llib -lfdb_c fdb_c_tests_HEADERS := -Ibindings/c +CLEAN_TARGETS += fdb_c_tests_clean + ifeq ($(PLATFORM),linux) fdb_c_LIBS += lib/libstdc++.a -lm -lpthread -lrt -ldl fdb_c_LDFLAGS += -Wl,--version-script=bindings/c/fdb_c.map -static-libgcc -Wl,-z,nodelete @@ -98,4 +100,7 @@ packages/fdb-c-tests-$(VERSION)-$(PLATFORM).tar.gz: bin/fdb_c_performance_test b fdb_c_tests: packages/fdb-c-tests-$(VERSION)-$(PLATFORM).tar.gz +fdb_c_tests_clean: + @rm -f packages/fdb-c-tests-$(VERSION)-$(PLATFORM).tar.gz + packages: fdb_c_tests From f23df874cea07658ce0891aa370a35fd788c42ac Mon Sep 17 00:00:00 2001 From: Yichi Chiang Date: Tue, 20 Mar 2018 16:45:34 -0700 Subject: [PATCH 19/26] Fix review comments --- bindings/java/src/main/com/apple/foundationdb/Transaction.java | 2 +- .../com/apple/foundationdb/directory/DirectoryPartition.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/bindings/java/src/main/com/apple/foundationdb/Transaction.java b/bindings/java/src/main/com/apple/foundationdb/Transaction.java index 7cf3cdc9ea..e2af113a5b 100644 --- a/bindings/java/src/main/com/apple/foundationdb/Transaction.java +++ b/bindings/java/src/main/com/apple/foundationdb/Transaction.java @@ -39,7 +39,7 @@ import com.apple.foundationdb.tuple.Tuple; * Transactions group operations into a unit with the properties of atomicity, isolation, and * durability. Transactions also provide the ability to maintain an application's invariants or * integrity constraints, supporting the property of consistency. Together these properties are - * known as acid.
+ * known as ACID.
*
* Transactions are also causally consistent: once a transaction has been successfully committed, * all subsequently created transactions will see the modifications made by it. diff --git a/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java b/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java index 8dc9eb3c6f..6509984b3b 100644 --- a/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java +++ b/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java @@ -38,7 +38,7 @@ import com.apple.foundationdb.tuple.Tuple; *

* * For general guidance on partition usage, see - * The Developer Guide. + * Developer Guide. */ class DirectoryPartition extends DirectorySubspace { From 6fbef3ab394db0018d15a8b51aa38b24928e110f Mon Sep 17 00:00:00 2001 From: bnamasivayam <36455962+bnamasivayam@users.noreply.github.com> Date: Tue, 20 Mar 2018 17:24:47 -0700 Subject: [PATCH 20/26] Change Capitalization at certain places. --- documentation/sphinx/source/configuration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/sphinx/source/configuration.rst b/documentation/sphinx/source/configuration.rst index 08904f6632..742598af1c 100644 --- a/documentation/sphinx/source/configuration.rst +++ b/documentation/sphinx/source/configuration.rst @@ -521,7 +521,7 @@ Guidelines for setting process class In a FoundationDB cluster, each of the ``fdbserver`` processes perform different tasks. Each process is recruited to do a particular task based on its process ``class``. For example, processes with ``class=storage`` are given preference to be recruited for doing storage server tasks, ``class=transaction`` are for log server processes and ``class=stateless`` are for stateless processes like proxies, resolvers, etc., -The recommended minimum number of ``class=transaction`` (log server) processes is 8 (active) + 2 (standby) and the recommended minimum number for ``class=stateless`` processes is 4 (proxy) + 1 (resolver) + 1 (Cluster Controller) + 1 (Master) + 2 (standby). It is better to spread the transaction and stateless processes across as many machines as possible. +The recommended minimum number of ``class=transaction`` (log server) processes is 8 (active) + 2 (standby) and the recommended minimum number for ``class=stateless`` processes is 4 (proxy) + 1 (resolver) + 1 (cluster controller) + 1 (master) + 2 (standby). It is better to spread the transaction and stateless processes across as many machines as possible. ``fdbcli`` is used to set the desired number of processes of a particular process type. To do so, you would issue the ``fdbcli`` commands:: From 4da519bc4a7cbdb76387fab79d4f524738ad8f2a Mon Sep 17 00:00:00 2001 From: John Brownlee Date: Wed, 21 Mar 2018 08:58:00 -0700 Subject: [PATCH 21/26] Removes the logo from our sphinx template. --- documentation/sphinx/conf.py | 1 - 1 file changed, 1 deletion(-) diff --git a/documentation/sphinx/conf.py b/documentation/sphinx/conf.py index 70fbef16d0..b0fad9da5b 100644 --- a/documentation/sphinx/conf.py +++ b/documentation/sphinx/conf.py @@ -134,7 +134,6 @@ html_title = 'FoundationDB ' + version # The name of an image file (relative to this directory) to place at the top # of the sidebar. -html_logo = sys.prefix + '/_static/logo.svg' # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 From 5e95497401ea48329eaef8520b0d659a3ca7b2f1 Mon Sep 17 00:00:00 2001 From: Alvin Moore Date: Wed, 21 Mar 2018 10:55:43 -0700 Subject: [PATCH 22/26] Removed package version from documentation file name --- Makefile | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 05e29dc644..59b63d9b5a 100644 --- a/Makefile +++ b/Makefile @@ -181,13 +181,13 @@ docpreview: javadoc godoc docpreview_clean: CLEAN_TARGETS= $(MAKE) -C documentation docpreview_clean -packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz: FORCE javadoc godoc +packages/foundationdb-docs-$(VERSION).tar.gz: FORCE javadoc godoc TARGETS= $(MAKE) -C documentation docpackage @mkdir -p packages - @rm -f packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz - @cp documentation/sphinx/.dist/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz + @rm -f packages/foundationdb-docs-$(VERSION).tar.gz + @cp documentation/sphinx/.dist/foundationdb-docs-$(VERSION).tar.gz packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz -docpackage: packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz +docpackage: packages/foundationdb-docs-$(VERSION).tar.gz FORCE: From 3fca78c0f91f661b46bae6178e3163bc0772cf4e Mon Sep 17 00:00:00 2001 From: "A.J. Beamon" Date: Wed, 21 Mar 2018 11:11:18 -0700 Subject: [PATCH 23/26] Update docs for patch release 5.1.1-5.1.4 --- documentation/sphinx/source/downloads.rst | 24 ++++----- documentation/sphinx/source/release-notes.rst | 53 +++++++++++++++++++ 2 files changed, 65 insertions(+), 12 deletions(-) diff --git a/documentation/sphinx/source/downloads.rst b/documentation/sphinx/source/downloads.rst index c7c0f21267..edc604ecd4 100644 --- a/documentation/sphinx/source/downloads.rst +++ b/documentation/sphinx/source/downloads.rst @@ -10,38 +10,38 @@ macOS The macOS installation package is supported on macOS 10.7+. It includes the client and (optionally) the server. -* `FoundationDB-5.1.0.pkg `_ +* `FoundationDB-5.1.4.pkg `_ Ubuntu ------ The Ubuntu packages are supported on 64-bit Ubuntu 12.04+, but beware of the Linux kernel bug in Ubuntu 12.x. -* `foundationdb-clients-5.1.0-1_amd64.deb `_ -* `foundationdb-server-5.1.0-1_amd64.deb `_ (depends on the clients package) +* `foundationdb-clients-5.1.4-1_amd64.deb `_ +* `foundationdb-server-5.1.4-1_amd64.deb `_ (depends on the clients package) RHEL/CentOS EL6 --------------- The RHEL/CentOS EL6 packages are supported on 64-bit RHEL/CentOS 6.x. -* `foundationdb-clients-5.1.0-1.el6.x86_64.rpm `_ -* `foundationdb-server-5.1.0-1.el6.x86_64.rpm `_ (depends on the clients package) +* `foundationdb-clients-5.1.4-1.el6.x86_64.rpm `_ +* `foundationdb-server-5.1.4-1.el6.x86_64.rpm `_ (depends on the clients package) RHEL/CentOS EL7 --------------- The RHEL/CentOS EL7 packages are supported on 64-bit RHEL/CentOS 7.x. -* `foundationdb-clients-5.1.0-1.el7.x86_64.rpm `_ -* `foundationdb-server-5.1.0-1.el7.x86_64.rpm `_ (depends on the clients package) +* `foundationdb-clients-5.1.4-1.el7.x86_64.rpm `_ +* `foundationdb-server-5.1.4-1.el7.x86_64.rpm `_ (depends on the clients package) Windows ------- The Windows installer is supported on 64-bit Windows XP and later. It includes the client and (optionally) the server. -* `foundationdb-5.1.0-x64.msi `_ +* `foundationdb-5.1.4-x64.msi `_ API Language Bindings ===================== @@ -58,18 +58,18 @@ On macOS and Windows, the FoundationDB Python API bindings are installed as part If you need to use the FoundationDB Python API from other Python installations or paths, download the Python package: -* `foundationdb-5.1.0.tar.gz `_ +* `foundationdb-5.1.4.tar.gz `_ Ruby 1.9.3/2.0.0+ ----------------- -* `fdb-5.1.0.gem `_ +* `fdb-5.1.4.gem `_ Java 8+ ------- -* `fdb-java-5.1.0.jar `_ -* `fdb-java-5.1.0-javadoc.jar `_ +* `fdb-java-5.1.4.jar `_ +* `fdb-java-5.1.4-javadoc.jar `_ Go 1.1+ ------- diff --git a/documentation/sphinx/source/release-notes.rst b/documentation/sphinx/source/release-notes.rst index 385daf9982..479c104469 100644 --- a/documentation/sphinx/source/release-notes.rst +++ b/documentation/sphinx/source/release-notes.rst @@ -2,6 +2,59 @@ Release Notes ############# +5.1.4 +===== + +Fixes +----- + +* The master would recover twice when a new cluster controller was elected. +* The cluster controller could be elected on a storage process after restarting all processes in a cluster. +* Allow backup expiration to succeed if the backup is too new to be restorable. +* Process metric collection in status could sometimes fail. + +5.1.3 +===== + +Fixes +----- + +* The backup agents ran out of memory when heavily loaded. +* Storage servers were not marked as failed until after their files were deleted. +* The consistency check requested too many shards in the same request from the proxy. +* Client knobs for blob send/receive were reversed in meaning. +* fdbbackup status provides more information on reported errors. + +5.1.2 +===== + +Fixes +----- + +* Backup did not incrementally delete mutations from the mutation log. +* fdbcli status misreported completed backup/DR as running. +* Stopped producing the "fdbblob" alias for fdbbackup. + +5.1.1 +===== + +Fixes +----- + +* Bindings: Disposing a transaction during a commit resulted in a broken promise from ``get_versionstamp``. +* Bindings: Calling ``create_cluster`` before initializing the network would result in a crash. +* Latest restorable version of a running backup was not being updated in backup layer status.  +* Backup layer status would sometimes show an error or an incorrect value for the recent blob bandwidth metric. +* Backup deletions were not deleting all of the files related to the backup. +* The cluster controller was sharing a process with the master even when better locations existed. +* Blob credentials files were being opened in read-write mode. +* Sometimes fdbbackup did not write log files even when ``--log`` was passed on the command line. + +Performance +----------- + +* Backup file uploads will respond to server-side throttling in the middle of a chunk upload rather than only between chunks. + 5.1.0 ===== From cc5149949dd7f0d4a473796f00e45ff612e8a71e Mon Sep 17 00:00:00 2001 From: Alvin Moore Date: Wed, 21 Mar 2018 12:24:06 -0700 Subject: [PATCH 24/26] Removed the version id (git hash) from the document package name --- documentation/sphinx/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/sphinx/Makefile b/documentation/sphinx/Makefile index eaa5937f0d..f878d0b449 100644 --- a/documentation/sphinx/Makefile +++ b/documentation/sphinx/Makefile @@ -102,5 +102,5 @@ livehtml: html # removed html prerequisite because it is previously explictly invoked package: mkdir -p $(DISTDIR) - rm -f $(DISTDIR)/$(PROJECT_NAME)-$(VERSION)-$(PKGRELEASE).tar.gz - cd $(BUILDDIR)/html && tar czf $(DISTDIR)/$(PROJECT_NAME)-$(VERSION)-$(PKGRELEASE).tar.gz . + rm -f $(DISTDIR)/$(PROJECT_NAME)-$(VERSION).tar.gz + cd $(BUILDDIR)/html && tar czf $(DISTDIR)/$(PROJECT_NAME).tar.gz . From 53438477c1d6b1400591094fed30c1b398c82a91 Mon Sep 17 00:00:00 2001 From: AlvinMooreSr <36203359+AlvinMooreSr@users.noreply.github.com> Date: Thu, 22 Mar 2018 07:50:56 -0700 Subject: [PATCH 25/26] Documenation Package Fix (#113) * Moved clean up to external makefile * Fixed the name of the documentation package * Fixed name of the documentation package --- Makefile | 3 +-- documentation/sphinx/Makefile | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 59b63d9b5a..a2311cc52d 100644 --- a/Makefile +++ b/Makefile @@ -150,7 +150,6 @@ clean: $(CLEAN_TARGETS) docpreview_clean @rm -rf $(DEPSDIR) @rm -rf lib/libstdc++.a @rm -rf bin/coverage.*.xml - @rm -f packages/cie-foundationdb-backup_$(VERSION)-$(PKGRELEASE).tar.gz packages/cie-foundationdb-client_$(VERSION)-$(PKGRELEASE).tar.gz packages/cie-foundationdb-python_$(VERSION)-$(PKGRELEASE).tar.gz packages/cie-foundationdb-server_$(VERSION)-$(PKGRELEASE).tar.gz targets: @echo "Available targets:" @@ -185,7 +184,7 @@ packages/foundationdb-docs-$(VERSION).tar.gz: FORCE javadoc godoc TARGETS= $(MAKE) -C documentation docpackage @mkdir -p packages @rm -f packages/foundationdb-docs-$(VERSION).tar.gz - @cp documentation/sphinx/.dist/foundationdb-docs-$(VERSION).tar.gz packages/foundationdb-docs-$(VERSION)-$(PKGRELEASE).tar.gz + @cp documentation/sphinx/.dist/foundationdb-docs-$(VERSION).tar.gz packages/foundationdb-docs-$(VERSION).tar.gz docpackage: packages/foundationdb-docs-$(VERSION).tar.gz diff --git a/documentation/sphinx/Makefile b/documentation/sphinx/Makefile index f878d0b449..e72b0d63f8 100644 --- a/documentation/sphinx/Makefile +++ b/documentation/sphinx/Makefile @@ -103,4 +103,4 @@ livehtml: html package: mkdir -p $(DISTDIR) rm -f $(DISTDIR)/$(PROJECT_NAME)-$(VERSION).tar.gz - cd $(BUILDDIR)/html && tar czf $(DISTDIR)/$(PROJECT_NAME).tar.gz . + cd $(BUILDDIR)/html && tar czf $(DISTDIR)/$(PROJECT_NAME)-$(VERSION).tar.gz . From 5a63af8f3f187bc16e5fcbb7a7775cfdbcf909b7 Mon Sep 17 00:00:00 2001 From: Yichi Chiang Date: Thu, 22 Mar 2018 09:22:13 -0700 Subject: [PATCH 26/26] Add 'the' word back in javadoc in DirectoryPartition.java --- .../com/apple/foundationdb/directory/DirectoryPartition.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java b/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java index 6509984b3b..22a479f1e1 100644 --- a/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java +++ b/bindings/java/src/main/com/apple/foundationdb/directory/DirectoryPartition.java @@ -37,7 +37,7 @@ import com.apple.foundationdb.tuple.Tuple; * content. *

* - * For general guidance on partition usage, see + * For general guidance on partition usage, see the * Developer Guide. */ class DirectoryPartition extends DirectorySubspace {