Commit Graph

713 Commits

Author SHA1 Message Date
A.J. Beamon 050c1bcba6 Update transaction_too_old error text. 2017-10-18 08:49:17 -07:00
Stephen Atherton ef84e52127 Improved error handling and memory usage in AsyncFileBlobStoreWrite. Writes will now fail if any upload has already failed, rather than buffering unboundedly until sync() is called to complete the file. There is also a configurable limit on how many uploads can be pending before writes will stall waiting for one to finish. 2017-10-18 05:51:30 -07:00
Stephen Atherton ebd0234514 Rewrote most error handling in BlobStoreEndpoint to fix several shortcomings in error handling and logging. The request loop now logs but rate limits all errors, and the exceptions thrown are more appropriate. HTTP 503 is now treated as retryable. Callers of BlobStoreEndpoint::doRequest() now specify which codes they consider to be successful so that more error handling can take place in the main request loop. 2017-10-18 02:52:09 -07:00
Alec Grieser 479c9fab1b added experimental warning to external callbacks docs 2017-10-17 13:18:58 -07:00
Alec Grieser 191aca0a3c removed reference to cie in vexillographer.csproj 2017-10-17 09:30:53 -07:00
Alec Grieser fc2ff719d0 fixed some capitalization issues that slid in through the case-insenstive filesystems 2017-10-17 09:07:05 -07:00
Alex Miller 7b9bc1d715 Merge pull request #170 from cie/alexmiller/flowprofile
Add support for profiling a running fdb cluster to fdbcli, fix security issues, and add an improved backtrace.
2017-10-16 16:51:53 -07:00
Alex Miller cf646d4a99 Address review comments.
* Fixed fdbcli to be more idiomatic.
* Removed is_binary_serializable in favor of std::is_pod<>
* Removed custom enable_if<> in favor of std::enable_if<>
* Removed HEY REVIEWER comments
* Removed print from prof.py
* Added FLOW_PROFILER_ENABLED=yes to circus components that wished to enable the flow profiler.
2017-10-16 16:46:52 -07:00
Yichi Chiang a6ae89af1a Merge pull request #176 from cie/add-cluster-controller-process-class
Add cluster controller process class
2017-10-16 16:27:54 -07:00
Yichi Chiang af2aa41136 Downgrade Transaction process class for cluster controller 2017-10-16 16:27:01 -07:00
Yichi Chiang 76c5488421 Add cluster controller process class 2017-10-16 16:21:25 -07:00
Alex Miller 16e5b50685 Replace backtrace with absl::GetStackTrace on non-MacOS platforms.
backtrace() gives a list of return addresses, which means that addr2line will
print out the line after the caller. GetStackTrace returns the list of caller
addresses, so the addr2line results should be accurate.  The flow profiler was
also changed to use the new backtracing code, so flow profiles will now be
accurate as well.  Unfortunately, the abseil code doesn't work on MacOS, so we
still fall back to backtrace() in this case.

For the stack unwinder to work, we must disable -fomit-frame-pointer.  This can
result in a small performance penalty, as it effectively reduces the number of
general purpose registers available by one.  (I'm also curious if this has
anything to do with the overly frequent "<value optimized out>" messages from
gdb.)  If this shows up as a problem, we can make release builds still have
-fomit-frame-pointer, and fall back to backtrace when it's enabled then as
well.
2017-10-16 16:05:02 -07:00
Alex Miller 1a91aab1d7 Import //base/debugging:stacktrace from abseil.
This code is all Apache 2 licensed, and all headers were maintained when
concatinated, so we should be completely fine from a legal standpoint.

I've scriptified the steps that I took so that if we need to update this code
in the future, it hopefully shouldn't be too much of a hassle.
2017-10-16 16:05:02 -07:00
Alex Miller f997cb9038 Add a string knob to hold the Log directory, and write profiles to it.
This is the combination of two small changes.

1. Add support for a string knob type.
2. Change profiles to be written to the log directory instead of the working
   directory.

We have three options of where to write files: the working directory, the data
directory, and the log directory.

The working directory may be set to a non-writable location, and likely
contains the fdb binaries.  Allowing these files to be overwritten would likely
not be a wise idea.

The data directory hosts our sqlite b-trees.  It would also be very unfortunate
if these were ever overwritten by an unfortunate profile name.

The log directory contains logs.  Out of the three, these matter the least if
they disappear or become corrupted.

Thus, we write to the log directory.
2017-10-16 16:05:02 -07:00
Alex Miller c5fbe33df6 Disallow arbitrary paths for storing profiles.
Previously, one could request profiles to be stored at
"../../../../../../etc/passwd".  Now we expand the paths, including symlinks,
and ensure that the target is a child of the targetted subdirectory.  This was
the least convoluted way I could figure out to handle paths.
2017-10-16 16:05:02 -07:00
Alex Miller 91a26a170c Add toggleable profiling support to fdbserver+fdbcli.
This adds the fdbcli commands:
* profile list -- Lists all workers in a way that doesn't fill `kill`'s list.
* profile flow run -- Allows starting flow profiling on a set of hosts for a specified interval.

And threads through all the support for enabling and disabling profiling as an RPC.
2017-10-16 16:05:02 -07:00
Alvin Moore 770d9663d8 Prepared for new release 5.0.6
Updated WIX Product Id
Updated Artifactory links
2017-10-16 11:25:14 -07:00
Balachandar Namasivayam 312f614133 Add the new ops and AND to NON_ASSOCIATIVE_MASK.
In the storage server, read the entire value if the op is ByteMin or ByteMax.
2017-10-16 11:06:31 -07:00
Alec Grieser e0be1ef1e0 Merge branch 'release-5.0' 2017-10-16 10:08:11 -07:00
Alec Grieser 432726ba2d Merge branch 'release-4.6' into release-5.0 2017-10-16 09:54:21 -07:00
Alec Grieser d40eb1ef9a changed java package from com.apple.cie.foundationdb to com.apple.foundationdb 2017-10-16 08:31:44 -07:00
Alec Grieser 18432f235f moved java files from com/apple/apple/foundationdbdb to com/apple/foundationdb in preparation for package rename 2017-10-16 08:11:41 -07:00
Stephen Atherton 68eccb681e Merge pull request #173 from bmuppana/master
Backup log messages.
2017-10-13 18:31:53 -07:00
Evan Tschannen 1f90a27acf Merge pull request #174 from bmuppana/fix-windows
Fixing windows build. Fixing backup.actor.cpp
2017-10-13 17:43:56 -07:00
Bhaskar Muppana 36b15f029d Fixing windows build. Fixing backup.actor.cpp 2017-10-13 17:41:17 -07:00
Evan Tschannen 215bcb8d3e Merge pull request #157 from cie/choose-leader-on-stateless-processes
Catch and update processClass change from DBSource
2017-10-13 14:03:29 -07:00
Yichi Chiang 5bcdd37c0d Move UID generation and add initialClass 2017-10-13 13:46:37 -07:00
Alvin Moore b075fb30a0 Merge branch 'release-5.0'
Removed TLS from repository and packaging
2017-10-13 12:38:46 -07:00
Yichi Chiang 12edd27281 Introduce prevChangeID to CandidacyRequest and LeaderHeartbeatRequest 2017-10-12 17:11:58 -07:00
Bhaskar Muppana d1e9d28239 Backup log messages. 2017-10-12 16:12:42 -07:00
Balachandar Namasivayam 3aaa11977e Addressed Review Comments 2017-10-12 14:56:00 -07:00
Stephen Atherton ad0ed79d36 Merge pull request #172 from bmuppana/backup-refactor
Backup refactoring
2017-10-12 11:38:49 -07:00
Stephen Atherton 659e39103e Missed file from merge of master into backup-refactor 2017-10-12 11:25:29 -07:00
Stephen Atherton 11517f7bfc Merge branch 'master' into continuous-backup
# Conflicts:
#	fdbclient/FileBackupAgent.actor.cpp
2017-10-12 11:03:23 -07:00
A.J. Beamon b20ae356b1 Alloc instrumentation backtraces use format_backtrace; Magnesium detects backtraces from binaries besides fdbserver. 2017-10-12 08:39:13 -07:00
Stephen Atherton 40afa6a9b3 Merge pull request #151 from bmuppana/backup
Backup Refactoring
2017-10-11 22:40:32 -07:00
Balachandar Namasivayam fd4e62d4c9 Add documentation for the new atomic ops byte_min an byte_max as well as changing description for min and max atomic op. 2017-10-11 18:43:19 -07:00
Alex Miller 9648f96200 Also fix unforwarded Metric in IndexedSet.
This is simply an exceedingly minor performance fix rather than a correctness issue.
2017-10-11 17:40:48 -07:00
Alex Miller c24b941485 Fix erroneous std::move in indexed set, and clean up addMetric users.
This is a follow-on to c4eb73d0.  Thanks to Bala for pointing out the unchanged
std::move usage, and there appeared to not be many existing users of addMetric
anyway.
2017-10-11 17:36:51 -07:00
Alec Grieser 09e54aa23b ~~whitespace fixes~~ 2017-10-11 14:54:49 -07:00
Alec Grieser 2722e6d99e added a helper method to fdb.Subspace and updated some docs 2017-10-11 14:48:09 -07:00
Balachandar Namasivayam 8e0bea2795 Update API_VERSION from 500 to 510 2017-10-11 13:49:38 -07:00
Alec Grieser 5e44aa06e4 python tester now sometimes uses has_incomplete_versionstamp 2017-10-11 11:00:16 -07:00
Alec Grieser 9fd934e002 added packWithVersionstamp to subspace ; some testing/javadocs tweaks 2017-10-11 10:53:51 -07:00
Alec Grieser f95553aca2 updated javadocs 2017-10-10 16:56:32 -07:00
Stephen Atherton c3d8412abb Merge pull request #166 from cie/alexmiller/deathservice
Fix potential division by zero issues via RPC.
2017-10-10 16:47:38 -07:00
Evan Tschannen 8feb3b8fbc fixed conflict range workload by just disabling timeKeeper instead of the check, because it should be a more robust fix 2017-10-10 16:01:02 -07:00
Balachandar Namasivayam eeebf10030 Modified existing behavior of MIN and AND atomic ops. The new behavior results in a 'SET' if the atomic op is performed on a non -existing key.
Added new atomic ops ByteMin and ByteMax that does lexicographic comparison of byte strings.
2017-10-10 13:02:22 -07:00
Evan Tschannen c8525dc3e7 timekeeper is constantly changing keys in the system keyspace, so do not report errors on key mismatches on keys in the system keyspace 2017-10-10 12:04:56 -07:00
Alec Grieser fc97fa3c8d added documentation for versionstamped tuples to python docs 2017-10-10 09:12:42 -07:00