Commit Graph

103 Commits

Author SHA1 Message Date
Stephen Atherton 2efb6f4c0d Added cleanPath() which puts a path in a canonical form without .., ., or duplicate separators without using the filesystem or resolving symbolic links. absPath() redefined to use cleanPath() so it will return the same result for a path without symbolic links regardless of whether or not the path actually exists. Redefined parentDirectory() to use absPath() and error on certain inputs. Added comments describing behavior of these functions, and added a unit test which verbosely tests many inputs to them. 2019-03-15 23:54:33 -07:00
A.J. Beamon 809861ee88 Merge branch 'master' into fix-parent-directory
# Conflicts:
#	flow/Platform.cpp
2019-03-15 10:51:12 -07:00
A.J. Beamon 85b3f11e71 Fix various compiler warnings 2019-03-15 10:34:57 -07:00
Alex Miller bbd5cc2e95 Remove duplicate include due to concurrent update to same file
Co-Authored-By: mpilman <markus@pilman.ch>
2019-03-11 20:42:35 -07:00
mpilman bc4820865e Fix some Windows issues
1. We need to set a Windows target to get rid of warnings when compiling
   files that include windows.h
2. By default Boost will try to automatically link against boost_system.
   In order to prevent this, we define BOOST_ALL_NO_LIB
3. Somehow, I had to include winioctl.h in Platform.cpp. According to
   the documentation from MS, this shouldn't be necessary as windows.h
   includes this as well. However, for me it didn't compile otherwise.
2019-03-11 20:42:35 -07:00
Alex Miller 59f598ab51 Try adding a header? 2019-03-08 21:46:32 -08:00
Vishesh Yadav 57832e625d net: Support IPv6 #963
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.

- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.

- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.

- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
mpilman 7a858b902d Most of FDB compiling on Windows with cmake 2019-02-07 15:37:04 -08:00
A.J. Beamon efe06d71ce Change profilingEnabled to thread_local to eliminate a race. 2019-01-24 13:27:16 -08:00
Evan Tschannen 684a22a52b Merge branch 'release-6.0'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/BackupContainer.actor.cpp
#	fdbclient/HTTP.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/BackupCorrectness.actor.cpp
#	versions.target
2019-01-09 16:14:46 -08:00
A.J. Beamon d265517156 Fix: fast allocator would not cleanup memory for a thread if that thread never called getMagazine. This could happen if the first thing the thread did was to release memory.
Added a new metric for the number of threads that hold memory for each size and improve some existing metrics.
Fix: a failed ASSERT would crash if done early in the program lifetime.
2019-01-08 14:36:01 -08:00
Andrew Noyes 75fd8e6e17 Include <sys/sysmacros.h> 2019-01-03 12:44:29 -08:00
Markus Pilman dbe9baff1f Several small compilation fixes for new versions of gcc
There are several missing includes for cmath in the code, I added those.

Next, Coro returns a reference to a stack variable and this causes a
warning. As this is probably ok for Coro, I disabled the warning in
that file for GCC. I want to have this warning in the build system as
it is generally a very useful warning to have.

Another change is that major and minor are deprecated for a while now.
I replaced those with gnu_dev_major and gnu_dev_minor.

ErrorOr currently implements operators ==, !=, and <. These do not
compile because Error does not implement ==. This compiles on older
versions of gcc and clang because ErrorOr<T>::operator== is not used
anywhere. It is still wrong though and newer gcc versions complain.
I simply removed these methods.

The most interesting fix is that TraceEvent::~TraceEvent is currently
throwing exceptions. This is illegal behavior in C++11 and a idea in
older versions of C++. For now I simply removed the throw, but this
might need some more thought.
2019-01-03 12:44:19 -08:00
Stephen Atherton f64d5321e9 Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL. 2018-12-20 18:05:23 -08:00
Evan Tschannen d2d68aa171 Merge branch 'release-6.0'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/ManagementAPI.actor.cpp
#	versions.target
2018-12-03 18:26:52 -08:00
Evan Tschannen 55a9c4a0f0
Merge pull request #955 from ajbeamon/fix-bad-error-creation-and-whitespace
throw platform_error; -> throw platform_error();. Convert some spaces to tabs.
2018-12-03 15:12:37 -08:00
Stephen Atherton e2673b2bbc Fixed typo causing mac build failure. 2018-12-02 22:03:21 -08:00
A.J. Beamon 50c9dfdd01 Errors that occur in platform that are the result of IO issues are now raised as io_error rather than platform_error. 2018-11-30 10:55:19 -08:00
A.J. Beamon 97847f517b throw platform_error; -> throw platform_error();. Convert some spaces to tabs. 2018-11-28 12:56:57 -08:00
Alex Miller 6bb1f4093d
Merge pull request #856 from dropbox/pr/include-fix
Adjust all includes to be relative to the root.
2018-10-22 09:51:55 -07:00
Evan Tschannen 888378c8cb fixed windows build 2018-10-19 13:28:32 -07:00
Robert Escriva 268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
Evan Tschannen 8dd900a337 fixed the windows build 2018-10-18 20:26:45 -07:00
Stephen Atherton 22f8a4efa9 Normalized all unit test names to begin with "/" if they should be included in random unit testing. 2018-10-05 22:09:58 -07:00
Evan Tschannen 1c29275672 call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details. 2018-08-01 14:30:57 -07:00
Stephen Atherton e17c04f666 Changed assignments to substr() to resize() for clarity/simplicity. 2018-07-02 14:25:24 -07:00
A.J. Beamon 7378ec85f0 Fixes parentDirectory to work when our path is a child of the root directory. Also works when the path has trailing slashes. 2018-06-26 16:30:47 -07:00
A.J. Beamon f965954122 Merge commit '82be52205b95464e355c449fdf3e7d483fa06677' into trace-log-refactor
# Conflicts:
#	fdbserver/Status.actor.cpp
#	fdbserver/workloads/DDMetrics.actor.cpp
#	flow/Trace.cpp
2018-06-08 16:22:22 -07:00
Balachandar Namasivayam 8360f71cbb Merge branch 'master' of github.com:apple/foundationdb into save-fitness-info
# Conflicts:
#	fdbserver/worker.actor.cpp
2018-06-08 16:09:59 -07:00
A.J. Beamon 99c9958db7 Some more trace event normalization 2018-06-08 13:57:00 -07:00
A.J. Beamon 0ca51989bb Merge branch 'master' into trace-log-refactor
# Conflicts:
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/Status.actor.cpp
#	flow/Trace.cpp
2018-06-08 13:24:30 -07:00
A.J. Beamon e5488419cc Attempt to normalize trace events:
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.

Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.

This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
Balachandar Namasivayam 11b79c6c94 Save fitness info of a process to become a cluster controller. This info is currently lost after a reboot. Save this info and reload it to avoid unnecessary re-recruitments. 2018-06-07 13:07:19 -07:00
A.J. Beamon ce0c991e78 Refactor trace events to store a vector of fields that aren't encoded until write time. Better support for pre-network trace events. Rework how trace events are queried. Some initial work towards pluggable formatting of logs. 2018-05-02 10:44:38 -07:00
Alec Grieser a1faaafca3
Merge remote-tracking branch 'upstream/release-5.1' into merge-release-5.1 2018-04-27 16:38:18 -07:00
Alex Miller 11bd7d7daf Add a disgusting and terrible hack to avoid "undefined std::istream::ignore(long)" linking errors.
This makes building our bindings in the provided docker image possible again.
2018-04-26 16:07:50 -07:00
A.J. Beamon 06dd052770 Reduce severity of GetDiskStatisticsDeviceNotFound to SevWarn. 2018-03-14 12:17:54 -07:00
Alec Grieser e1162e9238 Merge remote-tracking branch 'upstream/release-5.1' 2018-02-22 11:16:12 -08:00
Alec Grieser 0bae9880f1 remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py 2018-02-21 10:25:11 -08:00
A.J. Beamon 9c7a39d5b2 Rename SevWarnAlways trace event DeviceNotFound to GetDiskStatisticsDeviceNotFound 2018-02-07 12:54:34 -08:00
A.J. Beamon 4bfbdbf454 Extract getLocalTime to platform.cpp 2018-01-17 11:35:34 -08:00
Stephen Atherton a77162b53d Merge branch 'master' into backup-container-refactor
# Conflicts:
#	fdbclient/BackupAgent.h
#	fdbclient/FileBackupAgent.actor.cpp
#	fdbclient/KeyBackedTypes.h
2017-11-15 08:14:47 -08:00
Stephen Atherton 3dfaf13b67 IBackupContainer has been rewritten to be a logical interface for storing, reading, deleting, expiring, and querying backup data. The details of how the data is organized or stored is now hidden from users of the interface. Both the local and blobstore containers have been rewritten, the key changes being a multi level directory structure and no more use of temporary files or pseudo-symlinks in the blob store implementation. This refactor has a large impact radius as the previous backup container was just a thin wrapper that presented a single level list of files and offered no methods for managing or interpreting the file structure so all of that logic was spread around other places in the code base. This made moving to the new blob store schema very messy, and without this refactor further changes in the future would only be worse.
Several backup tasks have been cleaned up / simplified because they no longer need to manage the ‘raw’ structure of the backup.  The addition of IBackupFile and its finish() method simplified the log and range writer tasks.  Updated BlobStoreEndpoint to support now-required bucket creation and bucket listing prefix/delimiter options for finding common prefixes.  Added KeyBackedSet<T> type.  Moved JSONDoc to its own header.  Added platform::findFilesRecursively().

Still to do:  update command line tool to use new IBackupContainer interface, fix bugs in Restore startup.
2017-11-14 23:33:17 -08:00
A.J. Beamon bf07fa3023 Untested changes to MemAvailable computation on kernels without MemAvailable 2017-11-06 09:35:05 -08:00
A.J. Beamon a5c2373dbb Spaces->tabs 2017-10-18 09:04:35 -07:00
Alex Miller 7b9bc1d715 Merge pull request #170 from cie/alexmiller/flowprofile
Add support for profiling a running fdb cluster to fdbcli, fix security issues, and add an improved backtrace.
2017-10-16 16:51:53 -07:00
Alex Miller 16e5b50685 Replace backtrace with absl::GetStackTrace on non-MacOS platforms.
backtrace() gives a list of return addresses, which means that addr2line will
print out the line after the caller. GetStackTrace returns the list of caller
addresses, so the addr2line results should be accurate.  The flow profiler was
also changed to use the new backtracing code, so flow profiles will now be
accurate as well.  Unfortunately, the abseil code doesn't work on MacOS, so we
still fall back to backtrace() in this case.

For the stack unwinder to work, we must disable -fomit-frame-pointer.  This can
result in a small performance penalty, as it effectively reduces the number of
general purpose registers available by one.  (I'm also curious if this has
anything to do with the overly frequent "<value optimized out>" messages from
gdb.)  If this shows up as a problem, we can make release builds still have
-fomit-frame-pointer, and fall back to backtrace when it's enabled then as
well.
2017-10-16 16:05:02 -07:00
A.J. Beamon b20ae356b1 Alloc instrumentation backtraces use format_backtrace; Magnesium detects backtraces from binaries besides fdbserver. 2017-10-12 08:39:13 -07:00
A.J. Beamon 4f97bd44a5 If we fail to get the interface name due to a platform error, don't kill the process. Instead, just leave the network counters alone. Change the GetInterfaceAddrs trace event to SevWarnAlways. 2017-09-28 13:32:39 -07:00
Evan Tschannen 7081136f74 added a fix 2017-09-22 15:08:14 -07:00
Evan Tschannen 4809bd8f62 fix: We cannot inject faults after renaming the file, because we could end up with two asyncFileNonDurable open for the same file 2017-09-21 18:11:18 -07:00
A.J. Beamon f66e22c89d fix: machine metrics could sometimes default to 0, which cause underflows when compared with prior results. 2017-08-03 15:49:30 -07:00
FDB Dev Team a674cb4ef4 Initial repository commit 2017-05-25 13:48:44 -07:00