af617d68e6
boost 1.52.0 -> 1.67.0 in all vcxproj files
2019-03-05 22:06:12 -08:00
87d0c5bae0
Bug/usability fix: The output URLs of fdbbackup list were meant to be directly usable for backup management operations but they were missing the bucket URL parameter.
2019-03-05 21:14:21 -08:00
82d957e0bb
Merge pull request #1178 from vishesh/task/issue-963-IPv6
...
IPv6 Support
2019-03-05 17:14:16 -08:00
96ee95b9ad
fix: macOS build #963
...
Use the boost representation of IPv6 address internally
and make sure it uses std::array.
2019-03-05 14:03:14 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
e93cd0ff21
Add some checks and comments to IPv6 changes #963
2019-03-04 14:12:45 -08:00
592e224155
net: add/use formatIpPort to format IP:PORT pairs #963
2019-03-04 14:12:45 -08:00
82b2da4b78
net: Add IPAddress::parse() util #963
2019-03-04 14:12:45 -08:00
25daabdc02
net: TraceEvent and toIPVectorString for new IPAddress structure #963
2019-03-04 14:12:45 -08:00
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
1a550712cb
Add serialization support for std::array
2019-03-04 13:33:10 -08:00
75b2546e35
Allow FlowLock::take to succeed if there's no other active requests.
2019-03-04 01:42:38 -08:00
98bb58628f
Upgrade FlowLock from int to int64_t.
2019-03-04 01:42:38 -08:00
a462359af1
Implement g_random->randomSkewedUInt32()
...
This generates an exponential distribution, implemented as being
a "log-uniform" distribution.
2019-03-04 01:42:38 -08:00
39f612d132
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-03-02 17:07:00 -08:00
94385447bc
Status: Get if client configured TLS
...
To understand if all clients have configured TLS,
we check the tlsoption when a client tries to open database.
This is similar to how we track the versions of multi-version clients.
2019-03-01 15:17:01 -08:00
655c9d82c7
Various cleanup from review
2019-03-01 14:06:47 -08:00
13f7f5a073
Bump protocol version
2019-03-01 13:01:10 -08:00
e2bcecb08f
Merge branch 'master' into ratekeeper-batch-priority-limits
2019-02-28 12:52:44 -08:00
eb629d87a5
Add information about batch ratekeeper to status. Make it possible to track latencies in the ReadWrite workload for concurrently run instances separately.
2019-02-28 09:53:16 -08:00
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
82e429f417
Upgrade BinaryReader to support calls ArenaReader has.
2019-02-26 18:00:55 -08:00
abfe057805
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-25 13:47:16 -08:00
78aad255b0
updateProcessStats gets process stats directly
...
Storage servers no longer parse ProcessMetrics trace lines to get their
own cpuUsage and diskUsage statistics.
2019-02-25 13:45:53 -08:00
b8910ba7cd
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.h
# fdbserver/DataDistribution.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-22 14:38:13 -08:00
d008de576e
Merge pull request #1139 from xumengpanda/mengxu/machine-team-upgrade-PR
...
Add background actor to remove redundant teams
2019-02-22 14:22:07 -08:00
6660ccd936
Large rewrite/refactor from the page format outward. Replaced the in-page format, PrefixTree, with more general and templated implementation of the same concept, DeltaTree, which will be incrementally modifiable soon. New implementation for in-page cursors to use a per-page arena and share decoded binary tree nodes between cursors in the same page. New implementation for internal record B+Tree cursor which uses reference counted path hops which makes cloning a cursor very cheap. New implementation of external facing B+Tree cursor which makes efficient use of two internal cursors to find user-visible KV pairs and return them without copying keys and values except in the case of a large sharded value.
2019-02-21 02:46:30 -08:00
171d119d95
another attempt to fix compiler errors
2019-02-20 18:35:08 -08:00
fa96b8dd33
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-20 16:56:16 -08:00
4be90c91d6
Moved THIS and THIS_ADDR to actorcompiler.h
2019-02-19 15:16:59 -08:00
3bd9b9047b
Minor fixes - flow now compiling with intellisense
2019-02-19 15:16:59 -08:00
2c16475906
Added a general copy-constructor for intellisense
...
If a actor returns something of a different type that is
assignable, it will work because the actor compiler will
compile this to an assignment. However, if you compile
without the actor compiler, this will fail.
2019-02-19 15:16:59 -08:00
eb8285f14f
Add mock description for actors for intellisense
2019-02-19 15:16:59 -08:00
9f8c060343
Put assertions in its own scope
...
Some if-else statements in the code don't use
curly-braces, which is problematic if the only
statement used is an assertion. This should fix
that.
2019-02-19 15:16:59 -08:00
801e4c4700
Make when-macros work by accepting varargs
2019-02-19 15:16:59 -08:00
e05b53d755
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-15 20:37:07 -08:00
6d09ac483c
Merge with master
2019-02-15 17:03:40 -08:00
83060c6e56
Merge pull request #1062 from jzhou77/PR
...
Add a new DataDistributor role.
2019-02-15 13:51:27 -08:00
8099a0b665
Merge pull request #1129 from alexmiller-apple/tstlog-1
...
Spill-by-reference TLog Part 1: IDiskQueue::read()
2019-02-15 11:48:14 -08:00
75f692b931
simplify actorcompiler and target to compile coveragetool
2019-02-15 00:01:42 -08:00
5e6577cc82
Final cleanup per review comments
...
Make distributor interface optional in ServerDBInfo and many other small
changes.
2019-02-14 16:37:17 -08:00
578473a974
Various review comments fixes
2019-02-14 16:37:16 -08:00
907446d0ce
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-14 11:37:38 -08:00
b435d51061
Merge branch 'master' into track-server-request-latencies
2019-02-14 08:07:32 -08:00
5481851e82
TeamCollection: Add knobs for team remover
...
Added three knobs to control team remover
bool TR_FLAG_DISABLE_TEAM_REMOVER:
Disable the teamRemover actor
double TR_REMOVE_MACHINE_TEAM_DELAY:
Wait for the specified time before try to remove next machine team
double TR_WAIT_FOR_ALL_MACHINES_HEALTHY_DELAY:
Wait before checking if all machines are healthy
2019-02-13 15:11:56 -08:00
9272a41e5f
Merge pull request #1146 from atn34/fix-actor-warning
...
Fix actor warning for cmake build
2019-02-13 11:01:37 -08:00
1ea58c1e5e
Actually change name of flag
2019-02-13 10:44:58 -08:00
3a38bff8ee
Use DISABLE_ACTOR_WITHOUT_WAIT_WARNING consistently
2019-02-13 10:30:35 -08:00
2570b37e6e
Add function to read pages from RawDiskQueue_TwoFiles
2019-02-12 23:43:10 -08:00
601b229c05
Add a no-op breakpoint_me for easy gdb'ing.
2019-02-12 23:43:10 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
f257cca1bc
Update `Void _ = wait(...)` occurrences
2019-02-12 17:30:30 -08:00
df3454114e
Add TODO
2019-02-12 11:03:08 -08:00
8b0e593f83
Add --disable-actor-without-wait-error flag to actorcompiler
2019-02-12 11:01:17 -08:00
d348846a10
Introduce ErrorMessagePolicy class
...
This encapsulates logic for how to report particular error messages.
So far only used for reporting an actor not containing a wait statement,
as this warning should be suppressed in certain contexts.
We could change other error messages to use this class, but since they
don't have any logic I don't think the extra indirection is appropriate.
2019-02-12 11:01:17 -08:00
6051c8027b
Allow osx tooling to understand c sharp
...
After this change I am able to load foundationdb.sln with vscode and get
features like jump to definition and autocomplete.
IntermediateOutputPath will now use a default value.
2019-02-12 11:00:24 -08:00
b717750c4f
Minor README changes.
2019-02-11 21:40:14 -08:00
685242fbfc
Merge pull request #1092 from atn34/trace-format-network-option
...
Add trace_format network option
2019-02-08 10:52:50 -08:00
06e216f26e
fixed linker error
2019-02-07 15:37:04 -08:00
e45295a1f5
Added support for TLS
2019-02-07 15:37:04 -08:00
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
7a858b902d
Most of FDB compiling on Windows with cmake
2019-02-07 15:37:04 -08:00
d4349293b9
Reworked the way latency counters are tracked. Report the latency bands in separate events from StorageMetrics and ProxyMetrics. Fix a problem when the latency band configuration was changed. Add correctness testing.
2019-02-07 13:39:22 -08:00
0750dc0418
Change store from (Future, T&) to (T&, Future).
...
LHS = RHS, and the name of what's being modified is easier to find.
2019-02-04 18:04:22 -08:00
fa6f43b1f3
Added missing 'static' for actor fwd declarations
...
There is a bug in the code that when an ACTOR forward
declaration is static, the static keyword doesn't get
generated in the forward declaration. This commit fixes
that.
2019-02-01 14:44:47 -08:00
e9ddd94e27
The failure monitor is given a list of all IP addresses associated with a process
...
The connect packet includes the correct remote address
Did a lot of code cleanup
Simulation test mixed TLS and non-TLS listeners on the same process
2019-01-31 18:20:14 -08:00
9087f2948e
Support serialization of std::map objects
2019-01-31 11:24:04 -08:00
0e5db1e7b7
Merge pull request #1096 from mpilman/features/actor-forward-declarations
...
Allow forward declarations to be actors
2019-01-30 13:30:49 -08:00
e055fdab03
Better diagnostics for unrecognized trace format
2019-01-29 13:08:47 -08:00
01730cc461
Allow forward declarations to be actors
...
If an actor is forward declared it is kind of
awkward as the signatures don't fit. For example:
Future<Void> foo(int const& bar);
is the correct forward declaration for this actor:
ACTOR Future<Void> foo(int bar);
This also means that a lot of tooling (like rtags/cquery etc)
doesn't work correctly. It will show errors that the function
calls are ambiguous and "find references"-features typically
don't work.
Allowing to do this in the actor compiler is the first step. The
next will be to refactor all forward declarations.
2019-01-29 10:57:54 -08:00
dc2b740415
Added server_overloaded error and client message
2019-01-25 13:15:19 -08:00
1d7fec3074
Merge commit '048bfc5c368063d9e009513078dab88be0cbd5b0' into task/tls-upgrade-2
...
# Conflicts:
# .gitignore
2019-01-24 17:43:06 -08:00
ec32d3308b
Merge pull request #1086 from mpilman/features/c++-compiler-errors
...
Fixed several minor code issues
2019-01-24 15:24:33 -08:00
79637f07ac
Fixed several minor code issues
...
These will become a problem as soon as we
switch to C++17
2019-01-24 14:43:12 -08:00
4925b61d0e
Bump protocol version.
2019-01-24 13:49:22 -08:00
7fde75c439
Avoid possibly concurrent reassignment of dl_iterate_phdr.
2019-01-24 13:28:45 -08:00
efe06d71ce
Change profilingEnabled to thread_local to eliminate a race.
2019-01-24 13:27:16 -08:00
2198d24ce1
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/ServerDBInfo.h
# fdbserver/Status.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/storageserver.actor.cpp
2019-01-24 11:43:26 -08:00
8e05e95045
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-18 16:18:34 -08:00
7498c2308c
Merge branch 'release-6.0' into track-server-request-latencies
2019-01-16 13:39:01 -08:00
8b3cfa469c
Fix spelling
2019-01-15 19:30:09 +00:00
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
6276d6b4f4
Merge pull request #1026 from etschannen/master
...
Merge 6.0 into master
2019-01-09 16:23:35 -08:00
684a22a52b
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BackupContainer.actor.cpp
# fdbclient/HTTP.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/BackupCorrectness.actor.cpp
# versions.target
2019-01-09 16:14:46 -08:00
5048795554
Add JsonTraceLogFormatter to flow/CMakeLists.txt
2019-01-09 15:58:18 -08:00
7c5b2ab330
Merge pull request #976 from alexmiller-apple/jsonlogs
...
Allow trace logs to be output as JSON instead of XML
2019-01-09 17:04:50 -05:00
5d2b11cba9
Merge pull request #1019 from satherton/http-request-id
...
Backup usability and sanity check improvements
2019-01-09 13:29:37 -08:00
51b89ae083
WIP
2019-01-09 07:41:02 -08:00
d265517156
Fix: fast allocator would not cleanup memory for a thread if that thread never called getMagazine. This could happen if the first thing the thread did was to release memory.
...
Added a new metric for the number of threads that hold memory for each size and improve some existing metrics.
Fix: a failed ASSERT would crash if done early in the program lifetime.
2019-01-08 14:36:01 -08:00
0ec216a1fa
Added X-Request-ID header to HTTP requests and verification of matching ID in response, if present.
2019-01-07 17:56:38 -08:00
75fd8e6e17
Include <sys/sysmacros.h>
2019-01-03 12:44:29 -08:00
dbe9baff1f
Several small compilation fixes for new versions of gcc
...
There are several missing includes for cmath in the code, I added those.
Next, Coro returns a reference to a stack variable and this causes a
warning. As this is probably ok for Coro, I disabled the warning in
that file for GCC. I want to have this warning in the build system as
it is generally a very useful warning to have.
Another change is that major and minor are deprecated for a while now.
I replaced those with gnu_dev_major and gnu_dev_minor.
ErrorOr currently implements operators ==, !=, and <. These do not
compile because Error does not implement ==. This compiles on older
versions of gcc and clang because ErrorOr<T>::operator== is not used
anywhere. It is still wrong though and newer gcc versions complain.
I simply removed these methods.
The most interesting fix is that TraceEvent::~TraceEvent is currently
throwing exceptions. This is illegal behavior in C++11 and a idea in
older versions of C++. For now I simply removed the throw, but this
might need some more thought.
2019-01-03 12:44:19 -08:00
3f399dd9be
Indent within anonymous namespace
2019-01-03 09:21:09 -08:00
aa2a76ef4c
Merge pull request #981 from alexmiller-apple/cmake
...
Add a CMake build system
2019-01-02 18:50:15 -08:00
80dd1efaf6
Assign to g_traceLog.formatter
2019-01-02 16:42:00 -08:00
6a4d87802b
Replace & operator with variadic function
2018-12-28 11:33:42 -08:00
f64d5321e9
Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL.
2018-12-20 18:05:23 -08:00
b8df5acc15
Add --trace_format flag to fdbserver
2018-12-20 15:02:01 -08:00
33a4aac249
Fix build
2018-12-20 11:55:59 -08:00
4b31d209da
Emit valid json for unprintable characters
2018-12-20 11:43:18 -08:00
cdc7e83993
Update flow/Trace.cpp
...
Co-Authored-By: alexmiller-apple <35046903+alexmiller-apple@users.noreply.github.com>
2018-12-14 11:30:40 -08:00
a5401b834b
Integrate JsonTraceLogFormatter into build system + more.
...
Where more is:
* Changed the default format to xml, so that all traces don't suddenly
change as of this commit. Someone is going to need to start passing a
flag to all their servers, but let's worry about that later.
* Changed the copyright headers to be the correct filename.
* Include paths should be specified from the root.
2018-12-13 22:07:16 -08:00
2389cf65f3
fixed escaping bug
2018-12-13 21:50:01 -08:00
aa1eff8346
formatted json log formatter
2018-12-13 21:49:46 -08:00
40890e9dbe
Implemented a json log formatter
2018-12-13 21:46:02 -08:00
a982b9da72
Additional changes from a merge commit.
2018-12-13 17:13:41 -08:00
c654b1bb61
First version of working rpm packages
2018-12-13 15:55:31 -08:00
550daa05f8
Default configuration now builds.
2018-12-13 15:52:27 -08:00
df0f491c29
Some more improvements to the build and preparations for packaging
2018-12-13 15:04:13 -08:00
dce290909d
fdbserver now compiling
2018-12-13 14:13:47 -08:00
51beb8b48c
fdbrpc compiling with cmake
2018-12-13 14:02:16 -08:00
f5fcb666c5
flow compiling with cmake on OS X
2018-12-13 13:58:06 -08:00
43e5a46f9b
Change Endpoint::address(NetworkAddress) to vector<NetworkAddress>
...
Extend `Endpoint` class to take multiple NetworkAddresses instead of
just one. Hence, to talk to an endpoint instead of one IP:PORT, we'll
have multiple IP:PORT pairs.
This patch simply adds the field and makes changes to compile the
codebase. The first element of of `address` field is used everywhere.
Hence the way we talk to remains same with this patch.
NOTE:
Directly accessing the first memeber of Endpoint::address is unsafe
as Endpoint() doesn't enforces non-empty address list. However, since
the correctness test pass for now and are anyway replacing all those
unsafe accesses with ones considering the whole vector, this patch
ignores to access them in safe way.
2018-12-13 13:36:52 -08:00
42dffd4dff
Take a vector of network addresses from CLI to start FDB server
...
Extends the CLI interface to take multiple public and listen addresses.
We however do not do anything with those extra addresses and just
consider the first one for now.
2018-12-13 13:36:52 -08:00
e8e01b2406
Remove unused localAddress parameter from newNet2 and Net2 classes
2018-12-13 13:36:52 -08:00
d2d68aa171
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# versions.target
2018-12-03 18:26:52 -08:00
55a9c4a0f0
Merge pull request #955 from ajbeamon/fix-bad-error-creation-and-whitespace
...
throw platform_error; -> throw platform_error();. Convert some spaces to tabs.
2018-12-03 15:12:37 -08:00
e2673b2bbc
Fixed typo causing mac build failure.
2018-12-02 22:03:21 -08:00
50c9dfdd01
Errors that occur in platform that are the result of IO issues are now raised as io_error rather than platform_error.
2018-11-30 10:55:19 -08:00
eb2f27b8e5
Work in progress implementation of server-side latency tracking. The intent of this is to be able to measure the number of requests that achieve certain latency targets across the system relative to the total number of requests.
2018-11-30 10:46:04 -08:00
97847f517b
throw platform_error; -> throw platform_error();. Convert some spaces to tabs.
2018-11-28 12:56:57 -08:00
1f3b6e8bdf
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/BackupContainer.actor.cpp
# fdbclient/BlobStore.actor.cpp
# versions.target
2018-11-27 14:41:46 -08:00
8b021d38d8
Merge pull request #939 from satherton/backup-smaller-snapshot-folders
...
New backup folder scheme which results in far fewer kv range folders
2018-11-27 11:18:52 -08:00
b91b26ef75
Attempt at workaround of a rare issue where long running backup processes reach a state where DNS resolution requests always time out but other processes on the same host can still resolve successfully. In case this was somehow caused by a bad boost tcp_resolver state, each request now uses a unique tcp_resolver instance.
2018-11-26 20:02:03 -08:00
32f434b2ee
Bug fix, dns resolution would throw an error if any of the results were IPv6 addresses, which could happen depending on the host networking configuration.
2018-11-26 19:53:46 -08:00
ec9410492d
Changed backup folder scheme to use a much smaller number of unique folders for kv range files, which will speed up list and expire operations. The old scheme is still readable but will no longer be written except in simulation in order to test backward compatibility.
2018-11-23 05:23:56 -08:00
4e54690005
Merge branch 'release-6.0'
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/MoveKeys.actor.cpp
2018-11-12 20:26:58 -08:00
9d04934f11
Don't allocate magazines backed by hugepages on systems that support them.
...
Doing so causes an excessive amount of memory to be wasted. See issue #909 .
2018-11-12 13:45:48 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
16909fb51d
HTTP header field names are now treated as case insensitive.
2018-11-09 16:57:01 -08:00
81dc8b2d05
Fix crash when bad filename is specified for profiling output. Also removed some unused code and added a couple return value checks.
2018-10-25 09:35:19 -07:00
7c6cf47e37
Revert "Fix operator problems like those mentioned in GitHub issue #606"
2018-10-23 17:21:48 -07:00
47dd24a9fa
Fix operator problems like those mentioned in GitHub issue #606
...
So the code as written works well on older gcc, but does not like newer gcc. I
don't fully know why member operators seem to be more in conflict with different
compiler versions, but experience has taught me to always use non-member
operators for maximum portability across compilers.
Make a small change to that effect here.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-23 23:24:34 +00:00
0a296fc471
Merge pull request #797 from apkar/master
...
Adding Flow documentation.
2018-10-22 16:50:37 -04:00
6bb1f4093d
Merge pull request #856 from dropbox/pr/include-fix
...
Adjust all includes to be relative to the root.
2018-10-22 09:51:55 -07:00
e2fc1c9b95
Remove specifying non-root directory as a path to search for includes.
2018-10-19 18:56:45 -07:00
888378c8cb
fixed windows build
2018-10-19 13:28:32 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
8dd900a337
fixed the windows build
2018-10-18 20:26:45 -07:00
db71b60d72
Merge pull request #819 from satherton/feature-redwood
...
Redwood storage engine, initial/experimental version
2018-10-18 18:38:11 -07:00
22f8a4efa9
Normalized all unit test names to begin with "/" if they should be included in random unit testing.
2018-10-05 22:09:58 -07:00
57ce1ad7c3
Merge pull request #817 from apple/release-6.0
...
Merge release-6.0 into master
2018-10-05 15:28:41 -07:00
664f64881c
Port truncate optimization from Snowflake PR in order to make quick changes for a patch release.
2018-10-05 15:05:26 -07:00
7c1dc305cb
Merge commit 'a72c8f5cb2e79a673abc0ed3d27ef1c51028fb13' into feature-redwood
2018-10-05 10:15:10 -07:00
5d9cd9acdc
Correctness test now has additional random reader which doesn't do verification but isn't stopped when the btree is closed. Fixed bug exposed by this where pager snapshots will still try to read pages after the pager has been shut down or even destroyed. Added new error type, shutdown_in_progress.
2018-10-04 23:46:37 -07:00
c06c214f0b
Fixing typos in flow/README.md
2018-10-04 17:33:16 -07:00
3922e477a5
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/LogSystemDiskQueueAdapter.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00
62fbd7958a
Adding Flow documentation.
2018-10-02 11:03:31 -07:00
31b297e756
Merge pull request #755 from ajbeamon/fix-actor-compiler-warnings
...
Fix actor compiler warnings related to TEST_CASEs not having wait() s…
2018-09-24 10:11:32 -07:00
2fc86c5ff3
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbrpc/AsyncFileCached.actor.h
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-09-20 03:39:55 -07:00
10a8e67266
Merge remote-tracking branch 'upstream/release-6.0' into merge-release-6.0
2018-09-11 21:49:59 -07:00
4773b788e9
Fix line endings
2018-09-10 09:44:47 -07:00
309159ea23
WriteValue() no longer uses format() which creates temporary strings, instead if writes directly into allocated space in the target buffer using snprintf().
2018-09-10 00:44:09 -07:00
ce3f01a0cf
Added concept of type to JsonString. Appending single items or key/value pairs is now type-safe and only allowed in certain cases. JsonString will refuse to produce invalid JSON. All duplicative functions have been replaced with templates. Encoding of values uses json_spirit's value writer which should be no worse performance than format() and it will escape everything properly. Final string form is now built directly using knowledge of type, such as when an instance becomes an Array or Object the appropriate opening character is written. This avoids a full copy just to prepend the opening character later. Index interface for key/value pairs no longer makes a temporary copy of the key string. JsonString is now only needed by Status.actor.cpp. Still more work to be done here.
2018-09-08 07:15:28 -07:00
d3c8d7ab4e
fix: status would generate invalid json
2018-09-07 18:26:05 -07:00
e918ba7a4f
Fix actor compiler warnings related to TEST_CASEs not having wait() statements and the DescrCompiler having unused variables.
2018-09-06 16:14:04 -07:00
471cf84a47
Merge pull request #751 from apkar/master
...
Merge release-6.0 into master
2018-09-06 14:33:08 -07:00
920fd3fe97
Merge branch 'release-6.0'
2018-09-06 14:24:02 -07:00
7f0a70db7f
Remove defunct DebugQueryRequest.
2018-09-06 13:44:25 -07:00
425ed6c693
Fixed memory leak in JsonString operator[](), fixed minor document byte count error.
2018-09-06 02:28:23 -07:00
fca1451186
fix: hasKey was not set properly
2018-09-05 23:46:31 -07:00
98651bafb1
removed _keyNames from JsonString
2018-09-05 22:51:15 -07:00
4dd2dda0a3
Merge branch 'release-6.0'
...
# Conflicts:
# fdbserver/worker.actor.cpp
2018-09-05 16:11:06 -07:00
df406a340e
Merge pull request #742 from ajbeamon/roles-in-trace-events
...
Add the roles running on a process as a field on trace events in the …
2018-09-05 16:08:12 -07:00
90301f497f
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/TLSConnection.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# versions.target
2018-09-05 16:06:33 -07:00
9e79f9ec59
Cleanup from review
2018-09-05 15:53:12 -07:00
2de0b5d6d7
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-05 15:06:14 -07:00
821adeae4f
Added method to swap in json text
2018-09-05 14:17:33 -07:00
6aa22af83b
Added explicit keyword for appropriate constructor
...
Templatized some methods
Removed unused hash function
2018-09-05 12:22:04 -07:00
4eaff42e4f
Merge pull request #712 from ajbeamon/remove-database-name-internal
...
Eliminate use of database names (phase 1)
2018-09-05 10:35:00 -07:00
43a2afc3b6
Added TODO comment
...
Removed debug comments
2018-09-05 08:06:19 -07:00
04a768042a
Added TraceEvent to measure time to create Status Json
...
Simplified JsonString class to use implementation method for reuse of methods
Removed quotes from non-string values within json
Added Tests for jsonstring
Removed hashing of names for JsonString
Switched name tracker to unordered set
2018-09-05 03:50:53 -07:00
affd7423b4
Added class to write json as objects are added
...
Integrated JsonString class into status
2018-08-31 01:21:24 -07:00
383cac4991
This checkin is an attempt to support long keys by placing oversize key suffixes (as much as will fit) into a page and forcing the prefix bytes that do not fit to be part of the parent's page boundary leading to this page. This strategy is being abandoned but there are some valuable refactorings and debugging enhancements that are still in the right direction so they are being committed.
2018-08-28 13:46:14 -07:00
717c43a69f
merge 6.0 into master
2018-08-22 00:28:04 -07:00
ec1dfe8ae1
Some code cleanup / renaming of things for clarity. Improved StringRef hex string output to omit a middle segment when there is an exceeded length limit.
2018-08-16 16:22:10 -07:00
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
0b4c06169b
Convert tabs to spaces in ActorParser
2018-08-14 15:50:26 -07:00
63b1e85338
Ban `Void _ = wait(...)` constructions, and require just `wait(...)`.
...
There's never any reason to save the value of a Void return, and it's
the easiest source of redefined variable bugs that will creep back in
over time. So just `wait(...)`, it's cleaner that way.
2018-08-14 15:50:26 -07:00
809b2ecdd7
Add a missing #include <algorithm> to flow/Util.h
2018-08-14 15:50:26 -07:00
7feb5d8209
Remove including flow.h in actorcompiler.h, and fix resulting breakage.
...
For files that required flow.h, and only got it through actorcompiler.h,
their version of flow.h would have the actorcompiler #defines defined.
Then, if it included a STL/boost file, the same breakage would result.
This needs to not happen, so the include of flow.h in actorcompiler.h
was removed.
2018-08-14 15:50:26 -07:00
bca324eaa6
More actorcompiler.h fixes and additions.
2018-08-14 15:50:26 -07:00
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
6e7f8da694
Add includes for missing definitions of size_t and uint32_t.
2018-08-14 15:50:26 -07:00
3a0366d4c5
Add clang as a supported compiler.
2018-08-14 15:50:26 -07:00
07e5281142
Restrict actor keyword #defines to actor files.
...
This introduces a new rule in our codebase, that any file that #includes
actorcompiler.h needs to do it as the last #include, and it needs to
then #include unactorcompiler.h at the end of the file.
The point of this is that it prevents our actorcompiler.h #defines from
leaking into boost or the c++ standard library. Both of these start
throwing errors if you s/state// their code, which `#define state `
effectively does.
2018-08-14 15:50:26 -07:00
805781ed32
Forward declare instead of #include in actorcompiler.
...
For some reason, clang thinks that the #ifndef's aren't matched by
#endif's if we #include "flow.h" in the middle. Regardless, it's
better for compile time slightly to forward declare, so let's just do
that anyway.
It also maybe gets us away from a weird
actorcompiler.h -> flow.h -> genericactor.actor.h -> actorcompiler.h
circular include.
2018-08-14 15:50:26 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
1901be1e63
Allow `wait()` instead of `Void _ = wait()`.
2018-08-14 15:49:47 -07:00
cdcf056aef
Merge branch 'release-6.0'
2018-08-14 09:43:51 -07:00
168dce94cb
Remove some trace event suppressions that were happening off the network thread. Downgrade some trace events related to trace logging problems from SevError to SevWarnAlways.
2018-08-14 09:00:43 -07:00
3186fac397
Make sure we still accept some connections even if we are CPU bound by high priority work
2018-08-10 17:47:21 -07:00
574c5576a2
Merge branch 'release-6.0' of github.com:apple/foundationdb
...
# Conflicts:
# fdbrpc/TLSConnection.actor.cpp
# versions.target
2018-08-10 14:31:58 -07:00
3535ddad80
Merge pull request #674 from alexmiller-apple/glibcxx-debug-fixes
...
Fix bugs uncovered by -D_GLIBCXX_DEBUG
2018-08-09 08:18:51 -07:00
f4acaabe4a
flow/Util.h gets a #pragma once
...
And you get a #pragma once! Everybody gets a #pragma once!
2018-08-08 16:05:49 -07:00
c8bc9b522c
Document our protocol version convention and add a static assert to help prevent erroneous edits.
2018-08-08 13:57:47 -07:00
db0a03a8aa
Increase the protocol version on master to be distinct from the 6.0 protocol version. Also realign it with our convention that the x and y digits match the release version.
2018-08-08 13:26:51 -07:00
b5a133865d
Merge branch 'release-6.0' of github.com:apple/foundationdb into release-6.0
...
# Conflicts:
# fdbrpc/TLSConnection.actor.cpp
2018-08-06 18:26:54 -07:00
fb46385a39
Merge pull request #628 from alexmiller-apple/reloadcertificates
...
Reload certificates if changed.
This is a cherry-pick of #628 back to release-6.0
2018-08-06 18:04:04 -07:00
c3c5dd2805
do not suppress loadlibrary and loadfunction trace events
2018-08-06 17:55:24 -07:00
6f328d41ac
suppressed spammy trace events
2018-08-06 12:12:55 -07:00
be1a4d74c7
tlogs serve reads to log routers at a low priority, to prevent them from using all their resources catching up a remote dc that has been down for a long time
...
increase the amount of memory ratekeeper budgets for tlogs so that there is a gap after the spill threshold to prevent temporarily overshooting the budget
2018-08-04 10:31:30 -07:00
538e684f1c
Merge branch 'release-6.0'
...
# Conflicts:
# versions.target
2018-08-03 11:41:46 -07:00
65057b4788
Merge branch 'release-5.2' into release-6.0
...
# Conflicts:
# documentation/sphinx/source/downloads.rst
# documentation/sphinx/source/release-notes.rst
# fdbclient/MasterProxyInterface.h
# packaging/msi/FDBInstaller.wxs
2018-08-02 11:29:40 -07:00
9f64dd945b
remove an unused length from trace event
2018-08-02 10:49:26 -07:00
1a7cda4149
Stop performing self-moves. (e.g. a = std::move(a))
...
self-moves are frowned upon in C++, and in our code this generally happens from
calls to swap as part of trying to implement a "unordered erase" function via
swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now
offered that performs this, while disallowing self-moves.
2018-08-01 18:09:54 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
a361a785e8
fix: clients which cannot talk to storage servers poll the proxy for new storage server interfaces. If there are too many clients polling, we saturate the proxies with these requests, prevents the storage servers from updating their interfaces.
2018-07-31 16:57:23 -07:00
40762d9f9b
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-07-25 17:58:52 -07:00
262af775eb
Implement overly simple file write timestamps for simulation, and clean up code.
2018-07-24 17:20:31 -07:00
9d391498e8
Refactored how key/value memory is held in PrefixTree and VersionedBtree, eliminated many unnecessarily copies of large strings. PrefixTree tests pass but btree is still broken, just committing this because it's a large change set.
2018-07-23 03:09:13 -07:00
168496f819
Poll the certificate files if TLS is enabled and reload them if changed.
...
This allows certificates to be changed/updated without having to restart fdbserver.
2018-07-20 19:00:32 -07:00
80d8197223
Spaces -> tabs
2018-07-18 08:40:26 -07:00
227b55c8f9
Merge branch 'release-6.0' into on-main-thread-fix
2018-07-16 16:12:30 -07:00
a067f8812d
When calling onMainThread from the network thread, first empty the queue of tasks scheduled from other threads. This way, tasks scheduled from the main thread can't jump ahead in line.
2018-07-16 16:10:24 -07:00
e0caa28758
code cleanup
2018-07-16 15:56:43 -07:00
f72a9f60c0
only disable fearless if a datacenter has actually been killed
...
fix: we must prevent recovery into the dead datacenter while reducing usable_regions
2018-07-16 10:06:57 -07:00
96389c74cd
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-07-10 16:42:34 -07:00
fed92be47d
bump protocol version for release
2018-07-10 10:17:51 -07:00
be9c34c6f8
Merge remote-tracking branch 'upstream/release-5.2' into merge-release-5.2
2018-07-10 10:04:48 -07:00
1bc95862b7
Merge branch 'release-6.0' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-07-10 04:16:02 -07:00
3ce7c78d36
If an HTTP request fails due to a connection failure or a timeout, do not convert the error to the more generic http_request_failed.
2018-07-09 18:58:33 -07:00
5a2cb3037b
merge 5.2 into 6.0
2018-07-08 20:14:06 -07:00
fee4234e6b
Bug fixes in memory activity logging.
2018-07-06 19:02:42 -07:00
2925b9b984
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-07-03 23:03:56 -07:00
e17c04f666
Changed assignments to substr() to resize() for clarity/simplicity.
2018-07-02 14:25:24 -07:00
e67f951c06
Merge branch 'master' into feature-remote-logs
2018-07-02 02:18:20 -04:00
c3f88dbfe1
Merge branch 'master' of github.com:apple/foundationdb into tls-static
2018-07-01 23:13:57 -07:00
b24e272394
Merge branch 'master' into feature-remote-logs
2018-07-02 00:07:25 -04:00
5054c194e2
Some trace events are logged before FLOW_KNOBS are initialized
2018-07-01 14:30:37 -04:00
b95a2bd6c1
Merge commit 'b17c8359ec22892ed4daeaa569f2f5e105477251' into feature-redwood
...
# Conflicts:
# flow/Trace.cpp
2018-06-30 23:18:29 -07:00
b17c8359ec
Merge pull request #549 from apple/release-5.2
...
Merge release-5.2 into master
2018-06-30 22:50:07 -07:00
3d32fc4e5d
Reduced MacOS aligned_alloc fix to the most minimal change which fixes the crash while not technically changing the aligned_alloc() contract. The previous version allowed new bad behavior (a non power of 2 alignment less than sizeof(void *)) to succeed and it also would throw an exception on failure instead of just returning nullptr as the other platform implementations would.
2018-06-30 21:37:32 -07:00
3f57d0226c
Crash fix on MacOS. Aligned_alloc() would silently fail for alignments less than 8, which happen in Deque<T> for small T's such as Void or Error.
2018-06-30 17:16:19 -07:00
a680837ee4
The Machine field was missing in early trace events. The logGroup field was not being properly set.
2018-06-28 15:28:58 -07:00
1ff42e078f
Add missing include for Windows, remove throw from TraceEvent destructor.
2018-06-28 14:59:23 -07:00
09624aeec9
Add include statement for std::function to try to make Windows build happy
2018-06-28 10:22:33 -07:00
cbc840ad0a
Move the spammy trace event backstop from the destructor to the constructor. This allows us to avoid doing needless work on a trace event that is going to be throttled.
2018-06-27 15:51:30 -07:00
7378ec85f0
Fixes parentDirectory to work when our path is a child of the root directory. Also works when the path has trailing slashes.
2018-06-26 16:30:47 -07:00
fec225075f
Merge branch 'master' into trace-log-refactor
2018-06-26 14:54:42 -07:00
fe956bc35a
Address review comments
2018-06-26 14:37:21 -07:00
ef8de426d3
Changed the TLS_DISABLED macro
...
Disable TLS within Windows until working
2018-06-26 12:08:32 -07:00
9f545ce002
Merge commit '892727e358c0b3f075564c60c2b7cedb64306f83' into trace-log-refactor
2018-06-26 11:37:23 -07:00
1dce97f28c
Merge branch 'release-5.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/SimulatedCluster.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2018-06-21 17:05:11 -07:00
8bd7eaebdb
fix: broken_promise from push can be throw into the proxy’s actor collection
2018-06-21 15:55:27 -07:00
ed8c2485c8
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-21 14:16:13 -07:00
afdd992fca
Move the computation of total unused allocated memory into FastAlloc.cpp
2018-06-20 14:33:59 -07:00
5e81f4ac7e
Track unused allocated memory in ProcessMetrics and report it in status.
2018-06-20 10:10:51 -07:00
90c8288c68
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-17 14:55:05 -07:00
889889323e
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there.
...
The cluster controller will not switch to the other data center if remote logs are too far behind.
We will not recruit in DCs with negative priority.
2018-06-13 18:14:14 -07:00
1eae9d621b
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-13 15:58:21 -07:00
2878f30f29
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/storageserver.actor.cpp
2018-06-13 15:56:06 -07:00
6f941a89b4
Fix bug in actor compiler that would cause multi-line comments to be marked with the wrong line numbers.
2018-06-13 10:15:20 -07:00
8518f6a8a8
smartQuorum shouldn't return if more responses are desired than futures provided.
2018-06-12 16:50:25 -07:00
372ed67497
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
2018-06-11 11:34:10 -07:00
f965954122
Merge commit '82be52205b95464e355c449fdf3e7d483fa06677' into trace-log-refactor
...
# Conflicts:
# fdbserver/Status.actor.cpp
# fdbserver/workloads/DDMetrics.actor.cpp
# flow/Trace.cpp
2018-06-08 16:22:22 -07:00
8360f71cbb
Merge branch 'master' of github.com:apple/foundationdb into save-fitness-info
...
# Conflicts:
# fdbserver/worker.actor.cpp
2018-06-08 16:09:59 -07:00
06ccd9a500
Allow trace event type names to end with an underscore.
2018-06-08 15:49:31 -07:00
1fdfe20908
Relax the rules on trace event Types a bit by allowing multiple underscores, as well as starting with an underscore and consecutive underscores.
2018-06-08 15:40:29 -07:00
99c9958db7
Some more trace event normalization
2018-06-08 13:57:00 -07:00
0ca51989bb
Merge branch 'master' into trace-log-refactor
...
# Conflicts:
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/Status.actor.cpp
# flow/Trace.cpp
2018-06-08 13:24:30 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
d7d38c3544
Merge pull request #430 from ajbeamon/rename-logGroup-attribute
...
Rename trace file logGroup attribute to LogGroup
2018-06-08 10:30:45 -07:00
69b713918b
VersionedBTree now uses PrefixTree based pages (with bugs). This required significant changes to both classes because the interface and semantics for building, seeking in, and iterating through pages is very different from the previous trivial approach which was based on serialized vectors. PrefixTree node format rewritten to support optional values without increasing overhead for common node scenarios. PrefixTree::Cursor rewritten to reuse path prefix memory instead of allocating new memory on each movement which is then 'leaked' until destruction. PrefixTree::Cursor movement modified to work better with VersionedBTree::InternalCursor, which was also heavily modified. Added knobs related to key arrangement in PrefixTree nodes. Added StringRef::toHexString() as an alternative to printable() to make reading raw PrefixTree data easier. PrefixTree performance is temporarily worse with this update and VersionedBtree fails its unit test.
2018-06-08 03:32:34 -07:00
e4d5817679
fix: we must server getTeam requests before readyToStart is set because we cannot complete relocateShard requests without getTeam responses from both team collections
2018-06-07 16:14:40 -07:00
11b79c6c94
Save fitness info of a process to become a cluster controller. This info is currently lost after a reboot. Save this info and reload it to avoid unnecessary re-recruitments.
2018-06-07 13:07:19 -07:00
529d0497f1
Proxy going OOM when applying high volumes of writes to a proxy, particular in a sudden fashion before ratekeeper can control the workload.
...
Address this issue by proactively monitoring the memory used by commit batches and dropping requests if a certain memory limit is exceeded.
2018-06-01 15:21:40 -07:00
3ea5fc72f0
Rename trace file logGroup attribute to LogGroup
2018-05-31 15:34:17 -07:00
5848055620
More line ending fixes
2018-05-31 13:44:38 -07:00
090117a674
Fix line endings.
2018-05-31 13:41:32 -07:00
f1dea3bed9
Rework the formatter a bit and extract out the writing logic so that it could be pluggable.
2018-05-31 13:27:35 -07:00
f7735d000f
Fix linking issue in flow bindings by using a different variant of printable().
2018-05-31 11:16:01 -07:00
78839b20fd
Merge branch 'master' into trace-log-refactor
...
# Conflicts:
# flow/Trace.cpp
2018-05-31 10:46:20 -07:00
d9c702a9e3
Merge release-5.1 into release-5.2
2018-05-30 09:09:55 -07:00
e2e69b5632
Rework how rolled events are logged. Not very efficient, but it doesn't modify the cached version returned in queries.
2018-05-11 11:24:20 -07:00
02df30149f
Merge branch 'release-5.2' into trace-log-refactor
2018-05-11 11:22:34 -07:00
3fba932f80
Updated Protocol Version
...
Updated version within download documentation
Added old release notes
Updated Wix GUID Product ID
2018-05-10 17:03:46 -07:00
67fa95264c
Merge branch 'master' into feature-remote-logs
2018-05-10 17:01:25 -07:00
c45599f2a9
fix: versionstamped_key uses 4 bytes for size
2018-05-10 17:00:35 -07:00
91338fc984
Merge branch 'master' into feature-remote-logs
2018-05-10 15:33:45 -07:00
8f984cb2c9
Merge branch 'release-5.2'
...
# Conflicts:
# fdbrpc/TLSConnection.h
2018-05-10 09:13:22 -07:00
d3450ce5b0
Merge pull request #343 from bnamasivayam/tls-plugin
...
Tls plugin
2018-05-09 16:35:53 -07:00
fe1c9d5be4
Make the trace log formatter reference counted.
2018-05-09 14:35:15 -07:00
8de914a81f
use contents() instead of address of in withPrefix and withSuffix ; whitespace fixes
2018-05-09 09:01:22 -07:00
d3b5cfb93c
Support latest TLS plugin.
...
Add support for https in backup.
2018-05-08 16:28:13 -07:00
464e2cdbf0
change SetVersionstampedKey and SetVersionstampedValue behavior based on API version to make them consistent
2018-05-08 08:57:09 -07:00
047563aa62
Support trace event overflow.
2018-05-04 15:19:56 -07:00
2a46fdb31d
Change formatv to vsformat, which now returns the length of the string (or <0 for error) and takes the output string as an argument.
2018-05-04 13:35:25 -07:00
ce0c991e78
Refactor trace events to store a vector of fields that aren't encoded until write time. Better support for pre-network trace events. Rework how trace events are queried. Some initial work towards pluggable formatting of logs.
2018-05-02 10:44:38 -07:00
10d25927cd
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
2018-04-30 22:15:39 -07:00
738f3036cf
Merge branch 'release-5.2' into enable-trace-logging-before-open
2018-04-30 15:43:17 -07:00
69e831d522
Merge remote-tracking branch 'upstream/release-5.2' into merge-release-5.2
2018-04-28 17:44:52 -07:00
a1faaafca3
Merge remote-tracking branch 'upstream/release-5.1' into merge-release-5.1
2018-04-27 16:38:18 -07:00
11bd7d7daf
Add a disgusting and terrible hack to avoid "undefined std::istream::ignore(long)" linking errors.
...
This makes building our bindings in the provided docker image possible again.
2018-04-26 16:07:50 -07:00
af61d3596d
Merge branch 'public-master' into feature-redwood
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/OldTLogServer.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2018-04-24 17:22:21 -07:00
bd26d6916c
Support boost 1.52 and newer
2018-04-23 14:51:15 +02:00
d660c96158
Compatible with newer boost versions
2018-04-23 14:07:17 +02:00
2dd0b25259
Use ASSERT_ABORT in destructor
2018-04-23 14:05:26 +02:00
2f8a0240f1
Fix some typos.
2018-04-19 11:44:01 -07:00
9cdf25eda3
Fix some typos.
2018-04-20 00:49:22 +07:00
20082e3228
Clang fixes.
2018-04-12 11:10:53 -07:00
19762b847d
Merge branch 'release-5.2'
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/SimulatedCluster.actor.cpp
2018-04-10 17:02:43 -07:00
c1ba16b3c8
Merge branch 'release-5.1' into release-5.2
...
# Conflicts:
# bindings/java/src/test/com/apple/foundationdb/test/AbstractTester.java
# bindings/java/src/test/com/apple/foundationdb/test/VersionstampSmokeTest.java
# bindings/nodejs/lib/fdb.js
# bindings/nodejs/src/Version.h
# bindings/nodejs/tests/tuple_test.js
2018-04-10 16:50:47 -07:00
b46c32535c
surpassed spammy trace events
2018-04-10 15:52:32 -07:00
7af892f50b
first working version of non-copying recovery working with fearless configurations
2018-04-08 21:24:05 -07:00
2752a28611
Merge branch 'release-5.2' of github.com:apple/foundationdb into feature-redwood
2018-04-06 16:29:37 -07:00
b36e08f08f
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-29 15:12:38 -07:00
62726efa82
Add support for logging trace events before the network is created and the trace log opened.
2018-03-28 08:21:22 -07:00
d3fb17d30a
Merge pull request #74 from bnamasivayam/client-profiling-tests
...
Client profiling tests - Part 1
2018-03-23 16:52:49 -07:00
5db52ab081
Merge pull request #87 from etschannen/feature-remote-logs
...
Feature remote logs
2018-03-23 12:55:17 -07:00
ddc0c613ed
Merge pull request #109 from apple/release-5.2
...
Merge Release 5.2 into master
2018-03-21 09:37:56 -07:00
d4007c27ea
Merge pull request #37 from bnamasivayam/release-5.2
...
Split ProcessMetrics into ProcessMetrics, NetworkMetrics and MemoryMe…
2018-03-20 17:31:46 -07:00
551ea9c7f8
Merge remote-tracking branch 'upstream/release-5.2' into master-release-5.2-merge
2018-03-19 12:34:50 -07:00
0b4564e484
Merge pull request #73 from ajbeamon/release-5.2
...
Reduce severity of GetDiskStatisticsDeviceNotFound to SevWarn.
2018-03-16 15:44:54 -07:00
820382ea68
optimized the log router commit path to avoid re-serializing the data
2018-03-16 11:40:21 -07:00
59723f51f8
fix: continue to attempt to lock logs until remote logs are recovered, this is so that remote logs get locked and readers know they will not have any more data
...
do not throttle trace events in simulation
2018-03-14 12:39:55 -07:00
856d2a0a9d
Add correctness tests for Client transaction profiling data format. It also includes format check across upgrades.
2018-03-14 12:39:50 -07:00
06dd052770
Reduce severity of GetDiskStatisticsDeviceNotFound to SevWarn.
2018-03-14 12:17:54 -07:00
70a05c1a9b
fix some compiler whinges
2018-03-13 15:00:16 -07:00
5390af8be4
suppress spammy logs
2018-03-09 09:40:36 -08:00
336a54faef
Split ProcessMetrics into ProcessMetrics, NetworkMetrics and MemoryMetrics
2018-03-06 15:52:03 -08:00
f2c804e14f
Reverting changes from merge of master into release-5.2 ( b25810711c). Note that we never intend to release master into release-5.2, but if we did we would need to revert this commit.
2018-03-06 10:15:04 -08:00
37a6a81634
Merge commit '7f6fc3e039c911cd84b8540f7f799fc38a1c1822' into feature-remote-logs
...
# Conflicts:
# fdbserver/workloads/RestartRecovery.actor.cpp
2018-02-23 12:33:28 -08:00
de1551c20d
Merge branch 'release-5.1'
2018-02-23 08:24:06 -08:00
a1382895a6
Fixed headers and some whitespace
2018-02-23 04:50:23 -08:00
e1162e9238
Merge remote-tracking branch 'upstream/release-5.1'
2018-02-22 11:16:12 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
dc93759e15
suppressed trace events that are spammy
2018-02-16 16:01:19 -08:00
2eeb714c4b
Trace memory usage of individual FastAllocator of different sizes as part of ProcessMetrics.
2018-02-16 13:30:00 -08:00
03501fc26c
TraceEvent::type is set in init so that it works for TraceIntervals too. ASSERT that the type is non-empty.
2018-02-13 13:29:08 -08:00
0a35f167e4
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/DiskQueue.actor.cpp
# fdbserver/IDiskQueue.h
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
# fdbserver/worker.actor.cpp
2018-02-12 01:30:02 -08:00
42405c78a5
Merge commit '4038bd2fd968d88861f2cebd442ce511724816cb' into feature-remote-logs
...
# Conflicts:
# fdbserver/ClusterController.actor.cpp
# fdbserver/Knobs.cpp
2018-02-10 12:08:52 -08:00
32762648fa
Merge branch 'release-5.1'
2018-02-08 18:38:48 -08:00
1f78b98ac9
Bug fix, result as a state variable causes 'this' to be captured instead of copying result.
2018-02-08 09:54:47 -08:00
69425a303b
Improved error handling for cases where blob account credentials are either not found in the provided credentials sources and/or some of the credentials sources provided are not readable or parseable.
2018-02-07 21:50:43 -08:00
9c7a39d5b2
Rename SevWarnAlways trace event DeviceNotFound to GetDiskStatisticsDeviceNotFound
2018-02-07 12:54:34 -08:00
ebd94bb654
removed a separately configurable storage team size for the remote data center, because it did not make sense
...
fix: the master did not monitor for the failure of remote logs
stop merge attempts when a data center is failed
fixed a variety of other problems with data distribution when a data center is failed
2018-02-02 11:46:04 -08:00
080a454051
fix: getVersionstamp would return broken promise if a transaction was disposed before being set. getAddressesForKey would not return when resetPromise was set.
2018-01-31 13:47:36 -08:00
29c5d4ad3d
upgrades from 5.X mostly supported, still some remaining correctness problems
2018-01-28 11:52:54 -08:00
79d94214a4
Merge commit 'f4ffc9752b5ec66ac47f5f684a5d8be06a7eae6e' into feature-remote-logs
2018-01-25 10:12:06 -08:00
7808099318
bump protocol version for release ; new product guid
2018-01-25 10:02:48 -08:00
66de9d392b
New error code, http_auth_failed, which is used when blob authentication fails instead of the previous generic http_request_failed.
2018-01-22 14:58:56 -08:00
698ef4117e
Merge branch 'master' into feature-remote-logs
2018-01-20 10:34:30 -08:00
4bfbdbf454
Extract getLocalTime to platform.cpp
2018-01-17 11:35:34 -08:00
93b34a945f
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 04:09:43 -08:00
21482a45e1
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DBCoreState.h
# fdbserver/LogSystem.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-01-14 13:40:24 -08:00
660cee0254
increased the priority of getKeyServersLocations, because once a client gets a read version, answering their reads should be higher priority than starting new transactions
2018-01-12 13:46:20 -08:00
de119f192d
fixed a priority inversion where the tlog would prefer to copy data from the previous generation rather than make data durable (leading to being ratekeeper controlled)
2018-01-11 16:09:49 -08:00
2f5073d00f
Some visual studio project cleanup.
2018-01-10 10:07:18 -08:00
3ec45d38a0
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-06 13:54:45 -08:00
5ac4f73978
Merge branch 'release-5.1' into feature-remote-logs
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/Locality.h
# fdbrpc/simulator.h
# fdbserver/ApplyMetadataMutation.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WorkerInterface.h
# fdbserver/masterserver.actor.cpp
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-05 11:33:42 -08:00
0f20068e82
Renamed all TaskBucket backup tasks to more appropriate names. Created the ability to make task aliases and used this to direct old task names to a task definition which will abort backups created before version 5.1.
2018-01-04 22:53:31 -08:00
653a46f12f
Update error string fro cluster_version_changed error
2018-01-04 15:06:09 -08:00
cec9f4d7a4
Bug fix in DNS resolution. When the result is an error the result promise was being set twice.
2018-01-03 13:05:38 -08:00
ec28c77353
Merge branch 'master' of github.com:apple/foundationdb
2017-12-21 01:58:47 -08:00
e3aee45a74
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 01:58:15 -08:00
982f0dcb1e
Merge pull request #222 from cie/alexmiller/drtimefix2
...
Fix yet another VersionStamp DR issue.
2017-12-20 15:09:23 -08:00
b5a6bc0ab7
Fix VersionStamp problems by instead adding a COMMIT_ON_FIRST_PROXY transaction option.
...
Simulation identified the fact that we can violate the
VersionStamps-are-always-increasing promise via the following series of events:
1. On proxy 0, dumpData adds commit requests to proxy 0's commit promise stream
2. To any proxy, a client submits the first transaction of abortBackup, which stops further dumpData calls on proxy 0.
3. To any proxy that is not proxy 0, submit a transaction that checks if it needs to upgrade the destination version.
4. The transaction from (3) is committed
5. Transactions from (1) are committed
This is possible because the dumpData transactions have no read conflict
ranges, and thus it's impossible to make them abort due to "conflicting"
transactions. There's also no promise that if client C sends a commit to proxy
A, and later a client D sends a commit to proxy B, that B must log its commit
after A. (We only promise that if C is told it was committed before D is told
it was committed, then A committed before B.)
There was a failed attempt to fix this problem. We tried to add read conflict
ranges to dumpData transactions so that they could be aborted by "conflicting"
transactions. However, this failed because this now means that dumpData
transactions require conflict resolution, and the stale read version that they
use can cause them to be aborted with a transaction_too_old error.
(Transactions that don't have read conflict ranges will never return
transaction_too_old, because with no reads, the read snapshot version is
effectively meaningless.) This was never previously possible, so the existing
code doesn't retry commits, and to make things more complicated, the dumpData
commits must be applied in order. This would require either adding
dependencies to transactions (if A is going to commit then B must also be/have
committed), which would be complicated, or submitting transactions with a fixed
read version, and replaying the failed commits with a higher read version once
we get a transaction_too_old error, which would unacceptably slow down the
maximum throughput of dumpData.
Thus, we've instead elected to add a special transaction option that bypasses
proxy load balancing for commits, and always commits against proxy 0. We can
know for certain that after the transaction from (2) is committed, all of the
dumpData transactions that will be committed have been added to the commit
promise stream on proxy 0. Thus, if we enqueue another transaction against
proxy 0, we can know that it will be placed into the promise stream after all
of the dumpData transactions, thus providing the semantics that we require: no
dumpData transaction can commit after the destination version upgrade
transaction.
2017-12-20 15:04:04 -08:00
7caa012fbf
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 00:49:08 -08:00
e0d9cea008
Merge branch 'master' into continuous-backup
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/BlobStore.actor.cpp
2017-12-19 23:02:14 -08:00
1488c12c18
Simulation will return and error and print if any non-suppressed SevError events were logged.
...
This means that loops like `seed=1; while ./fdbserver -r simulation -s $seed;
do seed=$(($seed+1)); done` to find an example of an often failing test. This
also means joshua will report ExitCode errors on anything that has a SevError
in the log.
As a part of this, we also implicitly downgrade any injected errors to SevWarnAlways.
2017-12-19 17:17:50 -08:00
abb2dd1ebc
Merge pull request #214 from cie/alexmiller/fallocate
...
Use fallocate to zero ranges instead of writing zeroes
2017-12-06 13:47:40 -08:00
f8e89a40ac
Bug fixes, take(1) is incorrect usage of FlowLock.
2017-12-04 10:25:47 -08:00
482ac38ca6
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-01 13:04:32 -08:00
196258080b
Refactor zeroing a chunk of a file from DiskQueue into IAsyncFile.
...
If we're going to do the work to provide more optimized ways to zero files,
then I'd feel better with this being in a more common place, so that any other
zero-ers are likely to reuse it. It also makes testing easier/more obvious.
Also, because it's needed for correctness, fix the aligned_alloc for OSX, which
wasn't aligned, and use an actually aligned allocation function.
2017-11-30 17:57:55 -08:00
aeebe711ce
TaskBucket’s saveAndExtend() is now accomplished through extendTimeout() with an option to save parameters. SaveAndExtendIncrementally() has been removed as it is no longer needed because TaskBucket’s normal execution loop calls extendTimeout() periodically as long as the TaskFunc’s execute() actor has not finished or thrown. If a TaskFunc wants to save changes to task parameters to checkpoint progress for task restarts to benefit from it can call extendTimeout() explicitly with the updateParams flag set to true.
2017-11-30 17:18:57 -08:00
a77162b53d
Merge branch 'master' into backup-container-refactor
...
# Conflicts:
# fdbclient/BackupAgent.h
# fdbclient/FileBackupAgent.actor.cpp
# fdbclient/KeyBackedTypes.h
2017-11-15 08:14:47 -08:00
3dfaf13b67
IBackupContainer has been rewritten to be a logical interface for storing, reading, deleting, expiring, and querying backup data. The details of how the data is organized or stored is now hidden from users of the interface. Both the local and blobstore containers have been rewritten, the key changes being a multi level directory structure and no more use of temporary files or pseudo-symlinks in the blob store implementation. This refactor has a large impact radius as the previous backup container was just a thin wrapper that presented a single level list of files and offered no methods for managing or interpreting the file structure so all of that logic was spread around other places in the code base. This made moving to the new blob store schema very messy, and without this refactor further changes in the future would only be worse.
...
Several backup tasks have been cleaned up / simplified because they no longer need to manage the ‘raw’ structure of the backup. The addition of IBackupFile and its finish() method simplified the log and range writer tasks. Updated BlobStoreEndpoint to support now-required bucket creation and bucket listing prefix/delimiter options for finding common prefixes. Added KeyBackedSet<T> type. Moved JSONDoc to its own header. Added platform::findFilesRecursively().
Still to do: update command line tool to use new IBackupContainer interface, fix bugs in Restore startup.
2017-11-14 23:33:17 -08:00
313e823629
Delete TDMetric data (tmpEventMetric) when a trace event is throttled.
2017-11-13 15:06:21 -08:00
bf07fa3023
Untested changes to MemAvailable computation on kernels without MemAvailable
2017-11-06 09:35:05 -08:00
7cf17df821
Merge branch 'master' into log-group-for-unsupported-clients
...
# Conflicts:
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-11-01 11:31:02 -07:00
b1e3864c0e
fix: stop after does not print errors if actor cancelled
2017-11-01 10:58:39 -07:00
e377d1985c
Merge pull request #195 from cie/throttle-trace-events
...
Increase message limit for throttling to 100,000 for circus runs.
2017-10-31 11:10:11 -07:00
dd6c24ce09
Addressed Review Comments.
2017-10-31 11:08:54 -07:00
80e5fecfe2
Increase message limit for throttling to 100,000 for circus runs.
...
Added an optimization to use a separate set for throttled events. Since this set is expected to be small, comparison of every event against this set is going to be cheaper.
2017-10-31 10:35:26 -07:00
54d82c0d92
Merge pull request #194 from cie/alexmiller/valgrind
...
Fix valgrind errors
2017-10-27 17:25:12 -07:00
3b61b76876
Fix a massive amount of valgrind errors and make them easier to debug in the future.
...
std::is_pod<> being less restrictive than is_binary_serializable<> meant that
structs that both were POD and had a serialize method defined would be binary
serialized instead of using the defined serialize(). This means that it would
also serialize any padding that the struct contained, which would cause mass
waves of valgrind failures from uninitialized memory.
Included in this change is additional uses of valgrind client requests so that
attempts to send uninitialized memory are reported at the sending site, versus
as part of checksum calculation in sending the packet.
2017-10-27 16:54:44 -07:00
1d3c88c147
Increase the spammy trace event threshold to 20000 in 20 minutes.
2017-10-27 12:22:56 -07:00
cfefab18fb
Merge branch 'master' into add-new-atomic-ops
2017-10-25 18:03:34 -07:00
9dd588dcce
Addressed review comments.
...
Changed naming for NewMin and NewAnd to MinV2 and AndV2
2017-10-25 14:48:05 -07:00
d852a53ae4
Merge pull request #181 from cie/throttle-spammy-logs
...
Throttle spammy logs
2017-10-25 13:45:55 -07:00
3afc85881e
Merge branch 'master' into backup-container-refactor
...
# Conflicts:
# fdbrpc/BlobStore.actor.cpp
2017-10-20 21:38:28 -07:00
42955012e9
Merge branch 'release-5.0'
...
# Conflicts:
# fdbrpc/BlobStore.actor.cpp
# flow/error_definitions.h
2017-10-20 21:16:55 -07:00
efe857fef6
Fixed inconsistent styles of recently changed error messages.
2017-10-20 12:56:00 -07:00
e2c1e87df6
made a large number of fixes to make fearless DR correctness clean.
2017-10-19 15:36:32 -07:00
dd6d8f3b0e
Merge branch 'master' into add-new-atomic-ops
2017-10-18 16:36:44 -07:00
d3df8469dd
Upgrade protocol version as ClientWorkerInterface was changed.
2017-10-18 14:56:31 -07:00
a5c2373dbb
Spaces->tabs
2017-10-18 09:04:35 -07:00
050c1bcba6
Update transaction_too_old error text.
2017-10-18 08:49:17 -07:00
ebd0234514
Rewrote most error handling in BlobStoreEndpoint to fix several shortcomings in error handling and logging. The request loop now logs but rate limits all errors, and the exceptions thrown are more appropriate. HTTP 503 is now treated as retryable. Callers of BlobStoreEndpoint::doRequest() now specify which codes they consider to be successful so that more error handling can take place in the main request loop.
2017-10-18 02:52:09 -07:00
fc2ff719d0
fixed some capitalization issues that slid in through the case-insenstive filesystems
2017-10-17 09:07:05 -07:00
7b9bc1d715
Merge pull request #170 from cie/alexmiller/flowprofile
...
Add support for profiling a running fdb cluster to fdbcli, fix security issues, and add an improved backtrace.
2017-10-16 16:51:53 -07:00
cf646d4a99
Address review comments.
...
* Fixed fdbcli to be more idiomatic.
* Removed is_binary_serializable in favor of std::is_pod<>
* Removed custom enable_if<> in favor of std::enable_if<>
* Removed HEY REVIEWER comments
* Removed print from prof.py
* Added FLOW_PROFILER_ENABLED=yes to circus components that wished to enable the flow profiler.
2017-10-16 16:46:52 -07:00
16e5b50685
Replace backtrace with absl::GetStackTrace on non-MacOS platforms.
...
backtrace() gives a list of return addresses, which means that addr2line will
print out the line after the caller. GetStackTrace returns the list of caller
addresses, so the addr2line results should be accurate. The flow profiler was
also changed to use the new backtracing code, so flow profiles will now be
accurate as well. Unfortunately, the abseil code doesn't work on MacOS, so we
still fall back to backtrace() in this case.
For the stack unwinder to work, we must disable -fomit-frame-pointer. This can
result in a small performance penalty, as it effectively reduces the number of
general purpose registers available by one. (I'm also curious if this has
anything to do with the overly frequent "<value optimized out>" messages from
gdb.) If this shows up as a problem, we can make release builds still have
-fomit-frame-pointer, and fall back to backtrace when it's enabled then as
well.
2017-10-16 16:05:02 -07:00
1a91aab1d7
Import //base/debugging:stacktrace from abseil.
...
This code is all Apache 2 licensed, and all headers were maintained when
concatinated, so we should be completely fine from a legal standpoint.
I've scriptified the steps that I took so that if we need to update this code
in the future, it hopefully shouldn't be too much of a hassle.
2017-10-16 16:05:02 -07:00
f997cb9038
Add a string knob to hold the Log directory, and write profiles to it.
...
This is the combination of two small changes.
1. Add support for a string knob type.
2. Change profiles to be written to the log directory instead of the working
directory.
We have three options of where to write files: the working directory, the data
directory, and the log directory.
The working directory may be set to a non-writable location, and likely
contains the fdb binaries. Allowing these files to be overwritten would likely
not be a wise idea.
The data directory hosts our sqlite b-trees. It would also be very unfortunate
if these were ever overwritten by an unfortunate profile name.
The log directory contains logs. Out of the three, these matter the least if
they disappear or become corrupted.
Thus, we write to the log directory.
2017-10-16 16:05:02 -07:00
91a26a170c
Add toggleable profiling support to fdbserver+fdbcli.
...
This adds the fdbcli commands:
* profile list -- Lists all workers in a way that doesn't fill `kill`'s list.
* profile flow run -- Allows starting flow profiling on a set of hosts for a specified interval.
And threads through all the support for enabling and disabling profiling as an RPC.
2017-10-16 16:05:02 -07:00
e934604f67
Added DNS resolution. Interface is INetworkConnections::resolveTCPEndpoint() to resolve, or for convenience INetworkConnections::connect(host, service) will resolve host and service (port number or service name like http) and connect to one of the addresses at random.
...
BlobStoreEndpoint now only accepts hostnames and an optional service, so this update is not compatible with the previous URL formats having many IP addresses.
2017-10-15 21:51:11 -07:00
3aaa11977e
Addressed Review Comments
2017-10-12 14:56:00 -07:00
ad0ed79d36
Merge pull request #172 from bmuppana/backup-refactor
...
Backup refactoring
2017-10-12 11:38:49 -07:00
11517f7bfc
Merge branch 'master' into continuous-backup
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2017-10-12 11:03:23 -07:00
b20ae356b1
Alloc instrumentation backtraces use format_backtrace; Magnesium detects backtraces from binaries besides fdbserver.
2017-10-12 08:39:13 -07:00
9648f96200
Also fix unforwarded Metric in IndexedSet.
...
This is simply an exceedingly minor performance fix rather than a correctness issue.
2017-10-11 17:40:48 -07:00
c24b941485
Fix erroneous std::move in indexed set, and clean up addMetric users.
...
This is a follow-on to c4eb73d0. Thanks to Bala for pointing out the unchanged
std::move usage, and there appeared to not be many existing users of addMetric
anyway.
2017-10-11 17:36:51 -07:00
eeebf10030
Modified existing behavior of MIN and AND atomic ops. The new behavior results in a 'SET' if the atomic op is performed on a non -existing key.
...
Added new atomic ops ByteMin and ByteMax that does lexicographic comparison of byte strings.
2017-10-10 13:02:22 -07:00
15962cf079
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbrpc/Locality.cpp
# fdbrpc/Locality.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/ClusterRecruitmentInterface.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WorkerInterface.h
# fdbserver/fdbserver.vcxproj.filters
# fdbserver/masterserver.actor.cpp
# fdbserver/worker.actor.cpp
# flow/error_definitions.h
2017-10-05 17:09:44 -07:00
0ac868ad5d
"Simplify" IndexedSet's insert and addMetric API.
...
The existing code tried to work around the complexities of optionally using
rvalue references' move capabilities if they exist. As seen in the previous
MapPair, there's a combinatorial explosion of prototypes to declare as the
parameter length increases. Because of this, addMetric ended up with a strange
API, and there was a wrapper to make a copy for insert.
Instead, we can apply the idiom of using universal/forwarding references and
std::forward to allow the compiler to instantiate the combinations that are
needed. There's a TagData struct with no copy constructor that validates that
move constructors can be properly called still.
I measured a 12-byte difference between before and after this change, so no
template bloat was introduced.
2017-10-03 20:15:12 -07:00
0e153cdd35
Throttle Spammy logs. Three knobs are added.
...
Trace Events are sampled and cached with an expiration set. Every TraceEvent above SevDebug is checked against this cache to see if it exceeded a set threshold. If yes, then throttle the TraceEvent.
If a TraceEvent is throttled, a warning msg is logged.
2017-10-02 18:43:11 -07:00
6ea9903c82
Merge branch 'release-5.0'
...
# Conflicts:
# fdbbackup/backup.actor.cpp
# fdbserver/ClusterController.actor.cpp
# versions.target
2017-10-01 18:46:44 -07:00
e2b65e86ed
added configurable memory limits for backup and dr executables
...
added a default memory limit of 8GB for fdbcli
2017-09-29 10:35:40 -07:00
ef41b07bb3
renamed past_version to transaction_too_old
...
implemented read_lock_aware option
2017-09-28 16:35:08 -07:00
d4f75630de
Support log group field in status json
2017-09-28 16:31:29 -07:00
7b60e26660
Merge pull request #160 from cie/use-error-descriptions
...
Add the ability to access name and description in Error. Update error…
2017-09-28 16:00:39 -07:00
4f97bd44a5
If we fail to get the interface name due to a platform error, don't kill the process. Instead, just leave the network counters alone. Change the GetInterfaceAddrs trace event to SevWarnAlways.
2017-09-28 13:32:39 -07:00
67d0eb5d66
Change a few more error descriptions; update sphinx error code documentation
2017-09-28 13:03:17 -07:00
d30c730f75
Add the ability to access name and description in Error. Update error descriptions.
2017-09-28 12:35:03 -07:00
7081136f74
added a fix
2017-09-22 15:08:14 -07:00
4809bd8f62
fix: We cannot inject faults after renaming the file, because we could end up with two asyncFileNonDurable open for the same file
2017-09-21 18:11:18 -07:00
489332533c
all timeouts longer than two minutes have been can be lowered to 60.0 with buggification
...
added a workload that tries for a 50 second maximum latency in the presence of one failure with both buggification and connection failures
2017-09-18 11:04:51 -07:00
76e7988663
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/OldTLogServer.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/WorkerInterface.h
# flow/Net2.actor.cpp
2017-09-11 15:15:56 -07:00
fe208d6adf
Merge branch 'master' of github.com:apple/foundationdb into backup
2017-09-06 10:01:55 -07:00
fe9abbfac9
revert 'Remove unused code' for function referenced in fdbrpc
2017-09-01 09:54:03 -07:00
3aaa131b4f
Merge branch 'master' of github.com:apple/foundationdb
2017-09-01 09:03:45 -07:00
439193d17b
Moving keyBackupContainer to BackupConfig.backupContainer()
2017-08-30 12:48:28 -07:00
5d49f2c710
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
2017-08-28 17:45:50 -07:00
9a0a3b6329
Merge commit '66528becb82d826e81fa644bb378212584ab580e'
2017-08-28 16:47:59 -07:00
86d025f943
Bug fix: Metric base enabled state was not being initialized. Metrics are configured to be disabled upon construction, however if during construction it appears that a metric was initially enabled then a crash would result if the MetricsCollection global was not created.
2017-08-27 22:22:32 -07:00
9ce8d3ae4f
Merge branch 'release-5.0'
2017-08-09 10:37:43 -07:00
5a8cd34224
Disable profiling if we aren't using the cached dl_iterate_phdr values.
2017-08-08 09:03:04 -07:00
c22708b6d6
added tag localities
...
fix: remote logs need to stop the master when they are stopped
2017-08-03 16:16:36 -07:00
03915ce4ea
Merge branch 'release-5.0'
2017-08-03 15:49:54 -07:00
f66e22c89d
fix: machine metrics could sometimes default to 0, which cause underflows when compared with prior results.
2017-08-03 15:49:30 -07:00
9fb709cf73
Merge branch 'release-5.0'
...
# Conflicts:
# versions.target
2017-07-27 16:51:57 -07:00
19fa31ffff
fix: uncancellable was not forwarding errors to the result promise
2017-07-27 13:29:08 -07:00
6f0062330b
Merge branch 'master' of github.com:apple/foundationdb
2017-07-26 11:02:13 -07:00
64e9560599
Merge pull request #128 from cie/maintain-incompatible-connections
...
Maintain incompatible connections
2017-07-17 16:28:22 -07:00
2113d47db6
Update protocol version for incompatible connection change
2017-07-17 16:16:05 -07:00
3700624fd7
Merge branch 'release-5.0'
2017-07-17 08:54:10 -07:00
eee492a05b
fix build issue from Notified.h not being shuffled in vcxproj files
2017-07-14 16:46:08 -07:00
c860f09d8a
Merge branch 'release-5.0'
2017-07-14 16:01:15 -07:00
660729839c
moved Notified.h from flow -> fdbclient ; flow bindings package does better job when excluding testers
2017-07-14 15:49:30 -07:00
b133862db6
added FLOW and FDB_FLOW targets to make packages of flow headers and libs
2017-07-13 10:21:36 -07:00
72de765083
Remove unused code
2017-07-13 08:18:00 -07:00
979ebcef6c
changed to using a vector of logSets instead of a duplicate set of logs for remote servers
...
finished porting changes to the tlog
everything but peeking is finished in the TagPartitionedLogSystem
2017-07-09 14:46:16 -07:00
0906250e78
merged everything from feature-remote-logs besides the tlog and tagpartitionedlogsystem
...
re-included tags in messages to the tlog
previously never committed the LogRouter
2017-06-29 15:50:19 -07:00
343d115b37
update Platform to handle newer version of clang that does have __rdtsc
2017-06-28 14:32:01 -07:00
8f7c76ddd3
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# fdbserver/Knobs.h
Updated Windows GUID
Updated and corrected format of Protocol Version
2017-06-26 15:54:57 -07:00
430bb6224e
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/Net2FileSystem.cpp
# fdbrpc/sim2.actor.cpp
2017-06-16 02:14:19 -07:00
4bdcd8fc12
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# bindings/bindingtester/run_binding_tester.sh
# fdbrpc/AsyncFileKAIO.actor.h
2017-06-14 16:43:53 -07:00
b65ad3563c
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2017-06-09 14:56:41 -07:00
Alex Miller