f73aad863b
adding missing logging for 8192 fast allocators
2019-03-20 13:48:45 -07:00
70ac5ffda0
added random logging for huge arenas
2019-03-20 11:20:47 -07:00
c33428b705
do not fail atomicReplace if the files does not exist
2019-03-19 13:38:30 -07:00
10a30fed42
Merge pull request #1316 from senthil-ram/atomicreplace
...
atomicReplace to preserve owner & permission info
2019-03-19 08:21:35 -07:00
ddf8e86730
Merge pull request #1275 from alexmiller-apple/tstlog9
...
Spill-By-Reference TLog Part 5: TLogs That Can Spill Large Amounts Of Data
2019-03-18 17:58:21 -07:00
33a211c834
Merge pull request #1315 from etschannen/feature-memory-logging
...
Occasionally log a backtrace when the fast allocators allocates a new magazine
2019-03-18 16:05:20 -07:00
fe18596c01
Add a comment about ReplaceFile API
2019-03-18 15:24:08 -07:00
c8122edb6b
avoid relying on 128<<10 in knobs.cpp
2019-03-18 12:40:15 -07:00
7711542b55
Update flow/FastAlloc.cpp
...
Co-Authored-By: etschannen <36455792+etschannen@users.noreply.github.com>
2019-03-18 12:34:46 -07:00
c0094830c3
addressing minor format related comments
2019-03-18 11:59:51 -07:00
51a655d9c7
atomicReplace to preserve owner & permission info
...
atomicReplace currently does not preserve the
ownership and permission info, this fix will
preserves the ownership and permission info - it
fails the call if it is not able to do so.
2019-03-18 00:18:05 -07:00
fa5d929106
switch to g_nondeterministic_random to make simulation deterministic
2019-03-17 22:48:43 -07:00
d2eb7578fd
Occasionally log a backtrace when the fast allocators allocates a new magazine to help track down memory leaks
2019-03-17 21:59:56 -07:00
9a4fc25de5
ParentDirectory() in Platform,h no longer uses absPath() so it does not resolve symbolic links, making it suitable for use in fdbmonitor. CleanPath() will return "." instead of empty string to mean a relative path that refers to the current directory. Updated tests, and added new tests for ".".
2019-03-17 15:58:04 -07:00
f88e53e640
Merge branch 'master' of https://github.com/apple/foundationdb into fix-parent-directory
2019-03-16 00:13:09 -07:00
98a295a803
Changed parentDirectory() trace events back to SevError outside of simulation, did the same for cleanPath().
2019-03-16 00:07:38 -07:00
2efb6f4c0d
Added cleanPath() which puts a path in a canonical form without .., ., or duplicate separators without using the filesystem or resolving symbolic links. absPath() redefined to use cleanPath() so it will return the same result for a path without symbolic links regardless of whether or not the path actually exists. Redefined parentDirectory() to use absPath() and error on certain inputs. Added comments describing behavior of these functions, and added a unit test which verbosely tests many inputs to them.
2019-03-15 23:54:33 -07:00
471ac206a3
Add new boost preprocessor definitions to windows build in release mode, still just for flow project to see if it's sufficient.
2019-03-15 21:03:03 -07:00
2c8224f0b7
Add new boost preprocessor definitions to windows build.
2019-03-15 21:03:03 -07:00
7f8475cf27
Removed flow/boost.cpp and the copied boost files it was using, added boost macros to makefile and cmake stuff to hopefully enable building without them.
2019-03-15 21:03:03 -07:00
9f713a6b3d
Testing to see if builds work without the contents of the boost source copies.
2019-03-15 21:03:03 -07:00
5407c87179
Updated two files copied from boost 1.52.0 to their 1.67.0 equivalents since the default boost version to compile against is now 1.67.0 and compilation (on MacOS at least) fails against boost 1.52.0 because some definitions have moved since.
2019-03-15 21:03:03 -07:00
7f5bc2981f
Checksum DiskQueue pages on read, but at a lower priority.
...
If a server has its data spilled, then it's behind the 5s window.
Feeding it data is less important than committing, so we can hide the
extra CPU usage from checksumming the read amplified disk queue pages.
2019-03-15 21:01:19 -07:00
809861ee88
Merge branch 'master' into fix-parent-directory
...
# Conflicts:
# flow/Platform.cpp
2019-03-15 10:51:12 -07:00
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
5a10bf5dfc
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-14 10:35:12 -07:00
c32504f705
io: Add DISABLE_POSIX_KERNEL_AIO knob to use EIO instead of Kernel AIO
...
- Some Linux filesystems don't support O_DIRECT which is required by
Kernel AIO to function properly. Instead of using O_SYNC, EIO is
much better options in terms of performance penalty.
- Some systems may not support AIO at all. Eg. Windows Subsystem for
Linux.
FIXES #842
RELATED #274
2019-03-13 13:39:45 -07:00
e068c478b5
merge master
2019-03-12 18:31:25 -07:00
a7e45cff91
Merge pull request #1176 from jzhou77/ratekeeper
...
Make Ratekeeper a separate role
2019-03-12 15:58:59 -07:00
85c24b0067
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-12 15:20:54 -07:00
5392742902
fixed review comments
2019-03-12 14:38:54 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
bbd5cc2e95
Remove duplicate include due to concurrent update to same file
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-03-11 20:42:35 -07:00
bc4820865e
Fix some Windows issues
...
1. We need to set a Windows target to get rid of warnings when compiling
files that include windows.h
2. By default Boost will try to automatically link against boost_system.
In order to prevent this, we define BOOST_ALL_NO_LIB
3. Somehow, I had to include winioctl.h in Platform.cpp. According to
the documentation from MS, this shouldn't be necessary as windows.h
includes this as well. However, for me it didn't compile otherwise.
2019-03-11 20:42:35 -07:00
880e8643d1
Fix Windows link errors
2019-03-11 17:49:03 -07:00
435e515985
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-11 11:17:40 -07:00
ccab3834e2
merge in master
2019-03-10 23:12:37 -07:00
2ff37f49da
fix: compiler error
2019-03-10 22:56:12 -07:00
c6e94293bf
reset a process to not be degraded after 2 days
2019-03-10 22:39:21 -07:00
c304397166
fixed a memory leak when allocating memory > 4K and < 8K
2019-03-10 21:13:00 -07:00
892b649e64
fixed a memory leak when allocating memory > 4K and < 8K
2019-03-10 21:09:47 -07:00
59f598ab51
Try adding a header?
2019-03-08 21:46:32 -08:00
f593562743
use 8KB pages in Arena
2019-03-08 15:03:17 -08:00
426da90493
Add 8192 bit fast allocator
...
StorageServer is getting close to 4KB on older gccs. However,
on some systems (for example Fedora) the storageServer actor
is already larger than 4KB. This results in a linker error.
2019-03-08 15:03:17 -08:00
4891d31cc0
Remove a broken ASSERT.
...
It's now totally valid to have:
permits=3
take(1)
take(2)
take(72)
and the take(72) will only be granted once the first two finish.
2019-03-07 18:31:18 -08:00
3c86643822
Separate Ratekeeper from data distribution.
...
Add a new role for ratekeeper.
Remove StorageServerChanges from data distribution.
Ratekeeper monitors storage servers, which borrows the idea from
DataDistribution.
2019-03-07 13:16:20 -08:00
04880e3d4d
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-06 13:41:16 -08:00
fd34626009
boost: Remove log from Makefile and version check in flow.h
2019-03-05 22:06:12 -08:00
c6a65389ae
Remove noexcept macro and replace with BOOST_NOEXCEPT.
...
BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a
way that is correctly maintained over time.
2019-03-05 22:06:12 -08:00
367bcff700
Error if boost != 1.67.0
2019-03-05 22:06:12 -08:00
af617d68e6
boost 1.52.0 -> 1.67.0 in all vcxproj files
2019-03-05 22:06:12 -08:00
87d0c5bae0
Bug/usability fix: The output URLs of fdbbackup list were meant to be directly usable for backup management operations but they were missing the bucket URL parameter.
2019-03-05 21:14:21 -08:00
82d957e0bb
Merge pull request #1178 from vishesh/task/issue-963-IPv6
...
IPv6 Support
2019-03-05 17:14:16 -08:00
96ee95b9ad
fix: macOS build #963
...
Use the boost representation of IPv6 address internally
and make sure it uses std::array.
2019-03-05 14:03:14 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
e93cd0ff21
Add some checks and comments to IPv6 changes #963
2019-03-04 14:12:45 -08:00
592e224155
net: add/use formatIpPort to format IP:PORT pairs #963
2019-03-04 14:12:45 -08:00
82b2da4b78
net: Add IPAddress::parse() util #963
2019-03-04 14:12:45 -08:00
25daabdc02
net: TraceEvent and toIPVectorString for new IPAddress structure #963
2019-03-04 14:12:45 -08:00
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
1a550712cb
Add serialization support for std::array
2019-03-04 13:33:10 -08:00
75b2546e35
Allow FlowLock::take to succeed if there's no other active requests.
2019-03-04 01:42:38 -08:00
98bb58628f
Upgrade FlowLock from int to int64_t.
2019-03-04 01:42:38 -08:00
a462359af1
Implement g_random->randomSkewedUInt32()
...
This generates an exponential distribution, implemented as being
a "log-uniform" distribution.
2019-03-04 01:42:38 -08:00
39f612d132
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-03-02 17:07:00 -08:00
94385447bc
Status: Get if client configured TLS
...
To understand if all clients have configured TLS,
we check the tlsoption when a client tries to open database.
This is similar to how we track the versions of multi-version clients.
2019-03-01 15:17:01 -08:00
655c9d82c7
Various cleanup from review
2019-03-01 14:06:47 -08:00
13f7f5a073
Bump protocol version
2019-03-01 13:01:10 -08:00
e2bcecb08f
Merge branch 'master' into ratekeeper-batch-priority-limits
2019-02-28 12:52:44 -08:00
eb629d87a5
Add information about batch ratekeeper to status. Make it possible to track latencies in the ReadWrite workload for concurrently run instances separately.
2019-02-28 09:53:16 -08:00
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
82e429f417
Upgrade BinaryReader to support calls ArenaReader has.
2019-02-26 18:00:55 -08:00
abfe057805
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-25 13:47:16 -08:00
78aad255b0
updateProcessStats gets process stats directly
...
Storage servers no longer parse ProcessMetrics trace lines to get their
own cpuUsage and diskUsage statistics.
2019-02-25 13:45:53 -08:00
b8910ba7cd
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.h
# fdbserver/DataDistribution.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-22 14:38:13 -08:00
d008de576e
Merge pull request #1139 from xumengpanda/mengxu/machine-team-upgrade-PR
...
Add background actor to remove redundant teams
2019-02-22 14:22:07 -08:00
6660ccd936
Large rewrite/refactor from the page format outward. Replaced the in-page format, PrefixTree, with more general and templated implementation of the same concept, DeltaTree, which will be incrementally modifiable soon. New implementation for in-page cursors to use a per-page arena and share decoded binary tree nodes between cursors in the same page. New implementation for internal record B+Tree cursor which uses reference counted path hops which makes cloning a cursor very cheap. New implementation of external facing B+Tree cursor which makes efficient use of two internal cursors to find user-visible KV pairs and return them without copying keys and values except in the case of a large sharded value.
2019-02-21 02:46:30 -08:00
171d119d95
another attempt to fix compiler errors
2019-02-20 18:35:08 -08:00
fa96b8dd33
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-20 16:56:16 -08:00
4be90c91d6
Moved THIS and THIS_ADDR to actorcompiler.h
2019-02-19 15:16:59 -08:00
3bd9b9047b
Minor fixes - flow now compiling with intellisense
2019-02-19 15:16:59 -08:00
2c16475906
Added a general copy-constructor for intellisense
...
If a actor returns something of a different type that is
assignable, it will work because the actor compiler will
compile this to an assignment. However, if you compile
without the actor compiler, this will fail.
2019-02-19 15:16:59 -08:00
eb8285f14f
Add mock description for actors for intellisense
2019-02-19 15:16:59 -08:00
9f8c060343
Put assertions in its own scope
...
Some if-else statements in the code don't use
curly-braces, which is problematic if the only
statement used is an assertion. This should fix
that.
2019-02-19 15:16:59 -08:00
801e4c4700
Make when-macros work by accepting varargs
2019-02-19 15:16:59 -08:00
e05b53d755
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-15 20:37:07 -08:00
6d09ac483c
Merge with master
2019-02-15 17:03:40 -08:00
83060c6e56
Merge pull request #1062 from jzhou77/PR
...
Add a new DataDistributor role.
2019-02-15 13:51:27 -08:00
8099a0b665
Merge pull request #1129 from alexmiller-apple/tstlog-1
...
Spill-by-reference TLog Part 1: IDiskQueue::read()
2019-02-15 11:48:14 -08:00
75f692b931
simplify actorcompiler and target to compile coveragetool
2019-02-15 00:01:42 -08:00
5e6577cc82
Final cleanup per review comments
...
Make distributor interface optional in ServerDBInfo and many other small
changes.
2019-02-14 16:37:17 -08:00
578473a974
Various review comments fixes
2019-02-14 16:37:16 -08:00
907446d0ce
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-14 11:37:38 -08:00
b435d51061
Merge branch 'master' into track-server-request-latencies
2019-02-14 08:07:32 -08:00
5481851e82
TeamCollection: Add knobs for team remover
...
Added three knobs to control team remover
bool TR_FLAG_DISABLE_TEAM_REMOVER:
Disable the teamRemover actor
double TR_REMOVE_MACHINE_TEAM_DELAY:
Wait for the specified time before try to remove next machine team
double TR_WAIT_FOR_ALL_MACHINES_HEALTHY_DELAY:
Wait before checking if all machines are healthy
2019-02-13 15:11:56 -08:00
9272a41e5f
Merge pull request #1146 from atn34/fix-actor-warning
...
Fix actor warning for cmake build
2019-02-13 11:01:37 -08:00
1ea58c1e5e
Actually change name of flag
2019-02-13 10:44:58 -08:00
3a38bff8ee
Use DISABLE_ACTOR_WITHOUT_WAIT_WARNING consistently
2019-02-13 10:30:35 -08:00
2570b37e6e
Add function to read pages from RawDiskQueue_TwoFiles
2019-02-12 23:43:10 -08:00
601b229c05
Add a no-op breakpoint_me for easy gdb'ing.
2019-02-12 23:43:10 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
f257cca1bc
Update `Void _ = wait(...)` occurrences
2019-02-12 17:30:30 -08:00
df3454114e
Add TODO
2019-02-12 11:03:08 -08:00
8b0e593f83
Add --disable-actor-without-wait-error flag to actorcompiler
2019-02-12 11:01:17 -08:00
d348846a10
Introduce ErrorMessagePolicy class
...
This encapsulates logic for how to report particular error messages.
So far only used for reporting an actor not containing a wait statement,
as this warning should be suppressed in certain contexts.
We could change other error messages to use this class, but since they
don't have any logic I don't think the extra indirection is appropriate.
2019-02-12 11:01:17 -08:00
6051c8027b
Allow osx tooling to understand c sharp
...
After this change I am able to load foundationdb.sln with vscode and get
features like jump to definition and autocomplete.
IntermediateOutputPath will now use a default value.
2019-02-12 11:00:24 -08:00
b717750c4f
Minor README changes.
2019-02-11 21:40:14 -08:00
685242fbfc
Merge pull request #1092 from atn34/trace-format-network-option
...
Add trace_format network option
2019-02-08 10:52:50 -08:00
06e216f26e
fixed linker error
2019-02-07 15:37:04 -08:00
e45295a1f5
Added support for TLS
2019-02-07 15:37:04 -08:00
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
7a858b902d
Most of FDB compiling on Windows with cmake
2019-02-07 15:37:04 -08:00
d4349293b9
Reworked the way latency counters are tracked. Report the latency bands in separate events from StorageMetrics and ProxyMetrics. Fix a problem when the latency band configuration was changed. Add correctness testing.
2019-02-07 13:39:22 -08:00
0750dc0418
Change store from (Future, T&) to (T&, Future).
...
LHS = RHS, and the name of what's being modified is easier to find.
2019-02-04 18:04:22 -08:00
fa6f43b1f3
Added missing 'static' for actor fwd declarations
...
There is a bug in the code that when an ACTOR forward
declaration is static, the static keyword doesn't get
generated in the forward declaration. This commit fixes
that.
2019-02-01 14:44:47 -08:00
e9ddd94e27
The failure monitor is given a list of all IP addresses associated with a process
...
The connect packet includes the correct remote address
Did a lot of code cleanup
Simulation test mixed TLS and non-TLS listeners on the same process
2019-01-31 18:20:14 -08:00
9087f2948e
Support serialization of std::map objects
2019-01-31 11:24:04 -08:00
0e5db1e7b7
Merge pull request #1096 from mpilman/features/actor-forward-declarations
...
Allow forward declarations to be actors
2019-01-30 13:30:49 -08:00
e055fdab03
Better diagnostics for unrecognized trace format
2019-01-29 13:08:47 -08:00
01730cc461
Allow forward declarations to be actors
...
If an actor is forward declared it is kind of
awkward as the signatures don't fit. For example:
Future<Void> foo(int const& bar);
is the correct forward declaration for this actor:
ACTOR Future<Void> foo(int bar);
This also means that a lot of tooling (like rtags/cquery etc)
doesn't work correctly. It will show errors that the function
calls are ambiguous and "find references"-features typically
don't work.
Allowing to do this in the actor compiler is the first step. The
next will be to refactor all forward declarations.
2019-01-29 10:57:54 -08:00
dc2b740415
Added server_overloaded error and client message
2019-01-25 13:15:19 -08:00
1d7fec3074
Merge commit '048bfc5c368063d9e009513078dab88be0cbd5b0' into task/tls-upgrade-2
...
# Conflicts:
# .gitignore
2019-01-24 17:43:06 -08:00
ec32d3308b
Merge pull request #1086 from mpilman/features/c++-compiler-errors
...
Fixed several minor code issues
2019-01-24 15:24:33 -08:00
79637f07ac
Fixed several minor code issues
...
These will become a problem as soon as we
switch to C++17
2019-01-24 14:43:12 -08:00
4925b61d0e
Bump protocol version.
2019-01-24 13:49:22 -08:00
7fde75c439
Avoid possibly concurrent reassignment of dl_iterate_phdr.
2019-01-24 13:28:45 -08:00
efe06d71ce
Change profilingEnabled to thread_local to eliminate a race.
2019-01-24 13:27:16 -08:00
2198d24ce1
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/ServerDBInfo.h
# fdbserver/Status.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/storageserver.actor.cpp
2019-01-24 11:43:26 -08:00
8e05e95045
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-18 16:18:34 -08:00
7498c2308c
Merge branch 'release-6.0' into track-server-request-latencies
2019-01-16 13:39:01 -08:00
8b3cfa469c
Fix spelling
2019-01-15 19:30:09 +00:00
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
6276d6b4f4
Merge pull request #1026 from etschannen/master
...
Merge 6.0 into master
2019-01-09 16:23:35 -08:00
684a22a52b
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BackupContainer.actor.cpp
# fdbclient/HTTP.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/BackupCorrectness.actor.cpp
# versions.target
2019-01-09 16:14:46 -08:00
5048795554
Add JsonTraceLogFormatter to flow/CMakeLists.txt
2019-01-09 15:58:18 -08:00
7c5b2ab330
Merge pull request #976 from alexmiller-apple/jsonlogs
...
Allow trace logs to be output as JSON instead of XML
2019-01-09 17:04:50 -05:00
5d2b11cba9
Merge pull request #1019 from satherton/http-request-id
...
Backup usability and sanity check improvements
2019-01-09 13:29:37 -08:00
51b89ae083
WIP
2019-01-09 07:41:02 -08:00
d265517156
Fix: fast allocator would not cleanup memory for a thread if that thread never called getMagazine. This could happen if the first thing the thread did was to release memory.
...
Added a new metric for the number of threads that hold memory for each size and improve some existing metrics.
Fix: a failed ASSERT would crash if done early in the program lifetime.
2019-01-08 14:36:01 -08:00
0ec216a1fa
Added X-Request-ID header to HTTP requests and verification of matching ID in response, if present.
2019-01-07 17:56:38 -08:00
75fd8e6e17
Include <sys/sysmacros.h>
2019-01-03 12:44:29 -08:00
dbe9baff1f
Several small compilation fixes for new versions of gcc
...
There are several missing includes for cmath in the code, I added those.
Next, Coro returns a reference to a stack variable and this causes a
warning. As this is probably ok for Coro, I disabled the warning in
that file for GCC. I want to have this warning in the build system as
it is generally a very useful warning to have.
Another change is that major and minor are deprecated for a while now.
I replaced those with gnu_dev_major and gnu_dev_minor.
ErrorOr currently implements operators ==, !=, and <. These do not
compile because Error does not implement ==. This compiles on older
versions of gcc and clang because ErrorOr<T>::operator== is not used
anywhere. It is still wrong though and newer gcc versions complain.
I simply removed these methods.
The most interesting fix is that TraceEvent::~TraceEvent is currently
throwing exceptions. This is illegal behavior in C++11 and a idea in
older versions of C++. For now I simply removed the throw, but this
might need some more thought.
2019-01-03 12:44:19 -08:00
3f399dd9be
Indent within anonymous namespace
2019-01-03 09:21:09 -08:00
aa2a76ef4c
Merge pull request #981 from alexmiller-apple/cmake
...
Add a CMake build system
2019-01-02 18:50:15 -08:00
80dd1efaf6
Assign to g_traceLog.formatter
2019-01-02 16:42:00 -08:00
6a4d87802b
Replace & operator with variadic function
2018-12-28 11:33:42 -08:00
f64d5321e9
Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL.
2018-12-20 18:05:23 -08:00
b8df5acc15
Add --trace_format flag to fdbserver
2018-12-20 15:02:01 -08:00
33a4aac249
Fix build
2018-12-20 11:55:59 -08:00
4b31d209da
Emit valid json for unprintable characters
2018-12-20 11:43:18 -08:00
cdc7e83993
Update flow/Trace.cpp
...
Co-Authored-By: alexmiller-apple <35046903+alexmiller-apple@users.noreply.github.com>
2018-12-14 11:30:40 -08:00
a5401b834b
Integrate JsonTraceLogFormatter into build system + more.
...
Where more is:
* Changed the default format to xml, so that all traces don't suddenly
change as of this commit. Someone is going to need to start passing a
flag to all their servers, but let's worry about that later.
* Changed the copyright headers to be the correct filename.
* Include paths should be specified from the root.
2018-12-13 22:07:16 -08:00
2389cf65f3
fixed escaping bug
2018-12-13 21:50:01 -08:00
aa1eff8346
formatted json log formatter
2018-12-13 21:49:46 -08:00
40890e9dbe
Implemented a json log formatter
2018-12-13 21:46:02 -08:00
a982b9da72
Additional changes from a merge commit.
2018-12-13 17:13:41 -08:00
c654b1bb61
First version of working rpm packages
2018-12-13 15:55:31 -08:00
550daa05f8
Default configuration now builds.
2018-12-13 15:52:27 -08:00
df0f491c29
Some more improvements to the build and preparations for packaging
2018-12-13 15:04:13 -08:00
dce290909d
fdbserver now compiling
2018-12-13 14:13:47 -08:00
51beb8b48c
fdbrpc compiling with cmake
2018-12-13 14:02:16 -08:00
f5fcb666c5
flow compiling with cmake on OS X
2018-12-13 13:58:06 -08:00
43e5a46f9b
Change Endpoint::address(NetworkAddress) to vector<NetworkAddress>
...
Extend `Endpoint` class to take multiple NetworkAddresses instead of
just one. Hence, to talk to an endpoint instead of one IP:PORT, we'll
have multiple IP:PORT pairs.
This patch simply adds the field and makes changes to compile the
codebase. The first element of of `address` field is used everywhere.
Hence the way we talk to remains same with this patch.
NOTE:
Directly accessing the first memeber of Endpoint::address is unsafe
as Endpoint() doesn't enforces non-empty address list. However, since
the correctness test pass for now and are anyway replacing all those
unsafe accesses with ones considering the whole vector, this patch
ignores to access them in safe way.
2018-12-13 13:36:52 -08:00
42dffd4dff
Take a vector of network addresses from CLI to start FDB server
...
Extends the CLI interface to take multiple public and listen addresses.
We however do not do anything with those extra addresses and just
consider the first one for now.
2018-12-13 13:36:52 -08:00
e8e01b2406
Remove unused localAddress parameter from newNet2 and Net2 classes
2018-12-13 13:36:52 -08:00
d2d68aa171
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# versions.target
2018-12-03 18:26:52 -08:00
55a9c4a0f0
Merge pull request #955 from ajbeamon/fix-bad-error-creation-and-whitespace
...
throw platform_error; -> throw platform_error();. Convert some spaces to tabs.
2018-12-03 15:12:37 -08:00
e2673b2bbc
Fixed typo causing mac build failure.
2018-12-02 22:03:21 -08:00
50c9dfdd01
Errors that occur in platform that are the result of IO issues are now raised as io_error rather than platform_error.
2018-11-30 10:55:19 -08:00
eb2f27b8e5
Work in progress implementation of server-side latency tracking. The intent of this is to be able to measure the number of requests that achieve certain latency targets across the system relative to the total number of requests.
2018-11-30 10:46:04 -08:00
97847f517b
throw platform_error; -> throw platform_error();. Convert some spaces to tabs.
2018-11-28 12:56:57 -08:00
1f3b6e8bdf
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/BackupContainer.actor.cpp
# fdbclient/BlobStore.actor.cpp
# versions.target
2018-11-27 14:41:46 -08:00
8b021d38d8
Merge pull request #939 from satherton/backup-smaller-snapshot-folders
...
New backup folder scheme which results in far fewer kv range folders
2018-11-27 11:18:52 -08:00
b91b26ef75
Attempt at workaround of a rare issue where long running backup processes reach a state where DNS resolution requests always time out but other processes on the same host can still resolve successfully. In case this was somehow caused by a bad boost tcp_resolver state, each request now uses a unique tcp_resolver instance.
2018-11-26 20:02:03 -08:00
32f434b2ee
Bug fix, dns resolution would throw an error if any of the results were IPv6 addresses, which could happen depending on the host networking configuration.
2018-11-26 19:53:46 -08:00
ec9410492d
Changed backup folder scheme to use a much smaller number of unique folders for kv range files, which will speed up list and expire operations. The old scheme is still readable but will no longer be written except in simulation in order to test backward compatibility.
2018-11-23 05:23:56 -08:00
4e54690005
Merge branch 'release-6.0'
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/MoveKeys.actor.cpp
2018-11-12 20:26:58 -08:00
9d04934f11
Don't allocate magazines backed by hugepages on systems that support them.
...
Doing so causes an excessive amount of memory to be wasted. See issue #909 .
2018-11-12 13:45:48 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
16909fb51d
HTTP header field names are now treated as case insensitive.
2018-11-09 16:57:01 -08:00
81dc8b2d05
Fix crash when bad filename is specified for profiling output. Also removed some unused code and added a couple return value checks.
2018-10-25 09:35:19 -07:00
7c6cf47e37
Revert "Fix operator problems like those mentioned in GitHub issue #606"
2018-10-23 17:21:48 -07:00
47dd24a9fa
Fix operator problems like those mentioned in GitHub issue #606
...
So the code as written works well on older gcc, but does not like newer gcc. I
don't fully know why member operators seem to be more in conflict with different
compiler versions, but experience has taught me to always use non-member
operators for maximum portability across compilers.
Make a small change to that effect here.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-23 23:24:34 +00:00
0a296fc471
Merge pull request #797 from apkar/master
...
Adding Flow documentation.
2018-10-22 16:50:37 -04:00
6bb1f4093d
Merge pull request #856 from dropbox/pr/include-fix
...
Adjust all includes to be relative to the root.
2018-10-22 09:51:55 -07:00
e2fc1c9b95
Remove specifying non-root directory as a path to search for includes.
2018-10-19 18:56:45 -07:00
888378c8cb
fixed windows build
2018-10-19 13:28:32 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
8dd900a337
fixed the windows build
2018-10-18 20:26:45 -07:00
db71b60d72
Merge pull request #819 from satherton/feature-redwood
...
Redwood storage engine, initial/experimental version
2018-10-18 18:38:11 -07:00
22f8a4efa9
Normalized all unit test names to begin with "/" if they should be included in random unit testing.
2018-10-05 22:09:58 -07:00
57ce1ad7c3
Merge pull request #817 from apple/release-6.0
...
Merge release-6.0 into master
2018-10-05 15:28:41 -07:00
664f64881c
Port truncate optimization from Snowflake PR in order to make quick changes for a patch release.
2018-10-05 15:05:26 -07:00
7c1dc305cb
Merge commit 'a72c8f5cb2e79a673abc0ed3d27ef1c51028fb13' into feature-redwood
2018-10-05 10:15:10 -07:00
5d9cd9acdc
Correctness test now has additional random reader which doesn't do verification but isn't stopped when the btree is closed. Fixed bug exposed by this where pager snapshots will still try to read pages after the pager has been shut down or even destroyed. Added new error type, shutdown_in_progress.
2018-10-04 23:46:37 -07:00
c06c214f0b
Fixing typos in flow/README.md
2018-10-04 17:33:16 -07:00
3922e477a5
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/LogSystemDiskQueueAdapter.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00
62fbd7958a
Adding Flow documentation.
2018-10-02 11:03:31 -07:00
31b297e756
Merge pull request #755 from ajbeamon/fix-actor-compiler-warnings
...
Fix actor compiler warnings related to TEST_CASEs not having wait() s…
2018-09-24 10:11:32 -07:00
2fc86c5ff3
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbrpc/AsyncFileCached.actor.h
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-09-20 03:39:55 -07:00
10a8e67266
Merge remote-tracking branch 'upstream/release-6.0' into merge-release-6.0
2018-09-11 21:49:59 -07:00
4773b788e9
Fix line endings
2018-09-10 09:44:47 -07:00
309159ea23
WriteValue() no longer uses format() which creates temporary strings, instead if writes directly into allocated space in the target buffer using snprintf().
2018-09-10 00:44:09 -07:00
ce3f01a0cf
Added concept of type to JsonString. Appending single items or key/value pairs is now type-safe and only allowed in certain cases. JsonString will refuse to produce invalid JSON. All duplicative functions have been replaced with templates. Encoding of values uses json_spirit's value writer which should be no worse performance than format() and it will escape everything properly. Final string form is now built directly using knowledge of type, such as when an instance becomes an Array or Object the appropriate opening character is written. This avoids a full copy just to prepend the opening character later. Index interface for key/value pairs no longer makes a temporary copy of the key string. JsonString is now only needed by Status.actor.cpp. Still more work to be done here.
2018-09-08 07:15:28 -07:00
d3c8d7ab4e
fix: status would generate invalid json
2018-09-07 18:26:05 -07:00
e918ba7a4f
Fix actor compiler warnings related to TEST_CASEs not having wait() statements and the DescrCompiler having unused variables.
2018-09-06 16:14:04 -07:00
471cf84a47
Merge pull request #751 from apkar/master
...
Merge release-6.0 into master
2018-09-06 14:33:08 -07:00
920fd3fe97
Merge branch 'release-6.0'
2018-09-06 14:24:02 -07:00
7f0a70db7f
Remove defunct DebugQueryRequest.
2018-09-06 13:44:25 -07:00
425ed6c693
Fixed memory leak in JsonString operator[](), fixed minor document byte count error.
2018-09-06 02:28:23 -07:00
fca1451186
fix: hasKey was not set properly
2018-09-05 23:46:31 -07:00
98651bafb1
removed _keyNames from JsonString
2018-09-05 22:51:15 -07:00
4dd2dda0a3
Merge branch 'release-6.0'
...
# Conflicts:
# fdbserver/worker.actor.cpp
2018-09-05 16:11:06 -07:00
df406a340e
Merge pull request #742 from ajbeamon/roles-in-trace-events
...
Add the roles running on a process as a field on trace events in the …
2018-09-05 16:08:12 -07:00
90301f497f
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/TLSConnection.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# versions.target
2018-09-05 16:06:33 -07:00
9e79f9ec59
Cleanup from review
2018-09-05 15:53:12 -07:00
2de0b5d6d7
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-05 15:06:14 -07:00
821adeae4f
Added method to swap in json text
2018-09-05 14:17:33 -07:00
6aa22af83b
Added explicit keyword for appropriate constructor
...
Templatized some methods
Removed unused hash function
2018-09-05 12:22:04 -07:00
4eaff42e4f
Merge pull request #712 from ajbeamon/remove-database-name-internal
...
Eliminate use of database names (phase 1)
2018-09-05 10:35:00 -07:00
43a2afc3b6
Added TODO comment
...
Removed debug comments
2018-09-05 08:06:19 -07:00
04a768042a
Added TraceEvent to measure time to create Status Json
...
Simplified JsonString class to use implementation method for reuse of methods
Removed quotes from non-string values within json
Added Tests for jsonstring
Removed hashing of names for JsonString
Switched name tracker to unordered set
2018-09-05 03:50:53 -07:00
affd7423b4
Added class to write json as objects are added
...
Integrated JsonString class into status
2018-08-31 01:21:24 -07:00
383cac4991
This checkin is an attempt to support long keys by placing oversize key suffixes (as much as will fit) into a page and forcing the prefix bytes that do not fit to be part of the parent's page boundary leading to this page. This strategy is being abandoned but there are some valuable refactorings and debugging enhancements that are still in the right direction so they are being committed.
2018-08-28 13:46:14 -07:00
717c43a69f
merge 6.0 into master
2018-08-22 00:28:04 -07:00
ec1dfe8ae1
Some code cleanup / renaming of things for clarity. Improved StringRef hex string output to omit a middle segment when there is an exceeded length limit.
2018-08-16 16:22:10 -07:00
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
0b4c06169b
Convert tabs to spaces in ActorParser
2018-08-14 15:50:26 -07:00
63b1e85338
Ban `Void _ = wait(...)` constructions, and require just `wait(...)`.
...
There's never any reason to save the value of a Void return, and it's
the easiest source of redefined variable bugs that will creep back in
over time. So just `wait(...)`, it's cleaner that way.
2018-08-14 15:50:26 -07:00
809b2ecdd7
Add a missing #include <algorithm> to flow/Util.h
2018-08-14 15:50:26 -07:00
7feb5d8209
Remove including flow.h in actorcompiler.h, and fix resulting breakage.
...
For files that required flow.h, and only got it through actorcompiler.h,
their version of flow.h would have the actorcompiler #defines defined.
Then, if it included a STL/boost file, the same breakage would result.
This needs to not happen, so the include of flow.h in actorcompiler.h
was removed.
2018-08-14 15:50:26 -07:00
bca324eaa6
More actorcompiler.h fixes and additions.
2018-08-14 15:50:26 -07:00
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
6e7f8da694
Add includes for missing definitions of size_t and uint32_t.
2018-08-14 15:50:26 -07:00
3a0366d4c5
Add clang as a supported compiler.
2018-08-14 15:50:26 -07:00
07e5281142
Restrict actor keyword #defines to actor files.
...
This introduces a new rule in our codebase, that any file that #includes
actorcompiler.h needs to do it as the last #include, and it needs to
then #include unactorcompiler.h at the end of the file.
The point of this is that it prevents our actorcompiler.h #defines from
leaking into boost or the c++ standard library. Both of these start
throwing errors if you s/state// their code, which `#define state `
effectively does.
2018-08-14 15:50:26 -07:00
805781ed32
Forward declare instead of #include in actorcompiler.
...
For some reason, clang thinks that the #ifndef's aren't matched by
#endif's if we #include "flow.h" in the middle. Regardless, it's
better for compile time slightly to forward declare, so let's just do
that anyway.
It also maybe gets us away from a weird
actorcompiler.h -> flow.h -> genericactor.actor.h -> actorcompiler.h
circular include.
2018-08-14 15:50:26 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
1901be1e63
Allow `wait()` instead of `Void _ = wait()`.
2018-08-14 15:49:47 -07:00
cdcf056aef
Merge branch 'release-6.0'
2018-08-14 09:43:51 -07:00
168dce94cb
Remove some trace event suppressions that were happening off the network thread. Downgrade some trace events related to trace logging problems from SevError to SevWarnAlways.
2018-08-14 09:00:43 -07:00
3186fac397
Make sure we still accept some connections even if we are CPU bound by high priority work
2018-08-10 17:47:21 -07:00
574c5576a2
Merge branch 'release-6.0' of github.com:apple/foundationdb
...
# Conflicts:
# fdbrpc/TLSConnection.actor.cpp
# versions.target
2018-08-10 14:31:58 -07:00
3535ddad80
Merge pull request #674 from alexmiller-apple/glibcxx-debug-fixes
...
Fix bugs uncovered by -D_GLIBCXX_DEBUG
2018-08-09 08:18:51 -07:00
f4acaabe4a
flow/Util.h gets a #pragma once
...
And you get a #pragma once! Everybody gets a #pragma once!
2018-08-08 16:05:49 -07:00
c8bc9b522c
Document our protocol version convention and add a static assert to help prevent erroneous edits.
2018-08-08 13:57:47 -07:00
db0a03a8aa
Increase the protocol version on master to be distinct from the 6.0 protocol version. Also realign it with our convention that the x and y digits match the release version.
2018-08-08 13:26:51 -07:00
b5a133865d
Merge branch 'release-6.0' of github.com:apple/foundationdb into release-6.0
...
# Conflicts:
# fdbrpc/TLSConnection.actor.cpp
2018-08-06 18:26:54 -07:00
fb46385a39
Merge pull request #628 from alexmiller-apple/reloadcertificates
...
Reload certificates if changed.
This is a cherry-pick of #628 back to release-6.0
2018-08-06 18:04:04 -07:00
c3c5dd2805
do not suppress loadlibrary and loadfunction trace events
2018-08-06 17:55:24 -07:00
6f328d41ac
suppressed spammy trace events
2018-08-06 12:12:55 -07:00
be1a4d74c7
tlogs serve reads to log routers at a low priority, to prevent them from using all their resources catching up a remote dc that has been down for a long time
...
increase the amount of memory ratekeeper budgets for tlogs so that there is a gap after the spill threshold to prevent temporarily overshooting the budget
2018-08-04 10:31:30 -07:00
538e684f1c
Merge branch 'release-6.0'
...
# Conflicts:
# versions.target
2018-08-03 11:41:46 -07:00
65057b4788
Merge branch 'release-5.2' into release-6.0
...
# Conflicts:
# documentation/sphinx/source/downloads.rst
# documentation/sphinx/source/release-notes.rst
# fdbclient/MasterProxyInterface.h
# packaging/msi/FDBInstaller.wxs
2018-08-02 11:29:40 -07:00
9f64dd945b
remove an unused length from trace event
2018-08-02 10:49:26 -07:00
1a7cda4149
Stop performing self-moves. (e.g. a = std::move(a))
...
self-moves are frowned upon in C++, and in our code this generally happens from
calls to swap as part of trying to implement a "unordered erase" function via
swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now
offered that performs this, while disallowing self-moves.
2018-08-01 18:09:54 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
a361a785e8
fix: clients which cannot talk to storage servers poll the proxy for new storage server interfaces. If there are too many clients polling, we saturate the proxies with these requests, prevents the storage servers from updating their interfaces.
2018-07-31 16:57:23 -07:00
40762d9f9b
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-07-25 17:58:52 -07:00
262af775eb
Implement overly simple file write timestamps for simulation, and clean up code.
2018-07-24 17:20:31 -07:00
9d391498e8
Refactored how key/value memory is held in PrefixTree and VersionedBtree, eliminated many unnecessarily copies of large strings. PrefixTree tests pass but btree is still broken, just committing this because it's a large change set.
2018-07-23 03:09:13 -07:00
168496f819
Poll the certificate files if TLS is enabled and reload them if changed.
...
This allows certificates to be changed/updated without having to restart fdbserver.
2018-07-20 19:00:32 -07:00
80d8197223
Spaces -> tabs
2018-07-18 08:40:26 -07:00
227b55c8f9
Merge branch 'release-6.0' into on-main-thread-fix
2018-07-16 16:12:30 -07:00
a067f8812d
When calling onMainThread from the network thread, first empty the queue of tasks scheduled from other threads. This way, tasks scheduled from the main thread can't jump ahead in line.
2018-07-16 16:10:24 -07:00
e0caa28758
code cleanup
2018-07-16 15:56:43 -07:00
f72a9f60c0
only disable fearless if a datacenter has actually been killed
...
fix: we must prevent recovery into the dead datacenter while reducing usable_regions
2018-07-16 10:06:57 -07:00
96389c74cd
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-07-10 16:42:34 -07:00
fed92be47d
bump protocol version for release
2018-07-10 10:17:51 -07:00
be9c34c6f8
Merge remote-tracking branch 'upstream/release-5.2' into merge-release-5.2
2018-07-10 10:04:48 -07:00
1bc95862b7
Merge branch 'release-6.0' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-07-10 04:16:02 -07:00
3ce7c78d36
If an HTTP request fails due to a connection failure or a timeout, do not convert the error to the more generic http_request_failed.
2018-07-09 18:58:33 -07:00
5a2cb3037b
merge 5.2 into 6.0
2018-07-08 20:14:06 -07:00
fee4234e6b
Bug fixes in memory activity logging.
2018-07-06 19:02:42 -07:00
2925b9b984
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-07-03 23:03:56 -07:00
e17c04f666
Changed assignments to substr() to resize() for clarity/simplicity.
2018-07-02 14:25:24 -07:00
e67f951c06
Merge branch 'master' into feature-remote-logs
2018-07-02 02:18:20 -04:00
c3f88dbfe1
Merge branch 'master' of github.com:apple/foundationdb into tls-static
2018-07-01 23:13:57 -07:00
b24e272394
Merge branch 'master' into feature-remote-logs
2018-07-02 00:07:25 -04:00
5054c194e2
Some trace events are logged before FLOW_KNOBS are initialized
2018-07-01 14:30:37 -04:00
b95a2bd6c1
Merge commit 'b17c8359ec22892ed4daeaa569f2f5e105477251' into feature-redwood
...
# Conflicts:
# flow/Trace.cpp
2018-06-30 23:18:29 -07:00
b17c8359ec
Merge pull request #549 from apple/release-5.2
...
Merge release-5.2 into master
2018-06-30 22:50:07 -07:00
3d32fc4e5d
Reduced MacOS aligned_alloc fix to the most minimal change which fixes the crash while not technically changing the aligned_alloc() contract. The previous version allowed new bad behavior (a non power of 2 alignment less than sizeof(void *)) to succeed and it also would throw an exception on failure instead of just returning nullptr as the other platform implementations would.
2018-06-30 21:37:32 -07:00
3f57d0226c
Crash fix on MacOS. Aligned_alloc() would silently fail for alignments less than 8, which happen in Deque<T> for small T's such as Void or Error.
2018-06-30 17:16:19 -07:00
a680837ee4
The Machine field was missing in early trace events. The logGroup field was not being properly set.
2018-06-28 15:28:58 -07:00
1ff42e078f
Add missing include for Windows, remove throw from TraceEvent destructor.
2018-06-28 14:59:23 -07:00
09624aeec9
Add include statement for std::function to try to make Windows build happy
2018-06-28 10:22:33 -07:00
cbc840ad0a
Move the spammy trace event backstop from the destructor to the constructor. This allows us to avoid doing needless work on a trace event that is going to be throttled.
2018-06-27 15:51:30 -07:00
7378ec85f0
Fixes parentDirectory to work when our path is a child of the root directory. Also works when the path has trailing slashes.
2018-06-26 16:30:47 -07:00
fec225075f
Merge branch 'master' into trace-log-refactor
2018-06-26 14:54:42 -07:00
fe956bc35a
Address review comments
2018-06-26 14:37:21 -07:00
ef8de426d3
Changed the TLS_DISABLED macro
...
Disable TLS within Windows until working
2018-06-26 12:08:32 -07:00
9f545ce002
Merge commit '892727e358c0b3f075564c60c2b7cedb64306f83' into trace-log-refactor
2018-06-26 11:37:23 -07:00
1dce97f28c
Merge branch 'release-5.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/SimulatedCluster.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2018-06-21 17:05:11 -07:00
8bd7eaebdb
fix: broken_promise from push can be throw into the proxy’s actor collection
2018-06-21 15:55:27 -07:00
ed8c2485c8
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-21 14:16:13 -07:00
afdd992fca
Move the computation of total unused allocated memory into FastAlloc.cpp
2018-06-20 14:33:59 -07:00
5e81f4ac7e
Track unused allocated memory in ProcessMetrics and report it in status.
2018-06-20 10:10:51 -07:00
90c8288c68
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-17 14:55:05 -07:00
889889323e
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there.
...
The cluster controller will not switch to the other data center if remote logs are too far behind.
We will not recruit in DCs with negative priority.
2018-06-13 18:14:14 -07:00
1eae9d621b
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-13 15:58:21 -07:00
2878f30f29
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/storageserver.actor.cpp
2018-06-13 15:56:06 -07:00
6f941a89b4
Fix bug in actor compiler that would cause multi-line comments to be marked with the wrong line numbers.
2018-06-13 10:15:20 -07:00
8518f6a8a8
smartQuorum shouldn't return if more responses are desired than futures provided.
2018-06-12 16:50:25 -07:00
372ed67497
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
2018-06-11 11:34:10 -07:00
f965954122
Merge commit '82be52205b95464e355c449fdf3e7d483fa06677' into trace-log-refactor
...
# Conflicts:
# fdbserver/Status.actor.cpp
# fdbserver/workloads/DDMetrics.actor.cpp
# flow/Trace.cpp
2018-06-08 16:22:22 -07:00
8360f71cbb
Merge branch 'master' of github.com:apple/foundationdb into save-fitness-info
...
# Conflicts:
# fdbserver/worker.actor.cpp
2018-06-08 16:09:59 -07:00
06ccd9a500
Allow trace event type names to end with an underscore.
2018-06-08 15:49:31 -07:00
1fdfe20908
Relax the rules on trace event Types a bit by allowing multiple underscores, as well as starting with an underscore and consecutive underscores.
2018-06-08 15:40:29 -07:00
99c9958db7
Some more trace event normalization
2018-06-08 13:57:00 -07:00
0ca51989bb
Merge branch 'master' into trace-log-refactor
...
# Conflicts:
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/Status.actor.cpp
# flow/Trace.cpp
2018-06-08 13:24:30 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
d7d38c3544
Merge pull request #430 from ajbeamon/rename-logGroup-attribute
...
Rename trace file logGroup attribute to LogGroup
2018-06-08 10:30:45 -07:00
69b713918b
VersionedBTree now uses PrefixTree based pages (with bugs). This required significant changes to both classes because the interface and semantics for building, seeking in, and iterating through pages is very different from the previous trivial approach which was based on serialized vectors. PrefixTree node format rewritten to support optional values without increasing overhead for common node scenarios. PrefixTree::Cursor rewritten to reuse path prefix memory instead of allocating new memory on each movement which is then 'leaked' until destruction. PrefixTree::Cursor movement modified to work better with VersionedBTree::InternalCursor, which was also heavily modified. Added knobs related to key arrangement in PrefixTree nodes. Added StringRef::toHexString() as an alternative to printable() to make reading raw PrefixTree data easier. PrefixTree performance is temporarily worse with this update and VersionedBtree fails its unit test.
2018-06-08 03:32:34 -07:00
e4d5817679
fix: we must server getTeam requests before readyToStart is set because we cannot complete relocateShard requests without getTeam responses from both team collections
2018-06-07 16:14:40 -07:00
11b79c6c94
Save fitness info of a process to become a cluster controller. This info is currently lost after a reboot. Save this info and reload it to avoid unnecessary re-recruitments.
2018-06-07 13:07:19 -07:00
529d0497f1
Proxy going OOM when applying high volumes of writes to a proxy, particular in a sudden fashion before ratekeeper can control the workload.
...
Address this issue by proactively monitoring the memory used by commit batches and dropping requests if a certain memory limit is exceeded.
2018-06-01 15:21:40 -07:00
3ea5fc72f0
Rename trace file logGroup attribute to LogGroup
2018-05-31 15:34:17 -07:00
5848055620
More line ending fixes
2018-05-31 13:44:38 -07:00
090117a674
Fix line endings.
2018-05-31 13:41:32 -07:00
f1dea3bed9
Rework the formatter a bit and extract out the writing logic so that it could be pluggable.
2018-05-31 13:27:35 -07:00
f7735d000f
Fix linking issue in flow bindings by using a different variant of printable().
2018-05-31 11:16:01 -07:00
78839b20fd
Merge branch 'master' into trace-log-refactor
...
# Conflicts:
# flow/Trace.cpp
2018-05-31 10:46:20 -07:00
d9c702a9e3
Merge release-5.1 into release-5.2
2018-05-30 09:09:55 -07:00
e2e69b5632
Rework how rolled events are logged. Not very efficient, but it doesn't modify the cached version returned in queries.
2018-05-11 11:24:20 -07:00
02df30149f
Merge branch 'release-5.2' into trace-log-refactor
2018-05-11 11:22:34 -07:00
3fba932f80
Updated Protocol Version
...
Updated version within download documentation
Added old release notes
Updated Wix GUID Product ID
2018-05-10 17:03:46 -07:00
67fa95264c
Merge branch 'master' into feature-remote-logs
2018-05-10 17:01:25 -07:00
c45599f2a9
fix: versionstamped_key uses 4 bytes for size
2018-05-10 17:00:35 -07:00
91338fc984
Merge branch 'master' into feature-remote-logs
2018-05-10 15:33:45 -07:00
8f984cb2c9
Merge branch 'release-5.2'
...
# Conflicts:
# fdbrpc/TLSConnection.h
2018-05-10 09:13:22 -07:00
d3450ce5b0
Merge pull request #343 from bnamasivayam/tls-plugin
...
Tls plugin
2018-05-09 16:35:53 -07:00
fe1c9d5be4
Make the trace log formatter reference counted.
2018-05-09 14:35:15 -07:00
8de914a81f
use contents() instead of address of in withPrefix and withSuffix ; whitespace fixes
2018-05-09 09:01:22 -07:00
d3b5cfb93c
Support latest TLS plugin.
...
Add support for https in backup.
2018-05-08 16:28:13 -07:00
464e2cdbf0
change SetVersionstampedKey and SetVersionstampedValue behavior based on API version to make them consistent
2018-05-08 08:57:09 -07:00
047563aa62
Support trace event overflow.
2018-05-04 15:19:56 -07:00
2a46fdb31d
Change formatv to vsformat, which now returns the length of the string (or <0 for error) and takes the output string as an argument.
2018-05-04 13:35:25 -07:00
ce0c991e78
Refactor trace events to store a vector of fields that aren't encoded until write time. Better support for pre-network trace events. Rework how trace events are queried. Some initial work towards pluggable formatting of logs.
2018-05-02 10:44:38 -07:00
10d25927cd
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
2018-04-30 22:15:39 -07:00
738f3036cf
Merge branch 'release-5.2' into enable-trace-logging-before-open
2018-04-30 15:43:17 -07:00
69e831d522
Merge remote-tracking branch 'upstream/release-5.2' into merge-release-5.2
2018-04-28 17:44:52 -07:00
a1faaafca3
Merge remote-tracking branch 'upstream/release-5.1' into merge-release-5.1
2018-04-27 16:38:18 -07:00
11bd7d7daf
Add a disgusting and terrible hack to avoid "undefined std::istream::ignore(long)" linking errors.
...
This makes building our bindings in the provided docker image possible again.
2018-04-26 16:07:50 -07:00
af61d3596d
Merge branch 'public-master' into feature-redwood
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/OldTLogServer.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2018-04-24 17:22:21 -07:00
bd26d6916c
Support boost 1.52 and newer
2018-04-23 14:51:15 +02:00
d660c96158
Compatible with newer boost versions
2018-04-23 14:07:17 +02:00
2dd0b25259
Use ASSERT_ABORT in destructor
2018-04-23 14:05:26 +02:00
2f8a0240f1
Fix some typos.
2018-04-19 11:44:01 -07:00
9cdf25eda3
Fix some typos.
2018-04-20 00:49:22 +07:00
20082e3228
Clang fixes.
2018-04-12 11:10:53 -07:00
19762b847d
Merge branch 'release-5.2'
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/SimulatedCluster.actor.cpp
2018-04-10 17:02:43 -07:00
c1ba16b3c8
Merge branch 'release-5.1' into release-5.2
...
# Conflicts:
# bindings/java/src/test/com/apple/foundationdb/test/AbstractTester.java
# bindings/java/src/test/com/apple/foundationdb/test/VersionstampSmokeTest.java
# bindings/nodejs/lib/fdb.js
# bindings/nodejs/src/Version.h
# bindings/nodejs/tests/tuple_test.js
2018-04-10 16:50:47 -07:00
b46c32535c
surpassed spammy trace events
2018-04-10 15:52:32 -07:00
7af892f50b
first working version of non-copying recovery working with fearless configurations
2018-04-08 21:24:05 -07:00
2752a28611
Merge branch 'release-5.2' of github.com:apple/foundationdb into feature-redwood
2018-04-06 16:29:37 -07:00
b36e08f08f
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-29 15:12:38 -07:00
62726efa82
Add support for logging trace events before the network is created and the trace log opened.
2018-03-28 08:21:22 -07:00
d3fb17d30a
Merge pull request #74 from bnamasivayam/client-profiling-tests
...
Client profiling tests - Part 1
2018-03-23 16:52:49 -07:00
5db52ab081
Merge pull request #87 from etschannen/feature-remote-logs
...
Feature remote logs
2018-03-23 12:55:17 -07:00
ddc0c613ed
Merge pull request #109 from apple/release-5.2
...
Merge Release 5.2 into master
2018-03-21 09:37:56 -07:00
d4007c27ea
Merge pull request #37 from bnamasivayam/release-5.2
...
Split ProcessMetrics into ProcessMetrics, NetworkMetrics and MemoryMe…
2018-03-20 17:31:46 -07:00
551ea9c7f8
Merge remote-tracking branch 'upstream/release-5.2' into master-release-5.2-merge
2018-03-19 12:34:50 -07:00
0b4564e484
Merge pull request #73 from ajbeamon/release-5.2
...
Reduce severity of GetDiskStatisticsDeviceNotFound to SevWarn.
2018-03-16 15:44:54 -07:00
820382ea68
optimized the log router commit path to avoid re-serializing the data
2018-03-16 11:40:21 -07:00
59723f51f8
fix: continue to attempt to lock logs until remote logs are recovered, this is so that remote logs get locked and readers know they will not have any more data
...
do not throttle trace events in simulation
2018-03-14 12:39:55 -07:00
856d2a0a9d
Add correctness tests for Client transaction profiling data format. It also includes format check across upgrades.
2018-03-14 12:39:50 -07:00
06dd052770
Reduce severity of GetDiskStatisticsDeviceNotFound to SevWarn.
2018-03-14 12:17:54 -07:00
70a05c1a9b
fix some compiler whinges
2018-03-13 15:00:16 -07:00
5390af8be4
suppress spammy logs
2018-03-09 09:40:36 -08:00
336a54faef
Split ProcessMetrics into ProcessMetrics, NetworkMetrics and MemoryMetrics
2018-03-06 15:52:03 -08:00
f2c804e14f
Reverting changes from merge of master into release-5.2 ( b25810711c). Note that we never intend to release master into release-5.2, but if we did we would need to revert this commit.
2018-03-06 10:15:04 -08:00
37a6a81634
Merge commit '7f6fc3e039c911cd84b8540f7f799fc38a1c1822' into feature-remote-logs
...
# Conflicts:
# fdbserver/workloads/RestartRecovery.actor.cpp
2018-02-23 12:33:28 -08:00
de1551c20d
Merge branch 'release-5.1'
2018-02-23 08:24:06 -08:00
a1382895a6
Fixed headers and some whitespace
2018-02-23 04:50:23 -08:00
e1162e9238
Merge remote-tracking branch 'upstream/release-5.1'
2018-02-22 11:16:12 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
dc93759e15
suppressed trace events that are spammy
2018-02-16 16:01:19 -08:00
2eeb714c4b
Trace memory usage of individual FastAllocator of different sizes as part of ProcessMetrics.
2018-02-16 13:30:00 -08:00
03501fc26c
TraceEvent::type is set in init so that it works for TraceIntervals too. ASSERT that the type is non-empty.
2018-02-13 13:29:08 -08:00
0a35f167e4
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/DiskQueue.actor.cpp
# fdbserver/IDiskQueue.h
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
# fdbserver/worker.actor.cpp
2018-02-12 01:30:02 -08:00
42405c78a5
Merge commit '4038bd2fd968d88861f2cebd442ce511724816cb' into feature-remote-logs
...
# Conflicts:
# fdbserver/ClusterController.actor.cpp
# fdbserver/Knobs.cpp
2018-02-10 12:08:52 -08:00
32762648fa
Merge branch 'release-5.1'
2018-02-08 18:38:48 -08:00
1f78b98ac9
Bug fix, result as a state variable causes 'this' to be captured instead of copying result.
2018-02-08 09:54:47 -08:00
69425a303b
Improved error handling for cases where blob account credentials are either not found in the provided credentials sources and/or some of the credentials sources provided are not readable or parseable.
2018-02-07 21:50:43 -08:00
9c7a39d5b2
Rename SevWarnAlways trace event DeviceNotFound to GetDiskStatisticsDeviceNotFound
2018-02-07 12:54:34 -08:00
ebd94bb654
removed a separately configurable storage team size for the remote data center, because it did not make sense
...
fix: the master did not monitor for the failure of remote logs
stop merge attempts when a data center is failed
fixed a variety of other problems with data distribution when a data center is failed
2018-02-02 11:46:04 -08:00
080a454051
fix: getVersionstamp would return broken promise if a transaction was disposed before being set. getAddressesForKey would not return when resetPromise was set.
2018-01-31 13:47:36 -08:00
29c5d4ad3d
upgrades from 5.X mostly supported, still some remaining correctness problems
2018-01-28 11:52:54 -08:00
79d94214a4
Merge commit 'f4ffc9752b5ec66ac47f5f684a5d8be06a7eae6e' into feature-remote-logs
2018-01-25 10:12:06 -08:00
7808099318
bump protocol version for release ; new product guid
2018-01-25 10:02:48 -08:00
66de9d392b
New error code, http_auth_failed, which is used when blob authentication fails instead of the previous generic http_request_failed.
2018-01-22 14:58:56 -08:00
698ef4117e
Merge branch 'master' into feature-remote-logs
2018-01-20 10:34:30 -08:00
4bfbdbf454
Extract getLocalTime to platform.cpp
2018-01-17 11:35:34 -08:00
93b34a945f
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 04:09:43 -08:00
21482a45e1
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DBCoreState.h
# fdbserver/LogSystem.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-01-14 13:40:24 -08:00
660cee0254
increased the priority of getKeyServersLocations, because once a client gets a read version, answering their reads should be higher priority than starting new transactions
2018-01-12 13:46:20 -08:00
de119f192d
fixed a priority inversion where the tlog would prefer to copy data from the previous generation rather than make data durable (leading to being ratekeeper controlled)
2018-01-11 16:09:49 -08:00
2f5073d00f
Some visual studio project cleanup.
2018-01-10 10:07:18 -08:00
3ec45d38a0
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-06 13:54:45 -08:00
Evan Tschannen