d6170210e7
AsyncFileCached: throw (new) exception, through helper static fn, if cache eviction polity not recognized.
2019-05-06 10:11:46 +02:00
fdb3992990
AsyncFileCached: switch to string for eviction policy knob
2019-05-03 14:04:43 +02:00
b2280e15bf
AsyncFileCached: support for lru cache eviction policy
...
- Added a knob to control page cache eviction policy (0=random default, 1=lru)
- Added page cache hit, miss, and eviction stats
2019-05-02 17:35:30 +02:00
e193cac5ef
Merge remote-tracking branch 'apple/master' into tlog
...
Resolve Conflicts: fdbserver/MasterProxyServer.actor.cpp
2019-05-01 17:18:00 -07:00
2d5043c665
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-30 18:27:04 -07:00
4fa1c008f9
Highly prioritize storageServerRejoin messages on the proxy, so that storage servers can rejoin the cluster even when a proxy is CPU saturated
2019-04-23 20:56:01 -07:00
e0f76edf77
Merge pull request #1471 from AlvinMooreSr/release-6.1-merge
...
Merge Release 6.1 Into Master
2019-04-23 11:08:21 -07:00
c25fac9421
Lower the priority of spilled peeks to below that of spilling.
...
This prevents peeking from starving the TLog of CPU time to spill, and
thus impacting write throughput at prolonged saturation.
2019-04-22 18:39:21 -07:00
8b67da57bb
Fix upgrade test failure
...
Serialize pseudoLocalities if protocol version is larger than 0x0FDB00B061060001LL.
Note this version may need to be changed to "currentProtocolVersion" when merging
into the master, and "currentProtocolVersion" should be incremented.
2019-04-21 10:41:07 -07:00
e7ee61660c
Fix unit test failure due to trace event names
...
Capitalize the first letter.
2019-04-20 10:19:53 -07:00
2bea99591e
Merge branch 'release-6.1' of copy of master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-04-17 15:51:48 -07:00
6888baf1c3
Fix compile error
2019-04-16 15:44:55 -07:00
08e66d1764
Prefer success to cast-to-void
2019-04-16 15:29:03 -07:00
781b6ece77
Fix OPEN_FOR_IDE -Wunused-variable warnings
...
CC #1255 , #1173
2019-04-16 15:28:01 -07:00
baa3e806ef
Address review comments from #1446
2019-04-16 09:48:15 -07:00
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
6c7fd1593f
Avoid potential index-out-of-bounds
2019-04-15 15:20:09 -07:00
dced9232d0
ASSERT_WE_THINK trailing byte is '\0'
2019-04-15 09:43:26 -07:00
369c1a0230
Exclude trailing \0 byte for string literals
...
This will "do the right thing" as long as no one expects something like
char a[] = {'a'};
TraceEvent("Event").detail("a", a);
to trace `{..., "a": "a"}`.
It will trace `{..., "a": ""}`
2019-04-11 10:39:48 -07:00
1135331ce6
Include space as a printable character
2019-04-11 10:01:06 -07:00
cd5c9d91fa
Merge pull request #1443 from etschannen/master
...
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
8e05713a5d
do not log a SevError trace event if we cannot deserialize the connect packet
2019-04-10 17:41:02 -07:00
a9b1b32fde
Handle negative input to base16Char
...
base16Char was being called with (c < 0), causing a failed assertion.
2019-04-09 18:14:09 -07:00
6220a5ce0f
Merge pull request #1370 from jzhou77/fix-unreferenced
...
Remove unused functions
2019-04-09 11:49:45 -07:00
058d028099
Merge pull request #1301 from mpilman/features/cheaper-traces
...
Defer formatting in traces to make them cheaper
2019-04-09 10:11:04 -07:00
af17babb9a
Restore a deleted comment line
2019-04-09 09:38:45 -07:00
e4746888eb
Removed dead code
2019-04-09 09:08:45 -07:00
21c0ba555c
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-08 18:38:42 -07:00
05869a8383
do not log a degraded reset message if the previous reset was more than a week ago
2019-04-07 23:00:58 -07:00
19d9b8fcd8
don’t log MachineLoadDetails from getSystemStatistics()
2019-04-07 22:55:19 -07:00
d7612a4426
Fix OPEN_FOR_IDE build errors
2019-04-05 16:30:42 -07:00
d01cbf3455
Addressed code review comments
2019-04-05 13:12:20 -07:00
4287b1d2a1
resolved minor merge issues
2019-04-05 13:12:19 -07:00
b0877e7dd0
Don't quote spaces in std::strings
2019-04-05 13:12:19 -07:00
1c6c73af6d
Fixed infinite recursion
2019-04-05 13:12:19 -07:00
c6bc23791d
Fixed format error in microbench
2019-04-05 13:12:19 -07:00
ea67b742c7
Implemented Traceable for printable types
2019-04-05 13:12:19 -07:00
c9ee7f54e7
Add compound type to micro-benchmark
2019-04-05 13:12:19 -07:00
8120a33bda
Fix `>`-direction when initializing TraceEvent
2019-04-05 13:12:19 -07:00
32a3aea8bb
Make strings random in tracing micro-benchmark
2019-04-05 13:12:19 -07:00
bb82f8560a
process all volatile ints correctly in traces
2019-04-05 13:12:19 -07:00
9938a49c1e
make primitive test longer
2019-04-05 13:12:19 -07:00
8af0f15a43
primitive performance test for TraceEvent
2019-04-05 13:12:19 -07:00
ae84f5424a
Fix potential segfault when init changes enabled
2019-04-05 13:12:19 -07:00
c008e16c81
Defer formatting in traces to make them cheaper
...
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:
- TraceEvent::detail now takes a c-string instead of std::string for
literals. This prevents unnecessary allocations if the trace is not
going to be printed in the first place (for example for SevDebug).
Before that `detail` expected a `std::string` as key, which mean that
any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
specialized for any type that needs to be printed. The actual
formatting will be deferred to after the `enabled` check. This
provides two benefits: (1) if a TraceEvent is disabled, we don't pay
for the formatting and (2) TraceEvent can trace types that it doesn't
know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
calls, a call to detail will only introduce a if-branch which is much
cheaper than a function call.
2019-04-05 13:12:19 -07:00
390ab9cfed
A process will mark itself as degraded if it continually disconnects from a different process which the failure monitor thinks is healthy
2019-04-04 14:11:12 -07:00
2b75c2e684
Restore removed functions.
...
crc32c.cpp is 3rd party code.
orYield() in genericactors.actor.h might be used in the future code.
2019-04-04 13:24:55 -07:00
39c595223b
Merge branch 'release-6.1'
2019-04-02 22:30:02 -07:00
985e1de467
Update documentation and protocol version for release 6.1.0
2019-04-02 18:39:24 -07:00
47b4b82628
Merge branch 'master' into fix-unreferenced
2019-04-01 14:07:19 -07:00
3f76be8f45
Merge remote-tracking branch 'apple/master' into fix-unreferenced
2019-04-01 14:00:43 -07:00
f7f8ddd894
Fix warnings on unused variables
...
Found by -Wunused-variable flag.
2019-04-01 14:00:20 -07:00
a46620fbee
Merge branch 'release-6.1'
2019-03-30 17:59:28 -07:00
d670b74d69
prevent trace event spam by combining huge arena samples
2019-03-30 13:36:13 -07:00
d882c060bf
Merge commit '5dd6396eed0de0dfea6cf9eecc307995eff5cedc'
2019-03-28 18:00:55 -07:00
aa368c08a2
changed NetworkAddress hash function to use more bytes from the IP address
2019-03-28 17:47:13 -07:00
80ecb12190
change the IPv6 hash function to be more efficient
2019-03-28 14:07:46 -07:00
b6008558d3
renamed BinaryWriter.toStringRef() to .toValue(), because the function now returns a Standalone<StringRef>()
...
eliminated an unnecessary copy from the proxy commit path
eliminated an unnecessary copy from buffered peek cursor
2019-03-28 11:52:50 -07:00
836bb95a7a
Merge pull request #1372 from etschannen/master
...
Merge 6.1 into master
2019-03-27 21:00:49 -07:00
34b9d5e722
Merge pull request #1364 from etschannen/feature-fast-serialize
...
A few performance optimizations
2019-03-27 20:57:25 -07:00
6997075917
changed back to isV6addr instead of isV4addr for compatibility
2019-03-27 19:55:36 -07:00
2fbecac10f
fix: allocation strategy did not take into account the data used by the ArenaBlock
2019-03-27 19:54:58 -07:00
e5a80f2c94
optimized IPaddress
2019-03-27 18:21:13 -07:00
a55f06e082
Remove unused functions
...
Found with -Wunused-function flag.
2019-03-27 15:45:28 -07:00
d5c8b6b083
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
# flow/flow.h
2019-03-27 13:37:15 -07:00
71e2fdafb8
Changes to ratekeeper camel case
2019-03-27 08:24:25 -07:00
9e8237955f
BinaryWriter uses arena allocation
2019-03-26 17:27:33 -07:00
38c6681349
Fix some signed and unsigned mismatch warnings.
2019-03-26 14:54:11 -07:00
ea38b78768
BinaryWriter uses fastAllocator
2019-03-26 12:06:39 -07:00
dbcf1d717c
Added calls to TRACEALLOCATOR(96) and DETAILALLOCATORMEMUSAGE(96)
2019-03-25 13:45:07 -07:00
4a3467296f
Removed Arena96
2019-03-25 13:44:53 -07:00
007abbc45b
Added 96-byte FastAllocator
...
Since storage queue nodes account for a large portion of memory usage,
we can save space by only allocating 96 bytes instead of 128 bytes for
each node.
2019-03-25 13:44:39 -07:00
5e03e178de
Merge pull request #1345 from ajbeamon/support-multiple-client-or-worker-issues
...
Add support for a client or worker having multiple issues.
2019-03-24 17:27:50 -07:00
0ad351751a
fix build failure on Windows from undefined constant and mising parenthesis
2019-03-24 14:03:24 -04:00
24c92a1870
Merge pull request #1352 from etschannen/feature-network-address-list
...
Changed NetworkAddressList to at most two addresses for performance
2019-03-24 10:22:38 -07:00
1fc6937802
changed NetworkAddressList to at most two addresses for performance
2019-03-23 17:54:46 -07:00
09f37cf3d2
Merge pull request #533 from ajbeamon/fix-parent-directory
...
Fixes to parentDirectory() and abspath()
2019-03-22 23:53:46 -07:00
eacde47050
Disable setMemoryQuota for ASAN
2019-03-22 18:47:38 -07:00
382a7bdc5f
Changed behavior regarding ~ and ~user paths to treat them as unresolvable symbolic links.
2019-03-22 16:21:12 -07:00
524a666290
Added back previous handling of ~ in paths, with the improvement that it only treats ~ as special if it is the first character.
2019-03-22 11:44:46 -07:00
4eb5715689
Add support for a client or worker having multiple issues.
2019-03-22 08:29:41 -07:00
c6d96498ca
parentDirectory() is now based on popPath(). Bug fix, abspath() would prepend current working directory even when not resolving symlinks. Added more unit tests. Ported path operation unit tests to fdbmonitor() since the path manipulation function implementations are significantly different. Clarified some comments. The flow project version of abspath() does not allow resolveLinks to be false, for now, because behavior of this on Windows is not well thought out or tested.
2019-03-21 16:56:36 -07:00
bec6f0a37a
Added flags to absPath() and parentDirectory() for whether or not symbolic links should be resolved when possible and whether the final resolved path must actually exist on the filesystem. Updated tests, added more cases.
2019-03-20 22:52:47 -07:00
436c9319ff
Merge pull request #1333 from vishesh/scratch
...
fix: Use '_' instead of ':' in IPv6 tracefile names
2019-03-20 17:00:43 -06:00
c37291a366
fix: Use '_' instead of ':' in IPv6 tracefile names
...
':' is not acceptable on Windows. Reason to choose '_' instead of '.'
is to differentiate b/w IPv4 and IPv6 easily, and also '..1' in
filename looks weirder than '__1', which would happen with shortened
IPv6 addresses. And non-shortened IPv6 addreses are just lots of 0s.
2019-03-20 14:00:33 -07:00
f73aad863b
adding missing logging for 8192 fast allocators
2019-03-20 13:48:45 -07:00
70ac5ffda0
added random logging for huge arenas
2019-03-20 11:20:47 -07:00
c33428b705
do not fail atomicReplace if the files does not exist
2019-03-19 13:38:30 -07:00
10a30fed42
Merge pull request #1316 from senthil-ram/atomicreplace
...
atomicReplace to preserve owner & permission info
2019-03-19 08:21:35 -07:00
ddf8e86730
Merge pull request #1275 from alexmiller-apple/tstlog9
...
Spill-By-Reference TLog Part 5: TLogs That Can Spill Large Amounts Of Data
2019-03-18 17:58:21 -07:00
33a211c834
Merge pull request #1315 from etschannen/feature-memory-logging
...
Occasionally log a backtrace when the fast allocators allocates a new magazine
2019-03-18 16:05:20 -07:00
fe18596c01
Add a comment about ReplaceFile API
2019-03-18 15:24:08 -07:00
c8122edb6b
avoid relying on 128<<10 in knobs.cpp
2019-03-18 12:40:15 -07:00
7711542b55
Update flow/FastAlloc.cpp
...
Co-Authored-By: etschannen <36455792+etschannen@users.noreply.github.com>
2019-03-18 12:34:46 -07:00
c0094830c3
addressing minor format related comments
2019-03-18 11:59:51 -07:00
51a655d9c7
atomicReplace to preserve owner & permission info
...
atomicReplace currently does not preserve the
ownership and permission info, this fix will
preserves the ownership and permission info - it
fails the call if it is not able to do so.
2019-03-18 00:18:05 -07:00
fa5d929106
switch to g_nondeterministic_random to make simulation deterministic
2019-03-17 22:48:43 -07:00
d2eb7578fd
Occasionally log a backtrace when the fast allocators allocates a new magazine to help track down memory leaks
2019-03-17 21:59:56 -07:00
9a4fc25de5
ParentDirectory() in Platform,h no longer uses absPath() so it does not resolve symbolic links, making it suitable for use in fdbmonitor. CleanPath() will return "." instead of empty string to mean a relative path that refers to the current directory. Updated tests, and added new tests for ".".
2019-03-17 15:58:04 -07:00
f88e53e640
Merge branch 'master' of https://github.com/apple/foundationdb into fix-parent-directory
2019-03-16 00:13:09 -07:00
98a295a803
Changed parentDirectory() trace events back to SevError outside of simulation, did the same for cleanPath().
2019-03-16 00:07:38 -07:00
2efb6f4c0d
Added cleanPath() which puts a path in a canonical form without .., ., or duplicate separators without using the filesystem or resolving symbolic links. absPath() redefined to use cleanPath() so it will return the same result for a path without symbolic links regardless of whether or not the path actually exists. Redefined parentDirectory() to use absPath() and error on certain inputs. Added comments describing behavior of these functions, and added a unit test which verbosely tests many inputs to them.
2019-03-15 23:54:33 -07:00
471ac206a3
Add new boost preprocessor definitions to windows build in release mode, still just for flow project to see if it's sufficient.
2019-03-15 21:03:03 -07:00
2c8224f0b7
Add new boost preprocessor definitions to windows build.
2019-03-15 21:03:03 -07:00
7f8475cf27
Removed flow/boost.cpp and the copied boost files it was using, added boost macros to makefile and cmake stuff to hopefully enable building without them.
2019-03-15 21:03:03 -07:00
9f713a6b3d
Testing to see if builds work without the contents of the boost source copies.
2019-03-15 21:03:03 -07:00
5407c87179
Updated two files copied from boost 1.52.0 to their 1.67.0 equivalents since the default boost version to compile against is now 1.67.0 and compilation (on MacOS at least) fails against boost 1.52.0 because some definitions have moved since.
2019-03-15 21:03:03 -07:00
7f5bc2981f
Checksum DiskQueue pages on read, but at a lower priority.
...
If a server has its data spilled, then it's behind the 5s window.
Feeding it data is less important than committing, so we can hide the
extra CPU usage from checksumming the read amplified disk queue pages.
2019-03-15 21:01:19 -07:00
809861ee88
Merge branch 'master' into fix-parent-directory
...
# Conflicts:
# flow/Platform.cpp
2019-03-15 10:51:12 -07:00
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
5a10bf5dfc
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-14 10:35:12 -07:00
c32504f705
io: Add DISABLE_POSIX_KERNEL_AIO knob to use EIO instead of Kernel AIO
...
- Some Linux filesystems don't support O_DIRECT which is required by
Kernel AIO to function properly. Instead of using O_SYNC, EIO is
much better options in terms of performance penalty.
- Some systems may not support AIO at all. Eg. Windows Subsystem for
Linux.
FIXES #842
RELATED #274
2019-03-13 13:39:45 -07:00
e068c478b5
merge master
2019-03-12 18:31:25 -07:00
a7e45cff91
Merge pull request #1176 from jzhou77/ratekeeper
...
Make Ratekeeper a separate role
2019-03-12 15:58:59 -07:00
85c24b0067
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-12 15:20:54 -07:00
5392742902
fixed review comments
2019-03-12 14:38:54 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
bbd5cc2e95
Remove duplicate include due to concurrent update to same file
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-03-11 20:42:35 -07:00
bc4820865e
Fix some Windows issues
...
1. We need to set a Windows target to get rid of warnings when compiling
files that include windows.h
2. By default Boost will try to automatically link against boost_system.
In order to prevent this, we define BOOST_ALL_NO_LIB
3. Somehow, I had to include winioctl.h in Platform.cpp. According to
the documentation from MS, this shouldn't be necessary as windows.h
includes this as well. However, for me it didn't compile otherwise.
2019-03-11 20:42:35 -07:00
880e8643d1
Fix Windows link errors
2019-03-11 17:49:03 -07:00
435e515985
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-11 11:17:40 -07:00
ccab3834e2
merge in master
2019-03-10 23:12:37 -07:00
2ff37f49da
fix: compiler error
2019-03-10 22:56:12 -07:00
c6e94293bf
reset a process to not be degraded after 2 days
2019-03-10 22:39:21 -07:00
c304397166
fixed a memory leak when allocating memory > 4K and < 8K
2019-03-10 21:13:00 -07:00
892b649e64
fixed a memory leak when allocating memory > 4K and < 8K
2019-03-10 21:09:47 -07:00
59f598ab51
Try adding a header?
2019-03-08 21:46:32 -08:00
f593562743
use 8KB pages in Arena
2019-03-08 15:03:17 -08:00
426da90493
Add 8192 bit fast allocator
...
StorageServer is getting close to 4KB on older gccs. However,
on some systems (for example Fedora) the storageServer actor
is already larger than 4KB. This results in a linker error.
2019-03-08 15:03:17 -08:00
4891d31cc0
Remove a broken ASSERT.
...
It's now totally valid to have:
permits=3
take(1)
take(2)
take(72)
and the take(72) will only be granted once the first two finish.
2019-03-07 18:31:18 -08:00
3c86643822
Separate Ratekeeper from data distribution.
...
Add a new role for ratekeeper.
Remove StorageServerChanges from data distribution.
Ratekeeper monitors storage servers, which borrows the idea from
DataDistribution.
2019-03-07 13:16:20 -08:00
04880e3d4d
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-06 13:41:16 -08:00
fd34626009
boost: Remove log from Makefile and version check in flow.h
2019-03-05 22:06:12 -08:00
c6a65389ae
Remove noexcept macro and replace with BOOST_NOEXCEPT.
...
BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a
way that is correctly maintained over time.
2019-03-05 22:06:12 -08:00
367bcff700
Error if boost != 1.67.0
2019-03-05 22:06:12 -08:00
af617d68e6
boost 1.52.0 -> 1.67.0 in all vcxproj files
2019-03-05 22:06:12 -08:00
87d0c5bae0
Bug/usability fix: The output URLs of fdbbackup list were meant to be directly usable for backup management operations but they were missing the bucket URL parameter.
2019-03-05 21:14:21 -08:00
82d957e0bb
Merge pull request #1178 from vishesh/task/issue-963-IPv6
...
IPv6 Support
2019-03-05 17:14:16 -08:00
96ee95b9ad
fix: macOS build #963
...
Use the boost representation of IPv6 address internally
and make sure it uses std::array.
2019-03-05 14:03:14 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
e93cd0ff21
Add some checks and comments to IPv6 changes #963
2019-03-04 14:12:45 -08:00
592e224155
net: add/use formatIpPort to format IP:PORT pairs #963
2019-03-04 14:12:45 -08:00
82b2da4b78
net: Add IPAddress::parse() util #963
2019-03-04 14:12:45 -08:00
25daabdc02
net: TraceEvent and toIPVectorString for new IPAddress structure #963
2019-03-04 14:12:45 -08:00
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
1a550712cb
Add serialization support for std::array
2019-03-04 13:33:10 -08:00
75b2546e35
Allow FlowLock::take to succeed if there's no other active requests.
2019-03-04 01:42:38 -08:00
98bb58628f
Upgrade FlowLock from int to int64_t.
2019-03-04 01:42:38 -08:00
a462359af1
Implement g_random->randomSkewedUInt32()
...
This generates an exponential distribution, implemented as being
a "log-uniform" distribution.
2019-03-04 01:42:38 -08:00
39f612d132
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-03-02 17:07:00 -08:00
94385447bc
Status: Get if client configured TLS
...
To understand if all clients have configured TLS,
we check the tlsoption when a client tries to open database.
This is similar to how we track the versions of multi-version clients.
2019-03-01 15:17:01 -08:00
655c9d82c7
Various cleanup from review
2019-03-01 14:06:47 -08:00
13f7f5a073
Bump protocol version
2019-03-01 13:01:10 -08:00
e2bcecb08f
Merge branch 'master' into ratekeeper-batch-priority-limits
2019-02-28 12:52:44 -08:00
eb629d87a5
Add information about batch ratekeeper to status. Make it possible to track latencies in the ReadWrite workload for concurrently run instances separately.
2019-02-28 09:53:16 -08:00
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
82e429f417
Upgrade BinaryReader to support calls ArenaReader has.
2019-02-26 18:00:55 -08:00
abfe057805
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-25 13:47:16 -08:00
78aad255b0
updateProcessStats gets process stats directly
...
Storage servers no longer parse ProcessMetrics trace lines to get their
own cpuUsage and diskUsage statistics.
2019-02-25 13:45:53 -08:00
b8910ba7cd
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.h
# fdbserver/DataDistribution.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-22 14:38:13 -08:00
d008de576e
Merge pull request #1139 from xumengpanda/mengxu/machine-team-upgrade-PR
...
Add background actor to remove redundant teams
2019-02-22 14:22:07 -08:00
6660ccd936
Large rewrite/refactor from the page format outward. Replaced the in-page format, PrefixTree, with more general and templated implementation of the same concept, DeltaTree, which will be incrementally modifiable soon. New implementation for in-page cursors to use a per-page arena and share decoded binary tree nodes between cursors in the same page. New implementation for internal record B+Tree cursor which uses reference counted path hops which makes cloning a cursor very cheap. New implementation of external facing B+Tree cursor which makes efficient use of two internal cursors to find user-visible KV pairs and return them without copying keys and values except in the case of a large sharded value.
2019-02-21 02:46:30 -08:00
171d119d95
another attempt to fix compiler errors
2019-02-20 18:35:08 -08:00
fa96b8dd33
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-20 16:56:16 -08:00
4be90c91d6
Moved THIS and THIS_ADDR to actorcompiler.h
2019-02-19 15:16:59 -08:00
3bd9b9047b
Minor fixes - flow now compiling with intellisense
2019-02-19 15:16:59 -08:00
2c16475906
Added a general copy-constructor for intellisense
...
If a actor returns something of a different type that is
assignable, it will work because the actor compiler will
compile this to an assignment. However, if you compile
without the actor compiler, this will fail.
2019-02-19 15:16:59 -08:00
eb8285f14f
Add mock description for actors for intellisense
2019-02-19 15:16:59 -08:00
9f8c060343
Put assertions in its own scope
...
Some if-else statements in the code don't use
curly-braces, which is problematic if the only
statement used is an assertion. This should fix
that.
2019-02-19 15:16:59 -08:00
801e4c4700
Make when-macros work by accepting varargs
2019-02-19 15:16:59 -08:00
e05b53d755
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-15 20:37:07 -08:00
6d09ac483c
Merge with master
2019-02-15 17:03:40 -08:00
83060c6e56
Merge pull request #1062 from jzhou77/PR
...
Add a new DataDistributor role.
2019-02-15 13:51:27 -08:00
8099a0b665
Merge pull request #1129 from alexmiller-apple/tstlog-1
...
Spill-by-reference TLog Part 1: IDiskQueue::read()
2019-02-15 11:48:14 -08:00
75f692b931
simplify actorcompiler and target to compile coveragetool
2019-02-15 00:01:42 -08:00
5e6577cc82
Final cleanup per review comments
...
Make distributor interface optional in ServerDBInfo and many other small
changes.
2019-02-14 16:37:17 -08:00
578473a974
Various review comments fixes
2019-02-14 16:37:16 -08:00
907446d0ce
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-14 11:37:38 -08:00
b435d51061
Merge branch 'master' into track-server-request-latencies
2019-02-14 08:07:32 -08:00
5481851e82
TeamCollection: Add knobs for team remover
...
Added three knobs to control team remover
bool TR_FLAG_DISABLE_TEAM_REMOVER:
Disable the teamRemover actor
double TR_REMOVE_MACHINE_TEAM_DELAY:
Wait for the specified time before try to remove next machine team
double TR_WAIT_FOR_ALL_MACHINES_HEALTHY_DELAY:
Wait before checking if all machines are healthy
2019-02-13 15:11:56 -08:00
9272a41e5f
Merge pull request #1146 from atn34/fix-actor-warning
...
Fix actor warning for cmake build
2019-02-13 11:01:37 -08:00
1ea58c1e5e
Actually change name of flag
2019-02-13 10:44:58 -08:00
3a38bff8ee
Use DISABLE_ACTOR_WITHOUT_WAIT_WARNING consistently
2019-02-13 10:30:35 -08:00
2570b37e6e
Add function to read pages from RawDiskQueue_TwoFiles
2019-02-12 23:43:10 -08:00
601b229c05
Add a no-op breakpoint_me for easy gdb'ing.
2019-02-12 23:43:10 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
f257cca1bc
Update `Void _ = wait(...)` occurrences
2019-02-12 17:30:30 -08:00
df3454114e
Add TODO
2019-02-12 11:03:08 -08:00
8b0e593f83
Add --disable-actor-without-wait-error flag to actorcompiler
2019-02-12 11:01:17 -08:00
d348846a10
Introduce ErrorMessagePolicy class
...
This encapsulates logic for how to report particular error messages.
So far only used for reporting an actor not containing a wait statement,
as this warning should be suppressed in certain contexts.
We could change other error messages to use this class, but since they
don't have any logic I don't think the extra indirection is appropriate.
2019-02-12 11:01:17 -08:00
6051c8027b
Allow osx tooling to understand c sharp
...
After this change I am able to load foundationdb.sln with vscode and get
features like jump to definition and autocomplete.
IntermediateOutputPath will now use a default value.
2019-02-12 11:00:24 -08:00
b717750c4f
Minor README changes.
2019-02-11 21:40:14 -08:00
685242fbfc
Merge pull request #1092 from atn34/trace-format-network-option
...
Add trace_format network option
2019-02-08 10:52:50 -08:00
06e216f26e
fixed linker error
2019-02-07 15:37:04 -08:00
e45295a1f5
Added support for TLS
2019-02-07 15:37:04 -08:00
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
7a858b902d
Most of FDB compiling on Windows with cmake
2019-02-07 15:37:04 -08:00
d4349293b9
Reworked the way latency counters are tracked. Report the latency bands in separate events from StorageMetrics and ProxyMetrics. Fix a problem when the latency band configuration was changed. Add correctness testing.
2019-02-07 13:39:22 -08:00
0750dc0418
Change store from (Future, T&) to (T&, Future).
...
LHS = RHS, and the name of what's being modified is easier to find.
2019-02-04 18:04:22 -08:00
fa6f43b1f3
Added missing 'static' for actor fwd declarations
...
There is a bug in the code that when an ACTOR forward
declaration is static, the static keyword doesn't get
generated in the forward declaration. This commit fixes
that.
2019-02-01 14:44:47 -08:00
e9ddd94e27
The failure monitor is given a list of all IP addresses associated with a process
...
The connect packet includes the correct remote address
Did a lot of code cleanup
Simulation test mixed TLS and non-TLS listeners on the same process
2019-01-31 18:20:14 -08:00
9087f2948e
Support serialization of std::map objects
2019-01-31 11:24:04 -08:00
0e5db1e7b7
Merge pull request #1096 from mpilman/features/actor-forward-declarations
...
Allow forward declarations to be actors
2019-01-30 13:30:49 -08:00
e055fdab03
Better diagnostics for unrecognized trace format
2019-01-29 13:08:47 -08:00
01730cc461
Allow forward declarations to be actors
...
If an actor is forward declared it is kind of
awkward as the signatures don't fit. For example:
Future<Void> foo(int const& bar);
is the correct forward declaration for this actor:
ACTOR Future<Void> foo(int bar);
This also means that a lot of tooling (like rtags/cquery etc)
doesn't work correctly. It will show errors that the function
calls are ambiguous and "find references"-features typically
don't work.
Allowing to do this in the actor compiler is the first step. The
next will be to refactor all forward declarations.
2019-01-29 10:57:54 -08:00
dc2b740415
Added server_overloaded error and client message
2019-01-25 13:15:19 -08:00
1d7fec3074
Merge commit '048bfc5c368063d9e009513078dab88be0cbd5b0' into task/tls-upgrade-2
...
# Conflicts:
# .gitignore
2019-01-24 17:43:06 -08:00
ec32d3308b
Merge pull request #1086 from mpilman/features/c++-compiler-errors
...
Fixed several minor code issues
2019-01-24 15:24:33 -08:00
79637f07ac
Fixed several minor code issues
...
These will become a problem as soon as we
switch to C++17
2019-01-24 14:43:12 -08:00
4925b61d0e
Bump protocol version.
2019-01-24 13:49:22 -08:00
7fde75c439
Avoid possibly concurrent reassignment of dl_iterate_phdr.
2019-01-24 13:28:45 -08:00
efe06d71ce
Change profilingEnabled to thread_local to eliminate a race.
2019-01-24 13:27:16 -08:00
2198d24ce1
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/ServerDBInfo.h
# fdbserver/Status.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/storageserver.actor.cpp
2019-01-24 11:43:26 -08:00
8e05e95045
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-18 16:18:34 -08:00
7498c2308c
Merge branch 'release-6.0' into track-server-request-latencies
2019-01-16 13:39:01 -08:00
8b3cfa469c
Fix spelling
2019-01-15 19:30:09 +00:00
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
6276d6b4f4
Merge pull request #1026 from etschannen/master
...
Merge 6.0 into master
2019-01-09 16:23:35 -08:00
684a22a52b
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BackupContainer.actor.cpp
# fdbclient/HTTP.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/BackupCorrectness.actor.cpp
# versions.target
2019-01-09 16:14:46 -08:00
5048795554
Add JsonTraceLogFormatter to flow/CMakeLists.txt
2019-01-09 15:58:18 -08:00
7c5b2ab330
Merge pull request #976 from alexmiller-apple/jsonlogs
...
Allow trace logs to be output as JSON instead of XML
2019-01-09 17:04:50 -05:00
5d2b11cba9
Merge pull request #1019 from satherton/http-request-id
...
Backup usability and sanity check improvements
2019-01-09 13:29:37 -08:00
51b89ae083
WIP
2019-01-09 07:41:02 -08:00
d265517156
Fix: fast allocator would not cleanup memory for a thread if that thread never called getMagazine. This could happen if the first thing the thread did was to release memory.
...
Added a new metric for the number of threads that hold memory for each size and improve some existing metrics.
Fix: a failed ASSERT would crash if done early in the program lifetime.
2019-01-08 14:36:01 -08:00
0ec216a1fa
Added X-Request-ID header to HTTP requests and verification of matching ID in response, if present.
2019-01-07 17:56:38 -08:00
75fd8e6e17
Include <sys/sysmacros.h>
2019-01-03 12:44:29 -08:00
dbe9baff1f
Several small compilation fixes for new versions of gcc
...
There are several missing includes for cmath in the code, I added those.
Next, Coro returns a reference to a stack variable and this causes a
warning. As this is probably ok for Coro, I disabled the warning in
that file for GCC. I want to have this warning in the build system as
it is generally a very useful warning to have.
Another change is that major and minor are deprecated for a while now.
I replaced those with gnu_dev_major and gnu_dev_minor.
ErrorOr currently implements operators ==, !=, and <. These do not
compile because Error does not implement ==. This compiles on older
versions of gcc and clang because ErrorOr<T>::operator== is not used
anywhere. It is still wrong though and newer gcc versions complain.
I simply removed these methods.
The most interesting fix is that TraceEvent::~TraceEvent is currently
throwing exceptions. This is illegal behavior in C++11 and a idea in
older versions of C++. For now I simply removed the throw, but this
might need some more thought.
2019-01-03 12:44:19 -08:00
3f399dd9be
Indent within anonymous namespace
2019-01-03 09:21:09 -08:00
aa2a76ef4c
Merge pull request #981 from alexmiller-apple/cmake
...
Add a CMake build system
2019-01-02 18:50:15 -08:00
80dd1efaf6
Assign to g_traceLog.formatter
2019-01-02 16:42:00 -08:00
6a4d87802b
Replace & operator with variadic function
2018-12-28 11:33:42 -08:00
f64d5321e9
Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL.
2018-12-20 18:05:23 -08:00
b8df5acc15
Add --trace_format flag to fdbserver
2018-12-20 15:02:01 -08:00
33a4aac249
Fix build
2018-12-20 11:55:59 -08:00
4b31d209da
Emit valid json for unprintable characters
2018-12-20 11:43:18 -08:00
cdc7e83993
Update flow/Trace.cpp
...
Co-Authored-By: alexmiller-apple <35046903+alexmiller-apple@users.noreply.github.com>
2018-12-14 11:30:40 -08:00
a5401b834b
Integrate JsonTraceLogFormatter into build system + more.
...
Where more is:
* Changed the default format to xml, so that all traces don't suddenly
change as of this commit. Someone is going to need to start passing a
flag to all their servers, but let's worry about that later.
* Changed the copyright headers to be the correct filename.
* Include paths should be specified from the root.
2018-12-13 22:07:16 -08:00
2389cf65f3
fixed escaping bug
2018-12-13 21:50:01 -08:00
aa1eff8346
formatted json log formatter
2018-12-13 21:49:46 -08:00
40890e9dbe
Implemented a json log formatter
2018-12-13 21:46:02 -08:00
a982b9da72
Additional changes from a merge commit.
2018-12-13 17:13:41 -08:00
c654b1bb61
First version of working rpm packages
2018-12-13 15:55:31 -08:00
550daa05f8
Default configuration now builds.
2018-12-13 15:52:27 -08:00
df0f491c29
Some more improvements to the build and preparations for packaging
2018-12-13 15:04:13 -08:00
dce290909d
fdbserver now compiling
2018-12-13 14:13:47 -08:00
51beb8b48c
fdbrpc compiling with cmake
2018-12-13 14:02:16 -08:00
f5fcb666c5
flow compiling with cmake on OS X
2018-12-13 13:58:06 -08:00
43e5a46f9b
Change Endpoint::address(NetworkAddress) to vector<NetworkAddress>
...
Extend `Endpoint` class to take multiple NetworkAddresses instead of
just one. Hence, to talk to an endpoint instead of one IP:PORT, we'll
have multiple IP:PORT pairs.
This patch simply adds the field and makes changes to compile the
codebase. The first element of of `address` field is used everywhere.
Hence the way we talk to remains same with this patch.
NOTE:
Directly accessing the first memeber of Endpoint::address is unsafe
as Endpoint() doesn't enforces non-empty address list. However, since
the correctness test pass for now and are anyway replacing all those
unsafe accesses with ones considering the whole vector, this patch
ignores to access them in safe way.
2018-12-13 13:36:52 -08:00
42dffd4dff
Take a vector of network addresses from CLI to start FDB server
...
Extends the CLI interface to take multiple public and listen addresses.
We however do not do anything with those extra addresses and just
consider the first one for now.
2018-12-13 13:36:52 -08:00
e8e01b2406
Remove unused localAddress parameter from newNet2 and Net2 classes
2018-12-13 13:36:52 -08:00
Nikolas Ioannou