325a8e4213
remove confusing USE_ODIRECT knob
2019-10-24 11:44:03 -07:00
f23392ec5a
Don't use O_DIRECT in EIO by default
2019-10-24 11:39:55 -07:00
7ad0e20e48
Added knob to disable O_DIRECT
2019-10-24 11:20:14 -07:00
f41f19b5f6
Introduced knob to set eio parallelism
2019-10-24 11:20:14 -07:00
85977fb8d5
Use O_DIRECT with EIO
2019-10-24 11:20:14 -07:00
562ce17eca
Initialize outgoingConnectionIdle in the constructor. Add back line to connectionKeeper that is needed in some looping cases
2019-10-10 12:48:35 -07:00
ad8604f24a
Fix spurious ConnectionClosed event when starting a connection.
2019-10-10 10:34:44 -07:00
a7237c4302
Merge pull request #2045 from atn34/disallow-scalar-network-messages
...
Disallow scalar network messages
2019-08-30 13:38:54 -07:00
cf56b005e8
Add comment for pinging incompatible clients
...
If client is incompatible, connectionMonitor relies on peer->resetPing
to be triggered whenever data is received to prevent ping timeout.
The server stopped sending pings since 6.2 which meant resetPing
doesn't get triggered.
2019-08-30 11:17:22 -07:00
1fdabe62c2
Merge pull request #2048 from etschannen/feature-fix-connections
...
Fixed two different ways useful connections were being closed
2019-08-30 11:05:02 -07:00
8fc28dd730
fix: continue pinging incompatible clients from the servers so that the the client knows the server process is active
2019-08-29 16:51:03 -07:00
1c0484cffc
fix: do not close connections which have outstanding tryGetReplies with the peer
2019-08-29 16:49:57 -07:00
6aa0ada7b1
Replace scalar root types with proper messages
2019-08-28 14:40:50 -07:00
41b908752e
increased move keys parallelism to be less of a decrease just in case lowering this could effect normal data distribution
...
raised target durability lag versions to give more time for batch limiting to come into play before this limit is hit
changed max_bad_options to better reflect the name
2019-08-21 14:55:21 -07:00
51cedd24c8
load balance will send reads to remote servers if more than one alternative is failed or overloaded
2019-08-19 13:59:49 -07:00
da8163fd5a
allow one requests every second to skip there all_alteratives_failed delay, because if a client has a timeout longer than the delay we will never invalidate the key servers cache
2019-08-09 13:03:40 -07:00
5f7d3498ea
The delay for all_alteratives_failed can scale all the way up to 30.0 at a much slow time ratio
2019-08-09 12:35:19 -07:00
0eb0e7a44a
made Peer reference counted to avoid other potential bugs involving accessing Peer after it has been destroyed
2019-08-09 11:52:12 -07:00
84fd1003a5
do not close idle network connections with incompatible servers
2019-08-08 23:47:00 -07:00
98b643b7ae
fix: connectionReader could access self in its destructor after it has already the object has already been deleted if an error can be thrown from connectionMonitor while still on the stack from scanPackets
2019-08-08 23:35:44 -07:00
370ba8b841
Remove --object-serializer flag from executables
2019-08-06 09:25:40 -07:00
863204a29d
Update names in CMakeLists, vcxproj
2019-08-01 08:48:25 -07:00
e0736232d4
Rename file in comment header
2019-08-01 08:40:45 -07:00
e61cac4ed4
Fix spacing issue; rename fdbrpc/Stats.h to fdbrpc/TimedRequest.h
2019-08-01 08:39:52 -07:00
0e474ed47e
Update fdbrpc/Stats.h
...
Co-Authored-By: Evan Tschannen <36455792+etschannen@users.noreply.github.com>
2019-07-31 19:56:21 -07:00
7d247af500
Two minor bug fixes from recent optimizations
2019-07-31 19:14:11 -07:00
0569df00f6
Remove indirection in LoadBalancedReply serialization
2019-07-31 17:59:35 -07:00
daf1e09af4
Explicitly check for clang and g++
2019-07-30 20:08:56 -07:00
638d2d05f4
Adds attribute to non-windows compilers
2019-07-30 15:49:25 -07:00
83922f1f37
Fix clang compiling error without sse4.2
2019-07-30 15:02:24 -07:00
90e3b50213
Merge branch 'master' into feature-coordinator-connection
...
# Conflicts:
# fdbclient/DatabaseContext.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/workloads/KillRegion.actor.cpp
2019-07-26 15:05:02 -07:00
4a866290b7
Clients keep a persistent connection open with coordinators to get updates to the list of proxies
...
Status still needs to be updated with client information with information from the coordinators
2019-07-23 19:22:44 -07:00
c70e762f0e
Merge pull request #1785 from xumengpanda/mengxu/server-team-remover-PR
...
Remove redundant server teams
2019-07-19 17:44:16 -07:00
1ac2d01b03
Merge remote-tracking branch 'upstream/master' into flatbuffers-fixes2
2019-07-18 09:50:08 -07:00
5d3e69b6fc
Merge pull request #1820 from fzhjon/load-balance-locality
...
Introduced a knob that can turn locality on/off
2019-07-16 16:40:43 -07:00
d5caf0c1b4
Merge branch 'flatbuffers-fixes2' of github.com:mpilman/foundationdb into flatbuffers-fixes2
2019-07-16 14:47:40 -07:00
7d37040725
split the single knob into two for finer-grained control
2019-07-16 12:46:02 -07:00
20f067e794
Merge with master:Resolve conflict with PR#1797
2019-07-16 10:52:28 -07:00
6c6a1ca8f4
Expose serialization context too all traits
2019-07-15 12:58:31 -07:00
1c0daa7f2c
Resolve review comments:Remove unneeded code
2019-07-12 18:10:04 -07:00
75d4b612cf
Make object serializer versioned
2019-07-12 11:53:14 -07:00
cf935ff9e6
Remove debug message and format code
2019-07-11 22:05:20 -07:00
c96854fce2
Simplify IReplicationPolicy serialization
2019-07-11 17:35:37 -07:00
be3a07826d
fixed serialization bug with ReplicationPolicy
2019-07-11 17:35:37 -07:00
ae6f17625e
Support PacketBuffer's of arbitrary size
2019-07-11 17:35:37 -07:00
59ddf091e8
Re-use writeToOffsets vector
2019-07-11 17:35:37 -07:00
b9af1f43b7
Implement new dynamic_size_traits
2019-07-11 17:35:37 -07:00
1700d492cf
Merge pull request #1823 from ajbeamon/cache-hit-rate-in-status
...
Tweak cache hit calculations and add cache hit rate to status
2019-07-11 14:06:06 -07:00
c6e42d6119
ReplicationPolicy:Add trace for the name of each keyIndex
2019-07-10 19:29:29 -07:00
b4dbc6d7fa
Change the way cache hits and misses are tracked to avoid counting blind page writes as misses and count the results of partial page writes. Report cache hit rate in status.
2019-07-10 14:43:20 -07:00
c694931e33
sim2: Remove obsolete comment
2019-07-10 14:06:06 -07:00
19a91765f6
introduced a knob that can turn locality on/off
2019-07-09 16:16:15 -07:00
4b8eb27134
fdbrpc: Move setStatus line in addPeerReference
2019-07-09 15:01:12 -07:00
983343978e
fdbrpc: ConnectionMonitor should close unreferenced after delay
...
Potentially for cases, where it goes up to 1 immediately.
2019-07-09 14:24:16 -07:00
22678267cd
fdbrpc: Don't drop idle connections from server
...
Instead try pinging the client and let that decide whether the client
is alive or not. Ideally, it should always be failed since a well
behaved client would have closed the connection.
2019-07-09 14:24:16 -07:00
1f9c80f633
fdbrpc: Instead of tracking last sent data, track last sent non-ping data
...
* This will allow client to continue monitoring peer connections while
connection stays open, so that there is no period of "uncertainity"
without previous no-monitoring approach.
* Use multiplier for incoming connection idle timeout
* Update idle connection timeout values and leaked connection timeout in
simulator.
2019-07-09 14:24:16 -07:00
867986cdea
fdbrpc: Reduced connection monitoring from clients
...
This patch does two changes to connection monitoring:
1. Connection monitoring at client side will check if the connection
has been stayed idle for some time. If connection is unused for a
while, we close the connection. There is some weirdness involved here
as ping messages are by themselves are connection traffic. We get over
this by making it two-phase process, first being checking idle
reliable traffic, followed by disabling pings and then checking for
idle unreliable traffic.
2. Connection monitoring of clients from server will no longer send
pings to clients. Instead, it keep monitor the received bytes and
close after certain period of inactivity.
2019-07-09 14:24:16 -07:00
7647d3e3c0
fdbrpc: Don't use RequestStream for pings in ConnectionMonitor
...
RequestStream add another count to peerReference, which means as long
as ConnectionMonitor is alive, we'll never get peerReference=0 keeping
unnecessary connections potentially alive.
2019-07-09 14:24:16 -07:00
3f4f71ff9f
fdbrpc: Increment peerReferences correctly
...
The constructor of FlowReceiver which handled reference counting
peerReferences relied on calling a virtual method from constructor
whose behaviour isn't correct. This patch, bubbles down result of that
virtual method from derived constructor to base contructor.
2019-07-09 14:24:16 -07:00
15c6f2b864
Explain SFINAE for has_serialization_done
2019-07-05 14:07:02 -07:00
7350b3db30
Don't assume serializeReplicationPolicy succeeds
2019-07-05 14:07:02 -07:00
e2ed56fa56
Convert ownedPtr to unownedPtr for IReplicationPolicy
...
Remove WriteRawMemory feature
Remove deserialization_done
2019-07-05 14:07:02 -07:00
4c5ebd7609
Avoid assert when collecting vtables
2019-07-05 14:07:02 -07:00
888f4f92e0
Fix errors and TaskPriority more priorities.
2019-07-03 21:03:58 -07:00
8e1ab6e7db
Merge remote-tracking branch 'upstream/master' into flowlock-api
2019-06-28 17:32:54 -07:00
5ccffe3cb3
Merge pull request #1684 from jzhou77/large-packet
...
Better handling for large packets
2019-06-28 16:19:01 -07:00
bc4548e0d3
Fix sed accidentally rewriting a trace event to have an invalid field name.
2019-06-27 17:55:41 -07:00
bf883d7055
Merge remote-tracking branch 'upstream/master' into flowlock-api
2019-06-25 14:26:50 -07:00
0fe6edc254
Merge pull request #1678 from mpilman/features/external-workload
...
Features/external workload
2019-06-25 13:53:19 -07:00
24937d8125
Merge pull request #1744 from vishesh/task/monitor-leader-on-demand
...
Fix setting enClientFailureMonitor global for client
2019-06-25 13:38:59 -07:00
e6ff23c420
Allow next packet size to be read when receiving data
...
For large packet, allocate sizeof(uint32_t) more bytes for next packet size.
Also add knob MIN_PACKET_BUFFER_FREE_BYTES, which is used to trigger allocation
of a new arena when free bytes are lower than this threshold.
2019-06-25 10:18:56 -07:00
c4e44e6697
Refactor: add missing include
2019-06-25 10:18:56 -07:00
54ac91342d
Refactor: include ordering in FlowTransport.actor.cpp
2019-06-25 10:18:56 -07:00
9d07e84dc8
Update fdbrpc/FlowTransport.actor.cpp
...
Co-Authored-By: Evan Tschannen <36455792+etschannen@users.noreply.github.com>
2019-06-25 10:18:56 -07:00
e0c3df899b
Break connection read into multiple smaller reads
...
And add yield between consecutive reads.
2019-06-25 10:18:56 -07:00
559a4b6e26
Add const qualifier
2019-06-25 10:18:56 -07:00
2363326ecb
Add knobs for various packet buffer sizes
2019-06-25 10:18:56 -07:00
b151141965
Better handling for large packets
...
On the sending side, a large packet is split into smaller pieces. On the
receiving side, use packet length to allocate buffer to avoid multiple memcpy
and allocations.
2019-06-25 10:18:56 -07:00
7a500cd37f
A giant translation of TaskFooPriority -> TaskPriority::Foo
...
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
cbc2398254
fdbrpc: Remove default parameter from FlowTransport::createInstance
2019-06-25 01:17:38 -07:00
189541d42c
LoadBalancedReplies were sending uninitialized versions of their subclasses when there was an error.
2019-06-20 13:57:56 -07:00
12dbe13c9c
Provide a no-op O_CLOEXEC on windows to fix the build.
2019-06-19 17:16:06 -07:00
df0baa0066
Merge pull request #1720 from mpilman/features/protocol-version
...
Make protocol version a type
2019-06-19 13:46:35 -07:00
2eff2b7e21
First simple test is working (but very buggy)
2019-06-19 13:03:41 -07:00
68ce9a5e75
ProtocolVersion type - second try
2019-06-18 17:55:27 -07:00
4fa5dc0502
Merge remote-tracking branch 'upstream/master' into cloexec
2019-06-18 16:35:18 -07:00
8576665a90
Revert "Revert "Make protocol version a type""
...
This reverts commit 455bf3b3ec
2019-06-18 14:49:04 -07:00
455bf3b3ec
Revert "Make protocol version a type"
2019-06-18 10:59:17 -07:00
dc9522bd86
Address code-review comments
2019-06-16 09:59:15 -07:00
da53a92bec
Make protocol version a type
...
This fixes #1214
The basic idea is that ProtocolVersion is now its own type. This
alone is an improvement as it makes many things more typesafe. For
each version, we can now add breaking features (for example Fearless).
After that, there's no need to test against actual (confusing) version
numbers. Instead a developer can simply test
`protocolVersion->hasFearless()` and this will return true iff the
protocolVersion is newer than the newest version that didn't support
fearless.
2019-06-16 09:59:15 -07:00
6ececa94ce
Merge pull request #1640 from vishesh/task/client-failmon
...
Clients will no longer get failure monitoring info from cluster controller
2019-06-13 17:31:17 -07:00
55f7e7d372
fix: The delay inside the disabledMap was causing the storage server updateStorage actor to run on the client process
2019-06-13 14:28:30 -07:00
dccb9bc26d
fixed a number of correctness problems
2019-06-12 19:40:50 -07:00
42fafe8a42
Addressed review comments
2019-06-11 18:58:00 -07:00
9fdbf0c846
Merge pull request #1477 from tclinken/features/local-rk
...
Local Ratekeeper
2019-06-11 13:24:08 -07:00
4c62458172
Make valgrind work on Fedora 30
2019-06-11 10:27:08 -07:00
8144882d7b
Merge branch 'apple-master' into features/local-rk
2019-06-10 19:40:25 -07:00
a8e408e268
run clang-format on changes
2019-06-10 14:10:24 -07:00
4316ef9ec6
failMon: For clients remove expireFailure and report failures only during connect
2019-06-09 00:43:38 -07:00
6fa7081a21
net: Don't make FailureMonitoring requests from client
...
This patch removes the need for clients to continuously contact
cluster coordinator for failure monitoring information. Instead, it
uses the FlowTransport to monitor the statuses of peers and update
FailureMonitor accordingly.
2019-06-09 00:43:38 -07:00
6b4d30c3ae
failmon: Identify client vs server when starting failure monitoring client
2019-06-09 00:43:12 -07:00
29b96414e2
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/NativeAPI.actor.cpp
# fdbserver/Coordination.actor.cpp
# flow/Arena.h
# versions.target
2019-06-03 18:49:35 -07:00
773f52d0a1
Merge remote-tracking branch 'upstream/master' into cloexec
2019-06-03 15:43:32 -07:00
172a0a9009
added an extra trace event
2019-05-30 17:25:42 -07:00
d68a229772
makefile changes to accommodate boost/process.hpp
2019-05-28 22:07:46 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
f4fbaac6b0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-05-19 10:27:59 -07:00
2b8b7954a9
in simulation, prevent data from being received over a connection 1 second after the connection is closed on the other end
2019-05-17 15:05:32 -07:00
5a8c97480a
Merge pull request #1506 from nikolas-ioannou/feature-pagecache-lru
...
AsyncFileCached: switch from a random to an LRU cache eviction policy
2019-05-17 13:42:21 -07:00
3acaa7343e
Enabled C++17 for all Windows projects
...
Set Visual Studio version to 2017 (first version to support C++17)
2019-05-16 17:44:13 -07:00
a745a8094e
do not close a connection due to a missed ping if the process is still receiving data from the connection
2019-05-16 17:26:48 -07:00
94aed513c7
Switched Windows tools within projects to 2017
2019-05-16 15:05:11 -07:00
69fb852ee0
Add more CLOEXEC-like things.
...
From missed call sites found during/after code review.
2019-05-14 20:30:58 -10:00
1f02cd30e2
Mark all opened files as close-on-exec.
2019-05-13 16:05:49 -10:00
46e7a0ca56
address reviews and make compile with `-Wunused-variable`
2019-05-13 14:15:23 -07:00
0713e06efc
Started to work on Windows
2019-05-13 14:15:23 -07:00
20c3f7f264
remove mixed-mode support
2019-05-13 14:15:23 -07:00
42385c2f81
Fixed issues introduced during rebase
2019-05-13 14:15:23 -07:00
f5fa3a65b4
some more fixes
2019-05-13 14:15:23 -07:00
44db3450ec
Several flatbuffers bug fixes
2019-05-13 14:15:23 -07:00
69fa3d3903
fixed compilation issues after rebase
2019-05-13 14:15:23 -07:00
3bc5ee8c58
fix indentation
2019-05-13 14:15:22 -07:00
47fdb78782
use EnsureTable for enums of enums
2019-05-13 14:15:22 -07:00
642a96807b
Fixed compilation issues after rebase
2019-05-13 14:15:22 -07:00
6afce01744
Implementation complete (not yet working)
2019-05-13 14:15:22 -07:00
9eeb48c43d
Allow to turn on object serializer
...
This commit includes functionality to turn on
the object serializer for network communication.
This is done the following way:
- On incoming connections, a process will detect
whether the client supports the object serializer
and will only serialize responses with it, if it does
- On outgoing connections, the command line flag is used
to determine whether the object serializer should be used
to send data.
This way, a cluster can run in mixed mode. To upgrade one
can upgrade one process at a time and set the flag one process
at a time.
This is how this is tested on the simulator:
- The command line flag can take three options: on, off,
and random.
- For off, the object serializer will never we used.
- For on, the object serializer will be always used.
- For random, the simulator will flip a coin for each
process it starts up.
2019-05-13 14:15:22 -07:00
ba83c458a6
types implemented
2019-05-13 14:15:22 -07:00
fe81454ec2
basic functionality for object serializer
...
This commit includes:
- The flatbuffers implementation
- A draft on how it should be used for network messages
- A serializer that can be used independently
What is missing:
- All root objects will need a file identifier
- Many special classes can not be serialized yet as the
corresponding traits are not yet implemented
- Object serialization can not yet be turned on (this will
need a network option)
2019-05-13 14:15:22 -07:00
8c3516951a
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-05-12 20:13:49 -07:00
c502ed3d15
Fix a variety of problems stemming from a wait() being added to push().
...
And that this code was previously insufficiently tested.
2019-05-10 14:55:11 -10:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
510b0b2fcd
Fix DiskQueue not replaceFile'ing frequently enough for the final time.
2019-05-08 23:08:25 -10:00
c2827f4fa3
Add page cache hit, miss, and eviction stats to SystemMonitor
2019-05-08 15:41:17 +02:00
af00248df6
fdbrpc: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
41ae9cd0d8
Remove ; for namespace declarations
2019-05-06 13:21:28 -07:00
d6170210e7
AsyncFileCached: throw (new) exception, through helper static fn, if cache eviction polity not recognized.
2019-05-06 10:11:46 +02:00
fdb3992990
AsyncFileCached: switch to string for eviction policy knob
2019-05-03 14:04:43 +02:00
b2280e15bf
AsyncFileCached: support for lru cache eviction policy
...
- Added a knob to control page cache eviction policy (0=random default, 1=lru)
- Added page cache hit, miss, and eviction stats
2019-05-02 17:35:30 +02:00
e193cac5ef
Merge remote-tracking branch 'apple/master' into tlog
...
Resolve Conflicts: fdbserver/MasterProxyServer.actor.cpp
2019-05-01 17:18:00 -07:00
2d5043c665
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-30 18:27:04 -07:00
024d58263e
Merge pull request #1505 from alexmiller-apple/reorder-unset-fit
...
UnsetFit > OkayFit, so order the cases that way
2019-04-30 13:21:37 -07:00
2462d29235
AsyncFileCached: switch from a random to an LRU cache eviction policy
2019-04-30 11:56:34 +02:00
48ce964225
UnsetFit > OkayFit, so order the cases that way
...
Keeping them sorted top->down makes keeping track of the rules of what
will be recruited where much easier.
2019-04-29 19:51:43 -07:00
b4f833c27e
Added sampled logging for slow ftruncate/fallocate calls.
2019-04-23 13:40:23 -07:00
e0f76edf77
Merge pull request #1471 from AlvinMooreSr/release-6.1-merge
...
Merge Release 6.1 Into Master
2019-04-23 11:08:21 -07:00
d19b0cf1c1
Refactor LogSet with two new constructors
2019-04-21 10:41:07 -07:00
2ad6d4d994
Do not rethrow server_overloaded error from load balancer
2019-04-20 15:38:57 -07:00
ddd6e26e18
Fix unused-variable warning in linux-only code
2019-04-20 10:39:20 -07:00
2967ceb807
Fixed checkAndProcessResult
...
checkAndProcessResult must handle errors in LoadBalancedReply objects in the
same way other errors are handled.
2019-04-19 18:29:58 -07:00
2bea99591e
Merge branch 'release-6.1' of copy of master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-04-17 15:51:48 -07:00
mpilman