a36caed3bd
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-09 10:22:55 -07:00
36b1247054
Merge pull request #2136 from ajbeamon/dont-require-future-get-database
...
Don't require fdb_future_get_database from new client binaries.
2019-10-09 09:58:56 -07:00
d146f1d636
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-07 11:27:15 -07:00
1b946d588f
Merge pull request #2208 from alexmiller-apple/faster-txstag-recovery
...
Recover Txs Faster [0/?]: Combine spill-by-value and spill-by-reference into one file/SharedTLog
2019-10-07 11:15:56 -07:00
28f6275f94
Use AssumeVersion instead of Unversioned
...
Which lets us revert the unversioned serilaization of TLogSpillType
2019-10-03 15:59:09 -07:00
d96a7b2c69
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-03 09:47:45 -07:00
d38a96ab73
Make LogData aware of the spill type it was created to perform.
...
The spilling type is now pulled out of the request, and then stored on
LogData for later access, and persisted in the tlog metadata per tlog
generation.
It turns out that serializing types as Unversioned is a bit wonky.
2019-10-03 01:45:10 -07:00
24c46337e1
Advance TLogVersions for 7.0 while adding a V5 that is TLogServer
...
Advancing the MIN_RECRUITABLE and DEFAULT is just following the standard
progression for 7.0. It was convenient to do while adding the V5 so
that we can hook TLogServer back into being used.
2019-10-03 01:43:01 -07:00
628b4e0220
added a warning if multiple log ranges exist for the same range
2019-10-02 17:06:19 -07:00
d0147e5e5d
Merge branch 'release-6.2' into mengxu/merge-release620-to-master-v3
...
Resolved Conflicts:
documentation/sphinx/source/release-notes.rst
fdbserver/DataDistribution.actor.cpp
versions.target
2019-10-02 13:22:56 -07:00
9ec9f41d34
backup and DR would not share mutations if they were started on different versions of FDB
2019-10-01 18:52:07 -07:00
656eacc965
Increase the default client shard location cache size by a factor of 2.
2019-10-01 14:31:48 -07:00
1d63ba6980
Use immediate priority for coordinator changes
2019-10-01 08:36:37 -07:00
e5a6ebae7e
fixed compiler error
2019-09-30 12:46:30 -07:00
4d659662b8
made cleanup handle retries better
2019-09-30 12:44:20 -07:00
045175bd0e
added tracking for the size of the system keyspace
2019-09-27 22:39:19 -07:00
324d0bd3b0
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-cleanup-mutations
2019-09-27 19:15:14 -07:00
3cc5d484a5
the include and exclude commands do not need to set the moveKeysLockOwnerKey, which will kill the data distribution algorithm
2019-09-27 18:33:56 -07:00
ef01ad2ed8
optimized log range clearing to clear everything for each possible hash (256 clears) if that would be more efficient than one clear per second that has elapsed
...
aborting a DR without the —cleanup flag will still attempt to cleanup for 30 seconds before giving up
added a cleanup command to fdbbackup which can remove mutations from orphaned DRs which were stopped without the —cleanup flag
2019-09-27 18:32:27 -07:00
81ebc7bfc9
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-09-27 13:14:13 -07:00
d41852abbc
Merge pull request #1988 from negoyal/negoyal_dequeue
...
Replace std::map with std::deque for versionedMap
2019-09-26 23:25:58 -07:00
0d06bf8a66
Update fdbclient/VersionedMap.h
...
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:58:04 -07:00
2526c1a0d2
Update fdbclient/VersionedMap.h
...
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:57:54 -07:00
d049105a69
Update fdbclient/VersionedMap.h
...
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:57:44 -07:00
f168c3fda9
Update fdbclient/VersionedMap.h
...
Renamed variable to be more specific
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:57:32 -07:00
381f3220b3
Update fdbclient/VersionedMap.h
...
Indentation Fix
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:57:11 -07:00
efbce26c78
Update fdbclient/VersionedMap.h
...
Indentation fix.
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:56:43 -07:00
efa2f5df6b
avoid double overwrite on failed servers and reordered removal of keys
2019-09-26 13:52:53 -07:00
450a09e117
Code Review Changes
2019-09-24 15:48:50 -07:00
a51ab0003d
Don't require fdb_future_get_database from new client binaries. The previous code, which technically incorrect, is believed to be asymptomatic because new versions of the client do have this symbol in order to support old API header versions.
2019-09-24 13:11:30 -07:00
40ad6f0931
created new reply types to work with flatbuffer serialization requirements
2019-09-18 13:40:18 -07:00
471e283128
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-09-18 11:49:07 -07:00
a4d38f1158
Fix #2057 snapshot cli to print UID in failure too
2019-09-17 05:18:28 -07:00
b495cc697b
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-09-13 09:25:08 -07:00
d9fb199486
Remove debug messages from checkDatabaseLock()
2019-09-12 22:11:25 -07:00
9b4f7626bb
cache the serialization of clientDBInfo
2019-09-11 15:19:42 -07:00
6100d3274d
Merge pull request #2058 from tclinken/expose-lock-status
...
Added lockUID to status output if database is locked
2019-09-11 08:47:35 -07:00
41fd3d9467
Merge branch 'master' into remove-unused-ssi-get-version
...
# Conflicts:
# fdbclient/StorageServerInterface.h
# fdbserver/storageserver.actor.cpp
2019-09-10 08:11:43 -07:00
94668c6f1f
Merge pull request #2063 from jzhou77/clang
...
Refactor deserialization of on-wire buffer with TagsAndMessage
2019-09-09 16:34:56 -07:00
8fbd90e2f6
Merge pull request #1985 from xumengpanda/mengxu/storage-engine-switch-PR-v2
...
Graceful storage engine migration
2019-09-09 13:51:53 -07:00
8c31a839be
s/lockUID/lock_uid in status
2019-09-06 22:20:55 -07:00
2d216f7ae5
Added database_lock_state to statusSchema
2019-09-06 22:20:50 -07:00
2d5ebebb7b
Use TagsAndMessage for deserialization in TLogServer
2019-09-05 16:53:10 -07:00
34dedc9a62
Fix whitespace
2019-09-05 16:44:58 -07:00
c18c4c1b83
Use a transaction option to control includePort behavior
2019-09-05 14:58:39 -07:00
2723922f5f
Replace -1 as VERSION_HEADER constant for serialization
2019-09-05 12:45:39 -07:00
f9357c5ad8
Fix side effect of ArenaReader
...
ServerPeekCursor::nextMessage() should only consume the message header, because
the reader() directly inherits the current position. The previous commit
changes the positon to the begining of the next message, which breaks storage
server code.
2019-09-05 11:07:07 -07:00
11f6adf645
Treat \xff\xff prefix as 'includePort' for get_addresses_for_key
2019-09-04 17:47:40 -07:00
c2355f721e
Merge branch 'master' into mengxu/performant-restore-PR
2019-09-04 17:11:42 -07:00
d160810662
FastRestore:Resolve review comments
2019-09-04 16:48:43 -07:00
cd3f1e33d4
Refactor deserialization of TagsAndMessages
...
Consolidate deserialization of TagsAndMessages in the structure itself and
change both TLog and ServerPeekCursor to use it.
2019-09-04 14:55:05 -07:00
9d334948b1
Final Cleanup (hopefully) and including some performance numbers.
...
I microbenchmarked the storage queue standalone. i.e. the set and clearrange mutations
were performed solely at the in-memory storage queue. No other FDB components were involved
in this test. And hence the numbers presented here the best case numbers.
Test setup:
- 100M mutations: about 5% clearRange and 95% set mutations
- 100M rangeReads
- Keys/Values generated using deterministicRandom()
- A new version generated for each mutation (i.e. it's an extreme version test)
Performance comparison between std::map and std::deque for VersionedMap"
std::map std::deque
Time to perform the mutations 220.066 218.784
Time to perform buffered readRange 184.423 171.578
2019-09-04 11:29:29 -07:00
91020abb5a
Fixed another Assert.
2019-09-04 10:59:44 -07:00
ce53f7a896
Attempting to fix a segv.
2019-09-03 15:53:03 -07:00
76d353a332
Merge remote-tracking branch 'neelam/negoyal_dequeue' into negoyal_dequeue
2019-09-03 14:14:56 -07:00
bd80a67d46
Merge branch 'master' into mengxu/storage-engine-switch-PR-v2
2019-09-03 14:11:33 -07:00
624618751b
Merge branch 'master' of https://github.com/apple/foundationdb into negoyal_dequeue
...
Attempt at fixing a segv
2019-09-03 14:06:44 -07:00
ed38c0595b
Tried to fix segfault
2019-09-03 13:37:27 -07:00
24aad14f06
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-08-30 17:23:58 -07:00
2352193b3f
Code cleanup.
...
Removed commented out code and added some comments for clarification.
2019-08-30 16:48:43 -07:00
dc1d055b27
Merge pull request #2042 from senthil-ram/snap_cli_fix
...
fix fdbcli --exec 'snapshot create.sh' failure
2019-08-30 13:40:38 -07:00
a7237c4302
Merge pull request #2045 from atn34/disallow-scalar-network-messages
...
Disallow scalar network messages
2019-08-30 13:38:54 -07:00
84e2c9e1a5
Merge pull request #2041 from senthil-ram/snap_error_reporting
...
improved error msgs for snapshot cmd
2019-08-30 12:58:41 -07:00
b3277f2982
Fix #2009 posix compliant args for snapshot binary
2019-08-30 12:54:09 -07:00
3f9e392668
Merge pull request #2014 from etschannen/feature-fdbcli-sleep
...
Added a sleep command to fdbcli
2019-08-30 11:22:13 -07:00
f3bc7e0abd
do not duplicate data distribution disabled fields in status
...
fixed a few bugs related to the existing data distribution disabled fields in status
2019-08-29 18:41:34 -07:00
b5f9e9f307
Move comment above if
2019-08-28 15:21:58 -07:00
6aa0ada7b1
Replace scalar root types with proper messages
2019-08-28 14:40:50 -07:00
7a9097ea01
make fdbcli --exec 'snapshot create.sh' to succeed
2019-08-27 16:44:19 -07:00
5d87443323
improved error msgs for snapshot cmd
2019-08-27 16:43:52 -07:00
7414ddd891
cleanup merge/rebase error
2019-08-27 14:39:44 -07:00
3666c0c776
added more trace lines and added timeout to safety check in test workload
2019-08-27 14:39:44 -07:00
04d514c483
added a wait to check for master proxies changed and put in a few more trace events
2019-08-27 14:39:44 -07:00
00c2025d4b
fixed removeKeys impl, adjusted test workload, and introduced extra safety checks to NativeAPI and proxy
2019-08-27 14:39:44 -07:00
a07cf5d038
addressed code review comments
2019-08-27 14:39:44 -07:00
5a877d6b14
added safety check on client to prevent removing all servers from a team
2019-08-27 14:39:43 -07:00
807b02551e
updated help message and changed existing workload to use mark as failed feature
2019-08-27 14:39:43 -07:00
c908c6c1db
added command to fdbcli and changes to SystemData and ManagementAPI
2019-08-27 14:39:43 -07:00
2b941f51bd
Revert "fix: Use getReply* instead of tryGetReply in `monitorProxies`"
...
This reverts commit e7c94a2411
2019-08-26 18:31:08 -07:00
e7c94a2411
fix: Use getReply* instead of tryGetReply in `monitorProxies`
...
`tryGetReply` is unreliable, and since `monitorProxies` expects reply
after long period, the connection to coordinator gets closed due to
idle timeout, only to get reopened again in next loop to make
`openDatabase` request.
When using `getReply` our reliable message queue won't be empty and
connection will stay open.
2019-08-26 18:24:49 -07:00
64ce0c3285
Remove the unused getVersion from StorageServerInterface.
2019-08-26 13:53:54 -07:00
0b0c9fe0ff
data distribution status was combined into regular status
2019-08-21 14:44:15 -07:00
2b80d836f4
Merge branch 'release-6.2' into add-coordinator-to-status-roles-list
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-08-19 15:03:59 -07:00
3965a959b4
Merge pull request #2017 from ajbeamon/fix-fileconfigure-error-cases
...
Fix some validation logic when using the fdbcli fileconfigure command
2019-08-19 14:49:51 -07:00
4ab322f52c
Merge branch 'master' into mengxu/storage-engine-switch-PR-v2
2019-08-19 13:06:32 -07:00
7953545331
Fix an unknown_error when the file passed to fileconfigure doesn't contain a valid object (e.g. if you omit the enclosing {} of your object).
...
Fix an internal error when configuring regions with some storage servers that don't have a datacenter set.
2019-08-19 11:28:15 -07:00
2a436d5f6f
fix: do not block fdbcli from starting if DataDistributionStatus is not available
2019-08-16 18:15:02 -07:00
b448f92d61
StorageEngineSwitch:Remove unnecessary code and format code
...
Uncessary code include debug code and the unnecessary calling of
the removeWrongStoreType actor;
Format the changes with clang-format as well.
2019-08-16 16:53:38 -07:00
9d045c51e4
Add suggested nodiscards, and mention UNCANCELLABLE actors
2019-08-16 09:24:57 -07:00
4b97a7506d
Add some prudent [[nodiscard]]'s
2019-08-16 09:24:57 -07:00
b8e57f37d7
Add 'coordinator' to the list of roles that a process can have in status.
2019-08-15 14:42:49 -07:00
bb72cdd36a
Report lag with the usual "seconds" and "versions" fields. Rename and deprecate the qos.*version_lag_storage_server fields.
2019-08-15 13:42:39 -07:00
7a58b17e4e
Fixed a memory bug.
...
While replacing std::map with std::deque. Don't store
any pointers because emplace can invalidate iterators.
2019-08-15 12:01:34 -07:00
6581161dd3
Add ratekeeper's durability lag statistics to status
2019-08-15 11:07:04 -07:00
863a7ea84c
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-08-13 18:33:52 -07:00
a067e6e812
Merge pull request #1990 from etschannen/release-6.2
...
Fixed status reporting bugs related to connected clients
2019-08-13 16:24:12 -07:00
70ce678879
fix: max_protocol_clients were being added to the connected_clients list
...
fix: the clientCount was included clients with unknown protocol versions. This has been changed back to the pre-6.2 behavior where it is just a count of clients with known versions, and now clients with unknown versions are tracked explicitly as its own supported_version section
2019-08-13 15:54:40 -07:00
e6284684f0
StorageEngineSwitch:Always remove wrong storeType SS
...
In the old logic of switching storage engines, it marks a storage server
with wrong store type as undesired even though this can lead to no healthy team.
In the first version of the new storage engine switch, we mimic the same logic
of the old version.
2019-08-13 14:59:46 -07:00
caf8737f16
This is a draft PR. Not yet open for review. Please ignore for now.
...
Replaces std::map with std::deque in versionedMap.
2019-08-12 22:52:10 -07:00
a588710376
StorageEngineSwitch:Graceful switch
...
When fdbcli change storeType for storage engines,
we switch the store type of storage servers one by one gracefully.
This avoids recruiting multiple storage servers on the same process,
which can cause OOM error.
2019-08-12 17:37:52 -07:00
c4004a4eea
Don't count read version requests if we've already started one. Also avoid some other work that only needs to be done if we haven't started a read version request.
2019-08-12 15:55:48 -07:00
4a63de16e9
Merge pull request #1945 from xumengpanda/mengxu/tLog-code-read-v2
...
Add comments to DiskQueue and tLog
2019-08-08 13:24:32 -07:00
7ff46e6772
Merge branch 'master' into mengxu/performant-restore-PR
2019-08-07 20:31:56 -07:00
c9fa7237f1
Merge branch 'master' of github.com:apple/foundationdb
2019-08-06 16:40:14 -07:00
ba54508c47
code cleanup
2019-08-06 16:30:30 -07:00
370ba8b841
Remove --object-serializer flag from executables
2019-08-06 09:25:40 -07:00
2602cb3591
FastRestore:Rename RestoreConfig to RestoreConfigFR to fix link problem in windows
...
Because the current restore has defined RestoreConfig, windows linker complains.
This commit rename the RestoreConfig used in FastRestore as RestoreConfigFR.
2019-08-02 23:00:12 -07:00
9cc832cfd6
FastRestore:Fix Mac and Windows compilation error
2019-08-02 14:33:08 -07:00
3b54363780
FastRestore:Apply Clang-format
2019-08-01 18:09:12 -07:00
c9c50ceff8
Comments:Add comments to DiskQueue
...
No functional change.
2019-08-01 15:20:01 -07:00
7ccaeddf05
Merge branch 'master' into mengxu/performant-restore-PR
2019-08-01 13:23:17 -07:00
e61cac4ed4
Fix spacing issue; rename fdbrpc/Stats.h to fdbrpc/TimedRequest.h
2019-08-01 08:39:52 -07:00
7d247af500
Two minor bug fixes from recent optimizations
2019-07-31 19:14:11 -07:00
f33969c9d4
Merge pull request #1949 from atn34/at-what-cost3
...
Improve read performance part 1 of 2
2019-07-31 18:05:05 -07:00
dabe516320
Avoid unnecessary timer calls
2019-07-31 17:59:35 -07:00
2dd3a6afe1
Fully qualify base class members
2019-07-31 17:59:35 -07:00
0569df00f6
Remove indirection in LoadBalancedReply serialization
2019-07-31 17:59:35 -07:00
7d7aa27c2d
Merge pull request #1814 from dongxinEric/feature/1508/finer-grained-dd-controls
...
Added finer grained controls to DataDistribution in fdbcli.
2019-07-31 17:36:20 -07:00
f4bcfcd53c
Merge pull request #1944 from etschannen/master
...
more bug fixes
2019-07-31 17:13:58 -07:00
0063ef62ea
fix: the client would not shrink the proxy list in all cases
2019-07-31 16:06:51 -07:00
e8a9931dbe
Merge pull request #1918 from atn34/at-what-cost
...
Avoid memcpy for small types
2019-07-31 11:39:38 -07:00
5d20364423
Address review comments
2019-07-30 22:24:30 -07:00
1922c39377
Resolve review comments. 100K run shows one suspecious ASSERT_WE_THINK failure which I think could be a race.
2019-07-30 22:24:30 -07:00
ae11efcb0a
Made following changes:
...
- Make sure the disabled data distribution won't be accidentally enabled by the 'maintenance' command
- Make sure the status json reflects the status of DD accordingly
- Make sure the CLI can play with the new DD states correctly, i.e. print out warns when necessary
2019-07-30 22:20:45 -07:00
4ecfc9830f
Added finer grained controls to DataDistribution in fdbcli. What's happening under the hood is:
...
- Use pre-existing 'healthZone' key and write a special value to it in order to disable DD for all storage server failures
- Use a new system key 'rebalanceDDIgnored' key to disable/enable DD for all rebalance reasons(MountainChopper and ValleyFiller)
Kicked off two 200K correctness and showed no related errors.
2019-07-30 22:17:21 -07:00
efb9131657
Merge pull request #1909 from etschannen/feature-client-proxy-connections
...
Clients only connect to three proxies to reduce the number of network connections
2019-07-30 17:53:41 -07:00
54df2abe8e
fix: trace event did not compile
2019-07-30 17:52:53 -07:00
85767f2034
Update fdbclient/MonitorLeader.actor.cpp
2019-07-30 17:19:33 -07:00
69e7ed3e53
Merge pull request #1932 from etschannen/master
...
Bug fixes for rare bugs found by simulation
2019-07-30 17:18:30 -07:00
7aece7398b
fix: it was reducing the list of proxies on the coordinators, which would have made all the clients talking to that coordinator connect to the same set of proxies
...
optimized the code to avoid re-randomizing the same list of proxies
2019-07-30 17:15:24 -07:00
2cd10bc96c
Merge branch 'master' into feature-client-proxy-connections
2019-07-30 17:06:23 -07:00
63941e0d96
disable DD with a in-memory flag and use in snapv2
2019-07-30 17:04:51 -07:00
9f11f2ec53
Merge branch 'master' of github.com:apple/foundationdb
2019-07-30 16:55:56 -07:00
5c978f6129
fix: switchConnectionFile could get the proxies out of the clientInfo and continue connecting to the wrong cluster
2019-07-30 16:32:26 -07:00
59dd5916c5
Merge branch 'master' into rename-fault-tolerance-status-fields
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-07-30 15:11:25 -07:00
14648e20f9
Merge pull request #1901 from ajbeamon/data-distribution-receives-bytes-input-rate
...
Send bytes input rate to data distribution
2019-07-30 15:01:36 -07:00
438bc636d5
Rename max_machine_failures_without_losing_X to max_zone_failures_without_losing_X in status.
2019-07-30 14:02:31 -07:00
49eaa31984
Add a trace event for snap create failure
2019-07-29 20:28:22 -07:00
5a56f6b456
minor snap create client improvement and bug fixes
2019-07-29 20:28:22 -07:00
9f32edf4df
Avoid memcpy for small types
...
This is undefined behavior, since it's potentially a misaligned access.
But it's _probably_ not worse than the status quo
2019-07-29 17:11:45 -07:00
8425f53fc5
clients only connect to three proxies
2019-07-28 23:52:29 -07:00
5c98dcce6d
revert the proxy forwarding path, because it is no longer necessary as clients keep a persistent connection open with coordinators
2019-07-27 16:46:22 -07:00
b509a441e7
Merge branch 'master' into feature-skip-confirm
...
# Conflicts:
# bindings/flow/tester/Tester.actor.cpp
# bindings/go/src/_stacktester/stacktester.go
# bindings/java/src/test/com/apple/foundationdb/test/AsyncStackTester.java
# bindings/java/src/test/com/apple/foundationdb/test/StackTester.java
# bindings/python/tests/tester.py
# bindings/ruby/tests/tester.rb
# documentation/sphinx/source/api-c.rst
# documentation/sphinx/source/api-python.rst
# documentation/sphinx/source/api-ruby.rst
# documentation/sphinx/source/data-modeling.rst
# documentation/sphinx/source/developer-guide.rst
# fdbclient/vexillographer/fdb.options
# fdbserver/MasterProxyServer.actor.cpp
2019-07-27 15:08:13 -07:00
d1c7ab325b
fix: getConnectionFile could crash when connectionFile is null
2019-07-27 13:02:06 -07:00
1c4028d71e
fixed merge conflict error
2019-07-26 16:09:46 -07:00
27d9ce1143
fixed merge conflict
2019-07-26 15:23:36 -07:00
90e3b50213
Merge branch 'master' into feature-coordinator-connection
...
# Conflicts:
# fdbclient/DatabaseContext.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/workloads/KillRegion.actor.cpp
2019-07-26 15:05:02 -07:00
8149b5b352
Merge pull request #1413 from atn34/change-connection-file
...
Switch cluster file feature
2019-07-26 13:27:37 -07:00
ee92f0574f
fix: lastRequestTime was not updated
...
fix: COORDINATOR_REGISTER_INTERVAL was not set
fixed review comments
2019-07-26 13:23:56 -07:00
1706aaf199
Merge branch 'master' into mengxu/performant-restore-PR
...
Fix conflict in TlogServer.actor.cpp by accepting master changes
2019-07-26 11:46:27 -07:00
b0c31f28af
FastRestore:Fix bug that blocks restore
...
1) Should recruit only configured number of roles;
2) Should never register a restore master interface as a restore worker (loader or applier) interface.
2019-07-25 17:55:37 -07:00
9afd162e2f
remove snap v1 related code
2019-07-25 17:29:31 -07:00
be5d144b8b
added status information on connected clients
2019-07-25 17:15:31 -07:00
b91795d288
Send bytes input rate to DD.
2019-07-25 16:27:32 -07:00
45083edf74
Merge branch 'master' into mengxu/performant-restore-PR
...
Fix conflicts as well.
2019-07-25 10:46:11 -07:00
a65c9f92ed
get rid of all timeouts and other changes
2019-07-24 15:36:28 -07:00
a2f2ad96ff
code review comments and merge to master changes
2019-07-24 15:36:28 -07:00
869f77aef1
Few cosmetic edits and fixes
2019-07-24 15:36:28 -07:00
ddd4523816
bug fix in timeout & header file re-arrange in DD
2019-07-24 15:36:28 -07:00
0598779a40
snap create timeout for client call
2019-07-24 15:36:28 -07:00
ec9af66fc5
timeout snapshotdatabase call
2019-07-24 15:36:28 -07:00
62c14dae72
disable dd during snap and enable in restore
2019-07-24 15:36:28 -07:00
d0793f5ca2
snap v2: master proxy related changes
2019-07-24 15:36:28 -07:00
209448807d
snap v2: fdbclient related changes
2019-07-24 15:36:28 -07:00
83f4b8ebb1
Merge pull request #1866 from senthil-ram/setDDBugFix
...
setDDMode should set moveKeysLockWriteKey
2019-07-24 15:35:18 -07:00
9ad9bd4c1f
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-24 15:22:26 -07:00
8b73a1c998
removed verbose trace messages
2019-07-24 15:07:41 -07:00
4a866290b7
Clients keep a persistent connection open with coordinators to get updates to the list of proxies
...
Status still needs to be updated with client information with information from the coordinators
2019-07-23 19:22:44 -07:00
c4d1e16ab5
Merge pull request #1873 from atn34/uninitialized-memory
...
Fix uninitialized memory
2019-07-23 10:36:01 -07:00
1968db17e3
Initialize in default constructor for GetReadVersionReply
...
==10473== Uninitialised byte(s) found during client check request
==10473== at 0x1BA9ACE: sendPacket(TransportData*, ISerializeSource const&, Endpoint const&, bool, bool) (FlowTransport.actor.cpp:1252)
==10473== by 0x877C05: (anonymous namespace)::NetworkSenderActorState<GetReadVersionReply, (anonymous namespace)::NetworkSenderActor<GetReadVersionReply> >::a_body1cont2(GetReadVersionReply const&, int) [clone .isra.0] (networksender.actor
.h:40)
==10473== by 0x877CC6: a_body1when1 (networksender.actor.g.h:147)
==10473== by 0x877CC6: a_callback_fire (networksender.actor.g.h:161)
==10473== by 0x877CC6: ActorCallback<(anonymous namespace)::NetworkSenderActor<GetReadVersionReply>, 0, GetReadVersionReply>::fire(GetReadVersionReply const&) (flow.h:894)
==10473== by 0xC343A7: send<GetReadVersionReply&> (flow.h:343)
==10473== by 0xC343A7: send<GetReadVersionReply&> (fdbrpc.h:124)
==10473== by 0xC343A7: (anonymous namespace)::ForwardProxyActorState<(anonymous namespace)::ForwardProxyActor>::a_body1loopBody1when2(ReplyPromise<GetReadVersionReply> const&, int) (MasterProxyServer.actor.cpp:1814)
==10473== by 0xC33C10: (anonymous namespace)::ForwardProxyActorState<(anonymous namespace)::ForwardProxyActor>::a_body1loopBody1(int) (MasterProxyServer.actor.g.cpp:8167)
==10473== by 0xC35434: a_body1loopHead1 (MasterProxyServer.actor.g.cpp:8152)
==10473== by 0xC35434: a_body1loopBody1cont2 (MasterProxyServer.actor.g.cpp:8327)
==10473== by 0xC35434: a_body1loopBody1cont1when1 (MasterProxyServer.actor.g.cpp:8333)
==10473== by 0xC35434: a_body1loopBody1cont1when1 (MasterProxyServer.actor.g.cpp:8331)
==10473== by 0xC35434: a_callback_fire (MasterProxyServer.actor.g.cpp:8347)
==10473== by 0xC35434: ActorCallback<(anonymous namespace)::ForwardProxyActor, 3, Void>::fire(Void const&) (flow.h:894)
==10473== by 0x7E7BE7: SAV<Void>::finishSendAndDelPromiseRef() (flow.h:375)
==10473== by 0x8319FD: a_body1when1 (genericactors.actor.g.h:10892)
==10473== by 0x8319FD: a_callback_fire (genericactors.actor.g.h:10920)
==10473== by 0x8319FD: ActorCallback<(anonymous namespace)::ChooseActorActor<Void>, 0, Void>::fire(Void const&) (flow.h:894)
==10473== by 0x891917: void SAV<Void>::send<Void>(Void&&) (flow.h:343)
==10473== by 0x1C47ADC: send<Void> (flow.h:674)
==10473== by 0x1C47ADC: execTask (sim2.actor.cpp:1632)
==10473== by 0x1C47ADC: Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1cont1(Void const&, int) (sim2.actor.cpp:975)
==10473== by 0x1C47FF2: a_body1loopBody1when1 (sim2.actor.g.cpp:5092)
==10473== by 0x1C47FF2: Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1(int) (sim2.actor.g.cpp:5037)
==10473== by 0x1C47A6C: a_body1loopHead1 (sim2.actor.g.cpp:5020)
==10473== by 0x1C47A6C: Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1cont1(Void const&, int) (sim2.actor.g.cpp:5086)
==10473== Address 0x12db1ba1 is 2,977 bytes inside a recently re-allocated block of size 4,096 alloc'd
==10473== at 0x1CC5D7F: FastAllocator<4096>::allocate() (FastAlloc.cpp:290)
==10473== by 0x1CFAA68: operator new (FastAlloc.h:193)
==10473== by 0x1CFAA68: PacketWriter::nextBuffer() (Net2Packet.cpp:59)
==10473== by 0x1CFABD6: PacketWriter::writeAhead(int, SplitBuffer*) (Net2Packet.cpp:81)
==10473== by 0x1BA97EB: sendPacket(TransportData*, ISerializeSource const&, Endpoint const&, bool, bool) (FlowTransport.actor.cpp:1199)
==10473== by 0x7DEAD1: a_body1cont2 (networksender.actor.h:40)
==10473== by 0x7DEAD1: a_body1when1 (networksender.actor.g.h:147)
==10473== by 0x7DEAD1: a_callback_fire (networksender.actor.g.h:161)
==10473== by 0x7DEAD1: ActorCallback<(anonymous namespace)::NetworkSenderActor<GetValueReply>, 0, GetValueReply>::fire(GetValueReply const&) (flow.h:894)
==10473== by 0xF22767: send<GetValueReply&> (flow.h:343)
==10473== by 0xF22767: send<GetValueReply&> (fdbrpc.h:124)
==10473== by 0xF22767: (anonymous namespace)::GetValueQActorState<(anonymous namespace)::GetValueQActor>::a_body1cont5(int) [clone .isra.0] (storageserver.actor.cpp:890)
==10473== by 0xF2305C: (anonymous namespace)::GetValueQActorState<(anonymous namespace)::GetValueQActor>::a_body1cont3(int) [clone .isra.0] (storageserver.actor.g.cpp:1592)
==10473== by 0xF23447: a_body1cont2when1 (storageserver.actor.g.cpp:1627)
==10473== by 0xF23447: (anonymous namespace)::GetValueQActorState<(anonymous namespace)::GetValueQActor>::a_body1cont2(Void const&, int) [clone .isra.0] (storageserver.actor.g.cpp:1512)
==10473== by 0xF23507: a_body1when1 (storageserver.actor.g.cpp:1523)
==10473== by 0xF23507: a_callback_fire (storageserver.actor.g.cpp:1537)
==10473== by 0xF23507: ActorCallback<(anonymous namespace)::GetValueQActor, 0, Void>::fire(Void const&) (flow.h:894)
==10473== by 0x891917: void SAV<Void>::send<Void>(Void&&) (flow.h:343)
==10473== by 0x1C47ADC: send<Void> (flow.h:674)
==10473== by 0x1C47ADC: execTask (sim2.actor.cpp:1632)
==10473== by 0x1C47ADC: Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1cont1(Void const&, int) (sim2.actor.cpp:975)
==10473== by 0x1C47FF2: a_body1loopBody1when1 (sim2.actor.g.cpp:5092)
==10473== by 0x1C47FF2: Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1(int) (sim2.actor.g.cpp:5037)
==10473== Uninitialised value was created by a stack allocation
==10473== at 0xC342D0: (anonymous namespace)::ForwardProxyActorState<(anonymous namespace)::ForwardProxyActor>::a_body1loopBody1when2(ReplyPromise<GetReadVersionReply> const&, int) (MasterProxyServer.actor.g.cpp:8213)
2019-07-22 13:54:52 -07:00
e29a6ea280
Merge pull request #1871 from bnamasivayam/tr-priority-add-client-log
...
Track the priority of sampled Transaction as part of GetReadVersion e…
2019-07-22 13:04:52 -07:00
df652155fc
Addressed review comments
2019-07-22 12:17:05 -07:00
3507bfd52f
Trigger masterProxiesChangeTrigger in switchConnectionFileImpl
2019-07-22 11:13:07 -07:00
af267ba053
Track the priority of sampled Transaction as part of GetReadVersion event.
2019-07-19 17:31:49 -07:00
041531d283
Merge pull request #1864 from ajbeamon/client-thread-safety-fix
...
Fix thread-safety issue with connection file when creating database on client
2019-07-19 16:42:26 -07:00
846038b0e6
Merge pull request #1858 from bnamasivayam/rk-ssfetch-throttle
...
Ratekeeper throttling aggressively when unable to fetch storage server list
2019-07-19 16:41:58 -07:00
3045826e3c
Merge pull request #1819 from mpilman/flatbuffers-fixes2
...
Flatbuffers fixes2
2019-07-19 16:33:50 -07:00
c3a8ae4752
Merge pull request #1791 from fzhjon/fetch-keys-requests-priority
...
Introduce priority to fetchKeys requests from data distribution
2019-07-19 14:54:51 -07:00
f6183df8b9
Merge pull request #1852 from vishesh/task/issue-1840-non-blocking-exclusion
...
fdbcli: Add `no_wait` option in `exclude` command to avoid blocking
2019-07-19 12:49:29 -07:00
b93a08ac6f
Don't wrap the connection file in a reference until after we are on the main thread because references aren't thread safe.
2019-07-19 11:16:30 -07:00
0962641540
setDDMode should set moveKeysLockWriteKey
...
After takeMoveKeysLock notes down the owner and the
moveKeysLockWriteKey value, it monitors the above two in
pollMoveKeysLock and checks if anything is changed, but
setDDMode was not setting the moveKeysLockWriteKey and
so a sequence like disable, enable and disable would not
really disable DD.
2019-07-19 11:13:29 -07:00
b242760adf
Use functional cast instead of explicit operator call
2019-07-19 09:58:40 -07:00
bc5c65e5ab
Merge pull request #1756 from jzhou77/db-option
...
Add transaction getApproximateSize() API
2019-07-19 08:33:24 -07:00
d9a8657096
fdbcli: Add `no_wait` option in `exclude` command to avoid blocking
...
RESOLVES #1840
2019-07-18 13:07:31 -07:00
94c66f8d58
Merge pull request #1738 from bnamasivayam/consistency-check-disable
...
Disable/Re-enable consistency check through a database key.
2019-07-18 10:56:02 -07:00
7c0aca5b0c
Fix review comments: release notes and formatting
2019-07-18 10:26:39 -07:00
4b36099097
Update fdbclient/ReadYourWrites.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-07-18 10:10:36 -07:00
1ac2d01b03
Merge remote-tracking branch 'upstream/master' into flatbuffers-fixes2
2019-07-18 09:50:08 -07:00
406bcebdc4
Ratekeeper to throttle tpsLimit to 1 if it is not able to fetch storage server list for some configurable amount of time.
2019-07-17 18:08:17 -07:00
e8e48e0dbd
Fix size calculation
...
Mutations in writeRangeToNativeTransaction() is already counted, so there is no
need to count them again.
2019-07-16 15:21:13 -07:00
d5caf0c1b4
Merge branch 'flatbuffers-fixes2' of github.com:mpilman/foundationdb into flatbuffers-fixes2
2019-07-16 14:47:40 -07:00
e7e48a40ce
Fix a few IncludeVersion bugs
2019-07-16 13:28:29 -07:00
14cb21285f
Remove futureGetVersion in C binding and FutureVersion in Python binding
2019-07-16 10:46:07 -07:00
d5cc2beb5f
Update fdbclient/ReadYourWrites.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-07-16 10:33:25 -07:00
a89d09f1a6
Merge pull request #1844 from ajbeamon/configurable-trace-field-length
...
Make trace event field lengths default knobified and individually configurable.
2019-07-15 15:53:22 -07:00
2cd05e9ac9
Merge pull request #1712 from tclinken/add-local-rk-to-status
...
Track the local ratekeeper rate in status
2019-07-15 15:17:11 -07:00
54416f46fd
Pass type as param to VectorRef instead of bool
2019-07-15 15:08:49 -07:00
68f2b7a7f3
Event subclasses are managed through templates and don't have virtual functions, so don't use override.
2019-07-15 15:01:24 -07:00
9169232fa9
Add the new messages to Schema.
2019-07-15 13:47:27 -07:00
c8e94e601a
Merge pull request #1729 from etschannen/feature-fast-txs-recovery
...
Improve the recovery speed of the txnStateStore
2019-07-15 13:27:41 -07:00
6c6a1ca8f4
Expose serialization context too all traits
2019-07-15 12:58:31 -07:00
1a18c859c7
knobified the durability lag rate controls
2019-07-12 18:50:56 -07:00
c4c9e6cee7
fixed compiler errors
2019-07-12 18:28:41 -07:00
02de53160d
only skip confirm epoch live if CAUSAL_READ_RISKY is enabled
...
time checked on the proxy should be less than the time waited by the master to account for clock speed differences
setting REQUIRED_MIN_RECOVERY_DURATION and ENFORCED_MIN_RECOVERY_DURATION to 0 will go back to the old behavior
2019-07-12 17:58:16 -07:00
562bf6511a
Fix approximate size calculation
2019-07-12 16:53:37 -07:00
d5051b08dd
Make trace event field lengths (and total event sizes) default knobified and configurable. Add a transaction option to control the field length of transaction debug logging. Make the program start command line field less likely to be truncated.
2019-07-12 16:12:35 -07:00
9001f5d23f
Fix bug introduced in merge commit
2019-07-12 14:02:38 -07:00
a63969afb3
enforce a minimum recovery duration, which allows proxies to avoid checking if the epoch is alive as long as its last commit has been less than MINIMUM_RECOVERY_DURATION ago
2019-07-12 13:10:21 -07:00
75d4b612cf
Make object serializer versioned
2019-07-12 11:53:14 -07:00
969957e619
Merge branch 'master' into change-connection-file
2019-07-12 11:39:19 -07:00
2dcc3cfd0a
Deprecate fdb_future_get_version for version 620
...
Use fdb_future_get_int64 in all bindings.
2019-07-11 21:17:31 -07:00
319a823b79
Fix whitespace
2019-07-11 17:35:37 -07:00
7f21fc2bde
Serialize range result to string for speed
2019-07-11 17:35:37 -07:00
ae6f17625e
Support PacketBuffer's of arbitrary size
2019-07-11 17:35:37 -07:00
b2a89c8b77
Address review comments for PR #1756
...
Use fdb_future_get_int64 for language bindings and get rid of using Version
with getApproximateSize API.
2019-07-11 16:41:29 -07:00
f31884c749
Merge branch 'master' into add-priority-starts-to-status
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-07-11 15:26:52 -07:00
1700d492cf
Merge pull request #1823 from ajbeamon/cache-hit-rate-in-status
...
Tweak cache hit calculations and add cache hit rate to status
2019-07-11 14:06:06 -07:00
97609ad991
Add information about transaction starts at different priorities to status.
2019-07-11 13:54:44 -07:00
f4366e69ca
Unknown options should not be used internally (i.e. underneath thread-safe API). This commit removes various checks that options exist and replaces them with an ASSERT.
2019-07-11 11:25:39 -07:00
652cd77689
fixed merge conflicts and use new TaskPriority enum class
2019-07-11 09:56:59 -07:00
f12a3909f3
renamed workloads and made code style adjustments
2019-07-11 09:56:58 -07:00
1e9d31597c
removed extra parameter from getRange, added knob to guard new changes, and adjusted style/formatting in several places
2019-07-11 09:56:58 -07:00
f707d186fe
added new priority for fetchkeys requests and adjusted ddmetrics workload to run parallel with mako
2019-07-11 09:56:58 -07:00
bbef631872
fix: do not access optionInfo unless the option already exists in the map
2019-07-10 18:48:54 -07:00
c70a426f04
Update approximate size calculation
2019-07-10 15:00:50 -07:00
2c2836c6c7
Require API version to 620 for approximate size
2019-07-10 15:00:50 -07:00
5d1437c8e0
Push int directly to stack for getApproximateSize
2019-07-10 15:00:50 -07:00
0802df2c8f
Convert size from int to string before pushing onto stack
...
Using int is troublesome because the size of int can be different from the
desired 64 bits. So, using a string representation seems to be more consistent.
2019-07-10 14:58:35 -07:00
9d12843a26
Push size as tuple to stack
2019-07-10 14:58:35 -07:00
d5aaba3b15
Minor code fix
2019-07-10 14:58:07 -07:00
0ad2d2d16e
Add binding test for getApproximateSize API
2019-07-10 14:58:07 -07:00
8ef8b59fcc
Use ThreadFuture for getApproximateSize
...
Change return type to int64_t and fix C and Python binding to use the correct
type.
2019-07-10 14:58:07 -07:00
4c0e824456
Include unactorcompiler.h at the end of *.actor.h
2019-07-10 14:51:52 -07:00
c50a675bf0
Add transaction getApproximateSize() API
...
The size is the summation of expected size of mutations, read conflict ranges,
and write conflict ranges.
2019-07-10 14:51:52 -07:00
b4dbc6d7fa
Change the way cache hits and misses are tracked to avoid counting blind page writes as misses and count the results of partial page writes. Report cache hit rate in status.
2019-07-10 14:43:20 -07:00
d8948c8be1
Merge branch 'master' into feature-fast-txs-recovery
...
# Conflicts:
# fdbserver/TagPartitionedLogSystem.actor.cpp
2019-07-10 13:59:52 -07:00
7e919e361c
Merge pull request #1817 from etschannen/feature-proxy-forward
...
Proxies will forward clients to the next generation
2019-07-10 13:53:12 -07:00
a72d5b526a
Merge pull request #1767 from ajbeamon/fix-mvc-default-options
...
Make default and persistent options specifyable via annotations to fdb.options...
2019-07-10 13:19:45 -07:00
69d7c4f79c
Merge branch 'master' into track-run-loop-busyness
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Net2.actor.cpp
# flow/network.h
2019-07-09 18:39:23 -07:00
a174178be1
Merge branch 'master' into fix-mvc-default-options
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-07-09 18:33:14 -07:00
c8d86516f0
Merge pull request #1800 from ajbeamon/rename-datacenter-version-difference
...
Rename datacenter_version_difference to datacenter_lag and include bo…
2019-07-09 17:29:27 -07:00
7ad0d1a12b
Merge branch 'master' into feature-proxy-forward
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-07-09 17:26:15 -07:00
cce00bb413
Merge pull request #1808 from ajbeamon/improved-transaction-metrics
...
Improve TransactionMetrics
2019-07-09 16:46:17 -07:00
15ecba59db
Merge branch 'master' into fix-mvc-default-options
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-07-09 15:28:12 -07:00
fdd580c878
Restore some variable initializations that were unintentionally removed.
2019-07-09 15:00:11 -07:00
ae6c3e013a
monitorClientInfo: Wait for master proxy endpoint failures than triggers
...
This will not initiate request to get get new set of proxy unless we
know for a fact that endpoint has indeed failed, not just because the
connection to Peer was closed as it was sitting idle.
2019-07-09 14:24:16 -07:00
764a4591ad
Add a comment to internal flag
2019-07-09 14:17:26 -07:00
1582a2a24d
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-09 13:41:54 -07:00
1bac04509e
Track the local ratekeeper rate as a percentage
...
This value is reported in status for each storage server.
2019-07-09 12:46:53 -07:00
eabc610daa
Merge pull request #1813 from alexmiller-apple/log-version-4
...
Add a TLogVersion::V4
2019-07-09 08:42:20 -07:00
Jon Fu