0da4c91ad5
Performance improvements in socket use and buffering which results in TLS network connections using around 3.5x less CPU.
...
PacketBuffers are 16k minimum sizes, using FastAlloc. All calling paths to PacketBuffer creation now pass a size hint if known. P2PNetworkTest improved in many ways, errors go to TraceEvents instead of stdout, error counts are reported in stats, range min/max parameters are combined into "min:max" strings, new options to "stutter" connections by delaying randomly before socket read or write, message size to send is now configured separately for incoming vs outgoing sessions.
2020-06-25 20:44:43 -07:00
8f842a66e4
Merge pull request #3397 from xumengpanda/mengxu/fr-addPrefix-PR
...
Fast Restore: Support and test addPrefix and removePrefix feature
2020-06-25 14:06:39 -07:00
af6b141736
Merge pull request #3407 from sfc-gh-tclinkenbeard/fix-vectorref-memory-leaks
...
Fix memory leaks in VectorRef
2020-06-25 00:54:33 -07:00
c13ba7aeb4
FastRestore:Cosmetic code style improvement
...
Also mute some trace events
2020-06-24 21:25:37 -07:00
94b1e17a96
FastRestore:Fix:Avoid reset txn backoff time in sanity check
2020-06-24 21:10:47 -07:00
d2b71e77c2
FastRestore:Fix:checkDatabaseLock tr should not reset
2020-06-24 20:21:49 -07:00
5cd3dad986
Merge pull request #3409 from sears/assert_strings
...
Print out the text (to stderr and as trace) of the failed condition o…
2020-06-24 14:58:12 -07:00
422003df34
Print out the text (to stderr and as trace) of the failed condition on assert failure
2020-06-24 14:21:48 -07:00
8de70432a6
Fix memory leaks in VectorRef
...
VectorRef<T> relies on T being trivially destructible, but this was not
yet enforced. By statically asserting that T is trivially destructible,
we avoid leaking memory by not calling the destructor for non-trivially
destructible types
2020-06-24 10:54:19 -07:00
b69c36f0d6
transformDatabaseContents:cosmetic change
2020-06-23 19:27:58 -07:00
198696bc1e
Move transformRestoredDatabase from server to client
...
AtomicRestore workload turns out to rely on the FileBackupAgent
client. Keeping transformRestoredDatabase in server makes linking harder.
2020-06-23 15:48:43 -07:00
6eb8dc58b6
FastRestore:addPrefixTest:Fix calling transformRestoredDatabase
2020-06-23 15:02:38 -07:00
84a640e4ee
AtomicRestore:Add addPrefix test for fast restore
2020-06-23 14:52:27 -07:00
4e27fd34e5
Refactor transformDatabaseContents into RestoreCommon
...
Prepare to enable addPrefix for atomicRestore
2020-06-23 14:33:13 -07:00
f011ff2c66
Merge remote-tracking branch 'upstream/master' into add-management-api-into-special-key-space
2020-06-23 11:23:18 -07:00
fc8715dacd
Adding write support for special-key-space
2020-06-23 11:21:03 -07:00
9cff90fb5f
runRYWTransaction:Add comment to remind func should be idempodent
2020-06-22 23:04:09 -07:00
dd5c1ec416
FastRestore:Add debug trace
...
The bug behavior is:
In transformDatabaseContents(), we write all new kvs into the DB in 100 keys batch.
We immediately read these keys out.
There are 100 keys missing.
2020-06-22 19:47:21 -07:00
42159ccfe0
Add -Wpessimizing-move and -Wredundant-move for clang
2020-06-22 23:45:03 +00:00
59361cedaa
Rename to have read and write impls and add set, clear, clearrange interfaces
2020-06-22 14:02:03 -07:00
5f4dde0b7a
Rename member variables
2020-06-22 13:59:16 -07:00
98026486fc
fix small issues
2020-06-22 13:53:24 -07:00
84c0856897
add management api to special-key-space getrange
2020-06-22 13:52:48 -07:00
3d6f69c8e2
FastRestore:addPrefix:Transform must clear both orignal and transformed range
...
Otherwise, anything left in the range can interfer with the result.
2020-06-21 22:18:12 -07:00
a59925dd73
Added more const-correctness improvements
2020-06-20 22:15:19 -07:00
85a284b13c
Use VectorRef::emplace_back in several places
2020-06-19 23:02:28 -07:00
c01a345652
Merge pull request #3386 from sfc-gh-tclinkenbeard/improve-const-correctness
...
Improve const-correctness
2020-06-18 12:40:08 -07:00
2fcef90cbc
FastRestore:Add addPrefix and removePrefix feature
2020-06-18 11:58:42 -07:00
ef3aa3b1c4
Replace comments with assertions in WriteMap.h
2020-06-18 11:53:36 -07:00
d7a8e554ed
FastRestore:Test for addPrefix and removePrefix
...
Add the skeleton code to test;
addPrefix and removePrefix are all empty for now.
2020-06-17 20:36:53 -07:00
5455467f31
Merge branch 'release-6.3' into mengxu/fr-addPrefix-PR
2020-06-17 19:16:22 -07:00
76ac5dfeaf
Merge branch 'release-6.3' into merge-release-6.3-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes/release-notes-630.rst
2020-06-17 18:31:06 -07:00
2a393633b6
Improved const-correctness
...
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185 ), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:
- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
2020-06-17 18:20:43 -07:00
e0fbbea79e
Merge pull request #3380 from sfc-gh-clin/update-cross-module-read-error-handler
...
Update special_keys_cross_module_read error handler
2020-06-17 18:17:32 -07:00
65916d7b10
Solve comments
2020-06-17 12:47:54 -07:00
9fbdd770ba
Update comments
2020-06-17 11:28:52 -07:00
0d9d20998e
Assert ryw not null, remove unit tests
2020-06-17 11:13:55 -07:00
32794a6917
Make key selectors clamp up to the boundary of the module rather than throw cross_module_error
2020-06-17 10:33:52 -07:00
bbdd451fb6
FastRestore:Add priorities for loader and applier endpoints
...
Endpoint for later restore step should have higher priority for
better performance: later step will not be blocked by the
next version batch's earlier steps.
2020-06-16 22:48:32 -07:00
c2a5b36eda
Change to initialization list for SpecialKeySpace constructor
2020-06-16 19:27:21 -07:00
327cc31e35
Revert "Request tracing"
2020-06-16 12:32:42 -07:00
96206a8032
Merge pull request #3368 from apple/release-6.3
...
Merge Release 6.3 to master
2020-06-15 20:15:22 -07:00
39c50798b8
Merge pull request #3341 from sfc-gh-clin/fix-rpc-calls-in-special-key-space
...
Fix fetchShardMetricsList_impl and add read cache in special key space
2020-06-15 16:12:38 -07:00
7a5fe3800d
Add comments, refine traces, add delay to dd rpc calls in the test
2020-06-15 15:31:55 -07:00
fabdffd6e4
Merge pull request #3332 from sfc-gh-tclinkenbeard/fix-memaccess-warnings
...
Fix class-memaccess warnings
2020-06-15 15:22:19 -07:00
4cd4c2ff9b
Added comment to TransactionOptions
2020-06-15 12:12:08 -07:00
acbfe2e4c9
Revert "Revert "Initial RocksDB""
2020-06-15 12:45:36 -04:00
3a914c71f2
Add TransactionOptions::clear function
2020-06-13 17:17:06 -07:00
8595813b7d
Merge pull request #3355 from apple/release-6.3
...
Merge Release 6.3 into master branch
2020-06-12 20:08:47 -07:00
f1ff29d70b
Revert debugTransaction in ReadYourWritesTransaction
...
This was accidentally smuggled in at PR 3329.
2020-06-12 15:23:31 -07:00
80334351c7
Merge pull request #3329 from sfc-gh-mpilman/features/flatbuffers-debugtx
...
Request tracing
2020-06-12 14:50:19 -07:00
5bc2e2e595
update comments, make isAsync() virtual, remove unused code
2020-06-12 10:12:44 -07:00
980bee1d13
Fix fetchShardMetricsList_impl and add read cache in special key space
2020-06-12 10:12:44 -07:00
1cbbd865de
Remove unused member from SSI.
2020-06-11 18:35:48 -07:00
9cd1614c82
Revert "Initial RocksDB"
2020-06-11 15:29:46 -07:00
ae6b716820
addressed more review comments
2020-06-11 09:58:41 -07:00
a4dbb5dd01
Merge branch 'trace-batch-thread-hostile' into rocksdb-6.3
2020-06-11 15:53:57 +00:00
9ef93714ac
Address review comments
2020-06-10 15:48:49 -07:00
4ab3441a95
Merge remote-tracking branch 'origin/master' into features/flatbuffers-debugtx
2020-06-10 15:31:29 -07:00
d9ddb84d34
remove weird include (added by clangd)
2020-06-10 13:18:26 -07:00
a332dff2ec
Fix class-memaccess warnings
2020-06-10 12:34:14 -07:00
ae10aaf5d6
Merge pull request #3323 from sfc-gh-tclinkenbeard/remove-boost-noexcept
...
Replace BOOST_NOEXCEPT with noexcept
2020-06-10 12:07:25 -07:00
f72d0a6a35
Merge pull request #3302 from sfc-gh-anoyes/anoyes/fix-watch-counting
...
Don't double count watches for too_many_watches
2020-06-10 10:40:11 -07:00
097261b7a4
Merge pull request #3326 from apple/release-6.3
...
Merge release 6.3 into master
2020-06-10 09:00:14 -07:00
99bf993815
Replace BOOST_NOEXCEPT with noexcept
2020-06-09 22:39:19 -07:00
164814cd23
Merge branch 'release-6.3' into mengxu/fr-addPrefix-PR
...
Also remove the unnecessary try catch block for unlockDatabase() in
parallelRestoreFinish actor
2020-06-09 22:04:19 -07:00
df064ac922
Merge pull request #3321 from xumengpanda/mengxu/fr-restore-ranges-PR
...
Fast Restore: Support restoring sub ranges in the framework
2020-06-09 21:01:39 -07:00
61fcce08cc
Merge pull request #3324 from apple/release-6.3
...
merge Release 6.3
2020-06-09 18:56:47 -07:00
dd8af487ec
FileBackupAgent:parallelRestoreFinish:verify if operation_cancelled error ever happens
2020-06-09 17:08:57 -07:00
8533daac50
parallelRestoreFinish:fix used_during_commit error
2020-06-09 15:40:24 -07:00
693d9e8b89
Merge branch 'master' into fdb_cache_wo_allocator
2020-06-09 15:09:58 -07:00
e10704fd76
Cherry-pick region related status changes from 6.3
2020-06-09 14:56:21 -07:00
e5e4cb5c90
parallelRestoreFinish:separate check restore finished and clear restoreRequestDoneKey
2020-06-09 11:32:20 -07:00
d85dc5a4d3
FastRestore:Only clear ranges that will be restored
...
Instead of clearning the entire normal key space.
This commit also removes some unnecessary tr->reset() which can invalid the txn backoff time.
2020-06-08 22:41:49 -07:00
e7fdd614fe
Remove assert on error_code_key_not_found in previous commit
...
Correctnss shows the assert does not hold in all scenarios
2020-06-08 19:02:56 -07:00
c829b12bbf
defined spans for read requests
2020-06-08 17:26:15 -07:00
5022566b35
Validate if key_not_found error ever happens
2020-06-08 16:59:00 -07:00
09c136e434
Framework for transaction tracing
2020-06-08 16:09:37 -07:00
55448a8d5a
Remove call in watch actor instead of Transaction::watch
2020-06-08 18:19:18 +00:00
3278222e2c
Merge pull request #3077 from tclinken/reenable-wclass-memaccess
...
Fix segmentation fault and reenable -Wclass-memaccess
2020-06-08 09:42:35 -07:00
731b77a8c2
Don't double count watches for too_many_watches
2020-06-07 23:54:22 +00:00
23a565ec63
Few bug fixes.
2020-06-05 16:27:04 -07:00
558fd6ba16
Merge pull request #3296 from apple/release-6.3
...
Merge release 6.3 to master
2020-06-05 10:49:06 -07:00
30bfd606c0
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/downloads.rst
# documentation/sphinx/source/release-notes.rst
# fdbserver/worker.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2020-06-04 19:21:32 -07:00
bf072d68ec
Workarounds for strange behaviors in Boost ssl sockets on MacOS and Linux. When writing to the ssl socket, write_some() would sometimes return BrokenPipe instead of WouldBlock unless onWriteable on the raw socket was checked first. On MacOS, even with the onWriteable check using a send size greater than 2016 (determined experimentally) would still result in the error. Also consolidated two identical copies of SendBufferIterator.
2020-06-04 01:44:05 -07:00
dfd44da02c
Fix enum ordering
2020-06-03 20:09:31 +00:00
cf13e00a8f
Merge remote-tracking branch 'origin/release-6.3' into fdb_cache_wo_allocator
2020-06-01 17:38:31 -07:00
414206015d
Merge pull request #3273 from apple/release-6.3
...
Merge release-6.3 into master
2020-05-29 16:58:36 -07:00
8329a242d2
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# documentation/sphinx/source/downloads.rst
# documentation/sphinx/source/release-notes.rst
2020-05-29 15:51:56 -07:00
ba63a66d12
Merge remote-tracking branch 'origin/master' into features/flatbuffers-debugtx
2020-05-29 15:38:09 -07:00
48eedb071f
Only destroy db if non-null
...
It could be the case that db is null for old client versions
2020-05-28 23:22:45 +00:00
c12c11acd3
Merge branch 'master' into dyoungworth/asyncFindFiles
2020-05-27 17:51:08 -07:00
8a7d98ab21
Started instrumentation
2020-05-27 15:36:04 -07:00
8731700d80
Merge remote-tracking branch 'upstream/release-6.3' into rocksdb-6.3
2020-05-27 20:02:25 +00:00
fa08384bc3
Merge pull request #3239 from apple/release-6.3
...
Merge release 6.3 into master
2020-05-27 08:29:28 -07:00
755267a7db
Merge pull request #3229 from ajbeamon/calling-run-network-twice-is-error
...
Eliminate the undefined behavior of calling run_network twice, instead returning an error
2020-05-26 21:24:52 -07:00
36ad1a95f4
Resolve conflicts when merge release-6.3 into master
2020-05-25 12:11:59 -07:00
1c35ad884f
Merge branch 'master' into mengxu/release-6.3-conflict-PR
...
Has conflict with master;
Next commit will fix the conflicts.
2020-05-25 12:01:49 -07:00
8a98ea3320
Changed non_flow_ref to flow_ref for clarity.
...
Also updated comment to explain that std::is_trivially_copyable
cannot be used to determine whether or not a type is a flow ref.
2020-05-23 22:57:41 -07:00
138569a489
another compile fix
2020-05-22 17:19:46 -07:00
ced65cd30b
finished explicitly versioning everything stored in the database
2020-05-22 17:14:21 -07:00
72ce997d22
explicitly versioned every key in systemData, so we only will update the associated protocolVersion when the serialization actually changes
2020-05-22 16:35:01 -07:00
86f712657e
Eliminate the undefined behavior of calling run_network twice, instead returning an error.
2020-05-22 13:31:06 -07:00
d128252e90
Merge release-6.3 into master
2020-05-22 09:25:32 -07:00
5d361fe532
Copy/paste rebase onto 6.3
2020-05-22 15:02:51 +00:00
7f9bdbb2b1
Update DDStatsRangeImpl
2020-05-21 18:26:34 +00:00
a2fd6d46a0
Cancel pending special key space reads on destroy
...
Previously the special key space would take a reference to the
transaction, but this doesn't make sense since the transaction might not
be refcounted
2020-05-21 17:37:11 +00:00
f22a8845e4
Finish putting actors in Platform.actor.cpp
2020-05-20 18:14:29 -07:00
cc9dd8ac72
Merge pull request #3214 from etschannen/release-6.3
...
Do not use tags in the keyServers map in the initial version of 6.3
2020-05-20 17:12:43 -07:00
3bf2ff9ae8
Merge pull request #2547 from tclinken/add-data-distribution-metrics
...
Add data distribution metrics
2020-05-20 17:11:40 -07:00
7d91a3a919
Merge branch 'release-6.3' of github.com:apple/foundationdb into release-6.3
2020-05-20 16:30:14 -07:00
afa7c0a9cc
Change the value format to as json string
2020-05-20 15:59:01 -07:00
3a8efb2d0b
Added advanceversion command to fdbcli
2020-05-20 15:47:31 -07:00
0d9135ee57
Merge remote-tracking branch 'upstream/release-6.3' into add-data-distribution-metrics
2020-05-20 15:21:26 -07:00
2aac9d1202
Merge pull request #3178 from ajbeamon/tag-throttling-by-priority
...
Tag (un)throttling by priority in fdbcli
2020-05-20 15:17:51 -07:00
f1dca05cf3
Merge pull request #3194 from etschannen/feature-small-endpoint
...
Optimized failure monitor performance
2020-05-20 14:48:13 -07:00
06676fc88f
Merge pull request #3166 from tclinken/6.3-avoid-ub
...
Eliminate some undefined behavior and const_cast usage
2020-05-20 14:36:30 -07:00
cc4874918a
Merge branch 'release-6.3' into tag-throttling-by-priority
...
# Conflicts:
# fdbserver/Ratekeeper.actor.cpp
2020-05-20 14:26:35 -07:00
6a006c4c1f
use the first RequestStream as the base endpoint
2020-05-20 13:52:22 -07:00
b589c3c940
Remove IncludeVersions.h, which is no longer needed for cmake-only builds
2020-05-20 13:23:02 -07:00
0f43ccbbf0
Merge pull request #3205 from bnamasivayam/add-locality-data
...
Add locality data
2020-05-20 10:56:31 -07:00
78bad4b07e
Merge pull request #3196 from ajbeamon/remove-deprecated-status-fields
...
Remove deprecated fields in status
2020-05-20 10:45:29 -07:00
298b94a044
clang-format the change
2020-05-19 20:51:02 -07:00
7971be2a19
update commetns
2020-05-19 20:48:21 -07:00
a6aff22e65
Update the special key prefix for dd-stats to \xff\xff/data_distribution_stats
2020-05-19 20:47:52 -07:00
2fbbf1e60c
Merge pull request #3203 from apple/release-6.3
...
Merge release-6.3 into master
2020-05-19 18:02:01 -07:00
2557da1d47
fix compiler errors
2020-05-19 16:53:14 -07:00
9313e490aa
change the first part of the UID to make comparisons cheaper
2020-05-19 16:48:20 -07:00
ead17d6a7f
another attempt at changing serialization
2020-05-19 16:28:01 -07:00
16990aeb6f
Add dcId to client transaction events.
2020-05-19 16:07:54 -07:00
7dd226d264
readded a missing line
2020-05-19 15:35:57 -07:00
93ca130602
Delete the unnecessay endKey field in DDMetricsRef, since each begin key is the previous end key
2020-05-19 13:32:42 -07:00
d636194d0d
Remove deprecated fields in status: worst_version_lag_storage_server and limiting_version_lag_storage_server
2020-05-19 13:12:10 -07:00
7a09d016a6
Merge branch 'release-6.3' into merge-release6.3-into-master
2020-05-19 12:52:44 -07:00
99e1fbdd47
another attempt at speeding up serialization
2020-05-19 12:36:11 -07:00
b7126f0acb
experimentally revert serialization of storage server interface
2020-05-18 19:54:09 -07:00
5013bc9106
fixed client compile issues; reduced the size of storage server interface
2020-05-18 17:20:15 -07:00
00dce3ca1d
clang-format
2020-05-18 14:23:17 -07:00
e3921504d7
Delete comments
2020-05-18 14:21:38 -07:00
fcca49f4d5
Add back an std::move that was accidentally removed. Some other small tweaks to avoid map lookups when tags aren't used.
2020-05-18 13:45:51 -07:00
6e684dda35
Using dd-stats zerobytes const in the workload
2020-05-18 11:53:26 -07:00
117ed14321
Add dd-stats zerobytes const to systemData
2020-05-18 11:52:36 -07:00
4bb90cb038
fix serilizer issue
2020-05-18 10:56:10 -07:00
bf62ef7c3e
Add dd-stats into special-key-space
2020-05-18 10:38:23 -07:00
f515e2c9d0
Add systemdata for dd-stats
2020-05-18 10:36:34 -07:00
d748166ed3
Merge branch 'master' of github.com:apple/foundationdb into features/flatbuffers-debugtx
2020-05-18 10:15:58 -07:00
6ce574f5ad
Merge remote-tracking branch 'upstream/release-6.3' into add-data-distribution-metrics
2020-05-17 23:36:52 -07:00
be2246c0ee
Merge branch 'release-6.3' of github.com:apple/foundationdb into release-6.3
2020-05-17 15:29:12 -07:00
8597b38d10
Add an assertion in SingleSpecialKeyImpl
2020-05-15 16:34:43 -07:00
87ac003967
add tests for no_module_found error, fix one bug caught by the test
2020-05-15 16:34:43 -07:00
032bc59cdb
Add tests for special_keys_cross_module_read cases, fixed one corner case
2020-05-15 16:34:43 -07:00
47f4f7085e
update special-key-space comments
2020-05-15 16:34:43 -07:00
2b84c3224f
update special-key-space comments
2020-05-15 16:34:43 -07:00
4a13107805
Add a test for special_keys_cross_module_read, fix two bugs, add several TODOs
2020-05-15 16:34:43 -07:00
8cc8ceb09c
Add check for range read cross module boundary, maybe improved later
2020-05-15 16:34:43 -07:00
c61e198e40
Using enum class instead of enum
2020-05-15 16:34:43 -07:00
43a28c254a
fix the bug when moving over null-impl-range
2020-05-15 16:34:42 -07:00
e4655f33fc
use std::unordered_map and ::at to get values
2020-05-15 16:34:42 -07:00
e1c4e5f619
Add enums for special-key-space modules and refactor related code
2020-05-15 16:34:42 -07:00
944f87e17b
Refactor special-key-space code that handles the normalization of keySelectors
2020-05-15 16:34:42 -07:00
03e395b353
Rename pks to sks, as the shorthand for special-key-space in header file
2020-05-15 16:34:42 -07:00
0e6d67aa5f
Rename pks to sks, as the shorthand for special-key-space
2020-05-15 16:34:42 -07:00
f605c8d74f
Narrow keyrange of special-key-space to [\xff\xff, \xff\xff\xff). Initilize new ones in correctness tests
2020-05-15 16:34:42 -07:00
bb1b5c4f9d
update special-key-space comments
2020-05-15 16:34:42 -07:00
14b23c146f
Support throttling and unthrottling tags by priority and their auto/manual state in fdbcli.
2020-05-15 12:47:55 -07:00
5e4a5ff49b
Walk TLogVersion forward for 7.0 now that 6.3 has been branched.
2020-05-14 18:39:46 -07:00
3ee4912312
Merge pull request #3152 from ajbeamon/tag-throttling-status-improvements
...
Add and fix tag throttling status fields
2020-05-14 16:08:05 -07:00
a160f9199f
Merge pull request #3171 from apple/release-6.3
...
Merge Release 6.3 into Master
2020-05-14 10:00:47 -07:00
667b224fc8
Several changes to reduce unnecessary moves/copies
2020-05-13 19:59:34 -07:00
1e0c10e9e8
control tag encoding of keyServersValue using a knob
2020-05-13 13:29:13 -07:00
27da91ab9e
Merge remote-tracking branch 'upstream/master' into mutation-debugging
2020-05-13 12:51:44 -07:00
d0c66d7282
Fix typo
2020-05-12 18:38:20 -07:00
634c988059
Tag throttles reacted poorly to the rate being set to max
2020-05-12 18:09:43 -07:00
e0526e0095
Add busiest read tags to storage server status
2020-05-12 15:49:40 -07:00
da14e1eb04
Fix: debug tagging was always enabled
2020-05-12 14:11:03 -07:00
f17f00fdd5
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-05-10 22:33:38 -07:00
048201717c
Fixed a number of problems with monitorLeaderRemotely
2020-05-10 14:20:50 -07:00
0957b33948
Merge pull request #3107 from etschannen/feature-fast-keyserver-decode
...
optimized decodeKeyServersValue on the proxy when loading all key ranges on startup
2020-05-09 18:05:29 -07:00
f9518c3441
Merge pull request #3069 from alexmiller-apple/tls-connection-count
...
YOLO at reducing TLS connection count via doing monitorLeader on coordinators
2020-05-09 17:12:27 -07:00
9d5e37f1a8
optimized decodeKeyServersValue on the proxy when loading all key ranges on startup
2020-05-09 16:54:35 -07:00
8bd5dcaff8
Merge branch 'master' into atn34/special-key-versioning
2020-05-09 15:34:20 -07:00
02307ba7b6
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/DatabaseContext.h
2020-05-09 07:50:29 -07:00
4cbf7b0fc0
Fix typo
2020-05-09 03:12:55 +00:00
c9205a5e1e
Explain module
2020-05-09 03:03:53 +00:00
d1b16eafc5
Only call getKeyRange once
2020-05-09 03:01:56 +00:00
d48185bbed
Remove unnecessary usages
...
I wish I remember why I added these
2020-05-09 02:47:05 +00:00
a53d05ea2d
Fix bug and add test
2020-05-09 02:46:40 +00:00
ef724bf939
Merge remote-tracking branch 'upstream/master' into add-data-distribution-metrics
2020-05-08 18:39:28 -07:00
ee0be88b99
consistency check can use a UIDtoTag map at a different version from the keyservers which could result in missing entries
2020-05-08 17:45:06 -07:00
e8b62e48f4
Rename DDMetrics to DDMetricsRef
2020-05-08 17:17:27 -07:00
f0f52fb2be
Merge branch 'master' into feature-small-endpoint
...
# Conflicts:
# fdbclient/StorageServerInterface.h
2020-05-08 16:37:35 -07:00
749fcd13b0
Merge branch 'master' into fdb_cache_wo_allocator
2020-05-08 16:23:29 -07:00
8166e1af8b
Storage server tags cannot change once used in the keyServer location map
2020-05-08 16:17:10 -07:00
049e3d36be
Revert custom allocator.
2020-05-08 15:43:48 -07:00
c6c0e2eb0f
Add several TEST macros
2020-05-08 05:37:37 +00:00
6389f6ac51
Combine \xff\xff/transaction/* into one module
2020-05-08 05:37:37 +00:00
697685574d
Fix worker_interfaces
...
They don't come sorted by address, they come sorted by process id
2020-05-08 05:37:37 +00:00
38830f0eb9
Fix range read bounds bug
2020-05-08 05:37:37 +00:00
76aed13bdc
Add prefix in comparator for worker_interfaces
2020-05-08 05:37:37 +00:00
524a5c19e4
Rename to onModuleRead, add missing usages
2020-05-08 05:37:37 +00:00
32d50fc359
Port over all existing special keys to framework
2020-05-08 05:37:37 +00:00
ce0e03a800
Add virtual destructor to SpecialKeyRangeBaseImpl
2020-05-08 05:37:37 +00:00
de53322d20
Prepare to transfer existing special key reads to framework
2020-05-08 05:37:37 +00:00
640eb41ba4
Introduce special key space in api version 630
2020-05-08 05:37:37 +00:00
6b35b1b686
Disallow no-module read by default
2020-05-08 05:37:37 +00:00
1d6209e304
Check for cross-module reads
2020-05-08 05:37:37 +00:00
a08bbcc539
Merge pull request #3046 from atn34/atn34/conflict-ranges
...
Add read/write conflict ranges to special key space
2020-05-07 22:36:18 -07:00
a833724322
Merge pull request #3078 from xumengpanda/mengxu/fr-circus-stall-PR
...
Performant restore: Various improvements based on circus test
2020-05-07 20:07:23 -07:00
62882f5552
Merge pull request #3088 from sears/lb_ub_perf_2
...
unrevert lb_ub_perf pull request
2020-05-07 18:23:17 -07:00
0e21c0c17e
More fixes to the merge problem
2020-05-07 16:06:46 -07:00
5f51d97444
Remove unneeded module violation.
2020-05-07 15:40:22 -07:00
a8e3ad3c15
Fix merge error
2020-05-07 15:10:50 -07:00
aed97a9f20
Merge branch 'master' into transaction-tagging
2020-05-07 14:52:22 -07:00
92730b2f05
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2020-05-07 14:50:25 -07:00
ff992060cd
Merge pull request #3073 from tclinken/fix-open-for-ide-build
...
Fix non-boost-related OPEN_FOR_IDE build errors
2020-05-07 14:47:59 -07:00
bb1724c3ae
remove obsolete less field from PTreeFingerEntry, increase finger size to 96 based on calculation in check()
2020-05-07 12:09:02 -07:00
c9489d45c5
remove the finger trimming loops from PTree::lower_bound and upper_bound
2020-05-07 12:08:54 -07:00
ea02c5b896
optimize PTreeFinger copy constructors
2020-05-07 12:08:47 -07:00
3569dfe6cb
stack allocate fingers
2020-05-07 12:08:39 -07:00
5e1c43f87b
replace double invocations of operator<() with three way compare().
2020-05-07 12:08:29 -07:00
99a805517e
reduce the number of calls to operator< made by lower_bound and upper_bound #2877
2020-05-07 12:08:21 -07:00
ae2d9744cb
simple benchmarks of IndexedSet, VersionedMap and std::map
2020-05-07 12:08:00 -07:00
9ccc417a4f
An indentifier must be set before calling LOG_TRANSACTION
2020-05-07 11:18:03 -07:00
90dbf33182
Merge pull request #3079 from bnamasivayam/client-transaction-info
...
Add read version and commit version info to getVersion and commit cli…
2020-05-07 09:59:35 -07:00
cd52005b4b
Merge branch 'master' into transaction-tagging
2020-05-07 09:16:52 -07:00
fbf436f45f
Various cleanup and knob adjustments.
2020-05-07 09:15:33 -07:00
5b7874c96d
Merge pull request #3084 from ajbeamon/fix-mutation-ref-serialization
...
MutationRef serialization depended on uninitialized data
2020-05-07 09:00:40 -07:00
c4aec8946d
Merge remote-tracking branch 'origin/master' into features/flatbuffers-debugtx
2020-05-06 14:19:32 -07:00
5559643403
Fix rebase issue
2020-05-06 13:39:18 -07:00
a09f30e48e
Make backupContainer's listFiles step asynchronous
2020-05-06 10:42:40 -07:00
8a6e177950
Merge remote-tracking branch 'upstream/master' into tls-connection-count
2020-05-05 16:49:36 -07:00
b1055a8501
Merge branch 'master' into transaction-tagging
2020-05-05 16:03:39 -07:00
f329164fb4
Merge pull request #2532 from dongxinEric/feature/hot-read-key-detection-part-2
...
Feature/hot read key detection part 2
2020-05-05 14:33:34 -07:00
37fe05fc76
Merge remote-tracking branch 'upstream/master' into atn34/conflict-ranges
2020-05-05 15:52:07 +00:00
8125096e9b
Revert "reduce the number of calls to operator< made by lower_bound and upper…"
2020-05-05 08:17:10 -07:00
38fa218a00
Remove unneeded duplicated code.
2020-05-05 01:02:01 -07:00
1117eae2b5
Rework to make ElectionResult code similar to OpenDatabase code.
...
And also restore and fix the delayed cluster controller code.
2020-05-05 01:00:17 -07:00
f4cc1d3a4f
Remove more auto-added headers
2020-05-04 20:44:57 -07:00
aaf87c0c5c
Remove headers vscode auto-added
2020-05-04 19:50:48 -07:00
dd033736ed
Merge branch 'master' into fdb_cache_subfeature2
2020-05-04 17:29:43 -07:00
46263f794a
Add missing line
2020-05-04 16:24:13 -07:00
9856099260
MutationRef serialization depended on type being initialized, but this wasn't true during vtable collection.
2020-05-04 16:07:04 -07:00
de1f85d106
Fix accounting bug in TagSet. Change serialization back to not including the number of elements. Fix knob usage.
2020-05-04 15:49:55 -07:00
4ae356b237
Merge pull request #2882 from sears/lb_ub_perf
...
reduce the number of calls to operator< made by lower_bound and upper…
2020-05-04 15:46:22 -07:00
362eb41509
Merge remote-tracking branch 'origin/master' into features/flatbuffers-debugtx
2020-05-04 14:17:26 -07:00
d039e75ac4
Merge remote-tracking branch 'origin/master' into features/flatbuffers-debugtx
2020-05-04 14:04:52 -07:00
b65c089edc
Don't assert extra conflict range is ready
2020-05-04 19:26:40 +00:00
d2e2532287
Fix memory error and avoid unnecessary copy
2020-05-04 18:50:59 +00:00
cd2176ac15
Adjust some knobs
2020-05-04 10:37:53 -07:00
36454bb3b8
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbclient/NativeAPI.actor.cpp
2020-05-04 10:23:25 -07:00
80a235aa80
Add some correctness tests
2020-05-04 10:15:18 -07:00
bb3d4b6b89
Add a bunch of TEST macros and some other little things
2020-05-04 10:11:36 -07:00
4a0925e9a6
Allow reading conflict ranges after ryw disabled tx commits
2020-05-04 16:43:20 +00:00
78d644f673
Merge branch 'master' into bugfixes/cmake-includes
2020-05-04 02:11:46 -07:00
8601deb3a1
Merge pull request #3038 from atn34/atn34/client-buggify-timeout-in-flight
...
Simulate a timeout for an in-flight commit with client buggify
2020-05-04 02:08:29 -07:00
d6bc5263ad
Add read version and commit version info to getVersion and commit client transaction events.
2020-05-03 22:42:06 -07:00
0c514e91c7
Use approximate write range for set_versionstamped_key
2020-05-04 04:43:50 +00:00
7b5d43da9c
FastRestore:Remove unused field in RestoreRequest
2020-05-03 20:59:47 -07:00
88fa9bdb63
Unrevert "Enable -Wclass-memaccess and fix warnings"
2020-05-02 20:43:50 -07:00
b33871a16b
Fix serialization bug for TagSet
2020-05-02 15:58:45 -07:00
ca1d457c6c
Fix bug where readVersion member future wasn't assigned
2020-05-02 07:59:37 -07:00
6caa714e8f
Commit some lines that were intended to be part of earlier commit
2020-05-02 06:47:25 -07:00
6ce4ddf1db
Fix a couple spots where we were missing checks for the new tag_throttled error
2020-05-01 21:47:12 -07:00
6db801ea53
shrink the serialized size of StorageServerInterface
2020-05-01 21:37:19 -07:00
decf3e82b0
Fix various bugs and make sure to cleanup throttles from the database when they expire
2020-05-01 21:36:28 -07:00
488bbcf33d
Merge pull request #2952 from xumengpanda/mengxu/fr-code-improvement-PR
...
Performant restore [29/XX]: Improve coding style and efficiency
2020-05-01 20:32:11 -07:00
9294f3ef4a
Merge pull request #3067 from atn34/atn34/set-versionstamped-key-no-ryw
...
Test SetVersionstampedKey without ryw enabled
2020-05-01 18:37:31 -07:00
494d1d3518
Fixed unused variable warning in atomicParallelRestore
2020-05-01 18:14:51 -07:00
175bdc842f
Merge pull request #3066 from tclinken/remove-dead-backup-agent-code
...
Remove dead backup agent code
2020-05-01 17:17:29 -07:00
ca92a39f5d
reduced the size of proxy and tlog interfaces
2020-05-01 16:41:20 -07:00
794e5eac59
Remove dead code
2020-05-01 22:58:55 +00:00
aca995d2c5
Removed dead backup agent code
2020-05-01 14:46:59 -07:00
43a63452d8
YOLO at reducing TLS connection count via doing monitorLeader on coordinators
2020-05-01 14:40:21 -07:00
9e5037291d
fix compiler errors
2020-05-01 14:30:50 -07:00
7edbbb642e
Test SetVersionstampedKey without ryw enabled
2020-05-01 20:58:37 +00:00
4d131bdd4a
Merge branch 'master' into feature-small-endpoint
2020-05-01 13:16:15 -07:00
c7cc546d80
build: Remove fdbclient/fdbclient.vcxproj
...
vcxproj files are unnecessary after CMake switch.
2020-05-01 12:33:46 -07:00
bd699f435c
fixed compiler errors
2020-05-01 11:01:09 -07:00
038f3834fc
Merge branch 'master' into mengxu/fr-code-improvement-PR
2020-05-01 09:26:29 -07:00
aed2d34bcb
Merge branch 'master' into feature-proxy-load-balance
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# flow/Knobs.cpp
2020-05-01 09:19:39 -07:00
ebe1a2e3dc
Merge remote-tracking branch 'upstream/master' into atn34/conflict-ranges
2020-05-01 15:56:16 +00:00
b3698ac921
Merge pull request #3060 from xumengpanda/mengxu/fr-agent-fix-PR
...
Fix fastrestore_agent errors when run in non-simulation mode
2020-04-30 22:46:39 -07:00
e9b9f0e845
onError wasn't properly retrying tag throttled errors
2020-04-30 21:12:34 -07:00
7f9e9224d2
Disallow reading write conflict range for SetVersionstampedKey
2020-05-01 01:41:25 +00:00
77b396d5ff
Allow reading tr.get_range('\xff\xff', '\xff\xff\xff')
2020-05-01 01:04:03 +00:00
ee104a116f
Merge pull request #3059 from xumengpanda/mengxu/too-many-files-fix-v2
...
Fix:Disable sanity check backup-metadata file
2020-04-30 16:41:40 -07:00
07a9a05683
FastRestore:Agent:Fix restore requests
2020-04-30 16:20:20 -07:00
6ee78aa3a4
Fix:Disable sanity check backup metadata file
...
Which can increase the false positive rate of TooManyFiles error
2020-04-30 16:16:14 -07:00
c14133a4ec
better serialization of empty StringRef
2020-04-30 11:13:59 -07:00
66228343f1
Merge branch 'master' into transaction-tagging
2020-04-30 08:12:03 -07:00
a0d67cac16
Merge branch 'master' into mengxu/fr-code-improvement-PR
2020-04-29 21:07:33 -07:00
0861785e80
Merge branch 'master' into feature-efficient-single-keyrange
2020-04-29 18:49:20 -07:00
fd0ee72293
Merge branch 'master' into feature-small-endpoint
2020-04-29 18:43:10 -07:00
f4af745a18
Merge branch 'master' into feature-tree-broadcast
2020-04-29 17:49:57 -07:00
f1d9c0cb97
Refactor/cleanup
2020-04-30 00:30:37 +00:00
99d5bb68dd
Make reading conflict ranges work for ryw disabled
2020-04-30 00:30:37 +00:00
3ddb0594c2
ASSERT(!tr.options.checkWritesEnabled)
2020-04-30 00:30:37 +00:00
ff2e82c7b2
Wait for timeout in pendingReads
2020-04-30 00:30:37 +00:00
94f8e090c8
Fix more bugs
2020-04-30 00:30:37 +00:00
fab33a8fd1
Improve test diagnostics, fix write_conflict_range bug
2020-04-30 00:30:37 +00:00
4fce3032d9
Rename debug_onIdle
2020-04-30 00:30:37 +00:00
5b365458cb
Fix bug
2020-04-30 00:30:37 +00:00
a60e9a7290
Fix bug - emit kv pair if we end in a conflict range
2020-04-30 00:30:37 +00:00
fdd6581167
Add write conflict range
2020-04-30 00:30:37 +00:00
e1c3c73d4c
Rudimentary read your own read conflict range
2020-04-30 00:30:37 +00:00
a998fa1e1a
Merge pull request #3053 from xumengpanda/mengxu/fr-bc-fix-PR
...
Fix performant restore bug introduced in PR 3041
2020-04-29 17:09:26 -07:00
3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd
...
Several bug fixes for new backups
2020-04-29 16:34:59 -07:00
84ca9c9176
Merge branch 'master' into mengxu/fr-bc-fix-PR
2020-04-29 16:25:12 -07:00
9da76c35ea
Fix a memory corruption error
...
The backup container URL should be Key instead of KeyRef.
2020-04-29 15:55:34 -07:00
519ac70a2a
Revert "Enable -Wclass-memaccess and fix warnings"
2020-04-29 15:51:29 -07:00
33dedaab35
FastRestore:Fix submitParallelRestore when targetVersion is unset
2020-04-29 15:44:54 -07:00
e7ab47b60d
Merge branch 'master' into feature-tree-broadcast
2020-04-29 14:35:42 -07:00
6ada5359b8
Merge branch 'master' into transaction-tagging
2020-04-29 14:27:21 -07:00
7de629e05f
Some knob tweaks
2020-04-29 14:26:29 -07:00
0420b3e786
fix compile error
2020-04-29 14:05:53 -07:00
7cebe743f9
A number of bug fixes of rare correctness errors
2020-04-29 13:50:13 -07:00
627eb93dd9
Merge pull request #2941 from etschannen/feature-tree-broadcast
...
Implemented a tree broadcast for the proxy and cluster controller
2020-04-29 13:38:09 -07:00
6156042f67
Merge pull request #3017 from tclinken/enable-wclass-memaccess
...
Enable -Wclass-memaccess and fix warnings
2020-04-29 10:38:59 -07:00
6ab3ad4037
remove obsolete less field from PTreeFingerEntry, increase finger size to 96 based on calculation in check()
2020-04-29 10:10:30 -07:00
fa5855c956
remove the finger trimming loops from PTree::lower_bound and upper_bound
2020-04-29 10:10:30 -07:00
bfc6d63986
optimize PTreeFinger copy constructors
2020-04-29 10:10:30 -07:00
5a8ec488d3
stack allocate fingers
2020-04-29 10:10:30 -07:00
dcdda88794
replace double invocations of operator<() with three way compare().
2020-04-29 10:10:30 -07:00
0115d94e7b
reduce the number of calls to operator< made by lower_bound and upper_bound #2877
2020-04-29 10:10:30 -07:00
14836619ab
simple benchmarks of IndexedSet, VersionedMap and std::map
2020-04-29 10:10:30 -07:00
76fb345dd1
Merge branch 'master' into feature-tree-broadcast
...
# Conflicts:
# fdbrpc/FailureMonitor.actor.cpp
2020-04-29 09:51:22 -07:00
a8becb9027
Use calculated block size for range files in unit tests
2020-04-28 21:13:18 -07:00
f4010c632a
Merge pull request #3041 from xumengpanda/mengxu/sim-restore-agent
...
Performant restore [31/xx]: Fix for fast restore agent
2020-04-28 20:28:18 -07:00
3adbc89510
update special-key-space argument
2020-04-28 16:50:24 -07:00
a69dda4263
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-28 16:33:35 -07:00
562456028c
Fix the interface of special-key-space, only keys in (\xff\xff, \xff\xff\xff) are valid
2020-04-28 16:33:10 -07:00
364142d02c
Fix backupWorkerEnabled flag not set bug for first backup
...
This can cause restore failures because the latestLogEndVersion could be wrong.
2020-04-28 15:39:24 -07:00
ba261eda36
Fix a backup container unit test
...
Write a valid range file instead of random data so that checking its content is
fine.
2020-04-28 15:39:24 -07:00
7d59e53349
Consolidate makePadding()
2020-04-28 15:39:23 -07:00
b315163033
Fix a memory corruption error
2020-04-28 15:39:23 -07:00
b7f5f3be48
merge in master
2020-04-28 13:11:47 -07:00
da24b5a1a9
Merge remote-tracking branch 'origin/master' into features/flatbuffers-debugtx
2020-04-28 18:20:55 +00:00
f5e8345496
FastRestoreAgent:Use atomicParallelRestore to kick off restore
...
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
2020-04-27 22:15:00 -07:00
77ea27dc56
Simulate a timeout for an in-flight commit with client buggify
2020-04-27 22:09:03 +00:00
573b10f9f5
Fix a merge bug and cleanup some argument passing.
2020-04-27 15:03:56 -07:00
41c517a5dd
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2020-04-27 13:05:24 -07:00
a65e97209a
Fix bug in flag management. Pass priority into ratekeeper's updateRate and use it when setting autothrottles.
2020-04-27 11:34:12 -07:00
32f4f013c2
Bug fix: HTTPS requests could hang. HTTP client was using stated contract for IConnection, which was different from the contract assumed by FlowTransport and implemented by SSLConnection. Updated HTTP client, added comments to IConnection to clarify the actual interface contract.
2020-04-24 17:59:32 -07:00
239876351b
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-24 11:31:16 -07:00
35c18ac60a
Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters.
2020-04-23 20:50:40 -07:00
3fb91e8f0c
Merge pull request #3016 from etschannen/master
...
Merge release 6.2 into master
2020-04-23 15:44:47 -07:00
59217ddf1e
Remove sanity check on metadata
...
The sanity check parses each range file to get the key range of each
range file.
The parsing incurs restore_unsupported_file_version error.
We need to include this sanity check before 6.3 release.
2020-04-23 14:59:58 -07:00
201553cfaf
Merge branch 'master' of github.com:apple/foundationdb into features/flatbuffers-debugtx
2020-04-23 14:05:53 -07:00
c87aa33941
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/go/src/fdb/generated.go
# documentation/sphinx/source/api-common.rst.inc
# documentation/sphinx/source/api-ruby.rst
# documentation/sphinx/source/release-notes.rst
# fdbclient/FailureMonitorClient.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-04-23 13:47:53 -07:00
76d4570d5d
Merge pull request #3008 from etschannen/release-6.2
...
Added logging for parallel peeks from TLogs
2020-04-23 10:27:49 -07:00
Steve Atherton