087c6fa33d
Merge branch 'master' into feature-redwood
2020-02-26 12:25:04 -08:00
636c256606
Simplified return value of CommitSubtree to a single ChildLinkSetRef, which doesn't contain a version. Removed unnecessary record copying in several places in CommitSubtree.
2020-02-24 15:18:34 -08:00
72a549df14
Added basic mutation buffer insert/lookup test.
2020-02-24 00:31:09 -08:00
1e45bb2be3
Further refactoring of MutationBuffer to make upcoming ART implementation easier to plug in. Key and mutation for an iterator position are now accessed via member functions which gives the implementation an opportunity to decode or reconstitute keys on-demand.
2020-02-23 23:44:15 -08:00
eb7016a09a
Some more refactoring of MutationBuffer so it will be easier to have multiple implementations.
2020-02-23 22:48:43 -08:00
710bc3ecb1
Added optimization for single key clears to avoid an additional mutation buffer boundary key insertion. Made mutation buffer members private and created additional member functions to avoid needing access to the private members.
2020-02-23 22:24:16 -08:00
3618dff07b
Corrected statement in test output to reflect new logic.
2020-02-23 00:21:39 -08:00
078f85fea7
Bug fix. Pager could return an an oldest version that is no longer readable because it is being expired in the current commit cycle.
2020-02-23 00:15:35 -08:00
648040b070
Removed versioned records from the mutation buffer and commit path because support for this was incomplete and has high overhead costs when only exact committed versions are being read. The BTree could still contain versioned records in the future as the read cursors still support this, but some optimizations were added for when all internal records are at version 0 which is the case and has been for quite some time. Mutation buffer now stores keys and values in one arena per buffer.
2020-02-22 20:21:06 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
f1ec780b31
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
2020-02-21 17:43:11 -08:00
5a1884d054
Leaf page modifications are now made incrementally until the DeltaTree is out of space or the tree is too imbalanced, at which point it falls back to a linear merge and page(s) rebuild.
2020-02-21 16:26:44 -08:00
986e533c8c
Bug fix, DeltaTree was not updating numItems after erase / insert operations which only change the deleted flag of an existing equivalent entry.
2020-02-20 22:49:36 -08:00
893729e3a8
Bug fix, Redwood was not using direct IO where available.
2020-02-20 10:29:20 -08:00
0c7c815396
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/CMakeLists.txt
2020-02-12 16:12:57 -08:00
93e3e36d52
Changed RedwoodRecordRef::compare() to include value and updated VersionedBTree to adapt to this change. This fixes an (uncommitted) bug where DeltaTree inserts of a record matching a deleted record except for value would simply unhide the deleted record. For DeltaTree delete/insert sequences to work correctly compare() must only return 0 when the records are fully equivalent.
2020-02-12 01:18:35 -08:00
151f070809
Track failed evictions in ObjectCache.
2020-02-10 12:00:40 -08:00
9f4e39f1ef
DeltaTree now supports inserting a deleted item that is still present as a deleted node by simply clearing the node's deleted flag. Insertion of duplicates (determined by T::compare() == 0) is no longer allowed.
2020-02-07 01:00:11 -08:00
df2b0452b4
Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef.
2020-02-06 13:19:24 -08:00
9cf9c46668
DeltaTree now tracks internal used/free/deleted space, and insert will fail safely if a new item does not fit.
2020-02-05 15:30:04 -08:00
9dbb9e4cec
DeltaTree now supports node deletion by flagging nodes as deleted (a state which the T::Delta must be able to store and return). DeltaTree::Cursor will hide deleted nodes from view.
2020-02-04 01:22:27 -08:00
460c9b78d8
DeltaTree improvements. DecodedNodes store non-parent ancestor pointer which enables moveNext() and movePrev() operations to be O(1) from any leaf node. This also enables seekLessThanOrEqual to accept a hint in the form of a cursor with a position close to the target to avoid traversal from the root when the destination is close to the hint. DeltaTree T::compare() now must take a skipLen argument which is used by insert() and seek operations to skip over bytes known to be shared between the new item or query and every node in the tree.
2020-01-31 00:32:48 -08:00
da73164eda
Move crc32c from fdbrpc to flow
...
So that we can use it from a piece of flow code without breaking module
boundaries.
Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
2020-01-13 18:19:30 -08:00
560c1da805
Fix UBSAN report
...
/home/anoyes/workspace/foundationdb/fdbserver/VersionedBTree.actor.cpp:1606:10: runtime error: null pointer passed as argument 2, which is declared to never be null
2019-12-04 09:42:34 -08:00
04b2338e60
Added sequential insert speed test.
2019-11-23 00:40:58 -08:00
5f1644f293
DeltaTree::Reader is now DeltaTree::Mirror and supports insertion into a DeltaTree. DeltaTrees now support an item count, so BTreePage no longer has an item count, so the VersionedBTree format version has been bumped.
2019-11-23 00:09:11 -08:00
a9af2de1d2
Added option in redwood prefixed set test to use randomly generated prefixes in sorted order.
2019-11-20 15:55:59 -08:00
d91d744fd7
Typedefs to simplify ObjectCache a bit.
2019-11-20 03:20:23 -08:00
8d973ce762
Bug fix: Any time the least recently used ObjectCache entry is not evictable the effective size of the cache would grow by one entry for each read and will never shrink even once there are enough evictable pages to return the cache to its configured size. This has probably never actually happened because evictability of Redwood pages is currently based on having no pending IO, but it would be more of a problem if evictability were redefined to require a reference count of 1.
2019-11-19 23:45:13 -08:00
9e1e0d731d
Incremental subtree deletion now processes pages in parallel.
2019-11-18 02:34:37 -08:00
4e404e34e5
Added prefix size comparison test which generates records with a configurable prefix pattern and compares storage size between Redwood and the SQLite storage engine.
2019-11-17 17:53:42 -08:00
f841d14141
Bumped format versions, also simplified version scheme to a pager version and a btree version, removing per-page versions for queue and btree pages.
2019-11-11 16:28:21 -08:00
9227de5c20
Redwood correctness unit test was using wallclock based time limit which breaks determinism.
2019-11-11 15:13:58 -08:00
61558eea04
Implemented page preloading on BTree cursor seeks to enable hiding latency on soon-to-be-read sibling pages. Added random scans with various preload sizes to the set performance unit test. ObjectCache now tracks hits, misses, and pages which were preloaded but then never used prior to eviction. BTree pages no longer store flags because height is sufficient. Removed virtual specifier in classes not designed to be further inherited. Removed old prototype code (PrefixTree, IndirectShadowPager, MemoryPager) as some interface changes are incompatible and they are no longer worth maintaining.
2019-11-11 09:54:22 -08:00
2aa672cb59
When bulk building pages, make most of them full.
2019-11-07 15:52:23 -08:00
5d00d93665
Fixed errors found by valgrind involving incorrect page memory lifetimes for IO operations plus some false positives for partially used pages.
2019-11-07 13:57:09 -08:00
ea428b72f5
Bug fix, DWALPager must flush its page ID queues in order to get an accurate user page count.
2019-11-05 01:11:34 -08:00
0c95fef8aa
Bug fix in tree clear and size check where sometimes there could still be old versions of pages in use because not enough commits have passed for them to be rolled off and freed.
2019-11-04 12:59:57 -08:00
4e9c3164a8
COWPager can now internally remap page IDs by version and has been renamed to DWALPager. This causes the B+Tree to no longer have to rewrite all ancestors of an updated page. FIFOQueue now has a read-only cursor and a peekAll() method to read an entire queue without popping it. Fixed some valgrind false positives, made some debug logging improvements. Fixed bug in pager shutdown where it could wait on an ActorCollection containing canceled futures.
2019-11-04 03:04:03 -08:00
547616d5f1
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
2019-10-29 16:33:22 -07:00
9c0d671d07
Two bug fixes in Redwood related to split KV pairs and one was masking the other. The first bug resulted in an incomplete erasure of fragments for a split KV pair and the second bug would generate an unnecessary explicit null record for the same key which would cause reads to correctly see the key as missing. Redwood correctness test now clears the tree and verifies expected resulting pager footprint, which succeeds due to the bug fixes.
2019-10-29 01:31:59 -07:00
40d53e23f5
Optimization, only the first btree mutation boundary for a subtree needs to be compared to the subtree's lower bound. Also removed a check for a condition which is no longer possible due to other changes.
2019-10-28 16:05:11 -07:00
0d993522d3
CommitSubtree() will now return an empty page set even for the tree root because commit_impl() handles this correctly. Improved commitSubtree() debug output related to which mutations are relevant to a subtree. Added random setting of range clear boundaries after clear() in Redwood correctness to make sure mutation buffer logic handles this correctly. B+Tree's dbEnd mutation is represented as a clear to prevent unnecessary rightmost subtree traversal during commit.
2019-10-28 04:00:37 -07:00
5cf0045bc0
Merge pull request #2294 from satherton/feature-redwood
...
Bug fixes in Redwood
2019-10-25 14:56:01 -07:00
2ee1782c19
Bug fixes in Redwood. BTree height was not being reset when a new empty root is written. IKeyValueStore wrapper was not obeying the row limit in a reverse range query. Added yields to and delays to break up tasks and set IO priorities.
2019-10-25 14:52:06 -07:00
eb910b850b
fixed a window build error
2019-10-23 13:48:24 -07:00
613bbaecc4
Bug fix in queue page footprint tracking. Added VersionedBTree::destroyAndCheckSanity() which clears the tree, processes the entire lazy delete queue, and then verifies some pager usage statistics. This check is currently disabled because it appears to find a bug where the final state has a few more pages in use than expected. StorageBytes now includes the delayed free list pages as free space since they will be reusable soon.
2019-10-23 09:31:06 -07:00
6a57fab431
Bug fixes in lazy subtree deletion, queue pushFront(), queue flush(), and advancing the oldest pager version. CommitSubtree no longer forces page rewrites due to boundary changes. IPager2 and IVersionedStore now have explicit async init() functions to avoid returning futures from some frequently used functions.
2019-10-22 17:17:29 -07:00
44175e0921
COWPager will no longer expire read Snapshots that are still in use.
2019-10-18 01:27:00 -07:00
0e9d082805
Bug fixes in FIFOQueue concurrent nested reads and writes caused by the pager/freelist circular dependencies.
2019-10-17 21:34:17 -07:00
Steve Atherton