6a57fab431
Bug fixes in lazy subtree deletion, queue pushFront(), queue flush(), and advancing the oldest pager version. CommitSubtree no longer forces page rewrites due to boundary changes. IPager2 and IVersionedStore now have explicit async init() functions to avoid returning futures from some frequently used functions.
2019-10-22 17:17:29 -07:00
44175e0921
COWPager will no longer expire read Snapshots that are still in use.
2019-10-18 01:27:00 -07:00
c3e2bde987
Deferred subtree clears and expiring/reusing old pages is complete. Many bug fixes involving scheduled page freeing, page list queue flushing, and expiring old snapshots (this was mostly written but not used yet). Rewrote most of FIFOQueue (again) to more cleanly handle queue cyclical dependencies caused by having queues that use a pager which in tern uses the same queues for managing page freeing and allocation. Many debug output improvements, including making BTreePageIDs and LogicalPageIDs stringify the same way everywhere to make following a PageID easier.
2019-10-15 03:10:50 -07:00
bb280e76db
Major refactor primarily to change BTree page ids from a single LogicalPageID to multiple, but also refactored write path data structures and memory lifetimes to use Refs and Arenas and carefully avoid unnecessary copying as it involved much of the same code. Pager reads can now explicitly avoid cache pollution. Refactored toString() helpers for easier debug output using common container types.
2019-09-27 21:46:24 -07:00
be37a7c01d
Added format versioning to COWPager page, header, BTreePage, BTree meta record. Added height to BTree pages.
2019-09-27 21:46:24 -07:00
b19ef86ab9
Pager2 interface now supports getting a read snapshot at a version and setting the oldest readable version. FIFOQueue now supports pushFront() which is needed for the BTree's incremental tree deletion process.
2019-09-27 21:46:24 -07:00
f81eeea495
Bug fixes. COWPager initialization was not flushing non-header pages with fsync() before writing and syncing the header. FIFOQueue was writing the initial page of a new queue multiple times. FIFOQueue::writePage() would unnecessarily (and invalidly) attempt to write if the page is not yet loaded.
2019-08-08 02:57:23 -07:00
da9c4e97d3
Added new pager interface, IPager2, whose write interface enables forcing the user to handle a page update causing a copy to a new Page ID. Implemented FIFOQueue<T> which uses pages of T stored in a Pager2 instance to implement a FIFO queue. Implemented COWPager, a copy-on-write Pager2 in which all page writes cause a change of Page ID. VersionedBTree, still only operating in single-version mode, now uses Pager2.
2019-08-07 02:36:33 -07:00
0fb8612ef5
debug_printf_noop() was incorrectly defined as a function, which still has a runtime cost of argument evaluation.
2019-05-22 03:40:18 -07:00
ebc96a7e0e
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-05-21 23:49:27 -07:00
13ba915a19
Fix more unused variable warnings
2019-04-17 15:38:08 -07:00
6660ccd936
Large rewrite/refactor from the page format outward. Replaced the in-page format, PrefixTree, with more general and templated implementation of the same concept, DeltaTree, which will be incrementally modifiable soon. New implementation for in-page cursors to use a per-page arena and share decoded binary tree nodes between cursors in the same page. New implementation for internal record B+Tree cursor which uses reference counted path hops which makes cloning a cursor very cheap. New implementation of external facing B+Tree cursor which makes efficient use of two internal cursors to find user-visible KV pairs and return them without copying keys and values except in the case of a large sharded value.
2019-02-21 02:46:30 -08:00
9d73166b3b
Many bug fixes related to concurrent page operations and pager shutdown.
2018-11-06 19:31:16 -08:00
df3bdde50b
Many bug fixes. AsyncFileCached write() on a page with a zero-copy read in progress would orphan the old page before the read was finished. Pager file operations were not converting page id to int64 for byte offset calculation. Pager was not calling releaseZeroCopy() after readZeroCopy() if there was an error or cancellation. Pager reads were using some variables that could go out of scope. BusyPage's mechanism for notifying when a physical page is no longer in use is itself no longer in use and therefore removed. Pager shutdown now cancels all outstanding reads. Improved some debug output.
2018-10-31 02:14:55 -07:00
0e84c1f438
Pager and btree debug output macro now prints local network address and time.
2018-10-25 03:57:09 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
8dd900a337
fixed the windows build
2018-10-18 20:26:45 -07:00
d7d2b58fef
Increased several testing parameters which exposed new bugs. Bug fix, PrefixTree's current format does not support trees larger than 64k. Cleaned up some debug output.
2018-09-24 02:42:23 -07:00
4df093f1ea
Implemented large key support by adding a superpage concept which is (currently) a BTree-level construct that treats multiple pages from the pager as a single contiguous page, splitting it up on write and reconstituting it on read. Refactored how pages are written along the way. The superpage construct will later be replaced by variable sized page support in IPager.
2018-09-19 00:32:39 -07:00
ee17854713
Bug fixes in internal page boundaries and prefix tree overhead calculations. Improved performance test output and increased amount of work it can do. Removed temporary restrictions on correctness test parameter ranges.
2018-06-14 17:52:25 -07:00
248dab79b6
Created “redwood” storage engine option and many changes to support that including IKeyValueStore::init() and custom DiskQueue file extensions.
2017-09-21 23:51:55 -07:00
d880569d52
Checkpointing progress on KeyValueStoreMVBTree. All methods are implemented to a usable point, and everything compiles, but Worker does not yet try to use it.
2017-09-21 04:43:49 -07:00
15622f026e
Forward range reads are done.
2017-09-15 17:27:13 -07:00
125d8168b4
Checkpointing progress on range reads (ordered iteration of user visible kv pairs via cursor). Added InternalCursor class which is used for all seeks and reads and sees the multi version, fragmented kv pairs and clears. Fixed a bug in commit which was discovered by the range read test where kv pairs (full or partial) could be missing from the tree but only for versions where they did not change. The write verification test did not find this because it only verifies exactly the changed versions of each key. The range test is not finished yet.
2017-09-15 05:19:39 -07:00
fe2d104000
Debug macro needs to be 1 instead of just being defined when it is in use.
2017-09-01 21:24:26 -07:00
d61f74e52b
Debug output changes.
2017-07-14 13:39:58 -07:00
fc0557252b
Bug fixes. Btree now uses page 0 as root and will write initial page only if necessary. Added debug printf macro.
2017-07-14 11:36:49 -07:00
b65ad3563c
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2017-06-09 14:56:41 -07:00
Stephen Atherton