d87e695fba
format files
2021-07-16 14:02:59 -07:00
29d61b708f
fix issues
2021-07-16 13:53:30 -07:00
1c7e3f08c2
format the files
2021-07-14 17:26:24 -07:00
126c87923f
switch unitmapper to static arry
2021-07-14 16:31:13 -07:00
737f053a7c
modify the reference
2021-07-13 10:10:23 -07:00
62ad629b44
fix comment typo
...
Co-authored-by: Steve Atherton <steve.atherton@snowflake.com>
2021-07-13 09:08:02 -07:00
79a2950b2f
move static potential combination outside the struct
2021-07-12 21:18:06 -07:00
7dbe10617b
format the code with devFormat
2021-07-12 16:07:18 -07:00
43e3e24bb7
resolve naming conflict
2021-07-11 14:11:30 -07:00
0104edb651
todo: raneg prefetch to leaf level, and format file
2021-07-11 14:09:48 -07:00
7935567f34
Update fdbserver/IPager.h
...
Co-authored-by: Steve Atherton <steve.atherton@snowflake.com>
2021-07-11 12:59:55 -07:00
fe575b995a
Added IO Priority to Redwood.
2021-07-09 05:05:08 -07:00
fa5a408eb4
change trace message
2021-07-06 22:31:37 -07:00
069d52a3cf
update trace information and fix minor errors
2021-07-06 22:31:37 -07:00
8ba2991263
check in idx
2021-07-06 22:31:37 -07:00
34cd0d04d7
update redwoodmetrics structure; need to fix bug
2021-07-06 22:31:37 -07:00
9d1aeb3e1e
initial draft on RedWoodMetrics changes
2021-07-06 22:31:37 -07:00
23d27663c0
Merge commit '98e463d642d1e674983842c8b2925981ff849729' into redwood-deltatree2-master
...
# Conflicts:
# fdbserver/IPager.h
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/VersionedBTree.actor.cpp
2021-06-09 01:31:16 -07:00
98e463d642
Merge pull request #4781 from sfc-gh-ngoyal/redwood_queue_improvements
...
Redwood queue format refactored to support faster cold start recovery of pager update log.
2021-06-09 00:11:57 -07:00
adcf126bfa
Removed commit read FlowLock because it costs too much overhead, will need another way to throttle. Removed readPage() fromCache argument as it is no longer useful.
2021-06-08 16:58:35 -07:00
a655ae3e48
Try not marking buffer undefined
2021-06-02 16:25:25 -07:00
c2c523d96f
Address review comments.
2021-05-25 22:52:10 -07:00
b37fb2ed5e
Merge branch 'master' into redwood_queue_improvements
2021-05-20 21:25:18 -07:00
f30793fd85
Implement getValuePrefix() using getValue() rather than the other way around to avoid the common getValue()'s actor state increasing from 128 to 256 bytes since it is a very hot code path.
2021-05-17 19:37:03 -07:00
e40538729e
Update fdbserver/IPager.h
...
Co-authored-by: Andrew Noyes <andrew.noyes@snowflake.com>
2021-05-17 14:46:06 -07:00
390b026f08
Merge branch 'master' into arena-page
2021-05-16 05:06:50 -07:00
cfeff9aa4b
Clarity improvement, loop was reusing variable name from enclosing scope.
2021-05-16 01:41:40 -07:00
2298567c2b
Use of aligned_alloc() for 4k pages causes too much wasted virtual memory. Added new 4k-aligned fast allocator, and changed Arena::allocatedAlignedBuffer() to be 4k-specific, now called Arena::allocate4kAlignedBuffer().
2021-05-14 23:12:00 -07:00
af0b51d684
Misc.
2021-05-09 00:44:07 -07:00
84a438b001
Merge branch 'master' into redwood_queue_improvements
2021-05-06 11:32:29 -07:00
e0e3cd39f1
Cherry picked #4678 changes onto #4780 to resolve conflicts.
2021-05-05 22:02:44 -07:00
799e7cc213
BTreeCursor now exposes the current Page and Mirror arenas so that users extracting Standalone Keys and Values can just reference those instead of copying.
2021-05-05 21:05:32 -07:00
f8a8bf315b
Added Arena::allocateAlignedBuffer() to get an aligned memory block owned by an Arena, and ArenaPage uses this.
2021-05-05 15:00:12 -07:00
06e8caa0aa
Replaced IPage with ArenaPage, a new Arena-based page class which enables StringRefs to hold a reference to Redwood page memory.
2021-05-05 12:35:39 -07:00
1467217d5c
Merge branch 'master' into redwood_queue_improvements
2021-05-04 23:03:21 -07:00
0a5c5a1fb5
Misc bug fixes.
2021-04-28 12:43:22 -07:00
5c3cb0da20
Pager now reports whether an uncacheable read hit the cache or not. CommitSubtree now does uncacheable reads so it can avoid copying the old version of the page for modification if it was not already in cache.
2021-04-16 20:13:23 -07:00
daa1796c99
Added Pager function for trying to evict a page from cache.
2021-04-15 00:08:29 -07:00
dbbb1bad13
First pass at Extent based queue for redwood remap queue.
2021-04-08 12:47:25 -07:00
90ebf90c8b
Refactored page rebuild logic to bulk build pages full and split pages more evenly.
2021-04-03 19:54:49 -07:00
e1b7dd0c7d
Merge remote-tracking branch 'upstream/release-6.3' into dyoungworth/fixMerge1
2020-08-22 12:25:19 -07:00
c18d16ad30
Multiple sibling page updates within the remap cleanup window can now avoid the second update step, copying updated contents over the original page, by instead updating the parent page to point to the new sibling locations.
2020-08-09 03:04:38 -07:00
2a393633b6
Improved const-correctness
...
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185 ), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:
- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
2020-06-17 18:20:43 -07:00
1966dbb576
Checkpointing large refactor of how commitSubtree() works regarding internal pages such that now they can be incrementally modified and detecting changes involves far fewer comparison and staging of records.
2020-05-08 07:11:51 -07:00
dda0993d16
Apply clang-format to Redwood source.
2020-04-24 14:12:40 -07:00
d4334256f1
Bug fix: BTree::writePages() was not accounting for the small amount of unpredictability in node overhead sizes which could lead to serializing a tree that doesn't fit into its destination page buffer. WritePages() now skips the prefix common to all records in the input set when calling deltaSize() to estimate resulting DeltaTree serialized sizes.
2020-04-08 20:18:41 -07:00
5a1884d054
Leaf page modifications are now made incrementally until the DeltaTree is out of space or the tree is too imbalanced, at which point it falls back to a linear merge and page(s) rebuild.
2020-02-21 16:26:44 -08:00
61558eea04
Implemented page preloading on BTree cursor seeks to enable hiding latency on soon-to-be-read sibling pages. Added random scans with various preload sizes to the set performance unit test. ObjectCache now tracks hits, misses, and pages which were preloaded but then never used prior to eviction. BTree pages no longer store flags because height is sufficient. Removed virtual specifier in classes not designed to be further inherited. Removed old prototype code (PrefixTree, IndirectShadowPager, MemoryPager) as some interface changes are incompatible and they are no longer worth maintaining.
2019-11-11 09:54:22 -08:00
4e9c3164a8
COWPager can now internally remap page IDs by version and has been renamed to DWALPager. This causes the B+Tree to no longer have to rewrite all ancestors of an updated page. FIFOQueue now has a read-only cursor and a peekAll() method to read an entire queue without popping it. Fixed some valgrind false positives, made some debug logging improvements. Fixed bug in pager shutdown where it could wait on an ActorCollection containing canceled futures.
2019-11-04 03:04:03 -08:00
0d993522d3
CommitSubtree() will now return an empty page set even for the tree root because commit_impl() handles this correctly. Improved commitSubtree() debug output related to which mutations are relevant to a subtree. Added random setting of range clear boundaries after clear() in Redwood correctness to make sure mutation buffer logic handles this correctly. B+Tree's dbEnd mutation is represented as a clear to prevent unnecessary rightmost subtree traversal during commit.
2019-10-28 04:00:37 -07:00
6a57fab431
Bug fixes in lazy subtree deletion, queue pushFront(), queue flush(), and advancing the oldest pager version. CommitSubtree no longer forces page rewrites due to boundary changes. IPager2 and IVersionedStore now have explicit async init() functions to avoid returning futures from some frequently used functions.
2019-10-22 17:17:29 -07:00
44175e0921
COWPager will no longer expire read Snapshots that are still in use.
2019-10-18 01:27:00 -07:00
c3e2bde987
Deferred subtree clears and expiring/reusing old pages is complete. Many bug fixes involving scheduled page freeing, page list queue flushing, and expiring old snapshots (this was mostly written but not used yet). Rewrote most of FIFOQueue (again) to more cleanly handle queue cyclical dependencies caused by having queues that use a pager which in tern uses the same queues for managing page freeing and allocation. Many debug output improvements, including making BTreePageIDs and LogicalPageIDs stringify the same way everywhere to make following a PageID easier.
2019-10-15 03:10:50 -07:00
bb280e76db
Major refactor primarily to change BTree page ids from a single LogicalPageID to multiple, but also refactored write path data structures and memory lifetimes to use Refs and Arenas and carefully avoid unnecessary copying as it involved much of the same code. Pager reads can now explicitly avoid cache pollution. Refactored toString() helpers for easier debug output using common container types.
2019-09-27 21:46:24 -07:00
be37a7c01d
Added format versioning to COWPager page, header, BTreePage, BTree meta record. Added height to BTree pages.
2019-09-27 21:46:24 -07:00
b19ef86ab9
Pager2 interface now supports getting a read snapshot at a version and setting the oldest readable version. FIFOQueue now supports pushFront() which is needed for the BTree's incremental tree deletion process.
2019-09-27 21:46:24 -07:00
f81eeea495
Bug fixes. COWPager initialization was not flushing non-header pages with fsync() before writing and syncing the header. FIFOQueue was writing the initial page of a new queue multiple times. FIFOQueue::writePage() would unnecessarily (and invalidly) attempt to write if the page is not yet loaded.
2019-08-08 02:57:23 -07:00
da9c4e97d3
Added new pager interface, IPager2, whose write interface enables forcing the user to handle a page update causing a copy to a new Page ID. Implemented FIFOQueue<T> which uses pages of T stored in a Pager2 instance to implement a FIFO queue. Implemented COWPager, a copy-on-write Pager2 in which all page writes cause a change of Page ID. VersionedBTree, still only operating in single-version mode, now uses Pager2.
2019-08-07 02:36:33 -07:00
0fb8612ef5
debug_printf_noop() was incorrectly defined as a function, which still has a runtime cost of argument evaluation.
2019-05-22 03:40:18 -07:00
ebc96a7e0e
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-05-21 23:49:27 -07:00
13ba915a19
Fix more unused variable warnings
2019-04-17 15:38:08 -07:00
6660ccd936
Large rewrite/refactor from the page format outward. Replaced the in-page format, PrefixTree, with more general and templated implementation of the same concept, DeltaTree, which will be incrementally modifiable soon. New implementation for in-page cursors to use a per-page arena and share decoded binary tree nodes between cursors in the same page. New implementation for internal record B+Tree cursor which uses reference counted path hops which makes cloning a cursor very cheap. New implementation of external facing B+Tree cursor which makes efficient use of two internal cursors to find user-visible KV pairs and return them without copying keys and values except in the case of a large sharded value.
2019-02-21 02:46:30 -08:00
9d73166b3b
Many bug fixes related to concurrent page operations and pager shutdown.
2018-11-06 19:31:16 -08:00
df3bdde50b
Many bug fixes. AsyncFileCached write() on a page with a zero-copy read in progress would orphan the old page before the read was finished. Pager file operations were not converting page id to int64 for byte offset calculation. Pager was not calling releaseZeroCopy() after readZeroCopy() if there was an error or cancellation. Pager reads were using some variables that could go out of scope. BusyPage's mechanism for notifying when a physical page is no longer in use is itself no longer in use and therefore removed. Pager shutdown now cancels all outstanding reads. Improved some debug output.
2018-10-31 02:14:55 -07:00
0e84c1f438
Pager and btree debug output macro now prints local network address and time.
2018-10-25 03:57:09 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
8dd900a337
fixed the windows build
2018-10-18 20:26:45 -07:00
d7d2b58fef
Increased several testing parameters which exposed new bugs. Bug fix, PrefixTree's current format does not support trees larger than 64k. Cleaned up some debug output.
2018-09-24 02:42:23 -07:00
4df093f1ea
Implemented large key support by adding a superpage concept which is (currently) a BTree-level construct that treats multiple pages from the pager as a single contiguous page, splitting it up on write and reconstituting it on read. Refactored how pages are written along the way. The superpage construct will later be replaced by variable sized page support in IPager.
2018-09-19 00:32:39 -07:00
ee17854713
Bug fixes in internal page boundaries and prefix tree overhead calculations. Improved performance test output and increased amount of work it can do. Removed temporary restrictions on correctness test parameter ranges.
2018-06-14 17:52:25 -07:00
248dab79b6
Created “redwood” storage engine option and many changes to support that including IKeyValueStore::init() and custom DiskQueue file extensions.
2017-09-21 23:51:55 -07:00
d880569d52
Checkpointing progress on KeyValueStoreMVBTree. All methods are implemented to a usable point, and everything compiles, but Worker does not yet try to use it.
2017-09-21 04:43:49 -07:00
15622f026e
Forward range reads are done.
2017-09-15 17:27:13 -07:00
125d8168b4
Checkpointing progress on range reads (ordered iteration of user visible kv pairs via cursor). Added InternalCursor class which is used for all seeks and reads and sees the multi version, fragmented kv pairs and clears. Fixed a bug in commit which was discovered by the range read test where kv pairs (full or partial) could be missing from the tree but only for versions where they did not change. The write verification test did not find this because it only verifies exactly the changed versions of each key. The range test is not finished yet.
2017-09-15 05:19:39 -07:00
fe2d104000
Debug macro needs to be 1 instead of just being defined when it is in use.
2017-09-01 21:24:26 -07:00
d61f74e52b
Debug output changes.
2017-07-14 13:39:58 -07:00
fc0557252b
Bug fixes. Btree now uses page 0 as root and will write initial page only if necessary. Added debug printf macro.
2017-07-14 11:36:49 -07:00
b65ad3563c
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2017-06-09 14:56:41 -07:00
Fuheng Zhao