da73164eda
Move crc32c from fdbrpc to flow
...
So that we can use it from a piece of flow code without breaking module
boundaries.
Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
2020-01-13 18:19:30 -08:00
560c1da805
Fix UBSAN report
...
/home/anoyes/workspace/foundationdb/fdbserver/VersionedBTree.actor.cpp:1606:10: runtime error: null pointer passed as argument 2, which is declared to never be null
2019-12-04 09:42:34 -08:00
f841d14141
Bumped format versions, also simplified version scheme to a pager version and a btree version, removing per-page versions for queue and btree pages.
2019-11-11 16:28:21 -08:00
9227de5c20
Redwood correctness unit test was using wallclock based time limit which breaks determinism.
2019-11-11 15:13:58 -08:00
61558eea04
Implemented page preloading on BTree cursor seeks to enable hiding latency on soon-to-be-read sibling pages. Added random scans with various preload sizes to the set performance unit test. ObjectCache now tracks hits, misses, and pages which were preloaded but then never used prior to eviction. BTree pages no longer store flags because height is sufficient. Removed virtual specifier in classes not designed to be further inherited. Removed old prototype code (PrefixTree, IndirectShadowPager, MemoryPager) as some interface changes are incompatible and they are no longer worth maintaining.
2019-11-11 09:54:22 -08:00
2aa672cb59
When bulk building pages, make most of them full.
2019-11-07 15:52:23 -08:00
5d00d93665
Fixed errors found by valgrind involving incorrect page memory lifetimes for IO operations plus some false positives for partially used pages.
2019-11-07 13:57:09 -08:00
ea428b72f5
Bug fix, DWALPager must flush its page ID queues in order to get an accurate user page count.
2019-11-05 01:11:34 -08:00
0c95fef8aa
Bug fix in tree clear and size check where sometimes there could still be old versions of pages in use because not enough commits have passed for them to be rolled off and freed.
2019-11-04 12:59:57 -08:00
4e9c3164a8
COWPager can now internally remap page IDs by version and has been renamed to DWALPager. This causes the B+Tree to no longer have to rewrite all ancestors of an updated page. FIFOQueue now has a read-only cursor and a peekAll() method to read an entire queue without popping it. Fixed some valgrind false positives, made some debug logging improvements. Fixed bug in pager shutdown where it could wait on an ActorCollection containing canceled futures.
2019-11-04 03:04:03 -08:00
547616d5f1
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
2019-10-29 16:33:22 -07:00
9c0d671d07
Two bug fixes in Redwood related to split KV pairs and one was masking the other. The first bug resulted in an incomplete erasure of fragments for a split KV pair and the second bug would generate an unnecessary explicit null record for the same key which would cause reads to correctly see the key as missing. Redwood correctness test now clears the tree and verifies expected resulting pager footprint, which succeeds due to the bug fixes.
2019-10-29 01:31:59 -07:00
40d53e23f5
Optimization, only the first btree mutation boundary for a subtree needs to be compared to the subtree's lower bound. Also removed a check for a condition which is no longer possible due to other changes.
2019-10-28 16:05:11 -07:00
0d993522d3
CommitSubtree() will now return an empty page set even for the tree root because commit_impl() handles this correctly. Improved commitSubtree() debug output related to which mutations are relevant to a subtree. Added random setting of range clear boundaries after clear() in Redwood correctness to make sure mutation buffer logic handles this correctly. B+Tree's dbEnd mutation is represented as a clear to prevent unnecessary rightmost subtree traversal during commit.
2019-10-28 04:00:37 -07:00
5cf0045bc0
Merge pull request #2294 from satherton/feature-redwood
...
Bug fixes in Redwood
2019-10-25 14:56:01 -07:00
2ee1782c19
Bug fixes in Redwood. BTree height was not being reset when a new empty root is written. IKeyValueStore wrapper was not obeying the row limit in a reverse range query. Added yields to and delays to break up tasks and set IO priorities.
2019-10-25 14:52:06 -07:00
eb910b850b
fixed a window build error
2019-10-23 13:48:24 -07:00
613bbaecc4
Bug fix in queue page footprint tracking. Added VersionedBTree::destroyAndCheckSanity() which clears the tree, processes the entire lazy delete queue, and then verifies some pager usage statistics. This check is currently disabled because it appears to find a bug where the final state has a few more pages in use than expected. StorageBytes now includes the delayed free list pages as free space since they will be reusable soon.
2019-10-23 09:31:06 -07:00
6a57fab431
Bug fixes in lazy subtree deletion, queue pushFront(), queue flush(), and advancing the oldest pager version. CommitSubtree no longer forces page rewrites due to boundary changes. IPager2 and IVersionedStore now have explicit async init() functions to avoid returning futures from some frequently used functions.
2019-10-22 17:17:29 -07:00
44175e0921
COWPager will no longer expire read Snapshots that are still in use.
2019-10-18 01:27:00 -07:00
0e9d082805
Bug fixes in FIFOQueue concurrent nested reads and writes caused by the pager/freelist circular dependencies.
2019-10-17 21:34:17 -07:00
6b7317da9b
Bug and clarity fixes to tracking FIFOQueue page and item count.
2019-10-15 03:36:22 -07:00
c3e2bde987
Deferred subtree clears and expiring/reusing old pages is complete. Many bug fixes involving scheduled page freeing, page list queue flushing, and expiring old snapshots (this was mostly written but not used yet). Rewrote most of FIFOQueue (again) to more cleanly handle queue cyclical dependencies caused by having queues that use a pager which in tern uses the same queues for managing page freeing and allocation. Many debug output improvements, including making BTreePageIDs and LogicalPageIDs stringify the same way everywhere to make following a PageID easier.
2019-10-15 03:10:50 -07:00
fa357ef1ca
Bug fixes. COWPager's page cache was being initialized too late in recovery, after it had already been used. Cursor's KeyValueRef memory sometimes pointed to freed memory from an InternalCursor that had been moved. Added valgrind macros to avoid false positives.
2019-10-02 06:43:11 -07:00
30c56536bd
Refactored FIFOQueue to support fixed or variable-sized types. Bug fixes in page deletion and lazy delete queuing.
2019-10-01 02:06:00 -07:00
24e03a55ad
Some code cleanup and updated TODOs.
2019-09-28 13:27:00 -07:00
2854087118
Implemented COWPager non-caching page reads.
2019-09-28 13:26:01 -07:00
0a3b7ff909
Cleanup of old or temporary code.
2019-09-28 00:26:57 -07:00
bb280e76db
Major refactor primarily to change BTree page ids from a single LogicalPageID to multiple, but also refactored write path data structures and memory lifetimes to use Refs and Arenas and carefully avoid unnecessary copying as it involved much of the same code. Pager reads can now explicitly avoid cache pollution. Refactored toString() helpers for easier debug output using common container types.
2019-09-27 21:46:24 -07:00
be37a7c01d
Added format versioning to COWPager page, header, BTreePage, BTree meta record. Added height to BTree pages.
2019-09-27 21:46:24 -07:00
b19ef86ab9
Pager2 interface now supports getting a read snapshot at a version and setting the oldest readable version. FIFOQueue now supports pushFront() which is needed for the BTree's incremental tree deletion process.
2019-09-27 21:46:24 -07:00
5384cf8f9c
Bug fixes in FIFOQueue. Read cursor would not start loading pages again after its end was pushed forward. Queue flushing of the free list queue would leave tail cursor in a bad state.
2019-09-27 21:46:24 -07:00
1bb323fa8c
Bug fix in FIFOQueue pop() when freeing an exhausted page causes a recursive pop() from the same queue, which happens when the queue is the freelist itself and the write cursor is also at the end of its page.
2019-09-27 21:46:24 -07:00
1882b58d21
COWPager dispose() was not deleting the page file.
2019-09-27 21:46:24 -07:00
8d2d1f4f24
Bug fix, COWPager recovery can't simulate header read failure using buggify anymore because the backup header is now a previous version and it is invalid to not recover with an fsync'd latest header. Debug output improvements.
2019-09-27 21:46:24 -07:00
61054492b6
Bug fix in the design of the COWPager commit sequence. Page 1 is now used to store a copy of the previous committed header rather than the new one, as recovering to an unsync'd new header from Page 1 is incorrect behavior since other pager writes may not have made it to disk. Also fixed header page size handling which would write unusable backup headers when using >4k pages.
2019-09-27 21:46:24 -07:00
ca11845934
Debug output tweaks.
2019-09-27 21:46:24 -07:00
65ddae1373
Bug fix, ObjectCache could evict the object it just added and then return an invalid reference to it.
2019-09-27 21:46:24 -07:00
95c8004049
Bug fixes. COWPager header recovery was using the wrong checksum input and did not work for physical page sizes other than 4k.
2019-09-27 21:46:24 -07:00
8c0b9b5111
COWPager now uses Page 1 as a write-ahead copy of the header which is written and sync'd before modifying Page 0.
2019-09-27 21:46:24 -07:00
af14bfc255
Changed COWPager page cache size argument to bytes instead of pages and changed initialization to use appropriate knobs in simulation.
2019-09-27 21:46:24 -07:00
537b8dc7ac
Bug fix, COWPager failed to reopen a created but unsync'd pager file. Added proper checksum error handling.
2019-09-27 21:46:24 -07:00
57f55c1e99
Bug fix - FIFOQueue design changed to not rely on the durability of unchanging bytes in modified pages that are not fsync'd.
2019-09-27 21:46:23 -07:00
e0873e2ba0
Removed COWPager snapshot lifetime management for now as it's the wrong strategy and causes crashes when snapshot references outlive the pager.
2019-09-27 21:46:23 -07:00
abc22d2610
COWPager bug fixes involving shut down while operations are in progress.
2019-09-27 21:46:23 -07:00
046dd76d73
Cleaned up the Redwood BTree correctness test, tweaked the parameters to favor shorter tests and added a time limit.
2019-08-08 23:08:08 -07:00
f81eeea495
Bug fixes. COWPager initialization was not flushing non-header pages with fsync() before writing and syncing the header. FIFOQueue was writing the initial page of a new queue multiple times. FIFOQueue::writePage() would unnecessarily (and invalidly) attempt to write if the page is not yet loaded.
2019-08-08 02:57:23 -07:00
bd8ed07f4d
Missing header.
2019-08-07 04:31:11 -07:00
10be4b8127
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-08-07 04:20:14 -07:00
a04e3cce23
Bug fixes in COWPager cache. Write while a read is outstanding is supported, the new write will wait on the prior read. New writes also wait on old writes. Cache entries no longer cancel in-progress operations when evicted.
2019-08-07 04:11:33 -07:00
Alex Miller