Stephen Atherton
|
a9f467c502
|
Btree correctness verification of contents is now asynchronous.
|
2018-09-28 00:35:03 -07:00 |
|
Stephen Atherton
|
5c3a93e41f
|
Bug fixes involving PrefixTree length encodings and page size limits, correctness test improvements such as better key size selection and random key reuse, reduced memory footprint of control set.
|
2018-09-27 16:07:29 -07:00 |
|
Stephen Atherton
|
d7d2b58fef
|
Increased several testing parameters which exposed new bugs. Bug fix, PrefixTree's current format does not support trees larger than 64k. Cleaned up some debug output.
|
2018-09-24 02:42:23 -07:00 |
|
Stephen Atherton
|
2fc86c5ff3
|
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
# Conflicts:
# fdbrpc/AsyncFileCached.actor.h
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
|
2018-09-20 03:39:55 -07:00 |
|
Stephen Atherton
|
90d9ef87d7
|
Changed simplePagerTest prints to use debug_printf() for output which is not reporting an error.
|
2018-09-19 19:16:18 -07:00 |
|
Stephen Atherton
|
6f84b6cee9
|
Bug fix, allocated superpage size was wrong. Also added initialization for valgrind build to prevent uninitialized read errors during page checksum calculation.
|
2018-09-19 18:54:50 -07:00 |
|
Stephen Atherton
|
ca9eb228bc
|
Bug fix, page write at version based on a reference page's latest didn't work if latest version of the page being written was already at that version.
|
2018-09-19 01:34:19 -07:00 |
|
Stephen Atherton
|
4df093f1ea
|
Implemented large key support by adding a superpage concept which is (currently) a BTree-level construct that treats multiple pages from the pager as a single contiguous page, splitting it up on write and reconstituting it on read. Refactored how pages are written along the way. The superpage construct will later be replaced by variable sized page support in IPager.
|
2018-09-19 00:32:39 -07:00 |
|
Stephen Atherton
|
383cac4991
|
This checkin is an attempt to support long keys by placing oversize key suffixes (as much as will fit) into a page and forcing the prefix bytes that do not fit to be part of the parent's page boundary leading to this page. This strategy is being abandoned but there are some valuable refactorings and debugging enhancements that are still in the right direction so they are being committed.
|
2018-08-28 13:46:14 -07:00 |
|
Stephen Atherton
|
2e651f02d3
|
Crash fixes related to initialization and memory ownership in cursor classes.
|
2018-07-25 02:29:17 -07:00 |
|
Stephen Atherton
|
4edcd8b0f0
|
Bug fixes.
|
2018-07-23 03:49:40 -07:00 |
|
Stephen Atherton
|
9d391498e8
|
Refactored how key/value memory is held in PrefixTree and VersionedBtree, eliminated many unnecessarily copies of large strings. PrefixTree tests pass but btree is still broken, just committing this because it's a large change set.
|
2018-07-23 03:09:13 -07:00 |
|
Stephen Atherton
|
7397ea4a79
|
Bug fixes. Memory corruption was happening due to memory ownership problems with using PrefixTree cursors. Now that internal page boundaries are truncated to the minimal necessary length they can be incomplete tuples of (key, version, offset) which was not being handled correctly (causing incorrect mutation buffer to page mappings during commit) for some partial tuple values. Tuple was modified to allow optional safe decoding of partial integer values and to assert if partial integer values are found but not allowed. Lots of debug output changes.
|
2018-07-18 03:19:35 -07:00 |
|
Stephen Atherton
|
540d88e85f
|
Minimal page boundary keys are now being used, though there are still bugs. Many debug output changes. Bug fix regarding common prefix length calculation which was reducing compression and causing incorrect boundary keys.
|
2018-07-17 00:41:42 -07:00 |
|
Stephen Atherton
|
bbf5ece5af
|
Btree pages can now borrow prefix bytes from their upper bound key (the next key after their lower bound key in their parent page) as well as their lower bound key.
|
2018-07-14 13:37:52 -07:00 |
|
Stephen Atherton
|
af9aa43f39
|
Tree-level prefix compression partly done. Btree page writing and reading now passes parent key lower bound to the PrefixTree builder / cursor so it can be used as a prefix.
|
2018-07-10 02:24:01 -07:00 |
|
Stephen Atherton
|
2cb0362102
|
AsyncFileCached now allows writing and truncation of whole pages previously read using readZeroCopy and not yet released without prior readers seeing the effects of the write.
|
2018-07-05 02:59:13 -07:00 |
|
Stephen Atherton
|
5936113cf4
|
Bug fix in shutdown. Normalized pager test filenames.
|
2018-07-04 21:12:09 -07:00 |
|
Stephen Atherton
|
f55e952307
|
Disabled memory pager unit test as it is not currently being maintained. Bug fix in unversioned IKeyValueStore shutdown.
|
2018-07-04 05:51:01 -07:00 |
|
Stephen Atherton
|
09e68a4335
|
Lots of bug fixes around page reads and concurrency.
|
2018-07-03 15:39:32 -07:00 |
|
Stephen Atherton
|
e4f347d3fd
|
Bug fix: PrefixTree::PathEntry was not safely copyable or swappable which caused a subtle bug at the higher level causing a Cursor state to appear valid but not make sense. More debug output improvements.
|
2018-06-17 06:48:41 -07:00 |
|
Stephen Atherton
|
ee17854713
|
Bug fixes in internal page boundaries and prefix tree overhead calculations. Improved performance test output and increased amount of work it can do. Removed temporary restrictions on correctness test parameter ranges.
|
2018-06-14 17:52:25 -07:00 |
|
Stephen Atherton
|
77df55d9e1
|
Bug fixes in page building causing memory corruption, improved debugging output, added memory initialization to new pages when valgrind is enabled to avoid unconcerning errors.
|
2018-06-14 04:15:14 -07:00 |
|
Stephen Atherton
|
d291cc9ec3
|
Bug fixes in cursor logic related to database and page ends. Rewrote cursor related toString() methods and debug output to make debugging easier.
|
2018-06-12 01:43:19 -07:00 |
|
Stephen Atherton
|
69b713918b
|
VersionedBTree now uses PrefixTree based pages (with bugs). This required significant changes to both classes because the interface and semantics for building, seeking in, and iterating through pages is very different from the previous trivial approach which was based on serialized vectors. PrefixTree node format rewritten to support optional values without increasing overhead for common node scenarios. PrefixTree::Cursor rewritten to reuse path prefix memory instead of allocating new memory on each movement which is then 'leaked' until destruction. PrefixTree::Cursor movement modified to work better with VersionedBTree::InternalCursor, which was also heavily modified. Added knobs related to key arrangement in PrefixTree nodes. Added StringRef::toHexString() as an alternative to printable() to make reading raw PrefixTree data easier. PrefixTree performance is temporarily worse with this update and VersionedBtree fails its unit test.
|
2018-06-08 03:32:34 -07:00 |
|
Stephen Atherton
|
4a58ed3b11
|
Debug output improvements, removed unused function.
|
2017-10-09 13:31:54 -07:00 |
|
Stephen Atherton
|
b15d29dd6b
|
VersionedBTrees now have names for logging purposes (trees do not deal with files directly but rather just an IPager). Bug fix in pager vacuumer which was not updating checksums. KeyValueStoreRedwood now relies on VersionedBTree to handle commit ordering, as there’s no reason to do this in both classes.
|
2017-10-09 13:24:16 -07:00 |
|
Stephen Atherton
|
0635a31604
|
Better error and dispose/close handling, closer to correct behavior for an IKeyValueStore.
|
2017-10-02 03:32:22 -07:00 |
|
Stephen Atherton
|
ed226e2e1c
|
VersionedBtree now allows writes while a commit is in progress. Also, lots of bug fixes, like the one where writes during commits weren’t prevented but also didn’t work.
|
2017-09-22 17:18:28 -07:00 |
|
Stephen Atherton
|
248dab79b6
|
Created “redwood” storage engine option and many changes to support that including IKeyValueStore::init() and custom DiskQueue file extensions.
|
2017-09-21 23:51:55 -07:00 |
|
Stephen Atherton
|
d880569d52
|
Checkpointing progress on KeyValueStoreMVBTree. All methods are implemented to a usable point, and everything compiles, but Worker does not yet try to use it.
|
2017-09-21 04:43:49 -07:00 |
|
Stephen Atherton
|
e16ce63db0
|
Bug fix, find was being done with a key ref that wasn’t living as long as the actor.
|
2017-09-21 00:58:56 -07:00 |
|
Stephen Atherton
|
eee8116fa4
|
Cleaned up how the root page is written and page write debug output.
|
2017-09-20 17:50:02 -07:00 |
|
Stephen Atherton
|
9b4cbe94aa
|
Bug fix, values exactly the size of the maximum part size were still split into a part set but of size 1, which is invalid. Also some debug output changes.
|
2017-09-19 13:03:30 -07:00 |
|
Stephen Atherton
|
25f958ac3b
|
Bug fixes. Finished backward iteration on Cursor. Correctness range verification now also reads the range in reverse and verifies the result.
|
2017-09-17 04:38:01 -07:00 |
|
Stephen Atherton
|
1aae32dc16
|
Some cleanup. InternalCursor does not need to be reference counted.
|
2017-09-16 02:09:09 -07:00 |
|
Stephen Atherton
|
c89b45158a
|
Removed InternalCursor::seekEqualOrGreaterThan because it isn’t needed, it’s always better to do a <= search so if the target key is found the initial cursor position will be at the last version prior to the target version.
|
2017-09-16 01:59:16 -07:00 |
|
Stephen Atherton
|
cea7903b46
|
Bug fixes involving edges cases with find >= and range read endpoints.
|
2017-09-16 01:45:39 -07:00 |
|
Stephen Atherton
|
15622f026e
|
Forward range reads are done.
|
2017-09-15 17:27:13 -07:00 |
|
Stephen Atherton
|
125d8168b4
|
Checkpointing progress on range reads (ordered iteration of user visible kv pairs via cursor). Added InternalCursor class which is used for all seeks and reads and sees the multi version, fragmented kv pairs and clears. Fixed a bug in commit which was discovered by the range read test where kv pairs (full or partial) could be missing from the tree but only for versions where they did not change. The write verification test did not find this because it only verifies exactly the changed versions of each key. The range test is not finished yet.
|
2017-09-15 05:19:39 -07:00 |
|
Stephen Atherton
|
919a3d1740
|
Iterating over and coalescing the “internal” records (not user visible, tuple of (key, version, valueIndex, value)) now works forward and backward, and findEqual() now works using this to read split values fully. This is also most of the work needed for range reads.
|
2017-09-09 01:29:25 -07:00 |
|
Stephen Atherton
|
b5f79d27f7
|
Large values are now split and stored, but not yet read correctly.
|
2017-09-05 16:59:31 -07:00 |
|
Stephen Atherton
|
9745334704
|
Changed random parameter range so that at least two internal keys can always fit in a single page.
|
2017-09-01 00:17:07 -07:00 |
|
Stephen Atherton
|
88e48e9a3e
|
Bug fix in detection of lack of mutations in a subtree.
|
2017-08-31 01:23:12 -07:00 |
|
Stephen Atherton
|
5049421a0e
|
More comments, debug output improvements.
|
2017-08-28 17:26:53 -07:00 |
|
Stephen Atherton
|
14ef5bed42
|
Major bug fix. In certain situations, some mutations were being applied to two adjacent pages, which had a lot of crazy side effects.
|
2017-08-28 06:28:49 -07:00 |
|
Stephen Atherton
|
d9136743f7
|
Bug fix, reformatted and cleaned up a lot of debug output.
|
2017-08-28 03:53:29 -07:00 |
|
Stephen Atherton
|
0fd57e9183
|
Many bug fixes.
|
2017-08-28 01:57:01 -07:00 |
|
Stephen Atherton
|
1070a5ed66
|
Several bug fixes.
|
2017-08-25 15:48:32 -07:00 |
|
Stephen Atherton
|
888093463b
|
Checkpointing progress on large rewrite of how mutations are stored and applied. Not working yet.
|
2017-08-24 17:25:53 -07:00 |
|
Stephen Atherton
|
0b817f95f2
|
Bug fixes, but there’s still an issue with crossing page boundaries at the same key.
|
2017-08-22 11:30:44 -07:00 |
|
Stephen Atherton
|
aff31b7f36
|
Checkpointing. Clears almost work, but two edge cases aren’t handled correctly yet involving range clears that must cross leaf page boundaries.
|
2017-08-21 22:29:57 -07:00 |
|
Stephen Atherton
|
de03491475
|
Switched to new commit buffer type which can support more than set, but so far only set is implemented.
|
2017-08-04 00:01:25 -07:00 |
|
Stephen Atherton
|
dbcd3526d7
|
Memory lifetime bug fix and previously missed changes for enforcing only reading committed versions.
|
2017-08-03 15:07:29 -07:00 |
|
Stephen Atherton
|
31314d06d4
|
Added test stats. Added enforcement of only reading committed versions, which is a currently an implementation limitation.
|
2017-07-25 16:10:19 -07:00 |
|
Stephen Atherton
|
d61f74e52b
|
Debug output changes.
|
2017-07-14 13:39:58 -07:00 |
|
Stephen Atherton
|
8b3569e27e
|
Bug fix.
|
2017-07-14 11:37:08 -07:00 |
|
Stephen Atherton
|
fc0557252b
|
Bug fixes. Btree now uses page 0 as root and will write initial page only if necessary. Added debug printf macro.
|
2017-07-14 11:36:49 -07:00 |
|
Stephen Atherton
|
d8b82ecbe4
|
Added use of IndirectShadowPager. Moved MemoryPager code into .cpp because of type conflicts and its implementation doesn’t need to be externally visible anyway. Added start of a performance test. Renamed tests. Correctness/set now has random reopen of disk based pager before verification. Added asserts for when keys or values that are too large to fit in a single page.
|
2017-07-13 22:11:48 -07:00 |
|
Stephen Atherton
|
b56f5643d5
|
Bug fixes.
|
2017-07-13 14:51:39 -07:00 |
|
Stephen Atherton
|
bea061fd75
|
Bug fix, building tree root (1 or more top levels) must happen inside commitSubtree() otherwise the root page logical ID (1) can be written out of version order.
|
2017-07-13 11:32:14 -07:00 |
|
Stephen Atherton
|
c508e8bdf9
|
Adjacent internal btree pages can now contain the same keys at different versions.
|
2017-07-04 23:49:18 -07:00 |
|
Stephen Atherton
|
1a71df1871
|
Lots of bug fixes and debug output added. Unitttest for set works…pretty often.
|
2017-07-04 23:41:48 -07:00 |
|
Stephen Atherton
|
b65ad3563c
|
Merge branch 'master' into feature-redwood
# Conflicts:
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
|
2017-06-09 14:56:41 -07:00 |