ebc96a7e0e
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-05-21 23:49:27 -07:00
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
b6008558d3
renamed BinaryWriter.toStringRef() to .toValue(), because the function now returns a Standalone<StringRef>()
...
eliminated an unnecessary copy from the proxy commit path
eliminated an unnecessary copy from buffered peek cursor
2019-03-28 11:52:50 -07:00
126252a274
Changed checksum to crc32. Disabled pager housekeeping for now. Added more btree read/write/commit metrics. Changed readPage to use disk read priority. Bug fix in CommitSubtree causing it to recurse to children unnecessarily. Added point read speed test at the end of set performance unit test.
2019-03-15 00:46:09 -07:00
6660ccd936
Large rewrite/refactor from the page format outward. Replaced the in-page format, PrefixTree, with more general and templated implementation of the same concept, DeltaTree, which will be incrementally modifiable soon. New implementation for in-page cursors to use a per-page arena and share decoded binary tree nodes between cursors in the same page. New implementation for internal record B+Tree cursor which uses reference counted path hops which makes cloning a cursor very cheap. New implementation of external facing B+Tree cursor which makes efficient use of two internal cursors to find user-visible KV pairs and return them without copying keys and values except in the case of a large sharded value.
2019-02-21 02:46:30 -08:00
9d73166b3b
Many bug fixes related to concurrent page operations and pager shutdown.
2018-11-06 19:31:16 -08:00
df3bdde50b
Many bug fixes. AsyncFileCached write() on a page with a zero-copy read in progress would orphan the old page before the read was finished. Pager file operations were not converting page id to int64 for byte offset calculation. Pager was not calling releaseZeroCopy() after readZeroCopy() if there was an error or cancellation. Pager reads were using some variables that could go out of scope. BusyPage's mechanism for notifying when a physical page is no longer in use is itself no longer in use and therefore removed. Pager shutdown now cancels all outstanding reads. Improved some debug output.
2018-10-31 02:14:55 -07:00
342466817a
Added pagefile name to debug output. Shutdown will no longer throw if actor collection or delete futures have errors.
2018-10-25 04:00:02 -07:00
f17cc1e20f
StorageServer will no longer send io_error or other inappropriate errors to a client (this would never happen on SQLite). Many bug fixes around error handling, initialization, and shutdown in Redwood. StorageServer now calls init() on its underlying storage engine.
2018-10-24 15:57:06 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
8dd900a337
fixed the windows build
2018-10-18 20:26:45 -07:00
22f8a4efa9
Normalized all unit test names to begin with "/" if they should be included in random unit testing.
2018-10-05 22:09:58 -07:00
5d9cd9acdc
Correctness test now has additional random reader which doesn't do verification but isn't stopped when the btree is closed. Fixed bug exposed by this where pager snapshots will still try to read pages after the pager has been shut down or even destroyed. Added new error type, shutdown_in_progress.
2018-10-04 23:46:37 -07:00
c5420218d0
Missed changes from previous checkin which stops freeing physical pages after a latest version logical page rewrite.
2018-10-03 12:10:07 -07:00
4ad44e72dd
Workaround for issue where calling sync on a cached file that doesn't exist yet causes an io_error after a simulated machine kill. Temporary disabled freeing physical pages on logical page replacement because it is a bug as currently implemented, it will reuse physical pages before the remap of its old logical page to a new physical page is durable.
2018-10-02 17:00:35 -07:00
15c1f87681
Rewrite superpage writes to fix bugs involving logical ID reuse and to make the logic cleaner. Correctness test is stronger, with pipelined writes/commits during verification.
2018-10-01 16:51:57 -07:00
d7d2b58fef
Increased several testing parameters which exposed new bugs. Bug fix, PrefixTree's current format does not support trees larger than 64k. Cleaned up some debug output.
2018-09-24 02:42:23 -07:00
2fc86c5ff3
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbrpc/AsyncFileCached.actor.h
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-09-20 03:39:55 -07:00
7c40ff831d
Bug fix, when a reference page version is needed for a page update the write must be converted to a normal update (not a replacement of the latest version).
2018-09-19 02:38:45 -07:00
ca9eb228bc
Bug fix, page write at version based on a reference page's latest didn't work if latest version of the page being written was already at that version.
2018-09-19 01:34:19 -07:00
4df093f1ea
Implemented large key support by adding a superpage concept which is (currently) a BTree-level construct that treats multiple pages from the pager as a single contiguous page, splitting it up on write and reconstituting it on read. Refactored how pages are written along the way. The superpage construct will later be replaced by variable sized page support in IPager.
2018-09-19 00:32:39 -07:00
540d88e85f
Minimal page boundary keys are now being used, though there are still bugs. Many debug output changes. Bug fix regarding common prefix length calculation which was reducing compression and causing incorrect boundary keys.
2018-07-17 00:41:42 -07:00
af9aa43f39
Tree-level prefix compression partly done. Btree page writing and reading now passes parent key lower bound to the PrefixTree builder / cursor so it can be used as a prefix.
2018-07-10 02:24:01 -07:00
7f5da0a59e
Bug fix, don't log error when cancelled.
2018-07-05 22:30:17 -07:00
5936113cf4
Bug fix in shutdown. Normalized pager test filenames.
2018-07-04 21:12:09 -07:00
09e68a4335
Lots of bug fixes around page reads and concurrency.
2018-07-03 15:39:32 -07:00
77df55d9e1
Bug fixes in page building causing memory corruption, improved debugging output, added memory initialization to new pages when valgrind is enabled to avoid unconcerning errors.
2018-06-14 04:15:14 -07:00
b15d29dd6b
VersionedBTrees now have names for logging purposes (trees do not deal with files directly but rather just an IPager). Bug fix in pager vacuumer which was not updating checksums. KeyValueStoreRedwood now relies on VersionedBTree to handle commit ordering, as there’s no reason to do this in both classes.
2017-10-09 13:24:16 -07:00
c0d1097bb4
Lots of debug output improvements on pager, fixed bug in file naming scheme for page file and log files.
2017-10-03 11:55:14 -07:00
0635a31604
Better error and dispose/close handling, closer to correct behavior for an IKeyValueStore.
2017-10-02 03:32:22 -07:00
c3e4364f08
Added checksums to pager.
2017-09-26 23:13:22 -07:00
248dab79b6
Created “redwood” storage engine option and many changes to support that including IKeyValueStore::init() and custom DiskQueue file extensions.
2017-09-21 23:51:55 -07:00
d880569d52
Checkpointing progress on KeyValueStoreMVBTree. All methods are implemented to a usable point, and everything compiles, but Worker does not yet try to use it.
2017-09-21 04:43:49 -07:00
bcc8a2cc82
Bug fix, page reference was not living as long as the file write call so the memory could be freed before the write happens if there is any delay.
2017-09-20 17:51:26 -07:00
125d8168b4
Checkpointing progress on range reads (ordered iteration of user visible kv pairs via cursor). Added InternalCursor class which is used for all seeks and reads and sees the multi version, fragmented kv pairs and clears. Fixed a bug in commit which was discovered by the range read test where kv pairs (full or partial) could be missing from the tree but only for versions where they did not change. The write verification test did not find this because it only verifies exactly the changed versions of each key. The range test is not finished yet.
2017-09-15 05:19:39 -07:00
919a3d1740
Iterating over and coalescing the “internal” records (not user visible, tuple of (key, version, valueIndex, value)) now works forward and backward, and findEqual() now works using this to read split values fully. This is also most of the work needed for range reads.
2017-09-09 01:29:25 -07:00
fc0557252b
Bug fixes. Btree now uses page 0 as root and will write initial page only if necessary. Added debug printf macro.
2017-07-14 11:36:49 -07:00
b65ad3563c
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2017-06-09 14:56:41 -07:00
Stephen Atherton