b4e42476b7
Unit test parameters are no longer global, they are accessible via a parameter to the unit test and initialized from otherwise unconsumed test options for the UnitTests workload in the test spec or from the fdbserver command line when using the unittests role.
2021-04-06 02:36:10 -07:00
f8711cf3c2
Merge commit 'e7573d546f094bd96e1b2fca0914c6e8d61c77e4' into redwood-improvements
2021-04-05 05:42:25 -07:00
e7573d546f
Some unit tests names had a prefixed "!" in order to be excluded from random selection, this has been changed to a ":" as it is less problematic on the command line. Some Redwood unit tests have been enabled for random selection.
2021-04-05 00:03:15 -07:00
ffeb94ada4
Updated Redwood set unit test to use unit test parameters.
2021-04-04 23:33:04 -07:00
5c93e684f8
Added comments.
2021-04-04 19:23:08 -07:00
90ebf90c8b
Refactored page rebuild logic to bulk build pages full and split pages more evenly.
2021-04-03 19:54:49 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
7d804f7bee
Setting stage to add support for super pages for Redwood FIFO queue.
2021-03-04 10:24:48 -08:00
79cec09255
Apply clang-tidy's performance-inefficient-vector-operation fix
...
I ran this command in my build directory after compiling with
OPEN_FOR_IDE. It took a few small tweaks to get it to compile, which is
outside the scope of this commit.
$ python run-clang-tidy.py -j $(nproc) -checks='-*,performance-inefficient-vector-operation' -fix
2021-03-04 03:58:25 +00:00
4ee97c0784
Use clang-tidy to automatically fix missing overrides
...
Use `clang-tidy -p . $file -checks='-*,modernize-use-override' -header-filter='.*' -fix`
to fix missing overrides, and then use git clang-format to reformat just
those changes. This went pretty well for most files.
Formatting the following files went off the rails, so I'm going to
follow up with a commit that's just clang-tidy and no clang-format.
- fdbclient/DatabaseBackupAgent.actor.cpp
- fdbclient/FileBackupAgent.actor.cpp
- fdbserver/OldTLogServer_4_6.actor.cpp
- fdbmonitor/SimpleIni.h
- fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
2021-01-26 02:04:12 +00:00
ff7d306b09
Merge branch 'release-6.3' into anoyes/merge-6.3-to-master
...
Include conflict markers for now. Will resolve.
2021-01-15 18:04:09 +00:00
2ce967bd77
Merge branch 'release-6.3' of github.com:apple/foundationdb into feature-redwood
2021-01-05 21:34:26 -08:00
4c28341351
Performance bug fix: Every page cache hit was calling forwardError() unnecessarily.
2021-01-05 21:34:16 -08:00
d7ebae07f5
Added yield to remap cleanup loop to prevent a stack overflow if too many queue entries are processed without waiting on IO.
2020-12-12 02:40:06 -08:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
73e77f31eb
Merge pull request #3928 from yliucode/merge
...
Merge release-6.3 into master
2020-10-20 15:48:46 -07:00
8cc3e4d3c6
Merge release-6.3 into master
2020-10-19 22:51:56 -07:00
410d418c3e
Revert "Merge pull request #3770 from dongxinEric/feature/ss-commit-pipelining"
...
This reverts commit e258dffc650e96233015
2020-10-19 09:28:38 -07:00
0b46af2925
Added a simulation-only check for pager remap cleanup writing the same page twice in a cleanup cycle, which should never happen, but a bug leading to this was fixed recently. Adjusted some buggify logic to widen edge case coverage around remap cleanup parameters.
2020-10-14 21:48:03 -07:00
dc35f2b4f5
Bug fix: In page remap cleanup, if a page update's next update was at exactly the oldest retained version then the earlier update would still choose to copy the updated page over top of the original (but it shouldn't) which would race with the later update's copy if the same remap cleanup cycle pops both updates from the queue.
2020-10-13 02:26:46 -07:00
dc4612712d
Merge remote-tracking branch 'origin/master' into use-override-more
2020-10-09 07:12:41 -07:00
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
8571dcfe28
Use override where applicable in fdbserver
2020-10-07 18:41:19 -07:00
34d1d04904
Commit pipelining attempt #2
2020-09-23 20:03:04 -07:00
12edadd059
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# fdbclient/Knobs.cpp
# fdbclient/MasterProxyInterface.h
# fdbrpc/simulator.h
# fdbserver/MasterProxyServer.actor.cpp
# tests/fast/CycleAndLock.txt
# tests/fast/TxnStateStoreCycleTest.txt
# tests/fast/VersionStamp.txt
# tests/slow/ParallelRestoreOldBackupApiCorrectnessAtomicRestore.txt
# tests/slow/ParallelRestoreOldBackupCorrectnessCycle.txt
# versions.target
2020-08-31 19:33:34 -07:00
89ae0aaf39
In Redwood unit test, if the verify actor falls far behind the latest commit version it will unnecessarily verify the latest version repeatedly which extends the test's run time needlessly.
2020-08-29 23:47:54 -07:00
167d3c5a8f
To avoid running too long, the Redwood correctness unit test now limits the size of the KV verification map. This map only grows as the test runs, as it contains a record for every key set or clear committed during the test.
2020-08-29 16:44:38 -07:00
e1b7dd0c7d
Merge remote-tracking branch 'upstream/release-6.3' into dyoungworth/fixMerge1
2020-08-22 12:25:19 -07:00
d53e6cf687
Bug fix in KV byte counts.
2020-08-18 17:58:57 -07:00
090241f0df
Count BTreeCursor page reads that come from the cursor's cache of pages as cache hits in RedwoodMetrics. BTree Unit Test now verifies all data at the initial version after recovery from disk.
2020-08-15 14:30:49 -07:00
051ec7066a
BTree unit test duration now targets a number of page read/writes so it should time out less often.
2020-08-13 01:50:51 -07:00
a817c9abbe
Redwood metrics was adding details with empty names.
2020-08-12 03:24:23 -07:00
c1ccbc3d27
Bug fixes. Internal page update would be missed if child page ID updates were expected but none were done. When rebuilding internal pages, child page ID updates were being written to the original page memory instead of the rebuild record set.
2020-08-12 01:33:07 -07:00
c18d16ad30
Multiple sibling page updates within the remap cleanup window can now avoid the second update step, copying updated contents over the original page, by instead updating the parent page to point to the new sibling locations.
2020-08-09 03:04:38 -07:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
5268dbe1ec
Use knob for remap cleanup window in write speed test.
2020-07-21 00:42:08 -07:00
d05b7ee785
Pager remap remover now accumulates a configurable version interval of page updates behind the oldest retained version which are used to coalesce updates to the same original page ID to reduce write amplification for many workloads.
2020-07-20 04:08:33 -07:00
f470ba8316
Remove using namespace std::rel_ops
...
This causes the following to not compile anymore
\#include <utility>
\#include <vector>
using namespace std::rel_ops;
int main() {
std::vector<int> xs;
return xs.rbegin() != xs.rend();
}
See https://godbolt.org/z/s1977n
2020-07-10 22:58:15 +00:00
1b68665228
Merge branch 'master' into release-6.3
2020-07-08 18:52:05 -07:00
1b55d75896
Remove TRIVIALLY_DESTRUCTIBLE macro
2020-07-04 19:28:10 -07:00
22f7f804b8
Merge branch 'release-6.3' into mengxu/merge-6.3-PR
2020-06-28 11:19:39 -07:00
8de70432a6
Fix memory leaks in VectorRef
...
VectorRef<T> relies on T being trivially destructible, but this was not
yet enforced. By statically asserting that T is trivially destructible,
we avoid leaking memory by not calling the destructor for non-trivially
destructible types
2020-06-24 10:54:19 -07:00
4216112e78
Fixed getWriteVersion and getLastCommittedVersion signatures
2020-06-17 22:38:54 -07:00
77e3b314f6
Mark IKeyValueStore::getStorageBytes const
2020-06-17 18:20:43 -07:00
2a393633b6
Improved const-correctness
...
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185 ), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:
- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
2020-06-17 18:20:43 -07:00
43d23bccb0
Applied clang-format.
2020-06-16 17:24:32 -07:00
ad68e44419
Change Redwood read concurrency lock to a reference because it must be able to live longer than the storage engine itself due to destruction order of actors if the storage engine is shut down while read operations are in progress.
2020-06-16 03:38:51 -07:00
9888fa7a10
Use a reference for m_commitReadLock to avoid invalid destruction.
2020-06-15 14:46:16 -07:00
c3c7db6a9d
Added flowlock around reads done on the write path so that those reads cannot starve reads to support reads done by the storage server.
2020-06-09 17:00:21 -07:00
8fdb81b48d
Tweaked BTree test random parameter limits to avoid test runs which take too long.
2020-06-06 21:10:52 -07:00
a28b5f0a8b
Possible bug fix, flow locks should be taken after initializing BTree cursor in KVStoreRedwood read functions otherwise it might be possible for the BTree to be closed before the flow lock wait returns, depending on destruction order of some things.
2020-06-06 21:10:13 -07:00
949fbd9145
New low level BTree cursor class which is designed for short lifetimes, does less memory allocation, and can be used to perform getRange() operations with less CPU overhead.
2020-06-04 23:23:14 -07:00
91f3b1c476
Merge pull request #3160 from satherton/feature-redwood
...
Update Redwood.
2020-05-20 15:04:01 -07:00
8c3cddc792
Metrics bug fix, page build count was being updated in an incorrect place.
2020-05-19 12:45:32 -07:00
53c4500694
Applied clang-format.
2020-05-19 04:11:39 -07:00
0d6f226938
Renamed some metrics to be clearer. Instead of "write" and "update" the terms are now "build" and "modify" which are less confusing and better represent what they mean in terms of DeltaTree operations.
2020-05-19 03:31:03 -07:00
7765168346
Redwood metrics now includes average item count, fill percentage, and record stored percentage for pages written or modified during the metrics interval.
2020-05-19 03:04:39 -07:00
f502236c31
Bug fix: BTree page header kvByte count was not being updated when pages are modified without rebuild.
2020-05-18 01:34:57 -07:00
ca8464881f
Applied clang-format.
2020-05-17 00:16:35 -07:00
d11fd75105
Bug fixes: No-hit page reads would count the access as a hit but would still move the page to the back of the eviction order. Page writes to uncached pages are no longer considered cache misses.
2020-05-17 00:16:10 -07:00
6347f76192
More Redwood metrics work. Reworked metrics standard output for unit tests.
2020-05-16 22:21:55 -07:00
e8626724f9
More refactor of Redwood metrics. Added per-level BTree stats. Added Pager total physical disk reads/writes, and uncacheable read hit/miss counts. Added logging to TraceEvents.
2020-05-16 02:51:57 -07:00
7cc8e281f2
Accidental override on non inherited function, which strangely only broke the Windows build.
2020-05-15 06:47:24 -07:00
0538edf7e7
Lazy / deferred subtree deletion had a few different names in the code, normalized everything to LazyClear. Removed accidental printfs.
2020-05-15 02:15:02 -07:00
32f4639168
Refactored page remap removal. The process is now called remap cleanup, does batches of remap entries in parallel, coalesces remaps of the same page ID to skip unnecessary writes, and has some knobs for controlling it. FIFOQueue now has peek() to support remap cleanup version lag limits. Added counters for remap cleanup and lazy subtree deletion. Refactored Redwood counters, normalized and grouped their names.
2020-05-15 02:10:51 -07:00
f51c1b352e
Applied clang-format.
2020-05-14 01:57:54 -07:00
7d84e1b7af
Eliminate undefined behaviour
2020-05-13 19:59:34 -07:00
421a6581c1
Lazy delete cycles now run more often and not just when the btree is having mutations actively merged into it. Specifically, it is launched upon recovery, then stopped and relaunched after every commit. This will make better use of I/O in between calls to commit(). Also added knobs to control lazy delete cycle parallelism and work limits.
2020-05-13 02:27:03 -07:00
3aef977af4
Removed a duplicative mutation buffer lookup when merging mutation buffer into internal page.
2020-05-12 20:48:31 -07:00
43f9e4dfad
Implemented concurrent read limit in IKeyValueStore interface for Redwood. Added knobs for Redwood page size, concurrent read limit, and page fill factor. Changed commitSubtree() recursion back to use a vector and waitForAll() because it seems to be lower overhead than ActorCollection.
2020-05-10 16:13:22 -07:00
ffbed1d84c
Use actor collection for subtree recursions.
2020-05-10 11:44:07 -07:00
f306666e21
Added commonPrefixLength for StringRefs that takes a prefix skip length.
2020-05-10 04:24:11 -07:00
5bb5bd2fd5
Improvements to Redwood set benchmark to improve readability and limit commit sizes to a record count or byte limit.
2020-05-10 04:21:25 -07:00
2ed43ebdf7
Added a non-persistent, memory-only option to DWALPager.
2020-05-10 02:19:19 -07:00
e4006c4b63
Apply clang-format.
2020-05-09 02:21:26 -07:00
cec63f2429
Merge pull request #3101 from satherton/feature-redwood
...
Redwood Update - Internal page incremental updates and efficient cleared/unchanged subtree skipping
2020-05-08 19:53:42 -07:00
163b291785
CommitSubtree() now skips recursing into adjacent subtrees that are uniformly cleared or unchanged very efficiently. This will be a huge reduction in CPU usage for random workloads.
2020-05-08 15:24:40 -07:00
1966dbb576
Checkpointing large refactor of how commitSubtree() works regarding internal pages such that now they can be incrementally modified and detecting changes involves far fewer comparison and staging of records.
2020-05-08 07:11:51 -07:00
92e672e2f0
Fixed compiler errors
2020-05-04 10:40:31 -07:00
dda0993d16
Apply clang-format to Redwood source.
2020-04-24 14:12:40 -07:00
ab46405bf4
Improvement to an edge case where writePages() would unexpectedly return only 1 page but not attempt to reuse the existing BTreePageID for it.
2020-04-23 04:23:05 -07:00
c80d28ac7e
Added several assertions and removed some checks for situations in commitSubtree() which should no longer be possible due to recent changes. This is mainly to test assumptions being made in the commitSubtree() refactor in progress. Set simulation to always choose Redwood, will revert for PR.
2020-04-21 04:12:39 -07:00
549ce29bdb
VersionDelta size options are now (0, 4, 6, 8), having removed the 2-byte option as it is not very useful.
2020-04-09 19:12:24 -07:00
6e5e7eea8d
Fixed a comment.
2020-04-09 17:14:25 -07:00
8de2bbb2dc
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2020-04-09 09:39:49 -07:00
d4334256f1
Bug fix: BTree::writePages() was not accounting for the small amount of unpredictability in node overhead sizes which could lead to serializing a tree that doesn't fit into its destination page buffer. WritePages() now skips the prefix common to all records in the input set when calling deltaSize() to estimate resulting DeltaTree serialized sizes.
2020-04-08 20:18:41 -07:00
9e82788c28
Bug fix, BTree header wasn't large enough to hold the largest possible root page references.
2020-04-08 03:38:37 -07:00
d979200799
Bump format version.
2020-04-08 03:29:12 -07:00
31fd4cd35a
Improved RedwoodRecord encoding/decoding tests, which found bugs in the version field scheme. Simplified it to select between different fixed size native integer formats.
2020-04-08 03:25:04 -07:00
7a4817b8bb
Re-enable ART mutation buffer in Redwood.
2020-04-07 11:47:19 -07:00
d7ca823a8d
Refactored RedwoodRecordRef::Delta to fix at least one bug and stop using variable length integers with extension bits. Instead, two (recently available) header bits now indicate which of four struct definitions the Delta is using. The resulting code is a bit repetitive but also easier to read and 15% faster in the RedwoodRecord Delta speed test vs the old version.
2020-04-07 04:47:08 -07:00
ab6e8e9b09
Bug fix, expanded page size calculation when the small page limit is reached was not correct in a couple of ways.
2020-04-06 23:37:32 -07:00
a354d7a037
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2020-04-06 20:29:12 -07:00
9bf0b5e71f
RedwoodRecord simplified to remove chunked key-value pair concept. Cursor and InternalCursor updated to be correct (but not optimal) for new record definition. DeltaTree::seek* refactored to support more seek types and with minor code duplication. Lots of debug output improvements. Refactored how bulk pages are written and introduced fill factor percentage (though since internal pages are not yet incrementally modified this is temporarily worse for those).
2020-04-06 20:27:30 -07:00
28c7899b76
Disable ART mutation buffer in Redwood until windows build issue is fixed.
2020-04-03 13:04:32 -07:00
1ed425233a
DeltaTree now has two size modes, 2 byte integer offers for <= 65k and 4 byte integer offsets for larger.
2020-04-01 04:24:40 -07:00
b10c6dad41
Added benchmark to DeltaTree RedwoodRecordRef test that does random seeks using a pool of cursors with different mirrors to test the speed of having to decode DeltaTree nodes much more often.
2020-04-01 04:12:18 -07:00
98061c80bb
Applying initial feedback from Steve
...
Removing iterator inheritance, and hence adding conversion operator
Using typedef to decide which MB implementation to use
2020-03-31 09:28:33 +02:00
59eede009c
Adding ART-based MutationBuffer
2020-03-30 13:10:44 +02:00
087c6fa33d
Merge branch 'master' into feature-redwood
2020-02-26 12:25:04 -08:00
636c256606
Simplified return value of CommitSubtree to a single ChildLinkSetRef, which doesn't contain a version. Removed unnecessary record copying in several places in CommitSubtree.
2020-02-24 15:18:34 -08:00
72a549df14
Added basic mutation buffer insert/lookup test.
2020-02-24 00:31:09 -08:00
1e45bb2be3
Further refactoring of MutationBuffer to make upcoming ART implementation easier to plug in. Key and mutation for an iterator position are now accessed via member functions which gives the implementation an opportunity to decode or reconstitute keys on-demand.
2020-02-23 23:44:15 -08:00
eb7016a09a
Some more refactoring of MutationBuffer so it will be easier to have multiple implementations.
2020-02-23 22:48:43 -08:00
710bc3ecb1
Added optimization for single key clears to avoid an additional mutation buffer boundary key insertion. Made mutation buffer members private and created additional member functions to avoid needing access to the private members.
2020-02-23 22:24:16 -08:00
3618dff07b
Corrected statement in test output to reflect new logic.
2020-02-23 00:21:39 -08:00
078f85fea7
Bug fix. Pager could return an an oldest version that is no longer readable because it is being expired in the current commit cycle.
2020-02-23 00:15:35 -08:00
648040b070
Removed versioned records from the mutation buffer and commit path because support for this was incomplete and has high overhead costs when only exact committed versions are being read. The BTree could still contain versioned records in the future as the read cursors still support this, but some optimizations were added for when all internal records are at version 0 which is the case and has been for quite some time. Mutation buffer now stores keys and values in one arena per buffer.
2020-02-22 20:21:06 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
f1ec780b31
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
2020-02-21 17:43:11 -08:00
5a1884d054
Leaf page modifications are now made incrementally until the DeltaTree is out of space or the tree is too imbalanced, at which point it falls back to a linear merge and page(s) rebuild.
2020-02-21 16:26:44 -08:00
986e533c8c
Bug fix, DeltaTree was not updating numItems after erase / insert operations which only change the deleted flag of an existing equivalent entry.
2020-02-20 22:49:36 -08:00
893729e3a8
Bug fix, Redwood was not using direct IO where available.
2020-02-20 10:29:20 -08:00
0c7c815396
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/CMakeLists.txt
2020-02-12 16:12:57 -08:00
93e3e36d52
Changed RedwoodRecordRef::compare() to include value and updated VersionedBTree to adapt to this change. This fixes an (uncommitted) bug where DeltaTree inserts of a record matching a deleted record except for value would simply unhide the deleted record. For DeltaTree delete/insert sequences to work correctly compare() must only return 0 when the records are fully equivalent.
2020-02-12 01:18:35 -08:00
151f070809
Track failed evictions in ObjectCache.
2020-02-10 12:00:40 -08:00
9f4e39f1ef
DeltaTree now supports inserting a deleted item that is still present as a deleted node by simply clearing the node's deleted flag. Insertion of duplicates (determined by T::compare() == 0) is no longer allowed.
2020-02-07 01:00:11 -08:00
df2b0452b4
Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef.
2020-02-06 13:19:24 -08:00
9cf9c46668
DeltaTree now tracks internal used/free/deleted space, and insert will fail safely if a new item does not fit.
2020-02-05 15:30:04 -08:00
9dbb9e4cec
DeltaTree now supports node deletion by flagging nodes as deleted (a state which the T::Delta must be able to store and return). DeltaTree::Cursor will hide deleted nodes from view.
2020-02-04 01:22:27 -08:00
460c9b78d8
DeltaTree improvements. DecodedNodes store non-parent ancestor pointer which enables moveNext() and movePrev() operations to be O(1) from any leaf node. This also enables seekLessThanOrEqual to accept a hint in the form of a cursor with a position close to the target to avoid traversal from the root when the destination is close to the hint. DeltaTree T::compare() now must take a skipLen argument which is used by insert() and seek operations to skip over bytes known to be shared between the new item or query and every node in the tree.
2020-01-31 00:32:48 -08:00
da73164eda
Move crc32c from fdbrpc to flow
...
So that we can use it from a piece of flow code without breaking module
boundaries.
Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
2020-01-13 18:19:30 -08:00
560c1da805
Fix UBSAN report
...
/home/anoyes/workspace/foundationdb/fdbserver/VersionedBTree.actor.cpp:1606:10: runtime error: null pointer passed as argument 2, which is declared to never be null
2019-12-04 09:42:34 -08:00
04b2338e60
Added sequential insert speed test.
2019-11-23 00:40:58 -08:00
5f1644f293
DeltaTree::Reader is now DeltaTree::Mirror and supports insertion into a DeltaTree. DeltaTrees now support an item count, so BTreePage no longer has an item count, so the VersionedBTree format version has been bumped.
2019-11-23 00:09:11 -08:00
a9af2de1d2
Added option in redwood prefixed set test to use randomly generated prefixes in sorted order.
2019-11-20 15:55:59 -08:00
d91d744fd7
Typedefs to simplify ObjectCache a bit.
2019-11-20 03:20:23 -08:00
8d973ce762
Bug fix: Any time the least recently used ObjectCache entry is not evictable the effective size of the cache would grow by one entry for each read and will never shrink even once there are enough evictable pages to return the cache to its configured size. This has probably never actually happened because evictability of Redwood pages is currently based on having no pending IO, but it would be more of a problem if evictability were redefined to require a reference count of 1.
2019-11-19 23:45:13 -08:00
9e1e0d731d
Incremental subtree deletion now processes pages in parallel.
2019-11-18 02:34:37 -08:00
4e404e34e5
Added prefix size comparison test which generates records with a configurable prefix pattern and compares storage size between Redwood and the SQLite storage engine.
2019-11-17 17:53:42 -08:00
f841d14141
Bumped format versions, also simplified version scheme to a pager version and a btree version, removing per-page versions for queue and btree pages.
2019-11-11 16:28:21 -08:00
9227de5c20
Redwood correctness unit test was using wallclock based time limit which breaks determinism.
2019-11-11 15:13:58 -08:00
61558eea04
Implemented page preloading on BTree cursor seeks to enable hiding latency on soon-to-be-read sibling pages. Added random scans with various preload sizes to the set performance unit test. ObjectCache now tracks hits, misses, and pages which were preloaded but then never used prior to eviction. BTree pages no longer store flags because height is sufficient. Removed virtual specifier in classes not designed to be further inherited. Removed old prototype code (PrefixTree, IndirectShadowPager, MemoryPager) as some interface changes are incompatible and they are no longer worth maintaining.
2019-11-11 09:54:22 -08:00
2aa672cb59
When bulk building pages, make most of them full.
2019-11-07 15:52:23 -08:00
5d00d93665
Fixed errors found by valgrind involving incorrect page memory lifetimes for IO operations plus some false positives for partially used pages.
2019-11-07 13:57:09 -08:00
ea428b72f5
Bug fix, DWALPager must flush its page ID queues in order to get an accurate user page count.
2019-11-05 01:11:34 -08:00
0c95fef8aa
Bug fix in tree clear and size check where sometimes there could still be old versions of pages in use because not enough commits have passed for them to be rolled off and freed.
2019-11-04 12:59:57 -08:00
4e9c3164a8
COWPager can now internally remap page IDs by version and has been renamed to DWALPager. This causes the B+Tree to no longer have to rewrite all ancestors of an updated page. FIFOQueue now has a read-only cursor and a peekAll() method to read an entire queue without popping it. Fixed some valgrind false positives, made some debug logging improvements. Fixed bug in pager shutdown where it could wait on an ActorCollection containing canceled futures.
2019-11-04 03:04:03 -08:00
547616d5f1
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
2019-10-29 16:33:22 -07:00
9c0d671d07
Two bug fixes in Redwood related to split KV pairs and one was masking the other. The first bug resulted in an incomplete erasure of fragments for a split KV pair and the second bug would generate an unnecessary explicit null record for the same key which would cause reads to correctly see the key as missing. Redwood correctness test now clears the tree and verifies expected resulting pager footprint, which succeeds due to the bug fixes.
2019-10-29 01:31:59 -07:00
40d53e23f5
Optimization, only the first btree mutation boundary for a subtree needs to be compared to the subtree's lower bound. Also removed a check for a condition which is no longer possible due to other changes.
2019-10-28 16:05:11 -07:00
0d993522d3
CommitSubtree() will now return an empty page set even for the tree root because commit_impl() handles this correctly. Improved commitSubtree() debug output related to which mutations are relevant to a subtree. Added random setting of range clear boundaries after clear() in Redwood correctness to make sure mutation buffer logic handles this correctly. B+Tree's dbEnd mutation is represented as a clear to prevent unnecessary rightmost subtree traversal during commit.
2019-10-28 04:00:37 -07:00
5cf0045bc0
Merge pull request #2294 from satherton/feature-redwood
...
Bug fixes in Redwood
2019-10-25 14:56:01 -07:00
2ee1782c19
Bug fixes in Redwood. BTree height was not being reset when a new empty root is written. IKeyValueStore wrapper was not obeying the row limit in a reverse range query. Added yields to and delays to break up tasks and set IO priorities.
2019-10-25 14:52:06 -07:00
eb910b850b
fixed a window build error
2019-10-23 13:48:24 -07:00
613bbaecc4
Bug fix in queue page footprint tracking. Added VersionedBTree::destroyAndCheckSanity() which clears the tree, processes the entire lazy delete queue, and then verifies some pager usage statistics. This check is currently disabled because it appears to find a bug where the final state has a few more pages in use than expected. StorageBytes now includes the delayed free list pages as free space since they will be reusable soon.
2019-10-23 09:31:06 -07:00
6a57fab431
Bug fixes in lazy subtree deletion, queue pushFront(), queue flush(), and advancing the oldest pager version. CommitSubtree no longer forces page rewrites due to boundary changes. IPager2 and IVersionedStore now have explicit async init() functions to avoid returning futures from some frequently used functions.
2019-10-22 17:17:29 -07:00
44175e0921
COWPager will no longer expire read Snapshots that are still in use.
2019-10-18 01:27:00 -07:00
0e9d082805
Bug fixes in FIFOQueue concurrent nested reads and writes caused by the pager/freelist circular dependencies.
2019-10-17 21:34:17 -07:00
6b7317da9b
Bug and clarity fixes to tracking FIFOQueue page and item count.
2019-10-15 03:36:22 -07:00
Steve Atherton