Commit Graph

40 Commits

Author SHA1 Message Date
Alex Miller 12123f41d6 Plumb a read function up the stack to IDiskQueue 2019-02-12 23:44:13 -08:00
Alex Miller 6c7229ec07 read fix while recovery 2019-02-12 23:44:13 -08:00
Alex Miller 8b21d1ac8f Add a standalone recovery initialization function. 2019-02-12 23:44:13 -08:00
Alex Miller 2f49acc8a0 Add a read function. 2019-02-12 23:44:13 -08:00
Alex Miller 63eb62cd36 Fix a bug when a read was delayed until after the entire disk queue has been rewritten. 2019-02-12 23:44:13 -08:00
Alex Miller 9886386a83 temporarily verify commited data as a test for read 2019-02-12 23:44:13 -08:00
Alex Miller efa8aa7e2e Adjust findPhysicalLocation to not spam.
Context is now optional, so that our high-volume calls don't get logged,
but low-volume calls still get logged the same way that they did before.
2019-02-12 23:44:13 -08:00
Alex Miller f1c31e2305 Add a read function to disk queue 2019-02-12 23:44:13 -08:00
Alex Miller 2d2b03a9ff prepare DiskQueue for actors 2019-02-12 23:44:13 -08:00
Alex Miller 40fe29c29b Abstract TrackMe into a reusable CRTP class. 2019-02-12 23:44:13 -08:00
Alex Miller 018d12fe90 use firstpages instead of recoveryfirstpages 2019-02-12 23:43:10 -08:00
Alex Miller dbf7cefcd8 Add firstPages to DiskQueue 2019-02-12 23:43:10 -08:00
Alex Miller 2570b37e6e Add function to read pages from RawDiskQueue_TwoFiles 2019-02-12 23:43:10 -08:00
Andrew Noyes 067a445e06 Replace unused _ variables with wait(success(...)) 2019-02-12 17:30:30 -08:00
Alex Miller 22a08b2b4e Change mutable ref to pointer so outparams are obvious. 2019-02-04 18:04:22 -08:00
Alex Miller 0efcccc06f Fix a long standing minor bug in disk queue that could lead to unnecessary commits.
If the disk queue is called with the following series of operations:

Push(a) -> 1
Commit()
Pop(1)
Push(b)
Commit()
Commit()

Then the last Commit() should be a no-op, and not actually run accordingly.

However, anyPopped was only set to `false` if no pages were pushed, and thus
we'd falsely think that an extra empty page commit needed to happen to log to
record the new popped position, but there actually was no new popped page
position to record.

Aside from the extra commit, it maybe makes getCommitOverhead slightly
inaccurate, but that's only used for some accounting inside of the memory
storage engine and at a quick glance doesn't look like it should have caused
any bad effects.

I dug through history, and this code has been this way since the initial commit
by Dave, and then no one has touched the anyPopped logic since.
2019-02-04 18:04:22 -08:00
Alex Miller bfab7c150a Require PageHeader to be 36 bytes, and don't use magic numbers. 2018-12-17 13:37:44 -08:00
Robert Escriva 268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
Stephen Atherton 2fc86c5ff3 Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
# Conflicts:
#	fdbrpc/AsyncFileCached.actor.h
#	fdbserver/IKeyValueStore.h
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/workloads/StatusWorkload.actor.cpp
#	tests/fast/SidebandWithStatus.txt
#	tests/rare/LargeApiCorrectnessStatus.txt
#	tests/slow/DDBalanceAndRemoveStatus.txt
2018-09-20 03:39:55 -07:00
Alex Miller fb31a6999f Rewrite all files to have #include actorcompiler.h as the last include. 2018-08-14 15:50:26 -07:00
Alex Miller 535b5701e5 Rewrite all `Void _ = wait(...)` -> `wait(...)`.
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
Evan Tschannen 1c29275672 call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details. 2018-08-01 14:30:57 -07:00
Stephen Atherton 2878f30f29 Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
# Conflicts:
#	fdbserver/IKeyValueStore.h
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/storageserver.actor.cpp
2018-06-13 15:56:06 -07:00
A.J. Beamon e5488419cc Attempt to normalize trace events:
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.

Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.

This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
Stephen Atherton 2752a28611 Merge branch 'release-5.2' of github.com:apple/foundationdb into feature-redwood 2018-04-06 16:29:37 -07:00
Alec Grieser 0bae9880f1 remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py 2018-02-21 10:25:11 -08:00
Stephen Atherton 0a35f167e4 Merge branch 'master' into feature-redwood
# Conflicts:
#	fdbserver/DiskQueue.actor.cpp
#	fdbserver/IDiskQueue.h
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/fdbserver.vcxproj
#	fdbserver/fdbserver.vcxproj.filters
#	fdbserver/worker.actor.cpp
2018-02-12 01:30:02 -08:00
Evan Tschannen 6d5dd9bd27 fix: we cannot pipeline disk queue commits until after the first commit is successful 2018-01-02 13:30:27 -08:00
Evan Tschannen a5601877b3 fix: valgrind issue with destruction ordering 2017-12-18 15:31:59 -08:00
Stephen Atherton abb2dd1ebc Merge pull request #214 from cie/alexmiller/fallocate
Use fallocate to zero ranges instead of writing zeroes
2017-12-06 13:47:40 -08:00
Evan Tschannen 5a947212ed fix: ensure all prior commits have completed before returning that a commit has committed from the disk queue 2017-12-06 12:31:07 -08:00
Evan Tschannen 49dac11a5f added a SevWarnAlways for when a disk queue file grows larger than 20GB 2017-12-01 15:05:17 -08:00
Alex Miller 196258080b Refactor zeroing a chunk of a file from DiskQueue into IAsyncFile.
If we're going to do the work to provide more optimized ways to zero files,
then I'd feel better with this being in a more common place, so that any other
zero-ers are likely to reuse it.  It also makes testing easier/more obvious.

Also, because it's needed for correctness, fix the aligned_alloc for OSX, which
wasn't aligned, and use an actually aligned allocation function.
2017-11-30 17:57:55 -08:00
Alex Miller c7a120c59d Rename IAsyncFile::incrementalDelete -> IAsyncFileSystem::incrementalDeleteFile.
`deleteFile` existed in IAsyncFileSystem, so an incremental delete function
seems to belong more as a virtual method on IAsyncFileSystem than a static
method on IAsyncFile, and the naming should match.

As long as we're here, change IAsyncFile to declare a virtual destructor, so
that it has good and proper C++ behavior.  I presume this is what was vaguely
intended by the default constructor definition that previously existed?
2017-11-30 17:19:10 -08:00
Stephen Atherton 248dab79b6 Created “redwood” storage engine option and many changes to support that including IKeyValueStore::init() and custom DiskQueue file extensions. 2017-09-21 23:51:55 -07:00
Alec Grieser 300b5a17ed Merge branch 'release-5.0' 2017-08-25 18:55:33 -07:00
Evan Tschannen 272b4b984c fix: fixed a rare bug where we do not wait for a file in the process of being deleted to shutdown before rebooting a machine 2017-08-25 10:12:58 -07:00
Evan Tschannen be941b4bd1 sending void to committed could cause self to be deleted, so call cleanup before sending 2017-08-23 13:56:18 -07:00
A.J. Beamon d8e308c18f Enable use of incremental delete when deleting disk queue and sqlite KVS sqlite files. 2017-07-26 14:11:11 -07:00
FDB Dev Team a674cb4ef4 Initial repository commit 2017-05-25 13:48:44 -07:00