8976694ba1
UDP implementation (untested)
2020-08-06 14:06:50 -06:00
d8bb36c4c8
Refactor applyMetadataMutations to accept less args
2020-08-06 11:40:40 -07:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
2b8aa296bb
Merge pull request #3482 from sfc-gh-tclinkenbeard/support-downgrades
...
Support and test downgrades
2020-07-28 17:00:54 -07:00
525f10e30c
Merge master branch
2020-07-22 16:08:49 -07:00
b2a3b4fd83
Merge branch 'master' into mengxu/merge-6.3-PR
2020-07-20 11:34:18 -07:00
8cffd30951
FastRestore:Replace master word with controller
2020-07-16 16:37:13 -07:00
5b06d69d25
Pass watches test
2020-07-15 00:37:41 -07:00
0a8f61b967
Make TOML mandatory.
2020-07-12 19:43:38 -07:00
c20fc87d6a
Fix the cmake things
2020-07-12 14:47:39 -07:00
f08f0255cc
Copy paste cmake code until it seems like it maybe downloads TOML11
2020-07-12 14:47:39 -07:00
373b5ffb4f
Added forward compatibility and Downgrade workload
2020-07-09 14:28:35 -07:00
1dead7c199
Merge pull request #3440 from sfc-gh-xwang/master
...
add HealthMetricsWorkload and WriteTagThrottlingWorkload
2020-07-09 10:04:46 -07:00
c76e0cfe04
Add a ReadAfterWrite workload, to measure TLog->SS propagation delay.
2020-07-01 02:17:43 -07:00
dc568cad21
add HealthMetricsWorkload and WriteTagThrottlingWorkload to prepare for new throttling feature
2020-07-01 04:56:21 +00:00
76ac5dfeaf
Merge branch 'release-6.3' into merge-release-6.3-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes/release-notes-630.rst
2020-06-17 18:31:06 -07:00
96206a8032
Merge pull request #3368 from apple/release-6.3
...
Merge Release 6.3 to master
2020-06-15 20:15:22 -07:00
acbfe2e4c9
Revert "Revert "Initial RocksDB""
2020-06-15 12:45:36 -04:00
9cd1614c82
Revert "Initial RocksDB"
2020-06-11 15:29:46 -07:00
a4dbb5dd01
Merge branch 'trace-batch-thread-hostile' into rocksdb-6.3
2020-06-11 15:53:57 +00:00
31db90963d
factor out duplicate include directories
2020-06-10 21:45:40 +00:00
744b6aabad
Merge branch 'rocksdb-build' of github.com:Daniel-B-Smith/foundationdb into rocksdb-build
2020-06-10 21:36:14 +00:00
f5562355e3
Fix comment
2020-06-10 21:36:13 +00:00
49562c2de3
Merge branch 'rocksdb-build' of github.com:Daniel-B-Smith/foundationdb into rocksdb-build
2020-06-10 21:29:39 +00:00
8c00b391da
Merge remote-tracking branch 'upstream/release-6.3' into rocksdb-build
2020-06-10 21:29:20 +00:00
dae91bb05f
Add RocksDB to FDBComponents.cmake
2020-06-10 21:07:49 +00:00
b8ac6ee41e
Get build to work with Ninja
2020-06-10 20:53:53 +00:00
2872673763
Remove cmake logic from initial PR
2020-06-10 19:40:06 +00:00
693d9e8b89
Merge branch 'master' into fdb_cache_wo_allocator
2020-06-09 15:09:58 -07:00
c54ad12072
Incremental improvements
2020-06-08 16:46:42 +00:00
c836a9f5a9
Add RocksDB to build Dockerfile and CMake to build RocksDB
2020-06-08 16:22:28 +00:00
aa9d30a111
Build RocksDB as an external package
2020-06-05 22:58:45 +00:00
4b402b213a
Enable FDB to be built without RocksDB
2020-06-03 20:32:39 +00:00
cf13e00a8f
Merge remote-tracking branch 'origin/release-6.3' into fdb_cache_wo_allocator
2020-06-01 17:38:31 -07:00
57390bfbc5
Merge pull request #3263 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-28 14:14:38 -07:00
74c11b4c9a
Merge pull request #3262 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-28 13:43:46 -07:00
382666d21e
Permissions were wrong on installed files. A missing DESTINATION for fdbserver caused the binary to be missing.
2020-05-28 10:33:09 -07:00
5af427f3b2
Merge remote-tracking branch 'upstream/release-6.3' into rocksdb-6.3
2020-05-28 15:42:34 +00:00
135b88d85f
Merge pull request #3254 from apple/release-6.3
...
Merge release-6.3 into master
2020-05-27 20:03:19 -07:00
b3c269351a
Merge pull request #3252 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-27 19:13:24 -07:00
96f396ec7f
Make sure to build the binary stripping targets when packaging stripped binaries
2020-05-27 16:40:07 -07:00
736a47837d
Merge pull request #3245 from apple/release-6.3
...
Merge release-6.3 into master
2020-05-27 13:27:47 -07:00
8731700d80
Merge remote-tracking branch 'upstream/release-6.3' into rocksdb-6.3
2020-05-27 20:02:25 +00:00
d6be580319
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# documentation/sphinx/source/command-line-interface.rst
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
2020-05-27 11:42:27 -07:00
d128252e90
Merge release-6.3 into master
2020-05-22 09:25:32 -07:00
5d361fe532
Copy/paste rebase onto 6.3
2020-05-22 15:02:51 +00:00
2f9cd383ea
Use stripped executables in packages when not building with debug symbols.
2020-05-21 13:39:55 -07:00
6ce574f5ad
Merge remote-tracking branch 'upstream/release-6.3' into add-data-distribution-metrics
2020-05-17 23:36:52 -07:00
27da91ab9e
Merge remote-tracking branch 'upstream/master' into mutation-debugging
2020-05-13 12:51:44 -07:00
ef724bf939
Merge remote-tracking branch 'upstream/master' into add-data-distribution-metrics
2020-05-08 18:39:28 -07:00
749fcd13b0
Merge branch 'master' into fdb_cache_wo_allocator
2020-05-08 16:23:29 -07:00
aed97a9f20
Merge branch 'master' into transaction-tagging
2020-05-07 14:52:22 -07:00
ff992060cd
Merge pull request #3073 from tclinken/fix-open-for-ide-build
...
Fix non-boost-related OPEN_FOR_IDE build errors
2020-05-07 14:47:59 -07:00
b1055a8501
Merge branch 'master' into transaction-tagging
2020-05-05 16:03:39 -07:00
f329164fb4
Merge pull request #2532 from dongxinEric/feature/hot-read-key-detection-part-2
...
Feature/hot read key detection part 2
2020-05-05 14:33:34 -07:00
dd033736ed
Merge branch 'master' into fdb_cache_subfeature2
2020-05-04 17:29:43 -07:00
80a235aa80
Add some correctness tests
2020-05-04 10:15:18 -07:00
943e8e7e84
More fixes for OPEN_FOR_IDE build
2020-05-01 23:04:12 -07:00
b7f5f3be48
merge in master
2020-04-28 13:11:47 -07:00
7dd7406c59
Merge branch 'master' into feature/hot-read-key-detection-part-2
2020-04-16 14:54:05 -07:00
b85dc16c6d
Merge branch 'master' into fdb_cache_subfeature2
2020-04-14 17:07:41 -07:00
65620b00f3
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-13 14:21:07 -07:00
07cc0a8d74
code cleanup
2020-04-10 17:02:11 -07:00
b4bde7f8f1
Added TPC-C workload
2020-04-03 11:33:24 -07:00
a0c8946f31
Merge branch 'master' into fdb_cache_subfeature2
2020-04-02 12:27:04 -07:00
a63f8b839c
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-03-31 12:05:42 -07:00
146787e8d6
Add MutationTracking to CMake.
2020-03-27 04:16:15 -07:00
289487559d
Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
...
This reverts commit 804fe1b22e
2020-03-24 18:11:15 -07:00
804fe1b22e
Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
...
This reverts commit 648dc4a933487d131b38
2020-03-19 21:34:28 -07:00
ab33e688c8
comment
2020-03-09 10:45:57 -07:00
7a76e9556d
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-04 11:24:39 -08:00
735e7c71f2
Add the cachetest back in.
2020-02-24 17:26:20 -08:00
be70b18c22
Remove Cachetest for merge.
2020-02-24 16:46:36 -08:00
28768715f2
Custom bookkeeping allocator for PTree. Simple Cache workload.
2020-02-14 11:42:47 -08:00
058de86b88
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-02-05 10:07:36 -08:00
2c83fbfe6c
Rename to BackupWorker.actor.cpp to be explicit
...
There is already one file named backup.actor.cpp in "fdbbackup/".
2020-01-22 19:38:46 -08:00
64052f6349
Check and fill backup gaps for old epochs and tags
...
Sometimes the backup worker has not updated progress to the system space and a
master recovery happens. As a result, next epoch doesn't know the progress of
previous ones. This change is to check for such missing gaps and fill them with
the whole range [startVersion, endVersion).
The code is refactored into BackupProgress.actor.* to consolidate backup
progress processing for the master server.
2020-01-22 19:38:46 -08:00
de8d953865
Add backup role, class, and worker skeleton
2020-01-22 19:35:30 -08:00
b0a1af1288
Added the actual read hot detection algorithm and logging machanism.
...
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
1d6ac716a1
Merge remote-tracking branch 'origin' into add-data-distribution-metrics
2020-01-15 13:20:04 -08:00
eacfdf6da3
Add a simple workload, ReportConflictingKeysWorkload, to test correctness of the API and performance overhead added to the resovler.
2019-12-06 16:21:03 -08:00
d46c7ded59
Merge remote-tracking branch 'origin/master' into storage-cache-subfeature1
2019-11-14 17:52:22 -08:00
d147071bfd
Merge branch 'release-6.2'
...
# Conflicts:
# fdbserver/MemoryPager.actor.cpp
2019-11-12 16:31:47 -08:00
a4a0bf18f9
Merging with Master.
2019-11-12 13:01:29 -08:00
61558eea04
Implemented page preloading on BTree cursor seeks to enable hiding latency on soon-to-be-read sibling pages. Added random scans with various preload sizes to the set performance unit test. ObjectCache now tracks hits, misses, and pages which were preloaded but then never used prior to eviction. BTree pages no longer store flags because height is sufficient. Removed virtual specifier in classes not designed to be further inherited. Removed old prototype code (PrefixTree, IndirectShadowPager, MemoryPager) as some interface changes are incompatible and they are no longer worth maintaining.
2019-11-11 09:54:22 -08:00
de8921b660
Move RestoreWorkerInterface to fdbclient
2019-10-25 10:42:22 -07:00
d4de608bb6
Fix OPEN_FOR_IDE build
2019-10-25 10:42:22 -07:00
60fb04ca68
Fork TLogServer into TLogServer_6_2
...
This prepares us for incoming modifications to the TLog that can't
easily coexist with our current on-disk state.
2019-10-03 01:41:25 -07:00
9cc832cfd6
FastRestore:Fix Mac and Windows compilation error
2019-08-02 14:33:08 -07:00
7ccaeddf05
Merge branch 'master' into mengxu/performant-restore-PR
2019-08-01 13:23:17 -07:00
45083edf74
Merge branch 'master' into mengxu/performant-restore-PR
...
Fix conflicts as well.
2019-07-25 10:46:11 -07:00
9ad9bd4c1f
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-24 15:22:26 -07:00
c3a8ae4752
Merge pull request #1791 from fzhjon/fetch-keys-requests-priority
...
Introduce priority to fetchKeys requests from data distribution
2019-07-19 14:54:51 -07:00
a79c5d8200
Move code in ApplyMetadataMutation from .h to .cpp
...
Not only can we now compile this giant code block in parallel with the
rest of the code, we also no longer compile this giant function twice!
(Once for the master, once for the proxy.)
2019-07-16 15:00:34 -07:00
f12a3909f3
renamed workloads and made code style adjustments
2019-07-11 09:56:58 -07:00
1582a2a24d
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-09 13:41:54 -07:00
0fe6edc254
Merge pull request #1678 from mpilman/features/external-workload
...
Features/external workload
2019-06-25 13:53:19 -07:00
afb0dbcd1c
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-06-20 19:11:29 -07:00
ab7562160c
Made JavaWorkload an external workload
2019-06-19 13:03:41 -07:00
1707f068e0
started implementation first c workload
2019-06-19 13:03:41 -07:00
c8957d93f8
Implementation code complete
2019-06-19 13:03:41 -07:00
5eb833759e
Extend RebootRequest API to include time to suspend the process before reboot. This is intended to be used for testing purposes to simulate failures.
2019-06-14 11:35:38 -07:00
e2b6acffce
Merge branch 'master' into add-data-distribution-metrics
2019-06-12 16:12:16 -07:00
02e173b601
Add changeConnectionFile method to Transaction
...
Also add tests
2019-06-11 13:58:22 -07:00
8144882d7b
Merge branch 'apple-master' into features/local-rk
2019-06-10 19:40:25 -07:00
477fd152c0
FastRestore:Refactor code
...
1) Use the runRYWTransaction for simple DB access
2) Replace some printf with TraceEvent
3) Remove printf not used in debugging
4) Avoid wait inside the condition in loop-choose-when for
the core routine of restore worker, loader and applier.
5) Rename Restore.actor.cpp to RestoreWorker.actor.cpp since
the file only has functionalities related to restore worker.
Passed correctness test
2019-06-04 11:22:47 -07:00
66811b7bd2
update to latest version
2019-06-03 16:49:19 -07:00
a6bee65f11
Merge branch 'master' into add-data-distribution-metrics
2019-05-30 11:11:49 -07:00
ec7834e2f7
code re-orgnaization and address comments
2019-05-28 22:07:46 -07:00
69edefe68b
Snapshot based backup and resotre implementation
2019-05-28 22:07:46 -07:00
fac63a83c4
FastRestore:Use NotifiedVersion to deduplicate requests
...
Add a NotifiedVersion into an applier data which represents
the smallest version the applier is at.
When a loader sends mutation vector to appliers, it sends
the request that contains prevVersion and commitVersion.
This commits also put actor into an actorCollector for
loop-choose-when situation.
2019-05-22 22:09:54 -07:00
74a705645b
changed kv format of metrics list and added simulation test
2019-05-22 12:44:32 -07:00
57968d9df7
Merge branch 'master' of https://github.com/apple/foundationdb into MakoWorkload
2019-05-21 16:24:11 -07:00
12a51b2d39
fix bugs, update naming and comments, refine functions
2019-05-20 18:26:30 -07:00
54f4df604a
CMake:Add FastRestore files
2019-05-16 19:43:19 -07:00
57912b33a5
fixed merge error
2019-05-13 14:15:23 -07:00
9c02354255
pass NDEBUG to sqlite to enable debug mode
2019-05-13 14:15:23 -07:00
5678a7417e
Mako Workload
2019-05-09 15:55:05 -07:00
22499666d0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/LogRouter.actor.cpp
# flow/Trace.cpp
# versions.target
2019-05-08 18:19:35 -07:00
a0cc3d98a1
Add a workload to trigger repeated recoveries.
2019-05-06 18:16:44 -07:00
25c75f4222
FastRestore: Add new empty files for restore roles
...
Add .h and .cpp files for RestoreLoader and RestoreApplier roles.
We will split the code for each restore role into a separate file.
This commit also fixes the bug in including RestoreCommon.actor.h, and
remove the unused code.
2019-05-06 16:59:41 -07:00
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
058d028099
Merge pull request #1301 from mpilman/features/cheaper-traces
...
Defer formatting in traces to make them cheaper
2019-04-09 10:11:04 -07:00
789cd67bcd
Don't compile JavaWorkload by default
2019-04-08 13:06:29 -07:00
bdba8e22eb
Added test and bugfixes
2019-04-08 11:05:29 -07:00
4287b1d2a1
resolved minor merge issues
2019-04-05 13:12:19 -07:00
02e3b634fb
Compile sqlite with NDEBUG so we can debug
2019-04-05 13:12:19 -07:00
101a05ae77
Merge branch 'master' into features/client-simulator
2019-04-03 10:03:56 -08:00
56a1128a9b
Enhance cmake's gperftools support
...
Add compiler flags and link flags for gperftools.
2019-04-02 17:34:29 -07:00
e23e63c6ac
Implemented JavaWorkload
...
This change allows a user to write a workload in Java.
The way this is implemented is by creating a JVM within the
simulator and calling the corresponding workload class. A
workload can then run in the simulator or on a testing cluster.
If the workload is executed within the simulator, the resulting
test will not be deterministic anymore as it will execute in a
different thread (and even without that it is not clear, whether
we could get determinism as the JVM does a lot of stuff that are
not deterministic).
This is intendet to get better testing of the Java client and
layer authors can use the simulator to test their layers on a single
machine but they can still simulate failing machines etc.
2019-03-31 17:57:43 -07:00
a7e45cff91
Merge pull request #1176 from jzhou77/ratekeeper
...
Make Ratekeeper a separate role
2019-03-12 15:58:59 -07:00
2537f26de6
First implementaion of more user-friendly cpack
...
Up unto here this code is only very rudiemantery tested.
This is a firest attempt of making cpack more user-friendly.
The basic idea is to generate a component for package type so
that we can have different paths depending on whether we build
an RPM, a DEB, a TGZ, or a MacOS installer. The cpack package
config file will then chose the correct components to use.
In a later point this should make it possible to build these
with `make packages` and the ugly iteration with calling cmake
between each package would be obsolete. While this solution is
a bit more bloated, it is also much more flexible and it will be
much easier to use.
Another benefit is, that this will get rid of all warnings during
a cpack run
2019-03-07 16:49:29 -08:00
36a51a7b57
Fix a segfault bug due to uncopied ratekeeper interface
2019-03-07 13:16:20 -08:00
3c86643822
Separate Ratekeeper from data distribution.
...
Add a new role for ratekeeper.
Remove StorageServerChanges from data distribution.
Ratekeeper monitors storage servers, which borrows the idea from
DataDistribution.
2019-03-07 13:16:20 -08:00
27d199409e
Add KillRegion.actor.cpp workload to cmake
2019-03-07 12:14:42 -08:00
981426bac9
More ide fixes
2019-03-05 18:03:57 -08:00
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
fa96b8dd33
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-20 16:56:16 -08:00
df61bd07db
Save an (unused) copy of the previous TLog.
2019-02-19 22:18:10 -08:00
91e05575a2
Rename OldTLogServer -> OldTLogServer_4_6
2019-02-19 22:18:10 -08:00
999ea09bfd
Use correct fwd decls in TesterInterface
...
Also TesterInterface.h -> TesterInterface.actor.h
2019-02-19 15:16:59 -08:00
699216f713
Use fwd decls in workloads
...
Also workloads.h -> workloads.actor.h
2019-02-19 15:16:59 -08:00
3f0fd2a20c
Use fwd decls in WorkerInterface
...
Also WorkerInterface.h -> WorkerInterface.actor.h
2019-02-19 15:16:59 -08:00
27a3153719
Use ACTOR forward declarations in MoveKeys
...
Also MoveKeys.h -> MoveKeys.actor.h
2019-02-19 15:16:59 -08:00
9b14aeb156
Tell cmake not to link/install on ide build
2019-02-19 15:16:59 -08:00
83060c6e56
Merge pull request #1062 from jzhou77/PR
...
Add a new DataDistributor role.
2019-02-15 13:51:27 -08:00
75f692b931
simplify actorcompiler and target to compile coveragetool
2019-02-15 00:01:42 -08:00
c35d1bf2ef
Fix according Alex's comment
2019-02-14 16:30:13 -08:00
8a17905621
Add a couple new files to CMakeLists
2019-02-14 08:08:44 -08:00
a09afe5906
Added Throttling workload to test native health metrics API
2019-02-04 13:04:25 -08:00
58964af7e1
ctest improvements - #1058
...
- A set of CMake variables controls whether to keep
the simfdb directory and the traces and whether we
want to aggregate the traces into a single file
- Test labels now contain the directory they are in
so that one can now run `ctest -R fast/`
- A different binary can be used for restart tests. CMake
will automatically look for an installed fdb and use that
by default. If none is found, it will use the built one
but it will also print a warning
- CMake will throw an error if there are any text files in
the tests directory that are not associated with a test.
- Moved testing from fdbserver/CMakeLists.txt to
tests/CMakeLists.txt
- Moved fdb testing functions to its own cmake module
2019-01-22 14:34:51 -08:00
b4a446756a
Remove more top-level tests that were out of order.
2019-01-14 20:28:40 -08:00
0d579b4730
Top level tests aren't run by default, and some fail.
2019-01-14 19:14:25 -08:00
b3e977d7c1
Apply test directory as a label.
...
This isn't ideal, as it makes `restarting/from_5.2.0/potato.txt` have
the label "from_5.2.0" instead of "restarting", but it does make the
fast label work right.
2019-01-14 19:14:25 -08:00
414fb0b6c8
made TestRunner work with XML traces
2019-01-14 19:14:25 -08:00
14f0a6958b
added all tests to ctest
2019-01-14 19:14:25 -08:00
b096b8e3f8
First tests working with ctest
2019-01-14 19:14:25 -08:00
550daa05f8
Default configuration now builds.
2018-12-13 15:52:27 -08:00
df0f491c29
Some more improvements to the build and preparations for packaging
2018-12-13 15:04:13 -08:00
e70e59a895
Change some file locations.
2018-12-13 14:53:19 -08:00
dce290909d
fdbserver now compiling
2018-12-13 14:13:47 -08:00
Markus Pilman