90ced244eb
Fix -Wunused-but-set-variable warnings
2021-12-01 18:15:53 -08:00
3b5d23ef88
Remove unnecessary temporary objects when appending to vector of pairs
2021-12-01 18:15:16 -08:00
a82845af43
Merge branch 'master' into bg_bindings
2021-12-01 16:55:28 -06:00
404bef2f03
Merge pull request #6076 from sfc-gh-ajbeamon/fix-cstate-registration-race
...
Fix a race condition with updating the coordinated state and updating the master registration
2021-12-01 12:56:12 -08:00
ec64890ac1
Remove some usages of PRId64 by using fmt library
2021-11-30 23:35:36 -08:00
72c5fb183d
Fix: avoid updating the master registration while the cstate is written but we are not accepting commits.
2021-11-30 15:44:04 -08:00
43b3e05fd5
Merge branch 'master' of github.com:apple/foundationdb into delay-prioritized-eviction
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2021-11-29 16:14:43 -08:00
ef248f3dbd
Merge pull request #6060 from sfc-gh-satherton/remove-try-evict
...
Remove unused / unwanted code in Redwood.
2021-11-29 13:26:05 -08:00
b6fd402a3c
Add option to use boost or libcoro
...
By default, use boost everywhere except windows and linux x86 (for
performance reasons)
2021-11-29 13:14:15 -08:00
1870e07ff4
Fixed pause racing with waitUntilHealthy
2021-11-29 14:19:15 -06:00
8216ad1e4e
When a cached page is hit, if it is in the prioritized eviction order don't move it to the normal order.
2021-11-28 21:04:35 -08:00
bed25f9571
Delay prioritized eviction of updated pages until after commit completes.
2021-11-28 21:03:44 -08:00
0f5535fce1
Remove explicit tryEvict(x) as it is unused and it is functionally replaced by prioritizeEviction(x).
2021-11-27 03:26:34 -08:00
e3819dad7c
fix: If a removed tlog never attempted a queue commit, the update storage loop could get stuck waiting for queueCommittingVersion to advance
2021-11-25 09:55:01 -08:00
80014d3247
Merge branch 'master' of https://github.com/apple/foundationdb into fix-best-team
...
# Conflicts:
# fdbclient/ServerKnobs.cpp
2021-11-24 11:56:23 -08:00
1afef44f19
fix: self->shardsAffectedByTeamFailure->moveShard must be called without any waits after getting the destination team or we could miss failure notifications for the storage servers in the destination team
2021-11-23 15:26:38 -08:00
37c9a1320c
added --print_sim_time to print simulated time to stdout
2021-11-23 15:01:44 -08:00
b0aad44831
change feed requests with an explicit end need to get empty versions at the end boundary
2021-11-23 15:01:44 -08:00
3360000a91
fix: an empty mutation vector did not mean that all of the data was returned
2021-11-23 15:01:44 -08:00
e97d337ffe
fix: when a feed transitions from atLatest to not atLatest stall all updates at the blocked version
2021-11-23 15:01:44 -08:00
10a925b7e9
do not return mutations larger than the dequeVersion
2021-11-23 15:01:44 -08:00
e19277160e
improved the change feed workload to test whenAtLeast
2021-11-23 15:01:44 -08:00
f3bb1d8f51
properly handle an active change feed stream when removed
2021-11-23 15:01:44 -08:00
12e68da635
Merge pull request #6036 from sfc-gh-satherton/redwood-debug-sim-fixes
...
Test-only bug fixes in Redwood along with debug logging detail improvements.
2021-11-22 11:13:28 -08:00
e43ddb09ac
Merge pull request #6034 from sfc-gh-satherton/remap-window-fix
...
Bug fix: incorrect remap cleanup window size in KeyValueStoreRedwood
2021-11-22 09:12:59 -08:00
d2149631d5
Bug fix, the remap cleanup window was being initialized incorrectly in the Redwood KVS wrapper.
2021-11-19 21:40:09 -08:00
3901d60548
Test-only bug fixes in Redwood along with debug logging detail improvements. Added clearRemapQueue() to Pager to more cleanly and reliably expire all old data and process the remap queue, fixing a bug where with certain configuration parameters and a lot of data the DestructiveSanityCheck would fail because it would not run cleanup long enough. Added more parameters to performance/set unit test.
2021-11-19 01:00:14 -08:00
c745002607
Merge remote-tracking branch 'origin/master' into add-format-warning
2021-11-18 11:13:03 -08:00
5879d611f5
Merge pull request #5899 from sfc-gh-jfu/jfu-fix-ss-segfault
...
Fix rare segfault that can occur when SS terminates while an actor that uses the StorageServer is still on the stack
2021-11-18 09:57:11 -08:00
2613ec7561
Expand use of fmt to get rid of %ld usage
2021-11-17 17:03:32 -08:00
07349869d9
Use fmt to address -Wformat warnings
2021-11-17 14:45:48 -08:00
766a05d33c
Merge remote-tracking branch 'origin/master' into add-format-warning
2021-11-17 12:14:01 -08:00
3caca74ac2
Merge commit 'fd707c6d7ee80de6d9fda5796da2d0add10abd79' into bit-flipping-workload
2021-11-16 21:54:27 -08:00
867ad8be46
Merge branch 'bit-flipping-workload' of github.com:sfc-gh-ngoyal/foundationdb into bit-flipping-workload
2021-11-16 18:12:18 -08:00
6e43dde613
Fixed bad merge resolution.
2021-11-16 18:04:37 -08:00
725d31e264
Fix parameters passes to simulatedMachine().
2021-11-16 17:50:12 -08:00
ce112e1f23
Enhance trace event.
2021-11-16 17:39:59 -08:00
8f6934c4d0
Merge branch 'master' of github.com:apple/foundationdb into jfu-fix-ss-segfault
2021-11-16 18:03:32 -05:00
557186ed17
Merge pull request #5909 from sfc-gh-jfu/jfu-cc-request-dbinfo
...
Change dbinfo broadcast to be explicitly requested by the worker registration message
2021-11-16 15:01:42 -08:00
035e0d6e52
Merge branch 'master' into bit-flipping-workload
2021-11-16 14:42:22 -08:00
8d6cfcb630
Merge pull request #6003 from sfc-gh-etschannen/fix-forwarding
...
fix: coordinators would process forwarding requests before making them durable
2021-11-16 14:24:21 -08:00
35bce4cd36
added a comment
2021-11-16 13:07:35 -08:00
fd635432c4
fix: coordinators would process forwarding requests before making them durable
2021-11-16 12:21:26 -08:00
21c3c585ca
Make file name parameter more user friendly in unit tests.
2021-11-16 04:03:11 -08:00
867999a41a
Rename wrong_format_version to unsupported_format_version.
2021-11-16 03:25:54 -08:00
7b29804a5e
Fix typo.
2021-11-16 02:30:57 -08:00
c53f5aa110
Renamed redwood to redwood-1-experimental and file extension to .redwood-v1.
2021-11-16 02:15:22 -08:00
964d0209ca
Merge pull request #5637 from sfc-gh-ljoswiak/features/data-loss-prevention
...
Data loss protection when joining new cluster
2021-11-15 15:26:32 -08:00
3d26d2372b
Merge pull request #5932 from sfc-gh-ahusain/ahusain-improveTesterLogging
...
Improve tester actor logging to track workload run & check status
2021-11-15 13:28:55 -08:00
02d0c43bc2
Merge pull request #5982 from sfc-gh-tclinkenbeard/improve-error-descriptions
...
Make snapshot errors more descriptive
2021-11-15 13:18:19 -08:00
6e81b83924
fix: cleanup change feeds which have been completely removed from a storage server
2021-11-15 11:47:42 -08:00
94a51e57a5
Merge branch 'master' into feature-changefeed-empty-versions
...
# Conflicts:
# fdbclient/StorageServerInterface.h
2021-11-14 19:13:05 -08:00
6909754b21
changefeeds now have a whenAtLeast function for efficiently learning when the version has updated but no mutations have been committed
2021-11-14 19:08:46 -08:00
0ba77ea79b
Fix proxySnapCreate trace typo
2021-11-14 16:12:28 -08:00
9422b8e5f2
Restricted getRangeAndFlatMap to snapshot
2021-11-12 15:12:37 -08:00
508429f30d
Redwood chunked file growth and low priority IO starvation prevention ( #5936 )
...
* Redwood files now growth in large page chunks controlled by a knob to reduce truncate() calls for expansion. PriorityMultiLock has limit on consecutive same-priority lock release. Increased Redwood max priority level to 3 for more separation at higher BTree levels.
* Simulation fix, don't mark certain IO timeout errors as injected unless the simulated process has been set to have an unreliable disk.
* Pager writes now truncate gradually upward, one chunk at a time, in response to writes, which wait on only the necessary truncate operations. Increased buggified chunk size because truncate can be very slow in simulation.
* In simulation, ioTimeoutError() and ioDegradedOrTimeoutError() will wait until at least the target timeout interval past the point when simulation is sped up.
* PriorityMultiLock::toString() prints more info and is now public.
* Added queued time to PriorityMultiLock.
* Bug fix to handle when speedUpSimulation changes later than the configured time.
* Refactored mutation application in leaf nodes to do fewer comparisons and do in place value updates if the new value is the same size as the old value.
* Renamed updatingInPlace to updatingDeltaTree for clarity. Inlined switchToLinearMerge() since it is only used in one place.
* Updated extendToCover to be more clear by passing in the old extension future as a parameter. Fixed initialization warning.
2021-11-12 13:47:07 -08:00
62efeb6812
Merge remote-tracking branch 'origin/master' into add-format-warning
2021-11-12 11:50:36 -08:00
08fd69e787
Merge pull request #5967 from nblintao/flatmap-exception
2021-11-12 10:19:44 -08:00
019fd50f46
Merge pull request #5971 from Daniel-B-Smith/rocksdb-error-logging
...
Clean up RocksDB error logging
2021-11-12 13:17:10 -05:00
82c3e8bf79
Trigger buildTeam operation if server transition from unhealthy -> healthy ( #5930 )
...
* Trigger buildTeam operation if server transition from unhealthy -> healthy
DataDistribution actor helps in building teams as server count changes
(add/removal), however, it is possible that total_healthy_server count
is insufficient to allow team formation. If happens, even healthy server
count recover, the buildTeam operation will not be triggered.
Patch proposal is to trigger `checkBuildTeam` operation if server
transitions from unhealthy -> healthy state. Incase system already
has created enough teams (desiredTeamCount/maxTeamCount), the operation
incurs a very minimal cost.
2021-11-12 09:41:01 -08:00
2d5f924278
GetKeyValuesAndFlatMap should return error if not retriable
2021-11-12 09:35:28 -08:00
9dccb0131e
Clean up RocksDB error logging
2021-11-12 12:14:12 -05:00
d73d2144fd
Adjust distributorSplitRange order.
2021-11-11 20:28:55 -08:00
984bc0fbea
Added Endpoints.
2021-11-11 16:56:04 -08:00
329091e14f
Merge branch 'master' into bg_bindings
2021-11-11 10:13:37 -06:00
b8ac4213a1
Switched BG APIs to transaction instead of database
2021-11-11 08:59:06 -06:00
0dfb72176e
Merge pull request #5857 from sfc-gh-vgasiunas/notify-client-lib-changes
...
A mechanism to notify MVC about relevant client library status changes on the cluster
2021-11-11 07:43:20 -07:00
e4c3f886da
Fix recovery issue
2021-11-10 16:15:13 -08:00
28b72550f3
Remove additional unused tracing
2021-11-10 13:33:49 -08:00
c93052121f
Fix issue where transaction spans would not be recorded
2021-11-10 13:33:49 -08:00
481cf9bb55
Merge pull request #5788 from Daniel-B-Smith/rocks-throttle
...
Throttle the number of concurrent reads to RocksDB
2021-11-10 15:20:30 -05:00
499dbcdb18
Don't fail fetchKeys when server overloaded is returned
2021-11-10 14:15:42 -05:00
db3c08c7cd
Merge pull request #5928 from sfc-gh-anoyes/anoyes/fix-heap-use-after-free
...
Fix a heap use after free
2021-11-10 10:21:05 -08:00
51b8ccf7d3
Merge remote-tracking branch 'apple/master' into notify-client-lib-changes
2021-11-10 18:40:34 +01:00
394b9dc619
Code review changes
2021-11-10 11:53:27 -05:00
b50b3de5d0
Allow SS to respond with server overloaded
2021-11-10 11:52:02 -05:00
66520eb1c1
Utilize read types to do selective throttling
2021-11-10 11:51:04 -05:00
470896bdc4
Redwood inline same-size value updates ( #5925 )
...
* Refactored mutation application in leaf nodes to do fewer comparisons and do in place value updates if the new value is the same size as the old value.
* Renamed updatingInPlace to updatingDeltaTree for clarity. Inlined switchToLinearMerge() since it is only used in one place.
2021-11-10 08:22:57 -08:00
ec99d1d888
Merge pull request #5946 from Daniel-B-Smith/cfoptions
...
Move table factory options to CFOptions
2021-11-10 11:20:41 -05:00
a246c1555d
Merge pull request #5931 from sfc-gh-mpilman/features/all-unit-tests-in-ci
...
Make sure unit tests are run often enough
2021-11-10 08:41:04 -07:00
8822c589de
Move table factory options to CFOptions
2021-11-09 17:29:06 -05:00
4b2757bf99
Fix memory bug in IndexPrefetchDemo
2021-11-09 13:52:28 -08:00
fdb3b72e35
Introduce GetRangeAndFlatMap to push computations down to FDB
...
Re-introduce #5609
2021-11-09 13:52:28 -08:00
15e0d5b29f
Add explicit transaction options when reading cluster ID
2021-11-09 12:29:49 -08:00
74cf64fe0f
Sync cluster ID through ServerDBInfo
2021-11-09 12:29:48 -08:00
4640045243
Fix rare simulation failures
...
When partitions appear before a cluster has fully recovered, it was
possible to have different tlogs persist different cluster IDs because
they were involved in different partitions. This would affect recovery
when a quorum was eventually reached. The solution to this is to avoid
persisting the cluster ID before a cluster has fully recovered, to make
sure all nodes agree on the cluster ID.
2021-11-09 12:29:48 -08:00
1fa726ca73
Fix compilation issue
2021-11-09 12:29:48 -08:00
3988b11fd6
Cleanup
2021-11-09 12:29:48 -08:00
aa3383f0e3
Exclude when joining new cluster
2021-11-09 12:29:48 -08:00
3e2c65bb11
Allow tlog to join another cluster but retain its data
2021-11-09 12:29:48 -08:00
30867750b5
Add protection against storage and tlog data deletion when joining a new cluster
2021-11-09 12:29:47 -08:00
2887e1c30a
set flag to true when doing first registration
2021-11-09 12:44:07 -05:00
5e4f25f96d
Improve tester actor logging to track workload run & check status
...
Patch improves logging in "tester.actor" to assist better tracking
of workload run & check status
2021-11-08 16:18:17 -08:00
d6fad2e489
readded old tlog tests
2021-11-08 15:52:08 -07:00
7df059570a
Make sure unit tests are run often enough
2021-11-08 15:43:32 -07:00
d97d968176
Added KeyBackedObjectMap and KeyBackedObjectProperty classes for storing serializable objects in FDB ( #5896 )
...
* Cleaned up some lambda capture workaround since x=y captures weren't available when these classes were originally written.
* Added KeyBackedObjectMap and KeyBackObjectProperty, which work like KeyBackedMap and KeyBackedProperty but use ObjectWriter/Reader for Value serialization so that the type can evolve over time.
* Disabled unit tests which shouldn't run as part of random selection.
2021-11-08 13:04:53 -08:00
b7e393587c
Fix a heap use after free
...
If we accept arena arguments by value, then the lifetime of any memory
allocated by that arena ends when the function returns. Given that we
seem to be appending to VectorRef's passed by pointer this is unlikely
to be what we want.
2021-11-08 12:51:32 -08:00
00f4bd8536
Check ccInterface against serverDbInfo's cc and make broadcast unconditional for first registration
2021-11-08 12:43:02 -05:00
9408c11c3d
MVC2.0: Remove client library status "available"
2021-11-08 18:10:36 +01:00
418cc4dc50
MVC2.0: Notifying clients about deleting or disabling client libraries that have upload or active status;
...
Declare library status access and change transactions as lock aware
2021-11-08 18:01:58 +01:00
396cd58b21
cancel ss core and ss actor collection after termination and before context switch
2021-11-04 16:05:23 -04:00
30cef51746
Improve tracing in ddSnapCreateCore
2021-11-04 12:59:50 -07:00
586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB"
2021-11-04 08:46:56 -07:00
679023ac51
Merge pull request #5609 from nblintao/index-prefetch-demo
2021-11-03 23:07:34 -07:00
4e8625ccc0
retain old behaviour along with explicit request
2021-11-03 17:23:07 -04:00
6c98e35893
Rename Hop to FlatMap
2021-11-03 13:32:01 -07:00
0853661d13
Introduce getRangeAndHop to push computations down to FDB
2021-11-03 13:21:16 -07:00
59f0a2c3e5
Change dbinfo broadcast to be explicitly requested by the worker registration message
2021-11-03 15:51:21 -04:00
5b2617a524
Added local granule file reading to mako
2021-11-03 09:33:30 -05:00
b4d69610ee
Remove unused variable and more clearly explain out of range annotation in Redwood debug output.
2021-11-02 21:48:37 -07:00
84854761cb
Change Redwood to use xxhash for checksums.
2021-11-02 21:47:31 -07:00
67bd4ddea0
Add a wait(delay(0)) to storage server termination to avoid a rare segfault
2021-11-02 16:24:40 -04:00
382882f1c1
mako successfully calls read_blob_granules and gets stuff back
2021-11-02 13:43:42 -05:00
d6a31078fe
C API for blob granules
2021-11-02 10:01:23 -05:00
45cff017c2
Remove Downgrade workload
2021-11-01 14:54:24 -07:00
d0c9cf4fb0
Enable mismatched-tags clang warning
2021-11-01 14:18:31 -07:00
ebcc023b6f
Enable missing-field-initializers clang warning
2021-11-01 14:18:31 -07:00
25257f6f87
Enable unused-function warning for clang
2021-11-01 14:18:31 -07:00
ee00135a6b
skip good recruitment errors when doing simulation only validation
2021-11-01 13:24:15 -07:00
78e36e7590
fix: simulation only validation could throw errors which would impact the behavior of the cluster controller
2021-11-01 13:24:15 -07:00
13bb7838aa
Enable clang -Wformat warning
2021-10-30 21:07:38 -07:00
ddf235713e
strengthen assert
2021-10-28 16:40:30 -07:00
4d8ee2ed33
fix: simple recruitment could succeed with less than the required replication factor
2021-10-28 16:38:04 -07:00
1e7338b6c3
Merge branch 'master' into bit-flipping-workload
2021-10-28 14:24:49 -07:00
88e66533ad
devFormat
2021-10-28 11:13:12 -07:00
875824b186
MVC2.0: Notify clients about relevant changes of client libraries
2021-10-27 23:43:40 +02:00
4f0991eb67
MVC2.0: Introducing client library status values for instructing clients to download and activate a library; Operations to read and change client library status
2021-10-27 23:43:40 +02:00
6174229a1b
Merge pull request #5694 from sfc-gh-vgasiunas/multi-version-client-2
...
Operations to upload and manage client binaries in the system keyspace
2021-10-27 14:28:10 -07:00
c8794ae993
MVC2.0: Adding a comment explaining buffer alignment in download & upload operations; checking additional details in testExpectedError
2021-10-27 19:40:22 +02:00
962a1aaf74
Fix race in configure database storage migration test
2021-10-27 11:38:15 -05:00
e4fd0023b7
don't disable machine team remover
2021-10-27 09:08:37 -07:00
75ef854563
format
2021-10-27 09:08:37 -07:00
db7ee9d389
disable team remover
2021-10-27 09:08:37 -07:00
14fa32f208
change boolean
2021-10-27 09:08:37 -07:00
1a2a838df3
add knob
2021-10-27 09:08:37 -07:00
c320391c4c
restartRecruiting
2021-10-27 09:08:37 -07:00
dc630d63bd
add asyncvar
2021-10-27 09:08:37 -07:00
654c0a1f14
format
2021-10-27 09:08:37 -07:00
8a10966126
wait extra time
2021-10-27 09:08:37 -07:00
d1959122af
consider wiggling when waitUntilHealthy
2021-10-27 09:08:37 -07:00
69190ed04e
format
2021-10-27 09:08:37 -07:00
0053b4793e
change knob and delete redundant doBuildTeam
2021-10-27 09:08:37 -07:00
db7b48b71c
wiggling teams calculation replace
2021-10-27 09:08:37 -07:00
3a6359e202
minus wiggling teams when build team
2021-10-27 09:08:37 -07:00
6b51e149ed
Fixing bug where SS would get removed when someone else got its tag, but its TSS pair wouldn't
2021-10-27 10:11:49 -05:00
61b8ee6eeb
Merge pull request #5327 from FuhengZhao/RedwoodSuperpage
...
Redwood Superpage Refactor from BTree to Pager concept
2021-10-27 01:40:27 -07:00
0794ebb7e8
Bump format versions for Pager and BTree as the multi-page BTree nodes are now stored in an incompatible way.
2021-10-26 22:17:55 -07:00
40da5a80f9
Merge remote-tracking branch 'apple/master' into multi-version-client-2
2021-10-26 19:29:10 +02:00
2208b04174
Merge pull request #5855 from sfc-gh-etschannen/blob_full_clean
...
Blob Granules V0
2021-10-26 09:57:35 -07:00
37bc41abbb
Merge remote-tracking branch 'apple/master' into multi-version-client-2
2021-10-26 18:51:43 +02:00
sfc-gh-tclinkenbeard