Commit Graph

18056 Commits

Author SHA1 Message Date
Evan Tschannen a546fb63ea
Merge pull request #5985 from sfc-gh-etschannen/feature-changefeed-empty-versions
Added a whenAtLeast function to change feeds to efficiently learn about empty versions
2021-11-15 10:51:28 -08:00
Markus Pilman daf6dc22d4
Merge pull request #5959 from mpilman/features/apple-silicon-3
FDB compiles on Apple Sillicon
2021-11-15 11:21:28 -07:00
Vaidas Gasiunas d7b5f9438a
Merge pull request #5972 from sfc-gh-ajbeamon/fix-client-threads-per-version-option-issues
Fix issues with setting CLIENT_THREADS_PER_VERSION option
2021-11-15 19:04:54 +01:00
Evan Tschannen 94a51e57a5 Merge branch 'master' into feature-changefeed-empty-versions
# Conflicts:
#	fdbclient/StorageServerInterface.h
2021-11-14 19:13:05 -08:00
Evan Tschannen 6909754b21 changefeeds now have a whenAtLeast function for efficiently learning when the version has updated but no mutations have been committed 2021-11-14 19:08:46 -08:00
sfc-gh-tclinkenbeard 0ba77ea79b Fix proxySnapCreate trace typo 2021-11-14 16:12:28 -08:00
sfc-gh-tclinkenbeard dc756228f2 Make snapshot errors more descriptive 2021-11-14 13:46:17 -08:00
Tao Lin c8cbb03f51
Merge pull request #5978 from nblintao/only-snapshot 2021-11-12 16:20:43 -08:00
Tao Lin 9422b8e5f2 Restricted getRangeAndFlatMap to snapshot 2021-11-12 15:12:37 -08:00
Steve Atherton 508429f30d
Redwood chunked file growth and low priority IO starvation prevention (#5936)
* Redwood files now growth in large page chunks controlled by a knob to reduce truncate() calls for expansion.   PriorityMultiLock has limit on consecutive same-priority lock release.  Increased Redwood max priority level to 3 for more separation at higher BTree levels.

* Simulation fix, don't mark certain IO timeout errors as injected unless the simulated process has been set to have an unreliable disk.

* Pager writes now truncate gradually upward, one chunk at a time, in response to writes, which wait on only the necessary truncate operations.   Increased buggified chunk size because truncate can be very slow in simulation.

* In simulation, ioTimeoutError() and ioDegradedOrTimeoutError() will wait until at least the target timeout interval past the point when simulation is sped up.

* PriorityMultiLock::toString() prints more info and is now public.

* Added queued time to PriorityMultiLock.

* Bug fix to handle when speedUpSimulation changes later than the configured time.

* Refactored mutation application in leaf nodes to do fewer comparisons and do in place value updates if the new value is the same size as the old value.

* Renamed updatingInPlace to updatingDeltaTree for clarity.  Inlined switchToLinearMerge() since it is only used in one place.

* Updated extendToCover to be more clear by passing in the old extension future as a parameter.  Fixed initialization warning.
2021-11-12 13:47:07 -08:00
John Brownlee 2f2e5219da Move the config file scheme into a separate package so it can be imported by the operator.
Move the argument for the number of servers per pod into a command-line argument so we can use the same config file at different storage server densities.
2021-11-12 11:58:38 -08:00
sfc-gh-tclinkenbeard 62efeb6812 Merge remote-tracking branch 'origin/master' into add-format-warning 2021-11-12 11:50:36 -08:00
Steve Atherton e752fdd69f
Optimization in FlowTransport's use of XXHash to avoid a malloc() when using the stream API and to not use the stream API when a message being sent is contiguous within a single packet buffer block. (#5970) 2021-11-12 11:45:21 -08:00
Tao Lin 08fd69e787
Merge pull request #5967 from nblintao/flatmap-exception 2021-11-12 10:19:44 -08:00
Daniel Smith 019fd50f46
Merge pull request #5971 from Daniel-B-Smith/rocksdb-error-logging
Clean up RocksDB error logging
2021-11-12 13:17:10 -05:00
A.J. Beamon d129b726e1 Don't use assertion for bad option usage; don't disable local client until setup network is called. 2021-11-12 10:12:49 -08:00
Ata E Husain Bohra 82c3e8bf79
Trigger buildTeam operation if server transition from unhealthy -> healthy (#5930)
* Trigger buildTeam operation if server transition from unhealthy -> healthy

DataDistribution actor helps in building teams as server count changes
(add/removal), however, it is possible that total_healthy_server count
is insufficient to allow team formation. If happens, even healthy server
count recover, the buildTeam operation will not be triggered.

Patch proposal is to trigger `checkBuildTeam` operation if server
transitions from unhealthy -> healthy state. Incase system already
has created enough teams (desiredTeamCount/maxTeamCount), the operation
incurs a very minimal cost.
2021-11-12 09:41:01 -08:00
Tao Lin 2d5f924278 GetKeyValuesAndFlatMap should return error if not retriable 2021-11-12 09:35:28 -08:00
Daniel Smith 9dccb0131e Clean up RocksDB error logging 2021-11-12 12:14:12 -05:00
He Liu 80d6debe14
Merge pull request #5966 from liquid-helium/pre-split-interfs
Interface for pre-split keyranges
2021-11-11 21:41:37 -08:00
He Liu d73d2144fd Adjust distributorSplitRange order. 2021-11-11 20:28:55 -08:00
He Liu 9f4a63dacf Adjusted endpoints order to be compatible. 2021-11-11 17:45:08 -08:00
Andrew Noyes 092af66198
Merge pull request #5965 from sfc-gh-anoyes/anoyes/fix-heap-use-after-free
Fix (test only) heap use after free
2021-11-11 17:27:12 -08:00
Renxuan Wang d9965e0e5a Add DNS mock in SimExternalConnection.
This is a substep in supporting hostnames in cluster files. So that in simulation, we can add <hostname, ip addresses> mappings to mock DNS.
2021-11-11 17:05:05 -08:00
He Liu 984bc0fbea Added Endpoints. 2021-11-11 16:56:04 -08:00
Andrew Noyes 5c9af1fdba Remove unnecessary calls to Standalone::contents 2021-11-11 16:39:39 -08:00
Andrew Noyes ed46a81bbc Fix (test only) heap use after free 2021-11-11 16:05:33 -08:00
Tao Lin 6dcc7f8ef0
Merge pull request #5955 from nblintao/change-getKeyValuesAndFlatMap-endpoint-order 2021-11-11 09:07:16 -08:00
Josh Slocum 329091e14f Merge branch 'master' into bg_bindings 2021-11-11 10:13:37 -06:00
Markus Pilman 28dde27cb1 Fix Linux compiler errors 2021-11-11 08:49:51 -07:00
Josh Slocum 77fa1e701a cleanup 2021-11-11 09:43:23 -06:00
Markus Pilman 0e6dd46f5a Correct code formatting 2021-11-11 08:38:25 -07:00
Josh Slocum b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Markus Pilman 0dfb72176e
Merge pull request #5857 from sfc-gh-vgasiunas/notify-client-lib-changes
A mechanism to notify MVC about relevant client library status changes on the cluster
2021-11-11 07:43:20 -07:00
Markus Pilman 5af465aa29 FDB compiles on Apple Sillicon 2021-11-10 20:05:38 -07:00
Lukas Joswiak e4c3f886da Fix recovery issue 2021-11-10 16:15:13 -08:00
Tao Lin 95aa3f9e00 Change SSI endpoints order to be consistent with 7.0 2021-11-10 14:22:06 -08:00
Lukas Joswiak e0953c381d Remove unused unit tests 2021-11-10 13:33:49 -08:00
Lukas Joswiak 069a04c5e5 Removed outdated definition 2021-11-10 13:33:49 -08:00
Lukas Joswiak 1da288822f Remove distributed trace database option 2021-11-10 13:33:49 -08:00
Lukas Joswiak 8bc0c3e8a2 Convert distributed trace sampling rate to a database option 2021-11-10 13:33:49 -08:00
Lukas Joswiak 28b72550f3 Remove additional unused tracing 2021-11-10 13:33:49 -08:00
Lukas Joswiak fc2bfdbacf Remove unit tests dependent on knob 2021-11-10 13:33:49 -08:00
Lukas Joswiak c93052121f Fix issue where transaction spans would not be recorded 2021-11-10 13:33:49 -08:00
Lukas Joswiak 5ed7abdc05 Fix bug where spans were not being properly constructed 2021-11-10 13:33:49 -08:00
Daniel Smith 481cf9bb55
Merge pull request #5788 from Daniel-B-Smith/rocks-throttle
Throttle the number of concurrent reads to RocksDB
2021-11-10 15:20:30 -05:00
Daniel Smith 499dbcdb18 Don't fail fetchKeys when server overloaded is returned 2021-11-10 14:15:42 -05:00
Andrew Noyes db3c08c7cd
Merge pull request #5928 from sfc-gh-anoyes/anoyes/fix-heap-use-after-free
Fix a heap use after free
2021-11-10 10:21:05 -08:00
Vaidas Gasiunas 51b8ccf7d3 Merge remote-tracking branch 'apple/master' into notify-client-lib-changes 2021-11-10 18:40:34 +01:00
Daniel Smith 394b9dc619 Code review changes 2021-11-10 11:53:27 -05:00