Commit Graph

994 Commits

Author SHA1 Message Date
Josh Slocum 43543b0d0e Removing incorrect CF validation asserts 2022-02-24 12:39:15 -06:00
Josh Slocum b6d02478cd Adding stream uid tracking for ss feed debugging 2022-02-24 12:19:25 -06:00
Josh Slocum 86236b766d Added print debugging for CF memory vs disk validation 2022-02-24 12:11:06 -06:00
Josh Slocum b67930cb93 Fixing bug in new change feed fetch logic 2022-02-24 11:31:36 -06:00
Josh Slocum 1de4fb1b9e Adding explicit stopVersion instead of stopped status to change feed durable data, several other minor fixes 2022-02-24 11:02:15 -06:00
Josh Slocum 6739d506c4 More fixes for handling change feed empty version >= ss version 2022-02-22 16:23:32 -06:00
Josh Slocum d89c6bb3de Fixing initial change feed stream reply to be in try/catch 2022-02-22 13:55:40 -06:00
Josh Slocum 5dca00a21d Cleaning up change feed on move triggers when stream is cancelled 2022-02-22 13:48:57 -06:00
Josh Slocum e75ab5f3a0 not advancing popped version of change feed sideband from another server 2022-02-22 11:22:35 -06:00
Josh Slocum f9f676abf9 Also re-persisting change feed metadata on fetch after cleanup 2022-02-17 19:21:40 -06:00
Josh Slocum ef25e4fdfb Sending empty version back immediately to establish stream connection and acknowledgement queue for change feed stream requests 2022-02-17 18:10:17 -06:00
Josh Slocum 38a75a8b89 Merge branch 'main' into blob_integration 2022-02-17 17:47:38 -06:00
Josh Slocum c87d868dd2 Fix SS restart and change feed pop races 2022-02-17 15:55:24 -06:00
Josh Slocum f60420a8e6 Correctly updating change feed storage/durable versions after fetch 2022-02-17 10:36:32 -06:00
Josh Slocum 26480c4705 change feed popping fixes 2022-02-15 15:12:23 -06:00
Josh Slocum 53573e7648 Don't reset CF fetch version on pop 2022-02-15 15:12:05 -06:00
Josh Slocum b5ff2006b2 Removing incorrect assert 2022-02-14 11:36:54 -06:00
Josh Slocum 9175ba354a Adding a separate phase for the end of change feed fetching to handle changes in fetchKeys available durability 2022-02-11 17:25:56 -06:00
Josh Slocum 0db843bc56 more CF movement and cleanup races 2022-02-11 16:04:28 -06:00
Josh Slocum 3bbddce668 Fixing shard move away and back race 2022-02-11 10:14:26 -06:00
Josh Slocum 8a56349ea3 Fixing shard move away then back then away race 2022-02-11 10:12:49 -06:00
Josh Slocum 08505325d3 Fixing change feed mapping in cleanup refactor 2022-02-11 08:41:01 -06:00
Josh Slocum f9c16ed937 misc change feed improvements 2022-02-11 08:14:00 -06:00
Josh Slocum 36742bcd7e Fixing durability model of change feed cleanup 2022-02-10 18:52:17 -06:00
Josh Slocum 631cf0b085 minor memory optimizations for SS change feeds 2022-02-10 12:21:22 -06:00
Josh Slocum f997b2dab4 Smarter change feed cancelling when data moved away 2022-02-10 12:21:22 -06:00
Josh Slocum 4af1b24c74 Using higher transferred version when fetching change feeds 2022-02-10 08:00:52 -06:00
Josh Slocum 9189ea0448 Fixed SS restart and mutation log race 2022-02-09 15:37:09 -06:00
Josh Slocum d7955a9017 Much better mechanism for change feed popped checking 2022-02-08 14:23:04 -06:00
Josh Slocum 6adf119e3d Fixing change feed durableFetchVersion not getting updated 2022-02-08 13:59:06 -06:00
Josh Slocum 90a870962d Unblocking bounded change feed request in changeFeedClientVersions once it hit end 2022-02-08 13:22:17 -06:00
Sreenath Bodagala 72e06369a4 Merge remote-tracking branch 'apple-upstream/main' into version-vector-prototype 2022-02-08 17:47:57 +00:00
Josh Slocum 9885d2cc90 More change feed fetch and read race fixes 2022-02-07 16:21:38 -06:00
Josh Slocum 9b3f6fe452 More bugs in streaming split 2022-02-07 13:25:14 -06:00
Josh Slocum 88cab7fb67 More change feed fetching improvements and optimizations 2022-02-07 11:19:51 -06:00
Josh Slocum ddfc301d74 Improving memory footprint of change feeds and making it configurable 2022-02-04 16:41:25 -06:00
Josh Slocum 0a3608a33b Fixed issue with change feed fetching durability 2022-02-04 11:03:29 -06:00
Josh Slocum 03d91fd42d Fixing change feed fetch durability semantics 2022-02-04 09:41:39 -06:00
Josh Slocum a974773676 Handled change feed getting moved away or cancelled during move 2022-02-04 09:33:04 -06:00
Josh Slocum d0c4c6856f Fix pop at end of CF fetch 2022-02-03 12:38:37 -06:00
Josh Slocum 0f9b2f7c48 Change feed fetch refactor and pop during fetch improvements 2022-02-03 10:49:30 -06:00
Josh Slocum b324cf4bc4 Fixing TSS buggify in restarting tests 2022-02-03 06:46:28 -06:00
Josh Slocum d3f9c01120 Handling change feed fetch for feeds added during fetching a range 2022-02-02 15:21:11 -06:00
Josh Slocum f187d01b71 Improving change feed performance and fixing durability issue 2022-02-02 11:41:29 -06:00
Josh Slocum d0e89ecdd5 Including Change Feeds in fetch keys budget 2022-02-01 14:39:10 -06:00
Josh Slocum 2c22d07b39 Several change feed popping fixes for behind SS 2022-02-01 10:27:41 -06:00
Josh Slocum 25a0d857fa Fixing issue with change_feed_popped 2022-01-31 17:19:52 -06:00
Josh Slocum d4931348c1 Fix for new change feed movement 2022-01-31 11:18:47 -06:00
Josh Slocum c3cece552c Fixes for previous 2022-01-28 15:19:00 -06:00
Josh Slocum cf7ed4ee40 Simplify change feed fetching to not inject anything at all during fetch and rely on reading from the remote feeds 2022-01-28 14:21:26 -06:00
A.J. Beamon 0803abedaa Adjust the name order of constructor variable initialization to match declaration order. 2022-01-28 10:27:13 -08:00
Josh Slocum 2d3b216f60 Persisting change feed popped version in all cases to enable popped checking 2022-01-28 12:27:04 -06:00
Josh Slocum ac1fd056dd Added change feed popped read guard for blob workers 2022-01-28 10:45:33 -06:00
Josh Slocum 10c3cc870f Fixing new race in change feed cleanup while still working for fetch+cleanup race 2022-01-28 09:26:36 -06:00
Josh Slocum 91df569fdd Fetch Change Feed metadata at the start of fetchKeys to fix mutation race 2022-01-27 18:56:07 -06:00
Josh Slocum c26e11c2c3 Adding extra debugid to change feed stream requests to track things better 2022-01-27 18:53:27 -06:00
Dan Lambright fe1ec33d32 Fix failures in flatmap tests 2022-01-27 16:19:33 -05:00
Josh Slocum bc7b76b407 Fix for gotAll in getChangeFeedMutations 2022-01-27 14:25:08 -06:00
Josh Slocum ce5faa470b Merge branch 'ss_cf_fix' into blob_integration 2022-01-26 16:49:08 -06:00
Josh Slocum 992fc8ed7c Using non-expanded clears in change feeds 2022-01-26 15:28:58 -06:00
Andrew Noyes c967e6246b
Fix uninitialized memory in storage server (#6282) 2022-01-26 13:24:35 -08:00
Josh Slocum 311a7656c0 Fixing change feed getting stuck not quite caught up, which causes empty version problems 2022-01-26 12:22:20 -06:00
He Liu 5e0b1fd13f Resolved comments. 2022-01-21 16:58:42 -08:00
He Liu 0421562ce9 Resolved comments. 2022-01-21 16:58:42 -08:00
He Liu b7388a714b Added get, scan, and commit counters. 2022-01-21 16:58:42 -08:00
He Liu da8453eb5d Added systemClearRange. 2022-01-21 16:58:42 -08:00
He Liu f14c832031 Added storage engine metrics in SS. 2022-01-21 16:58:42 -08:00
Josh Slocum 1180eb6e44 Fixed uncaught error in blob worker requests 2022-01-21 18:02:30 -06:00
Dan Lambright 1f67250402 Fix problem with PROXY_USE_RESOLVER_PRIVATE_MUTATIONS in resolver. 2022-01-21 10:39:06 -05:00
Josh Slocum 3be1bcd588 Fix move race in change feeds and added extra debugging to track down similar problems 2022-01-20 14:42:24 -06:00
Dan Lambright 9544379cdf rebase 2022-01-20 11:12:33 -05:00
Josh Slocum 215f5fae93 Reworked change feed initialization to handle more types of races 2022-01-19 15:20:23 -06:00
Josh Slocum 79704aa05f Fixing change feed private mutation handling 2022-01-18 20:17:39 -06:00
Josh Slocum 05d2eafd0c Correctly updating change feed storage and durable versions after a fetch 2022-01-12 18:28:31 -06:00
Josh Slocum cdb433569e fixed range bug in filterMutationsInverted 2022-01-12 13:17:11 -06:00
Evan Tschannen a7421e6ca2 fix: localChangeFeeds cannot get wrong_shard_server errors 2022-01-10 12:59:30 -08:00
Josh Slocum b0aea91895 Broadening explicit disconnect handling to explicit error handling of all types 2021-12-21 14:12:09 -06:00
A.J. Beamon d8e161f89e Refactor NativeAPI transactions to create and pass around a reference counted state object. Watches no longer use the tranasction info object but instead use their own state. 2021-12-17 11:57:39 -08:00
Josh Slocum f6ea67120e Adding explicit empty version back to change feeds for now 2021-12-10 10:04:05 -06:00
Evan Tschannen 9dcdda8192 fix: do not merge rollbacks when fetching 2021-12-07 20:34:01 -08:00
Andrew Noyes def41697bf
Merge pull request #6083 from sfc-gh-tclinkenbeard/remove-temporaries
Avoid creating unnecessary temporary objects
2021-12-06 13:24:56 -08:00
Evan Tschannen 98b4299fb2 .error() on a trace event must be before the details 2021-12-05 14:59:46 -08:00
Tao Lin 9b0a9c4503
Return error when getRangeAndFlatMap has more & Improve simulation tests (#6029) 2021-12-03 12:50:07 -08:00
Josh Slocum 03f341fd3a Merge branch 'bw_new_empty_versions' into blob_integration 2021-12-02 15:01:31 -06:00
Josh Slocum d85eb330e0 retooling some waitForVersion stuff and adding asserts 2021-12-02 14:52:16 -06:00
Josh Slocum 9f6c8a123b Fix: change feed update ordering 2021-12-02 13:29:30 -06:00
Josh Slocum 85c6e30aec Fix: several empty version issues with large single-version change feed entries 2021-12-02 09:56:15 -06:00
sfc-gh-tclinkenbeard 3b5d23ef88 Remove unnecessary temporary objects when appending to vector of pairs 2021-12-01 18:15:16 -08:00
Evan Tschannen f12ac4468e fix: fetched change feeds did not get their durable version updated 2021-11-30 13:32:12 -08:00
Josh Slocum efb21ca6a1 Merge branch 'master' into blob_integration 2021-11-24 13:17:54 -06:00
Evan Tschannen b0aad44831 change feed requests with an explicit end need to get empty versions at the end boundary 2021-11-23 15:01:44 -08:00
Evan Tschannen 3360000a91 fix: an empty mutation vector did not mean that all of the data was returned 2021-11-23 15:01:44 -08:00
Evan Tschannen e97d337ffe fix: when a feed transitions from atLatest to not atLatest stall all updates at the blocked version 2021-11-23 15:01:44 -08:00
Evan Tschannen 10a925b7e9 do not return mutations larger than the dequeVersion 2021-11-23 15:01:44 -08:00
Evan Tschannen f3bb1d8f51 properly handle an active change feed stream when removed 2021-11-23 15:01:44 -08:00
Jon Fu 8f6934c4d0 Merge branch 'master' of github.com:apple/foundationdb into jfu-fix-ss-segfault 2021-11-16 18:03:32 -05:00
Evan Tschannen 964d0209ca
Merge pull request #5637 from sfc-gh-ljoswiak/features/data-loss-prevention
Data loss protection when joining new cluster
2021-11-15 15:26:32 -08:00
Josh Slocum f140028086 Merge branch 'master' into blob_integration 2021-11-15 15:09:40 -06:00
Evan Tschannen 6e81b83924 fix: cleanup change feeds which have been completely removed from a storage server 2021-11-15 11:47:42 -08:00
Josh Slocum 1d1f3fe747 changing assert for now 2021-11-15 12:24:51 -06:00
Josh Slocum fd4f13fba1 Added fetch version to change feeds to avoid races and overwrites between updateStorage and fetchChangeFeed 2021-11-15 09:49:51 -06:00
Evan Tschannen 94a51e57a5 Merge branch 'master' into feature-changefeed-empty-versions
# Conflicts:
#	fdbclient/StorageServerInterface.h
2021-11-14 19:13:05 -08:00
Evan Tschannen 6909754b21 changefeeds now have a whenAtLeast function for efficiently learning when the version has updated but no mutations have been committed 2021-11-14 19:08:46 -08:00
Tao Lin 2d5f924278 GetKeyValuesAndFlatMap should return error if not retriable 2021-11-12 09:35:28 -08:00
Lukas Joswiak c93052121f Fix issue where transaction spans would not be recorded 2021-11-10 13:33:49 -08:00
Daniel Smith 499dbcdb18 Don't fail fetchKeys when server overloaded is returned 2021-11-10 14:15:42 -05:00
Daniel Smith b50b3de5d0 Allow SS to respond with server overloaded 2021-11-10 11:52:02 -05:00
Tao Lin fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce #5609
2021-11-09 13:52:28 -08:00
Lukas Joswiak 15e0d5b29f Add explicit transaction options when reading cluster ID 2021-11-09 12:29:49 -08:00
Lukas Joswiak 4640045243 Fix rare simulation failures
When partitions appear before a cluster has fully recovered, it was
possible to have different tlogs persist different cluster IDs because
they were involved in different partitions. This would affect recovery
when a quorum was eventually reached. The solution to this is to avoid
persisting the cluster ID before a cluster has fully recovered, to make
sure all nodes agree on the cluster ID.
2021-11-09 12:29:48 -08:00
Lukas Joswiak aa3383f0e3 Exclude when joining new cluster 2021-11-09 12:29:48 -08:00
Lukas Joswiak 3e2c65bb11 Allow tlog to join another cluster but retain its data 2021-11-09 12:29:48 -08:00
Lukas Joswiak 30867750b5 Add protection against storage and tlog data deletion when joining a new cluster 2021-11-09 12:29:47 -08:00
Jingyu Zhou b4ed47053b Remove toString for mutations in trace events 2021-11-08 10:04:16 -08:00
Jon Fu 396cd58b21 cancel ss core and ss actor collection after termination and before context switch 2021-11-04 16:05:23 -04:00
Tao Lin 586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin 6c98e35893 Rename Hop to FlatMap 2021-11-03 13:32:01 -07:00
Tao Lin 0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Jon Fu 67bd4ddea0 Add a wait(delay(0)) to storage server termination to avoid a rare segfault 2021-11-02 16:24:40 -04:00
Dan Lambright befe1993c4 fix conflict on rebase 2021-10-29 12:25:26 -04:00
Josh Slocum dbf46c200f disabling long keys and adding more debugging stuff 2021-10-28 13:53:54 -05:00
Josh Slocum 6b51e149ed Fixing bug where SS would get removed when someone else got its tag, but its TSS pair wouldn't 2021-10-27 10:11:49 -05:00
Josh Slocum 0ff8ddc2b6 Merge branch 'master' into blob_full_clean 2021-10-25 13:38:48 -05:00
Evan Tschannen 118c307b57 fixed formatting 2021-10-24 22:26:11 -07:00
Evan Tschannen 9a6384fc26 fixed merge conflicts 2021-10-24 21:18:49 -07:00
Evan Tschannen 6f7558b8ea Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	tests/CMakeLists.txt
2021-10-24 21:06:33 -07:00
A.J. Beamon e882eb33fc Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem. 2021-10-22 11:05:18 -07:00
Steve Atherton d153519188
Merge pull request #5813 from sfc-gh-jslocum/ss_ebrake_streaming_fix
Fixes to ss e-brake, tlog streaming, and their interaction
2021-10-22 10:46:17 -07:00
Josh Slocum 773886515e Merge branch 'feature-range-feed' into blob_full_clean 2021-10-22 11:07:51 -05:00
Evan Tschannen f03d32f3d4 fix: handle the case where a fetch happens at an earlier read version than the commit version of the change feed registration 2021-10-21 23:04:51 -07:00
He Liu 16ae2b76e5 Merge branch 'master' of https://github.com/apple/foundationdb into clean-sim-test-data-loss 2021-10-21 09:16:53 -07:00
Evan Tschannen f1158371a7 Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	flow/error_definitions.h
2021-10-21 00:55:12 -07:00
Evan Tschannen 4e79296a9f fixed a few bugs with fetching change feeds 2021-10-21 00:44:51 -07:00
Evan Tschannen 3ebabb6edc fixed incorrect use of change feed errors 2021-10-20 22:37:31 -07:00
Evan Tschannen e34d242581 fix: do not throw wrong_shard_server for local fetch keys 2021-10-20 22:04:39 -07:00
Josh Slocum 8dd7f8f447 Fixes to ss e-brake, tlog streaming, and their interaction 2021-10-20 10:48:29 -05:00
Evan Tschannen ad3dcd6a74 fix: memory replies were not being set 2021-10-19 21:07:36 -07:00
Evan Tschannen 3f7df58a77 fixed a number of issues 2021-10-19 13:56:52 -07:00
Josh Slocum 85f64bf42c more cleanup before merge 2021-10-18 17:11:14 -05:00
Josh Slocum 912ef76f1c cleanup before merge 2021-10-18 17:11:14 -05:00
Sreenath Bodagala 246f035afe Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 2021-10-18 19:07:33 +00:00
Daniel Smith 9713a14ef1 Reverse order of read type and debug ID args 2021-10-18 12:23:09 -04:00
Daniel Smith df53cc9580 Add an enum to IKeyValueStore to indicate the source/priority of the read 2021-10-15 14:35:59 -04:00
He Liu 66166e09be Clear range before setting the moved-in empty range as available. 2021-10-15 09:30:06 -07:00
He Liu dbfeb06c97 Reproduced user data loss incident, and tested the improved exclude tool
can fix the system metadata.
2021-10-14 14:08:39 -07:00
Josh Slocum 5f0ec0612a Merge branch 'feature-range-feed' into blob_full 2021-10-13 15:44:35 -05:00
Evan Tschannen 282f84c807 added handling for broken promise 2021-10-12 10:36:10 -07:00
He Liu 9f974ef21f Added logs for worker_removed() errors in SS. 2021-10-12 10:12:20 -07:00
Evan Tschannen 5b6f8b2abb added the known committed version to change feeds 2021-10-11 13:53:36 -07:00
Evan Tschannen 26b6f9a3f1 fix: resize changes to source vector even though m is a copy 2021-10-10 11:32:49 -07:00
Evan Tschannen 5c642f706e Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
2021-10-09 19:34:16 -07:00
Evan Tschannen efc4cec53f fixed a variety of bugs with change feeds 2021-10-09 19:24:01 -07:00
Dan Lambright f68d018a06 conflicts 2021-10-08 12:39:39 -04:00
Josh Slocum bbe7c7ca9f found change feed liveness bug 2021-10-06 12:32:16 -05:00
Chang Liu bf0b8510dc fix the problem about initialization list disorder in storageserver.actor.cpp file
Description

Testing
2021-09-28 10:19:51 -07:00
Chang Liu 76ab937e55 fix the warn about initialization order chaos in storageserver.actor.cpp
Description

Testing
2021-09-24 11:40:04 -07:00
Chang Liu 0af8aefeee Stop rolling trace events that are associated with a process role when the role terminates
Description

Testing
2021-09-24 09:53:32 -07:00
Chang Liu 612477c3ed Fix roll trace event issue
Description

Testing
2021-09-24 09:53:32 -07:00
Chang Liu 8427e40cbe Fix roll trace event issue
Description

Testing
2021-09-24 09:53:32 -07:00
Chang Liu 48990058a3 Fix roll trace event issue
Description

Testing
2021-09-24 09:53:32 -07:00
He Liu a1f6dcc2d5 merge from master 2021-09-22 13:07:38 -07:00
Josh Slocum ccb78b1ce5 more CF bug fixes 2021-09-21 13:20:27 -05:00
Sreenath Bodagala 950de29d5a Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 2021-09-20 21:29:57 +00:00
Evan Tschannen 5f59c037af buggify the streaming byte limit 2021-09-20 13:58:05 -07:00
Josh Slocum 8e4582673c more bug fixes 2021-09-20 10:21:52 -05:00
He Liu 4d5bf08da8 address comments 2021-09-19 17:03:06 -07:00
Sreenath Bodagala 2aa3b44d4e Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
- Conflicts:
	fdbserver/LogSystem.h
	fdbserver/LogSystemConfig.h
	fdbserver/TagPartitionedLogSystem.actor.cpp

- Files modified during merge:

modified:   fdbserver/LogSystem.cpp
modified:   fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
Xiaoge Su abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
Josh Slocum c9b8bdffbe Passes a single correctness test! 2021-09-15 17:18:04 -05:00
He Liu ef7fdc0781 fmt 2021-09-15 10:32:09 -07:00
Josh Slocum cefb66d64c fixing a couple bugs 2021-09-10 12:52:33 -05:00
Josh Slocum c2d1d1704f Merge branch 'feature-range-feed' into blob_full 2021-09-10 11:21:52 -05:00
Evan Tschannen ac5b580e2d Merge branch 'master' into feature-range-feed
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/StorageServerInterface.cpp
#	fdbclient/StorageServerInterface.h
#	fdbserver/ApplyMetadataMutation.cpp
#	fdbserver/TLogServer.actor.cpp
#	flow/error_definitions.h
2021-09-09 23:13:22 -07:00
helium 65e6f26f05 bug fix on unwaited actor 2021-09-09 20:43:29 -07:00
helium 2ea055371b Introduced serverKeysTrueEmptyRange to indicate the SS that the new
shard is empty.
2021-09-09 14:15:07 -07:00
Josh Slocum eb76343dfb Added blob granule reassignment and splitting 2021-09-08 14:09:14 -05:00
Evan Tschannen 590da022ee fix: missing mutations committed at newOldestVersion 2021-09-07 16:20:49 -07:00
Evan Tschannen f53ddf6720 fix: handle the case where I change feed is added while a fetch is in progress 2021-09-06 13:51:53 -07:00
Evan Tschannen 7b23894039 fixed a few bugs related to change feed fetches 2021-09-05 20:33:12 -07:00
Evan Tschannen 9fefd59076 Fixed a number of bugs, change feeds are correctness clean with the current workload 2021-09-05 18:17:23 -07:00
Evan Tschannen af0792b1f5 fixes that allow change feeds to pass correctness in tests with no failures 2021-09-03 17:24:33 -07:00
Evan Tschannen 8309c2ce26 filter mutations in change feeds to the requested range 2021-09-03 12:06:13 -07:00
Evan Tschannen 4bbae59bb0 Fixed a few bugs and added a backup popping mechanism 2021-09-02 21:11:44 -07:00
Evan Tschannen 27151b0831 support reads from range feeds that span multiple storage servers 2021-09-01 14:35:51 -07:00
Dan Lambright 8689e1f106 merge with master 2021-08-30 15:29:08 -04:00
FDB Formatster 2c788c233d apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-08-27 17:07:47 -07:00
Daniel Smith 45b40addb7
Merge pull request #5379 from neethuhaneesha/eagerReadsDisable
Disabling option for removing eagerReads for ClearRange mutations.
2021-08-19 12:39:30 -04:00
Evan Tschannen a278d2977a renamed range feeds to change feeds, data distribution support almost complete 2021-08-13 14:27:15 -07:00
Josh Slocum e444d3781c Various TSS improvements from snowblower testing 2021-08-13 10:24:15 -05:00
Neethu Haneesha Bingi c45daf6f51 Disabling option for removing eagerReads for ClearRange mutations. 2021-08-13 01:26:50 -07:00
Josh Slocum 1a4e70d22f Merge branch 'feature-range-feed' into blob_full 2021-08-12 10:22:28 -05:00
Josh Slocum 76a3c47e42 Got full demo working 2021-08-12 10:18:58 -05:00
Lukas Joswiak a605fb3852
Merge pull request #5026 from sfc-gh-ljoswiak/fixes/alp6
Actor sampling
2021-08-11 13:44:17 -07:00
Evan Tschannen 7891477a7c added support for streaming range feeds 2021-08-10 21:28:25 -07:00
Evan Tschannen db274c2431 a number of bug fixes 2021-08-10 16:47:01 -07:00
Sreenath Bodagala cec744cebf - Address the following issues:
- Sequencer should update the version vector once for a given commit
version (irrespective of the number of times that it receives and
processes the ReportRawCommittedVersionRequest message for that commit
version). Issue found by simulation tests.

- Storage server should take both its latest commit version and the
read version into account while processing a read request. This is to
address transaction_too_old error that we saw while running tests with
mako (and also in YCSB tests).

- Do not enable the tlog blocking-peek logic if ENABLE_VERSION_VECTOR
flag is set to false.
2021-08-10 19:47:18 +00:00
Josh Slocum 921a2cfca1 Merge branch 'feature-range-feed' into blob_full 2021-08-10 11:25:48 -05:00
Steve Atherton 54c7036eaf Move role UIDs for MutationTracking TraceEvents from various inconsistent detail fields into the TraceEvent UID field. 2021-08-10 01:52:36 -07:00
Steve Atherton faa4154a56 MutationTracking now uses a vector of keys to track. Removed "Mutation" detail from DEBUG_MUTATION() events because they are duplicates of the fields already logged in the returned MutationTracking event, which are now renamed and combined into "Mutation". Removed more toString() calls in TraceEvent detail values. 2021-08-09 23:30:45 -07:00
Evan Tschannen a1b0053b57 do not return send a reply to a pop request until the clear has been made durable 2021-08-09 22:06:53 -07:00
Evan Tschannen 208a5790ad fixed usage of durable version 2021-08-09 21:58:44 -07:00
Evan Tschannen ed28aecde0 Merge branch 'master' into feature-range-feed 2021-08-09 20:40:55 -07:00
Evan Tschannen 42ae870c84 added support for querying specific range feed versions 2021-08-09 20:39:28 -07:00
Evan Tschannen 52fcf3f565 fixed a few bugs 2021-08-09 17:16:53 -07:00
Evan Tschannen b03649d627 fixed assertion error 2021-08-09 12:55:57 -07:00
Evan Tschannen 27f87471ab fixed compile errors 2021-08-09 12:48:38 -07:00
Evan Tschannen bc9a0e1315 first attempt to add data distribution support for range feeds 2021-08-09 10:05:56 -07:00
Steve Atherton 2626fdc992 Don't loop over mutations with an empty loop if mutation tracking is disabled. 2021-08-08 00:44:17 -07:00
Steve Atherton 2bf7b9b7a9 Remove many calls to .toString(), .printable(), and printable(...) from TraceEvent .detail() values to avoid double-escaping characters such that the same values look different in different events and can't be easily searched for. Also fixed a few warnings found by clang. 2021-08-08 00:04:54 -07:00
Josh Slocum e5b0cf20aa Reader can read blob worker interface and mapping from db, read files, apply mutations, and return result 2021-08-06 11:02:05 -05:00
Sreenath Bodagala a081c0baa5 Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 2021-08-05 22:40:32 +00:00
Lukas Joswiak 5dc9a97230 Merge branch 'master' into fixes/alp6 2021-08-01 20:42:52 -07:00
Lukas Joswiak 78b5635d9f Remove hot lineage modification 2021-08-01 20:21:15 -07:00
Evan Tschannen 125241743d added support for removing range feeds 2021-07-30 16:01:46 -07:00
Evan Tschannen 0989c28a6b made range feeds durable on the storage server 2021-07-30 15:23:42 -07:00
sfc-gh-tclinkenbeard c74047c665 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-28 11:51:02 -07:00
Evan Tschannen 406562b282 Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed 2021-07-27 17:04:58 -07:00
Lukas Joswiak 3eed4084e2 Merge branch 'master' into fixes/alp6 2021-07-27 11:26:53 -07:00
Lukas Joswiak 59d535149e Merge branch 'master' into fixes/alp6 2021-07-27 10:07:18 -07:00
Steve Atherton 507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
sfc-gh-tclinkenbeard e5e449b340 Fix more -Wreorder-ctor warnings in storageserver.actor.cpp 2021-07-24 17:33:29 -07:00
sfc-gh-tclinkenbeard b9a22a61ef Fix many -Wreorder-ctor warnings 2021-07-23 17:33:18 -07:00
sfc-gh-tclinkenbeard 6f81155784 Merge remote-tracking branch 'origin/master' into const-serverdbinfo 2021-07-20 10:18:40 -07:00
Steve Atherton f596a81073 Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs. 2021-07-17 00:11:40 -07:00
Evan Tschannen a5e0e88df8 fix: reply promise streams could receive messages out of order if the sender and receiver were the same process 2021-07-15 15:27:49 -07:00
Sreenath Bodagala f77093ea61 - Address compilation errors 2021-07-15 19:49:20 +00:00
Sreenath Bodagala 0dbd917012 - Address review comments 2021-07-15 19:49:20 +00:00
Sreenath Bodagala b6f89df060 - Propagate the latest commit version of a storage server as part of read request.
Make storage server read at the specified version.
2021-07-15 19:48:42 +00:00
Evan Tschannen 15fe00927a Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/StorageServerInterface.h
#	fdbclient/SystemData.cpp
#	fdbclient/SystemData.h
#	fdbserver/storageserver.actor.cpp
#	flow/ProtocolVersion.h
2021-07-12 19:10:14 -07:00
sfc-gh-tclinkenbeard edbac4a26a Prevent storageServer from modifying ServerDBInfo object 2021-07-11 22:04:58 -07:00
sfc-gh-tclinkenbeard 8cc40e3a2b Expand use of BOOLEAN_PARAM 2021-07-02 21:41:50 -07:00
sfc-gh-tclinkenbeard 79ff07a071 Added *BOOLEAN_PARAM macros to enforce documentation of boolean parameters 2021-07-02 15:04:42 -07:00
Evan Tschannen ac72120739 fixed a trace event which used fetchVersion before it was defined in the actor 2021-06-25 16:43:19 -07:00
Evan Tschannen 1cec223434 cherry pick streaming replies to master 2021-06-19 09:47:13 -07:00
Josh Slocum d1d2ca9285 Don't inject TSS faults if speedUpSimulation is set 2021-06-18 12:41:48 -05:00
Neethu Haneesha Bingi 2f9cd71441 Histograms for storage server write path components. 2021-06-11 14:48:13 -07:00
Josh Slocum 65c49fc1e4
Merge pull request #4934 from sfc-gh-jslocum/tss_quarantine
Added TSS Quarantine and fixed TSS mapping bug in CommitProxy
2021-06-11 11:52:42 -07:00
Tao Lin 49211fd663 More fixes to wording 2021-06-09 16:47:04 -07:00
Tao Lin 568d1a97b6 Remove duplicated logging and improve comments/style 2021-06-09 16:47:04 -07:00
Tao Lin f64a73cc07 Consolidate the counters and add more comments 2021-06-09 16:47:04 -07:00
Tao Lin a00fa584c0 Metrics to compare the bandwidth used by data distributions and updates
And more comments about how data distribution and updates work in storage server.
2021-06-09 16:47:04 -07:00
Josh Slocum c6d96c4124 Added TSS Quarantine and fixed TSS mapping bug in CommitProxy 2021-06-09 14:42:50 +00:00
Andrew Noyes 402622ace9
Merge pull request #4909 from apple/anoyes/fix-ub
Fix several instances of undefined behavior
2021-06-07 08:58:45 -07:00
Lukas Joswiak 486a04659f Lazy inititialization 2021-06-04 15:01:18 -07:00
Lukas Joswiak 153de33f57 Revert "Merge pull request #4802 from sfc-gh-ljoswiak/revert/actor-lineage"
This reverts commit 6499fa178e, reversing
changes made to 1512631957.
2021-06-04 13:31:55 -07:00
Andrew Noyes ce25a99000 Disallow conversion from float in specialCounter 2021-06-04 12:09:13 -07:00
Josh Slocum b3e4f182ef TSS Mapping Change 2021-06-02 17:30:09 +00:00
Josh Slocum f6253db7dc Addressing final PR comments 2021-05-28 18:19:42 +00:00
Josh Slocum 4257ac2b4d More TSS Changes/Fixes 2021-05-25 20:37:48 +00:00
Josh Slocum ce82c9653e Testing Storage Server implementation 2021-05-25 20:28:50 +00:00