Commit Graph

183 Commits

Author SHA1 Message Date
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Josh Slocum aaaf42525a misc bg operational fixes and improvements 2022-04-05 12:26:00 -05:00
Josh Slocum cb918b9cef Added basic blob granule consistency check 2022-04-04 11:38:42 -05:00
Josh Slocum 377e252fcf
Better split sizing in blob manager (#6725) 2022-04-01 16:09:46 -07:00
Josh Slocum 7fc6dfa6c5 Adding useful debugging trace events 2022-03-29 14:48:28 -05:00
Jingyu Zhou da0673ccce
Merge pull request #6705 from RenxuanW/another
Add proxy option to backup and restore params.
2022-03-29 11:36:13 -07:00
Josh Slocum 2f8e9d9de0 misc bg fixes 2022-03-29 08:05:52 -05:00
Renxuan Wang 0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum 1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
Josh Slocum 0f9e88572a Cleaning up debugging and fixing race in blob manager recruitment 2022-03-17 14:57:43 -05:00
Josh Slocum a85b578d2b Added simulation validation that multiple blob maangers don't have same epoch 2022-03-17 14:57:43 -05:00
Evan Tschannen 2c88a189a9 more pruning bug fixes 2022-03-15 13:34:59 -07:00
Evan Tschannen d46e551f11 merge 2022-03-15 13:09:10 -07:00
Josh Slocum ce5b567971 Cleanup the cleanup 2022-03-15 15:02:28 -05:00
Josh Slocum 9d9b70abd4 Fixing another BM recovery and granule splitting race 2022-03-14 18:56:44 -05:00
Josh Slocum 8c35ceb684 switching bg file keys to sort on version then type, and using that to fix race where 2 blob workers try to split the same granule at different versions 2022-03-14 17:38:31 -05:00
Josh Slocum 1c14d4674d Fixing more BM races with granule splitting 2022-03-14 16:11:17 -05:00
Josh Slocum 3e278e8580 asserts for assign is empty is false if BM dies and rangeAssigner actor is dead 2022-03-14 15:53:09 -05:00
Josh Slocum e7aa534cae Forgot to update worker mapping after picking worker 2022-03-14 15:04:43 -05:00
Josh Slocum 8d88b7ca41 making range assigner boundary changes fully synchronous to ensure no races with things reading assignments 2022-03-14 14:39:00 -05:00
Josh Slocum 67dd3d071e Fixing incorrect assert in BM split handling 2022-03-11 10:50:46 -06:00
Josh Slocum 479ac313ca Fixing conflict handling in BM recovery 2022-03-10 16:35:08 -06:00
Josh Slocum 903f7adbc4 more cleanup 2022-03-10 13:53:06 -06:00
Josh Slocum c35e020da7 SS and BM test macros 2022-03-10 12:38:23 -06:00
Josh Slocum ba35e7908a off by one 2022-03-10 12:11:44 -06:00
Josh Slocum 4b254d259c Ensuring BM split retry is idempotent 2022-03-10 11:54:57 -06:00
Josh Slocum c8c97e0256 Blob Worker focused cleanup 2022-03-10 09:55:23 -06:00
Josh Slocum 1f964ac085 BM focused cleanup 2022-03-09 15:04:17 -06:00
Josh Slocum 9dbb6d216e SS-focused cleanup 2022-03-09 10:39:04 -06:00
Josh Slocum b21d0943b9 client-focused cleanup 2022-03-09 10:01:25 -06:00
Josh Slocum f230ac0e41 Fixing more merge issues 2022-03-09 09:13:03 -06:00
Josh Slocum e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
Jingyu Zhou 1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Evan Tschannen 976375466f fixed more pruning bugs 2022-03-02 17:03:37 -08:00
Josh Slocum e0578f24d2 Fixes for range streaming BG snapshot 2022-03-01 12:45:58 -06:00
Evan Tschannen 6f1fb71949 Fixed a number of different pruning problems 2022-03-01 10:13:20 -08:00
Evan Tschannen 8f8987ad67 Merge branch 'blob_integration' of https://github.com/apple/foundationdb into blob_integration 2022-03-01 10:12:36 -08:00
Josh Slocum 06f58f848d Improving change feed validation, and another popped while reading race fixed 2022-03-01 12:05:55 -06:00
Josh Slocum c27204d401 more BM split race fixes 2022-03-01 07:18:43 -06:00
Josh Slocum 839edbf5ea moved granule split evaluation to not be specific to any individual blob worker, to simplify it and handle additional cases 2022-02-28 16:49:58 -06:00
Josh Slocum 642aab2b8b blob manager range assignment race fixes 2022-02-28 16:12:27 -06:00
Josh Slocum a556582041 Fixes for new BM reecovery 2022-02-28 10:36:28 -06:00
Josh Slocum 921be1dedf fixed revoking when same worker has granule 2022-02-28 08:44:12 -06:00
Josh Slocum ed9f95b299 simplifying blob manager recovery by always treating the DB granule mapping as the source of truth, and persisting all boundary changes there BEFORE updating workers 2022-02-25 14:46:14 -06:00
Evan Tschannen 940f26cb0c Merge branch 'blob_integration' of https://github.com/apple/foundationdb into blob_integration 2022-02-25 11:00:13 -08:00
Josh Slocum bc7cc984b0 Fixing BGVerifyBalance test killing issues 2022-02-25 11:30:21 -06:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Josh Slocum 0714b7a250 Fixed blob manager split and reassign race 2022-02-23 08:48:45 -06:00
Evan Tschannen 330b2b48ec improved file cleanup execution and testing 2022-02-22 12:00:09 -08:00
Josh Slocum 38a75a8b89 Merge branch 'main' into blob_integration 2022-02-17 17:47:38 -06:00