Commit Graph

27 Commits

Author SHA1 Message Date
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Josh Slocum 1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Josh Slocum 8c35ceb684 switching bg file keys to sort on version then type, and using that to fix race where 2 blob workers try to split the same granule at different versions 2022-03-14 17:38:31 -05:00
Josh Slocum b21d0943b9 client-focused cleanup 2022-03-09 10:01:25 -06:00
Josh Slocum d58b0325b3 Added write-hot granule splitting and splitting based on write rate 2022-02-02 09:45:18 -06:00
Josh Slocum d0113a6776 Added mechanism for blob manager to poll blob workers for their granule assignments, and used that to improve manager recovery 2022-01-31 19:59:41 -06:00
Josh Slocum 62acbcfe19 Added explicit error for old blob manager instead of attaching it to response objects 2022-01-20 14:43:08 -06:00
Suraj Gupta 5a9d9921d0 Fixes and final cleanup for BM failure handling 2021-10-26 16:16:00 -04:00
Suraj Gupta 17b30f188a Working impl 2021-10-26 16:16:00 -04:00
Suraj Gupta 316a60b8cf Cleanup comments for BlobWorkerInterface. 2021-10-18 17:45:51 -04:00
Josh Slocum b5074fd597 Reworked all of the system data to encode granule data more efficiently for persistence 2021-10-13 16:28:04 -05:00
Josh Slocum f3c44c568f fixing merge conflicts 2021-10-13 16:26:44 -05:00
Suraj Gupta dfb9655c57 Handle blob work failure 2021-10-13 14:40:26 -04:00
Suraj Gupta 95166796cd Address PR comments. 2021-10-04 20:16:22 -04:00
Suraj Gupta 12870ef356 Serialize all streams. 2021-10-04 11:07:08 -04:00
Suraj Gupta 4d54669ccd Recruit the blob workers via blob manager.
In this PR, the blob manager now recruits blob workers
(via communication with the cluster controller). Blob workers
are onboarded as blob worker processes enter the cluster.
2021-10-04 11:07:08 -04:00
Suraj Gupta a4bcd3919d Add exclusive process class for Blob Worker.
Also introduces a specific machine in the simulated cluster
to test blob worker (similar to what's done for storage cache).
2021-09-23 16:54:44 -04:00
Josh Slocum 5ddf08dfe5 Got basic range reassignment working 2021-09-22 16:48:44 -05:00
Josh Slocum eb76343dfb Added blob granule reassignment and splitting 2021-09-08 14:09:14 -05:00
Josh Slocum 3b011408f8 Added sequence numbers and locks to blob worker and manager 2021-08-27 16:33:07 -05:00
Josh Slocum 714aca4f3c Refactored blob granule reads to actually be in NativeAPI 2021-08-25 15:01:38 -05:00
Josh Slocum fb2eef38fc Changing API and file format to full V1 specification 2021-08-24 10:05:46 -05:00
Josh Slocum 2ae447eaaa Refactored blob worker/manager to be in separate files 2021-08-23 14:16:09 -05:00
Josh Slocum 247602753d Blob worker handles multiple ranges now 2021-08-10 11:19:43 -05:00
Josh Slocum e5b0cf20aa Reader can read blob worker interface and mapping from db, read files, apply mutations, and return result 2021-08-06 11:02:05 -05:00