Commit Graph

52 Commits

Author SHA1 Message Date
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Josh Slocum cb918b9cef Added basic blob granule consistency check 2022-04-04 11:38:42 -05:00
Renxuan Wang 0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum 989dd8d7eb Finished implementing beginVersion 2022-03-24 17:15:11 -05:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Evan Tschannen 2c88a189a9 more pruning bug fixes 2022-03-15 13:34:59 -07:00
Evan Tschannen d46e551f11 merge 2022-03-15 13:09:10 -07:00
Josh Slocum c35e020da7 SS and BM test macros 2022-03-10 12:38:23 -06:00
Josh Slocum d7367dad99 Simplifying CF gap validation to ignore SS CF fetches 2022-03-10 11:56:22 -06:00
Josh Slocum b21d0943b9 client-focused cleanup 2022-03-09 10:01:25 -06:00
Josh Slocum f230ac0e41 Fixing more merge issues 2022-03-09 09:13:03 -06:00
Josh Slocum e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
Jingyu Zhou 1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Evan Tschannen 976375466f fixed more pruning bugs 2022-03-02 17:03:37 -08:00
Josh Slocum a14f3ff067 Do some verify tests with no pruning 2022-03-01 12:52:26 -06:00
Evan Tschannen 6f1fb71949 Fixed a number of different pruning problems 2022-03-01 10:13:20 -08:00
Evan Tschannen 1990216e89 use global knowledge inside simulation to determine when it should be possible to get a too_old error 2022-02-22 13:03:57 -08:00
Evan Tschannen 330b2b48ec improved file cleanup execution and testing 2022-02-22 12:00:09 -08:00
Josh Slocum f2b9eb1d4b Fixing potential bug of mixed-version FDB result in granule verifier 2022-01-31 10:44:09 -06:00
Evan Tschannen 966d9d9c15
Merge pull request #6273 from sfc-gh-etschannen/blob_integration
fix: do not report errors when the blob verifier is actor_cancelled
2022-01-21 12:06:34 -08:00
Evan Tschannen fee9e932e7 fix: do not report errors when the blob verifier is actor_cancelled 2022-01-21 12:06:00 -08:00
Josh Slocum 7f9bce40a7 retrying GRVs in BlobGranuleVerifier 2022-01-21 13:36:56 -06:00
Josh Slocum 4d2650f4dc Fixed a couple issues with failures and the final availability check 2022-01-07 11:21:05 -06:00
Josh Slocum 807ec37e84 BG request retry on failed blob worker properly 2022-01-05 17:38:49 -06:00
Josh Slocum bc69521a91 Several fixes with restarting BW/BM 2022-01-05 12:48:53 -06:00
Suraj Gupta a674edaa62 Address PR comments. 2021-12-10 16:43:58 -06:00
Suraj Gupta 310d990b12 Add debugging. 2021-12-10 16:43:58 -06:00
Suraj Gupta 968a4f9f50 Don't rely on database config to be updated. 2021-12-10 14:00:34 -06:00
Suraj Gupta fdc94d3356 Fix blobGranuleVerifier setup. 2021-12-10 14:00:34 -06:00
Suraj Gupta fc3376fe8f Move client knob to database config for blob granules. 2021-12-10 14:00:34 -06:00
Josh Slocum b438e98220 Fixed multiple availability bugs in blob worker 2021-12-08 15:47:16 -06:00
Josh Slocum 6c79750412 Added a bunch of debugging for tracking whenAtLatest problems, and fixed a bug there 2021-12-08 12:41:45 -06:00
Josh Slocum a82845af43 Merge branch 'master' into bg_bindings 2021-12-01 16:55:28 -06:00
sfc-gh-tclinkenbeard 07349869d9 Use fmt to address -Wformat warnings 2021-11-17 14:45:48 -08:00
Josh Slocum b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Josh Slocum d6a31078fe C API for blob granules 2021-11-02 10:01:23 -05:00
sfc-gh-tclinkenbeard 13bb7838aa Enable clang -Wformat warning 2021-10-30 21:07:38 -07:00
Josh Slocum 912ef76f1c cleanup before merge 2021-10-18 17:11:14 -05:00
Suraj Gupta e2e852e515 Mitigate transitive includes. 2021-10-18 10:49:25 -04:00
Josh Slocum b5074fd597 Reworked all of the system data to encode granule data more efficiently for persistence 2021-10-13 16:28:04 -05:00
Josh Slocum f3c44c568f fixing merge conflicts 2021-10-13 16:26:44 -05:00
Suraj Gupta 2ec8781224 Merge knobs into one. 2021-10-13 14:00:37 -04:00
Suraj Gupta 9d4b55c7fe Gate the blob verifier as well. 2021-10-13 11:45:51 -04:00
Suraj Gupta 4d54669ccd Recruit the blob workers via blob manager.
In this PR, the blob manager now recruits blob workers
(via communication with the cluster controller). Blob workers
are onboarded as blob worker processes enter the cluster.
2021-10-04 11:07:08 -04:00
Josh Slocum a986679b7f More bug fixes 2021-09-24 10:02:02 -05:00
Josh Slocum 57176a4955 Increasing workload request count 2021-09-16 10:03:30 -05:00
Josh Slocum c9b8bdffbe Passes a single correctness test! 2021-09-15 17:18:04 -05:00
Josh Slocum eb76343dfb Added blob granule reassignment and splitting 2021-09-08 14:09:14 -05:00
Josh Slocum 46adada5ff Cleaned up debugging and fixed a couple bugs 2021-08-31 12:30:43 -05:00