Commit Graph

196 Commits

Author SHA1 Message Date
Josh Slocum b2a96b64e6 refactoring one function into two 2022-07-13 07:49:19 -05:00
Josh Slocum f707626213 Cleanup 2022-07-12 17:43:44 -05:00
Josh Slocum 0b0ac16a4c Merge branch 'main' into granule_merging 2022-07-12 09:09:30 -05:00
Josh Slocum 48bcf6eaba
parameterizing min split size instead of hardcoding it to min_shard_bytes (#7404) 2022-06-30 12:28:22 -07:00
A.J. Beamon 26b35c07cd Refactor how tenant map entries are encoded and decoded. Add a specific version to the encoding that matches the version used when this feature was introduced (and the only version in which it was used). 2022-06-29 10:58:58 -07:00
A.J. Beamon 4bafe77889 Some refactoring of tenant code:
* extract tenant management into its own file and namespace
* rename the tenant management workload source file
* extract tenant special keys functions to a separate file
* extract some helper functions to GenericTransactionHelper.h
* convert StringRef -> TenantNameRef
* move some TenantMapEntry implementation into the cpp file
* add some helper functions to decode/encode a tenant mode
2022-06-27 12:32:49 -07:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Josh Slocum 710612442c small fixes 2022-06-09 17:21:21 -05:00
Josh Slocum d6920cde28 Implemented blob granule merging 2022-06-09 10:50:53 -05:00
Josh Slocum ae865027d6 fixes for blob metadata memory from valgrind 2022-06-07 13:50:11 -05:00
Josh Slocum fcd20c479d addressing review comments 2022-06-03 08:36:07 -05:00
Josh Slocum 567b1d35f7 Full integration with BlobConnectionProvider for blob granules 2022-06-01 12:39:51 -05:00
Josh Slocum 85af0a25b2 Enabling BM to understand tenant boundaries, and changing BlobGranuleCorrectness to use tenants 2022-05-25 17:16:56 -05:00
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Josh Slocum aaaf42525a misc bg operational fixes and improvements 2022-04-05 12:26:00 -05:00
Josh Slocum cb918b9cef Added basic blob granule consistency check 2022-04-04 11:38:42 -05:00
Josh Slocum 377e252fcf
Better split sizing in blob manager (#6725) 2022-04-01 16:09:46 -07:00
Josh Slocum 7fc6dfa6c5 Adding useful debugging trace events 2022-03-29 14:48:28 -05:00
Jingyu Zhou da0673ccce
Merge pull request #6705 from RenxuanW/another
Add proxy option to backup and restore params.
2022-03-29 11:36:13 -07:00
Josh Slocum 2f8e9d9de0 misc bg fixes 2022-03-29 08:05:52 -05:00
Renxuan Wang 0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum 1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
Josh Slocum 0f9e88572a Cleaning up debugging and fixing race in blob manager recruitment 2022-03-17 14:57:43 -05:00
Josh Slocum a85b578d2b Added simulation validation that multiple blob maangers don't have same epoch 2022-03-17 14:57:43 -05:00
Evan Tschannen 2c88a189a9 more pruning bug fixes 2022-03-15 13:34:59 -07:00
Evan Tschannen d46e551f11 merge 2022-03-15 13:09:10 -07:00
Josh Slocum ce5b567971 Cleanup the cleanup 2022-03-15 15:02:28 -05:00
Josh Slocum 9d9b70abd4 Fixing another BM recovery and granule splitting race 2022-03-14 18:56:44 -05:00
Josh Slocum 8c35ceb684 switching bg file keys to sort on version then type, and using that to fix race where 2 blob workers try to split the same granule at different versions 2022-03-14 17:38:31 -05:00
Josh Slocum 1c14d4674d Fixing more BM races with granule splitting 2022-03-14 16:11:17 -05:00
Josh Slocum 3e278e8580 asserts for assign is empty is false if BM dies and rangeAssigner actor is dead 2022-03-14 15:53:09 -05:00
Josh Slocum e7aa534cae Forgot to update worker mapping after picking worker 2022-03-14 15:04:43 -05:00
Josh Slocum 8d88b7ca41 making range assigner boundary changes fully synchronous to ensure no races with things reading assignments 2022-03-14 14:39:00 -05:00
Josh Slocum 67dd3d071e Fixing incorrect assert in BM split handling 2022-03-11 10:50:46 -06:00
Josh Slocum 479ac313ca Fixing conflict handling in BM recovery 2022-03-10 16:35:08 -06:00
Josh Slocum 903f7adbc4 more cleanup 2022-03-10 13:53:06 -06:00
Josh Slocum c35e020da7 SS and BM test macros 2022-03-10 12:38:23 -06:00
Josh Slocum ba35e7908a off by one 2022-03-10 12:11:44 -06:00
Josh Slocum 4b254d259c Ensuring BM split retry is idempotent 2022-03-10 11:54:57 -06:00
Josh Slocum c8c97e0256 Blob Worker focused cleanup 2022-03-10 09:55:23 -06:00
Josh Slocum 1f964ac085 BM focused cleanup 2022-03-09 15:04:17 -06:00
Josh Slocum 9dbb6d216e SS-focused cleanup 2022-03-09 10:39:04 -06:00
Josh Slocum b21d0943b9 client-focused cleanup 2022-03-09 10:01:25 -06:00
Josh Slocum f230ac0e41 Fixing more merge issues 2022-03-09 09:13:03 -06:00
Josh Slocum e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
Jingyu Zhou 1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Evan Tschannen 976375466f fixed more pruning bugs 2022-03-02 17:03:37 -08:00
Josh Slocum e0578f24d2 Fixes for range streaming BG snapshot 2022-03-01 12:45:58 -06:00
Evan Tschannen 6f1fb71949 Fixed a number of different pruning problems 2022-03-01 10:13:20 -08:00
Evan Tschannen 8f8987ad67 Merge branch 'blob_integration' of https://github.com/apple/foundationdb into blob_integration 2022-03-01 10:12:36 -08:00