Commit Graph

236 Commits

Author SHA1 Message Date
Josh Slocum 316b7a5344 Merge branch 'main' into granule_merging_converge 2022-07-20 12:13:48 -05:00
Josh Slocum 78b6a96006 Merge branch 'main' into granule_merging_batch 2022-07-20 07:42:26 -05:00
Josh Slocum 12b6f386cb Refactoring granule flush to retry properly on granule rollback 2022-07-19 19:49:20 -05:00
Josh Slocum 166de4b704 Fixing merge convergence bugs 2022-07-19 18:44:40 -05:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Josh Slocum 08186f9245 More efficient merge intent and granule history serialization 2022-07-15 20:38:06 -05:00
Josh Slocum 306610bfcb batch periodic merging in blob manager 2022-07-15 15:52:10 -05:00
Ata E Husain Bohra 1da88d5d8c BlobFile Encryption and compression support
Fix formatting issues

Description

Testing
2022-07-14 22:46:14 -07:00
Ata E Husain Bohra f288abebc2 BlobFile Encryption and compression support
Fix formatting issues and rename KNOB

Description

Testing
2022-07-14 17:22:00 -07:00
Ata E Husain Bohra f6f117592d BlobFile Encryption and compression support
- Limit verbose logging under DEBUG_MACRO
 - Update/Add code documentation

Description

Testing
2022-07-14 17:04:14 -07:00
Ata E Husain Bohra 3a3811d6cc BlobFile Encryption and compression support
Few fixes
 - Update BlobGranuleCipherKeysMeta to use 'std::string' to
   persist 'Initialization Vector'
 - Update WriteSnapshot to persist 'BlobGranuleCipherKeysMeta' as part
   of BlobFileIndex

Description

Testing
2022-07-14 17:04:14 -07:00
Ata E Husain Bohra 24b2de8de8 BlobFile Encryption and compression support
Description

Testing
2022-07-14 17:04:14 -07:00
Josh Slocum b85fbaef52
Merge pull request #7395 from sfc-gh-jslocum/bg_file_chunking
Chunked Snapshot Files
2022-07-13 17:22:34 -05:00
Josh Slocum b9437e9a8d BW re-sending merge candidate if manager changes 2022-07-13 10:39:55 -05:00
Josh Slocum f707626213 Cleanup 2022-07-12 17:43:44 -05:00
Josh Slocum 0b0ac16a4c Merge branch 'main' into granule_merging 2022-07-12 09:09:30 -05:00
Josh Slocum 9e64037b25 Merge branch 'main' into bg_file_chunking 2022-06-30 17:13:02 -05:00
A.J. Beamon 26b35c07cd Refactor how tenant map entries are encoded and decoded. Add a specific version to the encoding that matches the version used when this feature was introduced (and the only version in which it was used). 2022-06-29 10:58:58 -07:00
A.J. Beamon 4bafe77889 Some refactoring of tenant code:
* extract tenant management into its own file and namespace
* rename the tenant management workload source file
* extract tenant special keys functions to a separate file
* extract some helper functions to GenericTransactionHelper.h
* convert StringRef -> TenantNameRef
* move some TenantMapEntry implementation into the cpp file
* add some helper functions to decode/encode a tenant mode
2022-06-27 12:32:49 -07:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Josh Slocum 3f871f4123 Chunked Snapshot Files passes unit test 2022-06-15 14:52:28 -05:00
Josh Slocum 058f44a4ae Fixed req begin version when time travelling back from merge 2022-06-10 15:11:30 -05:00
Josh Slocum 710612442c small fixes 2022-06-09 17:21:21 -05:00
Josh Slocum d6920cde28 Implemented blob granule merging 2022-06-09 10:50:53 -05:00
Josh Slocum ae865027d6 fixes for blob metadata memory from valgrind 2022-06-07 13:50:11 -05:00
Josh Slocum ff643cce76 compile fix 2022-06-03 11:50:33 -05:00
Josh Slocum b650410a48 Merge branch 'main' into blob_granule_kms 2022-06-03 09:13:49 -05:00
Josh Slocum fcd20c479d addressing review comments 2022-06-03 08:36:07 -05:00
Josh Slocum 567b1d35f7 Full integration with BlobConnectionProvider for blob granules 2022-06-01 12:39:51 -05:00
Josh Slocum ffa4255c65 Added blob metadata concept as new secret type, and verified blob workers can load it 2022-05-27 15:15:56 -05:00
Josh Slocum 458e8018e1 review comments 2022-05-25 17:16:56 -05:00
Josh Slocum dd637dd6a7 passes tenant simulation test 2022-05-25 17:16:56 -05:00
Josh Slocum e3a570d50d Blob worker tenant integration mostly works 2022-05-25 17:16:56 -05:00
Josh Slocum ad54be5d0b Enabling tenant tests for blob granules and fixing bug 2022-05-25 17:16:56 -05:00
Josh Slocum 4015b5b2a0
Several Blob Granule bug fixes found in nightlies (#7170)
* Fixing simulation validation assert that was tripping incorrectly

* Commenting out debugging prints

* Fixed multiple error propagation issues in blob worker
2022-05-23 10:07:16 -07:00
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Josh Slocum aaaf42525a misc bg operational fixes and improvements 2022-04-05 12:26:00 -05:00
Evan Tschannen c168840b54 blob workers properly destroy change feeds when they are no longer needed 2022-04-05 11:02:32 -05:00
Josh Slocum 7fc6dfa6c5 Adding useful debugging trace events 2022-03-29 14:48:28 -05:00
Jingyu Zhou da0673ccce
Merge pull request #6705 from RenxuanW/another
Add proxy option to backup and restore params.
2022-03-29 11:36:13 -07:00
Josh Slocum 61474d5d54 Future-proof blob granules with full file size 2022-03-29 08:06:07 -05:00
Renxuan Wang 0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum 3cd1e5599e Cleanup 2022-03-24 17:15:11 -05:00
Josh Slocum 989dd8d7eb Finished implementing beginVersion 2022-03-24 17:15:11 -05:00
Josh Slocum 1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
Josh Slocum c72197122d bump 2022-03-23 08:07:49 -05:00
Josh Slocum 4e20da76fb bump 2022-03-22 15:37:16 -05:00
Josh Slocum 917c7acca5 Using desiredOldest for when change feed metadata is safe 2022-03-21 22:21:40 -05:00
Josh Slocum f4c2a789c6 Explicitly cancelling file updater future 2022-03-21 11:47:40 -05:00
Josh Slocum f7f2c9e979 Fixing BW erroneously asking BM to split even after it recieved a reply to continue without splitting 2022-03-20 08:13:38 -05:00