Commit Graph

28 Commits

Author SHA1 Message Date
Hui Liu f2289ced27 Add StorageServerInterface for BlobMigrator 2022-10-24 13:12:07 -07:00
Josh Slocum 89519343a7
adding new bg read metrics to client and refactoring them into a new trace event (#8493) 2022-10-18 14:13:35 -05:00
A.J. Beamon 4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
Hui Liu 45213175d0 fix unit test isRangeCoveredByBlob 2022-08-29 13:39:36 -07:00
Hui Liu b7d4453da7 check if fetchKeys is covered by blob storage 2022-08-26 14:21:44 -07:00
Josh Slocum 98a7ec1797
Blob Granules Cleanup (#7941)
* Cleaned up BlobGranule TODO + FIXMEs and addressed some

* popping feed at correct version

* blob worker taking over a granule will pop from where previous worker left off

* addressed fixme of blob worker not re-snapshotting from old change feed

* formatting

* more change feed popped fixes after pop updates

* Getting rid of change feed parallelism lock since it can cause deadlocks in fetching, and relying on full fetch lock

* New blob worker metric and fixing old one

* server-side popped checking still doesn't work because of pops at non-mutation versions

* format
2022-08-19 17:25:31 -07:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Josh Slocum 567b1d35f7 Full integration with BlobConnectionProvider for blob granules 2022-06-01 12:39:51 -05:00
Josh Slocum 989dd8d7eb Finished implementing beginVersion 2022-03-24 17:15:11 -05:00
Josh Slocum 1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
sfc-gh-tclinkenbeard 8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
Jingyu Zhou 1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Josh Slocum a82845af43 Merge branch 'master' into bg_bindings 2021-12-01 16:55:28 -06:00
sfc-gh-tclinkenbeard 2613ec7561 Expand use of fmt to get rid of %ld usage 2021-11-17 17:03:32 -08:00
Josh Slocum a5b2f464e3 Refactored granule materialization code to be called by c client as well as internal granule management 2021-11-03 13:19:07 -05:00
sfc-gh-tclinkenbeard 13bb7838aa Enable clang -Wformat warning 2021-10-30 21:07:38 -07:00
Josh Slocum 912ef76f1c cleanup before merge 2021-10-18 17:11:14 -05:00
Suraj Gupta e2e852e515 Mitigate transitive includes. 2021-10-18 10:49:25 -04:00
Josh Slocum eb76343dfb Added blob granule reassignment and splitting 2021-09-08 14:09:14 -05:00
Suraj Gupta 5aebd77f0a PR changes 2021-09-03 13:58:39 -04:00
Suraj Gupta a0b3446572 Add metrics for blob worker.
We want to add metrics for the blob worker to evaluate its
performance more concretely. We decided to track the following
information:
- s3 put requests
- s3 get requests
- S3 delete requests
- Delta files written
- Snapshot files written
- Delta bytes written
- Snapshot bytes written
- Number of current ranges assigned
- Bytes read from FDB (initial snapshot)
- Bytes read from S3 (compaction)
- Read requests count
- Read files returned
- Read deltas returned
- Read delta bytes returned
- Ranges assigned
- Ranges revoked
- Number of current ranges assigned
- Total mutation bytes buffered across all ranges // current or accumulated
- Range feed bytes input
- Range feed mutations input
- Wrong Shard Server count
2021-09-03 12:09:01 -04:00
Josh Slocum 8d49c98a41 Added simulation workload for blob granules and fixed some bugs 2021-08-26 13:48:05 -05:00
Josh Slocum 5259af787d Switched blob implementation to use backup container 2021-08-24 13:47:47 -05:00
Josh Slocum fb2eef38fc Changing API and file format to full V1 specification 2021-08-24 10:05:46 -05:00
Josh Slocum b3491d7338 Cleaned up code for demo 2021-08-12 14:30:33 -05:00
Josh Slocum 247602753d Blob worker handles multiple ranges now 2021-08-10 11:19:43 -05:00
Josh Slocum e5b0cf20aa Reader can read blob worker interface and mapping from db, read files, apply mutations, and return result 2021-08-06 11:02:05 -05:00