1817b135ac
Use a local keyRangeMap to avoid assigning ranges not part of client ranges.
2021-11-16 08:49:16 -06:00
ac0a5750e7
Clamp end of system keys down to normal keys.
2021-11-16 08:49:16 -06:00
f140028086
Merge branch 'master' into blob_integration
2021-11-15 15:09:40 -06:00
62efeb6812
Merge remote-tracking branch 'origin/master' into add-format-warning
2021-11-12 11:50:36 -08:00
bab8756e17
fixed bugs in small splits
2021-11-11 13:50:19 -06:00
db3c08c7cd
Merge pull request #5928 from sfc-gh-anoyes/anoyes/fix-heap-use-after-free
...
Fix a heap use after free
2021-11-10 10:21:05 -08:00
d97d968176
Added KeyBackedObjectMap and KeyBackedObjectProperty classes for storing serializable objects in FDB ( #5896 )
...
* Cleaned up some lambda capture workaround since x=y captures weren't available when these classes were originally written.
* Added KeyBackedObjectMap and KeyBackObjectProperty, which work like KeyBackedMap and KeyBackedProperty but use ObjectWriter/Reader for Value serialization so that the type can evolve over time.
* Disabled unit tests which shouldn't run as part of random selection.
2021-11-08 13:04:53 -08:00
b7e393587c
Fix a heap use after free
...
If we accept arena arguments by value, then the lifetime of any memory
allocated by that arena ends when the function returns. Given that we
seem to be appending to VectorRef's passed by pointer this is unlikely
to be what we want.
2021-11-08 12:51:32 -08:00
13bb7838aa
Enable clang -Wformat warning
2021-10-30 21:07:38 -07:00
0d1d1d7f9e
fix uninitialized memory and granule bounds on manager recovery
2021-10-28 18:23:43 -05:00
19076ad4d2
Changed blob manager checkin to by async and to have blob worker speculatively consume change feed ahead of re-snapshot
2021-10-27 16:23:51 -05:00
134aef6011
Fix compiler warning.
2021-10-27 10:40:03 -04:00
fd50a011b1
Address PR comments.
2021-10-26 21:39:41 -04:00
5a9d9921d0
Fixes and final cleanup for BM failure handling
2021-10-26 16:16:00 -04:00
17b30f188a
Working impl
2021-10-26 16:16:00 -04:00
99606482ea
initial thoughts
2021-10-26 16:16:00 -04:00
e57e2bec5f
Improve documentation.
2021-10-25 12:19:28 -04:00
912ef76f1c
cleanup before merge
2021-10-18 17:11:14 -05:00
e2e852e515
Mitigate transitive includes.
2021-10-18 10:49:25 -04:00
a9f23773ad
Default all debug flags to false.
2021-10-18 10:10:05 -04:00
90ce8bbe5b
Refactor retry loop to splitRange.
2021-10-18 09:56:47 -04:00
d1fe9d4c50
Remove old comments from BM.
2021-10-14 19:25:34 -04:00
b5074fd597
Reworked all of the system data to encode granule data more efficiently for persistence
2021-10-13 16:28:04 -05:00
f3c44c568f
fixing merge conflicts
2021-10-13 16:26:44 -05:00
cfb8368da6
Address PR comments.
2021-10-13 14:56:17 -04:00
ef67feed67
Clean up blob manager changes.
2021-10-13 14:40:26 -04:00
266a5b06fa
Fix infinite loop.
2021-10-13 14:40:26 -04:00
423a67f448
trying to fix infinite loop
2021-10-13 14:40:26 -04:00
dfb9655c57
Handle blob work failure
2021-10-13 14:40:26 -04:00
4e4a2534da
review comments and moving priority yield to correct spot
2021-10-06 09:24:48 -05:00
6a24ef9258
adding priorities to blob worker and fixing monitoring in blob manager
2021-10-05 16:51:19 -05:00
282f9d35cd
Cleanup comments and debugging code.
2021-10-04 11:07:08 -04:00
4d54669ccd
Recruit the blob workers via blob manager.
...
In this PR, the blob manager now recruits blob workers
(via communication with the cluster controller). Blob workers
are onboarded as blob worker processes enter the cluster.
2021-10-04 11:07:08 -04:00
8fb7c45e65
even more bug fixes
2021-09-25 10:30:27 -05:00
fa1fe5f08b
added blob worker rollbacks that handle (most) cases
2021-09-24 17:52:36 -05:00
a986679b7f
More bug fixes
2021-09-24 10:02:02 -05:00
889ae3f255
Address PR comments.
...
Remove nukeBlobWorkerData, simplify checkManagerLock call and make
final choose...when consistent.
2021-09-23 14:09:56 -04:00
d675e6b143
Kill BM by returning, instead of throwing.
2021-09-23 11:41:57 -04:00
8c49f1f238
Improve actor management and handling of failure in BM.
2021-09-23 10:58:38 -04:00
5fa6c687d6
Add blob manager as a singleton.
2021-09-23 10:45:37 -04:00
5ddf08dfe5
Got basic range reassignment working
2021-09-22 16:48:44 -05:00
c9b8bdffbe
Passes a single correctness test!
2021-09-15 17:18:04 -05:00
e2a51a4fe7
Fixing up after change feed updates
2021-09-10 11:49:41 -05:00
eb76343dfb
Added blob granule reassignment and splitting
2021-09-08 14:09:14 -05:00
fccfe3af78
Address PR comments.
2021-09-02 12:09:37 -04:00
c27d8f3336
Improve assignment of range granules.
...
Previously, we randomly picked a worker to assign a range to.
Now, we pick a worker that has the least number of range granules
already assigned, ultimately distributing the workload in a more
efficient manner.
Future iterations should also consider the number of read/writes
that a worker is already handling when picking a worker to assign
a range to. This could prevent us from assigning a range to a worker
that is already hot.
2021-09-01 23:36:52 -04:00
46adada5ff
Cleaned up debugging and fixed a couple bugs
2021-08-31 12:30:43 -05:00
bbeec49533
added range mover
2021-08-30 13:59:53 -05:00
b4bfd58bcb
multiple blob workers appears to work
2021-08-30 13:07:25 -05:00
3b011408f8
Added sequence numbers and locks to blob worker and manager
2021-08-27 16:33:07 -05:00
8d49c98a41
Added simulation workload for blob granules and fixed some bugs
2021-08-26 13:48:05 -05:00
ad899565e9
Using local files for blob granules in simulation
2021-08-24 14:15:14 -05:00
5259af787d
Switched blob implementation to use backup container
2021-08-24 13:47:47 -05:00
2ae447eaaa
Refactored blob worker/manager to be in separate files
2021-08-23 14:16:09 -05:00
b3491d7338
Cleaned up code for demo
2021-08-12 14:30:33 -05:00
247602753d
Blob worker handles multiple ranges now
2021-08-10 11:19:43 -05:00
Suraj Gupta