* Exposing writeEntireFile up through BackupContainerFileSystem, and using it in blob worker
* Adding blob worker latency metrics
* avoid writeEntireFile if object is too large
* gracefully falling back to multi-part upload if the file is too big
* Added SSPhysicalShard.
* Update physicalShards in StorageServer::addShard().
* Handle notAssigned shard.
* fetchKeys() are not stopped during TerminateStorageServer since
physicalShards are not cleared.
* Fixed addingSplitLeft unset shardId.
* Increased the timeout for Rocks reads in simulation.
* Cleanup.
* set SERVE_AUDIT_STORAGE_PARALLELISM to 1.
* Disabled ValidateStorage test.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
* Proactively clean up idempotency ids for successful commits
This change also includes some minor changes from my branch working on
an idempotency ids cleaner, that I'd like to get merged sooner rather
than later.
- Adding a timestamp to idempotency values
- Making IdempotencyId an actor file
- Adding commit_unknown_result_fatal
- Checking idempotencyIdsExpiredVersion in determineCommitStatus
- Some testing QOL changes
* Factor out decodeIdempotencyKey logic
* Fix formatting
* Update flow/include/flow/error_definitions.h
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Use KeyBackedObjectProperty for idempotencyIdsExpiredVersion
* Add IDEMPOTENCY_ID_IN_MEMORY_LIFETIME knob
* Rename ExpireIdempotencyKeyValuePairRequest
Also add a code probe for the case where an ExpireIdempotencyIdRequest is
received before the count is known, and add an assert
* Fix formatting and add TODO for nwijetunga
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Fix a test timeout due to buggified knob MAX_WRITE_TRANSACTION_LIFE_VERSIONS
The buggified knob MAX_WRITE_TRANSACTION_LIFE_VERSIONS can be only 1M. In some
tests, this transaction always end up commitVersion - readVersion is a little
above 1M, thus always getting transaction_too_old error.
* Change MAX_COMMIT_BATCH_INTERVAL instead
So that the master may give out versions fast enough.
* Fix an assertion failure in a unit test
48125>>8 = 187, 48125 = 0xbbfd
48128>>8 = 188, 48128 = 0xbc00
So if 48125 is chosen as the index, 48128 changes the higher order byte.
48125 & 0xff7f = 47997 = 0xbb7d. Thus +5 won't change the higher order byte.
Xiaoxi Wang