Commit Graph

9 Commits

Author SHA1 Message Date
Andrew Noyes 3a514d3451
Add design for avoiding storage servers for idempotency ids (#9113)
* Add design for avoiding storage servers for idempotency ids

* Update design/idempotency_ids.md
2023-01-20 11:26:54 -08:00
Andrew Noyes f0cb73a941 Address review comments 2022-12-28 13:29:57 -08:00
Andrew Noyes 4fb8e6d090 Update design doc, and make AUTOMATIC_IDEMPOTENCY visible 2022-12-19 10:24:19 -08:00
Andrew Noyes 0a15f081a1
Proactively clean up idempotency ids for successful commits (#8578)
* Proactively clean up idempotency ids for successful commits

This change also includes some minor changes from my branch working on
an idempotency ids cleaner, that I'd like to get merged sooner rather
than later.

- Adding a timestamp to idempotency values
- Making IdempotencyId an actor file
- Adding commit_unknown_result_fatal
- Checking idempotencyIdsExpiredVersion in determineCommitStatus
- Some testing QOL changes

* Factor out decodeIdempotencyKey logic

* Fix formatting

* Update flow/include/flow/error_definitions.h

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Use KeyBackedObjectProperty for idempotencyIdsExpiredVersion

* Add IDEMPOTENCY_ID_IN_MEMORY_LIFETIME knob

* Rename ExpireIdempotencyKeyValuePairRequest

Also add a code probe for the case where an ExpireIdempotencyIdRequest is
received before the count is known, and add an assert

* Fix formatting and add TODO for nwijetunga

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-10-28 09:07:54 -07:00
Andrew Noyes cd01ee8dcb Describe new design for cleaning idempotency ids 2022-10-12 10:21:25 -07:00
Andrew Noyes 13c5e7f240 Mention batch index is a 2 byte int 2022-10-11 14:15:04 -07:00
Andrew Noyes 7fc5f03b81 Update design/idempotency_ids.md
Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2022-10-11 14:04:29 -07:00
Andrew Noyes f0f2b944a2 Apply suggestions from code review
Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2022-10-11 13:46:40 -07:00
Andrew Noyes 64eaeb22a7 Add design doc 2022-10-11 13:46:39 -07:00