Commit Graph

136 Commits

Author SHA1 Message Date
Hao Fu f00606fd71
Enable MATCHED and UNMATCHED mode for index prefetch (#7162)
* Enable MATCHED and UNMATCHED mode for index prefetch

MATCHED mode returns index entries whose secondary KVs are present,
UNMATCHED mode returns index entries whose secondary KVs are absent.

Note that the conflict read range of this txn is set in 2 steps:
* Set the conflict range for primary query according to request
* Set the conflict ranges for secondary queries according to responses.

As a result, conflicts of different match_index mode are taken care of.

* Fix c binding
2022-05-18 09:16:28 -07:00
Hao Fu 853e6a346b
Optimization: support removing index conditionally (#7116) 2022-05-13 10:10:43 -07:00
Vaidas Gasiunas a3102c0db4 Fix updating database shared state on upgrades 2022-05-12 15:28:53 +02:00
Ray Jenkins dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Ray Jenkins 1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. (#6855)" (#6941)
This reverts commit 5df3bac110.
2022-04-25 09:29:56 -05:00
Vaidas Gasiunas e48ab69c6b An option to specify a custom directory for temporary client library copies 2022-04-21 19:27:19 +02:00
Ray Jenkins 5df3bac110
Migrate to OpenTelemetry tracing. (#6855) 2022-04-20 09:26:37 -05:00
Dan Lambright 1b3b4166c6
Merge branch 'main' into vv 2022-04-08 17:18:13 -04:00
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Dan Lambright 5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Jon Fu b0ae22f16d Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-04-06 12:28:28 -04:00
Jingyu Zhou 64d4658034 Merge branch 'main' into vv
Fix Conflicts:
	flow/error_definitions.h
2022-04-01 21:49:24 -07:00
A.J. Beamon 68f15650a1 Make sure closed and tenantUpdater are read/written in the same critical section. 2022-03-31 11:12:01 -07:00
Jingyu Zhou 4fd414a8ed Merge branch 'main' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
2022-03-31 09:38:36 -07:00
Jingyu Zhou cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou b34f4052cd Merge branch 'main-f28dfc12b' into vv
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
Jingyu Zhou 00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
A.J. Beamon c7d53b31ee Use a TenantState object in the MVC implementation to help manage tenant lifetime. 2022-03-30 12:52:27 -07:00
Jon Fu f266695a76 clear shared state map entries on version upgrade and use currentProtocolVersion instead of piping version through function parameters 2022-03-30 15:28:55 -04:00
Jon Fu 9d0a520fe0 add ProtocolVersion to shared state and state cluster map 2022-03-30 13:28:06 -04:00
Jon Fu 53d252fda0 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-29 17:23:08 -04:00
Jon Fu bdbe78e48f address code review comments 2022-03-29 17:20:22 -04:00
Josh Slocum 029c30d035 fix formatting 2022-03-24 16:05:51 -05:00
Josh Slocum b42c49ca0a Adding full file length for future-proofing 2022-03-24 15:56:03 -05:00
Josh Slocum b92c6ea92c granule parallelism, metrics, and other improvements to bg client bindings 2022-03-24 13:44:09 -05:00
Jon Fu 7c3151142e Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-23 16:41:33 -04:00
Jon Fu 4466e00716 Fix multithreaded GRV cache code and enable grv caching in multithreaded ctest suite 2022-03-23 16:09:12 -04:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Jon Fu 9731d96797 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-16 14:32:33 -04:00
Jon Fu 7ab205391b Revert "remove all multi-version client code related to grv cache"
This reverts commit 6d05ecffc2.
2022-03-16 14:26:36 -04:00
A.J. Beamon 68069c9784 Undo some unused changes. Fix tenant cache eviction. 2022-03-15 09:23:30 -07:00
A.J. Beamon 8bc2b283e1 Create a special keys API to create, delete, and read tenants. Remove the C API to create/delete tenants. 2022-03-15 09:23:30 -07:00
A.J. Beamon c89b1e3537 Use TenantName type instead of StringRef in various places. Add a function on transactions to get the tenant name being used for the transaction. 2022-03-15 09:21:27 -07:00
A.J. Beamon c635dcd3ad Add tenant support in the FDB native client 2022-03-15 09:21:27 -07:00
Tao Lin e2c7c30faf
GetMappedRange support serializable & check RYW & continuation (#6181) 2022-03-10 10:05:44 -08:00
Dan Lambright 9544379cdf rebase 2022-01-20 11:12:33 -05:00
Josh Slocum b7b2ad0a6f Handling timeout and transaction changes for ThreadResults properly 2021-12-10 12:47:53 -06:00
Josh Slocum 9cb6fb5114 fixing unrelated code formatting 2021-12-01 17:20:11 -06:00
Josh Slocum c11b8a3625 Improved comments 2021-12-01 17:04:55 -06:00
Josh Slocum a82845af43 Merge branch 'master' into bg_bindings 2021-12-01 16:55:28 -06:00
Josh Slocum 7f4fcc8c2c Added FDBResult and made readBlobGranules use it 2021-12-01 16:22:05 -06:00
A.J. Beamon 264c75b9a6 Add some extra client logging details:
1. Add a trace event when a database is created and move the cluster file / connection string from ClientStart to the new trace event
2. Add a detail for the path to the image being loaded
3. Add a detail for whether a client library is primary or not
4. Set a thread name for each external client thread that includes the release version
2021-11-29 09:57:10 -08:00
Josh Slocum 329091e14f Merge branch 'master' into bg_bindings 2021-11-11 10:13:37 -06:00
Josh Slocum b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Tao Lin fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce #5609
2021-11-09 13:52:28 -08:00
Tao Lin 586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin 6c98e35893 Rename Hop to FlatMap 2021-11-03 13:32:01 -07:00
Tao Lin 0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Josh Slocum 5b2617a524 Added local granule file reading to mako 2021-11-03 09:33:30 -05:00
Josh Slocum d6a31078fe C API for blob granules 2021-11-02 10:01:23 -05:00