Commit Graph

254 Commits

Author SHA1 Message Date
A.J. Beamon c4b0f6eaae
Add an internal C API to support connection to a cluster using a connection string (#7438)
* Add an internal C API to support memory connection records

* Track shared state in the client using a unique and immutable cluster ID from the cluster

* Add missing code to store the clusterId in the database state object

* Update some arguments to pass by const&
2022-07-07 10:12:49 +02:00
A.J. Beamon 4bafe77889 Some refactoring of tenant code:
* extract tenant management into its own file and namespace
* rename the tenant management workload source file
* extract tenant special keys functions to a separate file
* extract some helper functions to GenericTransactionHelper.h
* convert StringRef -> TenantNameRef
* move some TenantMapEntry implementation into the cpp file
* add some helper functions to decode/encode a tenant mode
2022-06-27 12:32:49 -07:00
Lukas Joswiak 88557d9169 Simplify function call when transaction is null 2022-06-22 14:50:17 -07:00
Lukas Joswiak b80ed948f1 Check initialization status before accessing field 2022-06-22 14:50:17 -07:00
Josh Slocum b3597ef3a8 Added plumbing for tenant-aware purge granules 2022-06-16 13:04:34 -05:00
Hao Fu 853e6a346b
Optimization: support removing index conditionally (#7116) 2022-05-13 10:10:43 -07:00
Vaidas Gasiunas a3102c0db4 Fix updating database shared state on upgrades 2022-05-12 15:28:53 +02:00
Ray Jenkins dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Josh Slocum 8284ec5712 Fixing memory leak when handling FDBResult in multi version client 2022-05-02 12:56:05 -05:00
Andrew Noyes 13a388ff14
Disable leak checks for some tests that intentionally leak (#6997) 2022-04-28 12:55:34 -07:00
Ray Jenkins 1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. (#6855)" (#6941)
This reverts commit 5df3bac110.
2022-04-25 09:29:56 -05:00
Vaidas Gasiunas 9897712307 Merge remote-tracking branch 'apple/main' into vgasiunas-client-tmp-dir 2022-04-22 13:00:09 +02:00
Vaidas Gasiunas 444b6c5479 Default the client temp directory to /tmp 2022-04-22 11:44:44 +02:00
Vaidas Gasiunas e48ab69c6b An option to specify a custom directory for temporary client library copies 2022-04-21 19:27:19 +02:00
Josh Slocum 40052c1394 Added Blob Granule API Tester 2022-04-20 10:15:43 -05:00
Ray Jenkins 5df3bac110
Migrate to OpenTelemetry tracing. (#6855) 2022-04-20 09:26:37 -05:00
Dan Lambright 1b3b4166c6
Merge branch 'main' into vv 2022-04-08 17:18:13 -04:00
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Dan Lambright c106847e3e
Merge branch 'main' into vv 2022-04-08 15:05:51 -04:00
Dan Lambright 5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Tao Lin 73ed34d2fb Fixes #6793: Upgrade from 7.0.0 to 7.1.0 fails because of missing function fdb_transaction_get_mapped_range 2022-04-08 09:32:18 -07:00
Jon Fu 1a3b5f78c4 trace sevError if no map entry is found 2022-04-08 10:32:24 -04:00
Jon Fu 68a75dd9cc additional safety checks in the upgrade scenario for GRV cache 2022-04-08 10:17:18 -04:00
Jon Fu b0ae22f16d Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-04-06 12:28:28 -04:00
Jon Fu 124aa6e5fd clean up debug traces and fix details with delRef placement 2022-04-06 12:27:39 -04:00
Jon Fu 1b9bd597c5 address code review comments and add some traces for manual testing 2022-04-05 15:29:02 -04:00
Jingyu Zhou 64d4658034 Merge branch 'main' into vv
Fix Conflicts:
	flow/error_definitions.h
2022-04-01 21:49:24 -07:00
Jon Fu c46ad3ce75 Only clear state map if it isn't the initial protocol version change from a null value 2022-04-01 13:51:57 -04:00
A.J. Beamon 68f15650a1 Make sure closed and tenantUpdater are read/written in the same critical section. 2022-03-31 11:12:01 -07:00
Jingyu Zhou 4fd414a8ed Merge branch 'main' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
2022-03-31 09:38:36 -07:00
Jingyu Zhou cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou b34f4052cd Merge branch 'main-f28dfc12b' into vv
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
Jingyu Zhou 00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
Jingyu Zhou e9659b5dd4 Merge branch 'master-PR-6500' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbserver/masterserver.actor.cpp
2022-03-30 14:53:49 -07:00
A.J. Beamon c7d53b31ee Use a TenantState object in the MVC implementation to help manage tenant lifetime. 2022-03-30 12:52:27 -07:00
Jon Fu f266695a76 clear shared state map entries on version upgrade and use currentProtocolVersion instead of piping version through function parameters 2022-03-30 15:28:55 -04:00
Jon Fu 9d0a520fe0 add ProtocolVersion to shared state and state cluster map 2022-03-30 13:28:06 -04:00
Jon Fu 53d252fda0 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-29 17:23:08 -04:00
Jon Fu bdbe78e48f address code review comments 2022-03-29 17:20:22 -04:00
Josh Slocum b92c6ea92c granule parallelism, metrics, and other improvements to bg client bindings 2022-03-24 13:44:09 -05:00
Jon Fu 7c3151142e Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-23 16:41:33 -04:00
Jon Fu 4466e00716 Fix multithreaded GRV cache code and enable grv caching in multithreaded ctest suite 2022-03-23 16:09:12 -04:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Jon Fu 9731d96797 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-16 14:32:33 -04:00
Jon Fu 7ab205391b Revert "remove all multi-version client code related to grv cache"
This reverts commit 6d05ecffc2.
2022-03-16 14:26:36 -04:00
A.J. Beamon 8a5107af03 Fix a couple variable initialization issues in the tenant MVC implementation. Encode tenant prefixes with unicode escape sequences for the list command. 2022-03-15 09:23:30 -07:00
A.J. Beamon 68069c9784 Undo some unused changes. Fix tenant cache eviction. 2022-03-15 09:23:30 -07:00
A.J. Beamon 8bc2b283e1 Create a special keys API to create, delete, and read tenants. Remove the C API to create/delete tenants. 2022-03-15 09:23:30 -07:00
A.J. Beamon c89b1e3537 Use TenantName type instead of StringRef in various places. Add a function on transactions to get the tenant name being used for the transaction. 2022-03-15 09:21:27 -07:00
A.J. Beamon c635dcd3ad Add tenant support in the FDB native client 2022-03-15 09:21:27 -07:00