Commit Graph

994 Commits

Author SHA1 Message Date
Josh Slocum f866ffc36b
Better granule conversion (#7787)
* better check for granule-ification

* Handling blob granule initial split too large

* Re-evaluating split size if too large, even if read doesn't get transaction_too_old

* reworked to have blob worker propose split key

* New GranuleStatusReply to avoid seqno issue stream side effects

* Handling retries on reevaluateInitialSplit properly

* Waiting for stream to be initialized

* Checking reevaluate split for additional split points beyond proposed

* Fixing more races in reevaluate initial split

* properly handling cleaning up old change feed after split re-evaluate

* fixing granule conversion bug with hard boundaries

* fixing clear and merge check race with cycle test

* refactor missed knob check for clearAndMerge

* Fixing formatting

* review comments and improving large range conversion

* fixing typo

* more formatting
2022-08-05 18:12:17 -05:00
Josh Slocum d721d1b850
More cf bug fixes (#7789)
* Fixing change feed fetch and rollback race

* Fixing validation issue for change feed validation

* Fixing shutdown segfault in blob worker
2022-08-04 15:57:42 -05:00
Jingyu Zhou 84d483605b
Merge pull request #7431 from xis19/main
Let the storage server reports busiest write tag
2022-08-04 10:23:31 -07:00
Yao Xiao 5ee40bbc5f
Clean up empty physical shards in KVS. (#7674) 2022-08-04 02:12:04 -07:00
He Liu fa418fd784
Change SHARD_ENCODE_LOCATION_METADATA to a server knob. (#7770)
Co-authored-by: He Liu <heliu@apple.com>
2022-08-03 13:51:40 -07:00
He Liu 013b9e3baa
Fixed ChangeServerKeysContext name issue. (#7761)
* Fixed ChangeServerKeysContext name issue.

* Update fdbserver/storageserver.actor.cpp

Co-authored-by: Andrew Noyes <andrew.noyes@snowflake.com>

Co-authored-by: He Liu <heliu@apple.com>
Co-authored-by: Andrew Noyes <andrew.noyes@snowflake.com>
2022-08-02 13:54:27 -07:00
Hao Fu 9db19b9a7c
Retain debug id in prefetch server-server call (#7754) 2022-08-01 19:46:45 -07:00
Xiaoge Su fd3c3f0774 fixup! Reformat source 2022-08-01 18:56:50 -07:00
Xiaoge Su 195890dd7b Add ratekeeper ID for storage server busiest write tag report 2022-08-01 18:56:50 -07:00
Xiaoge Su 90b887f394 fixup! Update per comments 2022-08-01 18:56:50 -07:00
Xiaoge Su 542b5e61cf Let the storage server reports busiest write tag
Issue #7258

The ratekeeper is recording the busiest write tag for *all* storage
servers, which throttles the traceevent. Distribute the busiest write
tag to corresponding storage servers should reduces this throttling
issue.
2022-08-01 18:56:50 -07:00
Nim Wijetunga 3e45b6657c
Merge pull request #7714 from sfc-gh-nwijetunga/nim/tenant-encryption-property
Encryption as a tenant property
2022-07-29 11:30:19 -07:00
Hao Fu 62bee7ac9a
Add g_traceBatch for prefetch query (#7730) 2022-07-28 17:54:31 -07:00
He Liu 35a4cb91d5
Disable ShardedRocksDB in simulation when shard_encode_location_metadata is disabled (#7726)
* Disabled tests for ShardedRocks.

Cleaned up ShardedRocks TraceEvent.

Added assertion in ShardManager::validate().

* Added test trace.

* Make sure TraceEvent contains `ShardedRocks`.

* Exclude ShardedRocksDB when SHARD_ENCODE_LOCATION_METADATA is disabled.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-28 13:54:29 -07:00
Junhyun Shim 1e83d7e1ab Apply code review suggestions 2022-07-28 00:47:14 +02:00
Junhyun Shim c6342a6e5b
Merge branch 'main' into features/authz 2022-07-27 20:51:32 +02:00
Nim Wijetunga 2cd73242de fix merge conflict 2022-07-27 10:58:41 -07:00
A.J. Beamon d39c0b773a Add a limit to the number of tenants that can be created in a cluster 2022-07-27 08:21:03 -07:00
Nim Wijetunga 50391c35b1 Merge branch 'main' of github.com:sfc-gh-nwijetunga/foundationdb into nim/tenant-encryption-property
* 'main' of github.com:sfc-gh-nwijetunga/foundationdb: (42 commits)
  Get ShardedRocks ready for simulation test. (#7679)
  fixing specific unit test
  formatting
  addressing review comments
  Fix incorrect deserialization of FdbClientLogEvents::Event (#7707)
  Fix a crash bug during CC shutdown process (#7705)
  addressing review comments
  remove runAfter
  Add comments explaining the use of the TransactionState tenant() and hasTenant() functions
  Make sure resumeFromDataMoves() starts after resumeFromShards().
  Fix: during recovery, it was possible for tenant operations to briefly fail because the tenant mode is not known
  formatting
  Fixed granule purging bug and improved debugging for purging
  making purge failures fail test
  cleanup and polish
  Bug fix and cleanup
  First version of key-sorted delta files
  Added full granule read unit test
  Completed delta format unit test
  delta file test and delta generation
  ...
2022-07-27 08:13:30 -07:00
Josh Slocum 1ea43aa878 Merge branch 'main' into improved_cf_testing 2022-07-26 18:12:58 -05:00
Josh Slocum 77956dc7ae
Merge pull request #7639 from sfc-gh-jslocum/cf_metadata_rewrite
Change Feed Metadata Rewrite and adding targeted fault injection
2022-07-26 18:10:37 -05:00
Junhyun Shim e2a3fedfc7
Merge branch 'main' into features/authz 2022-07-27 00:08:57 +02:00
Josh Slocum 5c2681f0b5 formatting 2022-07-26 16:47:07 -05:00
Josh Slocum 0609aa57d1 Merge branch 'main' into improved_cf_testing 2022-07-26 16:44:13 -05:00
Josh Slocum ba31229797 addressing review comments 2022-07-26 16:37:06 -05:00
Josh Slocum 4d2f90977d
Merge pull request #7656 from sfc-gh-jslocum/cf_bw_operational_fixes
Cf bw operational fixes
2022-07-26 16:24:26 -05:00
Nim Wijetunga 7af4a103f9 fix tests 2022-07-26 10:45:27 -07:00
Nim Wijetunga 33f236a068 merge 2022-07-26 10:37:26 -07:00
Nim Wijetunga 26d233c0f8 encryption as a tenant property 2022-07-25 17:19:14 -07:00
A.J. Beamon 6719e5a85b
Merge pull request #7676 from liquid-helium/ss-shard-management
Fixed compiler warnings.
2022-07-25 13:28:54 -07:00
Zhe Wu 2fdd9d7407 Add testing mechanism 2022-07-25 09:18:02 -07:00
Zhe Wu 98fbc98274 Error in FetchKey GRV shouldn't fail storageserver 2022-07-25 09:18:02 -07:00
He Liu 03755d2489 Fixed compiler warnings. 2022-07-23 20:06:26 -07:00
A.J. Beamon 9459d0c6ca
Merge pull request #7661 from sfc-gh-ajbeamon/tenant-metadata-key-backed-types
Use key-backed types for tenants
2022-07-22 14:01:45 -07:00
Andrew Noyes 79f60691e9
Don't pass non-pod to variadic function (#7668)
fdbserver/storageserver.actor.cpp:654:24: error: cannot pass object of non-trivial type 'const std::string' (aka 'const basic_string<char, char_traits<char>, allocator<char>>') through variadic function; call will abort at runtime [-Wnon-pod-varargs]

Just use fmt instead
2022-07-22 12:51:44 -07:00
He Liu 7a8be255cd
Ss shard management (#7340)
* Storage server shard management with physical shards.

* Cleanup.

* Resolved comments.

* Added `UnlimintedCommitBytes`.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-22 09:30:44 -07:00
A.J. Beamon 17146c484b Use key-backed types for tenants. Add a tenant state field that will be used in upcoming work. Some other tenant related refactoring. 2022-07-21 20:33:28 -07:00
A.J. Beamon 410f27412b
Merge pull request #7620 from sfc-gh-ajbeamon/make-tuple
Add a Tuple::makeTuple function to easily construct a tuple
2022-07-20 17:09:10 -07:00
A.J. Beamon 4d036ae339
Merge pull request #7626 from sfc-gh-ajbeamon/tenant-metadata-change
Some changes to the tenant metadata
2022-07-20 14:50:00 -07:00
Josh Slocum e0f6897481 fixing merge conflicts 2022-07-20 09:11:37 -05:00
Josh Slocum 3f8f44dd16 Merge branch 'main' into improved_cf_testing 2022-07-20 08:19:52 -05:00
Josh Slocum 0b674bf0c4 Merge branch 'main' into cf_bw_operational_fixes 2022-07-20 08:03:59 -05:00
Josh Slocum fd9201f60b Merge branch 'main' into cf_metadata_rewrite 2022-07-20 07:55:00 -05:00
Jingyu Zhou 72e72f6d87 Add getMappedRange latency metrics 2022-07-19 16:31:24 -07:00
Jingyu Zhou fde5aeeb62 Remove unused "readQueueSizeMetric" 2022-07-19 14:38:10 -07:00
A.J. Beamon 537ceff8ac Remove the ability to configure a tenant subspace. Rename the prefixes used for tenant metadata. 2022-07-19 14:32:05 -07:00
A.J. Beamon 190ad8c7e9 Convert existing tuple usages to use Tuple::makeTuple() 2022-07-19 13:45:59 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Josh Slocum 19893b4b7a Persisting change feed metadata version to fix fetch/destroy race across restarts 2022-07-19 14:20:14 -05:00
Josh Slocum acc5aee89e Fixed change feed validation issue 2022-07-19 14:10:10 -05:00