Commit Graph

9800 Commits

Author SHA1 Message Date
Josh Slocum ee1b0cdd43 making purge failures fail test 2022-07-23 13:10:03 -05:00
A.J. Beamon 9459d0c6ca
Merge pull request #7661 from sfc-gh-ajbeamon/tenant-metadata-key-backed-types
Use key-backed types for tenants
2022-07-22 14:01:45 -07:00
Andrew Noyes 79f60691e9
Don't pass non-pod to variadic function (#7668)
fdbserver/storageserver.actor.cpp:654:24: error: cannot pass object of non-trivial type 'const std::string' (aka 'const basic_string<char, char_traits<char>, allocator<char>>') through variadic function; call will abort at runtime [-Wnon-pod-varargs]

Just use fmt instead
2022-07-22 12:51:44 -07:00
Lukas Joswiak 7fb427f4ec Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak a26344b877 Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak 8395c6cf3f Fix Tuple error 2022-07-22 10:37:29 -07:00
Lukas Joswiak 5fa8e3f7d0 Simplify refresh condition 2022-07-22 10:37:29 -07:00
Lukas Joswiak 703aa1d279 Mess with timeout values 2022-07-22 10:37:29 -07:00
Lukas Joswiak 40d403ed5f Reduce global configuration system key reads from proxy
Clients now poll the proxy for the latest global config for a specific
version. The proxy now periodically requests the latest global
configuration data and stores it in memory, enabling it to respond
immediately to clients with the appropriate version.
2022-07-22 10:37:29 -07:00
Lukas Joswiak 56dfdbda83 Add migration timeout 2022-07-22 10:37:29 -07:00
Lukas Joswiak e024db85a0 Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak 7a48a53778 Perform migration automatically on proxy boot 2022-07-22 10:37:29 -07:00
Lukas Joswiak 2e99d5f6cc Batch global config refresh requests 2022-07-22 10:37:29 -07:00
Lukas Joswiak c33e44b0f4 Proxy GlobalConfig reads through GRV proxies
Clients should avoid reading system keys unless authorized. Under global
config, each client reads from the system keyspace to check for new
global config keys. This commit moves these reads to a server role (the
GRV proxies) and sends the results back to GlobalConfig for an in-memory
update.
2022-07-22 10:37:29 -07:00
He Liu 7a8be255cd
Ss shard management (#7340)
* Storage server shard management with physical shards.

* Cleanup.

* Resolved comments.

* Added `UnlimintedCommitBytes`.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-22 09:30:44 -07:00
Zhe Wu 5441eb4610 Add CommitDebug event after Commit Proxy reporting committed version to master 2022-07-21 21:29:53 -07:00
A.J. Beamon 17146c484b Use key-backed types for tenants. Add a tenant state field that will be used in upcoming work. Some other tenant related refactoring. 2022-07-21 20:33:28 -07:00
Andrew Noyes 89141d4b3a
Validate subrange reads in simulation (#7597)
* Add extra validation to special key space reads in simulation

* Fix bugs turned up by validating subrange reads

* Change to validateSpecialSubrangeRead

It is in general not safe to expect that a read from the special key
space returns the same results if performed again, since the
transaction may be being modified concurrently.

* Add comment

* Add comment
2022-07-21 14:42:08 -07:00
He Liu 257bab7311
Allow read range on null shard (#7654)
* Fixed ShardedRocks unit test.

* Removed `UNIT` from noSim/ShardedRocksDBTest.toml.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-21 13:59:49 -07:00
Zhe Wu bd99f4fa3b Log tlog initialization 2022-07-21 13:54:44 -07:00
Trevor Clinkenbeard 9d63a859b0
Merge pull request #7443 from sfc-gh-xwang/readaware
[Read-aware DD] Use KeyRangeStorageMetrics instead of StorageMetrics
2022-07-21 12:37:51 -07:00
A.J. Beamon 410f27412b
Merge pull request #7620 from sfc-gh-ajbeamon/make-tuple
Add a Tuple::makeTuple function to easily construct a tuple
2022-07-20 17:09:10 -07:00
Xiaoxi Wang cd13356964 solve review comments 2022-07-20 16:09:38 -07:00
A.J. Beamon fee131f24d
Merge pull request #7591 from sfc-gh-ajbeamon/key-backed-types-range-read-more
Add support for returning a more flag from key backed types' range reads
2022-07-20 15:57:50 -07:00
A.J. Beamon 4d036ae339
Merge pull request #7626 from sfc-gh-ajbeamon/tenant-metadata-change
Some changes to the tenant metadata
2022-07-20 14:50:00 -07:00
Ata E Husain Bohra 9a3e88df46
Fix ASAN build (#7629)
* Fix ASAN build

Description
  -diff-2: Fix Mac build issues
  -diff-1: Address review comments

Patch addresses the issue where ASAN build failed after introducing
BlobGranule compression.

Testing

ASAN build
2022-07-20 14:14:46 -07:00
Zhe Wu aa2740e21e Increase AllTags field length in TLogReady 2022-07-20 13:17:21 -07:00
Josh Slocum 8ccf18f35c cleanup 2022-07-20 12:52:21 -05:00
Josh Slocum 316b7a5344 Merge branch 'main' into granule_merging_converge 2022-07-20 12:13:48 -05:00
Josh Slocum cc5ba89fcf
Merge pull request #7612 from sfc-gh-jslocum/granule_merging_batch
Granule merging batch
2022-07-20 11:59:55 -05:00
Josh Slocum 12ccd3b146 typo 2022-07-20 09:35:57 -05:00
Josh Slocum 44885aa719 cleanup 2022-07-20 07:46:57 -05:00
Josh Slocum 78b6a96006 Merge branch 'main' into granule_merging_batch 2022-07-20 07:42:26 -05:00
Josh Slocum 4000682578 moving comment 2022-07-20 07:35:30 -05:00
Jingyu Zhou 0f094d2aa1 Fix ScopeEventFieldTypeMismatch error for TLogMetrics
Backport changes made to TLogServer in #4616 back to old TLog implementations.
Old implementations were using SpecialCounter, which is string. The new type
is int64_t. Without the change, we can have many events like:

<Event Severity="30" Time="1658165532.958152" DateTime="2022-07-18T17:32:12Z" Type="ScopeEventFieldTypeMismatch" ID="0000000000000000" EventType="TraceEvent.TLogMetrics" FieldName="QueueDiskBytesUsed" OldType="Int64" NewType="String" ThreadID="2451559325190099131" Machine="10.242.248.26:4703" LogGroup="xxxx" Roles="TL" />
2022-07-19 22:27:29 -07:00
Josh Slocum 12b6f386cb Refactoring granule flush to retry properly on granule rollback 2022-07-19 19:49:20 -05:00
Josh Slocum 166de4b704 Fixing merge convergence bugs 2022-07-19 18:44:40 -05:00
Jingyu Zhou 72e72f6d87 Add getMappedRange latency metrics 2022-07-19 16:31:24 -07:00
Jingyu Zhou fde5aeeb62 Remove unused "readQueueSizeMetric" 2022-07-19 14:38:10 -07:00
A.J. Beamon 537ceff8ac Remove the ability to configure a tenant subspace. Rename the prefixes used for tenant metadata. 2022-07-19 14:32:05 -07:00
A.J. Beamon 190ad8c7e9 Convert existing tuple usages to use Tuple::makeTuple() 2022-07-19 13:45:59 -07:00
Xiaoxi Wang 7865ead54d Merge branch 'main' into readaware 2022-07-19 13:35:33 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
A.J. Beamon 1b81e72604 Add a Tuple::makeTuple function to easily construct a tuple. Update Tuple to allow all types to be passed via .append() so they can be used with makeTuple. 2022-07-19 11:50:58 -07:00
A.J. Beamon c4cd72bea1 Fix a few compile warnings about unused variables 2022-07-19 10:01:41 -07:00
A.J. Beamon 0e03fbd9e7
Merge pull request #7605 from sfc-gh-ajbeamon/fix-trace-log-detail-formatting
Remove some instances of using printable or toString in a trace event detail
2022-07-19 08:24:58 -07:00
Trevor Clinkenbeard 1eec6f993d
Merge pull request #7572 from sfc-gh-akejriwal/tenantquota
Introduce storage quotas per tenant
2022-07-18 20:00:22 -07:00
A.J. Beamon 38855fd786 Remove some instances of using printable or toString in a trace event detail. 2022-07-18 16:24:33 -07:00
Josh Slocum bfab550435 cleanup 2022-07-18 16:26:03 -05:00
Josh Slocum 340a01763b disabling merge convergence check for now 2022-07-18 16:09:14 -05:00