180b531691
Merge pull request #8856 from jzhou77/fix
...
Fix a log router race condition that blocks remote tlogs forever
2022-11-17 13:15:39 -08:00
f285a91f6c
Add more debug events
2022-11-17 11:54:21 -08:00
c6ebdd8ae8
Send error when LogRouterPeekPopped happens
...
Otherwise, the remote tlog won't get a response and the parallel peek requests
will never be cleared, blocking subsequent peeks. As a result, remote tlog will
no longer be able to pop the log router, which in turn can no longer peek tlogs.
The whole remote side will become blocked.
2022-11-17 11:54:14 -08:00
8e8c4b4489
Merge pull request #8170 from sfc-gh-sgwydir/ddsketch
...
Use DDSketch for sample data
2022-11-17 10:38:12 -08:00
349f5821d7
Merge pull request #8592 from sfc-gh-jfu/tenant-list-filter
...
Add option to filter the tenant list command for metacluster
2022-11-17 09:04:58 -08:00
950ca22ee2
Merge pull request #8633 from sfc-gh-xwang/feature/main/dataApi
...
Add MockStorageServer data API implementations
2022-11-16 20:28:02 -08:00
3a5cf32f0b
Trying to re-use existing physical shard during data movement
2022-11-16 16:36:00 -08:00
8971b5907c
add comments; mark some methods const
2022-11-16 13:18:40 -08:00
2edb2fbe44
Merge pull request #8833 from sfc-gh-satherton/actor-collection-constant-time
...
Rewrote ActorCollection to be O(1) instead of O(log n) for actor add and completion
2022-11-16 11:11:58 -08:00
91fc3fef4a
[EAR]: Remove usage of EncryptDomainName for Encryption at-rest operations ( #8715 )
...
* [EAR]: Remove usage of EncryptDomainName for Encryption at-rest operations
Description
diff-1: Address review comments
EncryptDomainName is an auxillary information, given EAR encryption domain
matches with Tenants, EncryptDomainName maps to TenantName in the current
code. However, this mapping adds EAR depedency has multiple drawbacks:
1. In some scenarios obtaning consistent mapping of TenantId <-> TenantName
is difficult to maintain. For instance: StorageServer (SS) TLog mutation
pop loop, it is possible that same commit batch contains: TenantMap update
mutation as well as a Tenant user mutation. SS would parse TenantMap update
mutation (FDB System Keyspace encryption domain), process the mutation, but,
doesn't apply it to the process local TenantMap. SS then attempts to process,
Tenant user mutation and fails to decrypt the mutation given TenantMetadaMap
isn't updated yet.
2. FDB codebase uses EncryptDomainId matching TenantId, TenantName is used as
an auxillary information source and feels better to be handled by an
external KMS.
Major changes include:
1. EAR to remove TenantName dependency across all participating processes
such as: CommitProxy, Redwood, BlobGranule and Backup agent.
2. Update EKP and KmsConnector APIs to avoid relying on "domainName"
information being passed around to external KMS EAR endpoints.
Testing
devRunCorrectness - 100K
EncryptKeyProxyTest - 100K
EncryptionOps Test - 100K
2022-11-16 10:26:39 -08:00
72642c7142
Fix StorageServerCollection::addCost write cost calculation
2022-11-16 08:01:54 -08:00
9a258e087b
Track commit costs while ReportCommitCostEstimationRequest is in-flight
2022-11-16 07:37:00 -08:00
6b77bcfc35
fix merge conflict upstream/main
2022-11-16 07:34:42 -08:00
a21f657429
Add Encryption Code Probes for BlobGranule ( #8779 )
...
* add code probes
* address pr comments
* address pr comments
* Trigger Build
* fix
2022-11-16 06:13:55 -08:00
0af0079a62
several getChangeFeedMutations performance improvements based on cpu profiling ( #8741 )
2022-11-16 07:56:54 -06:00
bfefbfee8c
Merge pull request #8705 from sfc-gh-jshim/authz-accept-base64-for-jwt-tenant-name
...
Make token's 'tenants' field base64-encoded (cf. base64url)
2022-11-16 10:17:10 +01:00
959f77d03d
Merge pull request #8730 from sfc-gh-xwang/fix/main/restoreStats
...
Fix the omit StorageWiggler.restoreStats() bug
2022-11-15 22:37:39 -08:00
f9e87240d0
Add unit test that verifies choose/when behavior needed by ActorCollection.
2022-11-15 18:58:50 -08:00
b2451e600a
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/dataApi
2022-11-15 17:04:49 -08:00
70a26ac314
Adding delays between requests to force sim time to advance at a reasonable rate ( #8784 )
2022-11-15 16:29:22 -08:00
56c2651dfb
Simplification after understanding why i is always valid after runnerHandler() is started.
2022-11-15 16:24:39 -08:00
503769ef05
Merge pull request #8496 from sfc-gh-mpilman/bugfixes/machines-attrition-debugging
...
Enable machine attrition injection
2022-11-15 16:32:33 -07:00
3036f2458f
Merge pull request #8803 from sfc-gh-akejriwal/commitproxies
...
Add a transaction option to bypass storage quota enforcement
2022-11-15 15:22:11 -08:00
3e5293d14c
Don't multiply by fungibility ratio in ThroughputCounters::updateCost
...
(We already multiply by this ratio in getWriteOperationCost)
2022-11-15 15:00:39 -08:00
575793db1c
In assignMutationsToStorageServers, divide cost of writes across all
...
storage servers
(write amplification is already handled by
CLIENT_KNOBS->GLOBAL_TAG_THROTTLING_RW_FUNGIBILITY_RATIO)
2022-11-15 15:00:39 -08:00
d1529aafaf
Mark proxy_tag_throttled error retryable
2022-11-15 15:00:39 -08:00
7c76566746
Avoid computing average transaction cost when less than 1 TPS is being run
2022-11-15 15:00:39 -08:00
907d7af966
solve merge conflict upstream/main
2022-11-15 14:59:31 -08:00
82db9415fb
Add const qualifier to more methods
...
This commit mainly targets get* and is* methods
2022-11-15 14:57:32 -08:00
ec615181ce
Mark several more methods const
2022-11-15 14:57:32 -08:00
c9968f5c0c
Mark several methods of ILogSystem, IPeekCursor and LogPushData const
2022-11-15 14:57:32 -08:00
ba04aea3e2
Mark PromiseStream::getFuture non-const
2022-11-15 14:57:06 -08:00
cecb41694f
Storage change feed fetch performance improvements ( #8736 )
2022-11-15 16:22:45 -06:00
16e6ad6afc
Add decorator to new code probes
2022-11-15 23:05:12 +01:00
41ea1678d0
Merge remote-tracking branch 'origin/main' into authz-accept-base64-for-jwt-tenant-name
2022-11-15 22:57:49 +01:00
37f920a9d2
Merge pull request #8829 from sfc-gh-vgasiunas/vgasiunas-lock-ports-for-tests
...
Avoid port collisions in ctests
2022-11-15 13:49:51 -08:00
214db4d17e
formatting
2022-11-15 13:38:55 -08:00
47eef84d8d
disable attrition injection for another rocks specific test
2022-11-15 14:32:58 -07:00
c03f60c618
Update rare code probe annotations
2022-11-15 13:21:25 -08:00
95cf6f9a0f
Avoid Sev40 for rare code probe miss
2022-11-15 13:21:25 -08:00
c9a601cf4b
Add MissedNonRareProbes field to Joshua results
2022-11-15 13:21:25 -08:00
009d8b7d06
Print rareness of code probes in Joshua results summary
2022-11-15 13:21:25 -08:00
99d4bacf5d
Merge remote-tracking branch 'origin/main' into ddsketch
2022-11-15 13:19:42 -08:00
5aedba5c03
Merge branch 'pml-kill-bug' into actor-collection-constant-time
2022-11-15 13:12:57 -08:00
4ce6b28319
Preserve original FIFO destruction order of ActorCollection so the refactor is a 1:1 replacement.
2022-11-15 13:05:16 -08:00
7ed8838094
Update testing section of global-tag-throttling.md
2022-11-15 12:58:34 -08:00
696c451fa6
Reenable ThroughputQuota.toml test
2022-11-15 12:58:34 -08:00
de983818eb
Merge branch 'main' of github.com:apple/foundationdb into commitproxies
2022-11-15 12:56:20 -08:00
13513252f8
Merge pull request #8832 from sfc-gh-satherton/pml-kill-bug
...
Bug fix: Use-after-destruct in PriorityMultiLock
2022-11-15 12:52:38 -08:00
9b1d86a069
Merge branch 'main' of github.com:apple/foundationdb into tenant-list-filter
2022-11-15 12:44:43 -08:00
Jingyu Zhou