24ce8c0fd0
Commit Proxy Encryption Code Probes ( #8618 )
...
* add commit proxy encryption code probes
* fix comment
* address pr comments
* address pr comments
* address pr comments
* address pr comments
* Trigger Build
2022-10-31 20:04:42 -07:00
bb0236433c
Merge pull request #8540 from sfc-gh-xwang/feature/main/storageMetrics
...
Make MockStorageServer serve StorageMetrics related request
2022-10-25 17:29:21 -07:00
74212eeacf
Encapsulate CounterCollection
2022-10-25 10:17:15 -07:00
3c67b7df39
extract serveStorageMetricsRequests template function
2022-10-24 09:58:41 -07:00
df5825ff65
Merge pull request #8398 from sfc-gh-anoyes/anoyes/idempotency-id2
...
Initial work for automatic idempotency
2022-10-13 13:07:14 -07:00
ac6aaf3785
encryption: fix some data not being encrypted ( #8403 )
...
Changes:
1. Change `isEncryptionOpSupported` to not check against `clientDBInfo.isEncryptionEnabled`, but instead against ENABLE_ENCRYPTION server knob. The problem with clientDBInfo is before its being broadcast to the workers, its content is uninitialized, during which some data (e.g. item 2) is not getting encrypted when they should.
2. Fix CommitProxy not encrypting metadata mutations which are recovered from txnStateStore
3. Fix KeyValueStoreMemory (thus TxnStateStore) partial transaction coming from recovery is not encrypted
4. new CODE_PROBE for the above fixes
5. Logging changes
2022-10-12 14:18:56 -07:00
aad3899cb7
Enforce the lifetime of a write tx is actually <= MAX_WRITE_TRANSACTION_LIFE_VERSIONS
2022-10-11 13:46:39 -07:00
23edfd0d59
Fix formatting
2022-10-04 18:33:30 -06:00
550488b020
Merge remote-tracking branch 'origin/main' into bugfixes/open-for-ide
...
# Conflicts:
# bindings/c/CMakeLists.txt
# fdbclient/include/fdbclient/GetEncryptCipherKeys.actor.h
# fdbserver/BackupWorker.actor.cpp
# fdbserver/BlobWorker.actor.cpp
# fdbserver/CommitProxyServer.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/include/fdbserver/GetEncryptCipherKeys.actor.h
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/PhysicalShardMove.actor.cpp
# flow/CMakeLists.txt
2022-10-04 18:27:48 -06:00
97dfc6823f
fixed build with OPEN_FOR_IDE
2022-10-04 17:01:02 -06:00
8d591fc5e7
address pr comments
2022-08-02 10:51:41 -07:00
af6db42b1b
temp
2022-08-01 15:19:21 -07:00
1de37afd52
Make TEST macros C++ only ( #7558 )
...
* proof of concept
* use code-probe instead of test
* code probe working on gcc
* code probe implemented
* renamed TestProbe to CodeProbe
* fixed refactoring typo
* support filtered output
* print probes at end of simulation
* fix missed probes print
* fix deduplication
* Fix refactoring issues
* revert bad refactor
* make sure file paths are relative
* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
4103b16793
Fix a performance bug in Resolver
...
The erased bytes from recent state transactions was accidently set to 0 and
cause anyPopped to be always false. Thus, when totalStateBytes will never
decrease and can cause high latency between "Resolver.resolveBatch.Before" and
"Resolver.resolveBatch.AfterQueueSizeCheck", in hundreds of milliseconds. This
will cause high server side commit latency.
2022-07-08 09:58:28 -07:00
ec9c3cf05c
Remove unused variables
2022-06-28 16:41:58 -04:00
6246664006
Support encrypting TxnStateStore ( #7253 )
...
Adding encryption support for TxnStateStore. It is done by supporting encryption. for KeyValueStoreMemory. The encryption is currently done on operation level when the operations are being write to the underlying log file. See inline comment for the encrypted data format.
This PR depends on #7252 . It is part of the effort to support TLog encryption #6942 .
2022-06-14 13:26:32 -07:00
3caa3bc595
Rename variable names
2022-05-06 12:33:19 -07:00
0cce5feae3
Optimize LogPushData to avoid constructing LogSystemConfig
...
Which seems to be a CPU hot spot in our testing.
2022-05-06 12:33:19 -07:00
b970d507c0
Avoid creating LogPushData when PROXY_USE_RESOLVER_PRIVATE_MUTATIONS is off
...
To save CPU cost, especially for creating LogSystemConfig.
2022-05-05 08:45:40 -07:00
dc9e782ccc
OpenTelemetry Tracing Perf Fixes ( #6990 )
2022-05-02 14:56:51 -05:00
1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. ( #6855 )" ( #6941 )
...
This reverts commit 5df3bac110
2022-04-25 09:29:56 -05:00
5df3bac110
Migrate to OpenTelemetry tracing. ( #6855 )
2022-04-20 09:26:37 -05:00
e43fde16ec
formatting
2022-04-08 17:28:16 -04:00
62975f87d1
Formatting
2022-04-08 15:04:46 -04:00
cfcf0f152c
Merge branch 'main-4a085fc84' into vv
...
Fix Conflicts:
fdbclient/NativeAPI.actor.cpp
fdbserver/ClusterRecovery.actor.cpp
fdbserver/MasterInterface.h
fdbserver/masterserver.actor.cpp
flow/error_definitions.h
2022-03-30 22:28:06 -07:00
e9659b5dd4
Merge branch 'master-PR-6500' into vv
...
Fix Conflicts:
fdbclient/CommitProxyInterface.h
fdbclient/NativeAPI.actor.cpp
fdbserver/masterserver.actor.cpp
2022-03-30 14:53:49 -07:00
a71099471b
Update copyright header dates
2022-03-21 13:36:23 -07:00
250a88e682
Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement.
2022-02-24 12:25:52 -08:00
608dabbf41
Fix Valgrind error of uninitialized data in CommitTransactionRef
2022-02-19 09:02:11 -08:00
1f67250402
Fix problem with PROXY_USE_RESOLVER_PRIVATE_MUTATIONS in resolver.
2022-01-21 10:39:06 -05:00
9544379cdf
rebase
2022-01-20 11:12:33 -05:00
49e89571fa
Set recoverAt to max(all tlogs rv) for recovered (crashed) tLogs in UNICAST mode.
2022-01-04 12:27:20 -05:00
f91afcd626
do not broadcast unless shard changed.
2021-12-14 16:21:58 -05:00
9f4ac866cd
Avoid context switch between appending version list and updating dv
...
Port PR 6117 (Resolver saves shardChanged in recent state transactions)
2021-12-13 13:02:32 -05:00
0222d8669d
fix simulation failures
2021-12-10 09:56:21 -05:00
90ced244eb
Fix -Wunused-but-set-variable warnings
2021-12-01 18:15:53 -08:00
23062b892e
Calculate tpcv on resolvers
2021-10-15 16:40:00 -04:00
e288348675
Fix forceRecovery flag to control private mutations
...
The flag change from initial commit should be ignored, while the rest should be
honored.
2021-10-15 09:47:23 -04:00
fc7325e923
Fix force recovery when generating private mutations
...
Resolver need to skip transactions after forceRecovery flag is true.
2021-10-15 09:47:23 -04:00
0a88d6a94c
Limit resolution balancing to normal keys
...
This is for resolver private mutation feature if used.
2021-10-15 09:47:23 -04:00
34e9932778
Separate \xff keyspace from keyResolvers
...
This is needed to make sure all Resolvers process all metadata mutations.
For commit proxy, we add all read/write conflict ranges for metadata mutations
to all resolvers.
We also send lock_aware flag for each transaction to resolvers.
2021-10-15 09:47:23 -04:00
0dc9c607f4
Add knob PROXY_USE_RESOLVER_PRIVATE_MUTATIONS
...
To control proxy to use private mutations from resolvers or not.
2021-10-15 09:47:23 -04:00
c1d7b03087
CommitProxy uses Resolver's private mutations
...
Note the commit proxy still runs applyMetadataMutations for setting up memory
states, and no longer generates private mutations.
2021-10-15 09:47:23 -04:00
90a27bedf5
fix conflict fdbserver/ApplyMetadataMutation.cpp
2021-10-15 09:47:20 -04:00
9ecdbc39ee
Add private mutations to Resolver reply messages
2021-10-15 09:45:57 -04:00
ad64ee9858
Format fix
2021-10-15 09:45:57 -04:00
ff75d22ce8
Fix test failures
...
Need to be careful what values applyMetadataMutation needs.
Need to call txnStateStore->enableSnapshot() after receiving the broadcast to
allow semiCommit to happen.
2021-10-15 09:45:57 -04:00
fbc6f45190
Generate private mutations at Resolvers
2021-10-15 09:45:57 -04:00
f79ca8d7fb
Add actor error handling in Resolvers
2021-10-15 09:45:57 -04:00
e0184bc8aa
resolve conflict
2021-10-15 09:45:52 -04:00
Nim Wijetunga