bcb9bc7e62
Set up authz integration test environment using Python venv
...
Add Authz-related support to {tmp|local}_cluster.py
Also fix public key set load/refresh mechanism
2022-08-17 14:57:05 +02:00
c8741385b2
Fix Windows build broken by #7823
2022-08-16 15:39:24 +02:00
18aa638951
Merge pull request #7877 from sfc-gh-mdvorsky/mdvorsky/fix_net2_unittest
...
Fix Net2 unittest flakiness
2022-08-15 13:58:40 -06:00
03435b5133
Update BlobCipher cache to respect EKP/KMS cipherKey TTL ( #7885 )
...
Description
FDB native encryption data at-rest supports two type of cipher-keys
in-memory caching:
1. Revocable keys - with a definite expiry (future timestamp)
2. Non-revocable keys - with or without expiry timestamp and/or
refreshAt timestamp.
Patch update BlobCipherKey in-memory cache to respect EKP/KMS
supplied 'refreshAt' and 'expireAt' timestamp. GetLatestCipher
validates `cipher key freshness' as well as GetCipherKey checks
for 'cipher key liveness' before replying details to the caller.
Patch also optimizes the BlobCipher module logging by taking
following measures:
1. BLOB_CIPHER_DEBUG macro to guard spammy log messages needed
mostly for debugging failures.
2. Minimize log volume by logging cipherKey details for any new
key added to the cache, key-refreshes are not logged.
3. Categorize logs into: debug, info and warn on per-usecase basis
Testing
devRunCorrectness - 100K
EncryptOps.toml - 100K
2022-08-15 11:17:26 -07:00
8e6009427e
Fix Net2 unittest flakiness
2022-08-13 18:51:30 +02:00
9133d4e16d
Merge pull request #7803 from sfc-gh-xwang/feature/main/ddvisibility
...
Add server selection counter in DDQueue
2022-08-12 15:10:25 -07:00
aa29de8d4c
Merge pull request #7838 from xis19/proxyerror
...
Split the proxy_memory_limit_exceeded to commit/grv version
2022-08-12 10:16:06 -07:00
3cf0914ce0
Fix incorrect parenthesis
2022-08-12 13:52:50 +02:00
312d3da040
Merge remote-tracking branch 'origin/main' into fix-skewed-u32-out-of-range
2022-08-12 13:01:43 +02:00
e57d3a6f9b
Address review comments
2022-08-12 10:24:23 +02:00
0326d53965
Split proxy_memory_limit_exceeded to commit/grv specific exceptions
...
Currently GRV is reporting proxy_memory_limit_exceeded error which has
error message claiming Commit proxy failing. This split should remove
such confusion.
2022-08-12 00:45:57 -07:00
44f8bdd258
Blob Worker memory limit ( #7858 )
...
* Simulation version of blob_worker_full
* tracking blocked BM assignments
* actual memory estimation implementation
2022-08-11 15:07:08 -07:00
583021c2d9
Merge pull request #7772 from sfc-gh-tclinkenbeard/global-tag-throttling6
...
Add status section for global tag throttler
2022-08-11 17:38:31 -03:00
a41f8d2354
Merge pull request #7768 from sfc-gh-ajbeamon/metacluster-api-changes
...
Metacluster metadata changes
2022-08-10 13:19:58 -07:00
1cff154adb
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/ddvisibility
2022-08-10 12:03:42 -07:00
ea0c60381f
merge upstream/main
2022-08-09 12:28:57 -07:00
e89b077b97
Update ParseTree.cs
2022-08-09 14:27:48 -05:00
dfdfe3f9a1
Update Program.cs
2022-08-09 14:27:37 -05:00
0dce3e5d91
Update ActorCompiler.cs
2022-08-09 14:27:15 -05:00
cd82f65ef5
Change the data cluster entry to use a state field rather than a locked boolean
2022-08-09 11:53:57 -07:00
86ee76ec50
Merge pull request #7816 from sfc-gh-mdvorsky/mdvorsky/coverage
...
Flush gcov coverage upon SIGTERM
2022-08-08 12:03:22 -06:00
c9865809e0
Merge pull request #7786 from sfc-gh-mdvorsky/mdvorsky/net2_unittest
...
Add Net2 unittests for ThreadSafeQueue and Net2::onMainThread
2022-08-08 11:23:31 -06:00
e268ae9a86
Run clang-format
2022-08-08 18:48:15 +02:00
b8370e5129
Print SIGNAL output to stdout
2022-08-08 18:28:11 +02:00
77269d0506
Print to stderr only upon errors
2022-08-08 18:11:35 +02:00
79571dd2b4
Testing upgrades to a future version of FDB ( #7780 )
...
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli
* Auto format python files used in upgrade tests
* Add a test for upgrading to a future FDB version
* Emphasize that the options for using future protocol version are intended for test purposes only
* Make the global variable for current protocol version visible only locally
* Refactirng to avoid using currentProtocolVersion() in static intialization
* Update go bindings
2022-08-08 17:29:49 +02:00
722f66beb7
Flush gcov coverage upon SIGTERM
2022-08-08 17:11:20 +02:00
b18e29dd87
Merge remote-tracking branch 'upstream' into feature/main/ddvisibility
2022-08-06 21:43:36 -07:00
c199ea054c
format code
2022-08-06 20:01:35 -07:00
f9866f419a
Propagate environment variables to spaned fdbserver process
2022-08-06 11:09:09 -07:00
671a2f9f8e
Add comment to INetwork
2022-08-05 19:59:06 +02:00
6b790ef4b8
Disable the new onMainThread test in simulation
2022-08-05 17:06:34 +02:00
7c00c57253
Small fixes
2022-08-05 16:30:05 +02:00
7fe2088e09
Add ITagThrottler::getRates method
2022-08-04 16:58:58 -07:00
ff23d5994e
Merge pull request #7729 from sfc-gh-ajbeamon/feature-metacluster
...
Metacluster
2022-08-04 15:29:44 -07:00
90b0d74875
DebugRelocationTraceEvent; TraceInterval randomId:
2022-08-04 15:28:33 -07:00
84d483605b
Merge pull request #7431 from xis19/main
...
Let the storage server reports busiest write tag
2022-08-04 10:23:31 -07:00
9e4af40b0f
Run clang-format
2022-08-04 16:20:42 +02:00
70748f51e4
Add Net2 unittests for ThreadSafeQueue and onMainThread
2022-08-04 16:03:59 +02:00
07eafcec93
Merge pull request #7763 from sfc-gh-xwang/feature/main/unittest
...
move waitForMost into generic actors
2022-08-02 13:30:30 -07:00
4a25f8b692
Merge remote-tracking branch 'origin' into env
2022-08-02 14:51:50 -04:00
473feef246
Set knobs using environment variables
2022-08-02 14:29:37 -04:00
3c76ad9e72
move waitForMost into generic actors
2022-08-02 10:38:11 -07:00
4b66645d80
Granule file performance benchmark and improvements ( #7742 )
...
* added cpu microbenchmark for blob granule files
* Added edge case read benchmarks, and sorting memory deltas
* Sorted merge for granule files
* key block comparison optimization in granule files
* More performance improvements to granule file read
* fixing zlib not supported build
* fixing formatting
* Added debug macro for new debugging prints
* review comments
* more strict compression size validation assert
2022-08-02 11:36:44 -05:00
75b69d7774
fixup! gcc build error
2022-08-01 18:56:50 -07:00
ead1aa5bc1
fixup! Remove C++20 attribute
2022-08-01 18:56:50 -07:00
ec40c6bfec
fixup! Add a wrapper of ResourceWeakRef for better support of self pointer
2022-08-01 18:56:50 -07:00
2ce456bd3f
fixup! Fix minor issues
...
* Reintroduce the anonymous namespace deleted by accident
* Rename the guardian definition in OwningResource.h
2022-08-01 18:56:50 -07:00
bc9d2e8cbe
fixup! Fix per upstream change
2022-08-01 18:56:50 -07:00
cf04afe925
fixup! Non-owning reference to an object
...
See documents in flow/OwningResource.h
2022-08-01 18:56:50 -07:00
26877a8924
fixup! Remove the boost/bind.hpp in TLSTest.cpp
...
Not used, and causes gcc emit warning
2022-08-01 18:56:50 -07:00
1b693a588a
Merge branch 'main' into feature-metacluster
2022-08-01 14:43:14 -07:00
51b92d59b9
Merge pull request #7733 from RenxuanW/hostname-backup
...
Prefer IPv6 in hostname resolving.
2022-08-01 12:34:02 -07:00
4a0bea2230
Document for pickOneAddress().
2022-08-01 10:54:04 -07:00
ad52c575a8
Merge branch 'main' into feature-metacluster
2022-07-30 18:33:19 -07:00
dd902fc772
create ShardsAffectedByTeamFailure.h
2022-07-29 15:12:15 -07:00
8b7b6d1d4c
Various cleanup; change some test parameters; add a test for metacluster management operations
2022-07-29 09:24:06 -07:00
316ea60232
Prefer IPv6 in hostname resolving.
2022-07-28 20:07:16 -07:00
e8e4f3ad3a
Merge branch 'main' into feature-metacluster
...
# Conflicts:
# fdbclient/include/fdbclient/Tenant.h
2022-07-28 16:53:29 -07:00
5db1135ac6
Add MetaclusterOperationContext to make it easier to run multi-transaction operations
2022-07-28 16:44:20 -07:00
65cfc839a6
Some various cleanup
2022-07-28 13:35:24 -07:00
ade9d20780
Merge pull request #7628 from sfc-gh-mpilman/features/code-coverage-test-harness
...
Change test harness to include comments
2022-07-28 13:54:21 -06:00
7f53b340f5
Merge branch 'main' into feature-metacluster
...
# Conflicts:
# bindings/python/tests/fdbcli_tests.py
# fdbcli/TenantCommands.actor.cpp
# fdbclient/ClientKnobs.cpp
# fdbclient/include/fdbclient/ClientKnobs.h
# fdbclient/include/fdbclient/Tenant.h
# fdbserver/BlobManager.actor.cpp
# fdbserver/BlobWorker.actor.cpp
# fdbserver/TenantCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/TenantManagementWorkload.actor.cpp
# flow/include/flow/error_definitions.h
2022-07-28 11:43:21 -07:00
65b441bd1b
Watch public key file at CLI-arg path
...
Argument name: authorization-public-key-file
2022-07-28 20:03:19 +02:00
1dbb13a899
Accept non-mTLS clients as untrusted
2022-07-28 20:03:07 +02:00
5469e58562
Merge remote-tracking branch 'origin/main' into features/code-coverage-test-harness
2022-07-28 10:01:20 -06:00
1e83d7e1ab
Apply code review suggestions
2022-07-28 00:47:14 +02:00
c6342a6e5b
Merge branch 'main' into features/authz
2022-07-27 20:51:32 +02:00
ef46f72d0c
Merge remote-tracking branch 'origin/main' into features/code-coverage-test-harness
2022-07-27 10:41:18 -06:00
7c6b3fb0b8
Merge branch 'main' into feature-metacluster
2022-07-27 08:55:10 -07:00
d39c0b773a
Add a limit to the number of tenants that can be created in a cluster
2022-07-27 08:21:03 -07:00
dec6dbfbfb
Merge pull request #7549 from sfc-gh-ajbeamon/feature-tenant-groups
...
Add support for tenant groups
2022-07-27 07:56:27 -07:00
d5f7294bd2
Fix failing Mac/BoringSSL build
2022-07-27 01:05:56 +02:00
5169616b16
Fix unresolved merge conflicts
2022-07-27 00:38:16 +02:00
e2a3fedfc7
Merge branch 'main' into features/authz
2022-07-27 00:08:57 +02:00
0609aa57d1
Merge branch 'main' into improved_cf_testing
2022-07-26 16:44:13 -05:00
c32e1da908
Merge pull request #7673 from sfc-gh-jslocum/delta_files_v2
...
Sorted Delta Files
2022-07-26 16:04:55 -05:00
7cf6037191
remove runAfter
2022-07-26 12:12:56 -07:00
a64693518a
Add support for tenant groups
2022-07-26 09:04:29 -07:00
dc9599f2e9
Get network interfaces ready for https proxy. ( #7556 )
...
* Move HTTP from fdbclient/ to fdbrpc/.
* Move md5 and libb64 to contrib/.
* Get network interfaces ready for https proxy.
* Rebase
2022-07-25 17:08:32 -07:00
077003c4f5
Cleanup tenant tombstones periodically
2022-07-24 18:24:19 -07:00
742b73028c
Fix incorrect range in randomSkewedUInt32()
...
Currently it invokes pow(10, uniform_rand(log_e(range_begin), log_e(range_end))),
which may overflow beyond UINT32_MAX.
Fix it by preventing log_e(0) case and change pow base to M_E
2022-07-24 23:02:48 +02:00
83a317ef88
Disable guard pages for valgrind build. Closes #7610 ( #7672 )
2022-07-22 16:51:05 -07:00
524c3131c5
Use a more specific error if we reject a tenant property change due to it needing to be moved
2022-07-22 15:43:51 -07:00
ea9018460a
cleanup and polish
2022-07-22 15:13:32 -05:00
9b66e0c1c7
Fix some subtle differences between branches
2022-07-22 12:51:24 -07:00
b308aa6604
Merge branch 'feature-tenant-groups' into feature-metacluster
2022-07-22 11:38:23 -07:00
237081905d
Add support for tenant groups
2022-07-22 09:27:31 -07:00
3e1763e7ac
Merge branch 'main' into feature-metacluster
2022-07-21 14:48:53 -07:00
45c8a4e09d
Build fixes/improvements for sanitizers ( #7657 )
...
* Don't build fdb c shim with ubsan
This avoids duplicate symbols when linking. It doesn't really make sense
to assemble files with -fsanitize=undefined anyway, since it won't
insert instrumentation.
* Consolidate boost_asan with boost_target
2022-07-21 12:43:38 -07:00
908b8faa88
Fix unmatched signatures for BoringSSL
2022-07-21 15:17:34 +02:00
97880880cd
Defer Simulation check in OSSL-IRandom binding until g_network is set
2022-07-21 14:52:02 +02:00
f76021fe7b
Apply Clang format
2022-07-21 12:12:48 +02:00
e6fe4a62b4
Ensure DeterministicRandom gets used by OpenSSL exclusively in Simulation
2022-07-21 12:11:33 +02:00
190711ccd8
Merge pull request #7622 from sfc-gh-ajbeamon/eat-and-connection-string-changes
...
Add some helper functions to ClusterConnectionString and an improvement to StringRef.eat
2022-07-20 16:10:24 -07:00
4d036ae339
Merge pull request #7626 from sfc-gh-ajbeamon/tenant-metadata-change
...
Some changes to the tenant metadata
2022-07-20 14:50:00 -07:00
9a3e88df46
Fix ASAN build ( #7629 )
...
* Fix ASAN build
Description
-diff-2: Fix Mac build issues
-diff-1: Address review comments
Patch addresses the issue where ASAN build failed after introducing
BlobGranule compression.
Testing
ASAN build
2022-07-20 14:14:46 -07:00
279296c29f
Merge branch 'tenant-metadata-change' into feature-metacluster
...
# Conflicts:
# fdbclient/SystemData.cpp
# fdbclient/Tenant.cpp
# fdbclient/include/fdbclient/SystemData.h
# fdbclient/include/fdbclient/Tenant.h
# fdbclient/include/fdbclient/TenantManagement.actor.h
# fdbserver/TenantCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-20 09:18:27 -07:00
b521dc2f58
Bind DeterministicRandom to OpenSSL RNG
2022-07-20 16:03:09 +02:00
5540f92e2e
Refactoring randomExp to the proper place
2022-07-20 08:36:03 -05:00
Junhyun Shim