Commit Graph

1759 Commits

Author SHA1 Message Date
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
A.J. Beamon ad52c575a8 Merge branch 'main' into feature-metacluster 2022-07-30 18:33:19 -07:00
A.J. Beamon 5a071e1b19 Fix bug in token cache unit test where the expiration time was underflowing. 2022-07-30 14:22:24 -07:00
A.J. Beamon 296f17e3e5 Merge branch 'main' into feature-metacluster 2022-07-29 15:06:48 -07:00
Markus Pilman b600966cf9 fix token cache unit test 2022-07-29 12:47:55 -06:00
A.J. Beamon e8e4f3ad3a Merge branch 'main' into feature-metacluster
# Conflicts:
#	fdbclient/include/fdbclient/Tenant.h
2022-07-28 16:53:29 -07:00
Junhyun Shim 78ea2b40b6 Add assertion to check g_currentDelivery* is unset before delivery 2022-07-28 01:16:42 +02:00
Junhyun Shim da16c59059 Fix Windows build error 2022-07-27 21:13:17 +02:00
Junhyun Shim c6342a6e5b
Merge branch 'main' into features/authz 2022-07-27 20:51:32 +02:00
A.J. Beamon 7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
Josh Slocum 77956dc7ae
Merge pull request #7639 from sfc-gh-jslocum/cf_metadata_rewrite
Change Feed Metadata Rewrite and adding targeted fault injection
2022-07-26 18:10:37 -05:00
Junhyun Shim 5169616b16 Fix unresolved merge conflicts 2022-07-27 00:38:16 +02:00
Junhyun Shim e2a3fedfc7
Merge branch 'main' into features/authz 2022-07-27 00:08:57 +02:00
Renxuan Wang dc9599f2e9
Get network interfaces ready for https proxy. (#7556)
* Move HTTP from fdbclient/ to fdbrpc/.

* Move md5 and libb64 to contrib/.

* Get network interfaces ready for https proxy.

* Rebase
2022-07-25 17:08:32 -07:00
Junhyun Shim 868f070c79 Simplify RequestStream verify() condition for TenantInfo 2022-07-21 13:18:30 +02:00
Junhyun Shim f76021fe7b Apply Clang format 2022-07-21 12:12:48 +02:00
Junhyun Shim e6fe4a62b4 Ensure DeterministicRandom gets used by OpenSSL exclusively in Simulation 2022-07-21 12:11:33 +02:00
Junhyun Shim d35c0f5a3c Make TokenCache cache complete token strings as keys 2022-07-21 11:39:36 +02:00
A.J. Beamon 279296c29f Merge branch 'tenant-metadata-change' into feature-metacluster
# Conflicts:
#	fdbclient/SystemData.cpp
#	fdbclient/Tenant.cpp
#	fdbclient/include/fdbclient/SystemData.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-20 09:18:27 -07:00
Josh Slocum fd9201f60b Merge branch 'main' into cf_metadata_rewrite 2022-07-20 07:55:00 -05:00
Junhyun Shim 0874e553cd Place generateRandomData() under {I|Deterministic}Random 2022-07-20 13:21:11 +02:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
A.J. Beamon 3ceb2a0639 Merge branch 'main' into feature-metacluster 2022-07-19 13:04:54 -07:00
Josh Slocum 78d4d85f3b Adding non-tss delay injection to SS as well 2022-07-19 09:59:14 -05:00
Josh Slocum 0d9bb9f4a5 Added targeted storage server restarts at critical metadata points 2022-07-19 08:33:43 -05:00
Andrew Noyes 989e323991
Attempt to fix windows build (#7600)
* Attempt to fix windows build

The windows build is complaining that some symbols in flow are
duplicate, which seems fair since we're currently compiling all the flow
sources and _also_ linking to flow for flowlinktest. This change makes
it so that we only link flow, and don't compile the flow source files
again.

* Remove source files from fdbclient and fdbrpc link tests
2022-07-18 08:59:03 -07:00
Junhyun Shim d1dc82629e Clean up TenantInfo and related headers 2022-07-18 14:37:17 +02:00
Junhyun Shim ca29cd9f41 Make RequestStreams accept empty TenantInfo
+ Make Authz tokens survive Transaction resets
+ Make tokens explicitly resettable by assigning empty value to authz token TR option
+ Fill out missing trace for some failed token verifications
+ Trace non-sensitive token parts upon failed verification
2022-07-18 13:30:55 +02:00
A.J. Beamon cb499fa5db Merge branch 'main' into feature-metacluster 2022-07-13 15:28:34 -07:00
Junhyun Shim 11a9fe9aff Move JWT "kid" field from claims to header 2022-07-13 20:21:27 +02:00
Junhyun Shim 24317aa6be Fix incorrect signature extraction function 2022-07-13 19:28:00 +02:00
Junhyun Shim ac5436a090 Catch up on changes to main 2022-07-12 16:48:09 +02:00
Josh Slocum 0b0ac16a4c Merge branch 'main' into granule_merging 2022-07-12 09:09:30 -05:00
Markus Pilman 2edbcf2c65
Merge pull request #44 from apple/main
Merge main
2022-07-12 07:51:22 -06:00
Junhyun Shim 545a9a8043 Fix bugs and add token timeout-in-cache test 2022-07-11 16:58:04 +02:00
Junhyun Shim 7f5049239d Fix LRUCache test 2022-07-11 15:27:22 +02:00
Junhyun Shim 61f3d14ec6 Add TokenCache test
Add function authz::jwt::TokenRef::toStringRef() for token tracing
Add StringRef::toStringView()
2022-07-11 13:42:41 +02:00
Markus Pilman f8d66c53a3 add LRU cache and unit test
the boost LRU cache copies the key into the list when
doing `lru::get`. This is problematic if the memory for
the key is owned by the value (which is the case in our
use-case).
2022-07-08 15:54:09 -06:00
Markus Pilman 13684042c1 use a VectorRef for tokens to improve memory&copy
the number of tenants in a token will be typically very small
(maybe 1-2), so hashing seems unnecessary (and potentially expensive).

Also copying the cache entry will be cheaper if VectorRef is used as the
memory will just be ref-counted
2022-07-08 14:38:43 -06:00
Markus Pilman 2ae17233d8 addressed review comments 2022-07-08 11:19:14 -06:00
Markus Pilman f268265d96
Apply suggestions from code review
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
Co-authored-by: Junhyun Shim <junhyun.shim@snowflake.com>
2022-07-08 10:42:44 -06:00
Markus Pilman a2ef5dc72a set token properly in test 2022-07-07 12:00:32 -06:00
Markus Pilman 5a880fd15e Create keys when running simulation 2022-07-07 11:24:45 -06:00
Markus Pilman 56541167c2 send token with tenant info object 2022-07-06 20:58:17 -06:00
A.J. Beamon c9b553663e Merge branch 'main' into feature-metacluster
# Conflicts:
#	bindings/java/src/main/com/apple/foundationdb/TenantManagement.java
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/TenantSpecialKeys.actor.cpp
#	fdbclient/include/fdbclient/KeyBackedTypes.h
#	fdbclient/include/fdbclient/RunTransaction.actor.h
#	fdbclient/include/fdbclient/SpecialKeySpace.actor.h
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-06 15:44:21 -07:00
Lukas Joswiak 618f8455c4 Add test executables to catch missing symbols
Currently, we have code in different folders like `flow/` and `fdbrpc/`
that should remain isolated. For example, `flow/` files should not
include functionality from any other modules. `fdbrpc/` files should
only be able to include functionality from itself and from `flow/`.
However, when creating a shared library, the linker doesn't complain
about undefined symbols -- this only happens when creating an
executable. Thus, for example, it is possible to forward declare an
`fdbclient` function in an `fdbrpc` file and then use it, and nothing
will break (when it should, because this is illegal).

This change adds dummy executables for a few modules (`flow`, `fdbrpc`,
`fdbclient`) that will cause a linker error if there are included
symbols which the linker can't resolve.
2022-07-06 14:49:33 -07:00
Junhyun Shim db449b80f7
Authz JWKS (#7519)
* Add JsonWebKeySet parser/stringifier

* Update header directory

* Make JWKS parser correctness clean for OpenSSL 1.x

Add RSA keygen support

* Make JWKS parser correctness clean for OpenSSL 3.x

+extend unique_ptr for scoped destruction of OpenSSL objects

* Use PKey::{sign|verify}() in TokenSign

* Apply AutoCPointer to MkCert

* Apply Clang format

* JWKS::toStringRef() returns StringRef > Optional<StringRef>

* Fix Mac/Windows build error

* Fix incorrect fix of Mac build

* Fix filename in license comment for AutoCPointer.h

* Refactor complex C macros into function templates
2022-07-05 10:13:09 +02:00
A.J. Beamon aea4d802c6 Merge branch 'main' into feature-metacluster 2022-06-30 15:08:09 -07:00
Xiaoxi Wang 0054b8a754 Merge branch 'main' of https://github.com/apple/foundationdb into bug/remotekv 2022-06-28 23:41:25 -07:00
Xiaoxi Wang c6ff556f06 fix bug by exclude the spawned kv process info when counting dead process; add method 2022-06-28 23:39:22 -07:00