Commit Graph

71 Commits

Author SHA1 Message Date
Andrew Noyes 989e323991
Attempt to fix windows build (#7600)
* Attempt to fix windows build

The windows build is complaining that some symbols in flow are
duplicate, which seems fair since we're currently compiling all the flow
sources and _also_ linking to flow for flowlinktest. This change makes
it so that we only link flow, and don't compile the flow source files
again.

* Remove source files from fdbclient and fdbrpc link tests
2022-07-18 08:59:03 -07:00
Lukas Joswiak 618f8455c4 Add test executables to catch missing symbols
Currently, we have code in different folders like `flow/` and `fdbrpc/`
that should remain isolated. For example, `flow/` files should not
include functionality from any other modules. `fdbrpc/` files should
only be able to include functionality from itself and from `flow/`.
However, when creating a shared library, the linker doesn't complain
about undefined symbols -- this only happens when creating an
executable. Thus, for example, it is possible to forward declare an
`fdbclient` function in an `fdbrpc` file and then use it, and nothing
will break (when it should, because this is illegal).

This change adds dummy executables for a few modules (`flow`, `fdbrpc`,
`fdbclient`) that will cause a linker error if there are included
symbols which the linker can't resolve.
2022-07-06 14:49:33 -07:00
Markus Pilman 20e7371036 some Linux fixes 2022-06-28 08:51:13 -06:00
Markus Pilman 3448bc6f61 fix fdbmonitor 2022-06-27 19:11:24 -06:00
Markus Pilman 88af6b403f fdbrpc compiling 2022-06-27 18:11:55 -06:00
Markus Pilman a47ed89018 Linux fixes and addressed review comments 2022-06-23 20:52:13 -06:00
Markus Pilman 8af056e7b0 fdbclient now compiling 2022-06-23 18:05:36 -06:00
Markus Pilman 9d80ee0cb6 fdbrpc is compiling 2022-06-23 17:15:49 -06:00
Markus Pilman ffaf15c12a moved wellknownendpoints and fixed some includes 2022-06-23 17:03:53 -06:00
Markus Pilman 10e478dfc3 Flow is compiling 2022-06-23 16:35:19 -06:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Trevor Clinkenbeard 942d687506
Clean up includes in actor header files (#7331)
* Remove unnecessary actorcompiler.h includes (from non-actor files)

* Make AsyncFileChaos a non-actor header file

* Add unactorcompiler.h include to the end of actor header files

* Add missing actorcompiler.h includes to actor header files
2022-06-13 13:26:51 -07:00
Junhyun Shim 3e79735b2f
Authz JWT support (#7279)
* Add JWT support to TokenSign

* Encapsulate OpenSSL public/private key type

Type-safe passing around of keys without having to DER/PEM-serialize
(OpenSSL doesn't have distinct types for public and private key)

* Apply Clang format

* Add verify benchmark for JWT and FlatBuffers token

* Unit test base64url::{encode, decode}

* Make all payload fields optional

Let user code validate non-signature fields

* Make all payload fields optional

Completely defer field check to user code

* Move rapidjson from fdbclient to contrib

* Make fdbrpc's rapidjson linkage private

Currently only sources include them.

* Modify rapidjson path in apiversioner.py

* Algorithm::Unknown > Algorithm::UNKNOWN
2022-06-02 13:22:50 +02:00
sfc-gh-tclinkenbeard 8ea68154bf Remove WITH_TLS CMake variable 2022-05-02 22:45:00 -07:00
Ata E Husain Bohra 333aadb903
Interface to enable clients to send/receive REST requests/responses (#6866)
* Interface to enable clients to send/receive REST requests/responses

Description

Major changes:
1. Add RESTClient interface enabling client to send/receive REST HTTP
   requests. Support REST APIs are: get, head, put, post, delete, trace
2. Add RESTUtil file introducing below interfaces:
 2.1. RESTUrl - Extract URI information: host, service, request-parameters.
 2.2. RESTConnectionPool-
      Connection establishment, life-cycle management, connection-pool (TTL)
 2.3. RESTClientKnobs - supports REST Knob parameter management and updates

Testing

Unit test - fdbrpc/RESTClient, fdbrpc/RESTUtils
2022-04-27 12:17:52 -07:00
Junhyun Shim b6a0c0f942 Merge remote-tracking branch 'upstream/main' into tenant-token-sign 2022-04-13 19:55:37 +02:00
Markus Pilman bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
Junhyun Shim 99fe104f98 Sign and verify auth tokens for multi-tenant FDB 2022-03-24 19:04:00 +01:00
Markus Pilman 117ee637db Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-15 17:17:47 +01:00
Mohamed Oulmahdi 9e81945301 Enable encryption for Windows 2022-03-11 10:06:07 +01:00
Markus Pilman dc973fb67e Allow List and first test 2022-02-22 11:15:16 +01:00
Andrew Noyes 23de546586 Only build libcoro if COROUTINE_IMPL == libcoro
It doesn't build at all on mac m1
2021-12-03 16:34:35 -08:00
Andrew Noyes b6fd402a3c Add option to use boost or libcoro
By default, use boost everywhere except windows and linux x86 (for
performance reasons)
2021-11-29 13:14:15 -08:00
sfc-gh-tclinkenbeard e918d20f68 Document "averaged" parameter of PerfMetric constructor 2021-08-29 15:38:21 -07:00
Mohamed Oulmahdi c1bb977a93 Set sampling dependencies for Windows only 2021-08-24 10:48:31 +02:00
Mohamed Oulmahdi 2c5510011c Add dependencies between sampling and non-sampling actors projects 2021-08-24 10:48:31 +02:00
Lukas Joswiak 59d535149e Merge branch 'master' into fixes/alp6 2021-07-27 10:07:18 -07:00
Lukas Joswiak e9a1679467 Disable sampling everywhere except fdbserver 2021-07-27 09:53:23 -07:00
sfc-gh-tclinkenbeard ad03a4787a Fix non-TLS build 2021-07-09 21:06:15 -07:00
sfc-gh-tclinkenbeard 81b8292094 Merge remote-tracking branch 'origin' into encrypt-backup-files 2021-06-25 12:21:12 -07:00
Evan Tschannen 1cec223434 cherry pick streaming replies to master 2021-06-19 09:47:13 -07:00
Josh Slocum ce82c9653e Testing Storage Server implementation 2021-05-25 20:28:50 +00:00
sfc-gh-tclinkenbeard a0c49234b2 Merge remote-tracking branch 'origin/master' into encrypt-backup-files 2021-03-19 20:47:53 -07:00
Markus Pilman d13e240496 use libcoroutine for Windows
The Windows CI still doesn't have boost context installed
so we use the old implementation for now
2021-02-17 09:23:41 -07:00
Markus Pilman 534fa45291 Merge branch 'master' of github.com:apple/foundationdb into features/coroutine2 2021-02-01 16:31:16 -07:00
sfc-gh-tclinkenbeard 511ec43e62 Merge remote-tracking branch 'origin/master' into encrypt-backup-files 2021-01-27 11:17:58 -08:00
Markus Pilman d3a85b555a CoroFlow and remove libcoro 2021-01-26 17:23:17 -07:00
sfc-gh-tclinkenbeard 1612c44988 Disable StreamCipher for Windows 2021-01-25 12:36:35 -08:00
sfc-gh-tclinkenbeard 037279c843 Disable StreamCipher if TLS is disabled 2021-01-25 10:42:14 -08:00
sfc-gh-tclinkenbeard 88bc157bd0 Added AsyncFileEncrypted 2021-01-24 22:53:04 -08:00
sfc-gh-tclinkenbeard 61a29ecfc8 Merge remote-tracking branch 'origin/master' into run-minio-joshua 2021-01-01 09:38:36 -04:00
sfc-gh-tclinkenbeard fed1a03950 Added SimExternalConnection (most methods not implemented yet) 2020-12-24 21:20:02 -04:00
Markus Pilman 390a82b6a2
Fix old typo
Co-authored-by: Andrew Noyes <63815641+sfc-gh-anoyes@users.noreply.github.com>
2020-12-09 13:51:50 -07:00
Markus Pilman 9344a5fd0a Fix Coro ucontext bug introduced in #4109 2020-12-09 12:25:31 -07:00
David Youngworth 3192984309 Remove unused zlib library 2020-08-11 17:15:07 -07:00
Meng Xu ef8c1060a2 Merge branch 'master' into mengxu/tmp-merge-6.3 2020-07-13 10:15:56 -07:00
A.J. Beamon b09dddc07e Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
# Conflicts:
#	cmake/ConfigureCompiler.cmake
#	documentation/sphinx/source/downloads.rst
#	fdbrpc/FlowTransport.actor.cpp
#	fdbrpc/fdbrpc.vcxproj
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogSystemPeekCursor.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/Status.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
A.J. Beamon 04d1217941 Track statistics about server-side request latency on each process, to include min, max, mean, and various percentiles. 2020-07-09 16:39:15 -07:00
Jingyu Zhou d972e072b7 Remove unused batcher.actor.h 2020-07-07 21:54:56 -07:00