Commit Graph

355 Commits

Author SHA1 Message Date
Ray Jenkins dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Binglin Chang 408c0cf1c9
Fix compile errors on ubuntu 20.04 (#4931) 2022-04-20 10:00:46 -07:00
Markus Pilman bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
Markus Pilman e2d7d4075d multiple bug fixes 2022-04-07 11:08:07 -06:00
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
Markus Pilman b595d4462f Throw error on unauthorized access 2022-03-29 14:58:43 -06:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Markus Pilman 1fbeca8038 fix memory issue 2022-03-19 11:03:32 +01:00
Josh Slocum 37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
Evan Tschannen 7908ea54f3 fix: NetNotifiedQueueWithAcknowledgements could miss disconnects that happen during the delay(0) in deliver() 2022-03-17 09:57:46 -07:00
Markus Pilman 158da462f7 fix memory bug 2022-03-17 14:37:05 +01:00
Markus Pilman 118b53b7cf Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-17 12:06:44 +01:00
Markus Pilman bed799220a Addressed review comments, added test 2022-03-15 16:57:26 +01:00
sfc-gh-tclinkenbeard baec03090e Fix "guarantee" misspelling 2022-03-14 16:33:09 -07:00
sfc-gh-tclinkenbeard 8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
Josh Slocum a2bbb188c3 Fixing delayed health updater to mark address failed when too many connections triggers before new connection starts, instead of after 2022-03-11 16:50:38 -06:00
Markus Pilman 8fac0081a8 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-09 11:00:00 +01:00
Markus Pilman 0aa7823622 Pass object as reference 2022-02-25 14:54:52 +01:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Markus Pilman 20bf3e1599 Address review coomments 2022-02-23 19:02:29 +01:00
Markus Pilman cf31e14904 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-02-23 10:29:32 +01:00
Markus Pilman 102169ba33 Ran clang-format 2022-02-23 10:23:27 +01:00
Markus Pilman dc973fb67e Allow List and first test 2022-02-22 11:15:16 +01:00
Markus Pilman c7899b9d39 Implemented public endpoints, started allow list 2022-02-16 18:12:03 +01:00
Trevor Clinkenbeard 3b4cbded96
Merge pull request #6374 from sfc-gh-ajbeamon/protocol-version-pretty-print
Log protocol version as a hex string rather than a uint64_t
2022-02-14 13:51:00 -08:00
A.J. Beamon 1b7e790211 When inserting multiple streams into an EndpointMap, don't assign them in the well-known endpoint space 2022-02-11 12:59:12 -08:00
A.J. Beamon 0a1bcf7a29 Log protocol version as a hex string rather than a uint64_t. 2022-02-10 13:57:48 -08:00
Renxuan Wang 46d17d748f Change member variable fromHostname to type bool.
So that it can be serialized.
2021-11-23 14:25:02 -10:00
Steve Atherton e752fdd69f
Optimization in FlowTransport's use of XXHash to avoid a malloc() when using the stream API and to not use the stream API when a message being sent is contiguous within a single packet buffer block. (#5970) 2021-11-12 11:45:21 -08:00
Renxuan Wang f15ceb5489 Add Hostname struct, and fromHostname in NetworkAddress struct. 2021-11-04 11:42:28 -07:00
Steve Atherton dc3f46c2ae Change asserts to throw internal_error as these function calls should never be compiled out. 2021-11-03 11:00:12 -07:00
Steve Atherton f3d62ce002 Assert success on XXHash stream functions. 2021-11-02 23:40:53 -07:00
Steve Atherton 313a3dcd63 Change FlowTransport to use 64 bit XXHash instead of CRC32. 2021-11-02 23:16:03 -07:00
Vaidas Gasiunas 16171d8252 Refactoring well-known endpoint registration
- List all well-known endpoints of FDB in a single enum
- Identify well-known endpoints by plain IDs
2021-09-21 11:05:31 -06:00
Lukas Joswiak d6a7a469ef Remove fdbclient dependency 2021-08-23 10:25:07 -07:00
sfc-gh-tclinkenbeard 658233d0e0 Merge remote-tracking branch 'origin/master' into paxos-config-db 2021-08-19 16:18:59 -07:00
sfc-gh-tclinkenbeard 7d87aa8203 Increase well known endpoint index for WLTOKEN_PROCESS 2021-08-16 12:09:56 -07:00
Xiaoxi Wang 2df0474fec merge master 2021-08-02 11:58:35 -07:00
Trevor Clinkenbeard 5e1639ad32
Merge pull request #5222 from sfc-gh-tclinkenbeard/paxos-config-db
Simple (non-fault tolerant) implementation of replicated configuration database
2021-08-02 10:03:24 -07:00
Xiaoxi Wang 10c82b422f merge master branch 2021-07-28 14:19:46 -07:00
sfc-gh-tclinkenbeard 59ddfd78bd Merge remote-tracking branch 'origin/master' into paxos-config-db 2021-07-28 12:54:02 -07:00
sfc-gh-tclinkenbeard c74047c665 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-28 11:51:02 -07:00
Evan Tschannen 81f93d794f Merge branch 'master' of https://github.com/apple/foundationdb into fix-ordered-delay 2021-07-27 13:56:35 -07:00
Evan Tschannen 256a18e43b Flow transport uses an ordered delay to avoid out of order reply promise stream messages 2021-07-27 12:01:32 -07:00
Steve Atherton 507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
Xiaoxi Wang 1587192331 temperary change to fix local out-of-order delivery 2021-07-26 09:36:23 -07:00
sfc-gh-tclinkenbeard b20e02ca25 Fix more -Wreorder-ctor warnings across several files 2021-07-24 11:43:19 -07:00
sfc-gh-tclinkenbeard 3442ebd3b7 Fix more -Wreorder-ctor warnings across many files 2021-07-24 11:20:51 -07:00
sfc-gh-tclinkenbeard 6f81155784 Merge remote-tracking branch 'origin/master' into const-serverdbinfo 2021-07-20 10:18:40 -07:00