Commit Graph

19514 Commits

Author SHA1 Message Date
Steve Atherton 38190ad7e7
Merge pull request #6737 from sfc-gh-satherton/fix-storage-timestamps
Change storage metadata and perpetual wiggle timestamps to double epoch seconds
2022-04-02 09:47:23 -07:00
Steve Atherton b179813989 Updated status schema and fixed spacing. 2022-04-01 17:21:35 -07:00
Steve Atherton 6eb1c2ae48
Merge pull request #6574 from sfc-gh-satherton/redwood-rare-bugs
Rare correctness bug fixes in Redwood
2022-04-01 16:40:22 -07:00
Josh Slocum 377e252fcf
Better split sizing in blob manager (#6725) 2022-04-01 16:09:46 -07:00
Bharadwaj V.R d248b73df5
Merge pull request #6648 from sfc-gh-bvr/ssupdateb4registration
Update StorageServers before registering SSI
2022-03-31 20:11:19 -07:00
Bharadwaj V.R f749aac223
Merge branch 'apple:main' into ssupdateb4registration 2022-03-31 18:59:44 -07:00
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
Bharadwaj V.R 8ff3b7d8a2
Merge branch 'apple:main' into ssupdateb4registration 2022-03-31 16:12:06 -07:00
Xiaoxi Wang c7d2f5fee2
Merge pull request #6739 from sfc-gh-jslocum/ddq_assert
fix destination limiting and cancelling logic in move_to_removed_serv…
2022-03-31 14:22:09 -07:00
Tao Lin 001909be08
Fixes for when getMappedRange cannot parse as tuple (#6665) 2022-03-31 14:06:45 -07:00
He Liu 966caadb3e
Merge pull request #6706 from kakaiu/Fix-block-cache-recreation-issue
Fix RocksDB Block Cache Recreation Problem
2022-03-31 13:50:15 -07:00
A.J. Beamon dee1293da2
Merge pull request #6731 from sfc-gh-ajbeamon/fix-tenant-memory-leak
Use a TenantState object in the MVC implementation to help manage tenant lifetime
2022-03-31 13:35:20 -07:00
A.J. Beamon b6a0eeda11
Merge pull request #6740 from sfc-gh-ajbeamon/docs-tenant-open-note
Add a note that opening a tenant does not check whether that tenant exists in the cluster
2022-03-31 13:35:04 -07:00
Josh Slocum 9e06881673 fix destination limiting and cancelling logic in move_to_removed_server case 2022-03-31 14:05:15 -05:00
A.J. Beamon 5469b57a2b Add a note that opening a tenant does not check whether that tenant exists in the cluster 2022-03-31 11:39:50 -07:00
A.J. Beamon 68f15650a1 Make sure closed and tenantUpdater are read/written in the same critical section. 2022-03-31 11:12:01 -07:00
A.J. Beamon 15b3465b49
Merge pull request #6647 from sfc-gh-ajbeamon/fdb-tenant-documentation
Add some basic documentation for tenants
2022-03-31 10:33:01 -07:00
A.J. Beamon d813caff10
Merge pull request #6644 from sfc-gh-ajbeamon/fdb-tenant-java
Java support for tenants
2022-03-31 08:32:18 -07:00
A.J. Beamon b6350a2535 Add a note that use of special keys may implicitly enable raw access on a transaction. 2022-03-31 08:29:31 -07:00
A.J. Beamon 860ede0c2e Add some basic documentation for tenants 2022-03-31 08:29:31 -07:00
Steve Atherton 35a5215702 Merge branch 'main' of https://github.com/apple/foundationdb into fix-storage-timestamps 2022-03-31 00:54:21 -07:00
Steve Atherton 4b4082fe62
Merge pull request #6738 from sfc-gh-xwang/cmake-fix
fix cmake bug when no link or compile options
2022-03-31 00:51:59 -07:00
Xiaoxi Wang 61e374f301 fix cmake bug when use gcc 2022-03-30 23:21:22 -07:00
Bharadwaj V.R 88c8b5f2a9
Merge branch 'apple:main' into ssupdateb4registration 2022-03-30 21:16:55 -07:00
Xiaoxi Wang d6d4596c35
Merge pull request #6709 from sfc-gh-ajbeamon/fix-fdbcli-bugs
Fix issues with command completion for exclude and storage_migration_type
2022-03-30 19:53:49 -07:00
Steve Atherton 26e34a947f
Merge pull request #6724 from sfc-gh-yiwu/file_latency
AsyncFileKAIO: add latency histograms
2022-03-30 19:17:34 -07:00
Steve Atherton 6744e9e4f9 Change timestamps used in storage server metadata and perpetual wiggle metrics to epoch seconds, stored as doubles, and stringified as either floating point epoch seconds or timestamp strings of the form "2013-04-28 20:57:01.000 +0000". 2022-03-30 18:57:06 -07:00
Steve Atherton 2c9b2dd005 Merge commit '1b919f52e928e8a72d5acba9175eae32ed4b0c90' into redwood-rare-bugs
# Conflicts:
#	flow/ThreadHelper.actor.h
2022-03-30 18:21:03 -07:00
Steve Atherton 84f9e00258 Remove duplicative generic actor repeatEvery() since recurring() exists. 2022-03-30 18:10:29 -07:00
A.J. Beamon be7315473a Move tenant creation/deletion into a TenantManagement class 2022-03-30 16:31:28 -07:00
A.J. Beamon 6570ae44c6 Use special keys to create/delete tenants 2022-03-30 16:31:28 -07:00
Jon Fu 995f1b3601 Support tuples in Java tenants 2022-03-30 16:31:28 -07:00
A.J. Beamon a2a97e7176 Add tenant support to the Java bindings 2022-03-30 16:31:28 -07:00
Andrew Noyes 1b919f52e9
Combine vector_like_traits::{insert,reserve} (#6689)
* Combine vector_like_traits::{insert,reserve}

and explain semantics better. This should make it more clear what
implementers need to do when implementing the vector_like_traits
concept.

* Update std::unordered_set vector_like_traits impl
2022-03-30 16:29:35 -07:00
Steve Atherton 1d90f17c20
Merge pull request #6721 from sfc-gh-satherton/threadmulticallback-refactor
Remove unordered_map of pointers from ThreadMultiCallback
2022-03-30 14:51:51 -07:00
Steve Atherton d6e2d2a1fe Fix nondeterminism in StorageWiggleMetrics caused by use of timer_int(). 2022-03-30 14:48:01 -07:00
Steve Atherton 2a52c76b7a Added INetwork::timer_int() for convenience. Clarified what timer_int() actually returns in header comments. 2022-03-30 14:47:24 -07:00
Andrew Noyes 88a439e156
Fix typo from pull request #6698 (#6729) 2022-03-30 13:40:57 -07:00
Steve Atherton 75247affa3 Renamed member for better readability. 2022-03-30 13:39:33 -07:00
A.J. Beamon c7d53b31ee Use a TenantState object in the MVC implementation to help manage tenant lifetime. 2022-03-30 12:52:27 -07:00
Steve Atherton 5d74e4d091 Added comments to explain some invariants with ThreadMultiCallback and ThreadCallback and how they are enforced. 2022-03-30 12:38:57 -07:00
Steve Atherton 637a6f654b Merge commit 'fc68cdf45f194d198beb8848c49e3081eff6c396' into threadmulticallback-refactor
# Conflicts:
#	fdbclient/BackupAgent.actor.h
2022-03-30 12:27:03 -07:00
A.J. Beamon fc68cdf45f
Merge pull request #6693 from sfc-gh-ajbeamon/python-tenant-management-module
Move Python tenant management to its own module
2022-03-30 12:03:08 -07:00
Steve Atherton e6457b1656 A few changes for clarity / readability. 2022-03-30 11:31:46 -07:00
A.J. Beamon 16cc74e910 Remove documenation for Database.delete_tenant. 2022-03-30 11:01:54 -07:00
Yi Wu acfee48894 AsyncFileKAIO: add latency histograms 2022-03-30 10:45:04 -07:00
Bharadwaj V.R d56066ee91
Merge branch 'apple:main' into ssupdateb4registration 2022-03-30 09:47:20 -07:00
Xiaoxi Wang e2c66de28f
Merge pull request #6722 from sfc-gh-satherton/fix-backup-backward-compatibility
Fixed Codec<Reference<IBackupContainer>> backward compatibility bug from #6705
2022-03-30 08:38:48 -07:00
Steve Atherton c5995f2e27
Can't enforce size limit or downgrades from future versions could break. 2022-03-29 23:53:45 -07:00
Steve Atherton bf505ed816
Simplify conditional.
Co-authored-by: Renxuan Wang <renxuan@apple.com>
2022-03-29 23:44:41 -07:00