wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,647 Commits 37 Branches 304 Tags 268 MiB
Commit Graph

81 Commits

Author SHA1 Message Date
sfc-gh-tclinkenbeard 04a1347df2 Merge remote-tracking branch 'origin/main' into dd-refactor 2022-02-08 00:33:27 -08:00
Xiaoxi Wang 6dc5921575
createdTime based storage wiggler (#6219) 
* add storagemetadata

* add StorageWiggler;

* fix serverMetadataKey bug

* add metadata tracker in storage tracker

* finish StorageWiggler

* update next storage ID

* change pid to server id

* write metadata when seed SS

* add status json fields

* remove pid based ppw iteration

* fix time expression

* fix tss metadata nonexistence; fix transaction retry when retrieving metadata

* fix checkMetadata bug when store type is wrong

* fix remove storage status json

* format code

* refactor updateNextWigglingStoragePID

* seperate storage metadata tracker and store type tracker

* rename pid

* wiggler stats

* fix completion between waitServerListChange and storageRecruiter

* solve review comments

* rename system key

* fix database lock timeout by adding lock_aware

* format code

* status json

* resolve code format/naming comments

* delete expireNow; change PerpetualStorageWiggleID's value to KeyBackedObjectMap<UID, StorageWiggleValue>

* fix omit start rount

* format code

* status json reset

* solve status json format

* improve status json latency; replace binarywriter/reader to objectwriter/reader; refactor storagewigglerstats transactions

* status timestamp
 2022-02-04 15:04:30 -08:00
sfc-gh-tclinkenbeard 68ec591cf9 Move DDTeamCollection into its own files 2022-02-04 00:39:42 -08:00
Xiaoxi Wang d1959122af consider wiggling when waitUntilHealthy 2021-10-27 09:08:37 -07:00
Xiaoge Su abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
Xiaoxi Wang 85cd2b9945 add perpetualStorageWiggler 2021-05-20 23:31:08 +00:00
Evan Tschannen d2f9bf7eb6 added comments and fixed style 2021-03-16 15:44:49 -07:00
FDB Formatster df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
Steve Atherton 76a4293d1d Merge branch 'release-6.3' 
# Conflicts:
#	fdbserver/fdbserver.actor.cpp
 2021-02-15 02:13:06 -08:00
Steve Atherton f4c9b88908 Merge branch 'release-6.2' into release-6.3 
# Conflicts:
#	cmake/CompileBoost.cmake
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/Error.cpp
 2021-02-15 02:05:03 -08:00
Andrew Noyes 4e184fe236 Fix memory errors 2021-02-11 02:58:21 +00:00
sfc-gh-tclinkenbeard 19816ccdbf Improve DataDistribution const-correctness 2020-12-26 20:22:27 -04:00
Andrew Noyes 7fbc4d7391 Resolve conflicts 2020-12-04 23:58:42 +00:00
Andrew Noyes 877997632d Merge branch 'release-6.3' into anoyes/merge-release-6.3-master 
Include conflict markers for review purposes
 2020-12-04 01:38:07 +00:00
Andrew Noyes dc2bac5670 Resolve conflicts 2020-11-24 19:09:42 +00:00
Andrew Noyes 1f541f02be Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge 
Merge, leaving conflict markers for now
 2020-11-24 16:55:34 +00:00
David Youngworth d64cf8b9e3 Merge branch 6.3 into master 2020-11-17 11:22:45 -08:00
David Youngworth 489ba20641 Fix several merge issues 2020-11-16 14:46:36 -08:00
David Youngworth d0391db862 Merge branch 'release-6.2' into release-6.3 2020-11-16 10:15:23 -08:00
sfc-gh-tclinkenbeard ca8ea3b6ff Fix memory issues caused by cancelling data distribution tracker 2020-11-15 23:52:36 -08:00
Meng Xu 222da17558 Merge branch 'release-6.2' into mengxu/ha-code-read 2020-11-12 13:39:27 -08:00
Meng Xu 063700e4d6 Add comments and questions to HA and tLog code reading 
The comments' correctness need to be confirmed by reviewers.
 2020-10-30 12:14:57 -07:00
Xin Dong 9ef29d0cea Changed getTeamID() to return a string instead of UID as suggested by reviews. 2020-10-26 16:44:52 -07:00
Xin Dong 9b5a02b552 Resolve review comments 2020-10-26 16:44:52 -07:00
Xin Dong 21ad448ad3 Fix macOS build. 2020-10-26 16:44:52 -07:00
Xin Dong 7ebb2e5c09 Piggy back this PR to polish more TraceEvent by: 
- Making it clear that it's tracking machine team info or server team info
- Added ID to both machine team and server team for better trackability
- Attach distributor id to some trace events.
 2020-10-26 16:44:09 -07:00
sfc-gh-tclinkenbeard e0b1f95740 Merge remote-tracking branch 'origin/master' into remove-global-ddenabled-flag 2020-10-21 18:22:08 -07:00
Young Liu 8cc3e4d3c6 Merge release-6.3 into master 2020-10-19 22:51:56 -07:00
sfc-gh-tclinkenbeard 652d753daf Remove global ddEnabled flag 2020-10-17 11:23:52 -07:00
Jingyu Zhou 8f17a1a5d6 Merge branch 'release-6.2' into release-6.3 2020-10-16 15:25:39 -07:00
sfc-gh-tclinkenbeard 91a8367acb Avoid slow task in ~DataDistributionTracker 2020-10-01 11:44:55 -07:00
Evan Tschannen a49cb41de7 Merge branch 'release-6.3' 
# Conflicts:
#	CMakeLists.txt
#	cmake/ConfigureCompiler.cmake
#	fdbserver/Knobs.cpp
#	fdbserver/StorageCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	flow/ThreadHelper.actor.h
#	flow/serialize.h
#	tests/CMakeLists.txt
 2020-07-29 00:31:55 -07:00
Russell Sears ab0d8b0626
Merge pull request #3509 from sfc-gh-anoyes/anoyes/remove-using-relops 
Remove using namespace std::rel_ops
 2020-07-22 11:58:25 -07:00
sfc-gh-tclinkenbeard 9a2ce4c981 Make IDataDistributionTeam const-correct 2020-07-21 11:05:34 -07:00
Andrew Noyes f470ba8316 Remove using namespace std::rel_ops 
This causes the following to not compile anymore

\#include <utility>
\#include <vector>

using namespace std::rel_ops;

int main() {
    std::vector<int> xs;
    return xs.rbegin() != xs.rend();
}

See https://godbolt.org/z/s1977n
 2020-07-10 22:58:15 +00:00
A.J. Beamon b09dddc07e Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3 
# Conflicts:
#	cmake/ConfigureCompiler.cmake
#	documentation/sphinx/source/downloads.rst
#	fdbrpc/FlowTransport.actor.cpp
#	fdbrpc/fdbrpc.vcxproj
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogSystemPeekCursor.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/Status.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	flow/flow.vcxproj
 2020-07-10 15:06:34 -07:00
Evan Tschannen 5e02fd490e fix: the check for if a teamCollection was tracking a source server was unreliable, leading to scenarios where we would temporarily replicate a shard less than teamSIze 2020-06-29 10:02:27 -07:00
A.J. Beamon d128252e90 Merge release-6.3 into master 2020-05-22 09:25:32 -07:00
Chaoguang Lin ef724bf939 Merge remote-tracking branch 'upstream/master' into add-data-distribution-metrics 2020-05-08 18:39:28 -07:00
chaoguang e8b62e48f4 Rename DDMetrics to DDMetricsRef 2020-05-08 17:17:27 -07:00
Evan Tschannen 07cc0a8d74 code cleanup 2020-04-10 17:02:11 -07:00
tclinken 247ab84323 Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics 2020-03-23 17:01:17 -07:00
Evan Tschannen e08f0201f1 merge release 6.2 into master 2020-03-17 12:51:47 -07:00
A.J. Beamon 555db50cd1 Avoid calling into SABTF so frequently. Use a cheaper call that only checks that shards exist. 2020-03-12 11:22:03 -07:00
Evan Tschannen 303df197cf Merge branch 'release-6.2' 
# Conflicts:
#	CMakeLists.txt
#	bindings/c/test/mako/mako.c
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/NativeAPI.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogRouter.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/flow.vcxproj
#	flow/flow.vcxproj.filters
#	versions.target
 2020-03-06 18:22:46 -08:00
Evan Tschannen 125bd13198 fix: in multi-region configurations, the data distribution queue could start too much work, expecting that the remote region would contribute to the read workload 2020-03-04 14:17:17 -08:00
Evan Tschannen 96258b9809 Merge branch 'release-6.2' 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
 2020-02-21 19:09:16 -08:00
Evan Tschannen 08914a2acd Once available space ratio falls below 0.3 avoid moving data to teams with less free space than the median team 2020-02-21 15:14:32 -08:00
A.J. Beamon e1fb568fd1 Merge branch 'release-6.2' into dd-use-available-space 
# Conflicts:
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
 2020-02-20 16:12:42 -08:00
A.J. Beamon e4b483796d Combine some logic that was doing similar computations for free space ratio. 2020-02-20 14:52:08 -08:00