Commit Graph

45 Commits

Author SHA1 Message Date
Meng Xu ced4f8c49f fix: getReadHotRanges should be const function 2020-08-12 22:18:28 -07:00
Meng Xu 002cca5e58 Merge branch 'master' into tmp/merge-PR
Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/StorageMetrics.actor.h
2020-08-12 20:08:13 -07:00
Xin Dong 3ac6996844 Increased some knobs to throttle the spammy read hot logging. Also added more details inside the read hot log to make it useful 2020-08-12 14:27:34 -07:00
Jingyu Zhou 2e5ae8eed5 Change iterator name to small case 2020-07-10 10:56:41 -07:00
sfc-gh-tclinkenbeard 2a393633b6 Improved const-correctness
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:

- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
2020-06-17 18:20:43 -07:00
Xin Dong 696d95ebca Fix the possible dead loop when endKey got reset to equal to the beginKey. 2020-05-12 08:12:15 -07:00
Xin Dong 81184e79c5 Fixed a bug where the loop might get stuck. 2020-05-11 15:56:51 -07:00
Evan Tschannen f329164fb4
Merge pull request #2532 from dongxinEric/feature/hot-read-key-detection-part-2
Feature/hot read key detection part 2
2020-05-05 14:33:34 -07:00
Xin Dong 3ba8daae9e Partially resolve some of the review comments 2020-04-30 11:31:59 -07:00
Xin Dong 76eb1c4376 Address review comments from Meng 2020-04-21 11:00:19 -07:00
Xin Dong ff8810ab45 Remove debug logs 2020-04-17 13:35:22 -07:00
Xin Dong 7dd7406c59
Merge branch 'master' into feature/hot-read-key-detection-part-2 2020-04-16 14:54:05 -07:00
Balachandar Namasivayam 73272fc72e Version difference is now the diff between TLog versions and SS version. 2020-04-03 19:04:43 -07:00
Balachandar Namasivayam ad1dd4fd9b Mark the storage servers that are continually lagging as unhealthy and so this will give the Data Distributor the chance to move data out of this server. 2020-03-31 18:25:39 -07:00
Evan Tschannen 96258b9809 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
2020-02-21 19:09:16 -08:00
A.J. Beamon 4c9c736253 Data distribution uses available space instead of free space when evaluating whether processes are low on space and penalizing them. 2020-02-20 11:21:03 -08:00
Xin Dong abbe16ca9a Fix the build failure on Windows 2020-01-21 11:19:52 -08:00
Xin Dong b0a1af1288 Added the actual read hot detection algorithm and logging machanism.
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
Xin Dong 33456e7276 Done the plumbing work at the SSI side. 2020-01-21 11:15:52 -08:00
Xin Dong 7708034cc9 Added the function used to determin sub-ranges within a read hot shard that has a readSize-to-byteSize ratio higher than the knob value. Alos added unit tests for that function. 2020-01-21 11:15:52 -08:00
Xin Dong 8a9be14171 Resolve review comments 2019-11-27 19:26:22 -08:00
Xin Dong c95fa062b2 For the read sampling, use a specialized notify function to avoid unnecessary stack object allocation and a lot branch misses. 2019-11-22 15:21:09 -08:00
Xin Dong 14dd5626d7 Resolve review comments 2019-11-22 10:11:45 -08:00
Xin Dong 82c63dd101 Fix macOS build failure 2019-11-21 15:51:42 -08:00
Xin Dong c14af54229 Only log code coverage in simulations 2019-11-20 14:03:20 -08:00
Xin Dong 0c28e099cb Addressed Review comments 2019-10-09 16:42:42 -07:00
Xin Dong 12293d5497 Added metrics for read hot key detection 2019-10-09 16:42:42 -07:00
Meng Xu e76f7adeb7 PhysicalDiskMetrics:Remove the random sampling 2019-08-14 13:57:22 -07:00
Meng Xu ea9c8c09f5 Suppress PhysicalDiskMetrics events for 60s
Because PhysicalDiskMetrics metric is quite stable within short period of time,
we should suppress it with 60 second interval to avoid spammy log messages.

Spammy warning log can cause false positive in correctness test.
2019-08-14 13:51:20 -07:00
A.J. Beamon b91795d288 Send bytes input rate to DD. 2019-07-25 16:27:32 -07:00
A.J. Beamon 5f55f3f613 Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used. 2019-05-10 14:01:52 -07:00
mpilman 1c16f87a4e Remove trace-calls to printable (in non-workloads) 2019-04-05 13:12:19 -07:00
Robert Escriva 268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
Stephen Atherton 22f8a4efa9 Normalized all unit test names to begin with "/" if they should be included in random unit testing. 2018-10-05 22:09:58 -07:00
Alex Miller bca324eaa6 More actorcompiler.h fixes and additions. 2018-08-14 15:50:26 -07:00
Evan Tschannen 6f328d41ac suppressed spammy trace events 2018-08-06 12:12:55 -07:00
A.J. Beamon e5488419cc Attempt to normalize trace events:
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.

Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.

This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
Alec Grieser 0bae9880f1 remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py 2018-02-21 10:25:11 -08:00
Evan Tschannen 370e8a9903 fix: split metrics could fail an assert in a very rare scenario 2018-01-08 18:20:22 -08:00
A.J. Beamon 0fea5e9c2f Convert client_invalid_operation errors to ASSERTs. 2017-11-13 11:38:34 -08:00
Alex Miller c24b941485 Fix erroneous std::move in indexed set, and clean up addMetric users.
This is a follow-on to c4eb73d0.  Thanks to Bala for pointing out the unchanged
std::move usage, and there appeared to not be many existing users of addMetric
anyway.
2017-10-11 17:36:51 -07:00
Alex Miller 706427ee62 Fix potential division by zero issues via RPC.
A carefully crafted SplitMetricRequest could have caused division by zero.
It's not really great to offer Division By Zero As A Service, so let's just
return an error instead.
2017-10-03 22:11:08 -07:00
Yichi Chiang 37e5e2acbb Fix parentheses issue in StorageMetrics.actor.h 2017-07-27 12:03:36 -07:00
Yichi Chiang 53e1ae9f60 shard system keyspace 2017-07-26 13:47:31 -07:00
FDB Dev Team a674cb4ef4 Initial repository commit 2017-05-25 13:48:44 -07:00