877997632d
Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
...
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
1f541f02be
Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
...
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
1343f40117
don't allow empty coments
2020-11-11 14:07:54 -07:00
bdd3dbfa7d
remove duplicates
2020-11-10 14:01:07 -07:00
e6aedfbf57
Report latency/bandwidth/size in each fetchKEys call
2020-11-09 15:23:58 -08:00
daead84fdd
Merge master
2020-08-14 14:07:08 -07:00
ced4f8c49f
fix: getReadHotRanges should be const function
2020-08-12 22:18:28 -07:00
002cca5e58
Merge branch 'master' into tmp/merge-PR
...
Conflicts:
fdbclient/NativeAPI.actor.cpp
fdbserver/StorageMetrics.actor.h
2020-08-12 20:08:13 -07:00
3ac6996844
Increased some knobs to throttle the spammy read hot logging. Also added more details inside the read hot log to make it useful
2020-08-12 14:27:34 -07:00
a46e620baa
Merge branch 'master' into feature/range-split-points-based-on-size
2020-08-06 15:23:11 -07:00
2e5ae8eed5
Change iterator name to small case
2020-07-10 10:56:41 -07:00
00e9f8b9bf
Added bindings tests; Protected new SSI endpoints under new 7.0 ProtocolVersion
2020-07-02 14:05:12 -07:00
eb28492900
Fixed a bug which leads to inaccurate metrics being reported. Added tests for the new API.
2020-07-02 14:05:11 -07:00
ea08bc5462
Added server side code for range split support
2020-07-02 14:05:11 -07:00
2a393633b6
Improved const-correctness
...
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185 ), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:
- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
2020-06-17 18:20:43 -07:00
696d95ebca
Fix the possible dead loop when endKey got reset to equal to the beginKey.
2020-05-12 08:12:15 -07:00
81184e79c5
Fixed a bug where the loop might get stuck.
2020-05-11 15:56:51 -07:00
f329164fb4
Merge pull request #2532 from dongxinEric/feature/hot-read-key-detection-part-2
...
Feature/hot read key detection part 2
2020-05-05 14:33:34 -07:00
3ba8daae9e
Partially resolve some of the review comments
2020-04-30 11:31:59 -07:00
76eb1c4376
Address review comments from Meng
2020-04-21 11:00:19 -07:00
ff8810ab45
Remove debug logs
2020-04-17 13:35:22 -07:00
7dd7406c59
Merge branch 'master' into feature/hot-read-key-detection-part-2
2020-04-16 14:54:05 -07:00
73272fc72e
Version difference is now the diff between TLog versions and SS version.
2020-04-03 19:04:43 -07:00
ad1dd4fd9b
Mark the storage servers that are continually lagging as unhealthy and so this will give the Data Distributor the chance to move data out of this server.
2020-03-31 18:25:39 -07:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
4c9c736253
Data distribution uses available space instead of free space when evaluating whether processes are low on space and penalizing them.
2020-02-20 11:21:03 -08:00
abbe16ca9a
Fix the build failure on Windows
2020-01-21 11:19:52 -08:00
b0a1af1288
Added the actual read hot detection algorithm and logging machanism.
...
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
33456e7276
Done the plumbing work at the SSI side.
2020-01-21 11:15:52 -08:00
7708034cc9
Added the function used to determin sub-ranges within a read hot shard that has a readSize-to-byteSize ratio higher than the knob value. Alos added unit tests for that function.
2020-01-21 11:15:52 -08:00
8a9be14171
Resolve review comments
2019-11-27 19:26:22 -08:00
c95fa062b2
For the read sampling, use a specialized notify function to avoid unnecessary stack object allocation and a lot branch misses.
2019-11-22 15:21:09 -08:00
14dd5626d7
Resolve review comments
2019-11-22 10:11:45 -08:00
82c63dd101
Fix macOS build failure
2019-11-21 15:51:42 -08:00
c14af54229
Only log code coverage in simulations
2019-11-20 14:03:20 -08:00
0c28e099cb
Addressed Review comments
2019-10-09 16:42:42 -07:00
12293d5497
Added metrics for read hot key detection
2019-10-09 16:42:42 -07:00
e76f7adeb7
PhysicalDiskMetrics:Remove the random sampling
2019-08-14 13:57:22 -07:00
ea9c8c09f5
Suppress PhysicalDiskMetrics events for 60s
...
Because PhysicalDiskMetrics metric is quite stable within short period of time,
we should suppress it with 60 second interval to avoid spammy log messages.
Spammy warning log can cause false positive in correctness test.
2019-08-14 13:51:20 -07:00
b91795d288
Send bytes input rate to DD.
2019-07-25 16:27:32 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
22f8a4efa9
Normalized all unit test names to begin with "/" if they should be included in random unit testing.
2018-10-05 22:09:58 -07:00
bca324eaa6
More actorcompiler.h fixes and additions.
2018-08-14 15:50:26 -07:00
6f328d41ac
suppressed spammy trace events
2018-08-06 12:12:55 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
370e8a9903
fix: split metrics could fail an assert in a very rare scenario
2018-01-08 18:20:22 -08:00
0fea5e9c2f
Convert client_invalid_operation errors to ASSERTs.
2017-11-13 11:38:34 -08:00
Andrew Noyes