77f596743f
Bump persistFormat in TLogServer to differ from OldTLogServer*
...
Though this format is being deprecated in favor of an eventual plumbing
through of TLogVersion, we should probably bump it anyway.
And also remove the fallback to OldTLogServer code. It should never be
executed, as OldTLogServer_6_0 is entirely relied upon to execute
OldTLogServer_4_6.
2019-03-15 21:01:13 -07:00
4f98634f59
Add LogId to all TLog TraceEvents that have it.
2019-03-15 21:01:12 -07:00
12ddd56698
Fix Ratekeeper and DataDistributor placement
...
Make sure both RateKeeper and DataDistributor are placed in the same data
center as the Master. Make sure only one RateKeeper is live in the cluster as
well.
2019-03-15 17:09:28 -07:00
bb5686eb75
Fix monitoring of DD and RK
2019-03-15 16:02:17 -07:00
9f6fe5f649
Merge remote-tracking branch 'apple/master' into ratekeeper
2019-03-15 11:30:04 -07:00
40860e0093
Attempt to fix.
2019-03-15 11:29:04 -07:00
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
9e59c9c253
Check DataDistributor and RateKeeper fitness
...
Fail the test if they are not put in the best fitness.
2019-03-14 16:14:57 -07:00
99d521ef4f
Monitor Ratekeeper and DataDistributor to use stateless processes
...
Since Ratekeeper and DataDistributor are no longer running with Master, they
might be running with stateful processes before a new Master becomes alive,
which is undesirable.
This PR adds a monitoring of both Ratekeeper and DataDistributor at Cluster
Controller -- if Master runs on a stateless class and RK/DD runs at a worse
class, then RK/DD will be killed. I.e., RK/DD should be running at their own
classes or on the same stateless process as Master. After restart, RK/DD should
be running at a better process class.
2019-03-14 15:00:57 -07:00
2ac07fe7e0
Merge pull request #1248 from satherton/feature-backup-json
...
JSON output options for fdbbackup status and describe
2019-03-14 13:41:28 -07:00
5a10bf5dfc
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-14 10:35:12 -07:00
e30e2af1f3
ClientKnobs: Add CHECK_CONNECTED_COORDINATOR_NUM_DELAY
2019-03-13 16:54:56 -07:00
e7d1f9e5f1
fixed review comments
2019-03-13 15:59:03 -07:00
7f48025348
optimize confirm epoch alive
2019-03-13 14:47:17 -07:00
dbacfcbc82
Merge branch 'master' into feature-backup-json
2019-03-13 13:30:45 -07:00
a2108047aa
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-13 13:14:39 -07:00
e068c478b5
merge master
2019-03-12 18:31:25 -07:00
8aab719c22
Merge branch 'master' into feature-backup-json
2019-03-12 18:23:16 -07:00
a7e45cff91
Merge pull request #1176 from jzhou77/ratekeeper
...
Make Ratekeeper a separate role
2019-03-12 15:58:59 -07:00
85c24b0067
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-12 15:20:54 -07:00
5392742902
fixed review comments
2019-03-12 14:38:54 -07:00
c5a18945b6
Merge pull request #1260 from vishesh/task/tls-upgrade
...
Allows cluster string to contain coordinators with different TLS states
2019-03-12 13:45:08 -07:00
a25e224cda
Merge pull request #1213 from etschannen/feature-metadata-version
...
Added a metadata version key
2019-03-12 13:36:33 -07:00
2b0139670e
Fix review comment for PR 1176
2019-03-12 12:02:30 -07:00
f0eae0295f
Merge branch 'master' of https://github.com/apple/foundationdb into feature-backup-json
2019-03-12 03:35:03 -07:00
e9b8bf601e
Added backup status JSON output to backup workload to get sim coverage.
2019-03-12 03:34:38 -07:00
880e8643d1
Fix Windows link errors
2019-03-11 17:49:03 -07:00
46f4b02807
TLS Status: Resolve review comments
...
Use connectedCoordinatorsNumDelayed to reduce the load on cluster controller;
Set connectedCoordinatorsNum to null by default for monitorLeader()
2019-03-11 17:10:08 -07:00
5873705228
tlog commits very rarely take an additional 6 seconds
2019-03-11 12:11:17 -07:00
435e515985
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-11 11:17:40 -07:00
80c3f2f8e2
added status fields detailing which processes are degraded, and also the total number of degraded processes
2019-03-10 22:58:15 -07:00
c6e94293bf
reset a process to not be degraded after 2 days
2019-03-10 22:39:21 -07:00
2627bcd35e
Merge branch 'master' into feature-metadata-version
2019-03-10 21:13:28 -07:00
1be9ae5ce3
fixed merge conflict
2019-03-08 22:51:06 -05:00
044b6b4f8a
Merge branch 'master' into feature-degraded-tlog
...
# Conflicts:
# fdbserver/ClusterController.actor.cpp
2019-03-08 22:50:41 -05:00
ebffe8c633
print correct pahes in alloc instrumentation
2019-03-08 15:03:17 -08:00
45fe6b369b
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes
...
better master exists will switch the master to avoid degraded processes
2019-03-08 14:40:00 -05:00
53f16b5347
when a tlog queue commit takes longer than 5 seconds, its process is marked as degraded
2019-03-08 11:46:34 -05:00
710a64dc4e
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-08 11:25:07 -05:00
2537f26de6
First implementaion of more user-friendly cpack
...
Up unto here this code is only very rudiemantery tested.
This is a firest attempt of making cpack more user-friendly.
The basic idea is to generate a component for package type so
that we can have different paths depending on whether we build
an RPM, a DEB, a TGZ, or a MacOS installer. The cpack package
config file will then chose the correct components to use.
In a later point this should make it possible to build these
with `make packages` and the ugly iteration with calling cmake
between each package would be obsolete. While this solution is
a bit more bloated, it is also much more flexible and it will be
much easier to use.
Another benefit is, that this will get rid of all warnings during
a cpack run
2019-03-07 16:49:29 -08:00
cdfe906c30
Data distributor pulls batch limited info from proxy
...
Add a flag in HealthMetrics to indicate that batch priority is rate limited.
Data distributor pulls this flag from proxy to know roughly when rate limiting
happens.
DD uses this information to determine when to do the rebalance in the background,
i.e., moving data from heavily loaded servers to lighter ones. If the cluster is
currently rate limited for batch commits, then the rebalance will use longer
time intervals, otherwise use shorter intervals. See BgDDMountainChopper() and
BgDDValleyFiller() in DataDistributionQueue.actor.cpp.
2019-03-07 13:16:20 -08:00
f43277e819
Format Ratekeeper.actor.cpp code
2019-03-07 13:16:20 -08:00
dc129207a9
Minor fix after rebase.
2019-03-07 13:16:20 -08:00
835cc278c3
Fix rebase conflicts.
2019-03-07 13:16:20 -08:00
7340998261
Fix status message for ratekeeper
2019-03-07 13:16:20 -08:00
517966fce2
Remove lastLimited from rate keeper
...
Refactor code to make IDE happy.
2019-03-07 13:16:20 -08:00
d52ff738c0
Fix merge conflicts during rebase.
2019-03-07 13:16:20 -08:00
b2ee41ba33
Remove lastLimited from data distribution
...
Fix a serialization bug in ServerDBInfo, which causes test failures.
2019-03-07 13:16:20 -08:00
36a51a7b57
Fix a segfault bug due to uncopied ratekeeper interface
2019-03-07 13:16:20 -08:00
e6ac3f7fe8
Minor fix on ratekeeper work registration.
2019-03-07 13:16:20 -08:00
3c86643822
Separate Ratekeeper from data distribution.
...
Add a new role for ratekeeper.
Remove StorageServerChanges from data distribution.
Ratekeeper monitors storage servers, which borrows the idea from
DataDistribution.
2019-03-07 13:16:20 -08:00
27d199409e
Add KillRegion.actor.cpp workload to cmake
2019-03-07 12:14:42 -08:00
9e4c780baa
Merge pull request #1249 from xumengpanda/mengxu/status/teamcollection-info
...
Status:healthy: Add optimizing_team_collections
2019-03-07 11:44:24 -08:00
f3391ea413
Merge pull request #1240 from satherton/feature-restore-by-timestamp
...
Restore by timestamp
2019-03-06 16:21:06 -08:00
ed49d603a0
Allows cluster string to contain coordinators with different TLS states
...
During live TLS upgrades, we can hence switch one coordinator at a time
to TLS than all of them together.
2019-03-06 16:05:10 -08:00
845f8fdcbc
Status:healthy: Add optimizing_team_collections
...
Change removing_redundant_teams status name to
optimizing_team_collections.
The new name is more general and can be applied in the future
when we switch storage engines.
2019-03-06 15:05:23 -08:00
04880e3d4d
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-06 13:41:16 -08:00
7778112f6a
Bug fix, restore was using the destination cluster to look up timestamps when printing the backup description instead of (optionally) the original cluster which generated the backup. Made missing cluster file errors more clear.
2019-03-06 02:45:55 -08:00
c6a65389ae
Remove noexcept macro and replace with BOOST_NOEXCEPT.
...
BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a
way that is correctly maintained over time.
2019-03-05 22:06:12 -08:00
af617d68e6
boost 1.52.0 -> 1.67.0 in all vcxproj files
2019-03-05 22:06:12 -08:00
820548223a
Status: connected_coordinators misc minor changes
...
Change the rst document file;
Change the coding style to be consistent with the nearby code;
Ensure we always initilize the connectedCoordinatesNum to 0
even when the variable is not used.
2019-03-05 21:45:18 -08:00
b7a52e81e2
Status: Count connected coordinators per client
...
A client will always try to connect all coordinators.
This commit let Status track the number of connected coordinators
for each client.
This allows us to do canary in coordinators. For example,
when we switch from non-TLS to TLS, we can switch 1 coordinator
from non-TLS to TLS. This can help check if a client has the ability
to connect through TLS.
We can make the non-TLS to TLS switch for each coordinators
one by one. This avoid the risk of losing connection in the switch.
2019-03-05 21:21:23 -08:00
ad0aca21b5
Update fdbserver/fdbserver.vcxproj
...
Co-Authored-By: atn34 <anoyes34@gmail.com>
2019-03-05 18:03:57 -08:00
981426bac9
More ide fixes
2019-03-05 18:03:57 -08:00
82d957e0bb
Merge pull request #1178 from vishesh/task/issue-963-IPv6
...
IPv6 Support
2019-03-05 17:14:16 -08:00
a9562f61be
fix: missing argument to printf in fdbserver
2019-03-05 14:03:09 -08:00
21f55e1878
Merge pull request #1190 from bnamasivayam/restore-multiple-ranges
...
Add support for restoring multiple ranges.
2019-03-05 10:15:55 -08:00
f1897f3eb6
Merge branch 'master' into feature-metadata-version
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-03-04 21:06:16 -08:00
69d7633d5b
Merge pull request #1217 from alexmiller-apple/tstlog-goodref
...
Spill-By-Reference TLog Part 4: Actually Usable Reference Spilling
2019-03-04 20:58:24 -08:00
3d196c9e97
fix: metadataVersionKey log range removal needs to be checked for each logDestination
2019-03-04 20:56:31 -08:00
988add9fb5
cache the metadataVersion for commits, so that doing setVersion with the commit version of a different transaction will
2019-03-04 16:48:34 -08:00
c0535c49bb
Status: TLS client status
...
Use ClientStatusInfo structure for each network address (client),
instead of passing each status info as a parameter.
2019-03-04 16:35:10 -08:00
89cbb77b4e
Merge branch 'master' of https://github.com/apple/foundationdb into lazily-fetch-health-metrics
2019-03-04 14:17:58 -08:00
56ae46f89e
Client lazily fetches health metrics from proxies
2019-03-04 14:16:39 -08:00
e93cd0ff21
Add some checks and comments to IPv6 changes #963
2019-03-04 14:12:45 -08:00
592e224155
net: add/use formatIpPort to format IP:PORT pairs #963
2019-03-04 14:12:45 -08:00
cc9ad0e202
net: Use IPv6 in simulation testing #963
...
25% times we will use IPv6 addresses
2019-03-04 14:12:45 -08:00
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
baa3e1af2c
Replace `/sizeof(Page)*sizeof(Page)` with `pageFloor()`.
2019-03-04 01:42:39 -08:00
ee64b43366
Change DQ shrink logic to consider "active" bytes rather than file size.
...
We know what the current ideal size of the DQ file should be, so we
should use it.
2019-03-04 01:42:39 -08:00
244903a9de
Spill txsTag by value under TagMsg/ and not TagMsgRef/
...
There's not a tremendous reason as to why this matters now, but I feel
like I might regret sometime later not keeping the same schema under the
same key.
2019-03-04 01:42:39 -08:00
72c2cf11ab
Replace ResourceLimiter with FlowLock.
2019-03-04 01:42:38 -08:00
94bf75cb00
Allow the disk queue to shrink if it has unneeded slack space.
2019-03-04 01:42:38 -08:00
52d5a721a6
Don't allocate 2x the memory for a read to save 1% of allocated memory.
2019-03-04 01:42:38 -08:00
aff9ebe21a
Spill (start,length) instead of (begin,end) to save a few bytes.
2019-03-04 01:42:38 -08:00
2aa527c0ef
Fix a bug resulting from concurrent TLog changes.
...
TLogServer was forked into OldTLogServer_6_0 at the same time that
3247d59478976161
2019-03-04 01:42:38 -08:00
fb4cb8c3a8
Print out configuration changes in ConfigureTest.
2019-03-04 01:42:38 -08:00
9ef283d4e7
Implement hard limiting of memory used to serve peek requests.
2019-03-04 01:42:38 -08:00
e3506ad9af
Add a yield to parseMessagesForTag
2019-03-04 01:42:38 -08:00
742f6e1847
Solve overreading via pre-calculating tag bytes per commit
2019-03-04 01:42:38 -08:00
e7d8520c63
Batch more when spilling data.
2019-03-04 01:42:38 -08:00
71a794ccc3
Re-enable spill-by-reference testing.
2019-03-04 01:42:38 -08:00
04e1170c88
Spill txsTag by value
2019-03-04 01:42:38 -08:00
4d4e0a1d54
Fix the build on -O0.
...
C++ < 17 requires definitions of declared static constexpr variables.
2019-03-04 01:42:38 -08:00
db546af4a3
Fix the build on -O0.
...
C++ < 17 requires definitions of declared static constexpr variables.
2019-03-04 01:38:58 -08:00
075fdef31a
Merge branch 'master' into feature-metadata-version
...
# Conflicts:
# fdbclient/DatabaseContext.h
2019-03-03 22:58:45 -08:00
057ebe56e4
fix: unknownCommit handling relied on soleOwnership of the version stamp keys, so we need to use a second key to track the commit version for the metadataVersionKey
...
renamed a confusing option
2019-03-03 21:31:40 -08:00
39f612d132
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-03-02 17:07:00 -08:00
2940b8d5fd
Update all per-process storage server health metrics at once
...
Ratekeeper updates all storage servers' health metrics in updateRate
with only a single map lookup
2019-03-02 16:08:28 -08:00
c1de93f467
fix: binary search could get stuck in an infinite loop
...
fix: avoid picking a read version which could be before the last real commit
fix: we must wait on metadataVersionKey in case it is not already cached
fixed review comments
2019-03-02 13:55:41 -08:00
841eb10af0
Replace g_network->now() with now()
...
Co-Authored-By: tclinken <trevorclinkenbeard@gmail.com>
2019-03-02 08:15:56 -08:00
a37a65a7ff
Replace g_network->now() with now()
...
Co-Authored-By: tclinken <trevorclinkenbeard@gmail.com>
2019-03-02 08:15:05 -08:00
2a47fbb5a2
Fix tab spaces again
2019-03-01 15:30:40 -08:00
4324434b99
Fix tab spaces
2019-03-01 15:29:09 -08:00
74f64e4570
Fix FuzzApiCorrectness failure.
2019-03-01 15:25:37 -08:00
94385447bc
Status: Get if client configured TLS
...
To understand if all clients have configured TLS,
we check the tlsoption when a client tries to open database.
This is similar to how we track the versions of multi-version clients.
2019-03-01 15:17:01 -08:00
655c9d82c7
Various cleanup from review
2019-03-01 14:06:47 -08:00
2af99080ac
Merge branch 'master' into ratekeeper-batch-priority-limits
2019-03-01 13:02:07 -08:00
2168b14834
Merge branch 'master' into feature-metadata-version
2019-02-28 17:45:55 -08:00
3da85f3acd
implemented the \xff/metadataVersion key, which can be used by layers to help them cheaply cache metadata and know when their cache is invalid
2019-02-28 17:45:00 -08:00
d08aad7fc4
Old build system now working on both docker images
2019-02-28 14:24:52 -08:00
d0c06ec8c0
Make old build system work on build/cmake/Dockerfile
2019-02-28 14:24:52 -08:00
e2bcecb08f
Merge branch 'master' into ratekeeper-batch-priority-limits
2019-02-28 12:52:44 -08:00
759c53a333
Add some very basic exercising of priority batch to simulation
2019-02-28 12:18:45 -08:00
93f7849261
Fix typo
2019-02-28 12:02:47 -08:00
3e6a6a6569
Update status schema for correctness. Send the count of batch transactions started back to ratekeeper so that it can be logged with other ratekeeper metrics.
2019-02-28 12:00:58 -08:00
eb629d87a5
Add information about batch ratekeeper to status. Make it possible to track latencies in the ReadWrite workload for concurrently run instances separately.
2019-02-28 09:53:16 -08:00
af69ba035a
Merge pull request #1200 from bnamasivayam/master
...
User provided Client Transaction id merging
2019-02-28 11:16:44 -05:00
59df4ab39a
Disable testing spill-by-reference.
...
There was a bad interaction between Spill-by-reference work (96f5c811
2019-02-28 00:16:37 -08:00
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
3f59f82670
Calculate durabilityLag instead of NDV for health metrics
2019-02-27 16:30:01 -08:00
934e0457e9
Merge branch 'master' of github.com:apple/foundationdb
...
# Conflicts:
# fdbclient/DatabaseConfiguration.cpp
2019-02-27 15:21:06 -08:00
976309de97
fix: removed upgrade code from when transactions logs did not have known committed versions, because it could cause remote logs to choose a start version in a range of versions between and epoch end and the following recovery version
2019-02-27 15:15:20 -08:00
51e6b1dda0
Merge pull request #1197 from ajbeamon/avoid-transient-errors-in-incorrect-cluster-file-contents-detection
...
Improve avoidance of transient issues when logging IncorrectClusterFileContents...
2019-02-27 14:44:45 -08:00
fafb66b0a8
Merge pull request #1126 from bnamasivayam/ratelimit-consistencycheck
...
Dynamically rate limit consistency check.
2019-02-27 14:43:09 -08:00
96f5c8114b
Merge pull request #1195 from alexmiller-apple/tstlog-refspill
...
Spill-By-Reference TLog Part 3: Reference Spilling
2019-02-27 14:42:04 -08:00
bccc065fdd
Update fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
...
Co-Authored-By: bnamasivayam <36455962+bnamasivayam@users.noreply.github.com>
2019-02-27 14:24:16 -08:00
cc0aac588a
Merge branch 'client-transaction-id'
...
* client-transaction-id:
Add user provided transaction id as part of the format in ClientTransactionProfileCorrectness workload.
Addressed review comments.
Apply suggestions from code review
Add support for client provided transaction identifier to be logged as part of trace logging or transaction profiling.
2019-02-27 13:11:15 -08:00
00acef6d4a
Add user provided transaction id as part of the format in ClientTransactionProfileCorrectness workload.
2019-02-27 13:10:49 -08:00
d48a2d7b54
Fix problem caused by merge
2019-02-27 12:21:07 -08:00
a6205391ae
Fix line endings
2019-02-27 12:05:32 -08:00
a8cd446a58
Merge branch 'master' into ratekeeper-batch-priority-limits
2019-02-27 11:09:48 -08:00
a051055caf
Initial implementation of adding separate limits for batch priority in ratekeeper
2019-02-27 10:31:56 -08:00
999ee68609
Improve avoidance of transient issues when logging IncorrectClusterFileContents SevWarnAlways events by making it time based.
2019-02-27 10:08:24 -08:00
c46252eacd
fix: Don't disallow FDB listen addresses which are not in cluster file #1189
...
Rejects listen addresses if the IP:PORT matches with that found in
cluster file, but the TLS state is different.
2019-02-26 21:26:27 -08:00
ba31f8f1f9
Remove all code related to writing and cleaning up old-style spilling.
2019-02-26 18:13:49 -08:00
0539c1df00
Peek from new spilled data and not old spilled data.
2019-02-26 18:13:49 -08:00
d687a4bb85
Implement proper cleanup of disk queue when spilling refs.
2019-02-26 18:00:55 -08:00
84dc41c206
Add a comment.
2019-02-26 18:00:55 -08:00
cade914645
Temporarily add verification of spilled tag data.
2019-02-26 18:00:55 -08:00
f659825575
Persist start,end of message when spilling tags.
2019-02-26 18:00:55 -08:00
232f2434a5
Allow IDiskQueue::location to be serialized.
2019-02-26 18:00:55 -08:00
8d76cbed02
Track both start and end versions in versionLocation.
2019-02-26 18:00:55 -08:00
ee8964c8ec
Plumb through getNext{Commit,Push}Location
2019-02-26 18:00:55 -08:00
8afb7fbb9d
Merge pull request #1160 from alexmiller-apple/tstlog-fork
...
Spill-By-Reference TLog Part 2: New and Old TLogServers co-exist harmoniously
2019-02-26 18:00:04 -08:00
2af5025185
Don't bias log_spill in SimulatedCluster.
2019-02-26 17:58:59 -08:00
d4fe9f905c
Let log_spill/log_version default in config, and clean up serialization.
...
We don't need to abide by object serializer rules yet, and the minor
change to Simulation config lets us test config being the default at the
start of the test.
2019-02-26 17:14:41 -08:00
2dc57568cb
Change many things about log_version.
...
* log_version in the database (`/conf/log_version`) is now a hint that gets
rounded to the nearest supported version.
* fdbcli and FDB enforce that only a valid log_version can be configured to
* TLogVersion is persisted in CoreTLogSet (and LogSet and TLogSet)
* Some comments here and there
* Add an assert on filename length to make sure KV-pairs in filename
don't exceed a maximum length.
2019-02-26 16:47:04 -08:00
ab99497695
Addressed review comments.
2019-02-25 18:29:30 -08:00
7eba50b086
Add support for restoring multiple ranges.
2019-02-25 18:00:28 -08:00
Alex Miller