57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
d0c06ec8c0
Make old build system work on build/cmake/Dockerfile
2019-02-28 14:24:52 -08:00
b8910ba7cd
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.h
# fdbserver/DataDistribution.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-22 14:38:13 -08:00
9b14aeb156
Tell cmake not to link/install on ide build
2019-02-19 15:16:59 -08:00
0bb60e5a3b
Use proper fwd decl in NativeAPI
...
Also NativeAPI.h -> NativeAPI.actor.h
2019-02-19 15:16:59 -08:00
3cb2391b58
use proper fwd declarations in ManagementAPI
...
Also ManagementAPI.h -> ManagementAPI.actor.h
2019-02-19 15:16:59 -08:00
065a45e05f
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-18 17:09:06 -08:00
8f2af8bed1
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller.
...
fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date.
fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-18 14:54:28 -08:00
e05b53d755
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-15 20:37:07 -08:00
c03de6c7b6
Update CLI to take addresses with mixed TLS states
2019-02-15 20:23:07 -08:00
75f692b931
simplify actorcompiler and target to compile coveragetool
2019-02-15 00:01:42 -08:00
b435d51061
Merge branch 'master' into track-server-request-latencies
2019-02-14 08:07:32 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
6da5971e79
Guard all versions.h to not break old WIN32 build
2019-02-08 16:06:00 -08:00
7e26b4ef0d
Address comments from PR
2019-02-07 15:37:04 -08:00
449b9bf79c
Fix typo that breaks linux build
2019-02-07 15:37:04 -08:00
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
750124f083
Accounting only non-excluded processes in the user output if recruiting is going on.
2019-02-06 17:04:38 -08:00
9c19d9e9eb
Improve status output to reflect that tlogs are recruited across availability zones according to the configured replication factor.
2019-02-04 18:14:00 -08:00
2198d24ce1
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/ServerDBInfo.h
# fdbserver/Status.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/storageserver.actor.cpp
2019-01-24 11:43:26 -08:00
8e05e95045
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-18 16:18:34 -08:00
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
7c5b2ab330
Merge pull request #976 from alexmiller-apple/jsonlogs
...
Allow trace logs to be output as JSON instead of XML
2019-01-09 17:04:50 -05:00
7eb6765698
Mention that xml is the default
2019-01-03 08:48:31 -08:00
bce5b03340
Fix whitespace
2019-01-02 15:24:11 -08:00
1bca665b29
Document --trace_format flag
2018-12-20 16:22:41 -08:00
03b48fb452
s/--trace-format/--trace_format/
2018-12-20 15:58:26 -08:00
b8df5acc15
Add --trace_format flag to fdbserver
2018-12-20 15:02:01 -08:00
df0f491c29
Some more improvements to the build and preparations for packaging
2018-12-13 15:04:13 -08:00
c06bb33730
CMake compiles all binaries except for bindings
2018-12-13 14:15:44 -08:00
d2d68aa171
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# versions.target
2018-12-03 18:26:52 -08:00
6e94d7d71a
changing the database configuration now checks that enough processes exist to support the new configuration
2018-11-30 18:52:24 -08:00
4e54690005
Merge branch 'release-6.0'
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/MoveKeys.actor.cpp
2018-11-12 20:26:58 -08:00
ccdc83036d
simplified region configuration protections
2018-11-12 17:45:20 -08:00
6353a6724b
strengthened the protections related to changing regions
2018-11-12 17:40:40 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
...
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
c02690471d
added protection against configuration changes which cannot be immediately reverted
...
the configure database workload tests region configurations
2018-11-04 19:53:55 -08:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
3eb4355a48
Some various cleanup and fixes. Added "Cluster" to TransactionMetrics trace event.
2018-09-25 15:06:19 -07:00
c831051474
This removes the idea of clusters from IClientApi.
2018-09-21 15:58:14 -07:00
90301f497f
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/TLSConnection.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# versions.target
2018-09-05 16:06:33 -07:00
4eaff42e4f
Merge pull request #712 from ajbeamon/remove-database-name-internal
...
Eliminate use of database names (phase 1)
2018-09-05 10:35:00 -07:00
40f5dbe423
fixed issues from review, added a safeguard to prevent configuring a cluster to an invalid configuration
2018-09-04 22:16:35 -07:00
e1fafbf259
Add some missing actorcompiler.h includes
2018-08-22 09:40:45 -07:00
d8ea3dbf9a
Added the ability to configure a cluster from a JSON file
2018-08-16 17:34:59 -07:00
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
bb2eb2fe53
Merge pull request #565 from etschannen/feature-remote-logs
...
Simulation did not permanently kill machines in most tests
2018-07-05 15:07:28 -07:00
6c98aa8aac
Fix not depending on FDBLibTLS.a and default TLS_LIBDIR to a sensible place.
2018-07-05 13:23:20 -07:00
507b3bacb0
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1.
...
added more recovery states.
2018-07-05 00:08:51 -07:00
01659e34cc
Move TLS libs into STATIC_LIBS to avoid having a make dependency on them.
...
And fix STATIC_LIBS to be cross platform.
2018-07-04 00:29:53 -07:00
cbdf598fa2
Add force_recovery_with_data_loss to hidden command list.
2018-07-03 15:04:11 -07:00
e67f951c06
Merge branch 'master' into feature-remote-logs
2018-07-02 02:18:20 -04:00
c3f88dbfe1
Merge branch 'master' of github.com:apple/foundationdb into tls-static
2018-07-01 23:13:57 -07:00
132e2d9267
Defined TLS build flags for projects
...
Updated TLS documentation
2018-07-01 22:49:39 -07:00
7a12d3e130
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 09:39:04 -04:00
45849d1f95
Added support for no-op legacy TLS options
2018-06-27 09:25:05 -07:00
dd967bd9e2
Removed developmental debug messages
2018-06-26 14:44:21 -07:00
ef8de426d3
Changed the TLS_DISABLED macro
...
Disable TLS within Windows until working
2018-06-26 12:08:32 -07:00
df1e3770b1
Fixed problem with adding xls libraries to linker
2018-06-25 14:11:51 -07:00
8a8914f046
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 00:04:00 -07:00
f8ce1de601
Added support for compiling TLS into binaries
2018-06-20 09:21:23 -07:00
39894ea798
Merge remote-tracking branch 'apple/release-5.2'
2018-06-12 18:31:20 -07:00
cfa7fe8866
Identify processes with host:port regardless of if TLS is enabled or not.
...
This makes `kill` and `profile` behave like how `exclude` functions, and means
commands don't have to change depending on TLS status.
Verified via starting a TLS cluster and killing a process before and after this change.
2018-06-11 16:49:20 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
026458baf3
Merge release-5.2 into master
2018-05-23 15:32:56 -07:00
086700aeb1
Plumb through TLS key password to CLI and from environment
2018-05-21 10:56:10 -07:00
8f984cb2c9
Merge branch 'release-5.2'
...
# Conflicts:
# fdbrpc/TLSConnection.h
2018-05-10 09:13:22 -07:00
e8b7f4b190
Add password support for tls.
2018-05-08 20:46:31 -07:00
d3b5cfb93c
Support latest TLS plugin.
...
Add support for https in backup.
2018-05-08 16:28:13 -07:00
8d1bd95f36
Fix the printStatus redundancy mode output
...
The key used to get the redundancy mode should match what's in JSON
2018-05-08 11:40:47 -07:00
42c8527f43
Merge remote-tracking branch 'upstream/release-5.2' into merge-release-5.2
2018-04-11 18:35:32 -07:00
d8175471bc
Merge release-5.1.5
2018-04-11 17:55:10 -07:00
44c0eaf4d7
fix: transactions created in fdbcli did not wait on the result of committing
2018-04-11 11:24:24 -07:00
19762b847d
Merge branch 'release-5.2'
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/SimulatedCluster.actor.cpp
2018-04-10 17:02:43 -07:00
c1ba16b3c8
Merge branch 'release-5.1' into release-5.2
...
# Conflicts:
# bindings/java/src/test/com/apple/foundationdb/test/AbstractTester.java
# bindings/java/src/test/com/apple/foundationdb/test/VersionstampSmokeTest.java
# bindings/nodejs/lib/fdb.js
# bindings/nodejs/src/Version.h
# bindings/nodejs/tests/tuple_test.js
2018-04-10 16:50:47 -07:00
82ed956c65
renamed the multi_dc configuration to three_datacenter. The old three_datacenter configuration was not a useful configuration.
2018-03-26 18:31:26 -07:00
551ea9c7f8
Merge remote-tracking branch 'upstream/release-5.2' into master-release-5.2-merge
2018-03-19 12:34:50 -07:00
0853fcb052
switch to using zu for some size_t variables in printf
2018-03-14 18:07:05 -07:00
f2c804e14f
Reverting changes from merge of master into release-5.2 ( b25810711c). Note that we never intend to release master into release-5.2, but if we did we would need to revert this commit.
2018-03-06 10:15:04 -08:00
37a6a81634
Merge commit '7f6fc3e039c911cd84b8540f7f799fc38a1c1822' into feature-remote-logs
...
# Conflicts:
# fdbserver/workloads/RestartRecovery.actor.cpp
2018-02-23 12:33:28 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
ceec580d2f
fdbcli considered a backup running if the 'running_backup' field was present, not if it was true.
2018-02-16 10:09:03 -08:00
5ac4f73978
Merge branch 'release-5.1' into feature-remote-logs
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/Locality.h
# fdbrpc/simulator.h
# fdbserver/ApplyMetadataMutation.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WorkerInterface.h
# fdbserver/masterserver.actor.cpp
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-05 11:33:42 -08:00
486d089e98
Better message is displayed to the user.
2017-11-09 13:55:19 -08:00
9809e84806
Added a counter to keep track of active outgoing incompatible connections.
...
This counter is used to print a warning in fdbcli if there are incompatible peers.
Example Output:
./fdbcli
Using cluster file `fdb.cluster'.
WARNING: Incompatible peers exist.
The database is unavailable; type `status' for more information.
Welcome to the fdbcli. For help, type `help'.
fdb> status
WARNING: Incompatible peers exist.
Using cluster file `fdb.cluster'.
Could not communicate with a quorum of coordination servers:
127.0.0.1:4000 (unreachable)
2017-11-09 11:20:35 -08:00
fefbf22b6f
Remove redundant tagCount
2017-11-06 14:00:08 -08:00
7d63c2cfb7
Formatting fixes
2017-11-06 09:20:31 -08:00
3ae3200cc5
Untested implementation of feature to log basic backup and DR info to fdbcli status.
2017-11-03 14:02:03 -07:00
988bc0207f
Reset Client Transaction profiling parameters when the config keys are cleared.
2017-10-31 15:40:57 -07:00
54d82c0d92
Merge pull request #194 from cie/alexmiller/valgrind
...
Fix valgrind errors
2017-10-27 17:25:12 -07:00
e0d33ef8d7
Preemptively fix profiler-related valgrind errors/straight out bugs.
...
I forgot to initialize some fields in requests.
2017-10-27 17:20:19 -07:00
edfb839ad9
added a message to exclude to suggest retrying the exclude after 30 seconds
2017-10-25 13:32:25 -07:00
48901a9223
added a list of tlog IDs that are missing to status
2017-10-24 16:28:50 -07:00
df74e2a373
re-added support for non-copying tlog recovery
2017-10-24 15:09:31 -07:00
cf646d4a99
Address review comments.
...
* Fixed fdbcli to be more idiomatic.
* Removed is_binary_serializable in favor of std::is_pod<>
* Removed custom enable_if<> in favor of std::enable_if<>
* Removed HEY REVIEWER comments
* Removed print from prof.py
* Added FLOW_PROFILER_ENABLED=yes to circus components that wished to enable the flow profiler.
2017-10-16 16:46:52 -07:00
91a26a170c
Add toggleable profiling support to fdbserver+fdbcli.
...
This adds the fdbcli commands:
* profile list -- Lists all workers in a way that doesn't fill `kill`'s list.
* profile flow run -- Allows starting flow profiling on a set of hosts for a specified interval.
And threads through all the support for enabling and disabling profiling as an RPC.
2017-10-16 16:05:02 -07:00
2e662b6bb6
Fixing review comments.
...
* parse_with_units found a proper home in flow.h while this was pending
* atof->strtod for error checking
2017-10-04 14:00:38 -07:00
e55cc447d2
Address code review comments.
...
* Fixed memory corruption with SystemData key constants
* Removed duplication in ClusterController
* Reworked fdbcli actions to better represent explicit vs default assignments
2017-10-04 13:36:18 -07:00
80fa597422
Allow client profiling to be configured from fdbcli.
...
This adds the following commands:
* profile client status
* profile client on 0.001 100MB
* profile client off
2017-10-04 13:36:18 -07:00
6ea9903c82
Merge branch 'release-5.0'
...
# Conflicts:
# fdbbackup/backup.actor.cpp
# fdbserver/ClusterController.actor.cpp
# versions.target
2017-10-01 18:46:44 -07:00
e2b65e86ed
added configurable memory limits for backup and dr executables
...
added a default memory limit of 8GB for fdbcli
2017-09-29 10:35:40 -07:00
6e26ae2bb3
added a new multi_dc configuration
2017-09-01 15:45:27 -07:00
0c7fd0a23d
Unify and clarify interactive and --exec command error and parse error handling.
...
Previously, interactive execution and --exec used two models for how a failed
command would impact other commands in the execution.
As an example, consider:
fdb> set foo bar ; set bar f\00 ; set baz foo
In interactive mode, this had the effect of {foo=bar, baz=foo}.
In --exec mode, this had the effect of {foo=bar}.
With this change, both now have the effect of {foo=bar}. This is achieved by
prefixing the last parsed command, which is the one that had the error, with a
fake "parse_error" token.
The execution of this would now look like:
ERROR: malformed escape sequence
>>> set foo bar
ERROR: Command failed to completely parse.
ERROR: Not running partial or malformed command: set bar
Which indicates how much execution occurred and where it halted, identically in
both modes of execution.
2017-08-21 13:42:01 -07:00
d78b29625c
Make fdbcli gracefully handle malformed and partial command errors.
...
Previously, running a command like `set \xffx\02abcded/` would cause a crash.
The `x\02` is a malformed typo of `\x02`, and the previously existing code to
handle this case looks like
loop {
err = parse_command
if (err) continue;
// do things
}
Thus, if we hit an error, we'd go back to the top of the loop, and try again.
This should be an infinite loop. However, the actor compiler implementation of
loops involves function calls, so this actually turns into a series of the loop
head calling the loop body calling the loop head calling ... and we eventually
crash due to running out of stack.
This is now fixed by simply letting the code continue on to the check later
that does
if (there was an error) {
print nasty message
return error
}
With output that looks like
ERROR: malformed escape sequence
WARNING: the previous command failed, the remaining commands will not be executed.
And therefore the world becomes a happy place.
2017-08-15 17:49:39 -07:00
f6d282e66a
change exclusions so that they calculate free space based on storage servers only
2017-08-11 17:20:28 -07:00
9ce8d3ae4f
Merge branch 'release-5.0'
2017-08-09 10:37:43 -07:00
c7402eb10c
fix: call initSignalSafeUnwind in fdbcli and backup to avoid hangs in slow task profiling.
2017-08-07 15:55:08 -07:00
6a8a5c41b0
Add a switch to turn off data distribution in CLI
2017-07-28 18:14:55 -07:00
15cb498aa7
removed fast_recovery_double and fast_recovery_triple from the fdbcli
2017-06-23 16:18:23 -07:00
b28ed397a2
Fixed printf field width specifier to reduce compilation warnings within OS X
2017-05-26 14:51:34 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
Vishesh Yadav