5a56f6b456
minor snap create client improvement and bug fixes
2019-07-29 20:28:22 -07:00
1706aaf199
Merge branch 'master' into mengxu/performant-restore-PR
...
Fix conflict in TlogServer.actor.cpp by accepting master changes
2019-07-26 11:46:27 -07:00
9afd162e2f
remove snap v1 related code
2019-07-25 17:29:31 -07:00
45083edf74
Merge branch 'master' into mengxu/performant-restore-PR
...
Fix conflicts as well.
2019-07-25 10:46:11 -07:00
a2f2ad96ff
code review comments and merge to master changes
2019-07-24 15:36:28 -07:00
209448807d
snap v2: fdbclient related changes
2019-07-24 15:36:28 -07:00
83f4b8ebb1
Merge pull request #1866 from senthil-ram/setDDBugFix
...
setDDMode should set moveKeysLockWriteKey
2019-07-24 15:35:18 -07:00
0962641540
setDDMode should set moveKeysLockWriteKey
...
After takeMoveKeysLock notes down the owner and the
moveKeysLockWriteKey value, it monitors the above two in
pollMoveKeysLock and checks if anything is changed, but
setDDMode was not setting the moveKeysLockWriteKey and
so a sequence like disable, enable and disable would not
really disable DD.
2019-07-19 11:13:29 -07:00
b242760adf
Use functional cast instead of explicit operator call
2019-07-19 09:58:40 -07:00
d9a8657096
fdbcli: Add `no_wait` option in `exclude` command to avoid blocking
...
RESOLVES #1840
2019-07-18 13:07:31 -07:00
7a500cd37f
A giant translation of TaskFooPriority -> TaskPriority::Foo
...
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
844dd60202
FDB compiling with intel compiler
2019-06-20 09:29:01 -07:00
477fd152c0
FastRestore:Refactor code
...
1) Use the runRYWTransaction for simple DB access
2) Replace some printf with TraceEvent
3) Remove printf not used in debugging
4) Avoid wait inside the condition in loop-choose-when for
the core routine of restore worker, loader and applier.
5) Rename Restore.actor.cpp to RestoreWorker.actor.cpp since
the file only has functionalities related to restore worker.
Passed correctness test
2019-06-04 11:22:47 -07:00
4bcb590f12
g_random -> deterministicRandom()
2019-05-28 22:07:46 -07:00
ec7834e2f7
code re-orgnaization and address comments
2019-05-28 22:07:46 -07:00
61e93a9304
Address review comments and minor fixes
2019-05-28 22:07:46 -07:00
69edefe68b
Snapshot based backup and resotre implementation
2019-05-28 22:07:46 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
f4fbaac6b0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-05-19 10:27:59 -07:00
4166ecdfe7
Increase the configure timeout to 60 seconds, to avoid spurious configuration failures
2019-05-16 18:03:16 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
b5cbffc1b8
fdbclient: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
2d5043c665
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-30 18:27:04 -07:00
6124a7e145
Update fdbclient/ManagementAPI.actor.cpp
2019-04-30 16:30:19 -07:00
b2d19eebc4
fdbcli would return success even when configure failed for a variety of error types
...
the existing configure safety check would fail when attempting to change between three_datacenter and region configuration
2019-04-30 16:30:19 -07:00
529ce66b6c
Merge branch 'apple/master' into mengxu/performant-restore-PR
2019-04-18 18:02:45 -07:00
cd5c9d91fa
Merge pull request #1443 from etschannen/master
...
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
6220a5ce0f
Merge pull request #1370 from jzhou77/fix-unreferenced
...
Remove unused functions
2019-04-09 11:49:45 -07:00
3356ac27bf
added three_data_hall_fallback configuration
2019-04-07 22:58:18 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
c4a8a80d6f
Merge branch 'apple/master' into mengxu/performant-restore-PR
2019-04-04 22:51:00 -07:00
a38c396283
made all maintenance transactions lock aware
2019-04-02 14:27:48 -07:00
781cf9b5a0
added the ability to make a zoneId for maintenance in fdbcli
2019-04-01 17:55:13 -07:00
47b4b82628
Merge branch 'master' into fix-unreferenced
2019-04-01 14:07:19 -07:00
f7f8ddd894
Fix warnings on unused variables
...
Found by -Wunused-variable flag.
2019-04-01 14:00:20 -07:00
70d7c289f4
Merge branch 'master' into mengxu/restore/parallel-v7
2019-03-30 22:13:10 -07:00
b6008558d3
renamed BinaryWriter.toStringRef() to .toValue(), because the function now returns a Standalone<StringRef>()
...
eliminated an unnecessary copy from the proxy commit path
eliminated an unnecessary copy from buffered peek cursor
2019-03-28 11:52:50 -07:00
f9aad46573
made use_provisional_proxies a transaction option
2019-03-19 18:44:37 -07:00
5b9c45ea0b
clients do not attempt to connect to provisional proxies
2019-03-19 13:37:50 -07:00
eb54a700ba
changed the old memory configuration to memory-1
2019-03-18 15:10:04 -07:00
a372c7cf18
configure memory now selects the ssd engine for transaction log spilling. Transaction log spilling is only used when the transaction logs cannot keep all of the unpopped mutations it has in memory. If we are only using this data structure because we do not have enough memory, it is much less safe to use the memory storage engine for this purpose.
2019-03-16 22:48:24 -07:00
a2108047aa
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-13 13:14:39 -07:00
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
b8910ba7cd
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.h
# fdbserver/DataDistribution.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-22 14:38:13 -08:00
1bb384db4d
Merge branch 'master' of https://github.com/apple/foundationdb into add-no-assign-class
2019-02-20 13:13:12 -08:00
3a572b010f
fix: a forced recovery needed to force the data distributor to restart
2019-02-19 16:04:52 -08:00
0bb60e5a3b
Use proper fwd decl in NativeAPI
...
Also NativeAPI.h -> NativeAPI.actor.h
2019-02-19 15:16:59 -08:00
3cb2391b58
use proper fwd declarations in ManagementAPI
...
Also ManagementAPI.h -> ManagementAPI.actor.h
2019-02-19 15:16:59 -08:00
ed9e20ce17
forgot to fix merge conflicts
2019-02-18 17:09:55 -08:00
065a45e05f
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-18 17:09:06 -08:00
8f2af8bed1
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller.
...
fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date.
fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-18 14:54:28 -08:00
83060c6e56
Merge pull request #1062 from jzhou77/PR
...
Add a new DataDistributor role.
2019-02-15 13:51:27 -08:00
6a655143e8
A follow-on fix for config key usage
...
And some trace event cleanups.
2019-02-14 16:37:16 -08:00
be5c962bb7
Add a new configuration version key \xff/conf/version
...
This fixed a bug found by upgrade test, where the configuration monitor of the
data distributor was monitoring excludedServersVersionKey, which doesn't
change in ChangeConfig workload. As a result, data distributor was not aware of
configuration changes.
Adding this new key and make sure this key is updated in configuration changes
so that the monitor can detect configuration changes.
2019-02-14 16:37:16 -08:00
21ce26a681
Merge release-6.0 into master
2019-02-14 14:24:35 -08:00
b435d51061
Merge branch 'master' into track-server-request-latencies
2019-02-14 08:07:32 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
c4fa69c998
Remove duplicate comment
2019-02-06 09:38:08 -08:00
b11d2ce1ec
Fix bug that in `include` command
...
This bug results in `include` to include all
machines that have a prefix of a given machine.
This is a weird bug that has to do with ASCII
ordering. Instead this command executes two
clears: one for the IP address and one for
all IP:PORT pairs
2019-02-06 09:38:08 -08:00
550f2e2682
Merge with master to use the latest backup container
...
In fdb 6.0.15, backup container is changed on how to organize the backup data.
The backup made by fdb >6.0.15 has to be restored with fdb > 6.0.15.
Merge with master so that the fast restore uses fdb > 6.0.15
2019-01-30 12:05:15 -08:00
6d7d40bcb3
Update fdbclient/ManagementAPI.actor.cpp
2019-01-29 11:37:08 -08:00
a49aea29d1
Update fdbclient/ManagementAPI.actor.cpp
...
Co-Authored-By: etschannen <36455792+etschannen@users.noreply.github.com>
2019-01-29 11:34:50 -08:00
da1a8a9919
Prioritize processes with class coordinator when "coordinators auto" is run
2019-01-25 11:33:10 -08:00
2198d24ce1
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/ServerDBInfo.h
# fdbserver/Status.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/storageserver.actor.cpp
2019-01-24 11:43:26 -08:00
8e05e95045
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-18 16:18:34 -08:00
699f8dd617
fix: coordinators auto could put two coordinators in the same zone
...
simulation now tests two machines in the same zone
2019-01-18 15:42:48 -08:00
80b2f75187
debug why restore did not restore the complete data
2018-12-03 19:29:17 -08:00
d2d68aa171
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# versions.target
2018-12-03 18:26:52 -08:00
1b085a9817
sequantial restore: pass 1 test case
...
-r simulation --logsize 1024MiB -f foundationdb/tests/fast/ParallelRestoreCorrectness.txt -b off -s 95208406
2018-12-03 10:57:30 -08:00
6e94d7d71a
changing the database configuration now checks that enough processes exist to support the new configuration
2018-11-30 18:52:24 -08:00
4e54690005
Merge branch 'release-6.0'
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/MoveKeys.actor.cpp
2018-11-12 20:26:58 -08:00
ccdc83036d
simplified region configuration protections
2018-11-12 17:45:20 -08:00
6353a6724b
strengthened the protections related to changing regions
2018-11-12 17:40:40 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
...
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
c02690471d
added protection against configuration changes which cannot be immediately reverted
...
the configure database workload tests region configurations
2018-11-04 19:53:55 -08:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
22f8a4efa9
Normalized all unit test names to begin with "/" if they should be included in random unit testing.
2018-10-05 22:09:58 -07:00
3ea9193fa7
Renamed redwood to redwood-experimental. UnitTest names can now be hidden using # as the first character so that random correctness tests will not run them. Excluded redwood tests from correctness testing. Reverted default storage engine to ssd.
2018-10-05 14:43:54 -07:00
7c1dc305cb
Merge commit 'a72c8f5cb2e79a673abc0ed3d27ef1c51028fb13' into feature-redwood
2018-10-05 10:15:10 -07:00
3922e477a5
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/LogSystemDiskQueueAdapter.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00
dd9c6856e1
fix: forced recovery is not safe to send multiple times
2018-10-02 17:29:39 -07:00
c831051474
This removes the idea of clusters from IClientApi.
2018-09-21 15:58:14 -07:00
2fc86c5ff3
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbrpc/AsyncFileCached.actor.h
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-09-20 03:39:55 -07:00
200e65fe61
added a workload which tests killing an entire region, and recovering from the failure with data loss.
...
fix: we cannot pop the txs tag from remote logs until they have a full copy of the txnStateStore
fix: we have to modify all of history, we cannot stop after finding a local remote
2018-09-17 18:32:39 -07:00
10a8e67266
Merge remote-tracking branch 'upstream/release-6.0' into merge-release-6.0
2018-09-11 21:49:59 -07:00
d3c8d7ab4e
fix: status would generate invalid json
2018-09-07 18:26:05 -07:00
90301f497f
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/TLSConnection.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# versions.target
2018-09-05 16:06:33 -07:00
40f5dbe423
fixed issues from review, added a safeguard to prevent configuring a cluster to an invalid configuration
2018-09-04 22:16:35 -07:00
d8ea3dbf9a
Added the ability to configure a cluster from a JSON file
2018-08-16 17:34:59 -07:00
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
9d85a05372
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-07-05 12:52:06 -07:00
2925b9b984
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-07-03 23:03:56 -07:00
866ccfe344
added the ability to allow the master to finish recovery before all storage servers in both regions have their mutations. This allows you to recover from scenarios where you lose all your tlogs in one dc.
2018-07-04 01:59:04 -04:00
7a12d3e130
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 09:39:04 -04:00
b95a2bd6c1
Merge commit 'b17c8359ec22892ed4daeaa569f2f5e105477251' into feature-redwood
...
# Conflicts:
# flow/Trace.cpp
2018-06-30 23:18:29 -07:00
a66eda8baa
added the three_datacenter_fallback redundancy mode, which allows you to drop a down datacenter when configured in three_datacenter mode
2018-06-27 23:24:33 -07:00
8a8914f046
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 00:04:00 -07:00
e5c48d453a
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-18 22:45:27 -07:00
sramamoorthy