2bb4f2e59f
Merge branch 'release-6.3-pre-format' into master-format
...
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
e9c9bd5299
Merge commit 'e2d3ddee2cb4da0bba65346d29ad2212fbce16f0' into file-backup-async-rename
...
# Conflicts:
# fdbclient/BackupContainer.actor.cpp
# fdbclient/FileBackupAgent.actor.cpp
# fdbclient/Knobs.cpp
# fdbrpc/AsyncFileNonDurable.actor.h
# fdbrpc/sim2.actor.cpp
2021-03-08 02:00:40 -08:00
318b862aa3
Apply clang-format to backup changes.
2021-03-08 01:49:29 -08:00
a397d8625f
AsyncFileNonDurable now has a flag for whether or not to emulate AIO mode, so the previous behavior limits are preserved when this flag is set. AsyncFileDurable writes which are selected to be durable are again done in a single write, and the logic to handle non-aligned writes for non-AIO mode is more clear and commented. In simulation, backup files on local filesystems are written with random buffer sizes. Backup container unit test now uses more random file sizes and limits memory consumption for all files being written in parallel.
2021-03-07 22:21:10 -08:00
1d00c41385
Bug fixes: Avoid empty writes, avoid 0 length truncation, and make AsyncFileNonDurable support writes at arbitrary offsets and lengths. Increase randomness of IBackupFile appends in backup container unit test.
2021-03-05 17:45:36 -08:00
8a8c488ede
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-05 18:13:38 -06:00
0c214d71da
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-05 16:31:35 -06:00
f33ed86210
Merge pull request #4420 from apple/release-6.2
...
Merge 6.2 into 6.3
2021-03-03 15:57:36 -08:00
08c98ec2a9
Unit test for rename function
2021-03-03 14:25:00 -07:00
fd1a865737
fix botched include
2021-03-03 09:36:04 -07:00
7f758328c0
add renameFile to WinASIO
2021-03-03 09:35:19 -07:00
7cb529ddb3
address review comment
2021-03-02 17:34:30 -07:00
a33bf68444
make sure rename code is actually called
2021-03-02 17:34:23 -07:00
d2bd63d0cb
fix includes
2021-03-02 17:05:27 -07:00
cc47332478
Added an actor to allow for async file renames
2021-03-02 16:38:51 -07:00
346a4e3ecd
Merge branch 'release-6.3'
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/MultiInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2021-03-01 18:52:06 -08:00
3cd819df25
Merge pull request #4301 from sfc-gh-anoyes/anoyes/iterator-fix-6.3
...
Fix bugs turned up by _GLIBCXX_DEBUG
2021-03-01 13:53:00 -08:00
bec1cd15b9
Use default stack size for libeio threads
2021-03-01 16:41:09 +00:00
4f2ed263cc
Update comment
2021-02-24 15:15:56 -08:00
78721530e6
Resolve merge conflict
2021-02-24 15:13:58 -08:00
a17b17f896
add more clarification
2021-02-23 12:23:10 -08:00
25a7f87433
Revise documentation in QueueModel to clarify the meaning of 'outstanding' and 'penalty'
2021-02-23 12:20:05 -08:00
8ee7ee9e45
Apply clang-format to fdbrpc/QueueModel.h
2021-02-23 10:21:35 -08:00
309c6fddc9
Add documentation to client side load balancing algorithm
2021-02-22 15:29:18 -08:00
ef0bf2728e
Merge branch 'release-6.3' into mengxu/ha-code
...
Resolve Conflicts:
fdbserver/LogRouter.actor.cpp: Only conflicts at comments
2021-02-19 21:47:09 -08:00
471a3489fb
Resolve review comments and add trace fields to MasterRecoveryState
2021-02-17 14:44:14 -08:00
d13e240496
use libcoroutine for Windows
...
The Windows CI still doesn't have boost context installed
so we use the old implementation for now
2021-02-17 09:23:41 -07:00
f514194aec
Merge remote-tracking branch 'origin/master' into features/coroutine2
2021-02-17 09:03:07 -07:00
9122be4d81
Add comments to HA code and loadBalance code
2021-02-10 13:51:36 -08:00
e6d9ebd3d2
Merge pull request #4302 from sfc-gh-ajbeamon/make-ping-stable-interface
...
Make the ping enpoint a stable interface
2021-02-08 18:10:33 -08:00
9f0f36c9a6
Make the ping enpoint a stable interface. Fix bug where we would could start scanning for packets before the connect packet was finished.
2021-02-08 17:10:33 -08:00
cb196daefe
Fix bugs turned up by _GLIBCXX_DEBUG
...
Compiling with -D_GLIBCXX_DEBUG enables libstc++ "debug mode", where
additional debug information is tracked with iterators and reported if
iterators are misused. This turned up two bugs.
I threw in removing dead code and avoiding an unnecessary map lookup
while I was in the neighborhood.
2021-02-09 00:41:21 +00:00
aaf0a9aa7b
Merge branch 'release-6.3' into merge-release-6.3-into-master
...
# Conflicts:
# build/docker-compose.yaml
# cmake/ConfigureCompiler.cmake
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/IAsyncFile.h
# fdbrpc/IRateControl.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbservice/ServiceBase.cpp
2021-02-08 12:58:34 -08:00
6e571ab87b
Merge branch 'master' of github.com:apple/foundationdb into features/coroutine2
2021-02-08 10:25:29 -07:00
67e783acf8
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/CompileBoost.cmake
# cmake/FDBComponents.cmake
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/storageserver.actor.cpp
# flow/Knobs.h
# flow/network.h
2021-02-08 09:20:28 -08:00
8547abbbe4
Bug fix: In simulation, KVStoreSQLite can outlive its process during a simulated reboot so its file references must be cleared explicitly during shutdown instead of implicitly during destruction. Re-enabled a useful trace event. Some cleanup, removed unnecessary member var initializer and removed unnecessary method.
2021-02-05 21:26:39 -08:00
08997d8602
Fix bugs turned up by _GLIBCXX_DEBUG
...
Compiling with -D_GLIBCXX_DEBUG enables libstc++ "debug mode", where
additional debug information is tracked with iterators and reported if
iterators are misused. This turned up two bugs.
I threw in removing dead code and avoiding an unnecessary map lookup
while I was in the neighborhood.
2021-02-04 02:37:51 +00:00
1db4b73215
Merge pull request #4243 from sfc-gh-etschannen/feature-storage-io-timeout
...
throw an io_timeout if it takes more than 2 minutes to commit on the storage server
2021-02-03 18:10:16 -08:00
593165679a
Remove unnecessary header
2021-02-02 12:42:46 -08:00
17671604c0
Add option to define I/O size
2021-02-02 12:39:05 -08:00
534fa45291
Merge branch 'master' of github.com:apple/foundationdb into features/coroutine2
2021-02-01 16:31:16 -07:00
5a0502cbf4
Improve SimExternalConnection::read performance
2021-01-31 16:08:03 -08:00
b0fd6bfbcf
More bug fixes in SQLite file handle lifetimes after storage server restart (including simulated process reboot) when a write rate limit is used. Added several debug trace events, renamed some VFSAsync* events to be more consistent.
2021-01-30 18:18:16 -08:00
52c2695f59
Bug fixes involving use of AFCached rate control in simulation.
2021-01-30 18:13:44 -08:00
e56fe02a25
Merge branch 'add-throttling-on-AsyncfileCached' of github.com:sfc-gh-clin/foundationdb into afcache-write-limit
2021-01-30 18:11:54 -08:00
80c302e5c9
removed assert
2021-01-28 17:52:06 -08:00
aa9d97bf22
validate that failDisk is being triggered in simulation
2021-01-28 12:34:25 -08:00
cf6a8e5df7
waitUntilDiskReady is not an actor
2021-01-27 14:36:02 -08:00
36e4f82115
more complete support for simulated disk failures
2021-01-27 14:29:43 -08:00
f30ae07a2d
fixed compile error
2021-01-27 14:10:36 -08:00
b9a4c7418a
fixed typo
2021-01-27 14:10:06 -08:00
3ee5831287
fault injection can now cause a disk to stop responding
2021-01-27 13:57:54 -08:00
989fee908a
More bug fixes in SQLite file handle lifetimes after storage server restart (including simulated process reboot) when a write rate limit is used. Added several debug trace events, renamed some VFSAsync* events to be more consistent.
2021-01-27 00:01:33 -08:00
d3a85b555a
CoroFlow and remove libcoro
2021-01-26 17:23:17 -07:00
d718e7baf3
Merge pull request #4179 from sfc-gh-tclinkenbeard/run-minio-joshua
...
Support Testing S3BlobStore backup in simulation
2021-01-26 14:39:57 -08:00
27314b111c
Fix Windows build issues
2021-01-26 13:41:37 -08:00
3560003c50
Fix missing override in fdbrpc/FlowTests.actor.cpp
2021-01-26 17:29:54 +00:00
bb1dd21c22
Define BOOST_DATE_TIME_NO_LIB in sim2.actor.cpp
2021-01-25 18:06:55 -08:00
4ee97c0784
Use clang-tidy to automatically fix missing overrides
...
Use `clang-tidy -p . $file -checks='-*,modernize-use-override' -header-filter='.*' -fix`
to fix missing overrides, and then use git clang-format to reformat just
those changes. This went pretty well for most files.
Formatting the following files went off the rails, so I'm going to
follow up with a commit that's just clang-tidy and no clang-format.
- fdbclient/DatabaseBackupAgent.actor.cpp
- fdbclient/FileBackupAgent.actor.cpp
- fdbserver/OldTLogServer_4_6.actor.cpp
- fdbmonitor/SimpleIni.h
- fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
2021-01-26 02:04:12 +00:00
88cc2f752a
Define BOOST_DATE_TIME_NO_LIB in SimExternalConnection.actor.cpp
2021-01-25 16:23:36 -08:00
c3775ede37
add override keyword properly to squash compiler warnings
2021-01-25 13:24:15 -05:00
0d32bd0c44
Bug fixes involving use of AFCached rate control in simulation.
2021-01-24 01:50:09 -08:00
51cbc494f1
Merge remote-tracking branch 'origin/master' into run-minio-joshua
2021-01-23 11:31:19 -08:00
82a8f1c763
Fix -Wreturn-type warning
2021-01-22 12:13:29 -08:00
fb9b4b7626
Merge remote-tracking branch 'origin/master' into misc-changes
2021-01-21 10:45:50 -08:00
3f7e19a182
Add tags to spans to provide additional metadata
2021-01-20 13:57:36 -08:00
98e6ba6962
Add POSIX send functionality
2021-01-20 13:57:36 -08:00
d2143f184a
Add fast UDP tracer
2021-01-20 13:57:36 -08:00
fb64902d5c
Assign roles
2021-01-19 16:04:09 -07:00
523a8c08db
Merge branch 'add-throttling-on-AsyncfileCached' of github.com:sfc-gh-clin/foundationdb into afcache-write-limit
...
# Conflicts:
# flow/Knobs.h
2021-01-17 05:30:55 -08:00
191f215b9d
Remove chrono dependency
2021-01-13 21:54:22 -08:00
8e93573aad
Merge branch 'master' of github.com:apple/foundationdb into features/actor-lineage
2021-01-05 10:24:45 -07:00
61a29ecfc8
Merge remote-tracking branch 'origin/master' into run-minio-joshua
2021-01-01 09:38:36 -04:00
86c7c1e946
Fix IAsyncFileSystem method signatures
2020-12-28 01:57:42 -04:00
5bfa6cea98
Merge remote-tracking branch 'origin/master' into misc-changes
2020-12-26 20:47:00 -04:00
913401aaee
Move SendBufferIterator out of network.h
...
This class is only needed for implementation in Net2.actor.cpp and
SimExternalConnection.actor.cpp, so this class should not be included
everywhere network.h is included
2020-12-26 16:30:00 -04:00
0b45c969b9
Remove invalid assertion from SimExternalConnection::read
2020-12-24 21:20:04 -04:00
a7202a5b42
Use connectExernal in INetworkConnections::connect
2020-12-24 21:20:04 -04:00
1dfa2acb93
Reduce sleep in SimExternalConnection::write
2020-12-24 21:20:04 -04:00
d9b5309f4f
Add random delays to SimExternalConnection
2020-12-24 21:20:04 -04:00
917da45c3f
Miscellaneous improvements to SimExternalConnection unit test
2020-12-24 21:20:04 -04:00
555c3d95fc
Added fdbrpc/SimExternalClient unit test
2020-12-24 21:20:03 -04:00
201857541d
Remove unnecessary copies from INetworkConnections interface
2020-12-24 21:20:03 -04:00
94d7c17978
Implemented SimExternalConnection::onReadable
2020-12-24 21:20:03 -04:00
8cb029b777
Made SimExternalConnection reading and writing deterministic
2020-12-24 21:20:03 -04:00
16476ec9e1
Implemented SimExternalConnection::read
2020-12-24 21:20:03 -04:00
1e7ea8ece1
Implemented SimExternalConnection::write
2020-12-24 21:20:02 -04:00
87282eda47
Implemented SimExternalConnection::getPeerAddress
2020-12-24 21:20:02 -04:00
9f22ae7831
Implement SimExternalConnection::connect
2020-12-24 21:20:02 -04:00
fed1a03950
Added SimExternalConnection (most methods not implemented yet)
2020-12-24 21:20:02 -04:00
315557f58f
Not all flush will write to disk
2020-12-11 10:34:36 -08:00
9601769b01
Merge pull request #3858 from sfc-gh-rchen/stable_interfaces
...
Stable interfaces
2020-12-11 09:34:27 -08:00
5f57d72a59
remove print statements and format protocol version workload
2020-12-11 04:46:20 +00:00
f8e1df6c4f
Support for actor stack traces
2020-12-10 10:42:04 -07:00
265a88e2ac
Fix signature
2020-12-09 22:30:18 -08:00
390a82b6a2
Fix old typo
...
Co-authored-by: Andrew Noyes <63815641+sfc-gh-anoyes@users.noreply.github.com>
2020-12-09 13:51:50 -07:00
5b55216252
Add rate control to AsyncFileCached
2020-12-09 12:03:32 -08:00
9344a5fd0a
Fix Coro ucontext bug introduced in #4109
2020-12-09 12:25:31 -07:00
0d324cee80
Annotation framework and role lineage
2020-12-09 10:19:59 -07:00
f8ed207ee4
Change timer() to now() to preserve determinism
2020-12-08 16:44:39 -08:00
87a26ac55c
Remove unnecessay m_budget_max, and update the constructor
2020-12-08 16:26:09 -08:00
7ac5aa48ed
Fix UDP simulation bug with duplicate ports
...
Occasionally, a second UDP socket would be created with a randomly generated
port equal to the randomly generated port of an earlier socket.
2020-12-08 17:38:32 +00:00
c8786a65c3
Remove unnecessary uses of virtual
2020-12-08 09:09:33 -08:00
cb97ab8356
Use unique pointers in LatencyBands
2020-12-08 09:09:32 -08:00
d15441e85c
Replace non-standard sealed with final
2020-12-08 09:09:30 -08:00
f3c0d26806
Make ISimulator::BackupAgentType an enum class
2020-12-08 09:09:30 -08:00
c914620c10
Fix signatures for ISimulator methods
2020-12-08 09:09:29 -08:00
9d395a926d
Make ISimulator::ProcessInfo::getListener const
2020-12-08 09:09:29 -08:00
c77d9e4abe
merge conflicts
2020-12-02 21:53:19 +00:00
1f541f02be
Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
...
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
d64cf8b9e3
Merge branch 6.3 into master
2020-11-17 11:22:45 -08:00
d0391db862
Merge branch 'release-6.2' into release-6.3
2020-11-16 10:15:23 -08:00
953bb98f02
attempt to fix windows build
2020-11-13 15:39:51 -07:00
d3598bfaa4
address review comments
2020-11-13 15:37:03 -07:00
222da17558
Merge branch 'release-6.2' into mengxu/ha-code-read
2020-11-12 13:39:27 -08:00
0f205620bf
Merge branch 'master' into features/udp
2020-11-12 13:51:17 -07:00
d853892b41
Bugfix
...
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
2020-11-12 13:48:23 -07:00
c2dd7d1d38
Remove unresolved questions
2020-11-11 22:39:11 -08:00
32a089dde1
Merge pull request #4030 from vishesh/task/issue-connection-logging-count
...
FlowTransport: Sample connect latencies
2020-11-10 20:00:58 -08:00
bdd3dbfa7d
remove duplicates
2020-11-10 14:01:07 -07:00
1976f2c371
FlowTransport: Sample connect latencies
2020-11-10 12:12:01 -08:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
4086e3a275
Merge pull request #3999 from dongxinEric/misc/attach-primary-peek-location-to-logRouterMetrics
...
Add a way to decorate the trace event created by `traceCounters` with more details.
2020-11-05 09:54:39 -08:00
1c05b70942
Add INetwork::timer_monotonic
2020-11-05 17:07:49 +00:00
c69c1064cf
Merge pull request #4006 from vishesh/task/issue-connection-logging-count
...
FlowTransport: Count number of incoming/outgoing/failed connections in logs
2020-11-03 20:07:37 -08:00
7bdcb01bdf
FlowTransport: Count number of incoming/outgoing/failed connections with logging
2020-11-02 18:51:48 -08:00
32c87bbb33
Lightweight, power of two spaced histogram implementation + automatic reporting
2020-11-02 11:13:16 -08:00
1664e2ff7f
Add more comments and questions to LR tLog and loadbalance
2020-11-01 21:22:23 -08:00
9c6c2704d0
Merge branch 'master' of https://github.com/apple/foundationdb into stable_interfaces
2020-10-30 23:59:32 +00:00
af7e65110f
Allow the caller to decorate role metrics trace event with more details.
2020-10-30 16:20:08 -07:00
09350c760e
Merge remote-tracking branch 'origin/master' into merge
2020-10-30 08:36:19 -07:00
0faaecc780
Merge branch 'master' into merge
2020-10-29 22:27:40 -07:00
cf4c8e375f
Merge remote-tracking branch 'origin/release-6.3' into merge
2020-10-29 22:15:41 -07:00
55e86c1336
Disambiguate between S3BlobStore and other blob stores
2020-10-29 20:42:23 -07:00
96cc700440
Merge pull request #3961 from sfc-gh-tclinkenbeard/azure-backup
...
Add support for writing backup files directly to Azure blob storage
2020-10-29 00:30:40 -07:00
e82339f40e
Merge pull request #3906 from sfc-gh-anoyes/anoyes/use-gcov
...
Add USE_GCOV cmake option
2020-10-27 08:27:46 -07:00
5b546c4854
change protocol version to hex encoded string in status json. Move constant from flow transport header to cpp
2020-10-26 19:35:38 +00:00
82b6daa16b
First draft of Azure blob storage backup container
2020-10-24 16:47:51 -07:00
ce3f09699f
revert checksum
2020-10-23 21:23:28 +00:00
309e38dc06
add unit test and some cleanup
2020-10-23 19:34:18 +00:00
055add9682
conflicts
2020-10-23 06:33:00 +00:00
90caa24af1
add function to get correct checksum class. Remove IChecksum::checkSum function
2020-10-23 02:24:29 +00:00
ea5225c600
Merge pull request #3927 from ajbeamon/fix-roughness-elapsed
...
Fix roughness elapsed time
2020-10-21 09:52:28 -07:00
8cc3e4d3c6
Merge release-6.3 into master
2020-10-19 22:51:56 -07:00
c84a5f4014
change IChecksum interface and remove coordinator protocol version optional
2020-10-19 21:39:07 +00:00
00912f5855
Don't use the end of a window in calculating the elapsed time for roughness
2020-10-19 13:40:16 -07:00
f2a8687c7c
fix merge conflicts
2020-10-19 17:00:41 +00:00
a4ccf8c5b6
add checksum class to change checksum in the future. expose get server protocol api
2020-10-19 16:57:11 +00:00
545ee4269d
master conflicts
2020-10-19 01:03:54 +00:00
1c533e7363
fix flowtransport conflicts
2020-10-19 01:00:03 +00:00
Vishesh Yadav