55ef40864f
Add comments for touched functions
2021-03-29 10:09:07 -07:00
1a0ef39e5d
Fix keyRangesFilter with logsOnly flag
...
Remove the Sev 40 trace event.
2021-03-27 21:48:27 -07:00
a9d8806828
Use the restored range in the actual restore
2021-03-25 10:07:25 -07:00
720c0b3a4f
Added const.
...
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2021-03-24 14:36:55 -07:00
a6136ffc62
Applied clang-format to recent changes.
2021-03-24 06:51:08 -07:00
17a19021fb
Added mutation batch filtering during restore, to the extent possible where each restore mutation log task only sees one block of data (usually 1MB) which can contain incomplete and therefore unfilterable mutation sets.
2021-03-24 06:38:50 -07:00
3751ecd11f
Merge pull request #4503 from sfc-gh-ajbeamon/merge-release-6.3-into-master
...
Final merge of release-6.3 into master
2021-03-16 09:29:05 -07:00
25c4880ebe
Merge branch 'release-6.3' into merge-release-6.3-into-master (temporarily discard all changes to BackupContainer.actor.cpp)
...
# Conflicts:
# fdbclient/BackupContainer.actor.cpp
# fdbserver/Knobs.h
2021-03-15 16:41:04 -07:00
766b9d24d6
Merge branch 'master' into remove-invalid-backup-status-fields
2021-03-14 18:57:03 -07:00
cfbaaa6001
Merge remote-tracking branch 'origin/release-6.2' into release-6.3
2021-03-10 14:30:47 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
f81d4eb6d3
Remove invalid backup status fields
2021-03-08 15:48:46 -08:00
318b862aa3
Apply clang-format to backup changes.
2021-03-08 01:49:29 -08:00
8a4aca3f47
Bug fix: In simulation only, RangeFileWriter was not waiting on a write future in one location.
2021-03-07 22:19:04 -08:00
8a8c488ede
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-05 18:13:38 -06:00
aaf0a9aa7b
Merge branch 'release-6.3' into merge-release-6.3-into-master
...
# Conflicts:
# build/docker-compose.yaml
# cmake/ConfigureCompiler.cmake
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/IAsyncFile.h
# fdbrpc/IRateControl.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbservice/ServiceBase.cpp
2021-02-08 12:58:34 -08:00
3fc8870ce3
winbuild: fix duplicate definition of Codec<ERestoreState>::pack and unpack in Debug build
...
- defined twice in FileBackupAgent.actor.cpp and BackupAgent.actor.h
- only fails when building in Debug ?
2021-01-30 19:03:25 +01:00
cec050a1f8
Apply clang-tidy's modernize-use-override fix
2021-01-26 02:04:12 +00:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
51db9a7e0a
add static method to access backup pause key instead of constructing it manually
2020-11-06 14:03:29 -05:00
c1d88b39ab
Merge remote-tracking branch 'origin/master' into add-upgrade-and-backup-restore-test
2020-10-17 11:35:12 -07:00
a1e868a569
Merge pull request #3862 from sfc-gh-tclinkenbeard/use-override-more
...
Add uses of override keyword, remove unnecessary uses of virtual
2020-10-14 15:06:45 -07:00
ccf2ca73f3
Fix Windows CMake build
2020-10-08 23:58:58 -07:00
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
a3de8c333b
Use override where applicable in fdbclient
2020-10-07 16:57:38 -07:00
7218586f2f
adjust test spec and backup describe
2020-10-06 17:03:09 -04:00
43ac7f21a4
Mark toString methods const
2020-10-04 23:27:10 -07:00
8ae0679112
Remove unnecessary separation of declaration and definition of version field
2020-10-04 23:27:05 -07:00
6326de052b
Remove unused LogFileWriter::FFs
2020-10-02 17:03:08 -07:00
8da8c6ff0d
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-30 14:25:29 -04:00
44cd3b0999
add stopBackup to incrementalBackup workload
2020-09-30 14:24:52 -04:00
cfaa207f88
fix spacing and newlines at ends of files
2020-09-28 16:32:20 -04:00
3aa4070adb
remove trace events used for debugging
2020-09-28 16:27:30 -04:00
1fb11858e6
Fix failure caused by merge
2020-09-27 11:03:16 -07:00
bbc7ce581e
Resolve conflicts merging from 6.3 to master
2020-09-25 16:08:31 -07:00
6ed330572e
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-24 12:09:17 -04:00
d13a9d3c91
Make EBackupState an enum class
2020-09-23 15:14:09 -07:00
69580593dd
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-23 15:35:05 -04:00
9b7d75b6dd
WIP
2020-09-21 13:41:35 -04:00
a3c861b59d
Merge pull request #3777 from apple/release-6.3
...
Merge Release 6.3 into master
2020-09-14 10:32:50 -07:00
f10e9ea679
FastRestoreApplier:Add write conflict range
2020-09-09 12:12:14 -07:00
e69ee87f33
Refactor blob failure to simulateBlobFailure function
2020-09-09 10:54:21 -07:00
20733ae1eb
Simulate throwing blob http error immediately after read blob file
2020-09-09 06:47:19 -07:00
cffb54ba0b
Buggify decodeLogFileBlock with blob http error
2020-09-09 06:44:02 -07:00
f3986d201d
Buggify backup decodeRangeFileBlock with blob connection error
2020-09-09 06:40:41 -07:00
8ac3b21459
changed occurrences of getOrThrow to get
2020-09-02 13:45:15 -04:00
00c77ba2b4
Added beginVersion cmd line option and addressed code review comments
2020-08-28 14:29:22 -04:00
0270ee13c8
change targetversion from maxlogend to contiguouslogend. also adjusted test workload and specs
2020-08-28 12:46:34 -04:00
35aa1034cd
added new test spec and workload to verify correctness of incremental backup and restore
2020-08-19 13:45:51 -04:00
7dce3a9187
fixed issue with mutations not applying and allow backup to non-empty db
2020-08-11 15:39:21 -04:00
21635f8a28
update backup restore for local testing
2020-08-04 15:48:43 -04:00
fbdbe0f052
adjusted locations in which restore pipeline is modified
2020-07-28 12:39:35 -04:00
2d09d2f68e
re-ordered parameters to fit existing test workloads
2020-07-28 12:39:35 -04:00
71a1b2cb0d
first attempt at making restore pipeline incremental only
2020-07-28 12:39:35 -04:00
3c62651ad6
initial commit to introduce incremental backup only (ignore base snapshot)
2020-07-28 12:39:35 -04:00
b2a3b4fd83
Merge branch 'master' into mengxu/merge-6.3-PR
2020-07-20 11:34:18 -07:00
2e5ae8eed5
Change iterator name to small case
2020-07-10 10:56:41 -07:00
941cf7b03b
Fix const return warning
2020-07-09 22:17:54 +00:00
ab3de8700e
Rename fastRestoreAgent to fastRestoreTool
2020-07-03 00:16:39 +09:00
c13ba7aeb4
FastRestore:Cosmetic code style improvement
...
Also mute some trace events
2020-06-24 21:25:37 -07:00
94b1e17a96
FastRestore:Fix:Avoid reset txn backoff time in sanity check
2020-06-24 21:10:47 -07:00
d2b71e77c2
FastRestore:Fix:checkDatabaseLock tr should not reset
2020-06-24 20:21:49 -07:00
b69c36f0d6
transformDatabaseContents:cosmetic change
2020-06-23 19:27:58 -07:00
198696bc1e
Move transformRestoredDatabase from server to client
...
AtomicRestore workload turns out to rely on the FileBackupAgent
client. Keeping transformRestoredDatabase in server makes linking harder.
2020-06-23 15:48:43 -07:00
6eb8dc58b6
FastRestore:addPrefixTest:Fix calling transformRestoredDatabase
2020-06-23 15:02:38 -07:00
84a640e4ee
AtomicRestore:Add addPrefix test for fast restore
2020-06-23 14:52:27 -07:00
4e27fd34e5
Refactor transformDatabaseContents into RestoreCommon
...
Prepare to enable addPrefix for atomicRestore
2020-06-23 14:33:13 -07:00
2fcef90cbc
FastRestore:Add addPrefix and removePrefix feature
2020-06-18 11:58:42 -07:00
d7a8e554ed
FastRestore:Test for addPrefix and removePrefix
...
Add the skeleton code to test;
addPrefix and removePrefix are all empty for now.
2020-06-17 20:36:53 -07:00
164814cd23
Merge branch 'release-6.3' into mengxu/fr-addPrefix-PR
...
Also remove the unnecessary try catch block for unlockDatabase() in
parallelRestoreFinish actor
2020-06-09 22:04:19 -07:00
df064ac922
Merge pull request #3321 from xumengpanda/mengxu/fr-restore-ranges-PR
...
Fast Restore: Support restoring sub ranges in the framework
2020-06-09 21:01:39 -07:00
dd8af487ec
FileBackupAgent:parallelRestoreFinish:verify if operation_cancelled error ever happens
2020-06-09 17:08:57 -07:00
8533daac50
parallelRestoreFinish:fix used_during_commit error
2020-06-09 15:40:24 -07:00
e5e4cb5c90
parallelRestoreFinish:separate check restore finished and clear restoreRequestDoneKey
2020-06-09 11:32:20 -07:00
d85dc5a4d3
FastRestore:Only clear ranges that will be restored
...
Instead of clearning the entire normal key space.
This commit also removes some unnecessary tr->reset() which can invalid the txn backoff time.
2020-06-08 22:41:49 -07:00
e7fdd614fe
Remove assert on error_code_key_not_found in previous commit
...
Correctnss shows the assert does not hold in all scenarios
2020-06-08 19:02:56 -07:00
5022566b35
Validate if key_not_found error ever happens
2020-06-08 16:59:00 -07:00
a833724322
Merge pull request #3078 from xumengpanda/mengxu/fr-circus-stall-PR
...
Performant restore: Various improvements based on circus test
2020-05-07 20:07:23 -07:00
7b5d43da9c
FastRestore:Remove unused field in RestoreRequest
2020-05-03 20:59:47 -07:00
494d1d3518
Fixed unused variable warning in atomicParallelRestore
2020-05-01 18:14:51 -07:00
a998fa1e1a
Merge pull request #3053 from xumengpanda/mengxu/fr-bc-fix-PR
...
Fix performant restore bug introduced in PR 3041
2020-04-29 17:09:26 -07:00
3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd
...
Several bug fixes for new backups
2020-04-29 16:34:59 -07:00
9da76c35ea
Fix a memory corruption error
...
The backup container URL should be Key instead of KeyRef.
2020-04-29 15:55:34 -07:00
33dedaab35
FastRestore:Fix submitParallelRestore when targetVersion is unset
2020-04-29 15:44:54 -07:00
364142d02c
Fix backupWorkerEnabled flag not set bug for first backup
...
This can cause restore failures because the latestLogEndVersion could be wrong.
2020-04-28 15:39:24 -07:00
7d59e53349
Consolidate makePadding()
2020-04-28 15:39:23 -07:00
f5e8345496
FastRestoreAgent:Use atomicParallelRestore to kick off restore
...
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
2020-04-27 22:15:00 -07:00
3063611355
Write range files' begin & end keys to manifest file
...
This information can be very useful in knowing the content in these files,
especially for restores.
2020-04-20 22:26:42 -07:00
60407bdee3
Use LiteralStringRef for backup paused key
2020-04-07 16:02:25 -07:00
9fb3fb9d82
Add pause/resume for new backups
...
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
2020-04-06 14:29:46 -07:00
efbd01894d
Really fix the stuck of backup
2020-03-28 17:38:02 -07:00
f0d0c863e6
Attempt to fix
2020-03-28 17:38:02 -07:00
78f74ed6fe
Skip copying inside BackupLogsDispatchTask
2020-03-28 17:38:02 -07:00
a2b8be5dc6
Switch off old mutation logging on proxies for new backups
...
Specifically, stop writing LogRange mutation for new backup jobs.
2020-03-28 17:38:02 -07:00
9a9af7d8a8
Add more trace event details on partitioned log
2020-03-26 13:57:31 -07:00
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR
...
Add ApiCorrectnessAtomicRestore workload for the new performant restore
2020-03-25 18:05:26 -07:00
495afe2e0b
Improve how to wati for restore to finish
...
Remove default parameter for atomicRestore as suggested in review.
2020-03-25 13:54:21 -07:00
1ba11dc74b
Apply clang format
2020-03-25 11:20:17 -07:00
120272f025
Change unlockDB from RestoreMaster to Agent
2020-03-25 11:04:49 -07:00
130b91c820
Fix segmentation fault in submitParallelRestore
2020-03-24 23:52:34 -07:00
Jingyu Zhou