Commit Graph

325 Commits

Author SHA1 Message Date
Nim Wijetunga 70d30d93aa
Add CLI Argument to Override Snapshot Backup Encryption (#8661)
* add cli option

* fix compile error

* address pr comments

* trigger build

* trigger build
2022-11-07 20:39:18 -08:00
Nim Wijetunga 6f37f55917
Restore System Keys First in Backup/Restore Workloads (#8475)
* system key restore ordering

* restore system keys before regular data

* atomic restore backup fix

* change testing

* fix compile error

* fix compile issue

* fix compile issues

* Trigger Build

* only split restore if encryption is enabled

* revert knob changes

* Update fdbserver/workloads/AtomicSwitchover.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/AtomicSwitchover.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/BackupCorrectness.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/AtomicRestore.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* add todo

* strengthen check

* seperate system restore for atomic restore

* address pr comments

* address pr comments

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-10-26 09:38:27 -07:00
Nim Wijetunga 025359a974
Add Options to restore user/system keys in backup agent (#8497)
* add options to restore user/system keys

* format help commands

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbbackup/backup.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbbackup/backup.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* address pr comments

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-10-20 09:58:00 -07:00
Nim Wijetunga 232e23c87e
Snapshot Backup Encryption (#8095)
* inital commit

* add encryption support

* refactor passing encryption keys

* disable encryption codepath

* pass enable encryption cli arg for fdbbackup

* change encryption key passing

* only enable encryption for backup if tenant mode is required

* revert server knobs

* ignore json trace files

* --amend

* integrate ekp

* add comments

* comments

* comments

* revert knobs

* fix bug

* modify encrypted backup logic

* use common prefix

* check boundaries between begin and end key

* return system encrypt domain

* modify

* add tenant entry cache to backup

* slight revert

* format

* Trigger Build

* format

* use tenant name to fetch keys

* Trigger Build

* Trigger Build

* merge

* fix getting ekp

* fix bug

* address pr comments

* Trigger Build

* Trigger Build

* remove faulty check
2022-09-29 14:45:47 -07:00
A.J. Beamon fda0d7223d Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants. 2022-09-22 10:00:13 -07:00
A.J. Beamon 4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
Dennis Zhou 80a0816157
flow: switch from hard coded to ApiVersion like ProtocolVersion (#8071)
* flow: add ApiVersion to replace hard coding api version

Instead of hard coding api value, let's rely on feature versions akin to
ProtocolVersion.

* ApiVersion: remove use of -1 for latest and use LATEST_VERSION
2022-09-02 09:28:13 +02:00
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
A.J. Beamon 7bca503528 Use narrow includes in key backed types; remove some unnecessary return statements. 2022-07-14 13:26:43 -07:00
A.J. Beamon 1dc78a3b3f Add support for different transaction types (Reference<ITransaction>, Reference<ReadYourWritesTransaction>) to key backed types 2022-07-01 10:02:39 -07:00
Markus Pilman 03d913a1de Flow compiling 2022-06-27 17:05:55 -06:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Renxuan Wang 13b2b455b1 HTTP proxy for backup.
Only support non-TLS blobs now.
2022-05-09 13:56:33 -07:00
sfc-gh-tclinkenbeard 06825775db Fix formatting of lines with TLS_OPTION_FLAGS 2022-05-02 22:56:06 -07:00
sfc-gh-tclinkenbeard 7f05221cfe Removed TLS_DISABLED macro 2022-05-02 22:15:27 -07:00
Yi Wu 994b8c92f8
Add option to limit resident memory and remove default memory limit (#6719)
Changing `memory` option to limit resident memory instead of virtual memory, in config file and fdbserver/fdbbackup/fdbcli command-line argument. Since `rlimit` doesn't support limiting virtual memory, the current implementation have both of fdbmonitor and the fdbserver/fdbbackup process checking process RSS periodically and kill and restart the process if the limit is exceeded.

Adding a new `memory_vsize` option to limit virtual memory, if backward-compatible behavior is desired.

closes #6671, closes #6672
2022-04-06 20:06:24 -07:00
Jingyu Zhou f68fd28d73 Refactor duplicated code into IKnobCollection::setupKnobs() 2022-04-05 02:06:38 -07:00
Renxuan Wang e1775627ab Add a check on proxy format. 2022-03-29 10:17:14 -07:00
Renxuan Wang 0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
sfc-gh-tclinkenbeard 8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
Jingyu Zhou 1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Renxuan Wang 233c918ffb Replace printf() and fprintf() with fmt::print(). 2022-02-25 19:06:57 -08:00
Renxuan Wang f7eb66441d Try eliminating warnings in macOS and Windows CI builds.
MacOS warnings are format warnings, e.g., `format specifies type 'long' but the argument has type 'Version' (aka 'long long')`.
Windows warnings are `ACTOR does not contain a wait() statement`.
2022-02-25 19:06:57 -08:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
sfc-gh-tclinkenbeard ef711d405c Add fdbbackup tags command 2022-01-24 13:34:54 -08:00
A.J. Beamon ff1cb58174 Convert hyphens to underscores for all prefix-based arguments (e.g. --knob-, --locality-) 2021-12-14 12:01:44 -08:00
A.J. Beamon f24adc7b6a Fix a bunch of places where we used old-style arguments. Allow hyphens for profiler args. 2021-12-14 09:59:14 -08:00
A.J. Beamon f29f487823
Unify flags (#25)
* Unify flags implementation and change help text in backup.actor.cpp
Description

Testing

* Keep LOG_GROUP unchanged

Description

Testing

* Transfer the hyphens to underscores for internal options and user's input, EXCEPT leading hyphens

Description

Testing

* Use a deep copy of the user's input flag to do the match

Description

Testing

* Convert the _ to - in Option arrays of backup.actor.cpp

Description

Testing

* Transter _ to - for files:
        TLSConfig.actor.h, fdbcli.actor.cpp, fdbserver.actor.cpp, FileConverter.h, FileConverter.cpp

Description

Testing

* Change another way to unify flag: using SO_O_ICASE_HYPHEN_AND_UNDERSCORE to determine whether we do the conversion in function IsEqual

Description

Testing

* Change the config command's name from SO_O_ICASE_HYPHEN_AND_UNDERSCORE to SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Update the comment for the SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Fix left underscore in SOption arrays

Description

Testing

* Convert _ to - in several files for commands

Description

Testing

* Make the FDBService and fdbmonitor backward compatible

Description

Testing

* Fix bugs about pointers

Description

Testing

* Check underscore and hyphen at the same time for --knob_, --localily_ and --test_
And fix bugs in fdbmonitor and FDBService
Description

Testing

* Simplify the function in fdbmonitor and FDBService about retrieving arguments.
And fix some documents in masterserver.actor.cpp

Description

Testing

* Convert _ to - for knob in the setKnob functions

Description

Testing

* Convert - to _ in the setKnob functions

Description
Since key in the knob related maps only contain _

Testing

* Rename varialbe name in the fdbmonitor and FDBService for clarification

Description

Testing

Co-authored-by: Chang Liu <chang.liu@snowflake.com>
2021-12-14 08:44:39 -08:00
sfc-gh-tclinkenbeard 464d9488ef Merge remote-tracking branch 'origin/master' into fix-unused-warnings 2021-12-01 23:52:09 -08:00
sfc-gh-tclinkenbeard 90ced244eb Fix -Wunused-but-set-variable warnings 2021-12-01 18:15:53 -08:00
sfc-gh-tclinkenbeard ec64890ac1 Remove some usages of PRId64 by using fmt library 2021-11-30 23:35:36 -08:00
sfc-gh-tclinkenbeard deb0eb85e8 Initialize global flow knobs in setupNetwork 2021-11-16 00:41:34 -08:00
A.J. Beamon e882eb33fc Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem. 2021-10-22 11:05:18 -07:00
Xiaoge Su e68b131e4a fixup! Reformat source code 2021-09-16 19:40:28 -07:00
Xiaoge Su abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
A.J. Beamon d4c67eb13d Fix crash when DR binaries are invoked without a source cluster file. Update the usage text for dr_agent cluster files to include the long form options. 2021-08-20 12:55:13 -07:00
Vishesh Yadav de93e2f854
Update fdbbackup/backup.actor.cpp
Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2021-08-04 20:27:43 -07:00
Vishesh Yadav 98fa70da11 backup: Show error info for RemovedDeadBackupLayerStatus 2021-08-04 13:28:47 -07:00
Steve Atherton f596a81073 Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs. 2021-07-17 00:11:40 -07:00
sfc-gh-tclinkenbeard 020371a78f Merge remote-tracking branch 'origin/master' into add-boolean-param 2021-07-07 16:50:51 -07:00
Trevor Clinkenbeard f24581b644
Merge pull request #5120 from sfc-gh-tclinkenbeard/improve-poll-delay-safety
Improve safety of backup agent poll delay
2021-07-07 14:42:33 -07:00
sfc-gh-tclinkenbeard 3d0536dcec Improve safety of backup agent poll delay 2021-07-05 13:22:31 -07:00
sfc-gh-tclinkenbeard 8370d92c20 Expand use of BOOLEAN_PARAM in backup code 2021-07-04 18:30:55 -07:00
sfc-gh-tclinkenbeard cadfdc551e Use BOOLEAN_PARAM for backup agents 2021-07-04 16:27:26 -07:00
sfc-gh-tclinkenbeard 8cc40e3a2b Expand use of BOOLEAN_PARAM 2021-07-02 21:41:50 -07:00
sfc-gh-tclinkenbeard 79ff07a071 Added *BOOLEAN_PARAM macros to enforce documentation of boolean parameters 2021-07-02 15:04:42 -07:00
sfc-gh-tclinkenbeard 27e44c1bb9 Add support for encryption testing in BackupCorrectness workload 2021-06-26 11:15:12 -07:00
sfc-gh-tclinkenbeard 1afae7623b Added /backup/containers/localdir/encrypted unit test 2021-06-25 22:33:26 -07:00
sfc-gh-tclinkenbeard f5aa3df917 Add --encryption_key_file command line argument to fdbbackup and fdbrestore 2021-06-25 15:11:03 -07:00