Commit Graph

465 Commits

Author SHA1 Message Date
Nim Wijetunga 05a8a90830
Snapshot Backup Tenant Deletion Support (#9145)
Tenant deletion support for snapshot backups
2023-01-26 17:46:14 -08:00
Xiaoxi Wang bbcb3cc018 extract KeyBackedConfig, StorageWiggleData class; solve template resolution problem; solve MV txn and native api conflict by splitting RunTransaction file 2023-01-02 23:34:39 -08:00
sfc-gh-tclinkenbeard 49ab88e912 Fix more clang 15 warnings 2022-12-08 14:25:36 -08:00
FoundationDB CI 86d6106dc1
format source code after switch to clang 15 2022-12-08 17:26:45 +00:00
sfc-gh-tclinkenbeard 82db9415fb Add const qualifier to more methods
This commit mainly targets get* and is* methods
2022-11-15 14:57:32 -08:00
Nim Wijetunga 70d30d93aa
Add CLI Argument to Override Snapshot Backup Encryption (#8661)
* add cli option

* fix compile error

* address pr comments

* trigger build

* trigger build
2022-11-07 20:39:18 -08:00
Nim Wijetunga 6f37f55917
Restore System Keys First in Backup/Restore Workloads (#8475)
* system key restore ordering

* restore system keys before regular data

* atomic restore backup fix

* change testing

* fix compile error

* fix compile issue

* fix compile issues

* Trigger Build

* only split restore if encryption is enabled

* revert knob changes

* Update fdbserver/workloads/AtomicSwitchover.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/AtomicSwitchover.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/BackupCorrectness.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/AtomicRestore.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* add todo

* strengthen check

* seperate system restore for atomic restore

* address pr comments

* address pr comments

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-10-26 09:38:27 -07:00
Nim Wijetunga 025359a974
Add Options to restore user/system keys in backup agent (#8497)
* add options to restore user/system keys

* format help commands

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbbackup/backup.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbbackup/backup.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* address pr comments

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-10-20 09:58:00 -07:00
Nim Wijetunga 232e23c87e
Snapshot Backup Encryption (#8095)
* inital commit

* add encryption support

* refactor passing encryption keys

* disable encryption codepath

* pass enable encryption cli arg for fdbbackup

* change encryption key passing

* only enable encryption for backup if tenant mode is required

* revert server knobs

* ignore json trace files

* --amend

* integrate ekp

* add comments

* comments

* comments

* revert knobs

* fix bug

* modify encrypted backup logic

* use common prefix

* check boundaries between begin and end key

* return system encrypt domain

* modify

* add tenant entry cache to backup

* slight revert

* format

* Trigger Build

* format

* use tenant name to fetch keys

* Trigger Build

* Trigger Build

* merge

* fix getting ekp

* fix bug

* address pr comments

* Trigger Build

* Trigger Build

* remove faulty check
2022-09-29 14:45:47 -07:00
A.J. Beamon fda0d7223d Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants. 2022-09-22 10:00:13 -07:00
A.J. Beamon 4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
Dennis Zhou 80a0816157
flow: switch from hard coded to ApiVersion like ProtocolVersion (#8071)
* flow: add ApiVersion to replace hard coding api version

Instead of hard coding api value, let's rely on feature versions akin to
ProtocolVersion.

* ApiVersion: remove use of -1 for latest and use LATEST_VERSION
2022-09-02 09:28:13 +02:00
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
A.J. Beamon 7bca503528 Use narrow includes in key backed types; remove some unnecessary return statements. 2022-07-14 13:26:43 -07:00
A.J. Beamon 1dc78a3b3f Add support for different transaction types (Reference<ITransaction>, Reference<ReadYourWritesTransaction>) to key backed types 2022-07-01 10:02:39 -07:00
Markus Pilman 60bb5442f3 fdbbackup compiling 2022-06-27 19:24:26 -06:00
Markus Pilman 03d913a1de Flow compiling 2022-06-27 17:05:55 -06:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Robert Barabas 8606923da2
Arm64 related build fixes (#7319)
* Add missing include

* Fix open call on arm64

* Bump up doctest to 2.4.8
2022-06-08 11:20:27 -07:00
Renxuan Wang 13b2b455b1 HTTP proxy for backup.
Only support non-TLS blobs now.
2022-05-09 13:56:33 -07:00
sfc-gh-tclinkenbeard 06825775db Fix formatting of lines with TLS_OPTION_FLAGS 2022-05-02 22:56:06 -07:00
sfc-gh-tclinkenbeard 7f05221cfe Removed TLS_DISABLED macro 2022-05-02 22:15:27 -07:00
Yi Wu 994b8c92f8
Add option to limit resident memory and remove default memory limit (#6719)
Changing `memory` option to limit resident memory instead of virtual memory, in config file and fdbserver/fdbbackup/fdbcli command-line argument. Since `rlimit` doesn't support limiting virtual memory, the current implementation have both of fdbmonitor and the fdbserver/fdbbackup process checking process RSS periodically and kill and restart the process if the limit is exceeded.

Adding a new `memory_vsize` option to limit virtual memory, if backward-compatible behavior is desired.

closes #6671, closes #6672
2022-04-06 20:06:24 -07:00
Mohamed Oulmahdi 5dc07c55f7 Add missing Windows headers 2022-04-06 14:43:59 +02:00
Jingyu Zhou dcfb6ee520
Added an option to fdbdecode for saving processed files and fixed a memory corruption issue in clang build (#6767)
* Add save file locally feature to fdbdecode

* Fix use-after-free in fdbdecode

This only happens in clang, though.

* Minor refactor for fdbdecode

* Fix a small memory leak in fdbdecode
2022-04-05 16:16:59 -07:00
Jingyu Zhou f68fd28d73 Refactor duplicated code into IKnobCollection::setupKnobs() 2022-04-05 02:06:38 -07:00
Renxuan Wang e1775627ab Add a check on proxy format. 2022-03-29 10:17:14 -07:00
Renxuan Wang 0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Jingyu Zhou 2532ac5882 Add knob value changes to fdbdecode
E.g., HTTP_VERBOSE_LEVEL, and HTTP_REQUEST_AWS_V4_HEADER knobs.
2022-03-24 09:26:41 -07:00
Jingyu Zhou cff1bc24fc Fix decoder for missing ClearRanges
If the ClearRange mutation happens within the given key prefix space,
previously the mutation is not printed. Fix by checking the prefix range
overlaps with the ClearRange mutation.
2022-03-22 16:04:56 -07:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
sfc-gh-tclinkenbeard 8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
Jingyu Zhou 1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Renxuan Wang 233c918ffb Replace printf() and fprintf() with fmt::print(). 2022-02-25 19:06:57 -08:00
Renxuan Wang f7eb66441d Try eliminating warnings in macOS and Windows CI builds.
MacOS warnings are format warnings, e.g., `format specifies type 'long' but the argument has type 'Version' (aka 'long long')`.
Windows warnings are `ACTOR does not contain a wait() statement`.
2022-02-25 19:06:57 -08:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
sfc-gh-tclinkenbeard ef711d405c Add fdbbackup tags command 2022-01-24 13:34:54 -08:00
A.J. Beamon ff1cb58174 Convert hyphens to underscores for all prefix-based arguments (e.g. --knob-, --locality-) 2021-12-14 12:01:44 -08:00
A.J. Beamon f24adc7b6a Fix a bunch of places where we used old-style arguments. Allow hyphens for profiler args. 2021-12-14 09:59:14 -08:00
A.J. Beamon f29f487823
Unify flags (#25)
* Unify flags implementation and change help text in backup.actor.cpp
Description

Testing

* Keep LOG_GROUP unchanged

Description

Testing

* Transfer the hyphens to underscores for internal options and user's input, EXCEPT leading hyphens

Description

Testing

* Use a deep copy of the user's input flag to do the match

Description

Testing

* Convert the _ to - in Option arrays of backup.actor.cpp

Description

Testing

* Transter _ to - for files:
        TLSConfig.actor.h, fdbcli.actor.cpp, fdbserver.actor.cpp, FileConverter.h, FileConverter.cpp

Description

Testing

* Change another way to unify flag: using SO_O_ICASE_HYPHEN_AND_UNDERSCORE to determine whether we do the conversion in function IsEqual

Description

Testing

* Change the config command's name from SO_O_ICASE_HYPHEN_AND_UNDERSCORE to SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Update the comment for the SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Fix left underscore in SOption arrays

Description

Testing

* Convert _ to - in several files for commands

Description

Testing

* Make the FDBService and fdbmonitor backward compatible

Description

Testing

* Fix bugs about pointers

Description

Testing

* Check underscore and hyphen at the same time for --knob_, --localily_ and --test_
And fix bugs in fdbmonitor and FDBService
Description

Testing

* Simplify the function in fdbmonitor and FDBService about retrieving arguments.
And fix some documents in masterserver.actor.cpp

Description

Testing

* Convert _ to - for knob in the setKnob functions

Description

Testing

* Convert - to _ in the setKnob functions

Description
Since key in the knob related maps only contain _

Testing

* Rename varialbe name in the fdbmonitor and FDBService for clarification

Description

Testing

Co-authored-by: Chang Liu <chang.liu@snowflake.com>
2021-12-14 08:44:39 -08:00
sfc-gh-tclinkenbeard 464d9488ef Merge remote-tracking branch 'origin/master' into fix-unused-warnings 2021-12-01 23:52:09 -08:00
sfc-gh-tclinkenbeard 90ced244eb Fix -Wunused-but-set-variable warnings 2021-12-01 18:15:53 -08:00
sfc-gh-tclinkenbeard ec64890ac1 Remove some usages of PRId64 by using fmt library 2021-11-30 23:35:36 -08:00
sfc-gh-tclinkenbeard deb0eb85e8 Initialize global flow knobs in setupNetwork 2021-11-16 00:41:34 -08:00
A.J. Beamon e882eb33fc Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem. 2021-10-22 11:05:18 -07:00
Xiaoge Su e68b131e4a fixup! Reformat source code 2021-09-16 19:40:28 -07:00
Xiaoge Su abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
FDB Formatster 2c788c233d apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-08-27 17:07:47 -07:00
A.J. Beamon d4c67eb13d Fix crash when DR binaries are invoked without a source cluster file. Update the usage text for dr_agent cluster files to include the long form options. 2021-08-20 12:55:13 -07:00
Steve Atherton e5e8a56b66 Merge branch 'master' of https://github.com/apple/foundationdb into debugging-cleanup
# Conflicts:
#	fdbbackup/FileDecoder.actor.cpp
2021-08-09 23:36:57 -07:00