Commit Graph

32 Commits

Author SHA1 Message Date
Renxuan Wang 2d78468cc0 Fix the workflow of not using proxy. 2022-08-22 11:16:48 -07:00
Renxuan Wang 657e6f8b60 Complete the proxy support for backup to S3. 2022-08-22 11:16:48 -07:00
Renxuan Wang dc9599f2e9
Get network interfaces ready for https proxy. (#7556)
* Move HTTP from fdbclient/ to fdbrpc/.

* Move md5 and libb64 to contrib/.

* Get network interfaces ready for https proxy.

* Rebase
2022-07-25 17:08:32 -07:00
Markus Pilman 8123070c2a move rapidxml to its own target 2022-06-29 16:10:14 -06:00
Markus Pilman 8af056e7b0 fdbclient now compiling 2022-06-23 18:05:36 -06:00
Markus Pilman ffaf15c12a moved wellknownendpoints and fixed some includes 2022-06-23 17:03:53 -06:00
Xiaoge Su 21ee76a44d fixup! Reformat source #2 2022-06-14 13:22:18 -07:00
Xiaoge Su c2676df2f8 fixup! Reformat source 2022-06-14 13:22:18 -07:00
Xiaoge Su 9fb6e5bb05 fixup! Fix the clang error when using std::move
This patch is to fix the compile error

/root/src/fdbclient/S3BlobStore.actor.cpp:410:9: error: moving a local
object in a return statement prevents copy elision
[-Werror,-Wpessimizing-move]
 return std::move(resource);
        ^
/root/src/fdbclient/S3BlobStore.actor.cpp:410:9: note: remove std::move
call here
 return std::move(resource);
        ^~~~~~~~~~        ~
1 error generated.
2022-06-14 13:22:18 -07:00
Zhanwei Wang e632aef1c7
Make backup work with s3 compatible service (#6355)(#6382) (#7324)
1. Support virtual hosting endpoint.

2. On-premise s3 compatible storage service may use IP instead of s3 form domain name,
especially for development/test environment.

Instead of parsing service and region from domain name,

1). Hard code "s3" as service name in v4 signature
2). Add new parameter to allow pass region name from url

3. Fix creating bucket issue on aws, adding request body.
2022-06-13 13:33:05 -07:00
Zhanwei Wang da346d718f Fix compatibility issue of s3 backup
1. For v4 signature, http request path should be encoded, currently only parameters are encoded.
2. For v2 signature, http request date header should be in form of either the http date header or x-amz-date, currently in form of ISO 8601
3. Empty response header value cause http_bad_response error.
4. Skip verifying MD5 for range get request and if http response code 206 is returned.
2022-05-25 09:48:54 -05:00
Renxuan Wang 9190fffbb6 Commit suggestion.
Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2022-05-09 13:56:33 -07:00
Renxuan Wang 13b2b455b1 HTTP proxy for backup.
Only support non-TLS blobs now.
2022-05-09 13:56:33 -07:00
Sam Gwydir 5403a29ecb
add WolfSSL support (#6682)
remove extraneous include
2022-04-28 16:53:38 -07:00
Renxuan Wang 465ff712b6
Move Hostname to its own files. (#6759)
* Change DNS cache to use std::map.

Revert commit 90c259d84e, because if we use unordered_map, toString() can be inconsistent.

* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.

* Move Hostname to its own files.

Also, add resolve-related variables and functions in Hostname.
2022-04-04 19:04:51 -07:00
Renxuan Wang e1775627ab Add a check on proxy format. 2022-03-29 10:17:14 -07:00
Renxuan Wang 0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum 5684fcace4 Merge branch 'main' into s3_sdk_credentials 2022-03-24 15:44:41 -05:00
Jingyu Zhou 902c10c299 Fix missing blob access key in credentials
This causes failures of backup_auth_missing: ErrorDescription="Cannot find
authentication details (such as a password or secret key) for the specified
Backup Container URL"

This is found by manually running "fdbdecode".
2022-03-24 09:26:41 -07:00
Josh Slocum ace2750ae3 Build AWS SDK as part of FDB, and use it to acquire blob credentials. This supports IAM role-based authentication, in addition to all existing credential types. 2022-03-24 10:09:06 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Bala Namasivayam 0f36157c2c Clang format files 2022-02-07 17:53:05 -08:00
Bala Namasivayam a420098fc5 Support AWS v4 header for s3 backup and restore 2022-02-07 17:53:05 -08:00
sfc-gh-tclinkenbeard ac274cf479 clang-format S3BlobStore* files 2021-05-10 10:25:32 -07:00
Kevin King b10ea7677e [fdbbackup] check for credentials inside of setAuthHeaders 2021-05-10 10:21:37 -07:00
Kevin King e3114a4656 [fdbbackup] make credentials optional
when blob storage is configured on a private network (such as an AWS
VPC), there isn't a need to specify http auth
2021-05-10 08:42:27 -07:00
FDB Formatster df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
sfc-gh-tclinkenbeard 5b2e88b187 Use structured bindings in for loops 2020-12-27 01:46:20 -04:00
sfc-gh-tclinkenbeard 4669f837fa Add uses of makeReference 2020-11-07 22:10:18 -08:00
sfc-gh-tclinkenbeard ca26cb57a2 Reapply BlobStore bug fix to S3BlobStore.actor.cpp
Because BlobStore.actor.cpp was renamed, this commit was lost in the
merge, so reapplying here
2020-10-30 10:06:35 -07:00
sfc-gh-tclinkenbeard 55e86c1336 Disambiguate between S3BlobStore and other blob stores 2020-10-29 20:42:23 -07:00