Commit Graph

366 Commits

Author SHA1 Message Date
Aaron Molitor ca1b606d70 move global variables into function args/local variables, build both regular and debug images by default 2022-08-23 11:43:49 -05:00
Vaidas Gasiunas d99d0370b1
C shim library: API for setting client library path; additional tests (#7702)
* Adding sources of the Implib.so project

* Run C unit tests and API tests with the shim library

* Reuse compilation of C test binaries with and without shim library

* Resolve client library path from an environment variable

* Refactoring: Reusable module for downloading FDB binaries

* Testing client shim library with current version and last release version

* Tests for specifying client library over an environment variable

* Enable C shim library tests on ARM

* Restore the original path for including fdb_api.hpp

* Improve cmake dependencies on doctest

* Remove unnecessary Implib.so files

* Shim library: API to specify path to the library, additional tests

* Adding shim library to Linux distribution packages

* Update link dependencies of fdb_c_shim_lib_tester

* Add dependencies on Implib.so sources

* Abort on fatal errors of shim library

* fix typo
2022-07-29 11:45:45 +02:00
Johannes M. Scheuermann b9afde3d4c Fix permission in Dockerfile 2022-07-05 20:02:51 -07:00
Johannes M. Scheuermann d1b085141d Remove mv flags 2022-06-27 17:44:27 -05:00
Johannes M. Scheuermann 0334cda487 Revert client lib change back 2022-06-27 17:44:27 -05:00
Johannes M. Scheuermann 4a333f5cba Correct script for different build setups 2022-06-27 17:44:27 -05:00
Johannes M. Scheuermann 2b83df5e58 Correct comment for script 2022-06-27 17:44:27 -05:00
Johannes M. Scheuermann 0d05f0f2d8 Refactor Docker setup for new binary location 2022-06-27 17:44:27 -05:00
Johannes Scheuermann 4b0c4a32b0
Add testing for Kubernetes sidecar (#7105)
* Refactor python sidecar and add unit tests

* Fix issue trying to send error response multiple times

* Fix imports and TLS handling

* Correct config variable in ssl reload
2022-06-21 19:39:53 +01:00
Johannes M. Scheuermann 8da4bb9d07 Add openssl for debugging in container image 2022-06-21 12:14:06 -05:00
Johannes Scheuermann c9b4ff3302
Add support for lumberjack in logging and update example to 7.1 (#7357) 2022-06-15 19:41:01 +01:00
Johannes M. Scheuermann b53c9fe851 Make use of relpath instead of basepath 2022-05-07 13:18:03 +01:00
Johannes M. Scheuermann 410bcc8d5d Format sidecar script 2022-05-06 08:40:18 -07:00
hao fu 24f6276e14 Eliminate vector string during mapped key processing
There was OOM if we pre-process all the strings.
2022-05-06 08:38:27 -07:00
Johannes M. Scheuermann d1c71a7903 Add sidecar method to check if a file is present 2022-04-29 13:10:05 +01:00
Aaron Molitor 74ec6962b0 update build-images.sh to play nice with AWS IMDSv2 2022-04-28 14:20:46 -05:00
Russell Sears 8819534a5e
Improve YCSB image logging; remove logic to support partial restarts (#6830) 2022-04-18 13:32:04 -07:00
Johannes M. Scheuermann 96181feab6 Fix Windows build path 2022-04-11 08:58:25 -05:00
Johannes M. Scheuermann d2d8f42222 Add new c libs to client package 2022-04-11 08:58:25 -05:00
Steven Li 01234720e7 update prerelease package format (#6733) 2022-04-08 18:15:40 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Aaron Molitor 4ff60db82b Revert "Use full version of fdb to support incompatible release candidates"
This reverts commit f379cf4d05.
2022-02-24 11:55:48 -06:00
Leonidas Tsampros f379cf4d05 Use full version of fdb to support incompatible release candidates 2022-02-24 11:03:42 -06:00
Chris Douglas 3793e4e5f0 Remove rundant WORKDIR directive 2021-12-17 11:34:32 -08:00
Chris Douglas fec0fb9e9f Move awscli to base image from YCSB 2021-12-17 11:25:22 -08:00
Aaron Molitor 95d33cb363 copy packaging/docker to PROJECT_BINARY_DIR (undoing part of #5994),
fetch commit_sha from source_code_directory (don't assume we're in the source tree anymore),
allow custom tag (if a parameter is passed in as $1)
update README.md
2021-12-15 15:23:17 -08:00
A.J. Beamon 5c9b64e414 Backup agent was mistakenly modified in conf files. 2021-12-14 12:02:13 -08:00
A.J. Beamon f24adc7b6a Fix a bunch of places where we used old-style arguments. Allow hyphens for profiler args. 2021-12-14 09:59:14 -08:00
A.J. Beamon f29f487823
Unify flags (#25)
* Unify flags implementation and change help text in backup.actor.cpp
Description

Testing

* Keep LOG_GROUP unchanged

Description

Testing

* Transfer the hyphens to underscores for internal options and user's input, EXCEPT leading hyphens

Description

Testing

* Use a deep copy of the user's input flag to do the match

Description

Testing

* Convert the _ to - in Option arrays of backup.actor.cpp

Description

Testing

* Transter _ to - for files:
        TLSConfig.actor.h, fdbcli.actor.cpp, fdbserver.actor.cpp, FileConverter.h, FileConverter.cpp

Description

Testing

* Change another way to unify flag: using SO_O_ICASE_HYPHEN_AND_UNDERSCORE to determine whether we do the conversion in function IsEqual

Description

Testing

* Change the config command's name from SO_O_ICASE_HYPHEN_AND_UNDERSCORE to SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Update the comment for the SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Fix left underscore in SOption arrays

Description

Testing

* Convert _ to - in several files for commands

Description

Testing

* Make the FDBService and fdbmonitor backward compatible

Description

Testing

* Fix bugs about pointers

Description

Testing

* Check underscore and hyphen at the same time for --knob_, --localily_ and --test_
And fix bugs in fdbmonitor and FDBService
Description

Testing

* Simplify the function in fdbmonitor and FDBService about retrieving arguments.
And fix some documents in masterserver.actor.cpp

Description

Testing

* Convert _ to - for knob in the setKnob functions

Description

Testing

* Convert - to _ in the setKnob functions

Description
Since key in the knob related maps only contain _

Testing

* Rename varialbe name in the fdbmonitor and FDBService for clarification

Description

Testing

Co-authored-by: Chang Liu <chang.liu@snowflake.com>
2021-12-14 08:44:39 -08:00
Aaron Molitor 8aab68303b add discrete tag_postfix to images, cleanup documentation 2021-12-10 15:37:04 -08:00
Aaron Molitor 661181d61c make curl and sha256sum more quiet, pull local binaries from the correct place in build_output_directory 2021-12-03 15:06:09 -08:00
Aaron Molitor 9f5197b54c use FDB_VERSION in lieu of PROJECT_VERSION (package scripts) 2021-11-29 15:11:20 -08:00
Aaron Molitor 77db63274b use FDB_VERSION in lieu of PROJECT_VERSION or CMAKE_PROJECT_VERSION
write FDB_VERSION to a file to avoid regex golf in other build scripts
2021-11-29 15:11:20 -08:00
Aaron Molitor 08b635d405 rename prerelease_string, replace PRERELEASE with SNAPSHOT 2021-11-29 15:11:20 -08:00
Russell Sears d332d03b34 Have individual YCSB containers record completion of their phase of the load. This allows loads to succeed even if the fdb-kubernetes-tests client crashes out mid run 2021-11-19 14:33:23 -08:00
Andrew Noyes 71fa5e86b9 Don't adduser in client postinstall 2021-11-18 10:52:37 -08:00
Aaron Molitor 124e7eb4f2 refactor docker directory
- make release images ALL based from centos:7
- keep eks images that are based from amazonlinux:2, but create strong alignment with release (centos7) images
- use multi-stage Dockerfile(s)
- have a single build-images script
- remove per image directories (which would only contain scripts that are copied into images)
- remove empty requirements.txt file for sidecar.py (the pip command, package and version are in the Dockerfile)
- don't copy docker into PROJECT_BINARY_DIR
- incorporate latest changes already in master
2021-11-17 16:29:07 -08:00
John Brownlee 01c37a053a
Merge pull request #5993 from brownleej/new-image-fixes
Updates to fdb-kubernetes-monitor to ease integration with the Kubernetes operator
2021-11-17 11:56:19 -08:00
Lukas Joswiak fc08313925 Use path relative to Python binary 2021-11-16 16:36:44 -08:00
Lukas Joswiak 8d4ec8562d Move generated versions.target file to build directory 2021-11-16 16:36:44 -08:00
John Brownlee 2f2e5219da Move the config file scheme into a separate package so it can be imported by the operator.
Move the argument for the number of servers per pod into a command-line argument so we can use the same config file at different storage server densities.
2021-11-12 11:58:38 -08:00
Yao Xiao 86ea63d1da
Add libsanitizer to fdb image. (#5918) 2021-11-04 23:21:54 -07:00
John Brownlee 45a66f9f75
Merge pull request #5456 from brownleej/fdb-kubernetes-monitor
Create fdb-kubernetes-monitor subproject
2021-10-29 11:29:23 -07:00
Aaron Molitor 504f08a102 consolidate docker stuff, add perf and flamegraph parts to release image 2021-10-29 09:58:55 -07:00
Leonidas Tsampros a4d784a3dc packaging: apt doesn't support >= and fix tini installation 2021-10-29 09:58:55 -07:00
Aaron Molitor 3305fe92f6 fix typo in script 2021-10-28 19:45:48 -07:00
John Brownlee a6b903e7f8 Move the new Kubernetes image to centos 7. 2021-10-28 17:40:05 -07:00
John Brownlee 1a5069a047 Use an write-and-rename pattern when copying files for atomicity.
Restructure the usage of the mutex in the monitor class.
2021-10-28 17:40:05 -07:00
John Brownlee c7858d2441 Add the init and sidecar modes for fdb-kubernetes-monitor.
Add support for using a special binary path during upgrades in fdb-kubernetes-monitor.
2021-10-28 17:40:05 -07:00
John Brownlee 7c36123cf8 Adds structured logging in fdb-kubernetes-monitor.
Adds a backoff window when restarting processes in fdb-kubernetes-monitor.
2021-10-28 17:40:05 -07:00