Commit Graph

8238 Commits

Author SHA1 Message Date
sevki f35b586052 bindings/go: explicit CFLAGS and LDFLAGS for macOS
When applied these changes will tell the CGO toolchain on macOS
the default installation location for the headers and libs.

I've recently started playing around with FDB go bindings on my mac.
I've followed the tutorial to set everything up but whilst doing so
I've started getting some error messages like so:

     /Users/sevki/go/pkg/mod/github.com/apple/foundationdb/bindings/go@v0.0.0-20200125010749-6945a6ea0141/src/fdb/cluster.go:26:11: fatal error: 'foundationdb/fdb_c.h' file not found
      #include <foundationdb/fdb_c.h>
                ^~~~~~~~~~~~~~~~~~~~~~

These errors are most probably caused by me unsetting LDFLAGS, CFLAGS
globally as the installer installs the header files and the libraries
to sane default locations.

However, I believe explicitly telling the CGO toolchain here is a
better experience for using this library. While unsetting LDFLAGS
and CFLAGS is not a very smart thing to do, some IDEs and other build
tools make it very hard to pass environment variables to set in this
context.
2020-01-30 13:54:54 +00:00
Alex Miller 99a1a3694b
Merge pull request #2610 from Daniel-B-Smith/clang-link
Add link option specific to Clang
2020-01-29 18:19:04 -08:00
Daniel Smith 9f1fda554c
Formatting 2020-01-28 21:32:56 -05:00
Daniel Smith 4ecf58adff
Apply suggestions from code review
Co-Authored-By: Markus Pilman <markus@pilman.ch>
2020-01-28 21:32:13 -05:00
Xin Dong 125fe0ac27
Merge pull request #2611 from alexmiller-apple/fix-the-build
Force setuptools to be upgraded during sphinx build
2020-01-28 15:17:05 -08:00
Alex Miller 714850bfa4 Set an actual minimal version for setuptools
It looks like PEP 508 was merged into setuptools 20?
2020-01-28 15:11:55 -08:00
Alex Miller ec87850bfa 45.0.0 not pushed to pypi but 44.0.0 is 2020-01-28 15:06:38 -08:00
Alex Miller e43ee302b7 Force setuptools to be upgraded during sphinx build
So that setuptools understands PEP 508, which is now required for a
transitive watchdog dependency.
2020-01-28 15:02:52 -08:00
A.J. Beamon 71d41d0981
Merge pull request #2609 from ajbeamon/merge-release-6.2-into-master
Merge release 6.2 into master
2020-01-28 14:22:13 -08:00
A.J. Beamon adc72dde43 Merge branch 'release-6.2' into merge-release-6.2-into-master
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
2020-01-28 12:16:24 -08:00
Daniel Smith b9de844ee7 Add link option specific to Clang 2020-01-28 12:51:05 -05:00
A.J. Beamon fa51a1abc5
Merge pull request #2604 from xumengpanda/mengxu/fast-restore-valgrind-fix-PR14
Performant restore [14/XX Add-on]: Fix initialized field in VersionBatch struct
2020-01-27 15:22:31 -08:00
Vishesh Yadav bfcdf6bc88
Merge pull request #2243 from canardleteer/go-pack-with-versionstamp
Add PackWithVersionstamp to Go Subpace & Directory bindings.
2020-01-27 08:34:20 -08:00
A.J. Beamon b566a0bcdf
Merge pull request #2602 from AlvinMooreSr/crc32c_update
Updated crc32c File Location
2020-01-27 08:09:52 -08:00
Meng Xu 76f30e71dc FastRestore:Init VersionBatch explicitly
Built-in variable may not be zero initialized by
compiler provided default constructor.
2020-01-26 13:15:45 -08:00
Alvin Moore d03e49b4a1 Fixed the location of crc32c.h from fdbrpc to flow 2020-01-26 07:01:25 -08:00
Evan Tschannen 5c325940ab
Merge pull request #2599 from etschannen/release-6.2
establishing 20 TLS connections in parallel is too expensive
2020-01-25 10:59:56 -08:00
Evan Tschannen 1ed3ba7170 establishing 20 TLS connections in parallel is too expensive 2020-01-25 10:59:20 -08:00
John Brownlee 8972027f43
Merge pull request #2456 from ajbeamon/document-api-version-upgrade-guide
Add an API version upgrade guide
2020-01-24 18:30:53 -08:00
Alex Miller 792e9ed85d
Merge pull request #2574 from ajbeamon/use-python3-in-docpreview
Use python3 in docpreview make target
2020-01-24 18:20:39 -08:00
Alex Miller 6945a6ea01
Merge pull request #2345 from zjuLcg/add-consistency-verification-in-mako-workload
Add consistency verification in mako workload
2020-01-24 17:07:49 -08:00
Alex Miller d06d664ed7
Merge pull request #2149 from tapaswenipathak/ticket-2135
Add comments to explain functions in ReplicationUtils.cpp
2020-01-24 16:35:11 -08:00
Alex Miller 2bc5b2cf8a
Merge pull request #2585 from Ma27/fix-glibc230-build
Fix build with glibc 2.30
2020-01-23 20:21:32 -08:00
Evan Tschannen 3453d5563d
Merge pull request #2591 from etschannen/master
fix: backupWorker would crash when run outside of simulation
2020-01-23 19:08:08 -08:00
Evan Tschannen 8f599e9d15 fix: backupWorker would crash when run outside of simulation 2020-01-23 19:06:39 -08:00
Evan Tschannen 76e192d490
Merge pull request #2538 from alexmiller-apple/hashlittle2-to-crc32c
Convert more hashlittle{,2} uses to crc32c_append
2020-01-23 17:54:38 -08:00
Evan Tschannen 6c0b934dda
Merge pull request #2242 from alexmiller-apple/fix-10min-stall-again
Fix the 10min multi-region recovery stall again
2020-01-23 17:53:02 -08:00
A.J. Beamon b2c8a4a34c
Merge pull request #2519 from xumengpanda/mengxu/fast-restore-versionBatch-fixSize-PR
Performant restore [14/XX]: Ensure each version-batch not exceed a configured size
2020-01-23 16:49:01 -08:00
A.J. Beamon 8a065b9da4
Merge pull request #2557 from alexmiller-apple/reduce-versionstamp-conflictranges
Narrow the unreadable range of keys after a versionstamped key operation
2020-01-23 11:14:47 -08:00
Evan Tschannen fe0e10c312
Merge pull request #1625 from jzhou77/backup-worker
Create a backup role in the cluster
2020-01-23 10:06:35 -08:00
Vishesh Yadav b5c1c8cdd0
Merge pull request #2560 from ajbeamon/java-bindingtester-error-retry
Add retry loop to Java binding tester instruction reader
2020-01-23 09:54:43 -08:00
A.J. Beamon 297a56c219
Merge pull request #2582 from mpilman/features/documentation-server
Add `docpreview` target
2020-01-23 09:43:18 -08:00
Maximilian Bosch e133cb974b
Fix build with glibc 2.30
The `gettid()` function is part of glibc 2.30[1]. I decided to keep the
`gettid` implementation here under a different name to remain compatible
to older glibc versions.

[1] https://sourceware.org/ml/libc-alpha/2019-08/msg00029.html
2020-01-23 09:28:18 +01:00
Jingyu Zhou 6ddf73e26a Remove code introduced when resolving merge conflicts 2020-01-22 21:23:38 -08:00
Jingyu Zhou 39fbacbc4f Address review comments 2020-01-22 19:43:40 -08:00
Jingyu Zhou acebfdc67b Restore storage queue limit to 0 in consistency check
The storage queue is no longer going to be a problem failing tests. Now the
backup worker life cycle is tied with backup. So consistency check only happens
after the backup workload is done. Thus, we no longer need to save backup
progress when consistency check is running.
2020-01-22 19:43:40 -08:00
Jingyu Zhou c6c39ca99d Update better master exist with backup workers
During recruitment, if there is no desired log router count, use tlog size
instead, because the number of backup workers has to be larger than 0.
2020-01-22 19:43:40 -08:00
Jingyu Zhou 8b67a89eed More review comments fixed. 2020-01-22 19:42:13 -08:00
Jingyu Zhou 1eaea91cb3 Address review comments 2020-01-22 19:42:13 -08:00
Jingyu Zhou 1311fec45a Add an option to get minKnownCommittedVersion from Proxies
The backup worker needs to use this version for popping when running in a NOOP
mode. This option is added to GetReadVersionRequest and proxies will send back
minKnownCommittedVersion if the option is set.

Also add a couple of knobs for backup workers.
2020-01-22 19:42:13 -08:00
Jingyu Zhou 7989f3f015 Add NOOP to backup worker
The backup worker just blindly pop tags if the "backupStartedKey" is not set.
Note the commit version from TLog cannot be used as the pop version, because
for a single region, during a recovery the log router tags are used to recover
mutations. The backup worker can potentially pop mutations that are needed for
recovery, causing consistency errors. So the solution for now is to use commit
version - 5,000,000, which is a version guaranteed to be persisted on all
replicas.
2020-01-22 19:42:13 -08:00
Jingyu Zhou c08a192c75 Add a backup start key
If the backup key is not set, do not recruit backup workers for old epoches.
2020-01-22 19:42:13 -08:00
Jingyu Zhou e14246ac16 Add more information for trace events 2020-01-22 19:42:13 -08:00
Jingyu Zhou 4bed33031f Set backup worker start version to be savedVersion + 1
If no progress found, start version is set to epochBegin. So the start version
is the one after the last saved (or from last epoch's saved) version.
2020-01-22 19:42:13 -08:00
Jingyu Zhou dcd0a46bc6 Fix a rare remote recovery bug
This bug was introduced when I added log router tags unconditionally to any
configurations. In newEpoch(), the wait for remote recovery is conditioned on
"logRouterTags == 0", which always becomes false. Thus remote recovery was not
performed and remote TLogs won't copy data from previous epoch's TLogs
(previous epoch is a single region configuration). As a result, storage servers
cannot peek/get the data, and won't pop tags. Thus, waitForFullReplication()
became stuck and eventually test timeout.
2020-01-22 19:42:13 -08:00
Jingyu Zhou 56a2c37071 Recruit backup workers for single region
Enable log router tags for single region, which are popped by backup workers.
Need to add noop for backup workers if there is no active backups.
2020-01-22 19:42:13 -08:00
Jingyu Zhou 0e5f5b50f0 Remove unused backup worker knobs 2020-01-22 19:38:46 -08:00
Jingyu Zhou 60f360c954 Log oldest backup epoch in the backup worker 2020-01-22 19:38:46 -08:00
Jingyu Zhou 690e93145e Fix some comments 2020-01-22 19:38:46 -08:00
Jingyu Zhou 06fb45f32a FileConverter skips mutation files without tag ID
Fileconverter doesn't know the format of old mutation logs.
2020-01-22 19:38:46 -08:00