41c790b299
Merge remote-tracking branch 'origin/master' into config-db
2021-06-10 22:31:23 -07:00
c6d96c4124
Added TSS Quarantine and fixed TSS mapping bug in CommitProxy
2021-06-09 14:42:50 +00:00
866f536983
Merge pull request #4888 from sfc-gh-tclinkenbeard/remove-fdbserver-includes
...
Remove fdbserver includes from fdbclient
2021-06-07 10:22:13 -07:00
b3e4f182ef
TSS Mapping Change
2021-06-02 17:30:09 +00:00
a775f92fca
Merge remote-tracking branch 'origin/master' into config-db
2021-06-01 15:39:34 -07:00
1a40c60674
Move RestoreWorkerInterface into fdbserver
2021-05-30 15:02:33 -07:00
594e8944ae
Move RestoreWorkerInterface into fdbserver
2021-05-30 11:51:47 -07:00
9515e8d44f
Add special configuration database keys to SystemData.h
2021-05-29 11:48:47 -07:00
4257ac2b4d
More TSS Changes/Fixes
2021-05-25 20:37:48 +00:00
ce82c9653e
Testing Storage Server implementation
2021-05-25 20:28:50 +00:00
6165af81fa
Merge branch 'apple:master' into feature-range-feed
2021-05-21 15:21:34 -07:00
a57061a5ed
change UID to PID meaning Process ID
2021-05-20 00:06:53 +00:00
6065d247f8
fix perpetualStorageWiggleKey
2021-05-17 20:59:38 +00:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
78697003ea
more formatting
2021-04-30 10:51:35 -07:00
d652287bae
Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
...
# Conflicts:
# fdbclient/DatabaseContext.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/StorageServerInterface.h
# fdbserver/storageserver.actor.cpp
2021-04-30 10:29:42 -07:00
6de28dd916
clang-format
2021-04-14 10:56:33 -07:00
c3f68831af
Move existing ClientDBInfo variables to global configuration
2021-04-14 10:56:32 -07:00
80c6048a01
Naming fixes
2021-04-14 10:56:32 -07:00
f1415412f1
Add global configuration framework implementation
2021-04-14 10:56:32 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
2bb4f2e59f
Merge branch 'release-6.3-pre-format' into master-format
...
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
0c72e5a9cb
fixed name of function
2021-03-04 20:22:42 -08:00
bc757f3e4b
basic framework for range feed support
2021-03-04 19:14:54 -08:00
37d9e975e9
Fix multiple compiler warnings
2021-03-03 10:18:03 -07:00
ff7d306b09
Merge branch 'release-6.3' into anoyes/merge-6.3-to-master
...
Include conflict markers for now. Will resolve.
2021-01-15 18:04:09 +00:00
53d23d95d8
initial commit in adding more documentation to special keys
2021-01-04 15:26:48 -05:00
bbb56e4089
Merge branch 'release-6.2' of https://github.com/apple/foundationdb into release-6.3
2020-12-23 14:26:59 -08:00
ac02329d7d
Added a command in fdbcli to allow user to manually trigger the detailed teams info loggings in data distributor
2020-12-03 14:42:30 -08:00
22996284c7
added changes to allow writing of last epoch end version to special keys when performing recovery due to snapshot
2020-09-11 15:00:11 -04:00
d334b6484e
attempt to write to system keys with snapshot
2020-09-02 15:17:54 -04:00
9c130b5ea2
added new system keys
2020-09-02 14:13:58 -04:00
12edadd059
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# fdbclient/Knobs.cpp
# fdbclient/MasterProxyInterface.h
# fdbrpc/simulator.h
# fdbserver/MasterProxyServer.actor.cpp
# tests/fast/CycleAndLock.txt
# tests/fast/TxnStateStoreCycleTest.txt
# tests/fast/VersionStamp.txt
# tests/slow/ParallelRestoreOldBackupApiCorrectnessAtomicRestore.txt
# tests/slow/ParallelRestoreOldBackupCorrectnessCycle.txt
# versions.target
2020-08-31 19:33:34 -07:00
29eec30183
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# build/Dockerfile
# build/Dockerfile.devel
# documentation/sphinx/source/downloads.rst
# fdbserver/Knobs.cpp
# fdbserver/LogSystem.h
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WaitFailure.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
# packaging/msi/FDBInstaller.wxs
2020-08-31 01:10:29 -07:00
9cf3041393
Add read conflict for metadata txs that aren't lock-aware
2020-08-20 02:52:37 +00:00
cf13e00a8f
Merge remote-tracking branch 'origin/release-6.3' into fdb_cache_wo_allocator
2020-06-01 17:38:31 -07:00
138569a489
another compile fix
2020-05-22 17:19:46 -07:00
72ce997d22
explicitly versioned every key in systemData, so we only will update the associated protocolVersion when the serialization actually changes
2020-05-22 16:35:01 -07:00
cc9dd8ac72
Merge pull request #3214 from etschannen/release-6.3
...
Do not use tags in the keyServers map in the initial version of 6.3
2020-05-20 17:12:43 -07:00
298b94a044
clang-format the change
2020-05-19 20:51:02 -07:00
a6aff22e65
Update the special key prefix for dd-stats to \xff\xff/data_distribution_stats
2020-05-19 20:47:52 -07:00
93ca130602
Delete the unnecessay endKey field in DDMetricsRef, since each begin key is the previous end key
2020-05-19 13:32:42 -07:00
5013bc9106
fixed client compile issues; reduced the size of storage server interface
2020-05-18 17:20:15 -07:00
00dce3ca1d
clang-format
2020-05-18 14:23:17 -07:00
117ed14321
Add dd-stats zerobytes const to systemData
2020-05-18 11:52:36 -07:00
f515e2c9d0
Add systemdata for dd-stats
2020-05-18 10:36:34 -07:00
1e0c10e9e8
control tag encoding of keyServersValue using a knob
2020-05-13 13:29:13 -07:00
9d5e37f1a8
optimized decodeKeyServersValue on the proxy when loading all key ranges on startup
2020-05-09 16:54:35 -07:00
02307ba7b6
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/DatabaseContext.h
2020-05-09 07:50:29 -07:00
ee0be88b99
consistency check can use a UIDtoTag map at a different version from the keyservers which could result in missing entries
2020-05-08 17:45:06 -07:00
f0f52fb2be
Merge branch 'master' into feature-small-endpoint
...
# Conflicts:
# fdbclient/StorageServerInterface.h
2020-05-08 16:37:35 -07:00
749fcd13b0
Merge branch 'master' into fdb_cache_wo_allocator
2020-05-08 16:23:29 -07:00
8166e1af8b
Storage server tags cannot change once used in the keyServer location map
2020-05-08 16:17:10 -07:00
dd033736ed
Merge branch 'master' into fdb_cache_subfeature2
2020-05-04 17:29:43 -07:00
f1d9c0cb97
Refactor/cleanup
2020-04-30 00:30:37 +00:00
fdd6581167
Add write conflict range
2020-04-30 00:30:37 +00:00
41c517a5dd
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2020-04-27 13:05:24 -07:00
35c18ac60a
Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters.
2020-04-23 20:50:40 -07:00
dfec896438
Enforce a throttle limit. Don't count transaction tags on RK if the proxy has updated us in a while.
2020-04-17 11:48:02 -07:00
78d48a0dad
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/Knobs.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/fdbclient.vcxproj
# fdbserver/MasterProxyServer.actor.cpp
2020-04-17 09:23:18 -07:00
b011efa8df
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-16 10:58:04 -07:00
7a9bcf8222
Review comments.
2020-04-14 17:45:34 -07:00
b85dc16c6d
Merge branch 'master' into fdb_cache_subfeature2
2020-04-14 17:07:41 -07:00
65620b00f3
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-13 14:21:07 -07:00
c8ab69cce6
Fix incorrect if/ASSERT logic from last minute cleanups.
...
My test code serialized UIDs and Tags, so that I could compare and
verify I got the same thing, and I undid this right before committing,
but forgot to also change the `if` to not rely on UIDs being first also.
2020-04-12 19:38:51 -07:00
db071256db
Merge remote-tracking branch 'upstream/master' into reduce-txn-state-store-size
2020-04-12 18:06:10 -07:00
ebeca10bce
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-09 16:55:56 -07:00
34cddd675c
clang-format
2020-04-08 14:33:41 -07:00
4a9658d6b8
fix issues according to andrew's comments
2020-04-08 13:38:12 -07:00
20b2984841
fix issues according to andrew's comments
2020-04-08 12:43:25 -07:00
36da61dd9c
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
2020-04-07 21:12:14 -07:00
60407bdee3
Use LiteralStringRef for backup paused key
2020-04-07 16:02:25 -07:00
9fb3fb9d82
Add pause/resume for new backups
...
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
2020-04-06 14:29:46 -07:00
6078fd1b18
Convert UID to Tag in keyServers to reduce txnStateStore size
2020-04-05 14:30:09 -07:00
b76baa9cc2
Update conflicting keys related constants
2020-04-03 16:08:57 -07:00
2336f073ad
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-03 15:24:14 -07:00
a0c8946f31
Merge branch 'master' into fdb_cache_subfeature2
2020-04-02 12:27:04 -07:00
81dd97d27f
Change special keyspace to \xff\xff, \xff\xff\xff
2020-03-31 19:06:45 -07:00
60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
...
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
113d0fb48b
Remove incorrect assertion
2020-03-27 12:13:30 -07:00
289487559d
Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
...
This reverts commit 804fe1b22e
2020-03-24 18:11:15 -07:00
804fe1b22e
Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
...
This reverts commit 648dc4a933487d131b38
2020-03-19 21:34:28 -07:00
0094293d50
add const vars
2020-03-11 23:11:49 -07:00
d1c56d3b57
add constant KeyRefs in SystemData
2020-03-11 12:25:50 -07:00
7a76e9556d
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-04 11:24:39 -08:00
cd949eca71
Merge branch 'master' into fdb_cache_subfeature2
2020-02-26 11:22:08 -08:00
a8c198ddbc
More compilation fixes.
2020-02-24 17:40:32 -08:00
735e7c71f2
Add the cachetest back in.
2020-02-24 17:26:20 -08:00
a12a161fb3
Merge branch 'master' into mengxu/fast-restore-pipeline-PR
2020-02-18 14:49:52 -08:00
c603b20e7e
FastRestore:Resolve review comments
2020-02-18 14:08:27 -08:00
28768715f2
Custom bookkeeping allocator for PTree. Simple Cache workload.
2020-02-14 11:42:47 -08:00
85cc35e81e
Merge branch 'master' into HEAD
2020-02-05 14:59:55 -08:00
5a602f58e8
Start backup with a wait on all backup workers running
...
This wait is to make sure that backup workers are already saving mutations so
that no mutations are missed. The idea is that the CLI sets a "backupStartedKey"
in the database and waits for allWorkerStarted() key of the backup to be set.
Backup workers monitor the changes to the "backupStartedKey" and start logging
mutations. Additionally, backup worker for Tag(-2,0) monitors all other workers
have started (checking their saved progress version is larger than the backup's
start version), and then sets the allWorkerStarted() key for the backup.
2020-01-31 19:29:09 -08:00
141609e80a
FastRestore:Improve code style and fix typos
2020-01-27 18:13:14 -08:00
16f9ec45bd
Merge branch 'master' into mengxu/fast-restore-pipeline-PR
2020-01-23 20:15:21 -08:00
1eaea91cb3
Address review comments
2020-01-22 19:42:13 -08:00
c08a192c75
Add a backup start key
...
If the backup key is not set, do not recruit backup workers for old epoches.
2020-01-22 19:42:13 -08:00
19d6a889ff
Recruit backup workers for old epochs
...
If there are unfinished ranges in the old epochs, the new master will recruit
backup workers responsible for finishing these ranges. These workers remains in
the cluster until the next epoch, when it will remove itself.
2020-01-22 19:38:45 -08:00
41f0cf2bb5
Add decode function for backup progress
2020-01-22 19:38:45 -08:00
7da9f47f26
Enable pop from backup workers
...
This is still WIP as some edge cases can trigger test failure, most likely due
to not popping mutations by backup workers when epoch ends.
2020-01-22 19:38:45 -08:00
sfc-gh-tclinkenbeard