56ddde4d58
Added VectorRef::clear
...
This allows us to avoid unnecessary rellocations
2020-06-21 17:58:35 -07:00
57fbc1a074
Added VectorRef::emplace_back
...
Also added VectorRef::emplace_back_deep.
2020-06-19 23:02:24 -07:00
327cc31e35
Revert "Request tracing"
2020-06-16 12:32:42 -07:00
80334351c7
Merge pull request #3329 from sfc-gh-mpilman/features/flatbuffers-debugtx
...
Request tracing
2020-06-12 14:50:19 -07:00
ae6b716820
addressed more review comments
2020-06-11 09:58:41 -07:00
10e094c6d7
Address review comments
2020-06-11 15:43:31 +00:00
4ab3441a95
Merge remote-tracking branch 'origin/master' into features/flatbuffers-debugtx
2020-06-10 15:31:29 -07:00
99bf993815
Replace BOOST_NOEXCEPT with noexcept
2020-06-09 22:39:19 -07:00
09c136e434
Framework for transaction tracing
2020-06-08 16:09:37 -07:00
3278222e2c
Merge pull request #3077 from tclinken/reenable-wclass-memaccess
...
Fix segmentation fault and reenable -Wclass-memaccess
2020-06-08 09:42:35 -07:00
8a98ea3320
Changed non_flow_ref to flow_ref for clarity.
...
Also updated comment to explain that std::is_trivially_copyable
cannot be used to determine whether or not a type is a flow ref.
2020-05-23 22:57:41 -07:00
131534e4a1
Added =default for Standalone constructors for clarity
2020-05-20 10:28:33 -07:00
02307ba7b6
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/DatabaseContext.h
2020-05-09 07:50:29 -07:00
5e1c43f87b
replace double invocations of operator<() with three way compare().
2020-05-07 12:08:29 -07:00
b1055a8501
Merge branch 'master' into transaction-tagging
2020-05-05 16:03:39 -07:00
8125096e9b
Revert "reduce the number of calls to operator< made by lower_bound and upper…"
2020-05-05 08:17:10 -07:00
4ae356b237
Merge pull request #2882 from sears/lb_ub_perf
...
reduce the number of calls to operator< made by lower_bound and upper…
2020-05-04 15:46:22 -07:00
b68980d686
Merge pull request #2028 from negoyal/cache_storageq_results
...
Cache storageq results
2020-05-04 11:27:02 -07:00
d0d859c313
Fix segmentation fault in VectorRef<T>::reallocate
2020-05-02 20:44:28 -07:00
88fa9bdb63
Unrevert "Enable -Wclass-memaccess and fix warnings"
2020-05-02 20:43:50 -07:00
66228343f1
Merge branch 'master' into transaction-tagging
2020-04-30 08:12:03 -07:00
519ac70a2a
Revert "Enable -Wclass-memaccess and fix warnings"
2020-04-29 15:51:29 -07:00
6ada5359b8
Merge branch 'master' into transaction-tagging
2020-04-29 14:27:21 -07:00
6156042f67
Merge pull request #3017 from tclinken/enable-wclass-memaccess
...
Enable -Wclass-memaccess and fix warnings
2020-04-29 10:38:59 -07:00
dcdda88794
replace double invocations of operator<() with three way compare().
2020-04-29 10:10:30 -07:00
41c517a5dd
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2020-04-27 13:05:24 -07:00
d654f75c46
Move instead of copy in VectorRef::reallocate
2020-04-25 12:24:06 -07:00
0efce542c8
Removed outdated comments
2020-04-23 15:27:11 -07:00
b1f525583a
Added -Wclass-memaccess compiler option and fixed warnings
2020-04-22 21:53:42 -07:00
2fa7d485f5
Merge branch 'master' into cache_storageq_results
2020-04-21 17:28:17 -07:00
29b2c2f3aa
Add hash to StringRef. Use unordered maps for storing tags. Create some helpful typedefs.
2020-04-10 12:54:59 -07:00
872877b221
Added StringRef::copyTo(), a prettier way to memcpy a StringRef somewhere and with a more useful return value.
2020-04-08 03:23:46 -07:00
6bce67ca75
FastRestore:Apply clang-format
2020-04-01 21:27:54 -07:00
1eea388278
Add unit test for splitMutation test file
...
Comment out debug trace as well.
2020-04-01 21:27:18 -07:00
7847d70e3a
StringRef:toString:Handle empty StringRef
2020-03-31 22:14:05 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
87ec36dc4c
Poison arena-internal buffers
2020-03-02 14:14:00 -08:00
f29d6c3f67
Move implementation of ArenaBlock members to Arena.cpp
2020-02-28 12:33:57 -08:00
087c6fa33d
Merge branch 'master' into feature-redwood
2020-02-26 12:25:04 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
f1ec780b31
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
2020-02-21 17:43:11 -08:00
0c7c815396
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/CMakeLists.txt
2020-02-12 16:12:57 -08:00
93e3e36d52
Changed RedwoodRecordRef::compare() to include value and updated VersionedBTree to adapt to this change. This fixes an (uncommitted) bug where DeltaTree inserts of a record matching a deleted record except for value would simply unhide the deleted record. For DeltaTree delete/insert sequences to work correctly compare() must only return 0 when the records are fully equivalent.
2020-02-12 01:18:35 -08:00
df2b0452b4
Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef.
2020-02-06 13:19:24 -08:00
3bf971ba8b
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/storageserver.actor.cpp
2019-12-12 07:13:12 -08:00
46d10dc7dc
Fix "null passed as argument declared not null"
...
Fix several such reports from ubsan
E.g.
/Users/anoyes/workspace/foundationdb/flow/Arena.h:794:16: runtime error: null pointer passed as argument 1, which is declared to never be null
2019-12-03 14:46:53 -08:00
3a9fd29d3c
Initialize memory for Optional and ErrorOr
...
This does _not_ fix any potential uses of uninitialized memory. Without
this change, gcc issues false-positive -Wuninitialized warnings
I'm hoping this does not have a noticeable impact on performance
2019-11-26 11:31:30 -08:00
b7b5d2ead3
Remove several nonsensical const uses
...
These seem to be all the ones that clang's -Wignored-qualifiers
complains about
2019-10-26 14:30:34 -07:00
042fb62771
Code cleanup
2019-09-04 16:44:19 -07:00
468cffa5d0
More cleanup around const versions of the VectorRef operators.
...
This PR changes the readRange() to cache the PTree results during the first traversal.
So that we avoid the second traversal during the merge between PTree data and Storage engine data.
I microbenchmarked the VersionedMap standalone. i.e. the set and clear-range mutations were
performed solely at the in-memory storage queue. No other FDB components were involved
in this test. And hence the numbers presented here are the best case numbers.
Test setup:
- 100M mutations: about 5% clearRange and 95% set mutations
- 100M rangeReads
- Keys/Values generated using deterministicRandom()
- Numbers presented below are for two setups, one with a single version generated for all set of mutations
and another where a new version is generated for each mutation (i.e. it's an extreme version test)
Single Versioned Extreme Versioned
Time to regular readRange 175.15 202.365
Time to readRange with cached results 158.798 184.423
2019-09-04 15:39:21 -07:00
476641a087
Merge pull request #1929 from jzhou77/fix-warning
...
Fix compiler warnings
2019-08-01 11:15:41 -07:00
3a0514851a
Fix a signed and unsigned mismatch
2019-07-31 17:01:23 -07:00
e8a9931dbe
Merge pull request #1918 from atn34/at-what-cost
...
Avoid memcpy for small types
2019-07-31 11:39:38 -07:00
b5d2234a13
Merge branch 'release-6.1' into merge-release-6.1-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbserver/MoveKeys.actor.cpp
# flow/FastAlloc.h
# versions.target
2019-07-30 16:23:42 -07:00
9f32edf4df
Avoid memcpy for small types
...
This is undefined behavior, since it's potentially a misaligned access.
But it's _probably_ not worse than the status quo
2019-07-29 17:11:45 -07:00
0bfd474e21
Fix huge arena tracking thread-safety issues. Fixes #1846 .
2019-07-16 15:22:19 -07:00
54416f46fd
Pass type as param to VectorRef instead of bool
2019-07-15 15:08:49 -07:00
6c6a1ca8f4
Expose serialization context too all traits
2019-07-15 12:58:31 -07:00
984b603c33
Avoid saving each item twice
2019-07-12 11:20:01 -07:00
74f992b2cc
fixed wrong assertion
2019-07-11 23:03:31 -07:00
b68f2d925f
Serialize range result to string for speed
2019-07-11 23:03:31 -07:00
f8584d0df8
Implement new dynamic_size_traits
2019-07-11 23:03:31 -07:00
e2ed56fa56
Convert ownedPtr to unownedPtr for IReplicationPolicy
...
Remove WriteRawMemory feature
Remove deserialization_done
2019-07-05 14:07:02 -07:00
f1f1081202
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-06-24 20:17:49 -07:00
362c2bf1e6
improved the cpu efficiency of printable
2019-05-29 14:55:45 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
ebc96a7e0e
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-05-21 23:49:27 -07:00
0713e06efc
Started to work on Windows
2019-05-13 14:15:23 -07:00
42385c2f81
Fixed issues introduced during rebase
2019-05-13 14:15:23 -07:00
186bff4ae5
removed unnecessary assert and added test
2019-05-13 14:15:23 -07:00
f5fa3a65b4
some more fixes
2019-05-13 14:15:23 -07:00
69fa3d3903
fixed compilation issues after rebase
2019-05-13 14:15:23 -07:00
6afce01744
Implementation complete (not yet working)
2019-05-13 14:15:22 -07:00
fe81454ec2
basic functionality for object serializer
...
This commit includes:
- The flatbuffers implementation
- A draft on how it should be used for network messages
- A serializer that can be used independently
What is missing:
- All root objects will need a file identifier
- Many special classes can not be serialized yet as the
corresponding traits are not yet implemented
- Object serialization can not yet be turned on (this will
need a network option)
2019-05-13 14:15:22 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
22499666d0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/LogRouter.actor.cpp
# flow/Trace.cpp
# versions.target
2019-05-08 18:19:35 -07:00
aefd68e1e7
The memory tracking trace events can crash if the memory that is being allocated is coming from a trace event itself. Specifically TDMetrics within trace events uses fast allocated memory. Trace events are supposed to be short lived, so this commits adds a global count of outstanding trace events, and disables memory tracking when a trace event exists.
2019-05-06 17:41:32 -07:00
d01cbf3455
Addressed code review comments
2019-04-05 13:12:20 -07:00
4287b1d2a1
resolved minor merge issues
2019-04-05 13:12:19 -07:00
ea67b742c7
Implemented Traceable for printable types
2019-04-05 13:12:19 -07:00
c008e16c81
Defer formatting in traces to make them cheaper
...
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:
- TraceEvent::detail now takes a c-string instead of std::string for
literals. This prevents unnecessary allocations if the trace is not
going to be printed in the first place (for example for SevDebug).
Before that `detail` expected a `std::string` as key, which mean that
any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
specialized for any type that needs to be printed. The actual
formatting will be deferred to after the `enabled` check. This
provides two benefits: (1) if a TraceEvent is disabled, we don't pay
for the formatting and (2) TraceEvent can trace types that it doesn't
know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
calls, a call to detail will only introduce a if-branch which is much
cheaper than a function call.
2019-04-05 13:12:19 -07:00
d670b74d69
prevent trace event spam by combining huge arena samples
2019-03-30 13:36:13 -07:00
d5c8b6b083
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
# flow/flow.h
2019-03-27 13:37:15 -07:00
1fc6937802
changed NetworkAddressList to at most two addresses for performance
2019-03-23 17:54:46 -07:00
70ac5ffda0
added random logging for huge arenas
2019-03-20 11:20:47 -07:00
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
c304397166
fixed a memory leak when allocating memory > 4K and < 8K
2019-03-10 21:13:00 -07:00
f593562743
use 8KB pages in Arena
2019-03-08 15:03:17 -08:00
c6a65389ae
Remove noexcept macro and replace with BOOST_NOEXCEPT.
...
BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a
way that is correctly maintained over time.
2019-03-05 22:06:12 -08:00
87d0c5bae0
Bug/usability fix: The output URLs of fdbbackup list were meant to be directly usable for backup management operations but they were missing the bucket URL parameter.
2019-03-05 21:14:21 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
2570b37e6e
Add function to read pages from RawDiskQueue_TwoFiles
2019-02-12 23:43:10 -08:00
6a4d87802b
Replace & operator with variadic function
2018-12-28 11:33:42 -08:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
2fc86c5ff3
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbrpc/AsyncFileCached.actor.h
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-09-20 03:39:55 -07:00
309159ea23
WriteValue() no longer uses format() which creates temporary strings, instead if writes directly into allocated space in the target buffer using snprintf().
2018-09-10 00:44:09 -07:00
383cac4991
This checkin is an attempt to support long keys by placing oversize key suffixes (as much as will fit) into a page and forcing the prefix bytes that do not fit to be part of the parent's page boundary leading to this page. This strategy is being abandoned but there are some valuable refactorings and debugging enhancements that are still in the right direction so they are being committed.
2018-08-28 13:46:14 -07:00
ec1dfe8ae1
Some code cleanup / renaming of things for clarity. Improved StringRef hex string output to omit a middle segment when there is an exceeded length limit.
2018-08-16 16:22:10 -07:00
9d391498e8
Refactored how key/value memory is held in PrefixTree and VersionedBtree, eliminated many unnecessarily copies of large strings. PrefixTree tests pass but btree is still broken, just committing this because it's a large change set.
2018-07-23 03:09:13 -07:00
sfc-gh-tclinkenbeard