Commit Graph

4418 Commits

Author SHA1 Message Date
Josh Slocum ed9f95b299 simplifying blob manager recovery by always treating the DB granule mapping as the source of truth, and persisting all boundary changes there BEFORE updating workers 2022-02-25 14:46:14 -06:00
Josh Slocum e5b4fb3d80 Adding explicit popVersion to change feed reply, like tlogs have 2022-02-24 15:20:18 -06:00
Josh Slocum 1de4fb1b9e Adding explicit stopVersion instead of stopped status to change feed durable data, several other minor fixes 2022-02-24 11:02:15 -06:00
Josh Slocum a7a3dccba5 TSS Mismatch Quarantine fix 2022-02-17 19:16:07 -06:00
Josh Slocum ef25e4fdfb Sending empty version back immediately to establish stream connection and acknowledgement queue for change feed stream requests 2022-02-17 18:10:17 -06:00
Josh Slocum 38a75a8b89 Merge branch 'main' into blob_integration 2022-02-17 17:47:38 -06:00
Zhe Wu 9da735c38e Batch empty peek reply 2022-02-16 15:28:56 -08:00
Josh Slocum 26480c4705 change feed popping fixes 2022-02-15 15:12:23 -06:00
Vaidas Gasiunas 092b5cee4b MVC2.0: Rollback added code 2022-02-14 13:50:42 -08:00
Trevor Clinkenbeard ef68e6fe0d
Merge pull request #6353 from sfc-gh-ljoswiak/fixes/dynamic-knobs
Fix dynamic knobs correctness issues
2022-02-10 22:13:02 -08:00
Zhe Wang d684508540 Add RatekeeperLimitReasonDetails traceevent for RK 2022-02-10 13:59:47 -08:00
Josh Slocum c8cd8c0622 Adding request timeout for blob worker 2022-02-09 15:49:33 -06:00
Lukas Joswiak cdc1549282 Fix client timeout errors 2022-02-09 13:43:33 -08:00
Lukas Joswiak dc4850e722 Don't specify types in format 2022-02-09 13:43:32 -08:00
Lukas Joswiak 990e215a8d Fix formatting
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2022-02-09 13:43:32 -08:00
Lukas Joswiak d5a562e6b8 Fix dynamic knobs correctness issues 2022-02-09 13:43:32 -08:00
Josh Slocum f4c235f134 propagating change_feed_popped correctly 2022-02-09 09:34:13 -06:00
Bala Namasivayam 9fe7a99e11 Make AWS V4 header default 2022-02-07 17:53:05 -08:00
Bala Namasivayam 208b5de572 Adding comment and address code style formatting. 2022-02-07 17:53:05 -08:00
Bala Namasivayam 0f36157c2c Clang format files 2022-02-07 17:53:05 -08:00
Bala Namasivayam a420098fc5 Support AWS v4 header for s3 backup and restore 2022-02-07 17:53:05 -08:00
Yi Wu cda68a0e4d Support xxhash3 for checksuming DiskQueue for TLogs 2022-02-07 13:32:52 -08:00
sfc-gh-tclinkenbeard 5b8f4d9be1 Move SimpleIni.h to fdbclient 2022-02-07 13:31:04 -08:00
Josh Slocum 9b3f6fe452 More bugs in streaming split 2022-02-07 13:25:14 -06:00
Josh Slocum 88b1593681 Removing incorrect assert 2022-02-07 11:56:14 -06:00
Josh Slocum 88cab7fb67 More change feed fetching improvements and optimizations 2022-02-07 11:19:51 -06:00
Josh Slocum c48ca9430d Fixes for streaming split keys 2022-02-07 10:36:22 -06:00
Josh Slocum 809ec24184 More change feed buffer tuning 2022-02-07 08:19:19 -06:00
Josh Slocum 1736eabd33 fixes for split metrics stream 2022-02-04 18:24:55 -06:00
Josh Slocum 41a274ed9c Fixing bug and refactoring iterative range splitting 2022-02-04 17:10:46 -06:00
Xiaoxi Wang 6dc5921575
createdTime based storage wiggler (#6219)
* add storagemetadata

* add StorageWiggler;

* fix serverMetadataKey bug

* add metadata tracker in storage tracker

* finish StorageWiggler

* update next storage ID

* change pid to server id

* write metadata when seed SS

* add status json fields

* remove pid based ppw iteration

* fix time expression

* fix tss metadata nonexistence; fix transaction retry when retrieving metadata

* fix checkMetadata bug when store type is wrong

* fix remove storage status json

* format code

* refactor updateNextWigglingStoragePID

* seperate storage metadata tracker and store type tracker

* rename pid

* wiggler stats

* fix completion between waitServerListChange and storageRecruiter

* solve review comments

* rename system key

* fix database lock timeout by adding lock_aware

* format code

* status json

* resolve code format/naming comments

* delete expireNow; change PerpetualStorageWiggleID's value to KeyBackedObjectMap<UID, StorageWiggleValue>

* fix omit start rount

* format code

* status json reset

* solve status json format

* improve status json latency; replace binarywriter/reader to objectwriter/reader; refactor storagewigglerstats transactions

* status timestamp
2022-02-04 15:04:30 -08:00
Josh Slocum ddfc301d74 Improving memory footprint of change feeds and making it configurable 2022-02-04 16:41:25 -06:00
Markus Pilman 4414df1914
Merge pull request #6279 from sfc-gh-rjenkins/threadname-issue-6064
Make better use of thread names.
2022-02-04 08:46:23 -07:00
Josh Slocum a974773676 Handled change feed getting moved away or cancelled during move 2022-02-04 09:33:04 -06:00
Josh Slocum 2592c3f0ae Implemented less efficient version of reply stream closing 2022-02-03 20:07:49 -06:00
Andrew Noyes 02fcb1a239
Deprecate transaction_include_port_in_address (#6332)
* Deprecate transaction_include_port_in_address

Update the description to start with Deprecated, so that it will be
deprecated in the generated java files.

* Update generated.go
2022-02-03 14:36:01 -08:00
Josh Slocum 9c8afdf35c Debugged and fixed stuck change feed issue 2022-02-02 15:15:54 -06:00
Josh Slocum d58b0325b3 Added write-hot granule splitting and splitting based on write rate 2022-02-02 09:45:18 -06:00
Ray Jenkins dd45805312
Merge branch 'apple:main' into threadname-issue-6064 2022-02-01 17:40:07 -06:00
Josh Slocum d0113a6776 Added mechanism for blob manager to poll blob workers for their granule assignments, and used that to improve manager recovery 2022-01-31 19:59:41 -06:00
Ata E Husain Bohra 591ef57857
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314)
* Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor

Major changes proposed are:
1. Refactor StreamCipher code to enable instantiation of
   multiple encryption keys. However, code still retains
   a globalEncryption key semantics used in Backup file
   encryption usecase.
2. Enhance StreamCipher to provide HMAC signature digest
   generation. Further, the class implements HMAC encryption
   key derivation function.
3. Upgrade StreamCipher to use AES 256 GCM mode from currently
   supported AES 128 GCM mode.
   Note: The code changes the encryption key size, however, the
         feature is NOT currently in use, hence, should be OK.
3. Add EncryptionOps validation and benchmark toml supported
   workload, it does the following:
   a. Allow user to configure encrypt-decrypt of a fixed size
      buffer or variable size buffer [100, 512K]
   b. Allow user to configure number of interactions of the runs,
      in each iteration: generate random data, derive an encryption
      key using HMAC SHA256 method, encrypt data and
      then decrypt data. It collects following metrics:
    i) time taken to derive encryption key.
    ii) time taken to encrypt the buffer.
    iii) time taken to decrypt the buffer.
    iv) total bytes encrypted and/or decrypted
   c. Along with stats it basic basic validations on the encrypted
      and decrypted buffer
   d. On completion for test, records the above mentioned metrics
      in trace files.
2022-01-31 19:52:44 -06:00
A.J. Beamon 8c00ba6716
Merge pull request #6324 from sfc-gh-ajbeamon/high-contention-allocator
Rename high contention allocator implementation in fdbclient
2022-01-31 15:51:03 -08:00
A.J. Beamon 6affc58e97 Rename high contention allocator implementation in fdbclient 2022-01-31 14:25:38 -08:00
Markus Pilman 60cbce72c2
Merge pull request #6317 from sfc-gh-ajbeamon/high-contention-allocator
Added a generic high contention allocator implementation to fdbclient
2022-01-31 14:50:51 -07:00
A.J. Beamon d7819f8387
Merge pull request #6304 from sfc-gh-ajbeamon/reset-copy-trstate
Copy transaction state during reset
2022-01-31 09:11:04 -08:00
Vaidas Gasiunas 95074c2b0e
Merge pull request #6313 from sfc-gh-ajbeamon/split-management-api
Split the management API into two parts
2022-01-31 13:22:07 +01:00
A.J. Beamon 027fe80594 Added a generic high contention allocator implementation to fdbclient. This is an adapted version of the flow bindings HCA implementation. 2022-01-28 15:34:30 -08:00
A.J. Beamon 89f4ed7481
Merge pull request #6296 from sfc-gh-ajbeamon/mark-external-system-transactions
Mark various externally exposed system transactions as read/access system keys
2022-01-28 14:16:40 -08:00
Renxuan Wang f9f3735f73 Add resolveHostnamesBlocking() in ConnectionString and IClusterConnectionRecord.
Also, combine IClusterConnectionRecord::getConnectionString() and IClusterConnectionRecord::getMutableConnectionString() to IClusterConnectionRecord::getConnectionString(), and rename setConnectionString() to setAndPersistConnectionString().
2022-01-28 12:20:41 -08:00
A.J. Beamon 2eda48416c Undo some changes from LOCK_AWARE to READ_LOCK_AWARE. Apparently we have some dependencies in composed functions where a read-only function is setting lock awareness to the benefit of another function that does writes. 2022-01-28 10:55:02 -08:00