Commit Graph

345 Commits

Author SHA1 Message Date
Junhyun Shim be225acd2a Merge remote-tracking branch 'origin/main' into authz-tenant-name-to-tenant-id 2023-02-06 23:13:43 +01:00
Junhyun Shim 1afd63d7e3 Minimize the risk of TracedTooManyLines in simulation
- Disable audit logging for simulation
- Relax the max_trace_lines knob limit to reduce false positives
2023-02-06 21:50:39 +01:00
Xiaoxi Wang 53ce648325 set clearAfterTest=false in RawTenantAccessClean test 2023-02-02 13:26:56 -08:00
Nim Wijetunga 86f3665514
Handle EKP Tenant Not Found Errors (#9261)
handle EKP tenant not found errors
2023-02-01 19:15:38 -08:00
Xiaoxi Wang 8f7a98fb9d remove test timeout 2023-02-01 11:04:39 -08:00
Xiaoxi Wang d1d860a342 merge upstream/main 2023-01-27 12:15:04 -08:00
Nim Wijetunga 05a8a90830
Snapshot Backup Tenant Deletion Support (#9145)
Tenant deletion support for snapshot backups
2023-01-26 17:46:14 -08:00
Xiaoxi Wang 33bddb31c7 parse tenant id during getResolution phase 2023-01-25 09:49:04 -08:00
Andrew Noyes 218cda3cf6
Lower ASAN memory usage (#9216)
* Print an asan heap profile on OOM

* Use 32KiB stacks for boost coro

* Print 100%, 10 max contexts for asan OOM

* Lower machineCount to 30 in DataLossRecovery test

* Add asanMachineCount override to control ASAN memory usage
2023-01-24 13:04:47 -08:00
Xiaoxi Wang 5a8cac6cf6 merge upstream/main; resolve merge change 2023-01-18 19:47:52 -07:00
Nim Wijetunga 330ac71630
Tenant Deletion Support for Backup Mutation Log (#9103)
tenant deletion support for backup mutation log
2023-01-18 15:11:58 -08:00
Xiaoxi Wang 853cafcb4c check mutation in applyMetadataEffect; modify RawTenantAccess test spec 2023-01-18 11:21:03 -07:00
Hui Liu 8d8cba5745 Add correct test for blob restore with large workload 2023-01-18 09:27:48 -08:00
Xiaoxi Wang 31c4cf8de2 add TenantRawAccess test 2023-01-17 16:16:58 -07:00
He Liu 00203c8732
Validate Storage part II (#8471)
* Implemented AuditUtils.actor.cpp

Moved AuditUtils to fdbserver/

* Persist AuditStorageState.

* Passed persisted AuditStorageState test.

* Added audit_storage_error to indicate a corruption is caught.

Throw/Send audit_storage_error when there is a data corruption.

Added doAuditStorage() for resuming Audit.

* Load and resume AuditStorage when DD restarts.

* Generate audit id monotonically.

* Fixed minor issue AuditId/Type was not set.

* Adding getLatestAuditStates.

* Improved persisted errors and added AuditStorageCommand.actor.cpp for
fdbcli.

* Added `audit_storage` fdbcli command.

* fmt.

* Fixed null shared_ptr issue.

* Improve audit data.

* Change DDAuditFailed to SevWarn.

* Sev.

* set SERVE_AUDIT_STORAGE_PARALLELISM to 1.

* Moved AuditUtils* to fdbclient/.

* Added getAuditStatus fdbcli command.

* Refactor audit storage fdb cli commands.

* Added auditStorage in sim.

* Cleanup.

* Resolved comments.

* Resolved comments.

* Test disabling audit for sims.

* Cleanup.

Co-authored-by: He Liu <heliu@apple.com>
2023-01-15 21:46:14 -08:00
Xiaoxi Wang 6592ecc3a5 merge upstream/main; confliction solve 2023-01-11 09:33:21 -08:00
Xiaoxi Wang c9aa3f1458 toml file format 2023-01-10 23:13:55 -08:00
Xiaoxi Wang 3bcb55dccb
Apply suggestions from code review
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-01-10 16:57:54 -08:00
Hui Liu 4af55a274d Add data verification at the end of BlobRestoreBasic.toml 2023-01-10 13:26:36 -08:00
Xiaoxi Wang b3886806c4 modify tests configuration 2023-01-06 16:01:24 -08:00
Xiaoxi Wang e3d09a5cbd Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/tenantCheck 2023-01-04 16:16:15 -08:00
Hui Liu e3bf79cf71 Add correctness test for blob restore 2023-01-04 11:10:34 -08:00
Xiaoxi Wang 8ff6d6c7ee merge upstream/main; solve conflicts 2022-12-17 15:57:07 -08:00
Xiaoxi Wang f03b327d06 disable tenant in DR backup tests 2022-12-17 14:57:21 -08:00
Andrew Noyes dd0036f09c
Automatically clean old idempotency ids (#9039)
* Add cleanIdempotencyIds

Delete zero or more idempotency ids older than minAgeSeconds

* Automatically clean idempotency ids from first proxy

* Add test for cleaner

* Fix formatting

* Address review comments
2022-12-14 14:24:24 -08:00
A.J. Beamon b29b20c1d9
Merge pull request #9011 from sfc-gh-nwijetunga/nim/fix-encrypted-backup-workload
Fix EncryptedBackupCorrectness Workload
2022-12-13 08:42:20 -08:00
Nim Wijetunga 3a238d638a
Commit Proxy uses Encryption DB Config (#8911)
* add encryption db config

* address pr comments

* address pr comments

* add comments

* cp uses db config

* remove includes

* fix tests

* fix tests

* modify comment

* add encryption enabled method

* modify simulation

* Trigger Build

* change commit proxy encrypt mode fetching

* address pr comments

* address pr comments

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build
2022-12-12 21:23:55 -08:00
Nim Wijetunga dbf6fc0fb8 Merge branch 'main' into nim/fix-encrypted-backup-workload
* main: (68 commits)
  Fix clang 15 warning in mako
  Fix more clang 15 warnings
  Fix clang 15 compiler warnings
  metrics: Break down OTELSum into smaller chunks to avoid UDP max packet size
  metrics: Make NumberDataPoint contain a variant<int, double> of the value
  ddsketch: Fix undefined behaviour in DDSketch::getValue
  metrics: Make DDSketch buckets uint32_t but double precision
  metrics: Add OTEL metric definitions
  metrics: Add simulation testing and fix incorrect TraceEvent names - Added a background actor that listens on METRICS_EMISSION_UDP_PORT for incoming metrics (and verifies they are in the correct format) - TraceEvent details have certain requirements for naming. This commit makes a seperate name for Counter/LatencySample and its underlying IMetric to avoid those issues
  metrics: Add verifyStatsdMessage() for testing
  metrics: Avoid name conflicts in MetricCollection - In Counter constructor, make sure that we combine CounterCollection name - In IMetric constructor, add a trace event and assertion in case of collision - Rename ClusterRecoveryData CounterCollection name from master to ClusterRecoveryData
  metrics: Fix bug in LatencySample::flush() - Make sure that we check batch.statsd_message to be non-empty before adding a newline - Make sure to actually add the message to the batch in flush()
  metrics: Added runMetrics() to workerServer
  metrics: Add IMetricClient and StatsdMetric to send batches over UDP
  metrics: Add MetricsLogger loop and more knobs
  metrics: Make sure LatencyBand counters are unique
  metrics: Add knobs and make Counter, LatencySample implement flush() method
  metrics: Add flush() definitions to Counter and LatencySample
  metrics: Make LatencySample and Counter inherit from IMetric
  metrics: Initial code - Define new IMetric interface - Rename MetricUpdateBatch to MetricBatch - Move previous fields of MetricUpdateBatch into a new struct called FDBScope - Define global MetricCollection class and enMetric enumerate
  ...
2022-12-09 10:17:12 -08:00
Nim Wijetunga eb1c0e11a2 fix test 2022-12-08 15:33:43 -08:00
Chaoguang Lin 4dbfa01fbf
Add a new robustness workload for testing special keys (#8957)
* Add a new robustness workload for testing special keys

* Fix a few robustness related issues and remove duplicate tests

* Add comments
2022-12-05 14:05:26 -08:00
Nim Wijetunga 97713cadff
Update Encryption Mode DB Config Values (#8839)
* add encryption db config

* address pr comments

* address pr comments

* add comments

* add comment

* modify check

* change condition

* address pr comments

* simplify check

* address pr comments
2022-11-22 16:33:59 -08:00
Junhyun Shim 7c65ca6648 Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-17 17:48:44 +01:00
Xiaoxi Wang 6b77bcfc35 fix merge conflict upstream/main 2022-11-16 07:34:42 -08:00
Junhyun Shim 632bddd4de Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-16 10:18:34 +01:00
Steve Atherton 959f77d03d
Merge pull request #8730 from sfc-gh-xwang/fix/main/restoreStats
Fix the omit StorageWiggler.restoreStats() bug
2022-11-15 22:37:39 -08:00
Xiaoxi Wang 907d7af966 solve merge conflict upstream/main 2022-11-15 14:59:31 -08:00
Junhyun Shim 7563d59688 Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-14 12:49:07 +01:00
Nim Wijetunga a1a2839cb3
Revert "Extend Transaction Window during Restore in Simulation" (#8806)
* fix resolver bug

* trigger build
2022-11-12 15:24:30 -08:00
Xiaoxi Wang b79268326a Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/dataApi 2022-11-11 08:29:22 -08:00
Xiaoxi Wang 93daa9ebb7 Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/restoreStats 2022-11-09 16:30:41 -08:00
Nim Wijetunga f1ef9d2671
Fix Bulk Load With Tenants Workload (#8754)
* fix bulk load

* Trigger Build
2022-11-09 13:18:34 -08:00
Xiaoxi Wang 62b88a0772 wait on lock take 2022-11-09 11:34:47 -08:00
Xiaoxi Wang fd425db1cf fix rawStartMovement bugs for merge shard; change the test toml file name 2022-11-08 23:37:58 -08:00
Xiaoxi Wang 317f0e9efe finish workload PerpetualWiggleStatsWorkload 2022-11-08 08:14:21 -08:00
Junhyun Shim 66e3050062 Merge remote-tracking branch 'origin/main' into authz-security-tests 2022-11-08 13:15:37 +01:00
Xiaoxi Wang 96cf3f855b add rawMoveShard function 2022-11-07 16:47:14 -08:00
Xiaoxi Wang fa72f2ef96 fix system access option 2022-11-07 16:13:17 -08:00
Junhyun Shim b64afded69 Parallelize LeakTLogInterface and AuthzSecurity workload 2022-11-08 01:07:26 +01:00
Junhyun Shim b6e5372a68 Keep up with test framework changes for AuthzSecurity workload 2022-11-07 12:07:25 +01:00
Xiaoxi Wang 8e6a9730ea move StorageWiggleMetrics out; add workload; try to fix the restore/reset bug (not test) 2022-11-03 23:42:44 -07:00