This fixes#2880.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Co-authored-by: Russell Cohen <rcoh@amazon.com>
This PR creates new-types for `Input`, `Output`, and `Error` so that the
type system can enforce that an output isn't given to an input and vice
versa. It also removes `TypedBox` since that isn't really needed
anymore.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
The publish tool doesn't consider unpublished crates when computing the
version tree for publish. This causes errors for examples that use
shared crates.
## Description
This adds support for considering unpublished crates when publishing. It
allows unpublished crates to have unpublished dependencies but published
crates can only have unpublished crates as dev dependencies.
## Testing
- [x] build these exact changes into the docker image and test
## Checklist
<!--- If a checkbox below is not applicable, then please DELETE it
rather than leaving it unchecked -->
- [ ] I have updated `CHANGELOG.next.toml` if I made changes to the
smithy-rs codegen or runtime crates
- [ ] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR changes the default runtime mode to orchestrator for generated
clients and the AWS SDK.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
During releases, if the toolchain version has changed, we'll try to
invoke a Rust version that doesn't exist. This pins it to the stable
version.
## Description
Set the `RUSTUP_TOOLCHAIN` environment variable in the release/CI
dockerfile
## Testing
- ran a release with this image
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
solved by deploying a newer version of smithy-rs instead
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR fixes Timestream in the orchestrator implementation.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR moves all the "runtime components", pieces that are core to the
operation of the orchestrator, into a separate `RuntimeComponents` type
for the orchestrator to reference directly.
The reason for this is so that these core components cannot be changed
by interceptors while the orchestrator is executing a request.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
The previous rust stable version is required to compile older versions
of the SDK
## Testing
built the docker image locally
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR fixes the S3 `alternative-async-runtime` retry tests. The retry
backoff time was being included in the operation attempt timeout, which
I think is undesirable as it makes retry config much harder to get right
since the backoff times are not predictable (they have randomness
incorporated into them). The overall operation timeout is the only one
that needs to incorporate backoff times.
In addition, this PR also:
- Updates READMEs for the `aws-smithy-runtime-api` and
`aws-smithy-runtime` crates
- Adds top-level crate docs to describe features to `aws-smithy-runtime`
- Copies `capture_test_logs` into `aws-smithy-runtime` so that it can be
used (just about) everywhere instead of just in `aws-config`
- Adds service/operation name to the tracing `invoke` span so it's
possible to tell which operation the events are for
- Makes the `Debug` impl for `Identity` useful
- Adds a ton of trace/debug spans and events to the orchestrator
- Fixes an issue in `aws-smithy-runtime` where a huge amount of the
orchestrator tests weren't being compiled due to a removed feature flag
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
When the WebAssembly SDK example was added some months ago, we changed
the build process to make each SDK example its own Cargo workspace. This
allowed the `.cargo/config.toml` that changed the compiler target to
work correctly. However, this has led to other issues: dependency
compilation is no longer shared between examples which greatly increases
the time it takes for that CI step to run, and now it is even running
out of disk space on the GitHub Actions runners.
This PR adds support for a new workspace layout where the
[`aws-doc-sdk-examples`](https://github.com/awsdocs/aws-doc-sdk-examples)
repo gets to decide how the workspaces are logically grouped. If a
`Cargo.toml` file exists at the example root, then the build system
assumes that the _old_ "one example, one workspace" layout should be
used. If there is no root `Cargo.toml`, then it assumes the new layout
should be used.
The `sdk-versioner` tool had to be adapted to support more complex
relative path resolution to make this work, and the `publisher
fix-manifests` subcommand had to be fixed to ignore workspace-only
`Cargo.toml` files.
The build system in this PR needs to work for both the old and new
examples layout so that the `sdk-sync` process will succeed. #2810 has
been filed to track removing the old example layout at a later date.
[aws-doc-sdk-examples#4997](https://github.com/awsdocs/aws-doc-sdk-examples/pull/4997)
changes the workspace structure of the actual examples to the new one.
## Testing
- [x] Generated a full SDK with the old example layout, manually
examined the output, and spot checked that some examples compile
- [x] Generated a full SDK with the new example layout, manually
examined the output, and spot checked that some examples compile
- [x] Examples pass in CI with the old example layout
- [x] Examples pass in CI with the new example layout
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR fixes the Smithy `@httpChecksumRequired` trait and idempotency
token auto-fill in the orchestrator implementation.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
CI uses an `OpenIdConnectProvider` to grant access to certain operations
in the CI AWS account to the GitHub Actions runners. This provider
checks the certificate thumbprints to validate the authenticity of
connect requests. GitHub recently [added a new
thumbprint](https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/),
which was causing CI and the PR bot to intermittently fail since the
OIDC provider wasn't aware of it. This PR corrects the thumbprints to
reestablish consistency in CI.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR:
- Renames "anonymous auth" to "no auth"
- Removes fallback to other auth schemes when an identity fails to
resolve (this was not complying to the reference architecture)
- Adds the ability to opt out of credentials in `ConfigLoader`, and
removes defaulting of the shared credentials cache if no credentials
provider is given
- Makes `ConfigBagAccessors` work on `FrozenLayer` and `CloneableLayer`
- Fixes STS and aws-config tests
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR fixes the Route53 tests for the client orchestrator
implementation.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR categorizes orchestrator TODO comments into `Launch` and
`Cleanup`:
- `enableNewSmithyRuntimeLaunch`: Comment needs to be addressed before
orchestrator launch
- `enableNewSmithyRuntimeCleanup`: Comment needs to be addressed after
launch when removing middleware
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR fixes presigning for Amazon Polly in the orchestrator
implementation.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
This PR fixes the protocol tests in orchestrator mode, and adds
`--all-targets` to the orchestrator CI checks.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Co-authored-by: Zelda Hessler <zhessler@amazon.com>
## Motivation and Context
This PR upgrades the MSRV to 1.68.2.
## Checklist
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the
smithy-rs codegen or runtime crates
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Co-authored-by: Zelda Hessler <zhessler@amazon.com>
Co-authored-by: 82marbag <69267416+82marbag@users.noreply.github.com>
Also, sort `Cargo.toml` dependencies if they were disordered.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
This PR implements ci-script for bench marking compile time.
Results are turned into a markdown file and compile time is normalized
to a value relative to the compile time of S3's sdk on dev profile
without any features.
Benchmark is triggered for every PR.
I considered using AWS batch, however, I decided not to move forward
with for following reasons
- I do not have access to your AWS account, thus making the debugging
extremely difficult
- Fork's github action will always fail if you block access
## Testing
NA
## Checklist
- [ ] I have updated `CHANGELOG.next.toml` if I made changes to the
smithy-rs codegen or runtime crates
- [ ] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Co-authored-by: Zelda Hessler <zhessler@amazon.com>
## Motivation and Context
Closes https://github.com/awslabs/smithy-rs/issues/2004
## Description
Run `mdbook test` over the `design` folder.
## TODO
- [x] Ignore the RFC sections using `ignore` tag on the code blocks.
- [ ] Fix the remaining examples.
- [x] Ensure local `rust-runtime` dependencies are being used.
---------
Signed-off-by: Daniele Ahmed <ahmeddan@amazon.de>
Co-authored-by: 82marbag <69267416+82marbag@users.noreply.github.com>
## Motivation and Context
This PR fixes S3 presigning in the client orchestrator implementation. A
Polly presigning fix will come in a separate PR.
This PR also:
- Renames `ClientProtocolGenerator` to `OperationGenerator` to better
represent what its generating.
- Moves `OperationCustomization` into `codegen-client` since it is
client specific and unused by the server codegen.
- Deletes the `ProtocolGenerator` base class in `codegen-core` since it
wasn't adding any value.
- Adds the concept of stop points to the orchestrator so that
orchestration can be halted before transmitting a request.
- Adds `DisableInvocationIdInterceptor`,
`DisableRequestInfoInterceptor`, `DisableUserAgentInterceptor`, and
`HeaderSerializationSettings` config bag configs to facilitate
presigning requirements.
- Fixes compilation of all S3 and Polly tests when in orchestrator mode.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
* Show how to plug a connector
* For TLS: keep rustls, only
## Checklist
<!--- If a checkbox below is not applicable, then please DELETE it
rather than leaving it unchecked -->
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the
smithy-rs codegen or runtime crates
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Signed-off-by: Daniele Ahmed <ahmeddan@amazon.de>
## Motivation and Context
This PR gets the Glacier integration tests to pass against the
orchestrator implementation by porting the Glacier customizations from
`make_operation` changes to interceptors.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
This PR removes all the middleware code from generated clients when the
`smithy.runtime.mode` flag is set to `orchestrator`. It also changes the
`aws-config` crate to use the fluent client instead of the Smithy client
for STS and SSO based credential providers.
The `polly` test no longer compiles in orchestrator mode with these
changes since presigning directly references middleware code. This will
get fixed in a later PR.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
- Controlling time is required for several testing use cases
- #2087
- #2262
## Description
Introduce `TimeSource` trait, a real implementation, and a test
implementation.
## Testing
These changes are used in the timestream PR
## Checklist
No changelog, these changes have no impact since the code is not yet
utilized
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
This PR adds back `--location` to the subcommand
`generate-version-manifest` for the `publisher`. Without this, CI would
fail due to the PR bot trying to use the already deleted command option
([link](https://github.com/awslabs/smithy-rs/actions/runs/5023168618/jobs/9007489752?pr=2712)).
It's important to note that all changes to `tools` need to be backwards
compatible across releases.
## Testing
Against the old and the new versions of
[`build.gradle.kts`](https://github.com/awslabs/smithy-rs/pull/2663/files),
manually ran
```
./gradlew :aws:sdk:assemble
```
and confirmed that `versions.toml` got created under
`smithy-rs/aws/sdk/build/aws-sdk`.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
Co-authored-by: Yuki Saito <awsaito@amazon.com>
## Motivation and Context
This PR adds the SDK smoke test with the orchestrator implementation to
CI, and excludes services that don't yet pass tests.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
This PR refactors the client protocol test generator machinery to use a
client instead of calling `make_operation` directly, and then fixes the
ad hoc tests for the orchestrator.
The ad hoc tests revealed that overriding the signing region/service via
endpoint config was lost when porting SigV4 signing to the orchestrator,
so this PR updates the SigV4 `HttpRequestSigner` implementation to
restore this functionality. It is doing this in the signer directly
rather than via an interceptor since it should only run this logic when
SigV4 is the selected auth scheme.
Other notable changes:
- Adds `--no-fail-fast` arg to `cargoTest` targets so that all Rust
tests run in CI rather than stopping on the first failure
- Changes `EndpointResolver::resolve_and_apply_endpoint` to just
`resolve_endpoint` so that the orchestrator can place the endpoint
config into the request state, which is required for the signer to make
use of it
- Adds a `set_region` method to SDK service configs
- Deletes the API Gateway model and integration test from the SDK smoke
test since it is covered by the ad hoc tests
- Adds a comment explaining where the API Gateway model comes from in
the ad hoc tests
- Adds a `smithy.runtime.mode` Gradle property to `aws:sdk` and
`aws:sdk-adhoc-test` to trivially switch between middleware and
orchestrator when testing/generating locally
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
We have found that `aws-sdk-rust` release-2023-05-01 brought in the
`aws-wasm` crate to `versions.toml`. The crate comes from an example in
the `aws-doc-sdk-examples` repository. This PR fixes gradle build in
`sdk` so that the `publisher` only considers crates under the `sdk`
directory when generating `versions.toml`.
## Testing
Manually ran
```
./gradlew :aws:sdk:assemble
```
and confirmed that `versions.toml` got created under
`smithy-rs/aws/sdk/build/aws-sdk`
~~`smithy-rs/aws/sdk/build/aws-sdk/sdk`~~ and `README` was successfully
hydrated.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Co-authored-by: Yuki Saito <awsaito@amazon.com>
## Motivation and Context
During a release for `aws-sdk-rust`, we've found that the canary test
received an improved speech text from Transcribe. This broke the canary
test because the test hard-coded the string that should be expected by
Transcribe. This PR updates the runner so that the expected speech text
can be passed-in from outside of the program.
## Testing
Added unit tests, mainly for argument parsing.
## Checklist
If the next release is created off of the main branch (because it's a
non-breaking release), we only need to merge this PR the main branch. If
we need to release from a release branch at some point in the future,
this PR will need to be duplicated and merged to the release branch as
well.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Co-authored-by: Yuki Saito <awsaito@amazon.com>
## Motivation and Context
Addresses vulnerabilities reported by `cargo audit` within the
repository.
## Description
This commit addresses vulnerabilities within the tools directory
reported by `cargo audit`. Mostly they have been fixed by regenerating
`Cargo.lock` files. Two exceptions:
- `crate-hasher` needs to drop the `temp_dir` crate and switch over to
the `tempfile` crate
- `canary-runner` needs to upgrade the `octorust` crate
## Testing
After the PR, no vulnerabilities reported from the crates that have been
patched. Ran `cargo t` on the updated crates.
Furthermore, no vulnerabilities reported currently within `rust-runtime`
and `aws/rust-runtime`:
```
➜ rust-runtime git:(ysaito/fix-cargo-audit) pwd
smithy-rs/rust-runtime
➜ rust-runtime git:(ysaito/fix-cargo-audit) rm Cargo.lock && cargo generate-lockfile && cargo audit
Updating crates.io index
Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
Loaded 543 security advisories
Updating crates.io index
Scanning Cargo.lock for vulnerabilities (314 crate dependencies)
➜ rust-runtime git:(ysaito/fix-cargo-audit) pwd
smithy-rs/aws/rust-runtime
➜ rust-runtime git:(ysaito/fix-cargo-audit) rm Cargo.lock && cargo generate-lockfile && cargo audit
Updating crates.io index
Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
Loaded 543 security advisories
Updating crates.io index
Scanning Cargo.lock for vulnerabilities (249 crate dependencies)
```
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
Co-authored-by: Yuki Saito <awsaito@amazon.com>
## Motivation and Context
cargo semver-checks 0.20.0 is something like 100x faster
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
## Motivation and Context
This PR adds support for Smithy's `@httpBasicAuth` and `@httpBearerAuth`
auth schemes, and ports the `@httpApiKeyAuth` scheme to the
orchestrator. This is prerequisite work for supporting Amazon
CodeCatalyst since that requires bearer auth.
This PR also fixes a bug in auth orchestrator that caused an error if no
identity is present for a scheme even when an identity for a lower
priority scheme is available.
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Co-authored-by: Zelda Hessler <zhessler@amazon.com>
## Motivation and Context
We want to automatically generate stubs in the codegen diff to ensure
they can be reviewed and have a simple way to generate and include the
stubs inside the Maturin wheel.
## Description
The Python example has been moved to the `examples` folder and
refactored. The refactoring
ensures the script `stubgen.py` is included in the codegeneration of the
SDK crate. The script is later used to generate stubs automatically
during testing and can be used by customers to add their own stubs
before the Maturin build
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
----
_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._
---------
Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>
Co-authored-by: Burak <unexge@gmail.com>
* Remove parameter from `Protocol`s `structuredDataParser`, `structuredDataSerializer`
No implementation of the `Protocol` interface makes use of the
`OperationShape` parameter in the `structuredDataParser` and
`structuredDataSerializer` methods.
* Remove the TypeConversionGenerator class in favor of using
customizations for JsonParserGenerator and ServerHttpBoundProtocolGenerator.
Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>
* Make the additionaParserCustomizations default to empty list
* Fix merge conflict
* Fix missing ;
* Use better defaults when checking for customizations
* Use better defaults when checking for customizations
* Add HttpBindingCustomization and relax the datetime symbol check
* Support recursive shapes and add a lot more models to the tests
Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>
* Support naming obstacle course
* Add support for constrained blobs conversions
* Support constraint traits
* Try to generate the full diff
Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>
* A better way of checking if we need to go into the Timestamp branch
* Remove wheels folder
---------
Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>
Co-authored-by: david-perez <d@vidp.dev>
This commit updates `ci-scripts/check-aws-sdk-service` to implicitly
check whether the top-level `Cargo.toml` is valid or not. In this case,
"valid" means that examples are properly excluded from the workspace.
Co-authored-by: Yuki Saito <awsaito@amazon.com>
* Add semver-checks CI job
* Add flag to skip generation
* Make it possible to suppress semver checks with label
* fix GitHub script
* Fix but in semver-checks python script
* Cleanup debug info
* Move to ci-pr
John DiSanti