mirror of https://github.com/smithy-lang/smithy-rs
40f22207ad
Avoid unnecessary Cargo crate rebuilds This commit modifies the Gradle buildscripts to avoid unnecessary Cargo rebuilds of the generated crates, decreasing development iteration cycles. Prior to this commit, if you ran a crate-generating command _twice_ on any of the `codegen-test`, `codegen-server-test`, `sdk-codegen-test`, and `codegen-server-test:python` subprojects (without making any changes to the codegen code), both invocations would take the same time: Cargo would recompile the crate and its dependencies, even when the generated crate is identical. This is because Gradle deletes everything under `buildDir` when running the `generateSmithyBuild` task: ``` > Task :codegen-server-test:smithyBuildJar Deleting stale output file: /local/home/davidpz/workplace/smithy-ws/src/SmithyRsSource/codegen-server-test/build/smithyprojections/codegen-server-test ``` So the files get recreated each time. While developing, it is likely that only a small number of the generated crate files are modified across rebuilds. [Cargo uses `mtime`](https://github.com/rust-lang/cargo/issues/6529) (among other factors) to determine whether it needs to recompile a unit. Indeed, running with `CARGO_LOG=cargo::core::compiler::fingerprint=trace` yields `err: current filesystem status shows we're outdated`. This commit adds a Gradle task that compares the hashes of the newly generated files with the (previously cached) old ones, and restores their `mtime`s if the hashes coincide. Another issue was causing unnecessary crate rebuilds. Prior to this commit, we were sending `RUSTFLAGS=-D warnings` when invoking Cargo. However, a common thing to do after generating a crate is to open its contents in an editor. The editor's `rust-analyzer` would compile the crate and its dependencies without the `RUSTFLAGS` we had used earlier. The next time you rebuilt the crate, Cargo would claim `err: RUSTFLAGS has changed: previously [], now ["-D", "warnings"]` and recompile everything again. This commit refactors the Gradle tasks so as to not send these flags when invoking Cargo, instead generating a `.cargo/config.toml` containing these flags. This way, `rust-analyzer` also picks them up and does not need to recompile the crates. With both patches, Cargo avoids unnecessary crate rebuilds. All in all, the second invocation of a `./gradlew --info -P modules='simple' -P cargoCommands='test' codegen-server-test:build` command now takes 15 seconds less than the first invocation on my `c5.4xlarge` machine; Cargo does not need to do _any_ work on the second invocation. This commit also refactors the `build.gradle.kts` files of the `sdk`, `sdk-codegen-test`, `codegen-test`, `codegen-server-test`, and `codegen-server-test:python` subprojects to make them DRYer and more consistent. The last 4 subprojects' buildscripts are now much shorter, with all the common logic having been moved to `CodegenTestCommon.kt`. Note that we have made the last 4 subprojects' `cargo check` and `cargo doc` invocations use the same set of flags than in the `sdk` subproject for consistency. Closes #1378. Closes #1412. |
||
|---|---|---|
| .cargo | ||
| .github | ||
| .pre-commit-hooks | ||
| aws | ||
| buildSrc | ||
| codegen | ||
| codegen-server | ||
| codegen-server-test | ||
| codegen-test | ||
| design | ||
| gradle | ||
| rust-runtime | ||
| tools | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| CHANGELOG.md | ||
| CHANGELOG.next.toml | ||
| CODEOWNERS | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| NOTICE | ||
| README.md | ||
| build.gradle.kts | ||
| ci | ||
| ci.mk | ||
| gradle.properties | ||
| gradlew | ||
| gradlew.bat | ||
| settings.gradle.kts | ||
README.md
Smithy Rust 
Smithy code generators for Rust that generate clients, servers, and the entire AWS SDK. The latest unreleased SDK build can be found in aws-sdk-rust/next.
All internal and external interfaces are considered unstable and subject to change without notice.
Setup
./gradlewwill setup gradle for you. JDK 11 is required.- Running tests requires a working Rust installation. See Rust docs for installation instructions on your platform. Minimum supported Rust version is the latest released Rust version, although older versions may work.
Development
For development, pre-commit hooks make it easier to pass automated linting when opening a pull request. Setup:
brew install pre-commit # (or appropriate for your platform: https://pre-commit.com/)
pre-commit install
Project Layout
aws: AWS specific codegen & Rust code (signing, endpoints, customizations, etc.) Common commands:./gradlew :aws:sdk:assemble: Generate (but do not test / compile etc.) a fresh SDK intosdk/build/aws-sdk./gradlew :aws:sdk:test: Generate & run all tests for a fresh SDK./gradlew :aws:sdk:{cargoCheck, cargoTest, cargoDocs, cargoClippy}: Generate & run specified cargo command.
codegen: Whitelabel Smithy client code generationcodegen-test: Smithy protocol test generation & integration tests for Smithy client whitelabel codedesign: Design documentation. See the design/README.md for details about building / viewing.codegen-server: Whitelabel Smithy server code generationcodegen-server-test: Smithy protocol test generation & integration tests for Smithy server whitelabel code
Testing
Running all of smithy-rs's tests can take a very long time, so it's better to know which parts to test based on the changes being made, and allow continuous integration to find other issues when posting a pull request.
In general, the components of smithy-rs affect each other in the following order (with earlier affecting later):
rust-runtimecodegenandcodegen-serveraws/rust-runtimeaws/sdk-codegen
Some components, such as codegen-test and codegen-server-test, are purely for testing other components.
Testing rust-runtime and aws/rust-runtime
To test the rust-runtime crates:
# Run all Rust tests for `rust-runtime/`
./gradlew rust-runtime:cargoTest
# Run clippy for `rust-runtime/`
./gradlew rust-runtime:cargoClippy
For aws/rust-runtime, just prefix with aws::
# Run all Rust tests for `rust-runtime/`
./gradlew aws:rust-runtime:cargoTest
# Run clippy for `rust-runtime/`
./gradlew aws:rust-runtime:cargoClippy
Some runtime crates have a additional-ci script that can also be run. These scripts often require
cargo-hack and cargo-udeps
to be installed.
Testing Client/Server Codegen
To test the code generation, the following can be used:
# Run Kotlin codegen unit tests
./gradlew codegen:check
# Run client codegen tests
./gradlew codegen-test:check
# Run server codegen tests
./gradlew codegen-server-test:check
Several Kotlin unit tests generate Rust projects and compile them. When these fail, they typically output links to the location of the generated code so that it can be inspected.
To look at generated code when the codegen tests fail, check these paths depending on the test suite that's failing:
- For codegen-test:
codegen-test/build/smithyprojections/codegen-test - For codgen-server-test:
codegen-server-test/build/smithyprojections/codegen-server-test
Testing SDK Codegen
See the readme in aws/sdk/ for more information about these targets as they can be configured
to generate more or less AWS service clients.
# Run Kotlin codegen unit tests
./gradlew aws:sdk-codegen:check
# Generate an SDK, but do not attempt to compile / run tests. Useful for inspecting generated code
./gradlew :aws:sdk:assemble
# Run all the tests
./gradlew :aws:sdk:test
# Validate that the generated code compiles
./gradlew :aws:sdk:cargoCheck
# Validate that the generated code passes Clippy
./gradlew :aws:sdk:cargoClippy
# Validate the generated docs
./gradlew :aws:sdk:cargoDoc
The generated SDK will be placed in aws/sdk/build/aws-sdk.