Design Doc Updates (#1347)

* Remove makefile, add mdbook-mermaid

* Add design doc section about backwards compatibility

* fix footnotes

.github/workflows/github-pages.yml

@@ -34,6 +34,7 @@ jobs:
 
         pushd design &>/dev/null
         cargo install mdbook
+        cargo install mdbook-mermaid
         mdbook build --dest-dir ../../output
         popd &>/dev/null
 

design/README.md

@@ -3,6 +3,7 @@ Design docs are hosted [here](https://awslabs.github.io/smithy-rs/design/).
 To render design docs locally:
 ```
 cargo install mdbook
+cargo install mdbook-mermaid
 mdbook serve &
 open http://localhost:3000
 ```

design/book.toml

@@ -1,6 +1,12 @@
 [book]
-authors = ["Russell Cohen"]
+authors = ["Russell Cohen", "aws-sdk-rust@amazon.com"]
 language = "en"
 multilingual = false
 src = "src"
 title = "AWS Rust SDK Design"
+
+[preprocessor.mermaid]
+command = "mdbook-mermaid"
+
+[output.html]
+additional-js = ["static/mermaid.min.js", "static/mermaid-init.js"]

design/src/SUMMARY.md

@@ -3,14 +3,15 @@
 - [Tenets](./tenets.md)
 - [Design FAQ](./faq.md)
 - [Transport](transport/overview.md)
-  - [Http Operations](transport/operation.md)
-  - [HTTP middleware](transport/middleware.md)
+  - [HTTP Operations](transport/operation.md)
+  - [HTTP Middleware](transport/middleware.md)
 
 - [Smithy](./smithy/overview.md)
     - [Simple Shapes](./smithy/simple_shapes.md)
     - [Recursive Shapes](./smithy/recursive_shapes.md)
     - [Aggregate Shapes](./smithy/aggregate_shapes.md)
     - [Endpoint Resolution](smithy/endpoint.md)
+    - [Backwards Compatibility](smithy/backwards-compat.md)
 
 - [RFCs](./rfcs/overview.md)
   - [RFC-0001: Sharing configuration between multiple clients](./rfcs/rfc0001_shared_config.md)
@@ -25,3 +26,4 @@
   - [RFC-0010: Waiters](./rfcs/rfc0010_waiters.md)
   - [RFC-0011: Publishing Alpha to Crates.io](./rfcs/rfc0011_crates_io_alpha_publishing.md)
   - [RFC-0012: Independent Crate Versioning](./rfcs/rfc0012_independent_crate_versioning.md)
+  - [RFC-0013: Body Callback APIs](./rfcs/rfc0013_body_callback_apis.md)

design/src/endpoint.md

@@ -1 +0,0 @@
-# Endpoint Resolution

design/src/faq.md

@@ -1,10 +1,19 @@
 # Design FAQ
+
 ### What is Smithy?
-Smithy is the interface design language used by AWS services. `smithy-rs` allows users to generate a Rust client for any Smithy based service (pending protocol support), including those outside of AWS.
+
+Smithy is the interface design language used by AWS services. `smithy-rs` allows users to generate a Rust client for any
+Smithy based service (pending protocol support), including those outside of AWS.
 
 ### Why is there one crate per service?
-1. Compilation time: Although it's possible to use cargo features to conditionally compile individual services, we decided that this added significant complexity to the generated code. In Rust the "unit of compilation" is a Crate, so by using smaller crates we can get better compilation parallelism.
 
-2. Versioning: It is expected that over time we may major-version-bump individual services. New updates will be pushed for _some_ AWS service nearly every day. Maintaining separate crates allows us to only increment versions for the relevant pieces that change.
+1. Compilation time: Although it's possible to use cargo features to conditionally compile individual services, we
+   decided that this added significant complexity to the generated code. In Rust the "unit of compilation" is a Crate,
+   so by using smaller crates we can get better compilation parallelism. Furthermore, ecosystem services like `docs.rs`
+   have an upper limit on the maximum amount of time required to build an individual crate—if we packaged the entire SDK
+   as a single crate, we would quickly exceed this limit.
 
-It is worth noting that this isn't a set-in-stone design decision. A parent crate may be even be created at some point!
+2. Versioning: It is expected that over time we may major-version-bump individual services. New updates will be pushed
+   for _some_ AWS service nearly every day. Maintaining separate crates allows us to only increment versions for the
+   relevant pieces that change. See [Independent Crate Versioning](./rfcs/rfc0012_independent_crate_versioning.md) for
+   more info.

design/src/middleware.md

@@ -1 +0,0 @@
-# HTTP middleware

design/src/operation.md

@@ -1 +0,0 @@
-# Http Operations

design/src/rfcs/overview.md

@@ -12,3 +12,4 @@
 - [RFC-0010: Waiters](./rfc0010_waiters.md)
 - [RFC-0011: Publishing Alpha to Crates.io](./rfc0011_crates_io_alpha_publishing.md)
 - [RFC-0012: Independent Crate Versioning](./rfc0012_independent_crate_versioning.md)
+- [RFC-0013: Body Callback APIs](./rfc0013_body_callback_apis.md)

design/src/rfcs/rfc0012_independent_crate_versioning.md

@@ -128,7 +128,14 @@ so all model changes will result in a `minor` version bump during this phase.
 
 Overall, determining a generated crate's version number looks as follows:
 
-![Phase 1: How to version a generated crate](rfc0012_independent_crate_versioning/phase1_generated_crate_version.svg)
+```mermaid
+flowchart TD
+    start[Generate crate version] --> smithyrschanged{A. smithy-rs changed?}
+    smithyrschanged -- Yes --> minor1[Minor version bump]
+    smithyrschanged -- No --> modelchanged{B. model changed?}
+    modelchanged -- Yes --> minor2[Minor version bump]
+    modelchanged -- No --> keep[Keep current version]
+```
 
 - __A: smithy-rs changed?__: Compare the `smithy_rs_version` in the previous `versions.toml` with the
   next `versions.toml` file, and if the values are different, consider [smithy-rs] to have changed.
@@ -146,8 +153,20 @@ and repeated when merging into `aws-sdk-rust/main`.
 
 The following checks need to be run for runtime crates:
 
-![Phase 1: How to validate a runtime version bump](rfc0012_independent_crate_versioning/phase1_runtime_crate_version_checks.svg)
-
+```mermaid
+flowchart TD
+    A[Check runtime crate] --> B{A. Crate has changed?}
+    B -- Yes --> C{B. Minor bumped?}
+    B -- No --> H{C. Version changed?}
+    C -- Yes --> K[Pass]
+    C -- No --> E{D. Patch bumped?}
+    E -- Yes --> F{E. Semverver passes?}
+    E -- No --> L[Fail]
+    F -- Yes --> D[Pass]
+    F -- No --> G[Fail]
+    H -- Yes --> I[Fail]
+    H -- No --> J[Pass]
+```
 - __A: Crate has changed?__ The crate's source files and manifest will be hashed for the previous version
   and the next version. If these hashes match, then the crate is considered unchanged.
 - __B: Minor bumped?__ The previous version is compared against the next version to see if the minor version
@@ -200,7 +219,7 @@ When stabilizing to 1.x, the version process will stay the same, but the minor v
 bumping runtime crates, updating models, or changing the code generator will be candidate for automatic upgrade
 per semver. At that point, no further API breaking changes can be made without a major version bump.
 
-- [aws-sdk-rust]: https://github.com/awslabs/aws-sdk-rust
-- [rust-semverver]: https://github.com/rust-lang/rust-semverver
-- [semver]: https://semver.org/
-- [smithy-rs]: https://github.com/awslabs/smithy-rs
+[aws-sdk-rust]: https://github.com/awslabs/aws-sdk-rust
+[rust-semverver]: https://github.com/rust-lang/rust-semverver
+[semver]: https://semver.org/
+[smithy-rs]: https://github.com/awslabs/smithy-rs

design/src/rfcs/rfc0012_independent_crate_versioning/Makefile

@@ -1,13 +0,0 @@
-# Requires @mermaid-js/mermaid-cli to be installed
-# Builds diagrams from the mermaid sources
-DIAGRAMS=$(wildcard *.mmd)
-SVGS=$(DIAGRAMS:.mmd=.svg)
-OPTIONS=-t dark -b transparent
-
-all: $(SVGS)
-
-%.svg: %.mmd
-	mmdc -i $*.mmd -o $*.svg $(OPTIONS)
-
-clean:
-	rm -f $(SVGS)

design/src/rfcs/rfc0012_independent_crate_versioning/phase1_generated_crate_version.mmd

@@ -1,6 +0,0 @@
-flowchart TD
-    start[Generate crate version] --> smithyrschanged{A. smithy-rs changed?}
-    smithyrschanged -- Yes --> minor1[Minor version bump]
-    smithyrschanged -- No --> modelchanged{B. model changed?}
-    modelchanged -- Yes --> minor2[Minor version bump]
-    modelchanged -- No --> keep[Keep current version]

design/src/rfcs/rfc0012_independent_crate_versioning/phase1_runtime_crate_version_checks.mmd

@@ -1,12 +0,0 @@
-flowchart TD
-    A[Check runtime crate] --> B{A. Crate has changed?}
-    B -- Yes --> C{B. Minor bumped?}
-    B -- No --> H{C. Version changed?}
-    C -- Yes --> K[Pass]
-    C -- No --> E{D. Patch bumped?}
-    E -- Yes --> F{E. Semverver passes?}
-    E -- No --> L[Fail]
-    F -- Yes --> D[Pass]
-    F -- No --> G[Fail]
-    H -- Yes --> I[Fail]
-    H -- No --> J[Pass]

design/src/smithy/backwards-compat.md

@@ -0,0 +1,132 @@
+# Backwards Compatibility
+
+AWS SDKs require that clients can evolve in a backwards compatible way as new fields and operations are added. The types
+generated by `smithy-rs` are specifically designed to meet these requirements. Specifically, the following
+transformations must not break compilation when upgrading to a new version:
+
+- [New operation added](#new-operation-added)
+- [New member added to structure](#new-member-added-to-structure)
+- [New union variant added](#new-union-variant-added)
+- New error added (todo)
+- New enum variant added (todo)
+
+However, the following changes are _not_ backwards compatible:
+
+- Error **removed** from operation.
+
+In general, the best tool in Rust to solve these issues in the `#[non_exhaustive]` attribute which will be explored in
+detail below.
+
+## New Operation Added
+
+**Before**
+
+```smithy
+$version: "1"
+namespace s3
+
+service S3 {
+    operations: [GetObject]
+}
+```
+
+**After**
+
+```smithy
+$version: "1"
+namespace s3
+
+service S3 {
+    operations: [GetObject, PutObject]
+}
+```
+
+Adding support for a new operation is backwards compatible because SDKs to not expose any sort of "service trait" that
+provides an interface over an entire service. This _prevents_ clients from inheriting or implementing an interface that
+would be broken by the addition of a new operation.
+
+## New member added to structure
+
+### Summary
+
+- Structures are marked `#[non_exhaustive]`
+- Structures must be instantiated using builders
+- Structures must not derive `Default` in the event that required fields are added in the future.
+
+In general, adding a new `public` member to a structure in Rust is not backwards compatible. However, by applying
+the `#[non_exhaustive]` to the structures generated by the Rust SDK, the Rust compiler will prevent users from using our
+structs in ways that prevent new fields from being added in the future. **Note**: in this context, the optionality of
+the fields is irrelevant.
+
+Specifically, [`#[non_exhaustive]`](https://doc.rust-lang.org/reference/attributes/type_system.html) prohibits the
+following patterns:
+
+1. Direct structure instantiation:
+   ```rust
+   # fn foo() {
+   let ip_addr = IpAddress { addr: "192.168.1.1" };
+   # }
+   ```
+   If a new member `is_local: boolean` was added to the IpAddress structure, this code would not compile. To enable
+   users to still construct
+   our structures while maintaining backwards compatibility, all structures expose a builder, accessible
+   at `SomeStruct::Builder`:
+
+   ```rust
+   # fn foo() {
+   let ip_addr = IpAddress::builder().addr("192.168.1.1").build();
+   # }
+   ```
+2. Structure destructuring:
+   ```rust
+   # fn foo() {
+   let IpAddress { addr } = some_ip_addr();
+   # }
+   ```
+   This will also fail to compile if a new member is added, however, by adding `#[non_exhaustive]`, the `..` multifield
+   wildcard MUST be added to support new fields being added in the future:
+   ```rust
+   # fn foo() {
+   let IpAddress { addr, .. } = some_ip_addr();
+   # }
+   ```
+
+### Validation & Required Members
+
+**Adding a required member to a structure is _not_ considered backwards compatible.** When a required member is added to
+a structure:
+
+1. The builder will change to become fallible, meaning that instead of returning `T` it will
+   return `Result<T, BuildError>`.
+2. Previous builder invocations that did not set the new field will still stop compiling if this was the first required
+   field.
+3. Previous builder invocations will now return a `BuildError` because the required field is unset.
+
+## New union variant added
+
+Similar to structures, `#[non_exhaustive]` also applies to unions. In order to allow new union variants to be added in
+the future, all unions (`enum` in Rust) generated by the Rust SDK must be marked with `#[non_exhaustive]`. **Note**:
+because new fields cannot be added to union variants, the union variants themselves do **not** need
+to be `#[non_exhaustive]`. To support new variants from services, each union contains an `Unknown` variant. By
+marking `Unknown` as non_exhaustive, we prevent customers from instantiating it directly.
+
+```rust
+#[non_exhaustive]
+#[derive(std::clone::Clone, std::cmp::PartialEq, std::fmt::Debug)]
+pub enum AttributeValue {
+    B(aws_smithy_types::Blob),
+    Bool(bool),
+    Bs(std::vec::Vec<aws_smithy_types::Blob>),
+    L(std::vec::Vec<crate::model::AttributeValue>),
+    M(std::collections::HashMap<std::string::String, crate::model::AttributeValue>),
+    N(std::string::String),
+    Ns(std::vec::Vec<std::string::String>),
+    Null(bool),
+    S(std::string::String),
+    Ss(std::vec::Vec<std::string::String>),
+
+    // By marking `Unknown` as non_exhaustive, we prevent client code from instantiating it directly.
+    #[non_exhaustive]
+    Unknown,
+}
+```

design/static/mermaid-init.js

@@ -0,0 +1 @@
+mermaid.initialize({startOnLoad:true});