* Upgrade minimum Tokio version to 1.8.4
This addresses RUSTSEC-2021-0072 and RUSTSEC-2021-0124.
The SDK, runtime crates, and server are set to the minimum secure
version of Tokio to allow a larger range of versions to be used by
consumers of those libraries. The tools and the Python server are
both set to the latest Tokio version since they are not intended
to be consumed as a libraries.
* Upgrade `tracing-subscriber` to 0.3.15
* Upgrade `criterion` to 0.3.6
This addresses RUSTSEC-2021-0093 by upgrading `criterion`, which
upgrades `rayon`, which upgrades `crossbeam-deque`.
* Fix bug in coroutine scheduling. Add ByteStream implementation and simplify Python symbol visitor.
* Bug in coroutine scheduling:
In the previous iteration, the runtime crate aws-smithy-http-server-python
was exposing the python application implementation as a struct, which
was wrapped by the codegenerated App to allow to dynamically building
the router.
This caused scheduling of coroutines (handlers prefixed with async def)
to block becuse we were passing the Python eventloop of the parent
process that was stored pre-fork().
This commit changes the runtime PyApp to become a trait, allowing us to
dynamically build the router post-fork() and with the right event loop.
This change also allowed us to remove a bunch of unnecessary Arc(s).
* Add ByteStream implementation
Implementation of a ByteStream type for Python that can be roughly used like this:
let b = await ByteStream.from_path("file.txt")
async for chunk in b:
print(chunk)
Implement futures_core::stream::Stream for Python ByteStream wrapper.
The BytesStream implementation in python can now use a sync Mutex from
parking_lot because we are now using pyo3_asyncio::tokio::local_future_into_py()
to read a chunk, which supports !Send futures.
This allows to simply forward the implementation of Stream (poll_next()
and size_hint()) directly to our inner SDK ByteStream.
* Simplify Python symbol visitor
Inherit from Symbol visitor and override just what is needed to swap Python complex
types.
Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>
The current approach that attempts to downcast never worked; all Rust
decorators were being loaded, and the cast was doing nothing, because
the generic type parameter is erased at runtime.
Attempting to downcast a generic class `C<T>` to `C<U>` where `U: T` is
not possible to do in Kotlin (and presumably all JVM-based languages)
_at runtime_. Not even when using reified type parameters of inline
functions. See https://kotlinlang.org/docs/generics.html#type-erasure
for details.
This commit thus goes for another approach, suggested in the linked
Stack Overflow question [0]: add a method to the loaded classes that
signals at runtime the generic type parameter (`ClientCodegenContext` or
`ServerCodegenContext`) they can work with, in order to filter them.
This commit also simplifies the way the Python server project loads the Python
server-specific decorators, by deleting the combined decorator
`PythonServerCodegenDecorator`, which was being loaded from the classpath, and
instead directly using `CombinedCodegenDecorator` and passing it the Python
server-specific decorators in the `extras` parameter.
[0]: https://stackoverflow.com/questions/5451734/loading-generic-service-implementations-via-java-util-serviceloader
* Exclude sigv4 test suite from pre-commit file endings hook
* Upgrade Ktlint
* Make Gradle ktlint target consistent with pre-commit
They were previously running two separate rulesets since pre-commit
pulls in the standard ruleset, and that wasn't pulled into the Gradle
dependencies.
* Fix Gradle Ktlint paths
* Establish editorconfig for Ktlint
* Fix Ktlint lints
* Enable `trailing-commas` rule
* Fix block quote indentation
Rust crate names are allowed to contain hyphens, while Rust identifiers
are not [0]. However, the Rust and Python server projects are currently
broken because we use the user-provided crate name verbatim
(`moduleName` in `settings`) in places where Rust expects valid
identifiers:
* The Rust server's generated operation registry Rust docs use the crate
name verbatim in import statements.
* The Python server uses the crate name verbatim to generate a shared
library, but library target names in `Cargo.toml` cannot contain
hyphens.
We never caught this bug because we don't generate any crates with
hyphens in their names.
This commit:
* fixes the above usages of verbatim crate names where identifiers are
expected by calling `toSnakeCase()`,
* modifies the unit test for the server's operation registry generator
to generate a crate with hyphens in its name,
* changes the Pokémon service server SDK crate names from
`pokemon_service_sdk` to `pokemon-service-server-sdk`, so that we
exercise generation of a crate with hyphens in its name, and to better
convey that the generated SDK is server-specific; and
* changes the Pokémon service client SDK crate name from
`pokemon_service_client` to `pokemon-service-client`, and changes the
Pokémon service crate name itself from `pokemon_service` to
`pokemon-service`, for symmetry.
[0]: https://doc.rust-lang.org/reference/identifiers.html
* Support server event streams
* Rename EventStreamInput to EventStreamSender
* Custom event stream errors
* EventStreamSender and Receiver are parametrized also on event stream error types
* Pokemon service model updated
* Pokemon server event handler
* Pokemon client to test event streams
* EventStreamDecorator to make optional using SigV4 signing
* Use forInlineFun for all errors
Closes: #1157
Signed-off-by: Daniele Ahmed <ahmeddan@amazon.de>
Co-authored-by: John DiSanti <jdisanti@amazon.com>
Despite all the decorators and protocols that we have, there aren't any
`RustCodegenDecorator`s or `ProtocolGeneratorFactory`s that are
modifying the base symbol provider that the Smithy plugin provides. So
we can use directly the base symbol provider everywhere and remove the
methods from these interfaces to make the code a little bit simpler.
Likewise, there isn't a single `ProtocolGeneratorFactory` that
transforms the base model that the Smithy plugin provides.
`ServerOperationRegistryGenerator` is the only server generator that is
currently not protocol-agnostic, in the sense that it is the only
generator that contains protocol-specific logic, as opposed to
delegating to the `Protocol` interface like other generators do.
With this change, we should be good to implement a
`RustCodegenDecorator` loaded from the classpath that implements support
for any protocol, provided the decorator provides a class implementing
the `Protocol` interface.
This commit also contains some style changes that are making `ktlint`
fail. It seems like our `ktlint` config was recently broken and some
style violations slipped through the cracks in previous commits that
touched these files.
This PR add several functionalities to the Python/Rust code-generation, making it usable for models that are not using Streaming and Union shapes.
* Export codegenerated shapes as shared library that can be imported by the Python interpreter.
* Add complete end-to-end testing running in CI.
* Register signals and handle workers processes lifecycle.
* Use a custom SymbolProvider to override non-primitive types that are exposed to Python.
* Implement support for DateTime.
* Cast Python exception into Rust error types.
* End to end testing suite with a Rust client calling a real Python server.
This commit splits each of:
* `CodegenContext`,
* `RustSettings`; and
* `CodegenConfig`
into two variants: one for the `rust-codegen` client plugin and another
for the `rust-server-codegen` plugin.
(`CodegenConfig` is a property of `RustSettings`, which is a property of
`CodegenContext`; they constitute pervasive read-only global data that
gets passed around to the code generators).
This allows each plugin to evolve separately, defining different
settings and passing around a tailor-made code-generation context.
Client-specific generators should now use:
* `ClientCodegenContext`,
* `ClientRustSettings`; and
* `ClientCodegenConfig`.
Likewise, server-specific generators should now use:
* `ServerCodegenContext`,
* `ServerRustSettings`; and
* `ServerCodegenConfig`.
This is a more robust and maintainable approach than the current one,
where both generator types have to read possibly null values (because
they don't make sense for a client/server target), and plugins have to
populate settings that are not relevant for the crate they're
generating. For example, the server plugin has to populate and pass
around the settings `renameExceptions`, `includeFluentClient` and
`addMessageToErrors` even though it never uses them and don't make sense
for it.
As of writing these client and server specific classes are very similar,
but in the future they will evolve in separate ways (for example, #1342
will add symbol providers for constrained shapes that only make sense in
a server code-generation context).
Common properties have not been duplicated however. Instead, utilizing
inheritance, code-generation context that is common to _all_ plugins can
be placed in:
* `CoreCodegenContext`,
* `CoreRustSettings`; and
* `CoreCodegenConfig`.
This way, generators that are shared by both client and server can take
in `CoreCodegenContext`. If they need to tweak the code they generate in
small and self-contained places, they can utilize the `CodegenTarget`
flag-like enum to disambiguate, like they have always been doing.
Note this change also makes `RustDecorator`s be targeted for clients or
servers, by making them generic over the code-generation context. When
loading the decorators from the classpath and combining them into one in
the executed smithy-rs plugin, all of them need to either target clients
or target servers. This makes sense: after all, all of the decorators
that conform the AWS Rust SDK are client-specific.
Closes#828.
* Refactor: Attribute.class used instead of helper methods in PythonServerAttributeUtils
Helper methods in PythonServerAttributeUtils have been deprecated in favor of using Attribute.class directly
Co-authored-by: Matteo Bigoi <1781140+crisidev@users.noreply.github.com>
* Smithy Python: PyException removed from enum
Smithy enums do not have ErrorTrait so there is no need to check and generate code for pyo3::exceptions::PyException
Co-authored-by: Fahad Zubair <fahadzub@amazon.com>
Co-authored-by: Matteo Bigoi <1781140+crisidev@users.noreply.github.com>
We are introducing code-generation for Python bindings of StructureShape, BlobShape, EnumShape, OperationShape.
This PR also add a runtime server implementation to support serving Python business logic directly and with an idiomatic experience.
Co-authored-by: david-perez <d@vidp.dev>
John DiSanti