In upstream, Census is renamed to "Instrumentation". `com.google.census` is renamed to `com.google.instrumentation.stats`.
In gRPC, "census" in every name is replaced by "stats".
Also removed warnings about protoc version matching runtime, since this
is no longer supposed to be a problem (starting with 3.0.0-beta-4) and
all our tests ran fine when using protoc 3.0.2 with protobuf runtime
3.1.0.
Fixes#2316
Highlights
==========
StatsTraceContext
-----------------
The bridge between gRPC library and Census. It keeps track of the total
payload sizes and the elapsed time of a Call. The rest of the gRPC code
doesn't invoke Census directly.
Context propagation
-------------------
StatsTraceContext carries CensusContext (and the upcoming TraceContext)
and is attached to the gRPC Context.
1. StatsTraceContext is created by ManagedChannelImpl, by calling
createClientContext(), which inherits the current CensusContext if available.
2. ManagedChannelImpl passes StatsTraceContext to ClientCallImpl, then
to the stream, then to the framer and deframer explicitly.
3. ClientCallImpl propagates the CensusContext to the headers.
1. ServerImpl creates a StatsTraceContext by implementing a new callback
method StatsTraceContext methodDetermined(MethodDescriptor, Metadata) on
ServerTransportListener.
2. NettyServerHandler calls methodDetermined() before creating the
stream, and passes the StatsTraceContext to the stream.
3. When ServerImpl creates the gRPC Context for the new ServerCall, it
calls the new method statsTraceContext() on ServerStream and puts the
StatsTraceContext in the Context.
Metrics recording
-----------------
1. Client-side start time: when ClientCallImpl is created
2. Server-side start time: when methodDetermined() is called
3. Server-side end time: in ServerStreamListener.closed(), but before
calling onComplete() or onCancel() on ServerCall.Listener.
4. Client-side end time: in ClientStreamListener.closed(), but before
calling onClonse() on ClientCall.Listener
Message sizes are recorded in MessageFramer and MessageDeframer. Both
the uncompressed and wire (possibly compressed) payload sizes are
counted.
TODOs
=====
The CensusContext created from headers on the server side should be
attached to the gRPC Context for the call. It's not done at this moment
because Census lacks the proper API to do it. It only affects tracing
and resource accounting, but doesn't affect stats functionality
protoc no longer builds in 3.0.0 because auto-download of the gmock zip
now fails. 3.0.2 has a fix to autogen:
bba446bbf2
All that was strictly necessary was to update .travis.yml and
buildscripts/, but it helps our sanity to keep the rest of the protobuf
versions in sync. Lite is left on its existing version, because it did
not see a bump of neither the java library nor the protoc plugin.
Protobuf-lite since beta-4 is now more of a fork than a subset of
protobuf-java, which may cause us problems later since lite API is not
stable. Also, lite-generated code may now depend on APIs only in
protobuf-lite, so our users must depend on the protobuf-lite runtime.
Having all our users explicitly override the dependency is bothersome to
them and can easily only expose problems only after we do a release.
So now we are doing the dependency overriding; most users should "just
work" and pick up the correct protobuf artifact. I've confirmed the
exclusion is listed in the grpc-protobuf pom and "gradle dependencies"
and "mvn dependency:tree" do not include protobuf-lite for the examples.
Vanilla protobuf users are most likely to experience any breakage, which
should detect problems more quickly since we use protobuf-java more
frequently than protobuf-lite during development.
protobuf-lite does not include pre-generated code for the well-known
protos, so users will need to generate them themselves for the moment
(google/protobuf#1889).
Note that today changing deps does not noticeably reduce the method code
for our users, since ProGuard already is stripping most classes. The
difference in output is only a reduction of 3 classes and 6 methods for
the android example.
partially resolving #1469
The added option for java_plugin `enable_deprecated` is `true` by default in `java_plugin.cpp`, so the generated code for `TestService.java` (`compiler/build.gradle` not setting this option) has all deprecated interfaces and static bindService method.
`./build.gradle` and `examples/build.gradle` set this option explicitly to `false`, so all the other generated classes do not have deprecated code.
Will set `enable_deprecated` to `false` by default in future PR when we are ready.
This allows us to play with zero-copy and proto3 support for lite.
Unfortunately, it introduced some warnings, so deprecated warnings are
now ignored for benchmarks and interop-testing.
It converts Google Auth Library Credentials to CallCredentials, and
supersedes ClientAuthInterceptor, which is now deprecated.
Also swaps out the ClientAuthInterceptor implementation.
Caveat: This in fact changes ClientAuthInterceptor's behavior. Before
this change, if multiple ClientAuthInterceptors were attached, their
effects would be additive. After this change, only the last executed one
would take effect, and it would also overwrite the CallCredentials set
in CallOptions. We don't think it's an issue, since other languages also
only allow one call credentials to be attached to an RPC.
Bumping tcnative was not required to be done at the same time as Netty.
But the new tcnative should now work correctly on Windows and be smaller
on Linux.
This now catches a few more places we needed -Xlint:-options.
InProcessSocketAddress is technically already in our stable API, so I
maintained its current serialVersionUID.
This improves our documentation for the gradle protobuf plugin, as its
version is dependent on the gradle version.
Gradle now has the --tests flag, performance improvements, and support
for OpenPGP subkeys.
The internal package is very unstable, so any users of it are not
compatible with alternative versions of grpc-core. The same is true for
HTTP/2 support in Netty. This primarily helps Maven users; Gradle
appears to ignore the distinction between '$version' and '[$version]'
and does not trigger a build failure.
Projects like grpc-stub don't depend on internal, so they aren't using
the version pinning. The whitelist is a bit prone to go out-of-date, but
introducing grpc-api would be the long-term solution.
Fixes#1459
When running :grcp-interop-testing:test, Travis has been hanging on OS X
or flaking on Linux with:
> Process 'Gradle Test Executor 4' finished with non-zero exit value 137
Exit code 137 indicates SIGKILL (128 + 9) and is most likely caused by
the JVM being killed by the kernel's OOM killer.
The limit in .travis.yml is 2x what was necessary to do a parallel
build. The main test memory limit in build.gradle is well above 16m
which is necessary for the tests. Interop-testing is well above 64m
which is necessary for interop-testing, but we use 1.5g to help prevent
timeouts on Travis.
Protobuf and protobuf-nano each have one tests that decodes a proto >64M
in size, which prevents them from running with less than 512m and 768m,
respectively.
The jsonp dependency string is no longer shared because it was only used
in one place and someone trying to compile the examples using a new
build.gradle will need to add that dependency. It was a bit complex to
follow how libraries.jsonp worked and it wasn't really adding any value
in this particular case.
A few things to note:
- ByteString has gone away in favor of AsciiString.
- Http2Headers now uses CharSequence for all methods, so there are a few places that we have to explicitly check for AsciiString to get the optimizations.
- We now have to specify a graceful shutdown timeout for our Netty handlers. Using 5 seconds.
This takes some steps towards #525, but it won't be fixed until we
officially support OpenSSL. This is due to the fact that ALPN->NPN fallback
isn't supported with Jetty (since only one of the bootstrap plugins can be provided).
The classes are available, even on Windows. Trying to use them though
won't work. You'll get an error like:
java.lang.UnsatisfiedLinkError: no netty-transport-native-epoll in java.library.path
Previously generated files were deleted at configuration time. Running
non-build tasks such as ``clean`` or ``tasks`` would delete the
generated files and tracked by git, which is annoying. Now we delete
them only before the ``generateProto`` task, which solves the problem.
After this change, the case that ``generateProto`` is not executed at
all, is no longer covered. However, it is much less likely than the
still-covered case that ``generateProto`` is not re-generating all files
due to misconfiguration.
This allows people who cannot run the pre-compiled ``protoc`` pulled
from Maven Central to use their own ``protoc``.
Upgrade to protobuf-gradle-plugin:0.4.1 to display error messages of
protoc failures.
Since commit 287a27a the ``grpc`` codegen plugin must be explicitly
added to the ``plugins`` block of the source set, while it didn't.
Remove the generated code before recompiling it to prevent such issue
from being missed by tests.
Resolves#404
Add the gRPC codegen executable and the root ``build.gradle`` (where the
protoc version is) to the inputs of ``generateProto`` task, so that the
task is rerun when either of them has changed.
- Switch all system properties to project properties.
- Use the ``javaLocalNamingStyle`` instead of the
``dot.delimited.style`` for property names, so that it can be directly
referenced by ``rootProject.propertyName``.
- Recommend users to put GRPC-specific properties in project-level
``build.properties`` instead of the user-level.
Resolves#357
- Add project property ``grpc.skip.codegen``, which is false by default.
People who don't change the codegen nor the proto files can set it to
true so that they don't need to set up C++ compilation.
- Check in all generated files under ``src/generated``.
After running tests, you can run jacocoTestReport. The jacocoTestReport
task does not depend on the tests, so they should be run separately.
There is still a lot of noise in the jacoco output since we aren't yet
filtering generated code.
It seems almost every developer has caused Travis CI to break due to
checkstyle failures. To prevent further breakages, checkstyle will now
fail the build by default.
Specifying checkstyle.ignoreFailures=true in ~/.gradle/gradle.properties
will turn the failures into warnings, which was the old behavior.
- Stop defining custom architectures because VisualCpp toolchain doesn't
allow it. Reference to built-in architectures 'x86' and 'x86_64' since
they are supported by all toolchains.
- Remove 'local_arch' from platforms. For unsupported platform, we just
do not specify the target.
- Target no more than one platform at a time. This simplifies the build
script a lot.
- Remove the TARGET_ARCHS environment variable. Add system property
``arch`` to override ``osdetector.arch``.
- Add ``vc.disable`` to override the default choice of VisualCpp on
Windows.
- Add ``vc.`` prefix to the properties that are only used with VC++
The plugin has been published on plugins.gradle.org
Conforming to the Gradle standard that plugin IDs must be name-spaced,
the plugin ID has been changed to com.google.protobuf.
Commit 76f0a09 after the previous release
(ws.antonov.gradle.plugins:gradle-plugin-protobuf:0.9.1) defers the
generation of generateProto tasks to post-evaluation of the project,
which make them no longer available in the evaluation phase. We need to
move the manipulation of these tasks to post-evaluation too.
1. Adds <property name="separateLineBetweenGroups" value="true"/> to CustomImportOrder to enfore blank line between imports groups.
2. Uses checkstyle 6.5, which fixed a bug of "CustomImportOrder checks import sorting according to ASCII order instead of case-insensitive alphabetical order".
Failing disabled by default, but setting checkstyle.ignoreFailures=false
in ~/.gradle/gradle.properties enables failing. Travis will always run
with such failing enabled.
The checkstyle.xml is a slightly modified version of the upstream Google
checkstyle configuration. All changes have comment describing them.
Lots of warnings were corrected. Examples is the only project that has
warnings still, as the necessary changes require some thought.
-Xlint:-options is not available on some earlier JDK 7s, but won't fail
if unsupported. It prevents the warning wanting bootclasspath specified
since target/source is 1.6.
We are already specifying to javac that our code is UTF-8, but we also
need to specify to JavaDoc, even though we don't have any non-ASCII
characters in JavaDoc comments.
Tested with LC_ALL=C
Protoc should be in PATH and the project properties protobuf.include and
protobuf.libs should be set. For example:
gradlew build -Pprotobuf.include=C:\path\to\protobuf-3.0.0-alpha-2\src ^
-Pprotobuf.libs=C:\path\to\protobuf-3.0.0-alpha-2\vsprojects\Release
When running more than once, it is probably more convenient to create
%HOMEDRIVE%%HOMEPATH%\.gradle\gradle.properties with contents like:
protobuf.include=C:\\path\\to\\protobuf-3.0.0-alpha-2\\src
protobuf.libs=C:\\path\\to\\protobuf-3.0.0-alpha-2\\vsprojects\\Release
Some early grpc users on Android are using a very old Guava. They are
working on upgrading, but it will take time. This alone is not enough
for "old guava" compilibility; expect more to come.
A Gradle protoc plugin is used for generating and compiling the grpc
codegen. The code organization was changed to match what Gradle expects.
Proto 3 is now required.
Kun Zhang