Commit Graph

237 Commits

Author SHA1 Message Date
Lidi Zheng 8ac9a4e7bd [xDS interop] add Docker tagging logic to the xds_url_map job 2021-10-07 09:08:23 -07:00
markb74 f57de6bd03
Make binder instrumentation tests run on kokoro. (#8563)
The tests run as part of the existing android-interop-testing job.

We needed to modify the manifest of the apk built under android-interop-testing to declare Android Services used by the binder tests.
2021-09-29 11:28:14 +02:00
Lidi Zheng 0287d83182 Add testing_version flag 2021-09-27 15:19:41 -07:00
Jan Tattermusch 3049c2c147 reenable previously disabled aarch64 tests 2021-09-22 10:53:10 -07:00
Lidi Zheng 40f70ca3c1
Change to a non-workload-identity GKE cluster (#8461)
Part of grpc/grpc#27189 and b/198291728.

By disabling the workload identity, we should be able to run tests faster and avoid future IAM policy size issue.

Kokoro run: https://fusion2.corp.google.com/invocations/b52b1684-47de-406d-a9f6-644909755f34/targets
2021-08-31 10:35:51 -07:00
Lidi Zheng c54fcba2ee
Extend the xds_url_map job's timeout to 90 minutes (#8429)
As title. We recently had one flake caused by the Kokoro job timeout.
2021-08-20 12:12:54 -07:00
Eric Anderson 3e9488be25 buildscripts: Increase memory for Gradle in Android CI
We've still been seeing random memory-related failures with the Android
CI, but it is nowhere near as severe as it was. But even when running
locally with "-Xmx512m -XX:MaxMetaspaceSize=512m" I get failures. Our CI
environment has lots of RAM; let's use it.
2021-08-17 12:31:49 -07:00
Lidi Zheng 6a6a5279c0
Add a branch name in xds_url_map's CloudBuild (#8405) 2021-08-12 13:45:19 -07:00
Lidi Zheng 2a636420ef
Update xDS client/server image per-branch tag after build (#8400) 2021-08-11 14:01:21 -07:00
Eric Anderson 57bd087cdf buildscripts: Build android instrumentation tests in android CI
Binder's :build was missing. Cronet build failed without specifying
Java 8 because of the transitive Guava dependency.
2021-08-02 16:52:30 -07:00
Eric Anderson f781d24ddd Mostly revert "Run binderchannel android tests. (#8306)"
This partilaly reverts commit 5e18ff208a.
It leaves the compilation fix that was made to
BinderClientTransportTest.

Running instrumentation tests via firebase requires a `--app` argument.
However, we don't have such an app and it isn't immediately clear how
we'll go about making one. Revert the change to let android-testing to
start passing again.

This problem wasn't noticed before merging the original commit because
android-testing is a post-commit CI.
2021-08-02 16:13:33 -07:00
Lidi Zheng 39b7415bc0
Fix a typo in xds_url_map.cfg (#8344) 2021-07-26 16:41:34 -07:00
Lidi Zheng b3a26b732e
Add xDS k8s url-map Kokoro job (#8340)
Related PR grpc/grpc#26764
Related CL cl/386366746

This test suite will be run every 6 hours, and takes around 30 minutes. The script is copied from xds-k8s.sh, and removed the generation of server image.
2021-07-26 10:14:27 -07:00
markb74 5e18ff208a
Run binderchannel android tests. (#8306) 2021-07-14 13:10:28 +02:00
Eric Anderson 79e75bace4
Port from Travis-CI to GitHub Actions
Travis-CI no longer has a free tier (only a free trial). That was a
major reason we used Travis-CI, so that external contributors would be
able to run the CI on their forks. Iterating on a Travis config in a
personal repo was also quite convenient. The other reason was that
Travis-CI was safe to run even with untrusted code.

Since the introduction of the permissions field in workflows, GitHub
Actions appears safe to run untrusted code and has a free tier for
external contributors. GitHub Actions and Google Cloud Build are the
main contenders for a Kokoro replacement, but Cloud Build isn't safe for
untrusted code. Instead of migrating to Travis-CI.com from
Travis-CI.org, let's migrate to GitHub Actions and gain some familiarity.

I've really appreciated Travis-CI.org and have wanted to pay for it for
years but wasn't about to give it write permission to the repo. I'm
disappointed to migrate off it, now that the permissions issues have
been sorted out.
2021-06-17 13:32:56 -07:00
Sergii Tkachenko 814655cdde buildscripts: add option to use xds-k8s test driver from a fork 2021-06-16 19:09:04 -04:00
Eric Anderson 5642e01243
Replace failOnVersionConflict() with custom requireUpperBoundDeps
failOnVersionConflict has never been good for us. It is equivalent to
Maven dependencyConvergence which we discourage our users to use because
it is too tempermental and _creates_ version skew issues over time.
However, we had no real alternative for determining if our deps would be
misinterpeted by Maven.

failOnVersionConflict has been a constant drain and makes it really hard
to do seemingly-trivial upgrades. As evidenced by protobuf/build.gradle
in this change, it also caused _us_ to introduce a version downgrade.

This introduces our own custom requireUpperBoundDeps implementation so
that we can get back to simple dependency upgrades _and_ increase our
confidence in a consistent dependency tree.
2021-06-11 14:01:18 -07:00
Eric Anderson 29618a6bb6 Bump Protobuf to 3.17.2
Removed some references to the protobuf version from COMPILING.md and
compiler/README.md to make updates a tiny bit easier.
2021-06-07 11:20:48 -07:00
Menghan Li d4e90a78fd
buildscript: fold header/path matching tests to all (#8054) 2021-05-27 09:47:50 -07:00
Sergii Tkachenko 976daf2dd0 buildscripts: switch xds-k8s cluster to 1.20.x 2021-05-05 20:01:15 -04:00
Lidi Zheng d2160ea703
Extend the xDS interop tests timeout to 360 mins (#8133) 2021-05-05 10:18:44 -07:00
Jan Tattermusch 72527708f5
skip flaky :grpc-xds:test in linux aarch64 tests (#8119)
Due to #8118 the :grpc-xds:test currently doesn't provide useful signal.
2021-04-27 10:18:39 -07:00
Sergii Tkachenko fcbc1abc44 buildscripts: xds-k8s pin pip to 21.0.1
pip 21.1 released on Apr 24 introduced a regression for python 3.6.1.
The regression was identified on Apr 24, the fix merged on Apr 25.
The fix is expected to be delivered in the 21.1.1 patch.

There's no clear date, when 21.1.1 will be released.
Until then, pin is temporarily pinned to the previous release, 21.0.1.
2021-04-26 18:15:18 -04:00
Jan Tattermusch b436d0dfb7
Improve emulated linux aarch64 tests, without protoc artifact build
Based on my testing, the tests are currently flaky, so it would be better to first merge it and start running it continuously to asses the situation.
2021-04-26 10:54:28 -07:00
Lidi Zheng 134e9cbc42
buildscripts: enable CSDS test 2021-04-21 20:48:34 -04:00
Sergii Tkachenko 278a336d1f buildscript: xds-k8s increase build timeout 2021-04-08 22:05:29 -04:00
Sergii Tkachenko c113ba1030 buildscript: add xds-k8s cluster endpoint override
Missed this in 1b86618ce9
2021-04-08 21:14:08 -04:00
Sergii Tkachenko 1b86618ce9 buildscript: use different xds-k8s cluster
In preparation to the Public Preview.
2021-04-08 19:00:34 -04:00
ZHANG Dapeng e4ccf0eeac
buildscript: enable fault injection interop test 2021-03-17 17:41:58 -07:00
Doug Fawley 3752b9e365
add timeout test case to xds interop tests (#7888) 2021-02-12 14:25:20 -08:00
Eric Gribkoff 0ffd942fd2
buildscripts: increase xds job timeouts (#7831) 2021-01-25 04:33:01 -08:00
ZHANG Dapeng 150f994315
buildscript: fix execution path of xds.sh in xds_v3.sh 2021-01-20 09:21:36 -08:00
ZHANG Dapeng c1f9658a72
buildscripts: fix buildscript unbound variable error 2021-01-19 18:33:55 -08:00
ZHANG Dapeng 56fc8b8626
buildscripts: add xds_v3 buildscripts 2021-01-19 15:01:21 -08:00
Jan Tattermusch 76c5a651a1
Add a simple way to run grpc-java test inside emulated ARM64 docker container 2021-01-19 09:05:22 -08:00
Sergii Tkachenko e5f57adb7f
buildscripts: xDS Kubernetes Interop tests buildscript 2020-12-23 09:52:01 -05:00
Chengyuan Zhang 1e49ac665a
buildscript: ensure dependency convergence (#7745)
Run grpc-all:dependency task in Kokoro. For some unknown reason, building grpc-all doesn't reliably catch dependency conflicts.
2020-12-22 09:09:11 -08:00
Eric Anderson 8ce6355e89 buildscripts: Combine android builds together
Previously the android projects were separate from the main build and
each other. For quite a while now they have been integrated in the main
project. There's no longer any need to build each separately.
2020-12-07 15:01:51 -08:00
Chengyuan Zhang 71a3c55177
buildscripts: add missing CI coverage for examples (#7708)
Adds CI coverage for building example/android/strictmode and examples/example-jwt-auth. Also cleans up existing Gradle and Maven build command in the CIs.
2020-12-07 14:43:32 -08:00
Eric Anderson f13471d68d buildscripts: Check for out-of-date codegen after android builds
Issues like that fixed in 05048cf3 should have failed the CI, like it
does for the rest of the build (on both Travis and linux_artifacts).
2020-12-07 11:56:06 -08:00
Eric Anderson ee9286a0f2 buildscripts: Fix grpc-java-artifacts indentation
Re-sort packages, now that the tabs aren't skewing the sorting.

I'm quite confident I had fixed this already, but I had multiple copies
of this file on multiple machines and must have fixed the wrong copy.
2020-12-04 16:42:26 -08:00
Eric Anderson 486d82a7e1 Build Linux artifacts using CentOS 7
CentOS 6 is dead and no longer has update servers. CentOS 7 is older
than Debian 9 (oldstable), so binaries hopefully work on both. More
testing is necessary, but everything's broken now, so this is better
than nothing.

We stop using protoc-artifacts because now the container is
straight-forward enough that we can just use our own. Previously the
"devtoolset" stuff made us want to share the container.
2020-12-03 22:56:31 -08:00
Sergii Tkachenko fd59f1de0c buildscripts: Add xDS GKE interop tests buildscript placeholder 2020-11-25 16:24:07 -05:00
Chengyuan Zhang c850840342
buildscripts: enable xDS circuit breaking test (#7615) 2020-11-16 19:27:57 -08:00
Chengyuan Zhang da100e8e49
build, examples, README.md: Update protobuf gradle plugin version to 0.8.13 (#7355)
Updated protobuf gradle plugin version to 0.8.13. Fixed Android Kokoro's memory issue by forcing to use a new Gradle daemon for building the previous commit.
2020-09-21 10:42:06 -07:00
Eric Anderson 96a5c5df57 buildscripts: Avoid signing sha256 and sha512 checksums
Starting in Gradle 6.0 maven-publish began including sha256 and sha512
checksums, in addition to the previous md5 and sha1 checksums. We don't want
.sha256.asc and .sha512.asc files, as they serve no purpose.
2020-07-31 09:52:36 -05:00
Menghan Li d2182fe197
interop-testing: add path_matching and header_matching (#7254) 2020-07-28 14:07:43 -07:00
Eric Gribkoff 4fbe6bef7d
interop-testing: add flags to xds test client 2020-07-28 12:43:00 -07:00
Eric Anderson 8a792c3367 buildscripts: Fix check for java-example-hostname:latest tagging
The previous 'git ls-remote' was returning the tag-referenced commits via
vF.O.O^{} which was confusing the version check (as v1.30.0 would not match
v1.30.0^{}). Passing --refs filters those ^{} tags.

There was also a missing 'v' in the version check. It was added explicitly to
generate the list but not to check the result.
2020-07-24 09:22:05 -05:00
ZHANG Dapeng 0d6546719a
all: bump protobuf to 3.12.0
Version 3.12.0 is needed for the feature of support for proto3 field presence #7051
2020-05-20 16:30:59 -07:00
Eric Anderson b1ca6e8d27 Automate release process for example-hostname 2020-05-12 14:30:27 -07:00
Eric Gribkoff 95b0550c59
buildscripts: use xds-test-server image (#6889) 2020-04-23 15:59:19 -07:00
Patrice Chalin da855f4164
Delete example-kotlin (#6936) 2020-04-16 13:55:05 -07:00
ZHANG Dapeng 5c31dc6d71
xds: migrating XdsNameResolver to "xds" scheme
Change "xds-experimenal" scheme to "xds" and add a deprecated provider with "xds-experimenal" scheme for keeping current users' tests working.
2020-04-07 11:52:41 -07:00
Eric Gribkoff e2bb44106a
buildscripts: allow per-xds test case artifacts (#6857) 2020-03-26 10:13:05 -07:00
Chengyuan Zhang 95e331a69b
buildscripts: clean up android related scripts (#6843)
Fix incorrect android-interop-testing apk path after flatten the project structure.
2020-03-19 19:09:54 -07:00
Eric Gribkoff 1acde4604f
buildscripts: checkout matching grpc/grpc branch for xds test (#6828) 2020-03-13 15:32:13 -07:00
Eric Gribkoff 99b6d0156e
buildscripts,interop-testing: Increase logging for xDS tests (#6818) 2020-03-11 20:34:26 -07:00
Eric Gribkoff 3b8e36358c
buildscripts: update xds test server arg (#6816) 2020-03-11 13:20:30 -07:00
Chengyuan Zhang 5ba663bcf6
android: add grpc-android into main build (#6793)
Add grpc-android into main build. grpc-android will be built if Gradle option skipAndroid is false. This change also migrates deprecated Robolectric methods to androidx.test methods.
2020-03-04 15:40:39 -08:00
Jihun Cho 0fd4975d4c
rls: add proto and converter (#6743) 2020-02-28 21:41:19 -08:00
Eric Gribkoff 5555ec9a35
buildscripts: fix url of grpc core repo (#6708) 2020-02-13 18:51:08 -08:00
Eric Gribkoff 84b10350eb buildscripts,interop-testing: Add xDS tests on GCP (#6662) 2020-02-09 19:43:11 -08:00
Eric Anderson 2e12b53f2f
examples: Add XDS client example 2020-01-31 11:13:44 -08:00
Eric Anderson e7d7c5bde4
examples: Add hostname example
This is placed in its own directory since it depends on grpc-services.
2020-01-17 15:50:49 -08:00
Eric Anderson 8cf0196548
Add Cronet to main build
Building now requires an Android SDK unless you specify
-PskipAndroid=true
2020-01-09 15:20:41 -08:00
Eric Anderson c606519a5d bazel: Support maven_install
maven_install is strongly superior to previous forms of grabbing dependencies
from Maven as it computes the appropriate versions to use from the full
transitive dependencies of all libraries used by an application. It also has
much less boilerplate and includes dependencies with generated targets.

In the future we will drop the jvm_maven_import_external usages and require
maven_install, at which point we can swap to using the `@maven' repository and
no longer depend on compat_repositories.

Fixes #5359
2019-12-30 12:08:42 -08:00
Liu sheng 470a4ab12d Add support for cross-compiling for aarch64 platform
This change modified the building scripts to support cross compile
grpc-java for aarch64 platform

For grpc/grpc-java#6364
2019-12-06 14:21:34 -08:00
Eric Anderson 54b7847a7f Update protobuf to 3.11.0 and gson to 2.8.6
Updating Protobuf to bring in improvements for Android.

Protobuf pulls in the newer version of gson and we are happy to update
to it.
2019-12-02 16:43:06 -08:00
Elliotte Rusty Harold 079219bbc9 Update to GSON 2.8.5 and protobuf 3.10.0 2019-10-09 07:47:34 -07:00
Chengyuan Zhang e866d3539c
buildscripts: add config for building grpc-cronet artifact (#6134) 2019-09-09 16:14:56 -07:00
ZHANG Dapeng c240f27077
buildscripts: android.sh to clean the build when building HEAD^
A gradle plugin [issue](google/protobuf-gradle-plugin#331) was seen for https://github.com/grpc/grpc-java/pull/6099. This PR try to workaround it.
2019-08-28 12:56:49 -07:00
Carl Mastrangelo 57e7bd394e
all: update to proto 3.9.0 2019-07-26 13:53:05 -07:00
Eric Anderson be819fa3fd
android: Convert to maven-publish
com.github.dcendents:android-maven-gradle-plugin is incompatible with
Gradle 5 and the project hasn't seen any activity in over a year, so it
seems unlikely to get fixed.

We want to use maven-publish anyway, since that's what we use elsewhere.
2019-06-06 14:53:27 -07:00
Eric Anderson 52dff83717 Update Protobuf to 3.7.1
This mainly avoids protoc from 3.7.0 which has a dependency on libatomic. Most
of our systems have libatomic, so it mostly works, but the interop docker
container does not, so building fails. Version 3.7.1 was rebuilt to avoid
needing the libatomic shared library.

This has the added benefit that Bazel is now on the same version as Gradle, as
3.7.1 included fixes for Bazel.
2019-04-05 10:55:14 -07:00
ZHANG Dapeng a17f8ab62c
buildscripts: add android-interop-test build (without runing test) in kokoro android 2019-04-03 14:22:39 -07:00
ZHANG Dapeng 755a22790b
android/android-interop-testing/examples: upgrade android plugin
This resolves #5523

While bumping `com.android.tools.build:gradle:3.1.2` to `3.3.0`, some other plugins/artifacts/maven repo/buildscripts have to be updated:

- gradle (wrapper) need to upgrade to 4.10.x
- protobuf gradle plugin need to bump a version compatible with gradle version.
- need add `google()` and `jcenter()` repos for android (otherwise `com.android.tools.build:aapt2:3.3.0x` and `trove4j` will not be found resp.)
- need to accept license for Android "build-tools;28.0.3" in kokoro env.
2019-04-02 14:52:27 -07:00
Eric Anderson e7a635d882
buildscripts: Update protoc-artifacts to 3.7.0
This was missed from d7e53e871

Also added missing --tmpdir, to avoid extracting protobuf
into the current directory.
2019-03-13 13:43:21 -06:00
Eric Anderson d7e53e871b
Merge pull request #5454 from ejona86/protobuf-3.7.0
Upgrade to Protobuf 3.7.0
2019-03-11 15:39:50 -06:00
Liam Miller-Cushon 952a767b9c Rename source jar outputs to avoid conflicts with java_common.compile
More information: https://github.com/bazelbuild/bazel/issues/5824
2019-03-05 08:44:22 -07:00
Rodrigo Queiro 0959a846c8 bazel: update to new ProtoInfo provider
This is part of #5383. As Bazel 0.22.0 is the only supported version
now, use this in the Kokoro build.
2019-02-28 21:30:48 -07:00
Jihun Cho 68f6ce9ddd
all: fix publish not having checksum (#5398) 2019-02-27 09:39:02 -08:00
Eric Anderson eaca73473c
Upgrade to protobuf 3.6.1
For Bazel, we upgrade to protobuf 3.6.1.2 and javalite HEAD to fix
incompatibilities in newer Bazel releases.

compiler/Dockerfile is unused, so it was removed instead of being updated.

protoc no longer includes codegen for nano, so we remain on the older protoc
any time nano is used.

Protobuf now requires C++11 when compiling, so windows was swapped to
VC 14.
2019-02-07 13:40:53 -08:00
Jihun Cho 1c3432c3fb
all: migrate gradle publish from maven to use maven-publish plugin (#5289) 2019-01-31 17:38:43 -08:00
Eric Gribkoff f973bbc06f
Revert "buildscripts: disable android size status check (#5278)" (#5284)
This reverts commit aaa3a86dd2.
2019-01-25 16:16:41 -08:00
Eric Gribkoff aaa3a86dd2
buildscripts: disable android size status check (#5278) 2019-01-24 12:21:58 -08:00
ZHANG Dapeng 788cdbd511
buildscripts: Show bazel version in CI 2019-01-14 16:54:51 -08:00
Eric Anderson c8817cf284
kokoro: Reduce gradle workers on Mac to reduce flakes
We've had a long run of test timeout failures on Mac with timeouts in
the seconds. Multiple of these haven't seemed like races, but simply
that the Mac machines were too slow. The most recent case for this is a
set of timeouts for
OkHttpTransportTest.earlyServerClose_serverFailure_withClientCancelOnListenerClosed
where it took over a second for a server stream to be created ("Timed
out waiting for server stream").

This commit reduces the number of workers to reduce the overall load,
hoping this lets tests complete within a reasonable amount of time.
2019-01-11 08:56:45 -08:00
Eric Anderson 186409f868
buildscripts: Add script to sync .protos from grpc-proto
Currently only protos in alts, grpclb, and services are synced. Once
some java_package options are "upstreamed", benchmarks and
interop-testing could also be synced.
2018-12-20 16:47:16 -07:00
ZHANG Dapeng 94e7339748
all: remove copies of gradle wrapper 2018-12-11 16:23:28 -08:00
ZHANG Dapeng dd1a7e19b9
buildscripts: remove jenkins-pre.bat 2018-12-11 10:36:58 -08:00
ZHANG Dapeng 3202fcc7d9
examples: move TLS example to a separate project 2018-12-06 12:03:25 -08:00
ZHANG Dapeng 00b4b8870b
examples: move alts bazel and rm alts from pom.xml 2018-12-03 17:12:30 -08:00
Eric Anderson 80883f0983 buildscripts: Reduce verbosity of protobuf builds
A substantial portion of our build logs are just compiling protobuf.
This reduces the noise while still keeping most of the useful
information.
2018-09-21 10:58:07 -07:00
Eric Gribkoff 03dab29e3a
buildscripts: run Android interop tests on Firebase (#4314) 2018-08-28 14:56:06 -07:00
zpencer b8f4c7a57a
buildscripts: disable gae jdk7 test (#4787)
The test has become very flakey recently. With GAE jdk7 going away
completely in early 2019, let's disable running the test in CI.
2018-08-23 17:14:49 -07:00
Eric Anderson 71e9841c80 travis.yml: Fix protobuf compilation caching
Wildcard apparently doesn't work... silently. Move versions into their
own directory so we can use a hard-coded string in the travis
configuration.
2018-08-17 17:27:12 -07:00
Eric Anderson 142851dcbb buildscripts: Use full SHA for protobuf download
Github used to produce tars with the full SHA in the folder name.
However, now they are providing tars with just as much SHA as was in the
initial request. This coincides with a change in its services:

Before:
Resolving github.com... 192.30.253.113, 192.30.253.112
Connecting to github.com|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/23357588/88f90e58-e598-11e7-9263-ef8b1fb345b8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180817%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180817T190158Z&X-Amz-Expires=300&X-Amz-Signature=b684e1ead24a96764fd1f29ac5eb1df693d47c5fe86962088aee2ccfb8e4e72a&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprotobuf-all-3.5.1.tar.gz&response-content-type=application%2Foctet-stream [following]

As of 2:00 PM PT today:
Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112
Connecting to github.com (github.com)|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/google/protobuf/tar.gz/61476b8e [following]

Note that repositories.bzl isn't impacted because it already specified
the full SHA.
2018-08-17 16:59:25 -07:00
Spencer Fang c3c810f5c3 buildscripts: Make kokoro script runnable from workstation
- Only cat /VERSION if it exists
- The host name is used as the version name, and if running locally
  the versions no longer necessarily start with 'kokoro'. Update the
  clean up command.
2018-08-16 15:03:03 -07:00
Spencer Fang 19f4491ce9 buildscripts: Clean up dummy default version check cmd
These statements can be simplified to be more readable.
2018-08-16 15:03:03 -07:00