Commit Graph

256 Commits

Author SHA1 Message Date
Sergii Tkachenko 1680ada2fd
buildscripts: Use the Kokoro shared install lib from the new repo (#10757)
Source: 4f7ead2c7b/tools/internal_ci/linux/grpc_xds_k8s_install_test_driver.sh
New repo: https://github.com/grpc/psm-interop
New path: `.kokoro/psm_interop_kokoro_lib.sh`
2023-12-15 15:27:58 -08:00
Eric Anderson 43e06372ec kokoro: Build android-interop-testing and binder separately
This avoids an OOM. We could probably configure things to run them
together, but that generally takes trial-and-error.
2023-11-27 10:15:42 -08:00
Eric Anderson a70dddd3dc kokoro: Use Firebase's arm virtual Android devices
They are a lot faster. Instead of 1-3 minutes of test execution, I now
see 2-22 seconds. There still may be 3 minutes of overhead for the
gcloud command to complete, but the reduction is noticable in the total
execution time. And it seems the tests are actually being run, as there
is some flakiness. The flakiness appears to be at a lower rate.

The script was slightly reorganized to make it easier to copy commands
to run locally.
2023-11-27 08:22:52 -08:00
Eric Anderson 0567ab5773 android-interop: Add testing for API levels 28-30
Note that this uses Pixel2 and Pixel3. Also swap 26-27 from Nexus6P to
Pixel2. We tend to prefer the latest (virtual) device for each API
level.

The current models and their supported API levels are available via:
```
gcloud firebase test android models list --filter=form=virtual
```

Pixel2.arm supports 31-32, but is beta, so I didn't swap to it. It also
supports the preview 33.
2023-11-22 14:49:19 -08:00
Terry Wilson 26be0c7665
buildscripts: add missing $GRADLE_FLAGS to enable AndroidX (#10504) 2023-08-17 16:32:23 -07:00
Terry Wilson beaeeefdd2
examples: Android examples to use AGP 7.4.0 (#10497)
- Upgrades AGP version to 7.4.0
- Upgrades appcomat to use AndroidX
- Updates the `android.sh` build script to set the `android.useAndroidX` flag
2023-08-17 11:24:42 -07:00
Zach Reyes 6d48271fa4
buildscripts: Add bootstrap generator to cron job that runs nightly (#10440) 2023-08-02 13:04:56 -07:00
Sergii Tkachenko 419767f968
Upgrade dependencies post v1.57.x branch cut (#10359)
### Dependency updates

#### Update successfully to the latest
- [x] `androidx.core:core 1.10.0 -> 1.10.1`
- [x] `com.google.api.grpc:proto-google-common-protos 2.17.0 -> 2.22.0`
- [x] `com.google.cloud:google-cloud-logging 3.14.5 -> 3.15.5`
- [x] `com.google.truth:truth 1.0.1 -> 1.1.5`
- [x] `com.puppycrawl.tools:checkstyle 8.28 -> 10.12.1`
- [x] `org.robolectric:robolectric 4.9.2 -> 4.10.3`
- [x] Auto-value
   - [x] `com.google.auto.value:auto-value 1.10.1 -> 1.10.2`
   - [x] `com.google.auto.value:auto-value-annotations 1.10.1 -> 1.10.2`
- [x] Protobuf
   - [x] `com.google.protobuf:protobuf-java 3.22.3 -> 3.23.4`
   - [x] `com.google.protobuf:protobuf-java-util 3.22.3 -> 3.23.4`
   - [x] `com.google.protobuf:protobuf-javalite 3.22.3 -> 3.23.4`
   - [x] `com.google.protobuf:protoc 3.22.3 -> 3.23.4`
- [x] Errorprone
   - [x] `com.google.errorprone:error_prone_annotations 2.18.0 -> 2.20.0`
   - [x] `com.google.errorprone:error_prone_core 2.18.0 -> 2.20.0`
   - ~`libs.checkstylejava8 = com.puppycrawl.tools:checkstyle 9.3 -> 10.12.1`~ -- pinned to last version supporting java8, update not needed

#### Updated to non-latest
- [x] `com.squareup.okio:okio 1.17.5 ->` ~`3.4.0`~ `2.10.0` -- updating to 3.x failed due to them [introducing gradle multiplatform artifacts](https://github.com/square/okio/blob/master/CHANGELOG.md#version-320). Error in [the comment below](https://github.com/grpc/grpc-java/pull/10359#issuecomment-1632853307.).
- [x] `org.checkerframework:checker-qual 3.33.0 ->` ~`3.36.0`~ -- removed, no longer needed
- [x] Mockito - updated to `4.11.0`. Versions 5.x break some tests. Errors in [the comment below](https://github.com/grpc/grpc-java/pull/10359#issuecomment-1632834435).
   - `org.mockito:mockito-android 3.12.4 ->` ~`5.4.0`~ `4.11.0`
   - `org.mockito:mockito-core 3.12.4 ->` ~`5.4.0`~ `4.11.0`

#### Not updated
- Cronet -- upgrade failed, created a blocker ticket https://github.com/grpc/grpc-java/issues/10396
   - ~`org.chromium.net:cronet-api 108.5359.79 -> 113.5672.61`~
   - ~`org.chromium.net:cronet-embedded 108.5359.79 -> 113.5672.61`~
- Google auth -- upgrade blocked by https://github.com/grpc/grpc-java/pull/9118
   - ~`com.google.auth:google-auth-library-credentials 1.4.0 -> 1.19.0`~
   - ~`com.google.auth:google-auth-library-oauth2-http 1.4.0 -> 1.19.0`~
- Guava -- not upgraded: 32.1.0 [broke gradle modules](https://github.com/google/guava/releases/tag/v32.1.0)
   - ~`libs.guava = com.google.guava:guava 32.0.1-android -> 32.1.1-jre`~
   - ~`libs.guava.jre = com.google.guava:guava 32.0.1-jre -> 32.1.1-jre`~
   - ~`libs.guava.testlib = com.google.guava:guava-testlib 32.0.1-android -> 32.1.1-jre`~


### checkForUpdates results
```
> Task :checkForUpdates
androidx.core:core 1.10.0 -> 1.10.1
com.google.auto.value:auto-value 1.10.1 -> 1.10.2
com.google.auto.value:auto-value-annotations 1.10.1 -> 1.10.2
org.checkerframework:checker-qual 3.33.0 -> 3.36.0
com.puppycrawl.tools:checkstyle 8.28 -> 10.12.1
org.chromium.net:cronet-api 108.5359.79 -> 113.5672.61
org.chromium.net:cronet-embedded 108.5359.79 -> 113.5672.61
com.google.errorprone:error_prone_annotations 2.18.0 -> 2.20.0
com.google.errorprone:error_prone_core 2.18.0 -> 2.20.0
com.google.errorprone:error_prone_core 2.10.0 -> 2.20.0
com.google.api.grpc:proto-google-common-protos 2.17.0 -> 2.22.0
com.google.auth:google-auth-library-credentials 1.4.0 -> 1.19.0
com.google.auth:google-auth-library-oauth2-http 1.4.0 -> 1.19.0
com.google.cloud:google-cloud-logging 3.14.5 -> 3.15.5
com.google.guava:guava 32.0.1-android -> 32.1.1-jre
com.google.guava:guava 32.0.1-jre -> 32.1.1-jre
com.google.guava:guava-testlib 32.0.1-android -> 32.1.1-jre
org.mockito:mockito-android 3.12.4 -> 5.4.0
org.mockito:mockito-core 3.12.4 -> 5.4.0
com.squareup.okio:okio 1.17.5 -> 3.4.0
com.google.protobuf:protobuf-java 3.22.3 -> 3.23.4
com.google.protobuf:protobuf-java-util 3.22.3 -> 3.23.4
com.google.protobuf:protobuf-javalite 3.22.3 -> 3.23.4
com.google.protobuf:protoc 3.22.3 -> 3.23.4
org.robolectric:robolectric 4.9.2 -> 4.10.3
com.google.truth:truth 1.0.1 -> 1.1.5
```
2023-07-21 10:15:05 -07:00
Eric Anderson d654707838
buildscripts: Build Android with main build in linux_artifacts
Note that this changes the JDK used to compile releases to Java 11. That
should only impact the appearance of the Javadoc.

This adds the Android SDK to the build container, removing the
dependency on the Android SDK being available on the CI host. This
allows running on newer Kokoro images. 'Android' and 'Android interop'
CIs still depend on the Android SDK being available on the host, but
since they aren't used as part of the release process, they can more
easily migrate off Kokoro as part of future work.

This also causes Android components to now be built with -Werror, as we
use -PfailOnWarnings=true in unix.sh but were missing it from the
Android build invocations.

Gradle will auto-download the necessary version of build-tools. We don't
want to download it ourselves because the version we specify might not
even be used. Looking at logs, we were previously downloading a version
that was unused.

We now fork javac to avoid OOM. The build fails 2/3 times before the
forking, and 0/3 after.
2023-06-28 16:24:21 -07:00
Eric Anderson 0e6c01837c buildscripts: Fix prepare_qemu to not assume system already has emulation
Apparently our Kokoro image has this done already, and my laptop as
well. But the newer Kokoro image and other computers like my desktop
don't have it already.
2023-06-23 10:24:25 -07:00
Xuan Wang 1c9ce5d76b
[PSM interop] Don't fail url_map target if sub-target already failed (#10295) 2023-06-22 15:42:21 -07:00
Eric Anderson d28e5f0fc5 buildscripts: Fail sonatype-upload for non-200 HTTP status code 2023-06-15 13:35:00 -07:00
Eric Anderson 0f2c43a8fd Download Maven from Maven Central
Maven deleted older binaries from their CDN, so the download was
failing. Maven Central seems it'll be more stable.
2023-06-09 10:32:12 -07:00
Xuan Wang 02ac6f4393
buildscripts, psm interop: Don't fail target if sub-target already failed
We configured TestGrid to file bug separately for each 
failed sub-target, if we still fail the main target, 
TestGrid will fail duplicate bugs.

The same change in core: 
https://github.com/grpc/grpc/pull/33222.
2023-05-31 17:37:46 -07:00
Stanley Cheung 4277867a7d
gcp observability: Update Docker images to eclipse-temurin (#10218)
Perform software update so that we install patches for latest
vulnerabilities. Similar to the fix in #10191
2023-05-30 08:03:36 -07:00
Sergii Tkachenko 16b84924a1
xds interop: Update Docker images to `eclipse-temurin:11-jre` (#10172)
- Update to xDS Test Client and xDS test server Docker images to `eclipse-temurin:11-jre`. 
- Perform software update so that we install patches for latest vulnerabilities.
2023-05-16 15:20:55 -07:00
Terry Wilson 0005029328
android,binder,cronet: .aar file when publishing (#10138) 2023-05-05 15:21:26 -07:00
Sergii Tkachenko c9fcd46e84
buildscripts: increase PSM Security test timeout to 4h (#10065)
The total time PSM Security takes to run was getting close to 3h lately.
Adding an extra hour to allow for a larger margin of error.

ref b/259690410
2023-04-19 10:35:12 -07:00
Eric Anderson 35852130d9 buildscripts: Cross-compile in Docker
Previously builds were done with Ubuntu 16.04, and now we are using
18.04. Thus the generated binaries will no longer work for
Ubuntu 16.04 and Debian 9 users, both of which are outside of their
support window and aren't supported by Abseil. RHEL users are
unaffected, as the binaries already didn't work on RHEL 7 and they will
remain working with RHEL 8. FWIW, Ubuntu 18.04 will leave its support
window in June.
2023-04-18 16:16:35 -07:00
Dirk Haubenreisser 99cbdd5d69
Add support for cross-compiling for s390x platform (#9455)
* Added s390x platform support
* Adapt to existing platform naming scheme
* Updated s390_64 library whitelist
* Use g++ compiler version 8.x for s390x
* Introduced dedicated Docker container for building s390x artifacts Minor fix

---------

Signed-off-by: Dirk Haubenreisser <haubenr@de.ibm.com>
Co-authored-by: Eric Anderson <ejona@google.com>
2023-03-23 13:21:31 -07:00
yifeizhuang c1ff4a856d
buildscript: iterate all example folder and build (#9961) 2023-03-17 15:09:58 -07:00
Stanley Cheung cc03b480b8 Pass interop parameters to each langs run.sh as-is. run.sh should just pass through to the interop client/server binaries 2023-03-10 10:34:11 -08:00
Terry Wilson 7ecb909776
buildscripts: Use Java 11 with android-interop (#9943) 2023-03-09 09:33:10 -08:00
Terry Wilson b5b7cacba9
Upgrade Android Gradle plugin to 7.4.0 (#9933)
This is the latest version of the plugin supported by the Gradle version
in use at the moment (7.6).

Note that this also upgrades the R8 optimizer to a version (4.0.48) that
now uses "full mode" optimization by default.

This also splits off Android projects to run under Java 11 (Gradle
plugin requirement) while the other projects continue to run under Java
8.
2023-03-08 11:18:27 -08:00
Eric Anderson 4229191a23 Move gcp-observability interop binary out of interop-testing
gcp-observability has many dependencies so is a bit annoying in some
build systems to get working... just for it not to be used in
non-observability scenarios.

grpc-go and c core are using separate binaries for gcp-observability
interop testing, so do the same in Java, which makes interop-testing a
bit lighter.
2023-03-07 10:57:38 -08:00
Stanley Cheung b55ecd5c35
GCP Observability Interop Testing Client/Server for Java (#9858) 2023-03-06 08:24:42 -08:00
Eric Anderson 43917f052e Use Apache CDN to download Maven
Maven seems to have improved their download management and instead of
having their webpage choose a host they now have a CDN domain.
apache.cs.utah.edu is slow and is failing to finish the downloading.
2023-02-02 16:42:06 -08:00
Sergii Tkachenko f2533f4fd8
xds interop: Fix buildscripts not continuing on a failed test suite (#9833)
Apparently there's a difference between bash 3 and bash 4.
OSX comes with bash 3 out-of-box, so for whoever wrote this logic
it "worked on my machine".

The `((` construct returns a 0 exit code if the value is non-zero.
Since the value starts at 0 and we do a post-increment,
it will always fail the first time.
Changing it to a pre-increment should fix it.
2023-01-17 13:15:40 -08:00
Sergii Tkachenko 42ba00ba65
Revert "xds interop: Fix buildscripts not continuing on a failed test suite (#9817)" (#9831)
This reverts commit d83a599c97.

Reverted in favor of better syntax suggested here: https://github.com/grpc/grpc-node/pull/2323#pullrequestreview-1248043432. This fix will be sent as another PR for the convenience of backporting.
2023-01-17 11:36:31 -08:00
Sergii Tkachenko d83a599c97
xds interop: Fix buildscripts not continuing on a failed test suite (#9817)
Apparently there's a difference between bash 3 and bash 4.
OSX comes with bash 3 out-of-box, so for whoever wrote this logic
it "worked on my machine".
2023-01-13 12:42:45 -08:00
Eric Anderson ec5bc6b1eb Drop Bazel 4 support
Bazel 6 has been released, and we track the two most recent major
versions.
2023-01-11 08:15:47 -08:00
Eric Anderson d17a2db4bd Upgrade to Checkstyle 8.28
Trying to upgrade Gradle to 7.6 improved the checkstyle plugin such that
it appears to have been running in new occasions. That in turn exposed
us to https://github.com/checkstyle/checkstyle/issues/5088. That bug was
fixed in 8.28, which also fixed lots of other bugs. So now we have
better checking and some existing volations needed fixing. Since the
code style fixes generated a lot of noise, this is a pre-fix to reduce
the size of a Gradle upgrade.

I did not upgrade past 8.28 because at some point some other bugs were
introduced, in particular with the Indentation module. I chose the
oldest version that had the particular bug impacting me fixed. Upgrading
to this old-but-newer version still makes it easier to upgrade to a
newer version in the future.
2023-01-05 17:07:04 -08:00
panxuefeng f40231e147
buildscripts: Enhancement for LoongArch64 (#9778) 2023-01-04 09:11:23 -08:00
Sergii Tkachenko f4cda008ee
buildscripts: drop xDS v2 interop test (#9783)
This disables xDS v2 support interop test.
xDS v2 support dropped in https://github.com/grpc/grpc-java/pull/9760.

Internal ref cl/499306755
2023-01-03 15:37:12 -08:00
Eric Anderson b118e00cf9
Use Protobuf 21.7 in our code generator
This aligns the C++ version we're using for gRPC-generated code with the
Java version. This should have no real impact to our users, as there
were no features added to .proto files or the like that would be visible
to users.
2022-10-04 10:22:52 -07:00
Terry Wilson bf692c04e6
Add custom_lb to the test suites. (#9502)
Co-authored-by: Sergii Tkachenko <hi@sergii.org>
2022-09-06 15:49:49 -07:00
Sergii Tkachenko c8c3d3d6b2
xDS interop: enable pod log collection in the buildscripts (#9495)
- Enables pod log collection in all PSM interop jobs implemented
  in https://github.com/grpc/grpc/pull/30594.
- Associate test suite runs with their own log file, so it's displayed
  on the "Target Log" tab
- Updates security job to not stop after a failed suite, so that
  authz_test run even if security_test failed
- Fix run_test not returning correct exit status, causing false
  positives in some cases. See https://github.com/grpc/grpc/pull/30768
2022-09-06 11:18:27 -07:00
Terry Wilson 1dd764a3a1
buildscripts: Add outlier_detection_test interop test. (#9461) 2022-08-22 14:27:20 -07:00
yifeizhuang 58cd6e1a7f
example: fix orca example to use new ORCA API (#9403) 2022-07-22 07:40:49 -07:00
Sergii Tkachenko 4aa9b92551
buildscripts: Fix kube contexts in the xds LB tests (#9389)
- The primary should've been `GKE_CLUSTER_PSM_LB`
- The secondary cluster was not activated for LB tests. This resulted
  in the failover test failing, as it relies on workloads running in
  different zones.
2022-07-18 17:40:46 -07:00
Sergii Tkachenko 10449d63bc Revert "buildscripts: Add missing secondary_kube_context to xds LB tests (#9380)"
This reverts commit e3e152a449.
2022-07-18 15:28:35 -07:00
Sergii Tkachenko e3e152a449
buildscripts: Add missing secondary_kube_context to xds LB tests (#9380)
Secondary cluster was not activated for LB tests. This resulted in the failover test failing, as it relies on workloads running in different zones.

ref b/238226704
2022-07-15 16:03:48 -07:00
Sergii Tkachenko d7a6c1ea31
xds: Allow Gradle to use more memory when building interop - GCE (#9354)
Same as #9347, but for GCE framework too (xds and xds_v3 jobs).

Should fix "Expiring Daemon because JVM heap space is exhausted".

PR #9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238334438
2022-07-08 19:28:23 -07:00
Eric Anderson 5f9ef98173 xds: Allow Gradle to use more memory when building interop
Should fix "Expiring Daemon because JVM heap space is exhausted".

https://github.com/grpc/grpc-java/pull/9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238334438
2022-07-08 12:59:27 -07:00
Eric Anderson 3de7e74c57
xds: Build third-party protos in separate build step
This dramatically shortens build time, even for full builds. A full
assemble of xds on my laptop goes from 1m 46s to 33s at least because
errorprone is disabled for the protos.
2022-07-07 07:26:38 -07:00
Sergii Tkachenko 91fcc33243
buildscripts: Fix Xmx JVM flag propagation in GRADLE_OPTS
* buildscripts: Fix Xmx JVM flag propagation in GRADLE_OPTS
* buildscripts: double Java memory allocation pool

To reduce periodic OOMs of the "GitHub Actions Linux Testing / tests (11) (pull_request)" job.
2022-06-30 14:39:03 -07:00
Eric Anderson c0790283ec
Bump protobuf to 3.21.1 (#9311)
Fixes #9264
2022-06-30 11:18:49 -07:00
sumitd2 04f9bdc1fd
compiler: Cross-compile for ppc64le platform (#9284)
Fixes #9274
2022-06-23 09:18:19 -07:00
Sergii Tkachenko 12984db6a7 kokoro: standard TESTING_VERSION in k8s framework build scripts
This Addresses the issue with skips not working due to the
missing/inconsistent `--testing_version` flag, ref b/235688697.

1. Uses the new `TESTING_VERSION` variable populated in the shared
grpc_xds_k8s_install_test_driver.sh - new approach for detecting
versions applicable to all languages.
2. Use `TESTING_VERSION` in all build files in `--testing_version` and
when tagging docker images. This will be backported to all active
test branches. Build Scripts in all other languages will be updated
as well.

Corresponding grpc core change: https://github.com/grpc/grpc/pull/30027
2022-06-21 09:08:24 -07:00
Lidi Zheng d6a23e4116 interop-test: Remove duplicated GCE tests 2022-05-25 15:56:29 -07:00