Commit Graph

387 Commits

Author SHA1 Message Date
Andrea Cirulli c1bd10d6bd unskip dependency check stage
refs DE-808

Change-Id: I1343bb774b0b0138e5d297ac84eb65937dd3264f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/272213
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Tested-by: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
2021-08-25 18:43:44 +00:00
Ryan Norton 556044d649 update post-merge slack failure message
this encourages the behavior of alerting the owning team if spec
flakiness is identified and adds a link to the spec tracker
dashboard for easier investigation

Change-Id: I68ca319866a0aa35bb306b4cb5a2791135669659
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/272131
Tested-by: Andrea Cirulli <andrea.cirulli@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
QA-Review: Ryan Norton <rnorton@instructure.com>
Product-Review: Ryan Norton <rnorton@instructure.com>
2021-08-24 18:01:36 +00:00
Andrea Cirulli 012d7d123c skip linters dependency check for a snyk dependent package issue
npx snyk is failing because a dependency (boolean) is requiring
node v16.7 and we have v14.
There is an open issue in github will re-enable the stage once
the issue is fixed

Change-Id: I22ad7082ac00e41ee1402fbbbbac3d5ce94fce59
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/271814
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Tested-by: James Butters <jbutters@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-08-19 16:16:16 +00:00
Ryan Norton 7e87dd9242 add detail to feature flag linter
this adds some details around who and what is changing when
a user modifies a feature flag

flag=none

Change-Id: I36f6107aa1a0509243c2b169d017cd037451f51a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/271559
Tested-by: James Butters <jbutters@instructure.com>
QA-Review: Ryan Norton <rnorton@instructure.com>
Product-Review: Ryan Norton <rnorton@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
2021-08-17 16:25:16 +00:00
Aaron Ogata f87f9b72e7 report feature flag changes for approval
refs DE-777

Change-Id: I4bfa74ec75894eb93660e2e8ef0e1d75ec51d927
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/269865
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
Reviewed-by: Ryan Norton <rnorton@instructure.com>
Product-Review: Ryan Norton <rnorton@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
2021-07-26 13:40:27 +00:00
Kyle Rosenbaum 10b19ee2b5 remove reporting is_kubernetes to splunk
refs DE-719

test plan:
  - is_kubernetes flag is no longer reported to Splunk

Change-Id: Ie7ef511fe641a76161a44505ff725276983924fc
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/267545
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
QA-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Product-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
2021-06-22 17:58:05 +00:00
Aaron Ogata 3dbf596e03 convert contract tests to EKS-compatible syntax
refs DE-707

Change-Id: I0d823124c25b879ff3c572240720f39ba9371909
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/267009
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-06-15 13:57:51 +00:00
Aaron Ogata 5d33184845 convert linters to EKS-compatible syntax
Test Plan
1. Linters works on EKS
2. Linters works on EC2
3. Linkers runs appropriate stages
4. Yarn changes run appropriate stages
5. Groovy changes run appropriate stages
6. Linters reports back to patchset via gergich

Change-Id: I6c50ab5a6be2e1b7031b56c70ee6e2727e92580e
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/266509
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
2021-06-11 20:04:35 +00:00
Aaron Ogata 9be94c6f23 restructure filesChangedStage for reusablity
refs DE-706

In order to access the buildConfig variables without duplicating constants in other shared library files, restructure this stage to be in its own files, and provide helper methods for getting / setting changed file state.

Test Plan
1. Normal build runs appropriate stages.
2. Local docker dev changes runs appropriate stages.
3. Groovy linter changes runs appropriate stages.
4. Migration changes run appropriate stages.
5. Spec changes run appropriate stages.
6. Yarn changes run appropriate stages.

Change-Id: I72e35d2b45d4842e195cc27351327a6907b073b7
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/266957
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-06-11 18:57:46 +00:00
Aaron Ogata 3400c31a83 convert dependency check stage to using npm
refs DE-706

The snykSecurity() plugin doesn’t seem to work on EKS, fortunately we can convert it to using the npm package instead which also reduces complexity.

Test Plan
1. Dependency check runs and reports results to snyk on EC2
2. Dependency check runs and reports results to snyk on EKS

Change-Id: I317ffe8c90cda6754a2ad5df7227224450b86546
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/266779
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-06-10 14:10:41 +00:00
Aaron Ogata 29e7aed396 merge all JS stages together
refs DE-687

Test Plan
1. All JS test counts align with latest post-merge

Change-Id: I2674f1dd594e561f1fbcc82d856342958ada19cc
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265466
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
2021-05-21 17:09:13 +00:00
Andrea Cirulli 70bb73c35b add jenkins file to the groovy linter stage
refs DE-690
flag = none

test-plan:
- Build passes
- Build trigger groovy linter stage if a change to Jenkinsfile is
made
- Build fails if a Jenkinsfile has linter issues

Change-Id: Ifdabb4f28f6124afcb6621844375a8de4b35d453
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265279
Reviewed-by: James Butters <jbutters@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-05-19 20:51:13 +00:00
Andrea Cirulli 796257539e fix linter rules
Fix following linter issues:
- ParameterName
- CouldBeSwitchStatement
- UnusedMethodParameter

refs DE-690
flag = none

test-plan:
- Build passes
- QA build that triggers FSC and executes the spec changed

Change-Id: I11baf5094e96d33bc6be7f5f03d680f34cc1d026
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265266
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
2021-05-19 15:43:33 +00:00
Andrea Cirulli f4757ca66a fix linter issues
Fix the following linter issues:
- UnusedVariable
- ClosureAsLastMethodParameter
- CatchException
- UnnecessaryGString
- ImplicitClosureParameter

test-plan:
- Build passes
- FSC qa build passes and it executes the changed test

Change-Id: I9f814e8f3466cd6a271a3b24afc410c32fc66022
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265218
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
2021-05-18 20:11:48 +00:00
Andrea Cirulli 3f057e9c0c fix jenkins files linter issues
Fix linter issues related to variable name, space after closing bracket.

refs DE-690
flag = none

test-plan:
- Build passes
- QA build with spec change executes fsc stage

Change-Id: I6f985fb3dfa8627703a27325b1eb25fae4585505
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265188
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-05-18 18:58:33 +00:00
Andrea Cirulli 67d6ccc360 fix jenkins file linter issues autofixable
refs DE-690
flag = none

test-plan:
- Build passes

Change-Id: Ie748a1e585ce8e3ec4978c7a5faa45dab524a905
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265184
Reviewed-by: James Butters <jbutters@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2021-05-18 16:48:15 +00:00
Andrea Cirulli 088c8f6200 fix yarn linter not being triggered
After the restructure of the Jenkinfile the yarn linter
is not triggered when a yarn files is changed. Add the yarn files
to the files changed detect stage

refs DE-688
flag = none

test-plan:
- Build passes and does not trigger yarn linter
- QA Build with a yarn change trigger yarn linter

Change-Id: If8eb0760dceff05c86a023ac8e88063000d69cfe
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265101
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-05-17 20:13:22 +00:00
Andrea Cirulli 7a01e4977e add groovy linter stage
refs DE-643
flag = none

Add a linter stage for groovy to the main build:

- Build passes
- Build passes and trigger groovy linter stage if no
linter errors are present
- Build fails on groovy linter stage if a linter issues
is found

Change-Id: I8f3c2f2f577227854e19b1af83400018a4c792db
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265024
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-05-17 19:43:03 +00:00
Aaron Ogata 9e10e34935 prepare rspec / selenium stage generation structure
refs DE-444

Modify the generation of these stages to align with our new extendedStage() conventions in preparation for a refactor of how the stage internals are implemented.

Test Plan
1. All RSpec stages are queued and run the correct tests
2. All Selenium stages are queued and run the correct tests
3. Parallel log build works correctly

Change-Id: Id0a35f84ddf8471f1f06eede608d6ad4dc964dea
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264870
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
2021-05-13 14:23:55 +00:00
Aaron Ogata 31096f13bf always use EC2 node for Builder stage
refs DE-670

For the initial iteration of the EKS project, we’re going to keep the Docker Build process on EC2 in order to avoid using docker-in-docker and its performance implications.

Test Plan
1. use-kubernetes flag continues to use EKS nodes, except for Builder stage which should use EC2
2. normal build continues to use EC2 nodes

[canvas-builds-refspec=c923c731de7a19030fd992ea34cbc47293e4a01d]

Change-Id: Ib7c95ca23e1bd87efcb111a4abbb081ed44f2c2a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264667
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-05-11 15:31:55 +00:00
Aaron Ogata 9fac78f6f1 move Detect Files Changed stage after Rebase
refs DE-670

As part of our work with the Linters stage, an issue was revealed where the build scripts would be stashed for other nodes -before- the rebase happens, causing that build to use the outdated scripts. Move Detect Files Changed to be after Rebase to fix this.

Test Plan
1. Detect Files Changed still works
2. Commits based off code before the Linters change run correctly.

Change-Id: I3b33a03a02e9374f79810ab934d0dc3270a5e867
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264659
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-05-11 14:44:47 +00:00
Aaron Ogata 35a01d7e8c move dependency check stage to linters node
refs DE-670

For the EKS transition, we need to ensure that the Linters stage doesn’t run on the EC2 builder node and instead properly uses EKS containers. As part of this work, transition the dependency check stage to running as part of the Linters set of stages.

Test Plan
1. Dependency Check runs correctly on post-merge build
2. Linters runs correctly on pre-merge build

Change-Id: I16d9fb63b40b6eb9f8e24b4f5c4253538e91bbf0
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264653
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-05-11 14:25:37 +00:00
Aaron Ogata db761b4f69 Revert "move dependency check stage to linters node"
This reverts commit 88829e1074.

Reason for revert: broke post-merge

Change-Id: I91384fd9c2a698eaabe02d1c63a959b8846a749b
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264016
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Tested-by: Aaron Ogata <aogata@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-05-10 23:47:19 +00:00
Aaron Ogata 88829e1074 move dependency check stage to linters node
refs DE-670

For the EKS transition, we need to ensure that the Linters stage doesn’t run on the EC2 builder node and instead properly uses EKS containers. As part of this work, transition the dependency check stage to running as part of the Linters set of stages.

Test Plan
1. Dependency Check runs correctly on post-merge build

Change-Id: Ifc9790e481e7d6cf7ba81e9f78b804a955f266b8
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264589
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-05-10 23:32:31 +00:00
Aaron Ogata f8f556d866 split linter stages into own node
refs DE-670

For the EKS transition, we need to ensure that the Linters stage doesn’t run on the EC2 builder node and instead properly uses EKS containers. As part of this work, transition linter stages into their own node as a sibling stage to Builder.

Test Plan
1. Linters run correctly
2. allow-stages directive works with Linters
3. allow-stages directive works without Linters

Change-Id: Ieae121ae3e79b5e30864f9060042e4453a8093e6
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264569
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-05-10 21:31:15 +00:00
Aaron Ogata 8f2885814c merge master bouncer into Linters parallel stages
refs DE-670

For the EKS transition, we need to ensure that the Linters stage doesn’t run on the EC2 builder node and instead properly uses EKS containers. As part of this work, transition Master Bouncer to using the Linters image since it uses Git.

Test Plan
1. Master Bouncer runs correctly

Change-Id: I8aa6d2f589326c4f88e313d423f023ec9af8a79f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264555
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-05-10 20:04:09 +00:00
Aaron Ogata c16a81dd02 always upload Linters image to starlord
refs DE-670
refs DE-500

For the EKS transition, we need to ensure that the Linters stage doesn’t run on the EC2 builder node and instead properly uses EKS containers. To do this, the Linters stage needs to always upload its image so that EKS can pull it.

Test Plan
1. Linters properly reports to the correct gerrit for main builds
2. Linters does not become the runtime bottleneck

Change-Id: Ibe28d47c13f965ee468b49ce6a74e3d08b75715d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264549
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-05-10 19:25:44 +00:00
Aaron Ogata fd6c9f8bcd send is_kubernetes flag to Splunk
refs DE-674

Start tracking Kubernetes enabled / disabled state separately on Splunk dashboards.

Test Plan
1. The value is correctly sent when builds run

Change-Id: I15f64f635f420c8b556166dc533f3a9ce8f5f58e
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264509
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-05-10 14:39:41 +00:00
Aaron Ogata 70b06aaa0b apply autoformatter to all files
refs DE-643

Change-Id: I9482afa27e6b8734640741fca784f971f782af26
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/264050
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2021-05-03 14:07:45 +00:00
Aaron Ogata 58bd7e4589 treat any non-SUCCESS status as failed
refs DE-656

ExtendedStage supports more alternative statuses than FAILED / SUCCESS now, such as NOT_BUILT, ABORTED_USER and ABORTED_PATCHSET. All of these need to be treated as FAILED conditions.

Test Plan
1. All patchsets start appearing in #canvas_builds-noisy

Change-Id: I0eae928a7d232c730e7c552336c216c09d62bfef
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263797
Reviewed-by: Ryan Norton <rnorton@instructure.com>
Product-Review: Ryan Norton <rnorton@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
2021-04-27 22:41:52 +00:00
Kyle Rosenbaum 1e43e05241 add podTemplate and container definition to all applicable stages
refs DE-558, DE-582

[use-kubernetes]
[canvas-builds-refspec=24d3d83e8034ac6ad5380936498e1dcf01ab50eb]

modify the existing nodeRequirements to add podTemplate and
container definitions. initially, the podTemplate to be used
is docker-in-docker. the "use-kubernetes" flag must be set
as well.

test plan:
- build executes on k8s nodes in EKS when "use-kubernetes"
flag is set.
- build executes on EC2 when "use-kubernetes" flag is not set.

Change-Id: I13c68e5420bbf414532518eaf69ac82f9310bfbf
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263257
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
QA-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Product-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
2021-04-27 19:47:25 +00:00
Aaron Ogata 8e88be86eb report unknown failure exceptions to Slack
refs DE-630

In order to start auto-killing bad nodes, we’ll need a list of exceptions that represent that state. Start collecting them in #canvas_builds-noisy.

Test Plan
1. Common failure cases don’t report to Slack

[canvas-builds-refspec=e4b78afbb94053b352b1ad7ef514ac1efdcf1558]

Change-Id: I93e57cfe49f4ad190769a101765cd870a6d16ca5
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263553
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-26 13:25:59 +00:00
Aaron Ogata 4afde2cb63 split linters stage into multiple stages
refs DE-628

Test Plan
1. Ensure that code / webpack always run
2. Ensure that yarn linters run only when package.json or yarn.lock changes
3. Ensure that build summary report works when each stage fails

[canvas-builds-refspec=f2c74ae95319648ac9fe19f306d7067adda9c534]

Change-Id: I64eb9f794e00737b2f0b87a1d962a4aaaeb174d2
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263365
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
2021-04-22 18:52:00 +00:00
Aaron Ogata 5ad6a24f34 split lintersStage setupNode / tearDownNode functions
refs DE-628

In order to improve the build summary report UX and also to increase compatibility with podTemplate / container, split the setup / tear down pieces of the Linters stage to prepare for moving each piece to its own Jenkins stage.

Test Plan:
1. All linters run and fail if error is found
2. Linters post status to gerrit
3. Plugin build Linters works

Change-Id: I4c4769eed80f201532edc34907f2428ceb465773
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263323
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-04-21 15:07:24 +00:00
Aaron Ogata 091df4a450 only process sub-job results if RunWrapper exists
refs DE-633

When a sub-job is cancelled, it won’t return a RunWrapper, so it can be null here.

Change-Id: Ia49b81268d6548c58ce9ba35d38175c5fd76705d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263295
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-20 22:42:56 +00:00
Kyle Rosenbaum 16469f7cfa add support for EKS container to protectedNode requirements
refs DE-558, DE-582

[canvas-builds-refspec=f9c80c0ac4b80de60ac50633523e30d5c2eb28b1]

modify the existing nodeRequirements call to pass a Map containing
the label as well as the possibility of container or podTemplate
params. this change only modifies the existing nodeRequirements to
continue using EC2 agents, not EKS.

test plan:
- builds still work as expected acquiring nodes

Change-Id: I0458a13a4c85e37aeb2f5dc56347e27101705646
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263208
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Product-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
2021-04-20 21:51:16 +00:00
Aaron Ogata 042f00b4d6 unify failed stage parsing logic
refs DE-619

Merge the current logic for handling the main build failures with the logic for sub-jobs to reduce complexity and ensure consistent handling.

Test Plan:
1. Ensure sub-job test results appear
2. Ensure rspec / selenium failure stages appear
3. Ensure sub-job failure stages appear
4. Ensure parent stages do not appear

[canvas-builds-refspec=6e70b5b572e9297f1cd06020ec61a551e7c0d323]

Change-Id: I7410cec5340714b3617a310f83549011a2a0ce0a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262979
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
2021-04-19 23:09:35 +00:00
Aaron Ogata 347c800d3f switch from on-demand to spot executors for primary node
refs DE-338

EKS requires that we stop using on-demand instances for any of our build nodes. Now that the pre-requisite work is done to allow this change, actually switch it over.

Test Plan
1. Spot instance termination of primary node works when Linters job is running

Change-Id: Idac911c2c8266e30dc9cb8f0b7478fc09c8aa543
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263160
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-19 17:07:37 +00:00
Aaron Ogata ef3b427766 fix stable branch building
Change-Id: Ie0569a6e9600f2fb73d931e93edf2c72468ae2fb
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263024
Reviewed-by: Ryan Norton <rnorton@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2021-04-15 21:19:37 +00:00
Aaron Ogata 5d1fb9b734 convert on*Result -> onStageEnded
refs DE-628

Test Plan:
1. Build summary report works for sub-jobs and normal stages
2. Sub-jobs run cleanupFn

[canvas-builds-refspec=82ccf7ae185871e45c658f225cc664ef05cda74d]

Change-Id: I9d4476b75e6488bcc388a89e9f7cf87150d0ae6d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262951
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-15 13:54:20 +00:00
Aaron Ogata 3027dcdbd7 rename handler to hooks
refs DE-628

[canvas-builds-refspec=a5aca2425b1f885f5fcb4298479c0081d73065ff]

Change-Id: I0581d820fc521fb4857c04a28804be1c9a2d19a5
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262947
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-14 22:50:58 +00:00
Aaron Ogata b3f6ee3b54 fix allow-stages directive
refs DE-621

Change-Id: I77e620defe993e37425cfd1f6e9c78d2574fb922
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262904
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-14 17:37:00 +00:00
Aaron Ogata 1a2b0052f1 convert extendedStage -> protectedNode to nodeRequirements() pattern
refs DE-621

Make the stage configuration aware of its node requirements for future EKS support and allowing nospot to be removed.

Test Plan:
1. RSpec / Selenium build works
2. Contract Tests works
3. JS works
4. FSC works
5. Vendored Gems works

[canvas-builds-refspec=480f3570a6a6ebee5cedaecbb3a089476a008215]

Change-Id: I6cb63df20abe8c1f913db4f1b21bb01bd4d30312
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262898
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-14 15:23:01 +00:00
Aaron Ogata de7559a9c2 avoid cps error when calling post actions
refs DE-624

Jenkins CPS transformation doesn’t correctly transform the Closure inside of the Expando object, but we can use a Map instead that has the same functionality (the property can be accessed using the dot operator).

Test Plan:
1. Build summary report works

Change-Id: Ic3446afa20f6c6ce5aac95e5e0cda06edfa39ef5
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262844
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-13 20:52:11 +00:00
Aaron Ogata e18887bf26 move docker cleanup to protectedNode() handler
refs DE-624

As part of making extendedStage() aware of stage node requirements, we need to remove consumers of the cleanupFn() functionality and replace them with extendedStage() compatible patterns.

Test Plan:
1. All stages execute correctly

[canvas-builds-refspec=07f1a75ee23f8f6a9a1bbe062316373b761e7a30]

Change-Id: Ica11b78ebead5c9239dea112677a077654c7090d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262814
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
2021-04-13 18:02:41 +00:00
Aaron Ogata c7ad6e2939 clean-up postFn caller implementation
refs DE-338

Test Plan:
1. Build succeeds when all stages pass, and uploads a summary report.
2. Build fails when any stage fails, and uploads a summary report.

Change-Id: I98d9ce4507ed805caad4fe4857aa9ecef9aceaf9
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262780
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
2021-04-13 14:45:34 +00:00
Aaron Ogata 2d7bf0e26f notify noisy channel on build success / failure
refs DE-620

It can be helpful for monitoring builds after potentially breaking changes are made to have a list of build success / failure in one place.

Change-Id: I5b06d3221e8292740fdcccdb4e1633f9ac30177a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262764
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-13 13:50:09 +00:00
Aaron Ogata f85b577b0e move rspec / selenium stages to root level
refs DE-338

Nospot removal requires all non-build stages to not be wrapped in the protectedNode() function.

Test Plan:
1. RSpec / Selenium stages pass
2. Build Summary Report works correctly

Change-Id: I313b865c1e2758177446905b62a5c7c6f1ed1830
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262748
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-12 21:10:50 +00:00
Aaron Ogata f89a7d81c1 fix naming of db image cache vars
refs DE-338

Eliminate the need for the withEnv() indentation by consistently naming all of these variables.

Test Plan:
1. Migrations caching works correctly

Change-Id: Iaef43c4e4278878e17b90e2190dc0d7a0a42cb26
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262743
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-12 20:17:54 +00:00
Aaron Ogata 18d8422aec actually execute post-build functions after entire build
refs DE-338

After we started moving stages out of the nospot block, we forgot to move the post-actions function out of that block as well. This results in postFn() being called too early.

Test Plan:
1. Build summary report actually captures all stages.

Change-Id: I743827c4f35682e8887918c9041ed833e51ad2f3
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262712
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2021-04-12 17:56:46 +00:00