Also remove the engine experiment, because that never really went
anywhere for canvas and was just one more "different" thing
[fsc-max-nodes=15]
[fsc-timeout=45]
Change-Id: Ib990deb94bf5e25d587d554f64c535c9b96cce26
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/314469
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Isaac Moore <isaac.moore@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Build-Review: Aaron Ogata <aogata@instructure.com>
Migration-Review: Aaron Ogata <aogata@instructure.com>
QA-Review: Jacob Burroughs <jburroughs@instructure.com>
Product-Review: Jacob Burroughs <jburroughs@instructure.com>
refs DE-966
flag=none
TEST PLAN:
ensure build completes and all tests run successfully
Change-Id: I82ef402eb790fafce53393b17b4695c9151e1745
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/281489
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ben Rinaca <brinaca@instructure.com>
QA-Review: Bobby Buten <bobby.buten@instructure.com>
Product-Review: Bobby Buten <bobby.buten@instructure.com>
refs FOO-2423
flag=none
TEST PLAN:
1) run pulsar specs lots of times with many seeds
2) they don't fail.
Change-Id: Ide7fbaebee5ddac37bf2db03d8699f32d1b57d56
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/275315
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
refs FOO-2423
flag=none
Patch multiple sources of unintentionally shared
state between specs, and reduce
intentional wait periods in
the pulsar consumer process.
TEST PLAN:
1) execute pulsar tests in vendor gems build
2) runtime is cut down to less than 4 minutes
Change-Id: I9a4d990b01f2c20ea78f4a59926dffbd91d3b32f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/275183
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
refs DE-690
flag = none
test-plan:
- Build passes
- Build trigger groovy linter stage if a change to Jenkinsfile is
made
- Build fails if a Jenkinsfile has linter issues
Change-Id: Ifdabb4f28f6124afcb6621844375a8de4b35d453
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265279
Reviewed-by: James Butters <jbutters@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Fix linter issues related to variable name, space after closing bracket.
refs DE-690
flag = none
test-plan:
- Build passes
- QA build with spec change executes fsc stage
Change-Id: I6f985fb3dfa8627703a27325b1eb25fae4585505
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/265188
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
refs DE-558, DE-582
[use-kubernetes]
[canvas-builds-refspec=24d3d83e8034ac6ad5380936498e1dcf01ab50eb]
modify the existing nodeRequirements to add podTemplate and
container definitions. initially, the podTemplate to be used
is docker-in-docker. the "use-kubernetes" flag must be set
as well.
test plan:
- build executes on k8s nodes in EKS when "use-kubernetes"
flag is set.
- build executes on EC2 when "use-kubernetes" flag is not set.
Change-Id: I13c68e5420bbf414532518eaf69ac82f9310bfbf
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263257
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
QA-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Product-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
refs DE-639
Pulsar has 2 services internally, dispatcher and admin server, make
sure both are ready before executing the admin work.
Test Plan
1. The build passes
2. Build is monitored for rest of day for any further intermittent failures
Change-Id: I14f9911285070e727003dc8c0ade5aa067388594
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263430
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
refs DE-558, DE-582
[canvas-builds-refspec=f9c80c0ac4b80de60ac50633523e30d5c2eb28b1]
modify the existing nodeRequirements call to pass a Map containing
the label as well as the possibility of container or podTemplate
params. this change only modifies the existing nodeRequirements to
continue using EC2 agents, not EKS.
test plan:
- builds still work as expected acquiring nodes
Change-Id: I0458a13a4c85e37aeb2f5dc56347e27101705646
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263208
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Product-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
refs DE-338
The protectedNode() function now handles automatic cleanup before / after a node is used, so much of the code here can be removed.
Test Plan:
1. Main build passes, FSC passes
[canvas-builds-refspec=81cc926fe89d6efd9f604d0c573630318fb4eb86]
Change-Id: Ie21d242aabe9826353f05149abb304a85223cebc
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/263149
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
refs DE-628
Test Plan:
1. Build summary report works for sub-jobs and normal stages
2. Sub-jobs run cleanupFn
[canvas-builds-refspec=82ccf7ae185871e45c658f225cc664ef05cda74d]
Change-Id: I9d4476b75e6488bcc388a89e9f7cf87150d0ae6d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262951
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
refs DE-621
Make the stage configuration aware of its node requirements for future EKS support and allowing nospot to be removed.
Test Plan:
1. RSpec / Selenium build works
2. Contract Tests works
3. JS works
4. FSC works
5. Vendored Gems works
[canvas-builds-refspec=480f3570a6a6ebee5cedaecbb3a089476a008215]
Change-Id: I6cb63df20abe8c1f913db4f1b21bb01bd4d30312
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262898
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Andrea Cirulli <andrea.cirulli@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
refs DE-624
As part of making extendedStage() aware of stage node requirements, we need to remove consumers of the cleanupFn() functionality and replace them with extendedStage() compatible patterns.
Test Plan:
1. Vendored Gems behaves correctly
Change-Id: I81d2e003abe64fd0414ee9434cf61c602bda9f08
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262862
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
refs DE-624
As part of making extendedStage() aware of stage node requirements, we need to remove consumers of the cleanupFn() functionality and replace them with extendedStage() compatible patterns.
Test Plan:
1. All stages execute correctly
[canvas-builds-refspec=07f1a75ee23f8f6a9a1bbe062316373b761e7a30]
Change-Id: Ica11b78ebead5c9239dea112677a077654c7090d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/262814
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Kyle Rosenbaum <krosenbaum@instructure.com>
refs FOO-1644
[fsc-propagate=false]
- create engines directory
- put one empty-ish engine in there
- add general engines dependency to
gemfile construction
- update build pipeline to run tests all engines
just like gems
- add engines directory to jenkins-cache build
- get rspec running successfully with a dummy app
inside the audit engine
- stop failing with no message if environment
unloadable in css config
- generally set pattern in build artifacts and in
code organization for pulling out an engine
Change-Id: I8650f5e0aa7d72556d2f02156886d2b503a111fb
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/258988
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
Add timeout to all builds and set the proper values for the existing
timeouts
refs DE-307
flag = none
test-plan:
- Force the build to timeout and check that jenkins fails the build
Change-Id: I2e3ae36f828da1bc047bf5a2cfbc79c167b84d5f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/256171
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
we want to always checkout master code for testing on jenkins, unless
there are changes to our build scripts then pull the gerrit refspec.
flag = none
Test Plan:
- Normal Jenkins build passes.
- changes to docker-compose.new-jeknins* causes sub builds
to checkout code from gerrit refspec.
- changes to any file in build/new-jenkins/ causes sub
builds to checkout code from gerrit refspec.
- Normal build with no changes to Jenkinsfile*, build/new-jenkins/,
or docker-compose.new-jenkins* sub builds checkout master.
Change-Id: If1224a44b52b00f0f056b9552cd562809fc7eaa8
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/252999
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
QA-Review: James Butters <jbutters@instructure.com>
Product-Review: James Butters <jbutters@instructure.com>
refs DE-351
flag = none
test-plan:
- Build passes
- Test FSC is working
Change-Id: Id3844dd12600e5d3a80d267bf67916c4d43d2548
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/251743
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Product-Review: Andrea Cirulli <andrea.cirulli@instructure.com>
These changes need to be accompanied by changes
to Jenkins config:
- main-canvas-postgres-12-ruby-2.6 and
main-canvas-postgres-12-ruby-2.6-postmerge, SCM
refspec changed to master
- sub-builds scm changed from canvas-lms-refspec
to JENKINSFILE_REFSPEC defaulting to master
- add Jenkins parameter JENKINSFILE_REFSPEC to sub-builds
- add new Jenkins build for main Jenkinsfile changes
- main build don't trigger on Jenkinsfile changes using
forbidden path.
Testing once merged:
- verify builds without Jenkinfile changes pull the master
version of Jenkinsfile
- verify sub-builds as well
- verify gerrit with Jenkinsfile changes triggers new build
- verify sub-builds get Jenkinsfiles from gerrit not master
- test a combination of Jenkinsfiles, not just the main one
- verify main build is not triggered
- run a plugin build without canvas-lms-refspec
- run a plugin build with canvas-lms-refspec
Change-Id: Ic6be8b446f80d57f7b0f25c77e6cffb959e03236
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/251634
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: James Butters <jbutters@instructure.com>
Product-Review: James Butters <jbutters@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Ryan Norton <rnorton@instructure.com>
refs DE-349
[canvas-builds-refspec=f04c7902ccabadb61cee13fa93f09b1184925b84]
Test Plan:
1. Build pulls master when flag does not exist
2. Build pulls specified commit when flag exists
3. Build pulls master for post-merge when flag exists
Change-Id: I365efd3a305d08f7a3df76fd44799756104b4c1f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/251635
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
refs DE-23
[change-merged]
[build-registry-path=jenkins/canvas-lms/de-23-test]
flag = none
Test Plan:
1. Jenkins pre-merge build with a new migration re-builds migrations
2. Jenkins pre-merge build with migrations cached does not re-build
3. Jenkins post-merge build with migrations cached re-builds migrations
4. skip-cache directive does not use the cached migration
5. Jenkins non-main builds works as expected
6. Jenkins FSC works as expected
Change-Id: I3221400a15220884740ad5136de7185a88ae2b39
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/241013
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Reviewed-by: Ryan Norton <rnorton@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
fixes: CCI-389
Test-Plan:
we need to run all the builds that we can for this:
- the current build needs to pass
- main-for-plugins needs to be ran
- master-bouncer
- dive?
- translations
Change-Id: Iae7f9dad7668b90653a0b9a8edc20ec69fa109c3
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/240345
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ryan Norton <rnorton@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Product-Review: Rex Fleischer <rfleischer@instructure.com>
Closes: CCI-368
flag = none
Test Plan:
- All builds now have timestamps in the logs
Change-Id: I76f6b329c1b7f04511c6f63643964d6ccc8aa350
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/236719
Reviewed-by: James Butters <jbutters@instructure.com>
Reviewed-by: Jacob Powell <spowell@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Derek Bender <djbender@instructure.com>
Product-Review: Derek Bender <djbender@instructure.com>
This patchset removes the much duplicated database.yml and instead
defaults to a DATABASE_URL scheme. There is however still present a
docker-compose/config/new-jenkins/database.yml file to temporarily allow
our internal portal service to still function until such time we can
reconfigure it.
To set up custom connection options all that is needed is setting
DATABASE_URL. See examples of this in docker-compose.yml and
docker-compose.new-jenkins.yml.
Test Plan
=========
Image:
- pulling canvas-lms:master image boots without database errors
Repo:
- given default dev environment where
COMPOSE_FILE=docker-compose.yml:docker-compose.override.yml
- build the web and postgres images: docker-compose build web postgres
(feel free to comment out `yarn install` and `compile_assets` in the
Dockerfile since we won't be needing them for this test and this
speeds things up)
- set a custom DATABASE_URL to change the database name and echo it to
see it gets into the image correctly:
$ docker run -it -e \
DATABASE_URL=postgres://postgres:sekret@postgres/canvas_custom_env \
$(docker-compose ps -q web) bash -c "echo \$DATABASE_URL"
=> postgres://postgres:sekret@postgres/canvas_custom_env
$ docker-compose run --rm -e \
DATABASE_URL=postgres://postgres:sekret@postgres/canvas_custom_env \
web bash -c "echo \$DATABASE_URL"
=> postgres://postgres:sekret@postgres/canvas_custom_env
- Test that setting a custom database name via DATABASE_URL can create
the database
$ docker-compose run --rm -e \
DATABASE_URL=postgres://postgres:sekret@postgres/canvas_custom_env \
web bash -c "bin/rails db:create"
=> Created database 'canvas_custom_env'
- This patchset boots on portal
Change-Id: Ic6f3819550df94b6c98b17ef05ac2029a578cfe4
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/234213
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Powell <spowell@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
QA-Review: Robin Kuss <rkuss@instructure.com>
QA-Review: Rex Fleischer <rfleischer@instructure.com>
QA-Review: Jacob Powell <spowell@instructure.com>
Product-Review: Rex Fleischer <rfleischer@instructure.com>
Closes: CCI-345
Test Plan:
- set your COMPOSE_FILE to docker-compose.new-jenkins.yml
- remove any existing database volumes via
`docker rm canvas-lms_pg_data` or `docker-compose down -v`
which will remove all volumes in the project.
- update the image with `docker-compose build postgres`
- start the container
$ docker-compose up --detach postgres
- create a second database:
$ docker-compose exec -T postgres createdb -U postgres \
-T template1 canvas_test1
- check both databases:
$ seq 0 1 | parallel "docker-compose exec -T postgres psql -x \
-U postgres \
-d canvas_test{} \
-c \"SELECT * FROM pg_available_extensions \
WHERE name = 'pg_collkey';\"
- Expect two records to be present:
-[ RECORD 1 ]-----+-------------------------------
name | pg_collkey
default_version | 0.5.1
installed_version | 0.5.1
comment | ICU collation function wrapper
-[ RECORD 1 ]-----+-------------------------------
name | pg_collkey
default_version | 0.5.1
installed_version | 0.5.1
comment | ICU collation function wrapper
Change-Id: If1417df3edab788372927b16632e146069a6829e
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/235310
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
Reviewed-by: Jacob Powell <spowell@instructure.com>
QA-Review: Jacob Powell <spowell@instructure.com>
Product-Review: Derek Bender <djbender@instructure.com>
Closes: CCI-344
Test Plan: No mention of spring in any stage that uses
build/new-jenkins/docker-compose-create-migrate-database.sh
Change-Id: Iafe5bb635bf936f83fdad1170398bb8852ebb804
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/235312
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Butters <jbutters@instructure.com>
Reviewed-by: Rex Fleischer <rfleischer@instructure.com>
QA-Review: James Butters <jbutters@instructure.com>
QA-Review: Derek Bender <djbender@instructure.com>
Product-Review: Derek Bender <djbender@instructure.com>
Closes: CCI-339
flag=none
test plan: spring is not referenced in builds
Change-Id: I6c3fb84f7e5d84d6507698b77495b8d376867fb3
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/234826
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Powell <spowell@instructure.com>
Reviewed-by: Rex Fleischer <rfleischer@instructure.com>
QA-Review: Derek Bender <djbender@instructure.com>
Product-Review: Derek Bender <djbender@instructure.com>
Jacob Burroughs