Commit Graph

1218 Commits

Author SHA1 Message Date
Jacob Burroughs 6af6d1bc34 Init consul before the shard cache
Change-Id: Ib01900d93d89527e1a9d8d5b3b1502d884fe3915
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242729
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Jacob Burroughs <jburroughs@instructure.com>
Product-Review: Jacob Burroughs <jburroughs@instructure.com>
2020-07-15 22:48:50 +00:00
Cody Cutrer 74d6822704 disable pg12 COLLATION for the moment
it doesn't work well in a mixed environment

Change-Id: I90b72e7a0fce1a9e4f815cc89f75b9a6b258ddfa
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242679
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-07-15 00:19:33 +00:00
James Williams d3189e9f01 fix remove_foreign_key with if_exists and column
refs #LS-1201

Change-Id: I4240f4b642b1c99bb6dfe3ce88713b597797bea1
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242590
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Keith Garner <kgarner@instructure.com>
Product-Review: Keith Garner <kgarner@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
2020-07-14 14:56:47 +00:00
Cody Cutrer 79ed9a51fe create collations in the public schema
simplifies lots of things, and it doesn't really matter _which_ schema a collation is in

Change-Id: Iaab931dfcb039cd29dfcabcf0d0b198d21e8b3cf
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242525
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-07-13 20:25:48 +00:00
Ethan Vizitei e5eb9c036f add shard/cluster data to DJ logs
fixes FOO-643
flag = none

TEST PLAN:
  1) start jobs
  2) run some jobs
  3) check logging output, should have shard/account

Change-Id: Id1ad972f2e1d81edf8491ec4751a6a6d8bc9a14b
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242275
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ahmad Amireh <ahmad@instructure.com>
QA-Review: Ahmad Amireh <ahmad@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
2020-07-10 22:42:48 +00:00
Ethan Vizitei be97356242 bulk_insert for partitioned models
fixes FOO-685
flag = none

TEST PLAN:
  1) bulk_insert a bunch of auditor records.
  2) they should not get sent to the root partition.

Change-Id: Ie9f6a3af8ba2a05c77ecec89742b651f3f00f7d8
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242163
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-07-09 16:23:22 +00:00
Ahmad Amireh 17f6cacaff fix collation migrations for PG < 12
the DETERMINISTIC option is introduced only in 12[1][2] but we were
version checking for >= 10, causing the migrations to fail on PG 10

relates to g/241190

[ TEST PLAN ]
  ---------

Verify you can run the migrations using PG 10 or 11.

[1]: https://postgresql.verite.pro/blog/2019/10/14/nondeterministic-collations.html
[2]: https://www.postgresql.org/docs/12/sql-createcollation.html

Change-Id: I049bc49d3598be5709c630980580bc8583dca121
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/241438
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-06-30 16:50:22 +00:00
Cody Cutrer 8936b5803a use PG10's built in collation support
test plan:
 * with PG <10, nothing should change/break (migrations shouldn't even try to run)
 * with pg >=10, run migrations
 * it should create collations, and recreate a couple indexes
 * going to the users page should not be broken
 * specs ensure that the sorting is to our liking

Change-Id: I3eaf42dd2207d733c164c12ef2a43a1c1b417ff2
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/241190
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Charley Kline <ckline@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Charley Kline <ckline@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-06-26 18:50:05 +00:00
Ethan Vizitei 58f188a064 connection-light find_in_batches strategy
Change-Id: If069b0dedd6209016a17b3b7963ca855b670b8c8
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/241048
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-06-24 20:21:04 +00:00
Ben Nelson 60200b8a05 schedule due date reminder smart alert
Adds a new model ScheduledSmartAlert which maintains records
or smart alerts to be run in the future, and methods to queue and
process them. Right now we only have the :due_date_reminder smart alert,
but it can be expanded to accomodate other types as well.

Test Plan:
(Verifications will most easily be done in a Rails console)
In a course with active student enrollments
 - Create an assignment with a due date in the future
 - Verify that a ScheduledSmartAlert record was created for that
     Assignment and due date
 - Edit the assignment and change the due date to a different future
     date
 - Verify that the existing ScheduledSmartAlert was updated, and a
     duplicate was not created
 - Edit the assignment and add a due date override with a date in the
     future
 - Verify that a ScheduledSmartAlert record was created for that
     AssignmentOverride and due date
 - Edit the assignment and remove the due date and save
 - Verify that the ScheduledSmartAlert record for that Assignment was
     deleted
 - Edit the assignment and set the due date to the past
 - Verify that there is no ScheduledSmartAlert for the Assignment
 - Edit the assignment and add a future due date, to create a
     ScheduledSmartAlert record
 - Delete the assignment and verify that the ScheduledSmartAlert for the
     Assignment AND the associated AssignmentOverride were deleted

closes VICE-466

Change-Id: I43fe6e2b9cb97f3ec78ffd53520443540bb7bbc4
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/238499
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Product-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Ahmad Amireh <ahmad@instructure.com>
2020-06-17 20:28:16 +00:00
Rob Orton 73b244e59a add ability to pass if_exists to remove_column
this is an option we have on removing indexes and other functions

Change-Id: I5b757be50e1b7ad229fbd65a6864045b8400c81b
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/240394
Reviewed-by: Matthew Lemon <mlemon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
2020-06-17 03:23:33 +00:00
Adrian Packel c46a1119c4 Turn off post-to-SIS for assignments in closed GP
flag=new_sis_integrations
flag=disable_post_to_sis_when_grading_period_closed
closes TALLY-632

Test plan:
- Enable the feature flag "New SIS Integrations" on the site-admin level
- Enable the feature flag "Disable Post to SIS When Grading Period
  Closed" on the *root* account you want to test with
- On the account settings page, check the "Automatically disable 'Post
  to SIS' on assignments when grading period closes" checkbox and save
  settings

- Have a configuration in said root account with multiple grading
  periods
  - Have one grading period (GP1) that closes in the next few minutes
- Have a course with some students and assignments
  - Set the due dates on the assignments
    - 1: Due within GP1
    - 2: Due before GP1's start date
    - 3: Due after GP1's end date
  - Set post_to_sis on all the assignments to true

- After the grading period has closed, allow the periodic job to run, or
  run it manually using:
  > Assignment.disable_post_to_sis_if_grading_period_closed
  - The assignment due within GP1 should have had post_to_sis set to
    false
  - The other assignments should not have been affected

Change-Id: I2bf64a7fb5b09abba0d8e892b70a38ed3e6a78c7
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/235637
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Adrian Packel <apackel@instructure.com>
Product-Review: Syed Hussain <shussain@instructure.com>
Reviewed-by: Spencer Olson <solson@instructure.com>
Reviewed-by: Gary Mei <gmei@instructure.com>
2020-06-10 16:35:47 +00:00
Cody Cutrer ec32fbefe1 add if_exists option to remove_index
and apply to recent non-transactional migrations

Change-Id: I3ef72208f4c39906905229eb3188eaf4df89c68a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239699
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-06-09 21:54:40 +00:00
Cody Cutrer fd03b83f1f add if_exists option remove_foreign_key
also fix foreign_key_for to prefer the obvious FK

Change-Id: I494c4a73aa136bc4a6735231c0957a9f062bdb11
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239688
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-06-09 21:54:19 +00:00
Cody Cutrer 4d1ac60e03 add if_not_exists to add_foreign_key
also fix delay_validation with rails 5.2, and constraints can be re-validated
without dropping them

Change-Id: Iab4d2dc374c7b4707d67323d05e8e8e8ff203e47
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239656
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-09 21:53:28 +00:00
Cody Cutrer 54b53bc094 add if_not_exists option to add_column
and use it for a recent non-transactional migration

Change-Id: I28e6fcc22ff1b22064cde2baf600c86a7728038a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239564
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-06-09 15:36:06 +00:00
Cody Cutrer 1ac2a1b581 add if_not_exists option to add_index
and use it in recent non-transactional migrations

Change-Id: Ie96e3cbf727403c5dfe9d04bc3403170c076d5b6
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239560
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-06-09 15:35:59 +00:00
Cody Cutrer 5c3e5df00f automatically drop duplicate NOT VALID indexes before creating them
Change-Id: I5a372be579aee0e8d72b1b8d39489bbb104a2ab7
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239556
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-06-09 15:35:44 +00:00
Cody Cutrer dd24be625b use SET LOCAL, not just SET
to ensure it goes away after the transaction

Change-Id: Ic616c4943ed5f360fdcf9d85383739859af17abb
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239551
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-06-09 15:34:52 +00:00
Cody Cutrer 029c20dff1 force cache warming query to avoid indexes
Change-Id: Ib8309376a7c1aec372149227fe167a66231226d7
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239474
Reviewed-by: Rob Orton <rob@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-06-06 03:30:01 +00:00
Augusto Callejas aaf457bc94 Outcomes Service content migration service
closes OUT-3655

flag=outcome_alignments_course_migration

test plan:
  - start up canvas, quiz_api and outcomes
  - for an account, provision outcomes service
    and sync some course outcomes (see g/235665)
  - enable the outcome_alignments_course_migration
    and outcome_alignment_non_scoring_content
    feature options in the root account of a course
  - in a course, create a canvas page and align
    an outcome
  - export a copy of the course
  - create a new course
  - import the course content into the new course
  - re-sync course outcomes and contexts
  - re-import the course content into the new course
  - confirm that the new course contains the canvas page
    with the outcome aligned to it

Change-Id: Ie5cce9e1f8829736c03fd5f6bc2c7a8c000c8333
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/238973
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Pat Renner <prenner@instructure.com>
Reviewed-by: Michael Brewer-Davis <mbd@instructure.com>
QA-Review: Michael Brewer-Davis <mbd@instructure.com>
Product-Review: Michael Brewer-Davis <mbd@instructure.com>
2020-06-05 15:25:43 +00:00
Ethan Vizitei 23f6ce8632 add jitter to partitioners
Change-Id: Ie1f441992ffba44ce710c3f6a5fb3ce2fe35cf77
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/238871
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-02 14:45:20 +00:00
Ethan Vizitei 98bfcba24c stop yaml_cache pollution
fixes CNVS-49087
flag = none

implement deep freezing
for the config values to
stop the pollution

dup all the locations
where the config is either
modified or passed off to
some other library where
it's not clear it was loaded
from a frozen config file

TEST PLAN:
 1) load a config file
 2) try to change it's key/vals
 3) you can't because it's frozen

Change-Id: I15faa230e3c99fe4806154493e238cecec526d1a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/236341
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-05-07 18:55:05 +00:00
Cody Cutrer 9d12ea69d0 apply pagination count timeout to non-grouped queries as well
test plan:
 * same as c8b0c29361
   (it uses a different query now, so implementation had to change)

Change-Id: I9ce37ef18a9b2040d9ba0e51626f7b6d39a90c40
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/235399
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-05-05 17:33:17 +00:00
Ethan Vizitei 0215a61aa8 bulk_insert can fail silently
TEST PLAN:
 - try to bulk_insert something with a missing parent
 - notice the actual error that occurs

Change-Id: Id93f3f52f0dc3e2ba50e98e144742b38d152bf5c
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/236204
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-05-05 14:40:29 +00:00
Cody Cutrer ab0d1fa3fb add vacuum helper method
refs USERS-538

Change-Id: Ic77533bf8cca5846a2f165ed3870baeef18adc48
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/236166
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
2020-05-04 20:33:03 +00:00
Ethan Vizitei bf534dab69 fix find_in_batches problem
FIXES CNVS-49060
flag = none

when the row count is exactly
the size of the batch, this
prevents the find_in_batches
code from erroring

TEST PLAN:
  1) find things in batches
  2) force it to make a temp_table with exactly batch_size
  3) batch iteration does not fail

Change-Id: I81ce7f6a58fa45e8041f814bac4cf5ec12879026
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/236102
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Gentry Beckmann <gbeckmann@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-05-03 15:47:15 +00:00
Cody Cutrer 5e5ebdebee re-arrange block nesting to avoid transaction when we're not going to query
Change-Id: I28b7fc558284a7b7d08d89a25605b6411001667e
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/235398
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-04-28 02:14:17 +00:00
Ethan Vizitei f558a7c3b3 host-specific circuit breaker
fixes CNVS-48937
flag = none

test plan:
  * start canvas
  * make it talk to some service for some reason
  * force the http call to return timeout errors
  * after enough of those within a short window
     we should start getting circuit breaker errors
  * make the http call not die
  * 15 seconds later circuit breaker errors should
     no longer happen

Change-Id: I3298932114bd2ed4a9e35e78ed41219403d4ee27
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/234637
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Williams <jamesw@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
2020-04-22 16:06:52 +00:00
Simon Williams 5ed76c8323 squash old migrations, p20
Change-Id: Ic005cc7e302f447b3cd640c7c666f820cdb952df
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/233756
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-04-21 15:40:55 +00:00
Cody Cutrer c8b0c29361 timeout when doing possibly unnecessary complicated counts for pagination
fixes USERS-274

test plan:
 * have a lot of users in an account (like 500)
 * go to /accounts/self/users
 * it should load, and show you how many pages there are
 * set the timeout to something absurdly low (like Setting.set('pagination_count_timeout', '1ms'))
 * go to the page again
 * it should still load the first page quickly, but the pages thingy at the bottom should only list
   1 and 2, and then an ellipsis

Change-Id: Ibc07036b39a9f4ed19d8824bb9254b23679298d1
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/234557
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-04-18 02:15:51 +00:00
Ethan Vizitei 95aca3787c auditor models and migrations
refs CNVS-48876
flag = none

The only goal for this PS is to get the
tables and models in place to handle auditors
data being written to PG instead of cassanrda.

No actual new writing happens in this change.

create the first couple partitions as a migration
and the previous 12 to have room for backfilling

include partitions migrations for keeping indices
on individual partitions

cron-ify partitioner to keep pre-creating
new partitions for auditor tables

TEST PLAN:
  * migrate the database
  * do things that trigger auditor writes
    - login
    - change grades
    - publish courses
  * nothing should have changed or broken
  * auditors tables should be completely empty in PG
  * create the first few partitions as a migration

Change-Id: If4e2446a82fb1c91e992a28e62918ff8d8a8c73a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/234064
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-04-16 20:02:49 +00:00
Cody Cutrer f58c60833b move Switchman.cache setup even earlier in the boot process
so that the default shard will use it as well

this also means MultiCache will load earlier, so move #cluster to Canvas
instead of ApplicationController to reduce how much is loaded. and don't
memoize the value of Rails.cache at that point, since it might not be
fully configured yet if you're not using ha_cache.yml

Change-Id: I104d7a3dcd69c6cb97e4083e198ac06c60700f9e
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/234253
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-04-16 17:24:58 +00:00
Cody Cutrer 56efa192bc fix serialization problems with sharding
tl;dr: set up the serialized settings column before any instances are
created (including the default shard), so that we don't end up caching
the string value of that column anywhere

Change-Id: I3249a2d910e84c539d4fbb88d61cdcdba103cac3
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/234252
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-04-16 14:25:26 +00:00
Mysti Lilla eb3a81894c Move JobLiveEvents module to a non-reloadable location
Test plan
- Specs pass
- Canvas starts
- Delayed Jobs run

Change-Id: I8899011c27dc5204a8e7d6d5e2cfc696bc55ca2b
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/233907
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-04-13 22:49:18 +00:00
James Williams 21ff8a5759 add last-modified header for template streaming
so rack doesn't try to digest it for an etag

Change-Id: I3ac1cce55a8a2f1a44c18cb05be620f511699d61
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/233405
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-04-13 17:54:34 +00:00
Cody Cutrer 4752c990a9 optimize find_in_batches_with_temp_table
* don't bother doing the query when we know there won't be rows to satisfy it
 * don't bother creating an index on a very small temp table

Change-Id: Ic24c83275da20cb000532dde895f2dae135e064e
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/233650
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-04-13 17:51:55 +00:00
Cody Cutrer 47e1389c16 fix autoloading of Canvas::Apm
Change-Id: I5f72b15bf29ce2b2ef25901416bbfce503aeb8c3
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/233587
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-04-13 12:37:08 +00:00
Rob Orton 6f625fff51 fix initializer
test plan
 - have sentry config
 - canvas should boot

Change-Id: Ifc0657050fa419bc72ba6c27a7ce6d1542a5f6f7
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/233075
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-04-06 19:17:07 +00:00
Rob Orton e1b2f967f7 ignore noisy errors p2
These errors are not meaningful to us in sentry and they produce a large
amount of errors. This will stop sending them to sentry, but will still
create an error report.

test plan
 - cause conditional release service error
 - it should not show up in sentry
 - it should still create an error report
 - it should increment counter in stats

flag=none

Change-Id: I0163e4c8c5090f7fafdbccd1fdc0e16964ba550c
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/232978
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
2020-04-06 16:34:39 +00:00
Rob Orton 336e188a78 ignore noisy errors
These errors are not meaningful to us in sentry and they produce a large
amount of errors. This will stop sending them to sentry, but will still
create an error report.

test plan TII
 - TII submissions should still work
 - on Failure
 - should not generate an error report
 - should not generate a sentry error
 - should send a stat to statsd
 - should retry
test plan grade passback
 - LTI tool with errors in grade passback
 - should not send to sentry
 - *should* still create an error report

fixes PLAT-5668
fixes PLAT-3155
flag=none

Change-Id: Iff28e6538798d2bfe80d91721559934d0d3e4329
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/232949
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
2020-04-06 16:33:47 +00:00
Rob Orton ccc216ecfc allow ignoring error types from setting
test plan
 - existing errors should work
 - have a raven.yml
 - populate Settings.set('ignorable_errors', 'some error')
 - 

Change-Id: I1bbfc003506ef5f15aa7fdb887e0249c84f09c56
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/232977
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
2020-04-06 16:33:00 +00:00
Cody Cutrer d30efbb215 configure switchman cache earlier in the boot process
so that it's used when loading the default shard's delayed jobs shard

Change-Id: I2a5f20f65720511255885da8bbff3ca54780a469
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/232127
Reviewed-by: James Williams <jamesw@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-04-01 13:44:35 +00:00
James Williams 4e04055993 disable periodic jobs queue queries
Change-Id: I4a79fe2fd2e9a4203073e1942eef9c8c2852c459
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/232319
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
2020-03-31 21:49:08 +00:00
Ethan Vizitei a3c2e318d6 enable instrumentation of APM for jobs
closes APM-16, APM-20
flag = none

also adds context id and user id
to request annotations for APM

configures host-level sampling

TEST PLAN:
  * enable apm collection on datadog agent on single test cluster
  * push consul config to same cluster for enabling apm sampling
  * push consul config depressing host sampling rate to 5%
  * delayed job telemetry should show up in ddog
  * trace count from active clusters should drop by an order of magnitude

Change-Id: I94d97b299ed14403e8b141629740a1627310b259
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/230592
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Reviewed-by: David Warkentin <dwarkentin@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-03-23 15:11:25 +00:00
Ethan Vizitei 38ccd99563 root account and shard tags for APM
closes APM-11
flag = none

TEST PLAN:
  * deploy to some environment with APM enabled
  * observe telemetry and confirm root_account
    and shard tags

Change-Id: I7246801c51617f10b8b7f3ca69b5d245375e81be
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/230415
Reviewed-by: Simon Williams <simon@instructure.com>
Reviewed-by: David Warkentin <dwarkentin@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Michael Hargiss <mhargiss@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-03-19 15:55:46 +00:00
Ethan Vizitei 8ce6b785aa enable auto-instrumentation for ddog APM
closes APM-7, APM-8
flag = none

hardcode set to a very low % of client sampling by default
to keep performance impact low.

test plan:
 * make sure telemetry is arriving in datadog APM
 * ensure trace traffic level is low, only from one cluster
     (proves settings are working)
 * canvas should not show degradation in response times post deployment

Change-Id: Ifca8d3f6239d6c4e70098dd2b68b9c2a1950e121
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/230064
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Michael Hargiss <mhargiss@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-03-17 19:15:15 +00:00
James Williams 6f775933d0 add caching around primary_keys db query
Change-Id: I04de69d1e8c0c13e4e3e110b603bdeb2ee975baa
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/229017
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-03-12 15:24:35 +00:00
Mysti Lilla cb2823193c Use same_site none for session store for relative file links
fixes LA-637
flag=none

Test plan
- Verify that as long as you don't have a
  separate files domain, you can't reproduce
  this issue in dev (should just work out of
  the box)
- Install puma-dev from github.com/puma/puma-dev
- Follow the Set up instructions, but configure
  at least 1 extra domain (.test and something else
  with the -d switch, it looks like `puma-dev -install -d
  test:files`)
- Make sure to get the symlink set up
- Add `export THREADS=1` to ~/.powconfig
- In Canvas, in domain.yml under development, add a
  files_domain that has the same app name, but a
  different domain (ie I set up my domain as
  canvas-lms.test and my files domain as
  canvas-lms.files), if you switch back to not using
  puma-dev, you will need to change these back to
  what they were
- Also add ssl: true
- In session_store.yml under development, add secure: true
- To start, set up puma-dev in launchctl `launchctl
  load ~/Library/LaunchAgents/io.puma.dev.plist`
- To view the server logs, use `less -R log/development.log`
  and then F
- To have Chrome accept the certificates, open your Keychain
  Access (on MacOS) and move the Puma-dev CA to System
- Create an HTML file that has a link like 'other_file.jpg'
  and upload both files to a Canvas course in the same folder
- In Chrome, turn on the flags for same site by default cookies
  and cookies without samesite must be secure
- Open the file and ensure that it is showing the linked
  file, and that it is serving it from another domain
- Get an old version of Safari somehow (XCode or otherwise)
  (version on iOS 12 will do it) and verify you can
  still log in to Canvas

Change-Id: I84e62cc16e3730ffb6bc2d4d7de9b772eedf260e
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/227939
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jeremy Stanley <jeremy@instructure.com>
Product-Review: Mysti Lilla <mysti@instructure.com>
2020-02-27 21:01:01 +00:00
Simon Williams f7bc9c4878 bump oj to 3.3.10
https://github.com/ohler55/oj/blob/v3.10.2/CHANGELOG.md

Change-Id: I8a1f29cb686a794c729a84966a1932242277ac4c
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/227905
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-02-26 17:22:18 +00:00