Commit Graph

1207 Commits

Author SHA1 Message Date
Rob Orton 553b2fe558 fix race condition on report runners
this is very difficult to test, because it's a race condition, but the
test plan would work...

the changes are just plumbing through if the primary db should be used
or a replica, and then methods that have been well tested and used.

for parallel account_reports we write rows to account_report_rows and
then read from account_report_rows to generate the csv file. If this is
done on a replica, it can be lagging and not get all the records.
Typical reports, this would not be a problem because it is old data for
a report...
but when we just wrote the data it may not exist, so use the primary
database.

test plan
 - run a report with a primary database and read_only replica
 - pause replication
 - the report should work

fixes PFS-16619
flag = none

Change-Id: I93140ce8f566a4e0553377388d6296ec36d800cb
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248863
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Mark Valentine <mvalentine@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
2020-09-29 22:47:34 +00:00
Rob Orton 6931030d92 remove extra return check on error state
the next line checks that workflow_state is running and returns false if
it is anything besides running.

Change-Id: I38d9015b7e0ab8908d5e7b8ea957d69f0feff024
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248864
Reviewed-by: Mark Valentine <mvalentine@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
2020-09-29 18:54:03 +00:00
Kyle Rosenbaum 025556caf3 bump rubocop version to 0.68.0 to support ruby 2.6
this changes upgrades to the earliest version of rubocop
supporting ruby 2.6 while also choosing a version that fixes
a breaking 0.60.0 change to revert psych version change.

namespace changes were made on multiple cops to support
the new rubocop version requirements as well as the change to
move performance cops out of the rubocop gem in to
rubocop-performance.

the Layout/IndentFirstParameter cop is currently broken but a
rubocop version upgrade will fix that. the FreezeConstants cop
also required a fix to handle nil types.

refs DE-289

test plan:
- ensure rubocop runs completely

Change-Id: I0ca68f426e287b5a973b96a4ae0e975f1cbc0240
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248786
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Product-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Reviewed-by: Ryan Norton <rnorton@instructure.com>
2020-09-29 16:09:54 +00:00
Cody Cutrer 4211945c1f use shard settings, not Setting, for common core group id
Change-Id: I3324d727ea16760c4b28a0b8d0d3cbea676948a5
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248463
Tested-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-09-28 17:22:17 +00:00
Ahmad Amireh 581baf6e37 highlight dependencies in canvas:compile_assets
refs FOO-966

as we're working on extracting Gradebook into its own client_app, the
canvas:compile_assets rake task had to be adjusted to build it too, then
i realized that the canvas_quizzes app has to be built _before_ we
generate i18n phrases while Gradebook (and canvas) have to go _after_
that task

the rake task did not appreciate any more complexity, so instead I
tried to focus on making the dependencies between these sub-tasks
clearer and at the same time maintain the parallel execution

changes:

- it's now possible to build a specific client_app via rake tasks
- the symlink needed for canvas_quizzes client app is now restored
  correctly in case of a failure during the i18n rake task. Previously,
  you had to restore it by hand before running any of the tasks again
- in case a client_app does not implement a `script/build`, we call its
  "npm build" script instead (eg `yarn run build`)

test plan:

- run "RAILS_LOAD_ALL_LOCALES=1 rake canvas:compile_assets" before you
  check out the patch and note the time taken
- check out the patch and re-run the task, verify it still works and
  that the time taken is not longer than what it was (if anything, it
  should be a little faster since we're batching more work)

Change-Id: I65e675ceeea93643a1c679ff29039ed5ef71b53d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247902
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-09-24 20:51:51 +00:00
Ethan Vizitei f9ee94c717 use AR scope for event_stream AR procs
refs EVAL-1229

Change-Id: I52e7357294d37f2a65b56942b646ae452e8bce19
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248468
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-09-24 20:34:14 +00:00
Cody Cutrer c925836467 bump some gems to reduce ruby 2.7 deprecation warnings
also fix some myself

Change-Id: I95e354d4da570deaf8aaa613e05ff361f49f0393
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248030
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-09-22 16:08:42 +00:00
Michael Brewer-Davis f979a12ef1 remove outcomes export i18n FF
closes OUT-3194
flag=none

Test plan:
qa-cr

Change-Id: I09cc9777dd92aa5d3c3274b4a0813e40d5731bce
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247418
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Augusto Callejas <acallejas@instructure.com>
Reviewed-by: Pat Renner <prenner@instructure.com>
QA-Review: Michael Brewer-Davis <mbd@instructure.com>
Product-Review: Michael Brewer-Davis <mbd@instructure.com>
2020-09-21 14:47:35 +00:00
Evan Francis 71cc43d2a1 reports: apply current proficiency scales to outcome reports
flag = account_level_mastery_scales

closes: OUT-3776

Test Plan:
1.Create Assignment
2.Create Account level Outcome attaching
Assignment to Account level Outcome via rubric
3.Create Outcome and attach to Assignment via
rubric
4.Run speed grader and grade both rubrics
5.Run the outcome_report and validate
we see scores associated with Outcome
6.Run the student competency report and
validate we see scores associated with Outcome
7.Enable FF
8.Run the outcome_report and validate we see
scores associated with Account level Outcome
9.Run the student competency report
and validate we see scores associated with
Account Outcome

Change-Id: Id20053c8be2ead467b8e7e54b699fc6e8d1e2b73
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246714
Reviewed-by: Augusto Callejas <acallejas@instructure.com>
QA-Review: Pat Renner <prenner@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Product-Review: Jody Sailor
2020-09-15 21:08:33 +00:00
Ethan Vizitei dfa1a99935 compare UNZIPPED size of sis imports
refs FOO-889

Change-Id: Ice551a462633fd571ec5c1e4b26c3ab5a70ea725
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247251
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Reviewed-by: Charley Kline <ckline@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-09-15 14:41:44 +00:00
James Williams f781e96cff use dummy root account for user and site admin content migrations
refs #LS-1361

Change-Id: Ie9ab343027b636cb28169385341aea4829938c7d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246934
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
2020-09-09 15:30:05 +00:00
Kai Bjorkman f6e8f0c3fa remove add grading scheme in admin reports grade export flag
closes EVAL-641
flag=add_grading_scheme_to_admin_grade_reports

Test Plan:
- Specs pass

Change-Id: I28001dd2047e4e4da48c425eb6c70f9595382013
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246694
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Gary Mei <gmei@instructure.com>
Product-Review: Syed Hussain <shussain@instructure.com>
Reviewed-by: Syed Hussain <shussain@instructure.com>
Reviewed-by: Gary Mei <gmei@instructure.com>
2020-09-09 03:01:56 +00:00
Cody Cutrer 42020b07bf delay loading live events context until we actually send a live event
Change-Id: Ieae2aaa3400c88959e2227874c6ca94794b258c6
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246222
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Ahmad Amireh <ahmad@instructure.com>
QA-Review: Ahmad Amireh <ahmad@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
2020-09-03 17:20:12 +00:00
James Williams a6e6fd0670 split built-in role rows by root account
test plan:
* a multi-tenant shard should still work
 creating enrollments and account admins before and after
 running postdeploy migrations
* the postdeploy should successfully run a datafixup
 to split up build-in role rows and reassign the role_id on
 other tables (account_notification_roles, account_users,
 enrollments, and role_overrides) accordingly

closes #INTEROP-6045

Change-Id: I8325acb635397ac1b008b9d09788595da597cec7
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243906
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Jeremy Stanley <jeremy@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Mysti Lilla <mysti@instructure.com>
2020-09-01 19:21:27 +00:00
Ethan Vizitei b7e45db77a allow client injection for dynamodb
refs FOO-782

Change-Id: I254ebdb633b399de320d4498b12e88a1bb2af01d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246603
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Michael Ziwisky <mziwisky@instructure.com>
QA-Review: Michael Ziwisky <mziwisky@instructure.com>
Product-Review: Michael Ziwisky <mziwisky@instructure.com>
2020-09-01 15:48:44 +00:00
Ethan Vizitei 16abf15de4 yank migrations and table management from canvas ddb
Change-Id: I3d2a60443e9b514f690a236b05dd2f437ad8e3c2
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246391
Reviewed-by: Michael Ziwisky <mziwisky@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-08-31 20:26:17 +00:00
Davis Hyer 2f1587387a fix slug generation exception
refs VICE-739
flag=none

test plan:
  - tests pass

qa risk: low

Change-Id: I05265f2b28e4c635b96783f27c1ff588c5831162
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246070
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Product-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
2020-08-27 16:37:05 +00:00
Rob Orton 196949740b stop generating messages in report specs
Change-Id: Ib880a3bd3f80d8e0ac60051e05be5834b6c6254c
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/245988
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
2020-08-26 17:16:34 +00:00
Davis Hyer 062555c451 filter profane slugs
fixes VICE-739
flag=none

See https://instructure.slack.com/archives/CGC0YJJDS/p1598370828002100
for context. tl;dr random strings can sometimes result
in vulgarity. This commit has us pick a new random
string if the previous was deemed vulgar

test plan:
  - :noideadog:

qa risk: low

Change-Id: Iddd4d0c944c44f85af9cf32352611acafd44f734
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246010
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Tested-by: Jeffrey Johnson <jeffrey.johnson@instructure.com>
Reviewed-by: Jeffrey Johnson <jeffrey.johnson@instructure.com>
QA-Review: Jeffrey Johnson <jeffrey.johnson@instructure.com>
Product-Review: Jeffrey Johnson <jeffrey.johnson@instructure.com>
2020-08-25 21:09:54 +00:00
Ethan Vizitei 42f635be1d allow passing consistency through batch ops
Change-Id: Ib87a7caf7285f1884ac461cbed348d31c8ecb811
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/245939
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-08-25 14:55:34 +00:00
Alex Anderson 6bbb46cf95 Fix sandbox "allow-popups" errors
Test Plan:
Let's make sure existing sandbox attributes are properly removed.
Change a page to have the following content:
<iframe src="https://docs.google.com/document/d/1054I9jXtjHmSwFUuj60YsyQ-DloZpORzBffO3KDdv2k" width="100%" height="600" sandbox="allow-scripts allow-forms allow-same-origin"></iframe>
Save
Clicking the 'Request Access' link should open up a confirm modal.
Don't see the error message in the devtools console

flag=none

Fixes CNVS-49914

Change-Id: I261e428185c01f2800c7a89131da2161b00e16ae
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/244965
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Product-Review: Peyton Craighill <pcraighill@instructure.com>
QA-Review: Robin Kuss <rkuss@instructure.com>
Reviewed-by: Ed Schiebel <eschiebel@instructure.com>
2020-08-13 19:30:06 +00:00
Rob Orton 3fe8c877fc report runner improvements
this doesn't really change anything with how report runners are run, but
how they might fail.
we don't want to abort the runner that may have failed, so we can get
data on which report had an error with state of error or running.

Also we don't want to write rows on a failed or aborted report runner
because then we can manually run that runner and any aborted to finish
the report in a console for troubleshooting.

test plan 
 - specs should pass

Change-Id: I5cfc3d975899f85aff64a6e20c7673b1f2cd1b88
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/244625
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Drake Harper <drake.harper@instructure.com>
QA-Review: Drake Harper <drake.harper@instructure.com>
Product-Review: Drake Harper <drake.harper@instructure.com>
2020-08-10 15:36:29 +00:00
Rob Orton 993d8b0349 run on master to simplify report_db activation
test plan
 - specs should pass

closes VICE-703

Change-Id: I5ac4d2c7c13b6637e68fc609e87bce1dd30ec71a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/244624
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-08-10 15:25:41 +00:00
Jackson Howe e4e4bb5487 Allow skype: protocol in RCE
This updates canvas_sanitize gem to allow the skype: protocol in
links in the RCE.

Fixes LS-248
flag = rce_enhancements

Test plan:
 - Open RCE (fixes for new and old RCE)
 - Switch to HTML editor
 - Add <a href="skype:whatever?call">Call whatever</a>
 - Save
 - Click the link
 - Expect browser to open Skype (may present warning first,
   depending on browser)

See http://enarion.net/programming/c-sharp/skype/html-call/ for
an explanation of the skype protocol.

Note: In both the new and old RCE, adding a skype: link via the
rich content mode, will only work if the link is skype://username
(with the slashes) - skype:username will not work, but is being
addressed in the new RCE in LS-1232.

Change-Id: I03cc1be046cf79231538bd9ae7c32fa41ba62cd1
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243978
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Clint Furse <cfurse@instructure.com>
QA-Review: Robin Kuss <rkuss@instructure.com>
Product-Review: Jackson Howe <jackson.howe@instructure.com>
2020-07-31 20:14:02 +00:00
Alex Anderson f354762d35 Include same-origin permission on iframes
The iframe sandboxing made it so media uploads
didn't properly render after being uploaded. This
is obviously a problem, so this patchset adds
the allow-same-origin sandbox permission to
all iframes.

This effectively reverts LS-743, so that ticket
was reopened so it can be addressed at a
later date.

This will need to be warm fixed.

Test Plan:
Upload media with the new RCE.
Save it.
The media should load correctly on the page.

Refs LS-743
Fixes LS-1260

Change-Id: I130af41a369d19ec02abf18682569a88d0a18f77
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243902
QA-Review: Robin Kuss <rkuss@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jackson Howe <jackson.howe@instructure.com>
Product-Review: Alex Anderson <raanderson@instructure.com>
2020-07-30 20:17:30 +00:00
Alex Anderson df88e1361b Sandbox the file preview iFrame
Since files are hosted on the same domain as the Canvas
instance, it has access to localStorage, cookies, and
autofill passwords. This can be used to capture user data
for nefarious purposes through XSS.

This adds a `sandbox="allow-scripts allow-forms"` attribute
to the preview iframe to disallow same-origin behaviors.

This patchset also applies sandbox rules to iframes
embedded in user-generated content. This is done
with the onChange handler.

Test Plan:
This will be a little involved, so bear with me.
Log out of Canvas. On the login page, save the raw HTML
to your computer as a .html file. In Chrome this is done
with the "Save Page As..." option.

This will create an HTML page that copies the Canvas Login.
Then, go to the Files page for a course and upload this
HTML file.

Preview the HTML file. If you have autofill passwords enabled,
the username and password fields should not be filled in.

If you don't have autofill passwords enabled, that's okay.
Instead, open the preview for the HTML file.
Then open the dev tools. In the console, change your
JavaScript context from "top" to "preview". Then, try
to execute the following JavaScript (sans quotes):
`localStorage`
If this throws an error, it's working as expected.
If it logs the `localStorage` object, then the frame
still has access to the top origin.

To test user-generated content, navigate to an RCE,
and use the Raw HTML Editor to add an iframe. Close
the Raw HTML editor, make a change in the editor,
then and open the HTML editor again. The sandbox
attributes should have been added to your iframe.

flag=none

Refs LS-743

Change-Id: I9fe5660d8592423e6435d2b7ed6804b05c74200a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243228
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jackson Howe <jackson.howe@instructure.com>
QA-Review: Robin Kuss <rkuss@instructure.com>
Product-Review: Alex Anderson <raanderson@instructure.com>
2020-07-29 13:01:19 +00:00
Ethan Vizitei 470ca950ad tune http throttle computation to seconds
refs FOO-739

TEST PLAN:
  1) make some long-running http requests
  2) if they are in serial, they should not throttle

Change-Id: I3c44fa3713859a31363980b9ae1b41742551fa04
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243503
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-07-28 01:12:49 +00:00
Aaron Ogata 2859749789 work around yarn intermitent failure in vendored-gems
Change-Id: Ie317a6e97f4023146eeedf993f4eee631460f049
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243566
Reviewed-by: Ryan Norton <rnorton@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Aaron Ogata <aogata@instructure.com>
Product-Review: Aaron Ogata <aogata@instructure.com>
2020-07-28 00:30:51 +00:00
Rob Orton ad13be6a8f spec: fix flakey spec
test plan
 - specs should pass.. all the time

@er was being set in other report when it failed so set it to nil after
it has been consumed.

fixes VICE-632

Change-Id: Iafa32ec7af0230ccd42df4d610bdb5c89551ea74
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243065
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
2020-07-21 16:18:49 +00:00
Derek Bender a09f6902ed Alpine canvas-lms based image
Drop-in replacement for existing Dockerfile

Closes: CCI-391

Change-Id: I9d21a74d72013be854631b2f77880eb0c47fded2
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/236853
Reviewed-by: Ryan Norton <rnorton@instructure.com>
Reviewed-by: Aaron Ogata <aogata@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: James Butters <jbutters@instructure.com>
Product-Review: James Butters <jbutters@instructure.com>
2020-07-15 17:21:56 +00:00
Ethan Vizitei b7e72994f5 Ascribe some throttling cost to http
closes FOO-690
flag = none

TEST PLAN:
  1) make some long-running requests
  2) they should eventually 403 if you
    make them over and over.

Change-Id: If4e399dc3f888545317fc135dac0e4752dd22467
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242287
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-07-10 20:06:46 +00:00
Ethan Vizitei be97356242 bulk_insert for partitioned models
fixes FOO-685
flag = none

TEST PLAN:
  1) bulk_insert a bunch of auditor records.
  2) they should not get sent to the root partition.

Change-Id: Ie9f6a3af8ba2a05c77ecec89742b651f3f00f7d8
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242163
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-07-09 16:23:22 +00:00
Ethan Vizitei 92b3deca92 canvas_http should log when we request through it
closes FOO-642

TEST PLAN:
  1) run a request through canvas that uses canvas_http
  2) you should see timestamped output in logs

Change-Id: I1756ddea835759e7da2f63cc13d792a316f252ba
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242027
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
2020-07-07 16:27:34 +00:00
Ethan Vizitei cbbfbe13a4 event_stream db_name is backend-dependent
refs CNVS-48876
flag = none

TEST PLAN:
  1) enable postgres read/write
  2) leave cassandra config in place
  3) drop cassandra server
  4) auditor actions should not generate cassandra
     errors

Change-Id: If355bccad34285b2157e608a329ef893e6a9491f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/242080
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-07-07 15:50:08 +00:00
Ethan Vizitei 4f0d4012a5 removes cassandra-specific interface
refs CNVS-48876
flag = none

TEST PLAN:
  1) tests should pass with internal
     plugins

Change-Id: I69333ca80feb8040d6830667a69604e0b6c72db0
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/241322
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-30 02:29:05 +00:00
Ethan Vizitei 560a5be0fb auditors x-shard auth records
refs CNVS-48876
flag = none

TEST PLAN:
  1) read auditors from postgres
  2) load auth records for a user that
     spans shards
  3) query returns auth auditor
     record results from all shards

Change-Id: Id5e85ddcf3b9888ff098a1eb523ab81a5d8cd4ef
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/240949
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-29 19:43:17 +00:00
Ethan Vizitei 1af520ef0c AR auditors handles nil index entries
refs CNVS-48876
flag = none

TEST PLAN:
  1) in console make a grade change auditor write
      which has a submission with `autograded?`
      set to true
  2) Auditor write does not raise error

Change-Id: I2915eeb5be96fb97984876a797e1c360e1b95a9d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/241167
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-25 18:02:12 +00:00
Ethan Vizitei 0870ec913f auditors needs to use bookmarks
refs CNVS-48876
flag = none

bookmarks stored as strings
require parsing for use
and validation

TEST PLAN:
  1) read auditors from postgres
  2) choose a case with more than 10 records to view
  3) scrolling down loads more

Change-Id: If4d05e4ce5a8240757d6e07598da51bba197e6f0
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/240146
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-13 12:38:56 +00:00
Ethan Vizitei 006589ac09 give DB auditors attribute manipulation
refs CNVS-48876
flag = none

and order records
in descending order just
like from cassandra

TEST PLAN:
  1) read auditors from postgres
  2) grade_changes records show up in auditors
      UI
  3) all auditors UI is ordered by created_at
    DESCENDING

Change-Id: Iceb862581453875d34810b3ccf9f8f3219a0c0e2
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/240017
Reviewed-by: Simon Williams <simon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-12 19:44:08 +00:00
Ethan Vizitei 2cd65fadb3 auditors use read setting at runtime
refs CNVS-48876
flag = none

also give course records
an "event_data" json parser

TEST PLAN:
  1) use a dual-writing system for a while
  2) flip auditors reading from cassandra
     to active_record without restart
  3) reading should happen exclusively
     from postgres, no CQL queries in logs.
  4) course records should be able to be shown
     in admin_tools logging UI

Change-Id: Ib5f626141d21acf925ff46e07c296bdd133fcde0
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239824
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-11 02:12:28 +00:00
Augusto Callejas 1e7ff0a8c4 Skip setting root account on AB imports
fixes OUT-3710

flag=none

Content migrations now automatically set their
root account from their context. This caused an
issue on Academic Benchmarks imports because the
context was the Site Admin context, which does
not exist in the current shard's accounts table,
causing a foreign key check to fail. Instead,
we skip setting the root account id altogether.

test plan:
  - set up Academic Benchmarks importing
  - create a second shard
  - confirm that importing works in that
    second shard without failing

Change-Id: Ibf1e4c1e94cd788b6c50203bf93bb1d80779b847
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239741
Reviewed-by: Michael Brewer-Davis <mbd@instructure.com>
Reviewed-by: Pat Renner <prenner@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Pat Renner <prenner@instructure.com>
Product-Review: Augusto Callejas <acallejas@instructure.com>
2020-06-10 18:45:43 +00:00
Cody Cutrer ec32fbefe1 add if_exists option to remove_index
and apply to recent non-transactional migrations

Change-Id: I3ef72208f4c39906905229eb3188eaf4df89c68a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239699
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-06-09 21:54:40 +00:00
Cody Cutrer 3045e2a3f4 lint for FK idempotency in non-transactional migrations
Change-Id: Iee7e823b5a2f43a5cd6e5eebf166a6df1b9668c4
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239689
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
2020-06-09 21:54:25 +00:00
Augusto Callejas 0d8da79090 Include concluded courses in outcomes report
fixes OUT-3595

flag=none

test plan:
  - create a course
  - import an account outcome into the course
  - create an assignment and align the outcome to it
  - enroll a student and submit to the assignment
  - assess the rubric of the submission
  - export the account Outcome Results report and
    confirm that the result for the student appears
    there
  - under the course settings, conclude the course
  - export the account Outcome Results report again
    and confirm that the result for the student still
    appears there

Change-Id: Ib61315cb83d15e41438eb9ae4ed3b5512741a9a3
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239575
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Michael Brewer-Davis <mbd@instructure.com>
QA-Review: Brian Watson <bwatson@instructure.com>
Product-Review: Jody Sailor
2020-06-09 20:08:07 +00:00
Ethan Vizitei 96c6d38b1a auditors read path
refs CNVS-48876
flag = none

make all auditors pass
backend strategy from dynamic
settings

configure indexes in
event_streams with columns
on db tables to search by

use a bookmarked collection
to return results with a simple
paginator

TEST PLAN:
  1) use a dual-writing system for a while
  2) examine API or UI results from auditor
     endpoints
  3) enable reading from postgres
  4) API and UI results from auditor endpoints
     should be equivalent
     (and loaded via SQL queries)

Change-Id: If5b10736077b7721348ede323a34dd843d08c75f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/236498
Reviewed-by: Simon Williams <simon@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-06-09 16:10:33 +00:00
Cody Cutrer ab30b3c8b3 lint for if_not_exists on non transactional migrations
Change-Id: I13708948205db4735280be100ac8191a6d3384e1
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239577
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-06-09 15:36:13 +00:00
August Thornton 23f1f83e0d add context to last_access_at for user access report
closes USERS-483
flag = none

test plan:
 - navigate to /accounts/self/settings
 - click on Reports tab
 - click on `?` bubble for Last User Access report
 - verify description makes sense given added context

Change-Id: Ia5085c3e2cfe4979614c239cc6f5bd0fdfc4cc41
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/239075
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: August Thornton <august@instructure.com>
2020-06-04 20:57:45 +00:00
Ethan Vizitei 0d9c3e3c21 add an ID-only scan for event_stream
refs CNVS-48876
flag = none

TEST PLAN:
 1) schedule backfill jobs
 2) they do not fail
  (nothing uses this code path yet, but the repair
   path for auditors backfill WILL use it,
   we just don't want current event_stream
   reads to be broken)

Change-Id: Ifbc4839eab3688a0c3ef0457e7edaef41ca43b73
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/238270
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
2020-05-22 18:49:41 +00:00
Jeremy Neander 8d12ab179b spec: remove explicit jest guidance for now
Change-Id: Ia1f1279db44bc8cc38a03a30163c9e3b41ebfa26
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/238044
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Spencer Olson <solson@instructure.com>
QA-Review: Jeremy Neander <jneander@instructure.com>
Product-Review: Jeremy Neander <jneander@instructure.com>
2020-05-20 19:21:42 +00:00
Ethan Vizitei 5decb828ec add a serialized fetch strategy
refs CNVS-48876
flag = none

TEST PLAN:
 1) backfill auditors
 2) log show CQL fetchs are max one ID per group
 3) throughput on cassandra ring does not spike as sharply with
higher parallelism

Change-Id: I96a24bf7273f089612ddf97fe4c038ff7cab490b
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/237810
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
2020-05-19 20:24:51 +00:00