fixes INTEROP-6250
flag=none
Test plan
- Set up tools with varying endpoints and verify
we're sending the "closest/latest" endpoint ids
from among a plethora of similar tools on courses/
sub-accounts/accounts
- Verify the migration does the same thing
Change-Id: Iac761e94cfeee01588cae5cd8b4797faa1bcb643
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/249736
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
Reviewed-by: Xander Moffatt <xmoffatt@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Tucker Mcknight <tmcknight@instructure.com>
Product-Review: Weston Dransfield <wdransfield@instructure.com>
fixes INTEROP-6247
flag=none
Test plan
- Create tools with different endpoints but matching
product families
- Ensure they get different subscription ids
- Delete those IDs and run the data fixup
and ensure they get different IDs that way as well
Change-Id: I5b427616b3caf5850ef54e033d3ab72f37b532ba
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/249673
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Tucker Mcknight <tmcknight@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Product-Review: Mysti Lilla <mysti@instructure.com>
test plan:
* before checking out the patchset,
have some editor button tools installed on a root account
and at least one set to be an rce favorite
* after checking out the patchset but before running
migrations, ensure that the previously configured
rce favorite tools still show up in the rce
and still are marked as a favorite in the root account settings
* after running migrations, ensure that everything appears the
same again
* now, should be able to configure sub-accounts external tools
to have their own set of rce favorites
closes #LS-1420
Change-Id: Ie95b9860149ab11ad5227544e4881d6b36bb6ee3
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248632
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jeremy Stanley <jeremy@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
test plan
- run the following with group category object and file contents
- GroupAndMembershipImporter.create_import_with_attachment(category, file)
- it should return a progress
- it should import groups and group members
closes VICE-813
flag=none
Change-Id: Iccc7a4f491d7efbe035057a2915f318fd695deb1
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/249030
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Matthew Lemon <mlemon@instructure.com>
QA-Review: Matthew Lemon <mlemon@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
test plan
- specs should pass
Change-Id: I207aa5f16f481db74a0d3dd6536bbc033a07f538
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/249336
Reviewed-by: John Koropchak <jkoropchak@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
Tested-by: Rob Orton <rob@instructure.com>
refs INTEROP-5767
flag=none
Test plan
- Set up a couple of submissions associated to a plagiarism
tool
- Set Setting.set('trigger_plagiarism_resubmit', '1,10')
in the console
- Change the resend_plagiarism_events.rb file and remove
the "if_production" parts, change to send_later_enqueue_args
so you can see the jobs try to work
- Run DataFixup::ResendPlagiarismEvents.run and verify
you get a couple of jobs in the Delayed::Jobs queue that have
reasonable run_at times.
- Ensure that the live-event sends properly when they fire
Change-Id: I3441ebbdf722961445703ba13f0ce03490a28c7a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248967
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Tucker Mcknight <tmcknight@instructure.com>
Product-Review: Weston Dransfield <wdransfield@instructure.com>
this changes upgrades to the earliest version of rubocop
supporting ruby 2.6 while also choosing a version that fixes
a breaking 0.60.0 change to revert psych version change.
namespace changes were made on multiple cops to support
the new rubocop version requirements as well as the change to
move performance cops out of the rubocop gem in to
rubocop-performance.
the Layout/IndentFirstParameter cop is currently broken but a
rubocop version upgrade will fix that. the FreezeConstants cop
also required a fix to handle nil types.
refs DE-289
test plan:
- ensure rubocop runs completely
Change-Id: I0ca68f426e287b5a973b96a4ae0e975f1cbc0240
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248786
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Product-Review: Kyle Rosenbaum <krosenbaum@instructure.com>
Reviewed-by: Ryan Norton <rnorton@instructure.com>
fixes INTEROP-6130
flag=plagiarism_tool_subscriptions (removes it)
Test plan
- Have a Lti::ToolProxy for TurnItIn or Urkund
or some other Plagiarism service that does
not have an associated subscription
- Run the data fixup. Verify the tool has
a working subscription
Change-Id: I21a85321caf1a735e0034661e175da3423548967
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246411
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Tucker Mcknight <tmcknight@instructure.com>
Product-Review: Mysti Lilla <mysti@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
closes FOO-1003
TEST PLAN:
1) aua log compaction continues to work
2) state on plugin setting and in metadata
table is in sync
Change-Id: Ie161fcb079f80a4072ddd99f3a561c97d33026fd
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248501
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
There are quiz questions which have a cross-shard quiz_id.
flag=none
refs INTEROP-6194
Test plan:
- before and after running migrations, try to create a QuizQuestion with
a cross-shard (global) quiz_id.
Change-Id: Ifd600774eb1aeea697e93a206cc4e26d0dd05530
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248376
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
QA-Review: Rob Orton <rob@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
fixes FOO-888
Repro (before applying this migration)
- create a course and publish it
- create a quiz in the course and publish it
- create two users, A and B, and enroll both in the course as students
- have user A start the quiz but don't finish it
- have user B start and finish the quiz
- merge user B into user A
- canvas go BOOM. unique key constraint violation.
After applying this migration, I went through the same repro, verified
that the issue was fixed, and indeed you end up with multiple
UserMergeDataRecords for the same QuizSubmission, but that doesn't seem
to be a problem. You can still do SplitUsers on the resulting user and
UserMergeData without issue.
Change-Id: I6752544cec8ed08e8229334d73efe02af5a31507
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/248267
QA-Review: Michael Ziwisky <mziwisky@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
Product-Review: Michael Ziwisky <mziwisky@instructure.com>
test plan:
* migrations should run
closes #LS-1464
Change-Id: I8da5b9e2347a6e32d42b738d9e38ebb237b1771a
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247651
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
closes OUT-3944
flag=none
test plan:
- confirm that these account permissions are
enabled for account admins (account roles)
and disabled but available for teachers and
designers (course roles)
- create a custom account role
- enable the "Learning Outcomes - add / edit / delete"
permission
- either run the data fixup code again or perform
the two steps above before running the migrations
from this patch set
- confirm the proficiency permissions are now enabled
on the custom role (reload Canvas permissions page
before confirming)
Change-Id: I6fb52c4f9eb0d5146453b55dba1fd14912818b7b
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247047
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Michael Brewer-Davis <mbd@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
QA-Review: Brian Watson <bwatson@instructure.com>
Product-Review: Jody Sailor
* run specs and test
Partially adapted from efe298c1c8
(clearing old User root_account_ids)
See also old CommunicationChannel backfill commit
8502957025
flag=none
closes INTEROP-6091
Test plan (multi-shard canvas):
- have some users and communication channels in different shards, with
some with cross-shard associations, and some comm channels in a
different shard as the user.
- run DataFixup::PopulateRootAccountIdOnModels.run on all shards to make
sure the Users all have root_account_ids.
- make sure you have some communication channels with root account ids
(this is not what we are testing here, if none are there, just set some)
- run migrations
- make sure the root account ids on the communication channels are
cleared (if you are running jobs they may start backfills so it's best
to not be running a jobs container)
- delete all jobs on all shards
(Shard.with_each_shard{Delayed::Job.delete_all}) to simulate the state
on production (with users backfill complete and no jobs)
- run DataFixup::PopulateRootAccountIdOnModels.run on all shards
- check that all communication channels have the correct root
account ids (comm channels should match the users)
Change-Id: I1ae5d4fb50bd5dadd62fe394ec8b4072f03edb3d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247183
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Mysti Lilla <mysti@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Evan Battaglia <ebattaglia@instructure.com>
and make role root_account_id not null
the datafix is done so we can remove the code
handling the old built-in roles
test plan:
* specs
closes #LS-1425
Change-Id: Ic8a115b71383c561b55275636bd93c194d752610
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247327
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Robin Kuss <rkuss@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
closes OUT-3890
flag=none
Test-plan:
- Verify that user interactions behave as expected
when testing with various different permissions
turned on/off when using various different user
roles
- Granular pages permissions can be found under
/accounts/self/permissions > Manage Pages
Change-Id: I0718a781991bf66fd90a14489710f0a68a4e4cb9
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/245714
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Pat Renner <prenner@instructure.com>
Reviewed-by: Augusto Callejas <acallejas@instructure.com>
Reviewed-by: Michael Brewer-Davis <mbd@instructure.com>
Reviewed-by: Charley Kline <ckline@instructure.com>
QA-Review: Augusto Callejas <acallejas@instructure.com>
Product-Review: Jody Sailor
also run it less often
the gist is to iterate by users, instead of alerts, so that indexes can be more
effective. also, don't join against the world - select only the columns we actually
need, and carefully form our joins. this way we can also avoid a bunch of queries
post-load as well. it also makes the final query much easier to grok
on a middle sized database, this changes from ~75 queries of ~22s each (with no
improvement with a warm cache) to ~22 queries of ~10s each (down to ~30ms on a
warm cache)
Change-Id: I9f2cbae3aa3fce4d5ff9a1e2fa424ffca11e042f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246829
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
flag = none
closes LS-1436
Change-Id: I4f303f09976bc4be377bc560dcb2c5ce5d9fc7dd
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247127
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: Robin Kuss <rkuss@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
closes FOO-937
* 7-tables by day-of-week
* infer write table from timestamp
* plugin with default settings values
* instructions on setting the local shard to use log-&-compact
* write path switches on setting to log compaction for view bumps
* compaction job
* single write per log segment (multpile updates)
* truncates previous partition
* periodic job harness with available settings
Change-Id: Idd5cbaf0d83436724c5d72038e0dbc78cdc23619
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/247083
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
Refs FOO-905
flag=none
We've removed this permission from Canvas, so let's just also
go remove all role overrides everywhere that reference it.
Test plan:
* If you had any RoleOverride records referencing the
:view_learning_analytics permission, they should be gone after
running DataFixup:ClearAnyCanvasInsightsViewRoleOverrides.run
Change-Id: I028eaa929dc4ae1176b683980ad2d3cb07c2dfbf
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246805
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: Charley Kline <ckline@instructure.com>
Product-Review: Charley Kline <ckline@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
test plan:
* a multi-tenant shard should still work
creating enrollments and account admins before and after
running postdeploy migrations
* the postdeploy should successfully run a datafixup
to split up build-in role rows and reassign the role_id on
other tables (account_notification_roles, account_users,
enrollments, and role_overrides) accordingly
closes #INTEROP-6045
Change-Id: I8325acb635397ac1b008b9d09788595da597cec7
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243906
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Jeremy Stanley <jeremy@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Mysti Lilla <mysti@instructure.com>
it's far more common for wiki pages to not be linked to wiki pages, so only look up their
assignment visibilities if necessary
Change-Id: Idf3f05e2c5e59a2163bd692497160c2388551be2
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246533
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
so that clean up is much faster
Change-Id: I265091dc328a54fb849b3eff39cad685e7889ae1
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/246525
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
flag=none
closes INTEROP-6094
Test plan
- Run the database migration
and ensure Account 0 is created
Change-Id: I9a5e6f8934c76ea97997a41eeecdd7fef7e76e95
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/245895
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Michael Guymon <mguymon@instructure.com>
Product-Review: Michael Guymon <mguymon@instructure.com>
flag=none
closes EVAL-1128
Test plan:
- Migrations on Cassandra run successfully
- Grade changes save the grading_period_id of the associated submission
in both Cassandra and AR
Change-Id: I7ae0ff54093b9bf1eaea43106d4788f22c6c920e
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/245734
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
Reviewed-by: Spencer Olson <solson@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Adrian Packel <apackel@instructure.com>
Product-Review: Jody Sailor
refs INTEROP-6080
flag=none
Test plan
- Have some developer keys with account ids
from a different shard
- Run DataFixup::CleanupCrossShardDeveloperKeys.run
and verify they get deleted
Change-Id: Ib389ef37b33a8a6e73c4926d9db775542dc6629f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/245434
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Mysti Lilla <mysti@instructure.com>
shares belong to the user's shard, not the content export's
so shares created for a course on a shard
that the user doesn't belong to will break
and shares sent to cross-shard users that create "received"
copies of the original share will also break
also semi-inadvertently fixing share sender info in API
and some problems on the import side when the correct
shard isn't active
test plan:
* cross-shard content shares should work
(both when sharing content from a course from a shard
the sending user is not native to, and when sharing content
to a user on a shard different than the content's course)
closes #LS-1337
Change-Id: Ib91529ffa5de0ee7a6f20f63281d6c5508982063
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/245496
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jeremy Stanley <jeremy@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
test plan:
* specs pass
Change-Id: Ib66137d96d8ae06634f4e305b9354b306a0a2405
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/245475
Reviewed-by: James Williams <jamesw@instructure.com>
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
closes OUT-3841
flag=none
test-plan:
- run migrations
- Ensure you can soft delete outcome proficiencies
(and their associated ratings)
- dc run --rm web script/rails c
> o = OutcomeProficiency.first (alternatively,
copy the factory code from factories/outcome_proficiency_factory
to create one for an account)
- ensure it has a workflow state (should be "active" by default)
- ensure the ratings have an "active" workflow state by default
> o.outcome_proficiency_ratings.map(&:workflow_state)
> o.destroy
- Ensure the outcome proficiency was soft deleted
(it should now have a workflow state of deleted)
- Ensure the ratings were soft deleted
> o.outcome_proficiency_ratings.map(&:workflow_state)
- Ensure calling undestroy restores both the proficiency and the ratings
> o.undestroy
> o.outcome_proficiency_ratings.map(&:workflow_state)
- Calling destroy_permanently! should hard delete
both the proficiency and all associated ratings (examine SQL output)
> o.destroy_permanently!
Change-Id: I6421fdfdd97b991e76fb08b528cddbc668495ac8
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/244166
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Reviewed-by: Augusto Callejas <acallejas@instructure.com>
Reviewed-by: Michael Brewer-Davis <mbd@instructure.com>
QA-Review: Augusto Callejas <acallejas@instructure.com>
Product-Review: Augusto Callejas <acallejas@instructure.com>
refs SAS-1540
* adds an audience setting to developer keys, so a key can be set to
target external audiences with its credentials grants
* when a key with an external audience grants credentials, the token is
signed with an asymmetric key instead of the internal symmetric key
* external audiences can retrieve the corresponding public keys from
/login/oauth2/jwks
* credentials issued by developer keys with an account id include the
account's guid in a custom claim
includes a refactor of key storage and rotation in consul, which had
already been done for LTI. but it wasn't really a feature of lti, just
something used by LTI, and we needed the same for key management for
this. moved it to be part of Canvas::Security
Change-Id: Ie5c0fcee6fc21687f31c109389a3bcc1ed349c5d
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/243606
QA-Review: Jonathan Featherstone <jfeatherstone@instructure.com>
Reviewed-by: Jonathan Featherstone <jfeatherstone@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Product-Review: Jacob Fugal <jacob@instructure.com>
Must be merged and deployed with the "Make Users root_account_ids
relative to user shard" (g/244742) commit.
The migration here is named to run before
20200806175207_populate_root_account_id_on_models
kicks off the users backfill.
flag=none
refs INTEROP-6063
Test plan:
- make sure you have some users with root account ids with all global
ids (ids are still global when on same shard as user)
- run migrations
- make sure those root account ids are cleared (if you are running jobs
you will also start backfills so you may notice users filling up again
but they will have local root account IDs for root accounts on the same
shard as the user)
Change-Id: I047afe45989f2e86ca711e24683beca4557ce1c6
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/244804
Reviewed-by: Mysti Lilla <mysti@instructure.com>
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Evan Battaglia <ebattaglia@instructure.com>
Mysti Lilla