[fsc-max-nodes=12]
[fsc-timeout=60]
* switch lots of parsing to Nokogumbo to keep things consistent
* deep CSS sanitization is now built in, and with a proper parser (meaning
we can drop our code to do it, and adjust some specs to account for things
that _are_ valid)
lots of changes because gumbo parsing<->serialization cycle is slightly different:
* better job preserving original whitespace
* literal non-breaking space characters are converted to entities
* <p> tags aren't inserted for the heck of it
* several _other_ entities are unnecessary, and output as literal characters
* some elements no longer have a closing tag
Change-Id: I7c5e36cbd04b8a05f64c9e0af00868dd6b00f4ce
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/256444
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
closes LS-1481
flag=none
though since media objects are never deleted, maybe this is a waste!?!
With this change, when presented with a mediai viewer URL (a url that
looks like "/media_objects_iframe/media-id-here") it
1. finds the Attachment with the matching media_entry_id which it uses
to figure out the context
2. finds the MediaObject
3. validates the MediaObject
test plan:
- in the RCE, embed media
- in /course/:id/link_validator, click the validate links button
> expect it to say all's good
- in the RCE's html editor, change the media object's id in the
media player's iframe's src
- go back and run the link validator
> expect the URL to be reported as
"Non-existent content referenced in this resource
- in the RCE, using the Media Options tray, change the media
to "Display Text Link" and save
- rerun the validator
> expect the URL to be reported as non-existent
Change-Id: Ia9f43c5cf16fa489b1ad055cb31de025926ca31c
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/256262
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Nate Armstrong <narmstrong@instructure.com>
QA-Review: Nate Armstrong <narmstrong@instructure.com>
Product-Review: Ed Schiebel <eschiebel@instructure.com>
Change-Id: Id39b1e8d3bebcc0f9b7ded920b450f275565f708
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/250923
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
also don't count a 429 request as unreachable
test plan:
- create links to a nonexistent domain such as bad_domain.net
- ensure the course link validator flags them as unreachable
- do Setting.set('link_validator_whitelisted_hosts', 'bad_domain.net')
- re-run the course link validator
- the links should no longer be flagged as unreachable
flag=none
fixes LA-97
Change-Id: I4e46ce252bed527ad8cb584992eb7d12c9df743f
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/220915
Tested-by: Jenkins
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
closes ADMIN-2334
Test plan
- In a course with a student enrolled
- Set up an assignment, a quiz, a wiki page
and a discussion topic
- In the body of the above, make sure to set
up at least one notifiable bad link (a link
to a course the student doesn't have access
to, something that 404s like offline_web_exports
if it's not turned on, something unpublished.
deleted doesn't work, because it redirects
right now)
- Set up an instructor with immediate, daily,
weekly and no notifications for Content
Link Errors
- As the student, visit the links
- Check /messages for the instructor(s) to ensure
the notifications were received (or not) as
expected and the notifications look right
according to ADMIN-2334
- (If you need to send daily/weekly notifications
you can do `DelayedMessage.where(workflow_state:
'pending').update_all(send_at: 1.day.ago)` and
then `SummaryMessageConsolidator.process` when
jobs are running to kick them off
Change-Id: Ia324b1f5360dda91641f22ff1a6a2ba528aa4669
Reviewed-on: https://gerrit.instructure.com/179823
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jeremy Stanley <jeremy@instructure.com>
Tested-by: Mysti Sadler <mysti@instructure.com>
Product-Review: Christi Wruck
these don't contain URLs
test plan: quiz answer comments containing "rel" attributes
should not be flagged as broken links
fixes ADMIN-2362
Change-Id: Ie4962f34a971f1af58e0fe469a00d106481b6dc7
Reviewed-on: https://gerrit.instructure.com/180486
Tested-by: Jenkins
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: Anju Reddy <areddy@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
even if they match the domain
closes #ADMIN-2325
Change-Id: I917ee322097091c2927a9eb09a37b80b48a6a61d
Reviewed-on: https://gerrit.instructure.com/176490
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Tested-by: Jenkins
QA-Review: James Williams <jamesw@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
so if flickr changes its "image unavailable" redirect target,
we can adapt without a hotfix
test plan:
- ensure deleted flickr photos still appear in the
course link validator
refs ADMIN-2280
Change-Id: I5c478d5da77c4857e4bf872fb1c04e108b7ee9ae
Reviewed-on: https://gerrit.instructure.com/174798
Reviewed-by: Carl Kibler <ckibler@instructure.com>
Tested-by: Jenkins
QA-Review: Carl Kibler <ckibler@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
this commit includes the following enhancements:
* broken course card images are detected
* deleted flickr images (redirects to image_unavailable.png)
are treated as broken
* the type and icon for each resource that has broken links
are included in the list
* broken links in each resource are grouped by the type of
brokenness, with explanatory text
* broken external URL items in modules are grouped by module
* link text for bad links is shown, rather than just
the URL
test plan:
- add the following to various objects in a course:
- a flickr image (that isn't deleted or broken)
- a flickr image where the name part of the URL has been changed
(so it redirects to an "image_unavailable.png")
- an image with an invalid hostname
(so it is unreachable)
- a broken image of some sort as the course card image
- links to an unpublished item
- some external URL module items in multiple modules
with broken URLs
- run the Course Link Validator (from Course Settings, right sidebar;
ensure jobs are running)
- make sure the good flickr image isn't reported as broken
- make sure the broken flickr image is
- make sure the results include the course card image
- make sure the unpublished item appears
when the "Show links to unpublished content" is checked
- make sure the broken external URL module items appear
under their corresponding modules
closes ADMIN-2280
Change-Id: Ia30d128eb19244cca34f64dc716d7c6f5ef1cdf8
Reviewed-on: https://gerrit.instructure.com/173440
Reviewed-by: Carl Kibler <ckibler@instructure.com>
Tested-by: Jenkins
QA-Review: Anju Reddy <areddy@instructure.com>
Product-Review: Christi Wruck
closes #ADMIN-700
Change-Id: If85d72013be018c4259ea23ebfae4121dbcb7b29
Reviewed-on: https://gerrit.instructure.com/138584
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Change-Id: I60ec3b519219fcb5256a132c05f63538600df73f
Reviewed-on: https://gerrit.instructure.com/121760
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
test plan:
* make sure canvas is configured with a
conditional release service with g/85046 merged or checked out
* create assignments in a course with conditional content
* after copying the course, or exporting/importing the course,
the new assignments should have conditional content
closes #CNVS-30371 #CYOE-235
Change-Id: I42693f2d12185f5c7f665d303baf938ad4af08ad
Reviewed-on: https://gerrit.instructure.com/85108
Tested-by: Jenkins
Reviewed-by: Michael Brewer-Davis <mbd@instructure.com>
Reviewed-by: Dan Minkevitch <dan@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Christian Prescott <cprescott@instructure.com>
QA-Review: Alex Morris <amorris@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
sometimes servers don't respond to HEAD requests, so
use GET as a fallback
also improve wording for link validator
test plan:
* add links to the first two urls referenced in the ticket
into course content
* the course link validator should not flag them
* (note: the last link referenced will still fail because it
redirects too many times)
closes #CNVS-28962
Change-Id: I80f2b8d3e3a0ad1d82e818af04d4c3b9a2a6c8c6
Reviewed-on: https://gerrit.instructure.com/78244
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Felix Milea-Ciobanu <fmileaciobanu@instructure.com>
QA-Review: Deepeeca Soundarrajan <dsoundarrajan@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
use HEAD instead of GET and accept only gzip compression
test plan:
- create a page and use these links:
- http://www.osa.org/en-us/home/
- http://www.aaes.org/
- start Course Link Validator
- the links above should not be listed under broken links
fixes CNVS-27460
Change-Id: I8b75b48249b1c8b0e87ee8fa417373357b69462d
Reviewed-on: https://gerrit.instructure.com/74461
Tested-by: Jenkins
Reviewed-by: James Williams <jamesw@instructure.com>
QA-Review: Deepeeca Soundarrajan <dsoundarrajan@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
test plan:
* use the wiki sidebar to "Link to a new page", using a page
title with a special character (e.g. ";" or "?")
* click the link and create the page
* run the link validator
* it should not flag the page link
closes #CNVS-27203
Change-Id: Ib0d33771bbaa5d83b60adaaf59eca46f49e59322
Reviewed-on: https://gerrit.instructure.com/72675
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jahnavi Yetukuri <jyetukuri@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
test plan:
* create a question bank
* create a question in the bank with an invalid link
* the course link validator should find it
* delete the question bank
* the course link validator should not flag the link
in the question anymore
closes #CNVS-27389
Change-Id: Icab0298a071dc4c668ebbd0bfaab1e616e0c71af
Reviewed-on: https://gerrit.instructure.com/72677
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jahnavi Yetukuri <jyetukuri@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
test plan:
* create a page or other html content with a link to a
file in the /public directory (e.g. "/images/avatar.png")
* the course link validator should not mark it as a broken link
closes #CNVS-26982
Change-Id: Ibba746679810ba93256e39635ebf8e5826bacd56
Reviewed-on: https://gerrit.instructure.com/72370
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Tested-by: Jenkins
QA-Review: Jahnavi Yetukuri <jyetukuri@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
closes #CNVS-26144
Change-Id: Iccf3c707cd3d1e09ef90d878d74b472f08ec9576
Reviewed-on: https://gerrit.instructure.com/70988
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
test plan:
* the course link validator should now do the following:
* not incorrectly flag wiki page urls for pages
with spaces in their name (and hence dashes in the urls)
* not incorrectly flag files as missing when
overwritten by one with the same name
(the link still works by finding the replacement file)
* should identify links that point to completely invalid
canvas routes (e.g. take a working url and add a typo)
closes #CNVS-26247 #CNVS-25877
Change-Id: Ia2fdff15504a3044eeb8b09e1f1d8c0b60459054
Reviewed-on: https://gerrit.instructure.com/69851
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jahnavi Yetukuri <jyetukuri@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
test plan:
* the course link validator should now detect
links that point to different courses
* also the special invalid types (linking to an
unpublished/deleted item or to a different course)
should have an explanatory label
* also there should be a checkbox to hide the
unpublished links in case they don't matter yet
closes #CNVS-25715
Change-Id: I71f1f38588c6c6392d508f848cf5287182633bb8
Reviewed-on: https://gerrit.instructure.com/68657
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Ryan Allen <rallen@instructure.com>
Product-Review: Cosme Salazar <cosme@instructure.com>
test plan:
* create an assignment/quiz/wikipage
* create a link to it in course content
* delete or unpublish the assignment/quiz/wikipage
* the link validator should point out the invalid link
closes #CNVS-25559
Change-Id: I2326dbdc81beaa8efb7726447688357e492a8398
Reviewed-on: https://gerrit.instructure.com/68266
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Tested-by: Jenkins
QA-Review: Ryan Allen <rallen@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
test plan:
* create course content with various valid and invalid links,
including the example links described in the ticket comments
* running the course link validator should not give false
positives for these example addresses
* should work otherwise as normal
closes #CNVS-25484
Change-Id: I5d3a98b37e5c7696a2c60248df2376cd95dc9b0d
Reviewed-on: https://gerrit.instructure.com/68164
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Ryan Allen <rallen@instructure.com>
Product-Review: Cosme Salazar <cosme@instructure.com>
Fixes CNVS-16333
Test Plan:
1. Post a few broken links in various places throughout the course,
such as in discussions or announcements
2. from the course settings, on the right-hand side, click
on the "Validate Links in Content" button to reach the
link navigator
3. Click Link Validator button
4. You should expect the button to change to a Loading message
then see a list of results
5. Click the link of one of the results
(it should take you to the location of the broken link)
6. Remove link and re-run Link Validator
to ensure that link no longer appears
Change-Id: I5cba118a1781eff427ef7d9690ed5bfa752dd785
Refs: CNVS-16333
Reviewed-on: https://gerrit.instructure.com/58125
Reviewed-by: Clay Diffrient <cdiffrient@instructure.com>
Tested-by: Jenkins
QA-Review: Ryan Allen <rallen@instructure.com>
Product-Review: Cosme Salazar <cosme@instructure.com>
test plan:
* using "mailto:example@example.com," in a link should
not break course copies (or anything else) because it
doesn't know how to rescue a URI::InvalidComponentError
closes #CNVS-22120
Change-Id: Iaac362a5bf33cd5fde05b01a578a1325f4126c6e
Reviewed-on: https://gerrit.instructure.com/59213
Tested-by: Jenkins
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Clare Strong <clare@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
closes CNVS-6016
No more error reports! (soon)
this commit builds up sentry integration through the new
Canvas::Errors module, along with other things that need
to happen on every exception. ErrorReports
should now get pushed towards just being used for representing
a complaint a user filed via the get help form.
I fixed about half the things that got linted as well
while I was in here, but because this touches to much
I fear divergence from tackling too many (I think we
can safely say it's "better than we found it")
I left a lot of the infrastructure for error reports in place
until other commits for plugins can be merged
TEST PLAN:
1) setup your raven.yml config file with the dsn for our
sentry install
2) force an error to happen in a request response cycle.
3) see the error in sentry
4) force an error to happen in a job
5) see the error in sentry
6) statsd increments shoudl still fire
7) for the moment, an error report should still get created.
Change-Id: I5a9dc7214598f8d5083451fd15f0423f8f939034
Reviewed-on: https://gerrit.instructure.com/51621
Reviewed-by: Simon Williams <simon@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
Tested-by: Jenkins
QA-Review: August Thornton <august@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
test plan:
* the course link validator should look at all user supplied
HTML content in a course and return a list of bad links:
- links to external content that does not return
a valid response (i.e. dead links).
- misconfigured urls
- links created by missing file references in imports
* use the (currently) undocumented link validator API
POST /courses/:course_id/link_validation
- Starts a link validation process
GET /courses/:course_id/link_validation
- Gets the current process status
and results (if processed)
* should return a list of items in a course, the
bad urls, as well as a direct link to the item (for the
future UI to follow)
refs #CNVS-15162
Change-Id: I613fff7d603e133cc155e5fb33c7f257affe0f48
Reviewed-on: https://gerrit.instructure.com/42821
Reviewed-by: James Williams <jamesw@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jahnavi Yetukuri <jyetukuri@instructure.com>
Product-Review: Hilary Scharton <hilary@instructure.com>
Cody Cutrer