for the convenience of api users
Test Plan:
* The top-level return value of the selective api call
should include urls to the sub-items lists
* the top-level url is: api/v1/courses/:course_id/content_migrations/:id/selective_data
refs CNVS-6061
Change-Id: I35b9288a1471dcbe66462703636e9b45dcafa31e
Reviewed-on: https://gerrit.instructure.com/21249
QA-Review: Clare Strong <clare@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Bracken Mosbacker <bracken@instructure.com>
Reviewed-by: Bracken Mosbacker <bracken@instructure.com>
test plan:
* enable cassandra page views
* visit /users/X for a user with page views
* search the rails log file for CQL lines, they should include the
cluster name and the environment name like "[page_views development]"
Change-Id: I61903e55c01c818f33022f2971905951a31b8f0e
Reviewed-on: https://gerrit.instructure.com/21302
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
this endpoints returns all the items for a migration so
that you could build a UI around selecting what to copy
when importing/copying courses
Test Plan:
* Start a content migration but don't finish the selection part
* use the /api/v1/courses/:id/content_migrations/:id/selective_data endpoint
* if called without args you should get the top-level areas
* if called with ?type=assignments (or the types in the top-level list)
you should get a list of just the specified type
closes #CNVS-5940
Change-Id: Idfc8bd9b02738801a3bf5b18df25314350929762
Reviewed-on: https://gerrit.instructure.com/20873
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Sterling Cobb <sterling@instructure.com>
QA-Review: Clare Strong <clare@instructure.com>
Product-Review: Bracken Mosbacker <bracken@instructure.com>
During the EventStream refactor we changed the page view inserts from
using PageView#changes to PageView#attributes. This was the right thing
to do, because #changes won't include columns that have the
default value for that column, but cassandra needs to be given any
non-nil default values. Not a problem for PageView, but it could be for
other models.
However, this introduced a performance regression, because #attributes
includes nil attributes which means the cassandra driver was sending a
DELETE (list of columns) command, creating a new tombstone record in
cassandra. The new insert_record() method on the driver removes any nil
values before calling update_record.
test plan: enable cassandra page views, then visit some pages. after the
insert job runs, the page views should have been created properly and
still display properly in the history.
Change-Id: Iccb967d3ec1f6b295d98b3f330b4b8ffe4508437
Reviewed-on: https://gerrit.instructure.com/20151
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
so that it can be used elsewhere
refs CNVS-4466
test plan: specs
Change-Id: I7246e5f878867c839bf276ffc857f30ee47b5bc8
Reviewed-on: https://gerrit.instructure.com/19646
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
Tested-by: Brian Palmer <brianp@instructure.com>
test plan: add "append_hostname: false" to your statsd config and
generate statsd metrics by visiting canvas pages. no hostname should be
appended to the key, so all servers in the environment will write to the
same metric.
Change-Id: I4662601ae04661aee2a75b39eeb58c7902870a38
Reviewed-on: https://gerrit.instructure.com/19342
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
filter_parameter_logging is deprecated in 3.0 and removed in 3.1
test plan: when logging in, your password should still be filtered from
the log file in both rails 2 and rails 3
Change-Id: I697d5a4aca1a7501fe84a52c0097f8ae81074a0a
Reviewed-on: https://gerrit.instructure.com/19085
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
ActiveSupport::SecureRandom was merged into the ruby stdlib. In rails
3.0 it is just an alias to ::SecureRandom, and is removed completely in
rails 3.1.
This stdlib exists in ruby 1.8.7+
Change-Id: I096b212c020fd60e3799a9d1635129944ac3b6e4
Reviewed-on: https://gerrit.instructure.com/19080
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
adds scopes to access token and a new scope, 'userinfo'. when this scope is
used, a user may choose to remember authorization for a 3rd party. when this
option is selected, subsequent requests for an access token scoped to userinfo
will skip the the step where the user authorizes the app and will return userinfo
but no access token.
test plan:
* follow the oauth token flow adding a param for scopes=%2Fauth%2Fuserinfo to the initial request
- check the box for to remember authorization
- click login
* repeat the above request
* you should not see the request access page
* delete the tokens that were generated above
* run the test above, this time not remembering access
* you should see the request access page on the second request
Change-Id: I303a55d3c71de517ce6aa5fd8acd74d89aa4c974
Reviewed-on: https://gerrit.instructure.com/17604
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
this will allow migrations to have flexible todo items
and allow us to make a super awesome UI for helping
teachers fix any migration problems
Test Plan:
* Run a migration that has issues. :)
* exercise the issue api
closes CNVS-4230
Change-Id: I4577f811dd3b16aa200d381f039632b7cc2cd184
Reviewed-on: https://gerrit.instructure.com/18639
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Adam Phillipps <adam@instructure.com>
Add a batch method that uses cassandra's BEGIN BATCH/APPLY BATCH
functionality to cut down on the number of network round trips, and
possibly save a bit of CPU time on the cassandra servers.
Since execute returns a result set, it can't be batched, so we add another
method update for INSERT/UPDATE CQL statements that don't return a
value.
Closes CNVS-3526
test plan: No behavioral change. Do a regression test that page view
information is still send to cassandra properly. You can check the debug
logs to see that the CQL statements are all sent in one big BEGIN BATCH
statement.
Change-Id: Ibca4f6fbd84f2644436599c017f1ec8c39783e36
Reviewed-on: https://gerrit.instructure.com/17156
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
fixes #CNVS-1886
SQL statements were noted that would fail
when a string was passed to a DeveloperKey
query when trying to match against the id column.
With grep I was only able to find 2 likely vectors,
one of which (developer key special keys) I
wrote a spec to confirm it wasn't an issue,
the other (canvas/oauth/provider) I wrote a spec
that uncovered it could be exploited in the
way described in the ticket, so I've closed
that possibility by including client_id validation
which confirms both that the client_id exists
and that it either is an integer or is
cast-able to an integer.
TEST PLAN:
No behavior has changed, this just closes
a vulnerability to bad data.
1) Try to login to canvas using a valid OAuth
login process, just to confirm there is no
regression.
2) Now, login through an OAuth pathway but make
sure the client ID is something nonsensical like
'XXXXX' or some other non-numeric string.
3) instead of causing a database error, you
should get a 400 and the JSON should have a
message: "invalid client_id"
Change-Id: Ic32a0a4498400eccc75c6d248e888439e61257d6
Reviewed-on: https://gerrit.instructure.com/17050
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
refs #CNVS-523
This will now pass the canvas_login parameter
on through the oauth workflow.
there are also several small refactors to pull
the oauth workflow out of the pseudonyms
controller and into a couple lib objects
TEST PLAN:
1) As a site admin, attempt to use the OAuth
through a domain that uses delegated authentication
2) add the "canvas_login=1" url parameter to your
oauth url
3) you should be taken to the regular canvas login
screen instead of the CAS url.
Change-Id: I8b68e5b1c6914b6109af6aabcbba03ed15c4f9cf
Reviewed-on: https://gerrit.instructure.com/16240
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Ethan Vizitei <ethan@12spokes.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Test Plan:
* Add a course to term with no start/end dates
* Add a teacher to the course and set start/end and check the checkbox to restrict users
* As the teacher (not an account admin) go to the course
* The teacher should still have access
* Students shouldn't have access
closes #cnvs-1189
Change-Id: I7c4d0d0d5075d0c52b5fc190f62b69fc8f0d5819
Reviewed-on: https://gerrit.instructure.com/15799
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Adam Phillipps <adam@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Tested-by: Bryan Madsen <bryan@instructure.com>
When calculating enrollment dates for admins the course
start date will be ignored. It can only be restricted by
the term dates
Test Plan:
* Create a course and set a start date in the future
* Check the box to restrict access to the dates
* Save the settings.
* Your teacher should still have access to the course
Change-Id: I25bad8d14b8eaeb3c7a823f61189078c533d3c6b
closes: #11954
Reviewed-on: https://gerrit.instructure.com/15572
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Ruby 1.9 uses a different YAML engine (although
we reset it to the 1.8 engine in the delayed
job plugin). This resulted in the yaml
dump to the assets.yml file (for plugin
asset inclusion) not producing
quite the expected output format at server
startup (before the delayed job plugin had
loaded), which caused
selenium tests to fail in the canvalytics
plugin because they ONLY loaded the assets.yml
file at startup rather than before each
request like in development.
A small regex change in the ruby block
in assets.yml fixed the actual problem. While
I was in here, I moved the ruby block into
it's own class so that all the processing
can be tested. the "subdoc" stuff in bundle_yml
could probably still stand to be slimmed down
a bit but this is a decent intermediate step.
Change-Id: I9300c167130fb3305c7f37710b0e3cfcfda19f48
Reviewed-on: https://gerrit.instructure.com/15111
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
fixes#9185
This adds a new version of export files and ensures
older exports will work. We are now using a
canvas_export.txt resource to identify if this is a
canvas cartridge or a common cartridge.
Test Plan
1. Copy a course
2. When copying, unselect copy all and course syllabus
3. The syllabus shouldn't be copied over.
Change-Id: I37c9aa12aabe453ef4481c6f39b7b33c97b130bb
Reviewed-on: https://gerrit.instructure.com/14971
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Made a helper class to make it more clear how the enrollment
dates are processed.
Test Plan: n/a
refs #11387
Change-Id: Id8458793faf0abfb4645cbf0364a4c945ec6023d
Reviewed-on: https://gerrit.instructure.com/14749
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Sterling Cobb <sterling@instructure.com>
Reviewed-by: James Williams <jamesw@instructure.com>
Adds a new back-end store for page_views, using a Cassandra cluster. All
the current page view queries are supported, many using denormalized
views on the data.
test plan:
first, canvas instances that are currently using AR page views
should function as before.
by Setting.set('enable_page_views', 'cassandra') and restarting, you will
switch to cassandra page views. a script to migrate the AR page views to
Cassandra is coming. all page view functionality should work as before.
note that the format of the pagination headers in the
/api/v1/users/X/page_views endpoint has changed.
Change-Id: I2d1feb4d83b06a0c852e49508e85e8dce87507b4
Reviewed-on: https://gerrit.instructure.com/14258
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
This adds an alternative method for uploading files by giving Canvas a
public URL in the first step, rather than uploading the file data directly.
test plan:
- create a course file via the API upload mechanism
- make sure the return values are as documented
- make sure the file was correctly uploaded
- create a course file via the URL approach
- make sure the return values are as documented
- make sure the file status endpoint returns valid responses
- make sure the file was correctly stored in Canvas
- repeat that process with a file that has at least one redirect
- repeat that process but creating a homework submission file
- try to create a course file with a malformed URL
- confirm that the appropriate error message is returned
- try to create a course file with a relative URL
- confirm that the appropriate error message is returned
- try to create a course file with a URL that doesn't return 200
- confirm that the appropriate error message is returned
Change-Id: I2dcf711347ec4ef26d767ae1c1fa0bb056986651
Reviewed-on: https://gerrit.instructure.com/12143
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Adds an infrastructure for using statsd (configured with
config/statsd.yml) and uses it to track a few basic stats. Stat names
are appended with the hostname.
test plan: without statsd enabled, make sure no errors are raised when
doing requests. add a statsd.yml configuration, restart the server, and
verify that stats are sent over UDP to the given host/port (this could
even be checked without statsd available, by monitoring UDP traffic)
Change-Id: Ie8c3ece7e08ff48616ffd968069bd760300e4fd2
Reviewed-on: https://gerrit.instructure.com/12673
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
fixes#9700
filter sensitive information from the data that is saved for an error report.
also add a few extra fields to the view, such as request_context_id and form
parameters.
test plan:
- make an action with the api that will cause an error report to be generated
- go view that error report
- it should have your access token filtered out
- it should have a request context id field
Change-Id: I3c4d0d8002b6f502fdeb9e4dd40f3fd5d51dc04d
Reviewed-on: https://gerrit.instructure.com/12625
Reviewed-by: Simon Williams <simon@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
This required building our own fork of the redis-store gem so that we
could update its dependency, and fix one small issue with redis connect
strings getting nil instead of the default value for the port number.
The redis 3.0.x gem now catches all Errno and Timeout errors and
re-raises them as subclasses of Redis::BaseConnectionError. It also now
handles EAGAIN internally, retrying when appropriate. So we've modified
our redis failure handling code to match.
test plan: verify the redis failure handling code still works (specs
pass). for instance, stop redis locally and see that canvas works in the
degraded state. make sure that redis still works for both caching and
non-caching code such as login attempts.
Change-Id: I9e8d3929afa06c522656d30f71efc0427e4ef7cc
Reviewed-on: https://gerrit.instructure.com/11521
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
This extends Attachment with support for having multiple thumbnail sizes,
and allow specified sizes to be generated dynamically the first time
that they are requested.
test plan: create two collection items, one with an explicit image_url
and one without. Verify they both get images after the jobs run, and the
images are properly scaled.
Also verify that giving a non-image or invalid image_url ends up with a
nil image_url.
Change-Id: I4d1b1de681f2cddda0c81f1b340d32c3b5b0097b
Reviewed-on: https://gerrit.instructure.com/10822
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
It's not valid for a zip to use backslashes, but some vendors
use them anyway. unzip handles it well so try to process it
anyway
Test Plan:
* Try to migrate a zip with backslashes
* It should not fail with a "could not unzip" error
closes#7379
Change-Id: Ie74506bffe70937871d865f63b1b5037b6678b2c
Reviewed-on: https://gerrit.instructure.com/9258
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Hudson <hudson@instructure.com>
This was causing media object failures for uses of the cloud kaltura
service, which began inserting these redirects recently.
fixes#7152
test plan: on an account that uses cloud kaltura, upload an audio or
video file, or do a canvas course import. verify that a MediaObject gets
created for that file.
Change-Id: Ia1380c012150329f09e1bdd0a17f0170e7bedfdc
Reviewed-on: https://gerrit.instructure.com/8544
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Hook into the redis library at a pretty low level, to try and do
everything we can to avoid erroring if redis goes down. This applies to
both redis-as-cache and redis-as-data-store.
test plan: Set up redis and caching in your local instance. Point it to
both an existing box on a port not running redis, and a non-existent IP.
In both situations, you should not see caching errors or redis data
errors. After the first error, it shouldn't attempt to hit redis again for 5
minutes.
Change-Id: I101b2d3d2123151b244eb82ba78b176ed1f4d5ad
Reviewed-on: https://gerrit.instructure.com/8097
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
The qti exporter uses the "Qti" module namespace but was in
the lib folder 'qti_exporter'. This moves everything to a 'qti'
folder and renames the 'exporter' to 'converter' to match
the convention of migration tool converters
The specs also weren't in spec_canvas to the CI tool wouldn't
run them, they were moved and made to only run if the python
qti tool is available
Test plan:
* Import some qti packages and make sure the still work correctly
refs #5178
Change-Id: I9277f2c4ecb0845b21ecb2e00102543e18a77aef
Reviewed-on: https://gerrit.instructure.com/7138
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
This uses redis to store the nonces as locks that expire after 90
minutes. Timestamps are epoch UTC values, as per the oauth spec.
testplan: send oauth requests to the api endpoint with the same nonce
more than once, or with a too-old timestamp
refs #5892
Change-Id: Id6130c2a07e206dad716673aa6adbe9d36565a7c
Reviewed-on: https://gerrit.instructure.com/6683
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Brian Whitmer <brian@instructure.com>
closes#5880
We track failed attempts for both (pseudonym) and (pseudonym, ip) in
Redis, the latter with a lower threshold. If either threshold is
exceeded, the user can't attempt to login for a given time period
(default 5 minutes). This protects against brute force auth attacks.
We've hooked into Authlogic for this, so it should apply to everywhere a
user is logged in -- login screen, API basic auth, Respondus API, etc.
It doesn't apply to SSO auth, where the SSO authority is assumed to have
existing protection of its own.
I refactored the Respondus SOAP API to use Authlogic in a more standard
manner, to make this work.
Change-Id: I569823f83c5c2855526464da270426275eb857cd
Reviewed-on: https://gerrit.instructure.com/6428
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Zach Wily <zach@instructure.com>
This will allow a UI where the user doesn't need to identify
the type of package they're uploading. For now it's only used
for common cartridge/canvas cartridge packages.
refs #4153
Change-Id: I2488777316660c9af60f544884429de7355f358f
Reviewed-on: https://gerrit.instructure.com/5701
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Cody Cutrer