especially when it's flat out wrong and causes it to not work
even when run in a sharded environment, it's run per-shard anyway
Change-Id: I6f568b44a6fe636468372e708cfcdcb329d4a733
Reviewed-on: https://gerrit.instructure.com/46744
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
test plan:
* enable page views in rails 3
* using the api, retrieve the page views for a user
* the 'controller' and 'action' fields should not be null
fixes #CNVS-13769
Change-Id: I19748f0f4bb455c8932283adf2a04b2f7d445554
Reviewed-on: https://gerrit.instructure.com/36886
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
fixes CNVS-13451
test plan:
* enable page view storage in cassandra
* open Canvas
* click "Settings", in the top right
* verify that the page renders successfully
Change-Id: Ia4193f59e41f6993b61a05bbb853a8e601285d8f
Reviewed-on: https://gerrit.instructure.com/35950
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Anthus Williams <awilliams@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Braden Anderson <banderson@instructure.com>
fixes CNVS-13024
test plan:
* configure cassandra page views
* as a user from shard 1, visit a course on shard 2
* go to /users/self on shard 2, and watch the network requests for
page views. in the response, the context should be a local id, and
the user should be a global id
Change-Id: I6bbc54d9b055aff74b81441b2449c4548fd96816
Reviewed-on: https://gerrit.instructure.com/35056
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Nick Cloward <ncloward@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
fixes: CNVS-10477
Test Plan:
Make sure auditing still works.
Change-Id: I021c0772ad0cf337d452b55bf690d15ce1a61a09
Reviewed-on: https://gerrit.instructure.com/31494
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Product-Review: Nick Cloward <ncloward@instructure.com>
instead of passing an :exportable option to
ActiveRecord::Associations, simply define a constant
on the class containing exportable associations and
attributes. This is due to :exportable breaking
ActiveRecord, and we can't simply monkey-patch in
config/initializers because models are included in
migrations before the initializers are run
Change-Id: I11f1a6b4570c397d8e01010c517bc6efdac7afca
Reviewed-on: https://gerrit.instructure.com/33235
Reviewed-by: Braden Anderson <banderson@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Anthus Williams <awilliams@instructure.com>
QA-Review: Anthus Williams <awilliams@instructure.com>
An audit of the related code revealed that this column is unused
test plan: After running the drop migrations, ensure that page views are
still logged correctly and can be viewed via the UI/API/CSV
Change-Id: I9141661440513d0c21ec0fdb5f0bd80dbb7c1c55
Reviewed-on: https://gerrit.instructure.com/30743
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
We attempted to make this backwards compatible when removing the cache
queue support, but the old setting was "cache" not "queue"
Change-Id: I7f419365089a1d99dae5dec3e3f7cb18a92ae05f
Reviewed-on: https://gerrit.instructure.com/31030
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
* small fix for creating - need to set created_at
* override finder methods more directly (in rails 3, they all go
through relation, which we can't override just for PageView)
Change-Id: I33476eaf67180a45c26b70110ca1f9801dff8f44
Change-Id: I783618bc5457ea53a9d59ab36d3ccf7b7fde7b2e
Reviewed-on: https://gerrit.instructure.com/29595
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Change-Id: If87f098435b873074295586a112734a7ce6087a1
Reviewed-on: https://gerrit.instructure.com/29038
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
in rails 3, request.method returns "POST" but in
rails 2, the result is "post".
Change-Id: Ia751b4db5f516bd3e62f15215d7918465fd7c418
Reviewed-on: https://gerrit.instructure.com/28543
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Anthus Williams <awilliams@instructure.com>
Product-Review: Anthus Williams <awilliams@instructure.com>
QA-Review: Anthus Williams <awilliams@instructure.com>
in order to avoid rails 3 deprecation warnings
Change-Id: If4c6ece9496e1161718770420384003de23f0421
Reviewed-on: https://gerrit.instructure.com/28206
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: James Williams <jamesw@instructure.com>
QA-Review: James Williams <jamesw@instructure.com>
The idea was to have a generic solution so
if we add fields to be tracked and exported
or displayed to users, we don't forget to force
the encoding and thus end up with an obscure
bug we only see in production.
Fixes CNVS-9528
Test plan
- in production environment
- Go to a users account details page where they
have page views.
- Download the CSV file from the top right corner
of the page views listing.
- Open the CSV file and check the "Remote IP"
column to the far right.
Change-Id: I83250f218936c8af0b3a7ae2ff69e777034e731c
Reviewed-on: https://gerrit.instructure.com/26798
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Anthus Williams <awilliams@instructure.com>
fixes: CNVS-348
This adds grade change auditing functionality. It uses cassandra and is built
after the authenication auditing. A new API endpoint was created to query these
logs. Permissions for this is a domain account. The logs can be quiried by
assignment, course, student, and grader.
Refer to: https://gollum.instructure.com/grade-auditor-implementation
Test Case:
Setup:
- Create a course with a student and an assignment.
1. Grade an assignment for a student.
2. The auditor should create a record in cassandra.
3. Query the endpoint should return the audit event.
- Query by Assignment
- Query by Course
- Query by Student
- Query by Grader
4. Ensure permissions are valid by querying with a non root account admin.
5. Permissions should be changable by site_admin under account permissions.
Change-Id: I0a1cf867d5d1b5bfbdeacc7eac81747f8732025a
Reviewed-on: https://gerrit.instructure.com/25961
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Product-Review: Nick Cloward <ncloward@instructure.com>
This wasn't working because we weren't loading the existing page view
from cassandra, so the user_id nil check was failing.
Also add the Setting to this codepath in case we need to turn off page
view updates for performance reasons.
fixes CNVS-2129
test plan:
* visit a course interaction page such as viewing a discussion topic
* turn on your web inspector
* spend some time on the page, scrolling around, clicking, doing things
that register as activity
* eventually you'll see an ajax request to /page_views to update the
interaction_seconds timer. this can take more than 5 minutes though --
the faster way to trigger it is to "interact" with the page for at least
30 seconds, and then navigate to another canvas page. that'll trigger
the page view update ajax call when the next page loads.
* visit /users/self and look at your page view history, the page view
should have the updated interaction seconds value. same with the csv
download of page view history.
Change-Id: I35a67450c0f3360720ff0c18fe9730ead607d5e0
Reviewed-on: https://gerrit.instructure.com/25871
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
This has been phased out, so is just needless complexity.
refs CNVS-2129
test plan:
* page views should still be stored and queried via cassandra or postgres
Change-Id: I0e5580086c912396b7e4cdfb139eb3a1acc4f67c
Reviewed-on: https://gerrit.instructure.com/25859
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
now that we have SIGHUP, we were changing everything to it anyway,
so just let caching in-proc be the default
Change-Id: Id1b44722522ac9693b17695da7107c99a359d5ac
Reviewed-on: https://gerrit.instructure.com/25020
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
refs CNVS-8392
also update the page view API documentation
test plan:
* with this commit applied, page views will now log the remote
ip of requests, and return that ip in the api and csv downloads
* regenerate api docs with rake doc:api, and verify that the user
section now has expanded information on the api response
Change-Id: If463e211515881db6ce1f75c9bbca8c7fbcf7589
Reviewed-on: https://gerrit.instructure.com/24880
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Jeremy Putnam <jeremyp@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
Sometimes these pieces of data coming from the request are marked as
binary (ascii-8bit) encoded, which causes the cassandra-cql gem to
hex-encode them before sending them to cassandra.
We never noticed before because the round-trip JSON encode/decode
through redis was forcing utf-8 encoding.
refs CNVS-7551
test-plan:
Without this commit, page views would show the user agent as unrecognzed
in the /users/X page views UI. However, depending on your test
environment this might not happen, it depends on the encoding of your
web/rack server. With this commit, it should be fixed regardless.
Change-Id: I85d6dc9eabe0052e9cc5e6b9e690b318d1a4d72b
Reviewed-on: https://gerrit.instructure.com/23381
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
fixes CNVS-7551
unless specifically told otherwise, write cassandra page views straight
to cassandra without putting them through the redis queue first. can
still cause them to go through redis queue by setting
'page_view_cassandra_uses_redis' to 'true'.
test-plan:
- have cassandra page views enabled and set up
- turn off delayed jobs
- navigate to some pages
- generated page views still show up
- set 'page_view_cassandra_uses_redis' setting to 'true'
- navigate to some more pages
- generated page views don't show up (yet)
- restart delayed jobs
- generated page views show up
Change-Id: I66a2991d41e18e1440e5eb68db6aa3d217577f41
Reviewed-on: https://gerrit.instructure.com/23325
Reviewed-by: Brian Palmer <brianp@instructure.com>
QA-Review: Clare Strong <clare@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
Tested-by: Brian Palmer <brianp@instructure.com>
This is no longer needed, so it's just unnecessary complexity.
closes CNVS-7232
test plan:
This is just removing functionality, there's no new functionality to
test. Verify that cassandra page views are still generated and displayed
as expected with Setting.set('enable_page_views', 'cassandra').
Change-Id: I8eaeb75dcfa94d57ea876d67fc39bdc0ae16c3cb
Reviewed-on: https://gerrit.instructure.com/22798
QA-Review: Jeremy Putnam <jeremyp@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
Filters on both generation of new page views, and reads of existing page
views already in the data store.
refs CNVS-7202
test plan:
* make an api request that gets logged as a page view such as
/api/v1/courses/X/discussion_topics?access_token=XXX
* verify that when you view your page views, the access token is
filtered out.
Change-Id: I517523c25f7518a1e8bb9745765d412db2f16eb6
Reviewed-on: https://gerrit.instructure.com/22724
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Jeremy Putnam <jeremyp@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
when querying page views in the API, allow specifying start_time and
end_time. these are passed to the underlying event stream.
fixes CNVS-6995
test-plan:
- create some page views (logging a user in and out)
- request the page views through the API
- choose a page view in the middle and note its timestamp
- pass that timestamp as the start_time in the query string (ISO8601
formatted); only the newer portion of page views should be returned,
including the selected page view.
- pass that timestamp as the end_time in the query string (ISO8601
formatted); only the older portion of page views should be returned,
including the selected page view.
- try combining start_time and end_time where
* start_time < end_time (subset of page views returned)
* start_time = end_time (only the page view with matching timestamp
returned)
* start_time > end_time (no page views returned)
Change-Id: Id419d3b7c170b5997b0b5513f9d50d220feaeae4
Reviewed-on: https://gerrit.instructure.com/22487
Reviewed-by: Brian Palmer <brianp@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Jeremy Putnam <jeremyp@instructure.com>
Product-Review: Jacob Fugal <jacob@instructure.com>
if recording a page view fails, we want to handle and stop the failure
so it doesn't fail out the whole batch. but we don't want to do so
silently; log the error instead. (in the case of cassandra page views,
this failure will actually be a failure to log the failure in the
event_stream_failures table.)
fixes CNVS-6199
test-plan:
- turn off cassandra for page views but leave redis queueing on
- queue a page view (via script/console) and note its request id
- queue a second page view with the same request id
- queue a third page view with a different request id
- process the queue (PageView.process_cache_queue)
- the first and third page views should be inserted into the database
- there should be crit-level lines in the log about the failure of the
second page view
Change-Id: Ie1849a73d8f625ab8f90c8c15ddff5f9150a0de4
Reviewed-on: https://gerrit.instructure.com/21263
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Jacob Fugal <jacob@instructure.com>
refs CNVS-6018
test plan:
* saving and loading of cassandra page views should work
Change-Id: Ic9730371bb1f4207cb06aa6da30baee9f4232d8f
Reviewed-on: https://gerrit.instructure.com/20979
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
test plan:
* in an environment that has the default shard not the birth
shard. and cassandra for page views.
* visit some stuff. then confirm you have page views for your user
Change-Id: I2a6a609a563ec0e2ce1845b82de4ed71f32b5c0e
Reviewed-on: https://gerrit.instructure.com/20879
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
fixes CNVS-5572
test-plan:
- have page views enabled and processing through a redis queue
- generate a new page view from a user with non-UTC time zone
- process redis queue to persist page view
- reload page view and check created_at; should equal the time the
page view was created
Change-Id: I8fa4da8f81d7f18815ef9223decea433e9a71172
Reviewed-on: https://gerrit.instructure.com/20171
Reviewed-by: Brian Palmer <brianp@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Clare Strong <clare@instructure.com>
Product-Review: Jacob Fugal <jacob@instructure.com>
fixes CNVS-4621
test plan:
- on the default shard, use db page views
Shard.default.tap { |s| s.settings[:page_view_method] = :db ; s.save! }
- on another shard, use cassandra page views
Shard.default.tap { |s| s.settings[:page_view_method] = :cassandra ; s.save! }
generate some page views on this second shard, and verify they appear
properly in the UI page view history after the periodic job runs.
Change-Id: I07e36bf95ec5c41f68faac6bdb70618493dfca53
Reviewed-on: https://gerrit.instructure.com/18579
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
fixes CNVS-4455
you can never have too much context
also, apparently this is the first integration spec that uses the
API from a session (not an access token), so fix API forgery
protection to respect the allow_forgery_protection option
(what's set for specs to not have to worry about forgery
protection), and clean up enabling of it in specs to use
stubbing
test plan:
* do an action that counts as participating, but wasn't a GET
(i.e. comment on a discussion)
* you should see a page view for the user in that course
Change-Id: I8714de45575123d6877e0265623e0fcaf9e7fa58
Reviewed-on: https://gerrit.instructure.com/18504
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
test plan: with cassandra page views activated on all shards, as a user
on the default shard, visit pages on another shard. these page views
should show up in your history.
Change-Id: I7abaa6eb1798760d422ca5faba2e4609ae467bb6
Reviewed-on: https://gerrit.instructure.com/18472
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
This is pointless extra load and leaves idle transactions visible in db
monitoring.
closes CNVS-4132
test plan: with cassandra page views enabled, monitor the delayed job
that inserts into page views. verify it doesn't send any begin/commit
commands to the db.
Change-Id: I6efbcd7a68811b2d5bddfba3d84382d6f03fce31
Reviewed-on: https://gerrit.instructure.com/18001
Reviewed-by: Zach Wily <zach@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
Add a batch method that uses cassandra's BEGIN BATCH/APPLY BATCH
functionality to cut down on the number of network round trips, and
possibly save a bit of CPU time on the cassandra servers.
Since execute returns a result set, it can't be batched, so we add another
method update for INSERT/UPDATE CQL statements that don't return a
value.
Closes CNVS-3526
test plan: No behavioral change. Do a regression test that page view
information is still send to cassandra properly. You can check the debug
logs to see that the CQL statements are all sent in one big BEGIN BATCH
statement.
Change-Id: Ibca4f6fbd84f2644436599c017f1ec8c39783e36
Reviewed-on: https://gerrit.instructure.com/17156
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
introduces a new BookmarkedCollection module with behavior similar to
PaginatedCollection in the simple case.
the primary advantage is that assigning to current_page (e.g. from the
:page parameter to paginate) expects a bookmark token value and
automatically deserializes into current_bookmark. the library client can
then use current_bookmark to skip forward in the collection, rather than
using (current_page - 1) * per_page as the number of items to skip. the
client then calls set_next_bookmark on the pager if there's more
results, and it automatically derives the bookmark for the next page and
serializes it into next_page, for use by Api.paginate, etc.
in addition to the PaginatedCollection.build analog, you can simply wrap
an existing scope to change it from something that will paginate by page
number into something that will paginate by bookmark.
finally, the key reason to use bookmarked pagination is to enable
composition of collections. you can merge multiple collections into one
collection which when paginated will pull results from each
subcollection, in order, to produce the page of results. you can also
concatenate multiple collections into one collection which when
paginated will exhaust the collections in order with seamless transition
from one to the next when a page spans both.
with collection merging available, you can paginate an association where
you'd like to use with_each_shard. one collection is created per shard,
and then they are merged together. this process is automated for you in
the BookmarkedCollection.with_each_shard method.
fixes CNVS-1169
Change-Id: Ib998eee53c33604cb6f7e338153428a157928a6d
Reviewed-on: https://gerrit.instructure.com/16039
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
These page views won't ever be queried anyway, so storing them is
pointless.
Change-Id: I8242c50be32f626d64caa528e7a5fd3a921af461
Reviewed-on: https://gerrit.instructure.com/16572
Reviewed-by: Brian Palmer <brianp@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
This avoids a sometimes-nasty query plan by removing request_id from the
order by clause, at the expense of selecting some duplicate rows and
having to query cassandra about them.
Also remove references to last_request_id since it's no longer used.
test plan: the page view cassandra migrator should still run, and not
process duplicate page view entries (as before).
refs #CNVS-868
Change-Id: I5933c695cc5f4c519f89362af315595221a70fdc
Reviewed-on: https://gerrit.instructure.com/15879
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
Store the page view method on the individual object at creation, so we
get the right method still when saving.
fixes #CNVS-868
test plan: with the global Setting set to cache page views, but a
specific shard set to cassandra page views, make sure that page views
created on both that shard and another shard go to the right place.
Change-Id: Ib236c8cd4401a919978423f777c5e95ce4f1ee85
Reviewed-on: https://gerrit.instructure.com/15825
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Tested-by: Clare Hetherington <clare@instructure.com>
refs #11772
The previous strategy didn't perform as hoped, so we've switched to the
(account_id, created_at) index. This required storing a separate
progress state per account_id so we can migrate one account at a time.
At the same time I decided to switch from newest-first going backwards
to oldest-first going forwards. This will make it easier to do the
entire migration while still on db page views, then switch to cassandra
page views, then do a 2nd migration just to move over the small # of
page views created in the interim.
test plan: generate some db page views in multiple accounts. then switch
configuration to cassandra page views. run rake
db:migrate_pageviews_to_cassandra and the existing db page views should
now be in cassandra, including showing up in the user's page view
history.
Change-Id: Ic748376032a4305a0537f398670fb37ee17b071c
Reviewed-on: https://gerrit.instructure.com/15489
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Cody Cutrer