because of nokogiri, we rely on multi-platform gem caching, which
only works right in bundler 2.2
Change-Id: Id207278946e849abec3418807c71e4650506a0cf
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/259142
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
closes FOO-1414
flag=none
TEST PLAN:
1) run a huge fleet of servers
2) none of them should accidentally share connections between the
passenger processes and the live events emitter
Change-Id: Ib121e4c5df2dc4c25867c9636cd035121304ee2c
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/256072
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
This has been done, but the inline comment
was never removed
flag=none
Test Plan
Verify only the comment to consider using
Kinesis put records was removed
Change-Id: I3e40382592758c2d2d9094e120c8d27c3c329371
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/254018
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Xander Moffatt <xmoffatt@instructure.com>
QA-Review: Xander Moffatt <xmoffatt@instructure.com>
Product-Review: Weston Dransfield <wdransfield@instructure.com>
Change-Id: Ib49bc8939cf1706e758429e531a87c57d0231a37
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/251156
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
Clarify that LiveEvents.statsd.time should only
be called once. Also, do not yield to it -- we
don't care about its return value for what we are
testing here.
Change-Id: I577254120d5126dd0300808b4fde7d2a9d10dbce
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/233985
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Tucker Mcknight <tmcknight@instructure.com>
PLAT-5353 is the ticket to fix this flakey spec.
flag = none
Change-Id: I19b25a427151a809492be49602373cbf9abd57fd
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/224845
Reviewed-by: Jacob Powell <spowell@instructure.com>
Reviewed-by: Rex Fleischer <rfleischer@instructure.com>
Tested-by: James Butters <jbutters@instructure.com>
QA-Review: James Butters <jbutters@instructure.com>
Product-Review: James Butters <jbutters@instructure.com>
Test Plan:
- tests pass, also make sure that events are
sent still
Change-Id: I61eb7fecd35c538de8c41eb7fa0ebc91c8cc220c
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/220986
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Marc Phillips <mphillips@instructure.com>
Product-Review: Marc Phillips <mphillips@instructure.com>
Reviewed-by: Xander Moffatt <xmoffatt@instructure.com>
Change-Id: Iee73340d67f1181dfd2e051fe27912174d30dc81
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/216372
Tested-by: Jenkins
Reviewed-by: Rex Fleischer <rfleischer@instructure.com>
Reviewed-by: S. Jacob Powell <spowell@instructure.com>
QA-Review: James Butters <jbutters@instructure.com>
Product-Review: James Butters <jbutters@instructure.com>
closes PLAT-5097
Test Plan:
- verify that normal live events get sent correctly
- once in production, we can look for the error log
to determine what request is causing the context
to not be set
Change-Id: I011c61b4f82546d4e6f2fdffb47074dcf88237ab
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/216231
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Tested-by: Jenkins
Product-Review: Clint Furse <cfurse@instructure.com>
fixes: COREFE-374
this is because we had a report of getting the wrong http hostname
on beta with this applied. I'll revert this for now and try to fix
that and submit this again later
test plan:
* live events should work the same as it always has and not
try to set up the context in a lambda
* live events events should be logged with the right hostname
This reverts commit 3c14453e30.
the jira of the reverted change is: COREFE-342
Change-Id: Ibcd94261980a04445c5d87dec98f50367a070c3c
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/214092
Tested-by: Jenkins
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Ryan Shaw <ryan@instructure.com>
This should ensure we capture live events on exit correctly.
Test Plan:
- clear your dev logs on your canvas
- start canvas, make some live events calls
- stop canvas, grep your dev logs for live events output
- should not see anything that is an error
Change-Id: I5faf89eb1b3d288287978680195a3fe3c0c35eb5
Reviewed-on: https://gerrit.instructure.com/212483
Tested-by: Jenkins
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Marc Phillips <mphillips@instructure.com>
Canvas::Errors was causing errors being used in
LiveEvents error reporting. Remove from LiveEvents
error reporting.
refs PLAT-5026
Test Plan:
- run canvas and have the stubbed client return an error
- note that things don't break
- for best testing, hook up canvas to your local kinesis
- see that things don't break
- try to set kinesis local to always fail and see that
the failures are caught and things don't break
Change-Id: Ie2a1709708bf0fd4f9638985aa7a66690bf0b73c
Reviewed-on: https://gerrit.instructure.com/212378
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Tested-by: Jenkins
Product-Review: Marc Phillips <mphillips@instructure.com>
This prevents us from having to run this block of code unless we
actually are going to post a live event
Closes: COREFE-342
Test plan:
* for rails requests that never post to live events, search the logs
For the string: “setup_live_events_context”
* it should never appear
* for requests that _do_ send a live events event, search the rails log
for “setup_live_events_context”
* it SHOULD appear
* you should see a few db/redis requests right after that for the live
Events stuff
* everything else with live events should work as it did before
Change-Id: I0edadd3230b93160d19eb7cd1b501df6c8c9c65c
Reviewed-on: https://gerrit.instructure.com/212039
Tested-by: Jenkins
Reviewed-by: Marc Phillips <mphillips@instructure.com>
QA-Review: Ryan Shaw <ryan@instructure.com>
Product-Review: Ryan Shaw <ryan@instructure.com>
Better tag metrics in datadog to allow aggregations
across all event_types. Also allow for request_id
to be sent to datadog when a failure is made so we
can easily track down the problem request in logs.
closes PLAT-5026
Test Plan:
- force a live event to fail putting to kinesis (you can
do this by updating the stubbed client to add error code)
- note that the enrichment is now present
- see that the dropped event is also pushed to sentry and
the logs
Change-Id: I6c82081c69ced40b32294bdd0e7f21849b19494e
Reviewed-on: https://gerrit.instructure.com/212192
Tested-by: Jenkins
QA-Review: Tucker Mcknight <tmcknight@instructure.com>
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Marc Phillips <mphillips@instructure.com>
refs QUIZ-6827
Test Plan:
Running the pact tests that fail will not fail if all are run
Change-Id: I0779dfbb6b0547713eaa1a6d7595a7e7c9db07ff
Reviewed-on: https://gerrit.instructure.com/208095
Tested-by: Jenkins
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Marc Phillips <mphillips@instructure.com>
Sometimes, the fclient object is nil, which causes the
expect_put_records method to throw an error.
fixes PLAT-4795
Change-Id: I6ed981c800e1c0793f69d1c2636ef49892edd7ec
Reviewed-on: https://gerrit.instructure.com/207290
Tested-by: Jenkins
Reviewed-by: James Butters <jbutters@instructure.com>
QA-Review: Tucker Mcknight <tmcknight@instructure.com>
Product-Review: Tucker Mcknight <tmcknight@instructure.com>
fixes PLAT-4773
Test Plan:
- Verify the new "compact_live_event_payloads"
release flag is visible in the root account and
defaults to "off"
- Enable the feature flag
- Verify null values are removed from live event
payloads
Change-Id: I8f91ead101fa3cd81fe8553f7f8ff6e94b294128
Reviewed-on: https://gerrit.instructure.com/206307
Reviewed-by: Xander Moffatt <xmoffatt@instructure.com>
Tested-by: Jenkins
QA-Review: Xander Moffatt <xmoffatt@instructure.com>
Product-Review: Oxana Jurosevic
Stub the aws kinsesis stream if
STUBBED_KINESIS_STREAM is set in the ENV.
This will print the event out to terminal so you
can see what events are being submitted.
refs PLAT-4639
Test Plan:
- live events doesn't break when configured
Change-Id: Ic182551e3304c9c6241091b7ae9988703093dfb2
Reviewed-on: https://gerrit.instructure.com/204138
Tested-by: Jenkins
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Product-Review: Marc Phillips <mphillips@instructure.com>
closes PLAT-4549
Test Plan:
- Live events should only be sent in batches
Change-Id: I72f8e210ae74b32b09e16a8d4fb515e0c4f699fc
Reviewed-on: https://gerrit.instructure.com/197993
Tested-by: Jenkins
Reviewed-by: Clint Furse <cfurse@instructure.com>
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
Reviewed-by: Brent Burgoyne <bburgoyne@instructure.com>
QA-Review: Tucker Mcknight <tmcknight@instructure.com>
Product-Review: Marc Phillips <mphillips@instructure.com>
closes PLAT-4550
test plan:
- tests pass
- this would be hard to manually qa, but if you want you could add a
sleep to the live event async worker loop, do some stuff that
queues events, ctrl-c the rails server, and watch the loggs for it
to process the events before finally exiting.
Change-Id: I8c82b07ff5eec75cf51a307a2571f3780e75a216
Reviewed-on: https://gerrit.instructure.com/197276
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins
closes PLAT-3761
test plan:
- configure live events plugin setting to write to a local kinesalite
stream
- make sure there aren't any errors writing to the stream
Change-Id: I88944ad3ad80ecaa18c6da3739b34b2551ba2366
Reviewed-on: https://gerrit.instructure.com/192000
Reviewed-by: Weston Dransfield <wdransfield@instructure.com>
QA-Review: Weston Dransfield <wdransfield@instructure.com>
Tested-by: Jenkins
Product-Review: Brent Burgoyne <bburgoyne@instructure.com>
fixes CORE-2538
test plan
- configure statsd to use data dog
- it should work
Change-Id: Ie8428e4e99973b35506bd7a8e4d1a18f5a7875a1
Reviewed-on: https://gerrit.instructure.com/182083
Tested-by: Jenkins
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
this allows us to not download the entire aws sdk and old versions of
gems for statsd
Change-Id: Ib8305a008c2270845fd4f7ca5311a86a2e73ce7e
Reviewed-on: https://gerrit.instructure.com/180784
Tested-by: Jenkins
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Simon Williams <simon@instructure.com>
refs QUIZ-4175
Test Plan:
- qa-cr for now. A subsequent commit will include a comprehensive
test plan once the Quiz LTI contract test is published.
Change-Id: I2ffe962d8a626994614f3cc904d131e41e3a00b1
Reviewed-on: https://gerrit.instructure.com/145857
Tested-by: Jenkins
Reviewed-by: Mark Grant <mgrant@instructure.com>
Product-Review: Michael Hargiss <mhargiss@instructure.com>
QA-Review: Michael Hargiss <mhargiss@instructure.com>
- also uses .to_json to convert hash to json string; ensures
no duplicate keys
closes PLAT-2588
Test plan:
Test alongside g/111746 (MRA)
Configure live events on local, or test against portal2.
Run canvas-lms/script/tail_kinesis [endpoint] [stream] [region]
to see a log of all live events.
For local (no creds required):
script/tail_kinesis http://kinesis:4567 live_events us-east-1
For portal2 (inseng creds):
script/tail_kinesis \
http://kinesis.us-west-2.amazonaws.com \
cyoe_live_events_qa \
us-west-2
Create a quiz in Canvas
As a student, take the quiz
Verify the ensuing grading events do not have duplicate keys
As a teacher, change the answers to a quiz question, and choose
to regarde the quiz
Verify the ensuing quiz regraded event does not have duplicate
keys
Change-Id: I0653363527ed9e39f56bcb84bcf0641e953d964c
Reviewed-on: https://gerrit.instructure.com/111745
Reviewed-by: Chris Wang <cwang@instructure.com>
Tested-by: Jenkins
QA-Review: Tucker McKnight <tmcknight@instructure.com>
Product-Review: Chris Wang <cwang@instructure.com>
Fixes: CNVS-35832
This will allow us to configure the feature via consul w/o storing
credentials in the clear. This has been made possible by our new app
servers having an instance role we can use.
Test Plan:
- Enable Kinesis and Consul for docker
- Create the kinesis stream according to doc/live_events.md
- Add some dummy AWS credentials to the web and jobs containers
- Configure live events at `/plugins/live_events`
- Check "Configure using Consul..."
- Leave the rest blank
- Tail the kinesis stream using script/tail_kinesis:
script/tail_kinesis http://kinesis.canvaslms.docker live-events us-east-1
- Create a course
- A live event should appear in the tailer window.
Change-Id: I136709697a7145491ec18fc712c6278f38f5eaef
Reviewed-on: https://gerrit.instructure.com/110200
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins
QA-Review: Tucker McKnight <tmcknight@instructure.com>
Product-Review: Tyler Pickett <tpickett@instructure.com>
Refs: CNVS-35832
This makes us a little more flexible in what we can supply without
having to resort to attempting to fake the PluginSetting API.
Change-Id: If37ab5242fb54dcfb59323c5ef95822a85f2bb80
Reviewed-on: https://gerrit.instructure.com/109956
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Tested-by: Jenkins
QA-Review: Tucker McKnight <tmcknight@instructure.com>
Product-Review: Tyler Pickett <tpickett@instructure.com>
as of this commit, all canvas gems should be on rspec 3.5, and pass
without deprecation warnings.
closes CNVS-34040
test plan: specs should pass without deprecation warnings
Change-Id: I556b1a4a5aeb791c6ddd50ee35b51c513e025019
Reviewed-on: https://gerrit.instructure.com/98414
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Product-Review: Landon Wilkins <lwilkins@instructure.com>
QA-Review: Landon Wilkins <lwilkins@instructure.com>
Tested-by: Jenkins
closes: CYOE-357
Test plan:
On a Canvas instance running in rails production mode:
1. Create an assignment with some graded submissions
2. Modify the points possible of the assignment so
that the submission scores will be updated
3. Verify that the generated live events are
- coming from a delayed job
- include the job information in the context
- include the account id, account lti_guid,
context id (course id) and context type ("Course")
Change-Id: I651b97a47f61f9d969161bc6c2360102e61eb2d3
Reviewed-on: https://gerrit.instructure.com/93663
Tested-by: Jenkins
Reviewed-by: Dan Minkevitch <dan@instructure.com>
QA-Review: Jahnavi Yetukuri <jyetukuri@instructure.com>
Product-Review: Michael Brewer-Davis <mbd@instructure.com>
Change-Id: I665840e2163184ffe6a81a7701f33db5257c42fb
Reviewed-on: https://gerrit.instructure.com/92742
Tested-by: Jenkins
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
it can coexist with v1, and now we can move our usages over to v2
one at a time
Change-Id: I6aa1ff3f46dbe3762ee2548d41ffd26e4c1b26fe
Reviewed-on: https://gerrit.instructure.com/92704
Tested-by: Jenkins
Reviewed-by: Simon Williams <simon@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
rely on test_all_gems.sh to output header and trailer,
and use `set -e` in each test.sh to simplify passing
through errors
Change-Id: I3ba724ad2539ddfe31195394c43f646acfc73920
Reviewed-on: https://gerrit.instructure.com/70469
Tested-by: Jenkins
Reviewed-by: Simon Williams <simon@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
closes CNVS-20323
canvas_statsd has been packaged and published as its own gem. remove
code from gems/ and other supporting code.
updated initializer to conform with new gem default tracking options
test plan:
- enable statsd in config/statsd.yml
- visit several pages in canvas
- create some things, updates some things
- reload a page or two
- canvas should continue to function as expected
- the following statsd request keys for controller actions should
continue to be sent:
- total
- db
- view
- sql.read
- sql.write
- sql.cache
- active_record
Change-Id: I28fbf8642a3d2719b08721a3df1c7b77ac52cb1f
Reviewed-on: https://gerrit.instructure.com/54251
Tested-by: Jenkins
Reviewed-by: Ethan Vizitei <evizitei@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Jason Madsen <jmadsen@instructure.com>
This commit adds a new module called LiveEvents that knows how to send a
certain set of events to Kinesis. The module is configured via
normal plugin settings per account. Once the plugin is configured with
a Kinesis stream, events will start getting sent to that stream.
Events are sent asynchronously, in a background thread.
test plan:
* See `doc/live_events.md` for instructions on how to setup a local
kinesis stream and configure the LiveEvents plugin.
* Start tailing the stream with the command specified in
`doc/live_events.md` in a terminal.
* Perform the actions described in `doc/api/live_events.md` and verify
that events show up in your Kinesis terminal with the correct data.
Change-Id: Id799688c972205a1eee84a673912f84b0c7abb57
Reviewed-on: https://gerrit.instructure.com/50324
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Jenkins
Reviewed-by: Jacob Fugal <jacob@instructure.com>
QA-Review: Jacob Fugal <jacob@instructure.com>
Product-Review: Zach Wily <zach@instructure.com>
Cody Cutrer