Ports the generation of stats for that question type to the CQS gem.
Changes:
- no longer exposing "user_ids"
- can now identify students who skipped the question
Closes CNVS-13089
TEST PLAN
---- ----
- create a quiz with multiple-answer question(s)
- take it by a number of students and cover the following cases:
- answer correctly by picking only the right choices
- answer almost correctly by:
1. picking only 1 right choice
2. picking 1 right and 1 wrong choices
3. picking everything
- answer incorrectly by picking only the incorrect choice(s)
- don't answer at all
- get the stats from the API:
- for "responses", "correct", and "partially_correct", verify they
meet the specification in the docs
- also for the "responses" field in each document in "answers"
- verify that there is an answer document with "none" for an id with
"responses" that reflect the number of students that skipped the
question
- visit ember quiz stats:
- verify the "Attempts: X out of Y" should read the "responses"
field out of the total quiz participant count
- verify the donut chart reads the correct "correct" response ratio
- verify there is a "No-Answer" bar
- expand the question details:
- verify that all the choices are displayed, and the correct
choices are highlighted in GREEEN
Change-Id: Ibc08b6f521f9cae35dd16950c68c164d7e27d95d
Reviewed-on: https://gerrit.instructure.com/35736
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Derek DeVries <ddevries@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
Closes CNVS-13386
TEST PLAN
---- ----
- similar to multiple-choice in https://gerrit.instructure.com/35734
but with this question type with a few minor differences:
- "correct" metric, see docs
- "other"/"unknown" answer should be generated only if any student
writes an answer that is not correct
Change-Id: I2d56afc0d9c1a3620d4de96059ebdadc56baecf9
Reviewed-on: https://gerrit.instructure.com/35735
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Derek DeVries <ddevries@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
Changes:
- no longer exposing user_ids
- no longer exposing response_values
- new metric: "responses" at the root-level
Closes CNVS-13385
TEST PLAN
---- ----
- create a quiz with those question types
- take the quiz by a number of students, choose different answers
- get the stats from the API:
[GET] /api/v1/courses/:course_id/quizzes/:quiz_id/statistics
- verify you get the documented metrics and they have the proper
values
- verify the relevant sections in ember quiz stats is still functional
Change-Id: I470912060042405629812585ef6667693b1302a1
Reviewed-on: https://gerrit.instructure.com/35734
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Derek DeVries <ddevries@instructure.com>
Product-Review: Derek DeVries <ddevries@instructure.com>
Adds numerical question support to CanvasQuizStatistics and the renderer
in the Ember quiz statistics (subclass of ShortAnswer question).
Closes CNVS-13087, CNVS-13086
TEST PLAN
---- ----
- create a quiz with a numerical question:
- mix and match between the answer types: exact, exact with margin,
and range answers
- take the quiz by a number of students, we need to score like this to
cover the possible cases:
- a correct answer
- a missing answer
- an incorrect/out-of-range answer
- hitting the stats api should return the documented metrics in the
output of "question_statistics":
/api/v1/courses/:course_id/quizzes/:quiz_id/statistics [GET]
- visiting ember quizzes should now render the numerical stats which
is similar to short answer (fill in the blank) and multiple answers:
- make sure the "Other" and "No Answer" answers and bars show up
- verify the API docs are updated
Change-Id: I0c9ec0dbc7c729410241177b6ed43b0cd4dad143
Reviewed-on: https://gerrit.instructure.com/35431
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Derek DeVries <ddevries@instructure.com>
Product-Review: Derek DeVries <ddevries@instructure.com>
Adds support for Matching question statistics. Similar to FIMB and
Mulitple Dropdowns in the output.
Closes CNVS-13088
TEST PLAN
---- ----
Similar to https://gerrit.instructure.com/#/c/35101/ but with Matching
questions instead of FIMB.
Change-Id: Iaa5da1631503a8750b9673d7cb6064a296a8aa55
Reviewed-on: https://gerrit.instructure.com/35331
Reviewed-by: Derek DeVries <ddevries@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
Add support for generating stats for File Upload and Formula questions.
Similar to Essay metrics but adjusted to calculate properly.
Closes CNVS-13169
TEST PLAN
---- ----
- create a quiz with those question types
- take the quiz:
- answer both question types by at least one student, but leave it
unanswered by another so we can test the "responses" metric
- test the stats:
GET /api/v1/courses/:course_id/quizzes/:quiz_id/statistics
- the "responses" metric should count the number of students who
provided an answer
- the "graded" metric should read the number of students whose
answers you've graded so far
- the "full_credit" one should read 0 until you grade it and give
them a score higher than, or equal to, the maximum points possible
- "point_distribution" is similar to that of Essays; it is an array
of objects that track every score you gave the students and the
number of students who received those scores
- grade the scores and re-test to verify the metrics update correctly
- verify that the API documentation is updated to include those
question types
PS: the "responses" field will read "the number of students who uploaded
a file" for File Upload, and "the number of students who wrote any
answer" for Formula.
Change-Id: I890eafe018e000eef88de782bd7e86b5259df5d5
Reviewed-on: https://gerrit.instructure.com/35112
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jason Madsen <jmadsen@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
Closes CNVS-13161
TEST PLAN
---- ----
- create a quiz with multiple-dropdown questions
- take the quiz by a number of students and type in different answers
- make an API request to stats:
GET /api/v1/courses/:course_id/quizzes/:quiz_id/statistics
- verify you get the documented metrics and they have the proper
values
- visit the ember quizzes page and verify that:
- the charts are still working
Change-Id: I37ef48f22c3c16170034b6dd887b40ff20da4af5
Reviewed-on: https://gerrit.instructure.com/35104
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Derek DeVries <ddevries@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
Support for FIMB question statistics in the CanvasQuizStatistics gem.
Closes CNVS-13160
TEST PLAN
---- ----
- create a quiz with FIMB questions
- take the quiz by a number of students and type in different answers
- make an API request to stats:
GET /api/v1/courses/:course_id/quizzes/:quiz_id/statistics
- verify you get the documented metrics and they have the proper
values
- visit the ember quizzes page and verify that:
- the charts are still working
Change-Id: I2d82c708614c41e222aa24d65de64555d8056a9a
Reviewed-on: https://gerrit.instructure.com/35101
Reviewed-by: Jason Madsen <jmadsen@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
Refactoring the generation of quiz question statistics into its own gem.
This patch adds support for Essay question statistics.
Closes CNVS-12725
TEST PLAN
---- ----
- create a quiz with an essay question
- perform an API request to retrieve the quiz statistics
- ensure that the following metrics are generated and correct:
- "graded": number of students whose answers have been graded by the
teacher
- "full_credit": number of students who received a full score
- "point_distribution": a list of scores and the number of students
who received them (so if 2 students got graded for 3 points, it
should have a key of 2 and a value of 3), un-graded submissions
should be keyed under null
- "responses": number of students who answered the question
(wrote anything)
- "user_ids": IDs of those students
- documentation for QuizStatistics -> Essay should be updated with the
new stats
> Other things to test
- verify that the old statistics page still renders:
/courses/:course_id/quizzes/:quiz_id/statistics
- verify that you can still generate both student and item analysis
CSV reports
API endpoint for quiz stats:
/api/v1/courses/:courseid/quizzes/:quiz_id/statistics [GET]
Change-Id: Ib15434ff4cef89ac211c1f4602d1ee609ef48ec4
Reviewed-on: https://gerrit.instructure.com/33990
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Derek DeVries <ddevries@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
Closes CNVS-8989
CHANGES
-------
- ItemAnalysis can now be generated to a Hash as well as CSV
TEST PLAN
---- ----
- create a quiz with a number of questions (true/false and
multiple-choice a must-have)
- take the quiz by 2-3 students
- perform an API request as described below and verify that you get a
statistics object that matches the API docs
- do it once in legacy JSON format, e.g:
"Accept: application/json"
- do it once in JSON-API format, e.g:
"Accept: application/vnd.api+json"
- read the API docs and report any mistakes or stuff that's not clear,
the stuff is written in the QuizStatistics module
Endpoint: [GET] /api/v1/courses/:course_id/quizzes/:quiz_id/statistics
Change-Id: I1588e1cdc8955b890c739df4c730507dad7118d1
Reviewed-on: https://gerrit.instructure.com/31141
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Derek DeVries <ddevries@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
This patch provides support for answering Quiz Questions via the API.
closes CNVS-9844, CNVS-10225
TEST PLAN
---- ----
Testing this will be a bit rough because there are many variations and
validations to cover. I'll spare the validations that are covered by
specs from the test plan.
Create a quiz with a question of *each* type except "Text" and "File
Upload". There's a script that creates a quiz with its questions
automatically for you if you don't want to keep doing this manually. See
references.
> Answering Questions
Now you need to answer each question via the API. Most of them vary in
formats, but they are fully specified in the API documentation page
(along with examples). See DOCUMENTATION for more info.
> Flagging Questions
Flagging, and unflagging, a question is the same regardless of its type,
see the "EXAMPLE REQUESTS" section.
> Access Validations
Here are some generic, non-question based validations to verify. You
should NOT be able to answer a question if:
- the quiz submission has been turned in
- the quiz submission is overdue
- the Access Code for the quiz is invalid
- the IP filter of the Quiz prohibits you from taking the quiz
- the quiz submission :validation_token is incorrectly specified (ie,
other students shouldn't be able to answer your questions)
- you don't specify the latest :attempt, so if the Quiz has multiple
attempts, and this is your 2nd take, you specify an :attempt of 1,
3, or anything but 2 should fail
- NEW: turn quiz into an OQAAT quiz with the "Can't go back" flag on;
the API should not reject all requests to modify any of the
questions with a 501 error saying that type of quizzes is not
supported yet (support will come in CNVS-10224)
> Grading
Also, when you're done answering the questions, take a look at the
grades and make sure everything gets graded just like it does when using
the UI directly.
> Verifying results in the browser
While taking a quiz in the canvas UI, the scripts perform backups in the
background that would overwrite any changes you do via the API. If you
want to verify the changes you make via the API from the UI, you must
append "?backup=false" to the take quiz page URL, something like this:
http://localhost:3000/courses/1/quizzes/1/take?backup=false
Setting that flag will (for now) disable the backup behaviour and should
make things tick.
EXAMPLE REQUESTS
------- --------
Don't forget to set the 'Content-Type' header to 'application/json'!
> Answering a Multiple-Choice question
[PUT] /api/v1/quiz_submissions/:quiz_submission_id/questions/:id
{
"attempt": 1,
"validation_token": "1babd0...",
"answer": 10
}
> Flagging a question
[PUT] /api/v1/quiz_submissions/:quiz_submission_id/questions/:id/flag
{
"attempt": 1,
"validation_token": "1babd0..."
}
> Unflagging a question
[PUT] /api/v1/quiz_submissions/:quiz_submission_id/questions/:id/unflag
{
"attempt": 1,
"validation_token": "1babd0..."
}
DOCUMENTATION
-------------
Run `bundle exec rake doc:api` and check out the Quiz Submission
Questions page. There's an Appendix that contains example requests for
each question type, as well as the errors produced by each handler.
LINKS
-----
- bootstrap script:
https://gist.github.com/amireh/e7e8f835ffbf1d053e4c
- direct link to the API documentation page:
http://canvas.docs.kodoware.com/quiz_submission_questions.html
Change-Id: I9a958323ece8854bc21a24c2affd8dc3972e46d5
Reviewed-on: https://gerrit.instructure.com/27206
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Derek DeVries <ddevries@instructure.com>
QA-Review: Myller de Araujo <myller@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
This patch ports the ability for teachers to manually grade questions
and add comments on student answers in a quiz submission.
fixes CNVS-8981
TEST PLAN
---- ----
Create a quiz with like two questions, take it as a student and turn it
in.
Endpoint: [PUT]
/api/v1/courses/:course_id/quizzes/:quiz_id/submissions/:id
1. Modify fudge points (Appendix A) with values like -2.5, 2.5 and:
a. verify that the score gets adjusted by the fudge amount
b. verify that the returned "fudge_points" in the output is adjusted
2. Modify a question score (Appendix B) and:
a. verify that the total score is adjusted by the correct amount
b. verify that no other question scores were affected
c. verify that no comments were affected
3. Modify a question comment (Appendix C) and:
a. verify that the total score is not affected
b. verify that no comments were affected
Tip: to verify stuff like single question scores and comments, just go
to the quiz history page in the web UI and keep refreshing after
sending API requests.
Tip: to get a list of all the question ids, perform a GET request to:
/api/v1/courses/:course_id/quizzes/:quiz_id/questions
Validations that should hold:
1. [403] if you're not a teacher
2. [400] if you specify an invalid attempt
3. [400] if you specify an attempt that is in progress (not turned in)
4. [400] if you pass in a funny score, like an array ([ 'hehe' ]) or a
Hash ({ hello: 'World' })
EXAMPLE REQUESTS
------- --------
Appendix A: Modifying fudge points
{
"quiz_submissions": [{
"attempt": 1,
"fudge_points": VALUE
}]
}
Appendix B: Modifying a question score
{
"quiz_submissions": [{
"attempt": 1,
"questions": {
"QUESTION_ID": {
"score": VALUE
}
}
}]
}
Appendix C: Modifying a question comment
{
"quiz_submissions": [{
"attempt": 1,
"questions": {
"QUESTION_ID": {
"comment": "HEHEHEHH"
}
}
}]
}
Change-Id: Id290d4e9f9cb9abcaa00eae857f4b0b7bd06e2ff
Reviewed-on: https://gerrit.instructure.com/28268
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Myller de Araujo <myller@instructure.com>
Reviewed-by: Jason Madsen <jmadsen@instructure.com>
Product-Review: Ahmad Amireh <ahmad@instructure.com>
The documentation regarding creating an assignment override with
`group_id` is made more clear and the prerequisites are better stated.
The group assignments feature and the dependency between
`group_category_id` and `group_id` in the stages of assignment creation
and assignment override creation are explained by a diagram sitting in an
appendix entry at the bottom of the Assignments API documentation page,
Appendix: Group assignments.
The diagram image in the appendix is exported out of the raw .xmind file
in doc/diagrams. You can edit that diagram using the XMind program
(http://www.xmind.net/), it's free to use and cross-platform (JAVA).
---
Testing:
1. generate the API docs: `bundle exec rake doc:api`
2. navigate to the Assignments API page
(link: public/doc/api/assignments.html)
3. take a look at the "Create an assignment override"
section
4. also take a look at the bottom of the page
fixes #CNVS-1204
Change-Id: I07660a7e8a58aba2e307849a79557067a7ea77fb
Reviewed-on: https://gerrit.instructure.com/17454
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Amber Taniuchi <amber@instructure.com>
Useful things the commit brings:
1. Source documentation can now include images and out-of-source examples
2. Source documentation can now be supplemented by "appendixes" for
documenting advanced or uncommon usage, auxiliary examples, or any
supplementary content
3. An implementation of the YARD @see tag that utilizes the canvas
YARD linkify helper
Necessary changes for integration were:
* Gemfile now includes 'yard-appendix'
* Rake task for generating API docs (doc:api) made more readable and
now supports asset migration (images and examples)
* Canvas YARD 'api' template now handles :appendix sections provided
by the plugin
* Canvas YARD 'linkify' helper modified:
* uses a shared linker to look up a topic and controller
* overrides default handling of 'Appendix: ' links
* defaults to using the @object title as the link body when no title
was explicitly passed instead of the path.to.object
* Canvas YARD 'fulldoc' handler respects a
DOC_OPTIONS[:all_resource_appendixes] that when turned on would
generate appendix entries in the All Resources section[1]
[1] I've already implemented this functionality because I misread the
requirement (as seen in PB 6) so I thought we could keep it around and
toggle it if need be. The options are inside lib/tasks/docs.rake
---
Testing:
To verify that the changes do not alter or affect the current API docs,
fire up a terminal and do the following (inline comments for directions):
```bash
cd /path/to/canvas;
# generate the original docs before pulling these changes
bundle exec rake doc:api
mv public/doc public/doc_original
# checkout these changes into a branch... after that:
bundle install
bundle exec rake doc:api
diff -r -y -q public/doc_original/api public/doc/api
```
The output of the last command should look like this:
Only in doc/api: examples
Only in doc/api: images
To test the actual @!appendix functionality:
* see https://github.com/amireh/yard-appendix for directions on how to
define Appendix entries
* write an Appendix in any controller, optionally reference it in some
method (using @see or {link})
* Appendix entry should be shown at the bottom of the controller's doc
page
* reference to the appendix entry should take you to it
Alternatively, you can check-out the gerrit change 17454 at
https://gerrit.instructure.com/#/c/17454/ which utilizes this
functionality.
Change-Id: Id667b77ff8d36b0f503e0f6752045e3d05bc3649
Reviewed-on: https://gerrit.instructure.com/17453
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
QA-Review: Simon Williams <simon@instructure.com>
Ahmad Amireh