also start moving the more complicated logic
from the rlint script into tested helper
classes
Change-Id: I5c3916b0c7e695389b552dc1a6b4353aaaeba290
Reviewed-on: https://gerrit.instructure.com/52743
Tested-by: Jenkins
Reviewed-by: Simon Williams <simon@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
lets rlint only run with local changeset unless you specify "--heavy"
Also checks for whether you actually touched a line
or not before deciding an INFO comment is relevant
Change-Id: I4a960c72644dfc46aca7a51d04321711cef0850c
Reviewed-on: https://gerrit.instructure.com/51992
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Tested-by: Ethan Vizitei <evizitei@instructure.com>
on jenkins this was taking up to 9 minutes.
By removing the file I/O and batching
the georgich commits, this ran locally on my
sentry patch set in 15 seconds.
Change-Id: I23b0df9b9396829de4b8ff084f4f92df6e698da2
Reviewed-on: https://gerrit.instructure.com/51987
Reviewed-by: Jon Jensen <jon@instructure.com>
Tested-by: Jenkins
Product-Review: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
Needed some better use of environment variables
to get the right SHA. also shouldn't care about
dirty working directory on jenkins.
Change-Id: I85d4b9468f629217adcdd25039e44f26fd20b1d5
Reviewed-on: https://gerrit.instructure.com/51907
Tested-by: Jenkins
Reviewed-by: Cody Cutrer <cody@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
add migration lint as rubocop cops
create frozen constant linter
find_ids datafixup lint
send_later lint
lint send_later in predeploys
add freeze_constant cop to default config
don't forget to include rubocop when running
get the rubocop runner into the script
lint for wrong algorithm name
lint primary key
lint remove_column in predeploys
get rubocop output as parsed json
diff munging for gergich
disable a few style cops
tweak rubocop setup to allow IDE plugins to work
get gergich comment format right
shell out to gergich if we're in jenkins-land
Change-Id: I6eecc8d8ede17a755c9d9a86121c3658776de9cd
Reviewed-on: https://gerrit.instructure.com/51755
Tested-by: Jenkins
Reviewed-by: Jason Madsen <jmadsen@instructure.com>
Product-Review: Ethan Vizitei <evizitei@instructure.com>
QA-Review: Ethan Vizitei <evizitei@instructure.com>
This commit adds a new module called LiveEvents that knows how to send a
certain set of events to Kinesis. The module is configured via
normal plugin settings per account. Once the plugin is configured with
a Kinesis stream, events will start getting sent to that stream.
Events are sent asynchronously, in a background thread.
test plan:
* See `doc/live_events.md` for instructions on how to setup a local
kinesis stream and configure the LiveEvents plugin.
* Start tailing the stream with the command specified in
`doc/live_events.md` in a terminal.
* Perform the actions described in `doc/api/live_events.md` and verify
that events show up in your Kinesis terminal with the correct data.
Change-Id: Id799688c972205a1eee84a673912f84b0c7abb57
Reviewed-on: https://gerrit.instructure.com/50324
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Jenkins
Reviewed-by: Jacob Fugal <jacob@instructure.com>
QA-Review: Jacob Fugal <jacob@instructure.com>
Product-Review: Zach Wily <zach@instructure.com>
vendor/plugins
When either gems/plugins or vendor/plugins doesn't exist in the repo,
the script would print an error as it tried to list the directory
contents through globbing. This fixes that error.
Fixes CNVS-19519
Test Plan:
- Create a new checkout with no vendor/plugins directory, or delete
the existing one (not recommended)
- Run the script and ensure it does not report an error with
a missing vendor/plugins/*
- Ensure the script completes without error
Change-Id: I4d49fc090d4abc21ac8e8005f4ed83c47b4e89c5
Reviewed-on: https://gerrit.instructure.com/50961
Reviewed-by: Cameron Sutter <csutter@instructure.com>
Product-Review: Cameron Sutter <csutter@instructure.com>
Tested-by: Jenkins
QA-Review: Benjamin Porter <bporter@instructure.com>
This makes a number of changes to get the compilation compatible with
node 12 and iojs.
- removes loom (no longer used)
- updates karma dependencies to more recent versions
- minor css change to make sass compilation work with node-sass
(lib-sass)
test plan:
- should be able to compile assets with node 12 and iojs
- JS specs still pass
Change-Id: Id4a0628ff0c6dd09e622d02329a4400d60459ec2
Reviewed-on: https://gerrit.instructure.com/49493
Tested-by: Jenkins
Reviewed-by: Nick Houle <nhoule@instructure.com>
Product-Review: Cameron Matheson <cameron@instructure.com>
QA-Review: Cameron Matheson <cameron@instructure.com>
bash best practice is to use double quotes to prevent globbing and word
splitting. This probably doesn't matter in the context we are using,
but it pollutes the output of the linter so might as well just fix it.
Also switch `` to $(). Backticks are deprecated
Fixes CNVS-18183
Change-Id: I03cef6a7b3e1740bfb46c422e89788cd5d1eb7e0
Reviewed-on: https://gerrit.instructure.com/47726
Reviewed-by: Mark Severson <markse@instructure.com>
Tested-by: Jenkins
QA-Review: Adam Stone <astone@instructure.com>
Product-Review: Benjamin Porter <bporter@instructure.com>
closes: CNVS-17331 CNVS-14993
refs: CNVS-17332
This is the first step to having a full theme editor where
accounts can pick their own color scheme in a UI and all of
canvas will be branded to their school colors.
test plan:
* copy config/brand_variables.scss.example to config/brand_variables.scss
* recompile sass
* users in accounts that have "new styles" or "k12" turned on that have
not turned on "high contrast". should see a new purple-orange canvas.
* but users that have turned on the "high_contrast" feature or accounts
that have not turned on either "k12" or "new styles" should not.
Change-Id: I16342d43b56e49d52fbee8fa5c6a0fd57ae6e602
Reviewed-on: https://gerrit.instructure.com/46085
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
QA-Review: Jeremy Putnam <jeremyp@instructure.com>
Product-Review: Colleen Palmer <colleen@instructure.com>
this will allow us to enforce linting in the build
also ignore plugins' vendored javascript
Change-Id: Ibf6e9b73e0051a0b581ee52b641f9ff00dd6b2f2
Reviewed-on: https://gerrit.instructure.com/46351
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Jon Jensen <jon@instructure.com>
no significant issues/vulnerabilities fixed in this commit, rather this is
a catch-all so we can enforce linting in the build.
to better understand what's going on here, see the following:
* public/javascripts/.xssignore
* script/xsslint.js
* https://github.com/jenseng/xsslint
high level summary of what's going on in this commit:
1. .html/.append/etc. now know what to do with a SafeString ... in many
cases we now put a $.raw around an .html argument to tell the linter
it's ok
2. although translation is an unlikely attack vector, we now htmlEscape
I18n.t calls used in html snippets, etc. this is a good thing, as it
ensures we don't create a vulnerability later (e.g. by interpolating
user content into a translation)
3. many vars were renamed (Html suffix, $ prefix) to let the linter know
it's something that was manually vetted
4. in some cases, rather than renaming or creating a superfluous var,
we add special xsslint comment overrides
test plan:
specs should all pass
Change-Id: Ide1df825b798d1b0c468a5308802543bf716c0d7
Reviewed-on: https://gerrit.instructure.com/46097
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Derek Hansen <dhansen@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
this swaps out our "parsing" with i18nliner-js
also remove localization abilities of I18n.beforeLabel, since it's not
worth the trouble to support. it can still be called to format a string.
notable changes/fixes in generated yml:
1. client_apps are picked up by i18n:rake so they'll get translated...
due to the old short-circuiting logic (`rc = rc && ...`), it would
stop processing scripts within a particular file after the first one
it found without an I18n.t
2. we no longer incorrectly double-escape special chars in our js strings
(e.g. newlines are actually newlines, not a literal "\n")
test plan:
1. verify string extraction:
1. `rake js:generate i18n:generate` before and after this commit
2. confirm `config/locales/generated/en.yml` is identical, except the
notable changes/fixes listed above
2. verify js translation file generation:
1. `rake i18n:generate_js` before and after this commit
2. confirm the files in public/javascripts/translations are identical
3. verify client_app checker still works:
1. `cd client_apps/canvas_quiz_statistics/`
2. `grunt check_i18n`
Change-Id: Ic8ad058bee1c9476f42916f10b612c1c08863fe3
Reviewed-on: https://gerrit.instructure.com/42809
Reviewed-by: Michael Ziwisky <mziwisky@instructure.com>
Product-Review: Michael Ziwisky <mziwisky@instructure.com>
QA-Review: Michael Ziwisky <mziwisky@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
closes CNVS-15881
This mostly involved adding hooks to the gem for things that we were
directly modifying in Canvas, then hooking into those points in the
canvas config/initializers/delayed_job.rb file.
Periodic jobs scheduling changed a bit as well -- there's no longer
support in the extracted gem for reading a special file under config/,
so I moved our periodic_jobs.rb file to a normal initializer.
test plan: delayed jobs should still work as before, including queuing
jobs, running the worker pool, and in tests.
Change-Id: I9ce57091d18f21f4355011fcb75230193d53facb
Reviewed-on: https://gerrit.instructure.com/42027
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
and my fingers still want to type it. at least this will remove
it from my tab-complete
Change-Id: Id826cff0612c27bb54dc2c74925193b8f33a66cf
Reviewed-on: https://gerrit.instructure.com/41607
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Rob Orton <rob@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
closes CNVS-14275
While we're at it, fix all the rspec deprecation warnings.
test plan: delayed jobs should still work as before, including queuing,
viewing in the UI, and running.
Change-Id: I36c6b74aa2b59a99e4f1f36e25e6d0e9e153f92a
Reviewed-on: https://gerrit.instructure.com/41211
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
We compile sass using npm run compile-sass now, but the
script was still referencing the old (now deprecated) rake
task, which was causing updates to fail.
Change-Id: Idbedcf61ccccbdb0f1deb5907afb25d5d6650000
Reviewed-on: https://gerrit.instructure.com/40392
Reviewed-by: Nick Houle <nhoule@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Dave Donahue <ddonahue@instructure.com>
QA-Review: Dave Donahue <ddonahue@instructure.com>
...and print a message of what went wrong
test plan:
write some bad syntax in a sass file,
run `npm run compile-sass`
it should exit with non-zero status code
it should tell you what line of css caused the error
Change-Id: I0febb3286f6100993d5d2d53083d6a53719cbedf
Reviewed-on: https://gerrit.instructure.com/39385
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Colleen Palmer <colleen@instructure.com>
Product-Review: Ryan Shaw <ryan@instructure.com>
QA-Review: Ryan Shaw <ryan@instructure.com>
after many steps towards this moment, we're finally here
This yanks sass and compass out of canvas-lms
completely and instead uses the libsass based
node-sass to compile our SASS files.
wins:
It is WAYYY faster!
as in, < 10 seconds to recompile all css in canvas
(compared to the 5+ minutes it used to take)
It is all in JS, helping use move to a completely
nodeJS based fronted tooling workflow.
next steps:
remove jammit: we don't need an assets.yml file
since node-sass can output compressed css for us
and we use sass to do all of our @import'ing of other
files (@colleen calls those "compiler" sheets), this
would simplify and speed up fronted asset building
even more
use gulp/broccoli/whatev to do cached, incremental builds
test plan:
all outputted css should look exactly the
same as it used to.
run `npm run compile-sass`, make sure it works
and is way faster than `rake css:generate` used to be
Change-Id: I7d865ea6b3e374cdc27a883d2019a4c15746c0e2
Reviewed-on: https://gerrit.instructure.com/38416
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Trevor deHaan <tdehaan@instructure.com>
Product-Review: Ryan Shaw <ryan@instructure.com>
test plan:
* run `script/canvas_update`
* verify that the log output goes in /log
* verify that canvas was updated
* run `script/canvas_update -q` for QUICK_MODE
* verify QUICK_MODE's output is:
Quick mode enabled (assumes you have guard running and don't want to
generate docs)
Bringing Canvas up to date ...
Log file is /Users/lwilkins/sandbox/canvas-lms/log/canvas_update.log
Updating plugin vendor/plugins/canvasnet_registration ...
Updating plugin vendor/plugins/instructure_misc_plugin ...
Updating plugin vendor/plugins/multiple_root_accounts ...
Pulling Canvas code ...
Checking your gems (bundle check) ...
Gems are up to date, no need to bundle install ...
Migrating DB ...
Installing npm packages ...
Tips:
- 'bundle exec guard': auto-compiles JS files while developing
- 'script/delayed_job run': run delayed jobs in the foreground
NOTE 1) if you see
Gems are up to date, no need to bundle install ...
then `bundle install` will not be run
NOTE 2) after running this command, you'll be on master branch,
if you want to continue testing this script, checkout the
branch you pulled down again.
Change-Id: I99c3097d452128d6fffa31d25e3b7e57a438555e
Reviewed-on: https://gerrit.instructure.com/37423
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Braden Anderson <braden@instructure.com>
Product-Review: Landon Wilkins <lwilkins@instructure.com>
QA-Review: Landon Wilkins <lwilkins@instructure.com>
self.up works in rails2 and rails3, but up only works in rails3+
also tag the switchman migrations as RAILS3 only, now that they actually
run in rails2
Change-Id: I768c8e657e86de6504a40444d127e2d875ce6934
Reviewed-on: https://gerrit.instructure.com/35390
Tested-by: Brian Palmer <brianp@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
fixes CNVS-12182
test plan
- regression test on incoming mail
- use script/process_incoming_emails to manually trigger the processing
of incoming mail
Change-Id: Iccd74d8fe2b5af3d5eefe25a2736273e3bf559b0
Reviewed-on: https://gerrit.instructure.com/32794
Reviewed-by: Simon Williams <simon@instructure.com>
Reviewed-by: Braden Anderson <banderson@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Trevor deHaan <tdehaan@instructure.com>
Product-Review: Joel Hough <joel@instructure.com>
fixes CNVS-12145
QA Test Plan:
- regression test incoming mail
- ensure that reply to discussion topic works
Change-Id: Iae88aa6da5cfe79e51609e233c05e356feacc198
Reviewed-on: https://gerrit.instructure.com/32610
Reviewed-by: Braden Anderson <banderson@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Trevor deHaan <tdehaan@instructure.com>
Product-Review: Joel Hough <joel@instructure.com>
fixes CNVS-12129
test plan
- regression on incoming mail
Change-Id: Ia9ab3419201c9fdbd89e2483a3fde51f54c7f982
Reviewed-on: https://gerrit.instructure.com/32594
Reviewed-by: Braden Anderson <banderson@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Trevor deHaan <tdehaan@instructure.com>
Product-Review: Joel Hough <joel@instructure.com>
...by parallelizing everything I could
fixes: CNVS-12362
`time bundle exec rake canvas:compile_assets`
(on my I7 2.3GHz macbook pro)
before: "322.56s user 26.62s system 173% cpu 3:21.05 total"
after: "425.91s user 30.71s system 413% cpu 1:50.35 total"
If you ever find yourself needing to run everything all
sequentially (and not in multiple threads), you can
set the environment variable CANVAS_BUILD_CONCURRENCY=1
by default it will use all the cores your system has
You're welcome, anyone that has ever had to build canvas
or wait for jenkins to run :)
Test Plan:
* run "time bundle exec rake canvas:compile_assets"
* checkout this code
* run it again
* verify that it is faster, and everything works
Change-Id: Ib01506bc9638e86f4a329284458706279ef751ab
Reviewed-on: https://gerrit.instructure.com/33127
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Nathan Rogowski <nathan@instructure.com>
Product-Review: Bracken Mosbacker <bracken@instructure.com>
refs CNVS-7597
render :json => thing will call ActiveSupport::JSON.encode(thing) unless
thing is a String. ActiveSupport::JSON.encode(thing) just calls
thing.to_json but with some circular reference checking that we want. we
may also want enhance ActiveSupport::JSON.encode to do additional
processing, and calling to_json straight up would bypass that.
in the cases where we do need to do the structural transformation before
passing to render :json (e.g. because of need to pass arguments), use
as_json to do structural transformation only, vs. to_json that does
serialization of the as_json result.
adds a rake task to lint the controllers to enforce as_json over to_json
in render json calls.
test-plan: heavy regression testing; no end behavior should change
(except a pair of serialization bugs that got fixed)
Change-Id: I7a91a9fe0eca70456bc5bca233f0ed6b27a54aaf
Reviewed-on: https://gerrit.instructure.com/23650
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Product-Review: Jacob Fugal <jacob@instructure.com>
QA-Review: Jacob Fugal <jacob@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
* automatically does with_exclusive_scope
* if the scope has an order or group by, use an alternate
implementation so it's not broken
* alternate methods are cursors (postresql only) or temporary
tables. if a transaction is already open, or we're on a slave,
prefer a cursor (even over the normal method, since the query
might be slow even without an order or group by)
test plan:
* migrations should still work (especially data fixups)
* lots of account reports use find_each; make sure they still run
Change-Id: If876b7b3401e6cda1d41f1c94b93af4810b78cf4
Reviewed-on: https://gerrit.instructure.com/22364
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
Change-Id: I391010689a6488c49a75eaef586783793fc29613
Reviewed-on: https://gerrit.instructure.com/20889
Reviewed-by: Cameron Matheson <cameron@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
Product-Review: Stanley Stuart <stanley@instructure.com>
QA-Review: Stanley Stuart <stanley@instructure.com>
Reviewed-by: Zach Pendleton <zachp@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
also added a warning to migration_lint about tables with no primary keys
test plan:
* run the migration; it should not fail
Change-Id: I67a065355bb516b84b00313725f079842bbd8974
Reviewed-on: https://gerrit.instructure.com/20705
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
- update the Gemfile to be 1.9 only, and raise an exception on wrong
ruby version
- remove RUBY_VERSION checks, replacing with the applicable code
- remove the FasterCSV compatibility shim, just use CSV now
test plan: trying to bundle install on ruby 1.8 or 2.0 should raise an
exception, specs should pass, canvas should work as normal on 1.9
Change-Id: I49088e9d227c59c6d5d5acb417c2df971129474a
Reviewed-on: https://gerrit.instructure.com/19806
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Clare Strong <clare@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
replaced the mailman gem with custom code with more error
handling. this will allow the incoming message processor to
continue processing messages after encountering a message with
an encoding or parsing error. the broken messages will be moved
aside to a separate folder for later inspection.
fixes CNVS-4970
test plan:
- read up on the new incoming_mail.yml configuration settings.
- configure incoming_mail.yml with the test imap accounts
using legacy settings and check for regressions.
- reconfigure incoming_mail.yml to read from a directory.
- copy some testing email files into the configured directory.
test files should be a mix of:
- emails with encoding errors
- emails with syntax errors
- normal emails
- all of the normal emails should be processed normally
- all of the error emails should be moved into the error
subdirectory
Change-Id: I0f946a56b41492f007db2775aa6da3cdfa4fdd3f
Reviewed-on: https://gerrit.instructure.com/19729
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Zach Pendleton <zachp@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
This can successfully load rails console and rails server. There are
many, many problems still. The idea is this won't change anything under
rails 2.3, it's all backwards compatible.
closes CNVS-4711
test plan: `touch RAILS3` in your Canvas Rails.root directory. The run
`bundle update` and verify that you get rails 3 installed. Run `bundle
exec rails c` to load console or `bundle exec rails s` to start a
webrick server. You can login, though the dashboard currently breaks.
Also jammit isn't working yet.
But more importantly, Rails 2.3 should still work same as ever. All
tests should pass, and a basic regression sanity check would be good too.
Change-Id: Idd6f35de88adde84cd2db3a650f44b71bd6e9684
Reviewed-on: https://gerrit.instructure.com/18453
Reviewed-by: Brian Palmer <brianp@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Clare Hetherington <clare@instructure.com>
Product-Review: Bracken Mosbacker <bracken@instructure.com>
Cody Cutrer