This course.destroy call was marking the workflow_state of the course as
sticky, so that it could never be un-deleted via a SIS import ever
again.
I think an argument could be made that actually courses whose
workflow_state is currently sticky because of a UI change, shouldn't
even be deleted by a full batch delete. That's a big change from how
batch deletes currently work, though, and might cause issues with
manually published courses.
sections and enrollments also get batch deleted, but their
workflow_state fields aren't sticky.
refs #9200
test plan: import a course via sis as active, then upload a full batch
sis that doesn't include the course. the course should now be deleted.
then import the course again, and see that it's no longer deleted.
Change-Id: I14925e60fd4bd24dafaf8143a555eec6cccc9236
Reviewed-on: https://gerrit.instructure.com/12014
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Bracken Mosbacker <bracken@instructure.com>
basically, use the API from the UI, and delete duplicate code.
the api specs already covered everything the non-api specs tested
also drop the unused sis_batch_log_entries table
test plan:
* validate an sis upload from the UI works, and shows progress,
and completes successfully
* validate the ui shows progress when the page is reloaded and
a batch is in progress
Change-Id: I9be1ac5901a5c6a1b01719718eeda106aaf3045d
Reviewed-on: https://gerrit.instructure.com/11595
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
And process all pending batches in the job, don't require one job per
batch.
This way we don't pile up jobs for accounts that send frequent sis
updates.
There is also an optional setting to add a delay before the job starts,
so that we can process more batches per job invocation if they are being
sent frequently.
test plan: make sure that sis batches are still processed, including
batches that were queued before the code update. verify that no matter
how many batches are queued, only one job per strand will be in the
queue (sometimes 2, if one is already running and a second is added).
Change-Id: I4d551ef9fa45d9d332244ab29ca8d061fa5d7b9c
Reviewed-on: https://gerrit.instructure.com/11465
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
fixes#8310
test plan:
* load some sis data for terms/courses/sections/enrollments into an
account
* do a batch mode import of an enrollments.csv with just the headers,
no data
* all enrollments should be deleted, but courses and sections should
be left alone
Change-Id: I05beb577bfb4573e97f8b398aeb4bfff5df78a4f
Reviewed-on: https://gerrit.instructure.com/10265
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
test plan:
* run the migration
* redo the migration
* do a batch mode sis import
Change-Id: I0204256f3825ffe0b4d252bbb83a0604d7a58942
Reviewed-on: https://gerrit.instructure.com/7904
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Jacob Fugal <jacob@instructure.com>
Reviewed-by: Ben Chobot <bench@instructure.com>
also remove existing attachments that aren't in a folder from their list
test plan:
* run specs
* create sis batches - make sure their position is NULL
* destroy their associated attachment, the query logs should not indicate
an UPDATE attachments SET position = position - 1... being executed
Change-Id: I6117c2ade0ef5ec41a3ffeeb13096b9d8473f3f9
Reviewed-on: https://gerrit.instructure.com/7843
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
test-plan:
- import a course, a section, and an enrollment as a sis batch
- import another sis batch that doesn't include the previous imported
entities without specifying a term
- none of the original course, section, and enrollment should be
deleted
Change-Id: Ia03d4abf8c86d2ebd14987357ad073033ab28700
Reviewed-on: https://gerrit.instructure.com/7168
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: JT Olds <jt@instructure.com>
notable changes:
* nothing is processed as a sis-import blocking error now. bad imports now
result in warnings, while just skipping bad data
* we no longer check for duplicates before going to the database
Change-Id: Iedc96b29d92caccdc6a71ae1de8100a1c82dd137
Reviewed-on: https://gerrit.instructure.com/5724
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
We decided this was extremely dangerous/error-prone and not that useful.
Change-Id: Ib0ff0c9d4ac8eeeb9d2088ed86be24991561a763
Reviewed-on: https://gerrit.instructure.com/3994
Reviewed-by: Zach Wily <zach@instructure.com>
Tested-by: Hudson <hudson@instructure.com>
In this mode, any courses, sections or enrollments that aren't
referenced in the sis import will be deleted.
This is the back-end piece, support in the UI and API are coming next.
Change-Id: Ica634f9fa7008ef7b612c4ea43163792f5d54fa5
Reviewed-on: https://gerrit.instructure.com/3768
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: JT Olds <jt@instructure.com>
Reviewed-by: Zach Wily <zach@instructure.com>
* Controlled by plugin settings; default 1
* Will re-balance incoming CSVs into at least <parallelism> new CSVs
(per import type)
* If there is a single CSV, it will be split optimally
* If there are multiple CSVs, no such guarantees are made
* Each CSV will be processed as a separate DelayedJob
* Accounts are not parallelized
* All other import types are parallelized within their own type. I.e.
all courses will be imported before any sections are started
Change-Id: Ifa6fe4e63812be95db999bdf963ab7cc1b15b75d
Reviewed-on: https://gerrit.instructure.com/3582
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: JT Olds <jt@instructure.com>
Refactor to use it in places where it makes sense. This eliminates all
the places where were were calling File.read or otherwise reading into
memory potentially large files just to write them back out.
Change-Id: I679ce9979933140340b30cc043540ca3f834097e
Reviewed-on: https://gerrit.instructure.com/3324
Tested-by: Hudson <hudson@instructure.com>
Reviewed-by: Bracken Mosbacker <bracken@instructure.com>
Brian Palmer