export legacy files to inst-fs on-demand

fixes SAS-1443

requires opt-in (ramping up rate in plugin setting, defaults to 0). will
initially opt in on sandbox accounts, then trial accounts, then a
gradual roll out to avoid hammering the service.

when opted in (100%), asking for a redirect target for a file that's not
yet in inst-fs will first try to add it to Inst-FS by posting the
metadata only (a nominally quick transaction). Inst-FS can use this to
create a "foreign" reference to the existing object in canvas' s3 bucket
(which inst-fs should be given access to), at which point it can
immediately start serving the object.

opting in at a level above 0% but below 100% will have the chosen chance
of doing the above per-access.

if successful, Inst-FS returns the new instfs_uuid with which the
attachment is updated, then the redirect continues to inst-fs instead of
s3. if unsuccessful, opted-out, or skipped due to opt-in threshold, the
existing s3 object is redirected to as before.

test-plan:
  (setup)
  - have inst-fs integrated locally
  - be using an s3 backing for non-instfs files
  - give inst-fs access to that s3 backing

  (testing)
  - turn off the inst-fs plugin setting
  - upload a file
  - turn on the inst-fs plugin setting, but leave the migrating flag
    unset
  - access the previously uploaded file, should be served from s3
  - turn on the migrating flag in the plugin setting
  - access the previously uploaded file, should be served (or at least
    attempted, if you haven't allowed inst-fs access to the canvas s3
    bucket) from inst-fs

Change-Id: I2b909d9b3f1c950e5a88f2af6ec7b40c05e71bb6
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/234167
Reviewed-by: Michael Jasper <mjasper@instructure.com>
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: Jacob Fugal <jacob@instructure.com>
Product-Review: Jacob Fugal <jacob@instructure.com>

app/models/attachment.rb

@@ -512,6 +512,14 @@ class Attachment < ActiveRecord::Base
     end
   end
 
+  def root_account
+    begin
+      root_account_id && Account.find_cached(root_account_id)
+    rescue ::Canvas::AccountCacheError
+      nil
+    end
+  end
+
   def namespace
     read_attribute(:namespace) || (new_record? ? write_attribute(:namespace, infer_namespace) : nil)
   end
@@ -836,11 +844,7 @@ class Attachment < ActiveRecord::Base
   end
 
   def url_ttl
-    settings = begin
-      root_account_id && Account.find_cached(root_account_id).settings
-    rescue ::Canvas::AccountCacheError
-    end
-    setting = settings&.[](:s3_url_ttl_seconds)
+    setting = root_account&.settings&.[](:s3_url_ttl_seconds)
     setting ||= Setting.get('attachment_url_ttl', 1.hour.to_s)
     setting.to_i.seconds
   end

app/views/plugins/_inst_fs_settings.html.erb

@@ -17,4 +17,20 @@
 %>
 
 <%= fields_for :settings, OpenObject.new(settings) do |f| %>
+  <table style="width: 500px;" class="formtable">
+    <tr>
+      <td colspan="2">
+        <p><%= t(:description, <<-TEXT)
+          Hosts uploaded files with the Inst-FS service.
+          TEXT
+        %></p>
+      </td>
+    </tr>
+    <tr>
+      <td><%= f.blabel :migration_rate, :en => "Chance of migrating older files into Inst-FS on access (0 to 100)" %></td>
+      <td>
+        <%= f.text_field :migration_rate %>
+      </td>
+    </tr>
+  </table>
 <% end %>

lib/canvas/plugins/default_plugins.rb

@@ -405,8 +405,11 @@ Canvas::Plugin.register('inst_fs', nil, {
   :author => 'Instructure',
   :author_website => 'http://www.instructure.com',
   :version => '0.0.1',
-  :settings => nil,
-  :settings_partial => 'plugins/inst_fs_settings'
+  :settings => {
+    :migration_rate => 0,
+  },
+  :settings_partial => 'plugins/inst_fs_settings',
+  :validator => 'InstFsValidator'
 })
 Canvas::Plugin.register('unsplash', nil, {
   name: lambda{ t :name, 'Unsplash' },

lib/canvas/plugins/validators/inst_fs_validator.rb

@@ -0,0 +1,33 @@
+#
+# Copyright (C) 2020 - present Instructure, Inc.
+#
+# This file is part of Canvas.
+#
+# Canvas is free software: you can redistribute it and/or modify it under
+# the terms of the GNU Affero General Public License as published by the Free
+# Software Foundation, version 3 of the License.
+#
+# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
+# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along
+# with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+module Canvas::Plugins::Validators::InstFsValidator
+  def self.validate(settings, plugin_setting)
+    if settings[:migration_rate].blank?
+      migration_rate = 0
+    else
+      migration_rate = settings[:migration_rate].to_f rescue nil
+    end
+    if migration_rate.nil? || migration_rate < 0 || migration_rate > 100
+      plugin_setting.errors.add(:base, I18n.t('Please enter a number between 0 and 100 for the migration rate'))
+      return false
+    end
+    settings[:migration] = migration_rate
+    settings.slice(:migration_rate).to_h.with_indifferent_access
+  end
+end

lib/file_authenticator.rb

@@ -51,19 +51,9 @@ class FileAuthenticator
     Digest::MD5.hexdigest("#{@user&.global_id}|#{@acting_as&.global_id}|#{@oauth_host}")
   end
 
-  def instfs_options(attachment, extras={})
-    {
-      user: @user,
-      acting_as: @acting_as,
-      access_token: @access_token,
-      root_account: @root_account,
-      oauth_host: @oauth_host,
-      expires_in: attachment.url_ttl,
-    }.merge(extras)
-  end
-
   def download_url(attachment)
     return nil unless attachment
+    migrate_legacy_attachment_to_instfs(attachment)
     if attachment.instfs_hosted?
       options = instfs_options(attachment, download: true)
       InstFS.authenticated_url(attachment, options)
@@ -75,6 +65,7 @@ class FileAuthenticator
 
   def inline_url(attachment)
     return nil unless attachment
+    migrate_legacy_attachment_to_instfs(attachment)
     if attachment.instfs_hosted?
       options = instfs_options(attachment, download: false)
       InstFS.authenticated_url(attachment, options)
@@ -93,4 +84,32 @@ class FileAuthenticator
       attachment.thumbnail_url(options)
     end
   end
+
+  private
+
+  def migrate_legacy_attachment_to_instfs(attachment)
+    # on-demand migration to inst-fs (if necessary and opted into). this is a
+    # server to server communication during the request (just before redirect,
+    # typically), but it's only a POST of the metadata to inst-fs, so should be
+    # quick, which we enforce with a timeout; inst-fs will "naturalize" the
+    # object contents later out-of-band
+    return unless InstFS.migrate_attachment?(attachment)
+    attachment.instfs_uuid = InstFS.export_reference(attachment)
+    attachment.save!
+  rescue InstFS::ExportReferenceError
+    # in the case of a recognized error exporting reference, just ignore,
+    # acting as if we didn't intend to migrate in the first place, rather than
+    # interrupting what could be a successful redirect to s3
+  end
+
+  def instfs_options(attachment, extras={})
+    {
+      user: @user,
+      acting_as: @acting_as,
+      access_token: @access_token,
+      root_account: @root_account,
+      oauth_host: @oauth_host,
+      expires_in: attachment.url_ttl,
+    }.merge(extras)
+  end
 end

lib/inst_fs.rb

@@ -22,6 +22,14 @@ module InstFS
       Canvas::Plugin.find('inst_fs').enabled? && !!app_host && !!jwt_secret
     end
 
+    def check_migration_rate?
+      rand < Canvas::Plugin.find('inst_fs').settings[:migration_rate].to_f / 100.0
+    end
+
+    def migrate_attachment?(attachment)
+      enabled? && !attachment.instfs_hosted? && Attachment.s3_storage? && check_migration_rate?
+    end
+
     def login_pixel(user, session, oauth_host)
       return if session[:oauth2] # don't stomp an existing oauth flow in progress
       return if session[:pending_otp]
@@ -153,6 +161,60 @@ module InstFS
       raise InstFS::DirectUploadError, "received code \"#{response.code}\" from service, with message \"#{response.body}\""
     end
 
+    def export_reference(attachment)
+      raise InstFS::ExportReferenceError, "attachment already has instfs_uuid" if attachment.instfs_hosted?
+      raise InstFS::ExportReferenceError, "can't export non-s3 attachments to inst-fs" unless Attachment.s3_storage?
+
+      # compare to s3_bucket_url in the aws-sdk-s3 gem's
+      # lib/aws-sdk-s3/customizations/bucket.rb; we're leaving out the bucket
+      # name from the url. otherwise, this is effectively
+      # `attachment.bucket.url`
+      s3_client = attachment.bucket.client
+      s3_url = s3_client.config.endpoint.dup
+      if s3_client.config.region == 'us-east-1' &&
+         s3_client.config.s3_us_east_1_regional_endpoint == 'legacy'
+        s3_url.host = Aws::S3::Plugins::IADRegionalEndpoint.legacy_host(s3_url.host)
+      end
+
+      body = {
+        objectStore: {
+          type: "s3",
+          params: {
+            host: s3_url.to_s,
+            bucket: attachment.bucket.name
+          }
+        },
+        # single reference
+        references: [{
+          storeKey: attachment.full_filename,
+          timestamp: attachment.created_at.to_i,
+          filename: attachment.filename,
+          displayName: attachment.display_name,
+          content_type: attachment.content_type,
+          encoding: attachment.encoding,
+          size: attachment.size,
+          user_id: attachment.context_user&.global_id&.to_s,
+          root_account_id: attachment.root_account&.global_id&.to_s,
+          sha512: nil, # to be calculated by inst-fs
+        }]
+      }.to_json
+
+      response = CanvasHttp.post(export_references_url, body: body, content_type: "application/json")
+      raise InstFS::ExportReferenceError, "received code \"#{response.code}\" from service, with message \"#{response.body}\"" unless response.class == Net::HTTPOK
+
+      json_response = JSON.parse(response.body)
+      well_formed =
+        json_response.is_a?(Hash) &&
+        json_response.key?("success") &&
+        json_response["success"].is_a?(Array) &&
+        json_response["success"].length == 1 &&
+        json_response["success"][0].is_a?(Hash)
+        json_response["success"][0].key?("id")
+      raise InstFS::ExportReferenceError, "import succeeded, but response body did not have expected shape" unless well_formed
+
+      json_response["success"][0]["id"]
+    end
+
     def duplicate_file(instfs_uuid)
       token = duplicate_file_jwt(instfs_uuid)
       url = "#{app_host}/files/#{instfs_uuid}/duplicate?token=#{token}"
@@ -387,6 +449,7 @@ module InstFS
   end
 
   class DirectUploadError < StandardError; end
+  class ExportReferenceError < StandardError; end
   class DuplicationError < StandardError; end
   class DeletionError < StandardError; end
 end