ylqjgm
/
canvas-lms
mirror of https://github.com/instructure/canvas-lms.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

refactor RevManifest to simplify the CDN interface 

refs FOO-2520
flag = none

[pin-commit-multiple_root_accounts=2a9bf89895f38df6bf8f54828af66aced594abf0]

revisit the API for resolving asset names to their (real)path on disk,
because adding to the existing logic to support an alternative bundler
made things hard to understand.

This patch brings a new simplified interface Canvas::Cdn::Registry to
query assets and resolve their location.

- Registry#include?(path) tells whether a realpath points to a static
  asset
- Registry#statics_available? tells whether static assets are available
- Registry#scripts_available? tells whether JS assets are available
- Registry#scripts_for(bundle) provides the realpaths to all the JS
  files in the specified bundle
- Registry#url_for(name) provides the realpath to the static asset

The Registry is a good place to house the BrandableCSS resolving logic
in the future for even more consistency. It can also support an
alternative bundler internally without leaking. Eventually, it would be
nice to have it as a gem.

CHANGES
-------

- helper "font_url_for()" has been removed as it was a duplicate of
  existing logic; instead use "font_path(...)" to achieve the correct
  result. As a result, BrandableCSS is no longer querying Gulp's
  manifest.
- preloaded fonts are now aware of the asset host and work for CDN
- InfoController uses the new Registry API to tell whether Gulp and
  Webpack have produced their assets successfully
- ApplicationHelper no longer re-computes the base URL for JavaScripts,
  now only the Registry is concerned with that
- ?optimized_js query parameter is no longer supported as it has no real
  benefit now that we have access to sourcemaps on production
- ENV['USE_OPTIMIZED_JS'] is now more consistent as there is a single
  source of truth for it. The Registry can be instantiated with
  {environment: "production"} to point to the optimized version of the
  scripts.
- "css:compile" task no longer writes BrandConfig records to the DB,
  that is now done as part of the "compile_assets" task, which you can
  opt out of doing by setting COMPILE_ASSETS_BRAND_CONFIGS=0

TEST PLAN
---- ----

- load your dashboard and verify all the assets are loaded correctly
- set up a CDN, restart your Rails server and reload the dashboard
  - verify all assets are loaded from the CDN
  - verify the Lato fonts are pre-loaded from the CDN
- (optional) add custom JS to a sub-account and visit it
  - verify the custom JS is loaded and evaluated *after* Canvas's main
    javascript bundles

Change-Id: I8198de747cdd5892d6a831cb6c61ba0ef9afa789
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/276537
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
QA-Review: James Butters <jbutters@instructure.com>
Reviewed-by: Charley Kline <ckline@instructure.com>
Product-Review: Charley Kline <ckline@instructure.com>
This commit is contained in:
Ahmad Amireh 2021-10-20 17:37:10 -06:00
parent 41be4400c6
commit e61659ffdf
24 changed files with 546 additions and 292 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
Dockerfile.jenkins.webpack-cache
View File

@ -3,7 +3,13 @@ COPY --chown=docker:docker --from=local/cache-helper-collect-webpack /tmp/dst ${
ARG JS_BUILD_NO_UGLIFY=0
ARG RAILS_LOAD_ALL_LOCALES=0
RUN COMPILE_ASSETS_API_DOCS=0 COMPILE_ASSETS_NPM_INSTALL=0 COMPILE_ASSETS_STYLEGUIDE=0 JS_BUILD_NO_UGLIFY="$JS_BUILD_NO_UGLIFY" RAILS_LOAD_ALL_LOCALES="$RAILS_LOAD_ALL_LOCALES" bundle exec rails canvas:compile_assets
RUN COMPILE_ASSETS_API_DOCS=0 \
COMPILE_ASSETS_BRAND_CONFIGS=0 \
COMPILE_ASSETS_NPM_INSTALL=0 \
COMPILE_ASSETS_STYLEGUIDE=0 \
JS_BUILD_NO_UGLIFY="$JS_BUILD_NO_UGLIFY" \
RAILS_LOAD_ALL_LOCALES="$RAILS_LOAD_ALL_LOCALES" \
bundle exec rails canvas:compile_assets
FROM local/ruby-runner AS webpack-cache
COPY --chown=docker:docker --from=webpack-runner /usr/src/app/public ${APP_HOME}/public

14
app/controllers/info_controller.rb
View File

@ -70,8 +70,12 @@ class InfoController < ApplicationController
# javascript/css build process didn't die, right?
asset_urls = {
common_css: css_url_for("common"), # ensures brandable_css_bundles_with_deps exists
common_js: ActionController::Base.helpers.javascript_url("#{js_base_url}/common"), # ensures webpack worked
revved_url: Canvas::Cdn::RevManifest.gulp_manifest.values.first # makes sure `gulp rev` has ran
common_js: ActionController::Base.helpers.javascript_path(
Canvas::Cdn.registry.scripts_for("main").first
),
revved_url: ActionController::Base.helpers.font_path(
"/fonts/lato/extended/Lato-Regular.woff2"
)
}
respond_to do |format|
@ -179,9 +183,7 @@ class InfoController < ApplicationController
# ensures brandable_css_bundles_with_deps exists, returns a string (path), treated as truthy
common_css: check.call { css_url_for("common") },
# ensures webpack worked; returns a string, treated as truthy
common_js: check.call do
ActionController::Base.helpers.javascript_url("#{js_base_url}/common")
end,
common_js: check.call { Canvas::Cdn.registry.scripts_available? },
# returns a PrefixProxy instance, treated as truthy
consul: check.call { DynamicSettings.find(tree: :private)[:readiness].nil? },
# returns the value of the block <integer>, treated as truthy
@ -195,7 +197,7 @@ class InfoController < ApplicationController
# nil response treated as truthy
ha_cache: check.call { MultiCache.cache.fetch("readiness").nil? },
# ensures `gulp rev` has ran; returns a string, treated as truthy
rev_manifest: check.call { Canvas::Cdn::RevManifest.gulp_manifest.values.first },
rev_manifest: check.call { Canvas::Cdn.registry.statics_available? },
# ensures we retrieved something back from Vault; returns a boolean
vault: check.call { !Canvas::Vault.read("#{Canvas::Vault.kv_mount}/data/secrets").nil? }
}

43
app/helpers/application_helper.rb
View File

@ -175,29 +175,6 @@ module ApplicationHelper
object.grants_any_right?(user, session, *actions)
end
# See `js_base_url`
def use_optimized_js?
if params.key?(:optimized_js)
params[:optimized_js] == "true" || params[:optimized_js] == "1"
else
ENV["USE_OPTIMIZED_JS"] == "true" || ENV["USE_OPTIMIZED_JS"] == "True"
end
end
# Determines the location from which to load JavaScript assets
#
# uses optimized:
# * when ENV['USE_OPTIMIZED_JS'] is true
# * or when ?optimized_js=true is present in the url. Run `rake js:build` to
# build the optimized files
#
# uses non-optimized:
# * when ENV['USE_OPTIMIZED_JS'] is false
# * or when ?debug_assets=true is present in the url
def js_base_url
(use_optimized_js? ? "/dist/webpack-production" : "/dist/webpack-dev").freeze
end
def load_scripts_async_in_order(script_urls)
# this is how you execute scripts in order, in a way that doesn’t block rendering,
# and without having to use 'defer' to wait until the whole DOM is loaded.
@ -227,14 +204,14 @@ module ApplicationHelper
# if there is a moment locale besides english set, put a script tag for it
# so it is loaded and ready before we run any of our app code
if js_env[:MOMENT_LOCALE] && js_env[:MOMENT_LOCALE] != "en"
moment_chunks =
Canvas::Cdn::RevManifest.all_webpack_chunks_for("moment/locale/#{js_env[:MOMENT_LOCALE]}")
@script_chunks += moment_chunks if moment_chunks
@script_chunks += ::Canvas::Cdn.registry.scripts_for(
"moment/locale/#{js_env[:MOMENT_LOCALE]}"
)
end
@script_chunks += Canvas::Cdn::RevManifest.all_webpack_chunks_for("main")
@script_chunks += ::Canvas::Cdn.registry.scripts_for("main")
@script_chunks.uniq!
chunk_urls = @script_chunks.map { |s| "#{js_base_url}/#{s}" }
chunk_urls = @script_chunks
capture do
# if we don't also put preload tags for these, the browser will prioritize and
@ -259,13 +236,12 @@ module ApplicationHelper
@rendered_preload_chunks ||= []
preload_chunks =
new_js_bundles.map do |(bundle, plugin, *)|
key = "#{plugin ? "#{plugin}-" : ""}#{bundle}"
Canvas::Cdn::RevManifest.all_webpack_chunks_for(key)
::Canvas::Cdn.registry.scripts_for("#{plugin ? "#{plugin}-" : ""}#{bundle}")
end.flatten.uniq - @script_chunks - @rendered_preload_chunks # subtract out the ones we already preloaded in the <head>
@rendered_preload_chunks += preload_chunks
capture do
preload_chunks.each { |url| concat preload_link_tag("#{js_base_url}/#{url}") }
preload_chunks.each { |url| concat preload_link_tag(url) }
# if you look the ui/main.js, there is a function there that will
# process anything on window.bundles and knows how to load everything it needs
@ -335,11 +311,6 @@ module ApplicationHelper
File.join("/dist", "brandable_css", base_dir, "#{bundle_path}-#{cache[:combinedChecksum]}.css")
end
def font_url_for(nominal_font_href)
cache = BrandableCSS.font_path_cache
cache[nominal_font_href] || nominal_font_href
end
def brand_variable(variable_name)
BrandableCSS.brand_variable_value(variable_name, active_brand_config)
end

6
app/views/layouts/_head.html.erb
View File

@ -21,9 +21,9 @@
<meta charset="utf-8">
<link rel="preconnect" href="https://fonts.gstatic.com/" crossorigin>
<% if Setting.get('disable_lato_extended', false) == 'false' %>
<link rel="preload" href="<%=font_url_for("/fonts/lato/extended/Lato-Regular.woff2")%>" as="font" type="font/woff2" crossorigin>
<link rel="preload" href="<%=font_url_for("/fonts/lato/extended/Lato-Bold.woff2")%>" as="font" type="font/woff2" crossorigin>
<link rel="preload" href="<%=font_url_for("/fonts/lato/extended/Lato-Italic.woff2")%>" as="font" type="font/woff2" crossorigin>
<%= preload_link_tag(font_path("/fonts/lato/extended/Lato-Regular.woff2"), { as: "font", type: "font/woff2", crossorigin: 'anonmyous' }) %>
<%= preload_link_tag(font_path("/fonts/lato/extended/Lato-Bold.woff2"), { as: "font", type: "font/woff2", crossorigin: 'anonmyous' }) %>
<%= preload_link_tag(font_path("/fonts/lato/extended/Lato-Italic.woff2"), { as: "font", type: "font/woff2", crossorigin: 'anonmyous' }) %>
<%= stylesheet_link_tag(css_url_for('lato_extended')) %>
<% else %>
<%= stylesheet_link_tag(css_url_for('lato')) %>

1
build/new-jenkins/library/vars/buildDockerImageStage.groovy
View File

@ -209,6 +209,7 @@ def i18nGenerate() {
-e RAILS_LOAD_ALL_LOCALES=1 \
-e COMPILE_ASSETS_CSS=0 \
-e COMPILE_ASSETS_STYLEGUIDE=0 \
-e COMPILE_ASSETS_BRAND_CONFIGS=0 \
-e COMPILE_ASSETS_BUILD_JS=0 \
$PATCHSET_TAG \
bundle exec rake canvas:compile_assets i18n:generate

1
build/new-jenkins/linters/run-gergich-webpack.sh
View File

@ -3,6 +3,7 @@
set -ex
export COMPILE_ASSETS_NPM_INSTALL=0
export COMPILE_ASSETS_BRAND_CONFIGS=0
export JS_BUILD_NO_FALLBACK=1
gergich capture custom:./build/gergich/compile_assets:Gergich::CompileAssets 'rake canvas:compile_assets'
yarn lint:browser-code

4
config/application.rb
View File

@ -289,7 +289,9 @@ module CanvasRails
config.exceptions_app = ExceptionsApp.new
config.before_initialize do
config.action_controller.asset_host = Canvas::Cdn.method(:asset_host_for)
config.action_controller.asset_host = lambda do |source, *_|
::Canvas::Cdn.asset_host_for(source)
end
end
if config.action_dispatch.rack_cache != false

1
config/brandable_css.yml
View File

@ -7,7 +7,6 @@ paths:
file_checksums: public/dist/brandable_css/brandable_css_file_checksums.json
output_dir: public/dist/brandable_css
browsers_yml: config/browsers.yml
rev_manifest: public/dist/rev-manifest.json
indices:
handlebars:

16
config/initializers/revved_asset_urls.rb
View File

@ -22,17 +22,9 @@
# eg: instead of '/images/whatever.png?12345', we want '/dist/images/whatever-<md5 of file>.png'.
# There is a different method that needs to be monkeypatched for rails 3 vs rails 4
require "action_view/helpers"
require_dependency "action_view/helpers"
require_dependency "canvas/cdn/revved_asset_urls"
module RevAssetPaths
def path_to_asset(source, options = {})
original_path = super
revved_url = ::Canvas::Cdn::RevManifest.url_for(original_path)
if revved_url
File.join(compute_asset_host(revved_url, options).to_s, revved_url)
else
original_path
end
end
Rails.configuration.to_prepare do
ActionView::Base.include(Canvas::Cdn::RevvedAssetUrls)
end
ActionView::Base.include(RevAssetPaths)

34
lib/brandable_css.rb
View File

@ -335,40 +335,6 @@ module BrandableCSS
fingerprint
end
# build a cache of nominal paths to font files to the decorated version we need to request
# e.g. "/fonts/lato/extended/Lato-Bold.woff2": "/dist/fonts/lato/extended/Lato-Bold-cccb897485.woff2"
# only track .woff2 font files since those will be the only ones ever asked for
# (truth be told, could limit it to just lato extended)
# this function is more or less modeled after combined_checksums
def font_path_cache
return @decorated_font_paths if defined?(@decorated_font_paths) && defined?(ActionController) && ActionController::Base.perform_caching
file = APP_ROOT.join(CONFIG.dig("paths", "rev_manifest"))
# in reality, if the rev-manifest.json file is missing you won't get this far, but let's be careful anyway
if file.exist?
return(
@decorated_font_paths =
JSON.parse(file.read).each_with_object({}) do |(k, v), memo|
memo["/#{k}"] = "/dist/#{v}" if /^fonts.*woff2/.match?(k)
end.freeze
)
end
# the file does not exist in production, we have a problem
if defined?(Rails) && Rails.env.production?
raise "#{file.expand_path} does not exist. You need to run brandable_css before you can serve css."
end
# for dev/test there might be cases where you don't want it to raise an exception
# if you haven't run `brandable_css` and the manifest file doesn't exist yet.
# eg: you want to test a controller action and you don't care that it links
# to a css file that hasn't been created yet.
@decorated_font_paths = {
anyfont: "Error: unknown css checksum. you need to run brandable_css"
}.freeze
end
def all_fingerprints_for(bundle_path)
variants.index_with do |variant|
cache_for(bundle_path, variant)

38
lib/canvas/cdn.rb
View File

@ -18,6 +18,7 @@
# with this program. If not, see <http://www.gnu.org/licenses/>.
require_dependency "canvas/cdn/s3_uploader"
require_dependency "canvas/cdn/registry"
require_dependency "config_file"
module Canvas
@ -33,15 +34,42 @@ module Canvas
end
end
# Provides an instance of Registry for the current Rails environment.
#
# Set ENV['USE_OPTIMIZED_JS'] to a truthy value to load the optimized
# version of the JavaScripts even if you're running a development Rails
# server.
def registry
@registry ||= begin
environment = if %w[1 True true].include?(ENV["USE_OPTIMIZED_JS"])
"production"
else
Rails.env
end
Registry.new(
environment: environment,
cache: if ActionController::Base.perform_caching
Registry::ProcessCache.new
else
Registry::RequestCache.new
end
)
end
end
def should_be_in_bucket?(source)
source.start_with?("/dist/brandable_css") || Canvas::Cdn::RevManifest.include?(source)
source.start_with?("/dist/brandable_css") || registry.include?(source)
end
def asset_host_for(source)
return unless config.host # unless you've set a :host in the canvas_cdn.yml file, just serve normally
config.host if should_be_in_bucket?(source)
# Otherwise, return nil & use the same domain the page request came from, like normal.
# use the :host specified in canvas_cdn.yml
if config.host && should_be_in_bucket?(source)
config.host
else
# Otherwise, use the same domain the page request came from, like normal.
nil
end
end
def push_to_s3!(*args, **kwargs, &block)

141
lib/canvas/cdn/registry.rb Normal file
View File

@ -0,0 +1,141 @@
# frozen_string_literal: true
#
# Copyright (C) 2021 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
require_dependency "canvas/cdn/registry/gulp"
require_dependency "canvas/cdn/registry/webpack"
module Canvas
module Cdn
# A registry of the available javascripts and static assets - like images
# and fonts - along with their location on disk. Since these assets have a
# dynamic filename that incorporates a hash fragment, we need a proxy
# interface like this one to resolve their location by their name.
#
# The resolving metadata is provided by the JS bundler, like Webpack, and
# the asset processor, like Gulp.
#
# @see Canvas::Cdn.registry
class Registry
attr_reader :cache, :environment
# @param [Union.<ProcessCache, RequestCache, StaticCache>] :cache
# Control the behavior of loading manifests, see the respective classes
# for more information.
#
# @param [String] :environment
# This only controls the variant of JavaScript assets to locate and has
# no effect on static assets
#
def initialize(cache:, environment: Rails.env)
@cache = cache
@environment = environment
end
# Whether the file is tracked by the registry (i.e. is a JS or static
# asset)
#
# Note that file is expected not to be qualified with a protocol/host; so
# something like /images/foo.png but not http://localhost/images/foo.png.
def include?(realpath)
bundler.include?(realpath) || gulp.include?(realpath)
end
# Whether static assets are locatable
def statics_available?
gulp.available?
end
# Whether JS files are locatable
def scripts_available?
bundler.available?
end
# @return [Array.<String>]
# Real paths to the JS files that make up the specified bundle
delegate :scripts_for, to: :bundler
# @return [String]
# Real path to the asset.
#
# @param [String] source
# Path to the source asset prior to any fingerprinting. This is relative
# to Rails public directory, like "/images/apple-touch-icon.png"
#
delegate :url_for, to: :gulp
private
def gulp
@cache.gulp
end
def bundler
@cache.webpack(environment: @environment)
end
end
# Load manifests at most once per instance
class Registry::ProcessCache
def gulp(*args, **kwargs)
@gulp ||= Registry::Gulp.new(*args, **kwargs)
end
def webpack(*args, **kwargs)
@webpack ||= Registry::Webpack.new(*args, **kwargs)
end
end
# (Re)load manifests on every request
class Registry::RequestCache
def gulp(*args, **kwargs)
::RequestCache.cache(["registry", "gulp"]) do
Registry::Gulp.new(*args, **kwargs)
end
end
def webpack(*args, **kwargs)
::RequestCache.cache(["registry", "webpack"]) do
Registry::Webpack.new(*args, **kwargs)
end
end
end
# Bypass the disk to supply pre-defined manifests
class Registry::StaticCache
def initialize(gulp:, webpack:)
@gulp_manifest = gulp
@webpack_manifest = webpack
end
def gulp(*args, **kwargs)
@gulp ||= Registry::Gulp.new(
*args,
**kwargs.merge(manifest: @gulp_manifest)
)
end
def webpack(*args, **kwargs)
@webpack ||= Registry::Webpack.new(
*args,
**kwargs.merge(manifest: @webpack_manifest)
)
end
end
end
end

80
lib/canvas/cdn/registry/gulp.rb Normal file
View File

@ -0,0 +1,80 @@
# frozen_string_literal: true
#
# Copyright (C) 2021 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
require "set"
module Canvas
module Cdn
class Registry
class Gulp
def initialize(manifest: nil)
@asset_dir = "dist"
@manifest = manifest || load_manifest_from_disk
@files = Set.new(@manifest.values) { |x| realpath(x) }
end
def available?
!@manifest.empty?
end
def include?(realpath)
@files.include?(realpath)
end
def url_for(file)
# source looks like "/images/apple-touch-icon.png"
# or like "/dist/images/apple-touch-icon.png"
# virtpath looks like "images/apple-touch-icon.png"
# realpath looks like "/dist/images/apple-touch-icon-585e5d997d.png"
if (fingerprinted = @manifest[virtpath(file)])
realpath(fingerprinted)
end
end
private
def load_manifest_from_disk
file = Rails.root.join("public/dist/rev-manifest.json")
if file.exist?
JSON.parse(file.read).freeze
elsif Rails.env.production?
raise "you must run \"gulp rev\" first"
else
{}
end
end
def virtpath(source)
normal = source.sub(%r{^/}, "")
if normal.start_with?(@asset_dir)
normal[@asset_dir.length + 1..]
else
normal
end
end
def realpath(vfile)
"/#{@asset_dir}/#{vfile}"
end
end
end
end
end

68
lib/canvas/cdn/registry/webpack.rb Normal file
View File

@ -0,0 +1,68 @@
# frozen_string_literal: true
#
# Copyright (C) 2021 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
require "set"
module Canvas
module Cdn
class Registry
class Webpack
def initialize(environment:, manifest: nil)
@asset_dir = if environment == "production"
"dist/webpack-production"
else
"dist/webpack-dev"
end
@manifest = manifest || load_manifest_from_disk
@files = Set.new(@manifest.values.flatten.uniq) { |x| realpath(x) }
end
def available?
!@manifest.empty?
end
def include?(realpath)
@files.include?(realpath)
end
def scripts_for(bundle)
@manifest.fetch(bundle, []).map { |s| realpath(s) }
end
private
def load_manifest_from_disk
file = Rails.root.join("public/#{@asset_dir}/webpack-manifest.json")
if file.exist?
JSON.parse(file.read).freeze
elsif Rails.env.production?
raise "you must run \"webpack\" first"
else
{}
end
end
def realpath(vfile)
"/#{@asset_dir}/#{vfile}"
end
end
end
end
end

142
lib/canvas/cdn/rev_manifest.rb
View File

@ -1,142 +0,0 @@
# frozen_string_literal: true
#
# Copyright (C) 2015 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
require "set"
# An interface to the manifest file created by `gulp rev` and webpack
module Canvas
module Cdn
module RevManifest
class << self
include ActiveSupport::Benchmarkable
# ActiveSupport::Benchmarkable#benchmark needs a `logger` defined
def logger
Rails.logger
end
def include?(source)
if webpack_request?(source)
true
else
gulp_revved_urls.include?(source)
end
end
def gulp_manifest
load_gulp_data_if_needed
@gulp_manifest
end
def webpack_manifest
load_webpack_data_if_needed
@webpack_manifest
end
def gulp_revved_urls
load_gulp_data_if_needed
@gulp_revved_urls
end
def webpack_request?(source)
source =~ Regexp.new(webpack_dir)
end
def webpack_prod?
ENV["USE_OPTIMIZED_JS"] == "true" || ENV["USE_OPTIMIZED_JS"] == "True"
end
def webpack_dir
if webpack_prod?
"dist/webpack-production"
else
"dist/webpack-dev"
end
end
def all_webpack_chunks_for(bundle)
webpack_manifest[bundle]
end
def webpack_url_for(source)
# source will look something like: "dist/webpack-prod/vendor.js"
# the manifest looks something like: {"vendor.js" : "vendor-c-d4be58c989364f9fe7db.js", ...}
# we want to return something like: "/dist/webpack-prod/vendor-c-d4be58c989364f9fe7db.js"
key = source.sub(webpack_dir + "/", "")
fingerprinted = webpack_manifest[key]
"/#{webpack_dir}/#{fingerprinted}" if fingerprinted
end
def revved_url_for(source)
fingerprinted = gulp_manifest[source]
"/dist/#{fingerprinted}" if fingerprinted
end
def url_for(source)
# remove the leading slash if there is one
source = source.sub(%r{^/}, "")
if webpack_request?(source)
webpack_url_for(source)
else
revved_url_for(source)
end
end
private
def load_gulp_data_if_needed
return if ActionController::Base.perform_caching && defined? @gulp_manifest
RequestCache.cache("rev-manifest") do
benchmark("reading rev-manifest") do
file = Rails.root.join("public/dist/rev-manifest.json")
if file.exist?
Rails.logger.debug "reading rev-manifest.json"
@gulp_manifest = JSON.parse(file.read).freeze
elsif Rails.env.production?
raise "you need to run `gulp rev` first"
else
@gulp_manifest = {}.freeze
end
@gulp_revved_urls = Set.new(@gulp_manifest.values.map { |s| "/dist/#{s}" }).freeze
end
end
end
def load_webpack_data_if_needed
return if (ActionController::Base.perform_caching || webpack_prod?) && defined? @webpack_manifest
RequestCache.cache("webpack_manifest") do
benchmark("reading webpack_manifest") do
file = Rails.root.join("public", webpack_dir, "webpack-manifest.json")
if file.exist?
Rails.logger.debug "reading #{file}"
@webpack_manifest = JSON.parse(file.read).freeze
else
raise "you need to run webpack" unless Rails.env.test?
@webpack_manifest = Hash.new(["Error: you need to run webpack"]).freeze
end
end
end
end
end
end
end
end

37
lib/canvas/cdn/revved_asset_urls.rb Normal file
View File

@ -0,0 +1,37 @@
# frozen_string_literal: true
#
# Copyright (C) 2021 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
require_dependency "canvas/cdn"
require_dependency "canvas/cdn/registry"
module Canvas
module Cdn
module RevvedAssetUrls
def path_to_asset(source, options = {})
original_path = super
revved_url = ::Canvas::Cdn.registry.url_for(original_path)
if revved_url
File.join(compute_asset_host(revved_url, options).to_s, revved_url)
else
original_path
end
end
end
end
end

3
lib/tasks/canvas.rake
View File

@ -36,6 +36,7 @@ unless $canvas_tasks_loaded
build_styleguide = ENV["COMPILE_ASSETS_STYLEGUIDE"] != "0"
build_i18n = ENV["RAILS_LOAD_ALL_LOCALES"] != "0"
build_js = ENV["COMPILE_ASSETS_BUILD_JS"] != "0"
write_brand_configs = ENV["COMPILE_ASSETS_BRAND_CONFIGS"] != "0"
build_prod_js = ENV["RAILS_ENV"] == "production" || ENV["USE_OPTIMIZED_JS"] == "true" || ENV["USE_OPTIMIZED_JS"] == "True"
# build dev bundles even in prod mode so you can debug with ?optimized_js=0
# query string (except for on jenkins where we set JS_BUILD_NO_UGLIFY anyway
@ -43,6 +44,7 @@ unless $canvas_tasks_loaded
build_dev_js = ENV["JS_BUILD_NO_FALLBACK"] != "1" && (!build_prod_js || ENV["JS_BUILD_NO_UGLIFY"] != "1")
batches = Rake::TaskGraph.draw do
task "brand_configs:write" if write_brand_configs
task "css:compile" => ["js:gulp_rev"] if build_css
task "css:styleguide" if build_styleguide
task "doc:api" if build_api_docs
@ -101,6 +103,7 @@ unless $canvas_tasks_loaded
ENV["COMPILE_ASSETS_NPM_INSTALL"] = "0"
ENV["COMPILE_ASSETS_STYLEGUIDE"] = "0"
ENV["COMPILE_ASSETS_API_DOCS"] = "0"
ENV["COMPILE_ASSETS_BRAND_CONFIGS"] = "0"
Rake::Task["canvas:compile_assets"].invoke
end

27
lib/tasks/css.rake
View File

@ -29,25 +29,12 @@ namespace :css do
end
task :compile do
# try to get a conection to the database so we can do the brand_configs:write below
begin
require "config/environment"
rescue => e
puts "WARN: failed to load rails environment before compiling: #{e}"
end
require "config/initializers/revved_asset_urls"
require "lib/brandable_css"
puts "--> Starting: 'css:compile'"
time = Benchmark.realtime do
if (BrandConfig.table_exists? rescue false)
Rake::Task["brand_configs:write"].invoke
else
puts "--> no DB connection, skipping generation of brand_config files"
end
BrandableCSS.save_default_files!
system("yarn run build:css")
raise "error running brandable_css" unless $?.success?
end
puts "--> Finished: 'css:compile' in #{time}"
require "action_view/helpers"
require "canvas/cdn/revved_asset_urls"
require "brandable_css"
ActionView::Base.include(Canvas::Cdn::RevvedAssetUrls)
BrandableCSS.save_default_files!
system("yarn run build:css")
raise "error running brandable_css" unless $?.success?
end
end

9
spec/controllers/info_controller_spec.rb
View File

@ -19,6 +19,8 @@
#
describe InfoController do
include_context "cdn registry stubs"
describe "GET 'health_check'" do
it "works" do
get "health_check"
@ -29,7 +31,6 @@ describe InfoController do
it "respond_toes json" do
request.accept = "application/json"
allow(Canvas).to receive(:revision).and_return("Test Proc")
allow(Canvas::Cdn::RevManifest).to receive(:gulp_manifest).and_return({ test_key: "mock_revved_url" })
get "health_check"
expect(response).to be_successful
json = JSON.parse(response.body)
@ -40,8 +41,8 @@ describe InfoController do
"revision" => "Test Proc",
"asset_urls" => {
"common_css" => "/dist/brandable_css/new_styles_normal_contrast/bundles/common-#{BrandableCSS.cache_for("bundles/common", "new_styles_normal_contrast")[:combinedChecksum]}.css",
"common_js" => ActionController::Base.helpers.javascript_url("#{ENV["USE_OPTIMIZED_JS"] == "true" ? "/dist/webpack-production" : "/dist/webpack-dev"}/common"),
"revved_url" => "mock_revved_url"
"common_js" => "/dist/webpack-dev/main-1234.js",
"revved_url" => "/dist/mock_revved_url"
}
})
end
@ -298,7 +299,7 @@ describe InfoController do
get "web_app_manifest"
manifest = json_parse(response.body)
src = manifest["icons"].first["src"]
expect(src).to start_with("/dist/images/")
expect(src).to eq("/dist/images/apple-touch-icon-1234.png")
end
end
end

6
spec/helpers/application_helper_spec.rb
View File

@ -703,12 +703,6 @@ describe ApplicationHelper do
end
end
describe "js_base_url" do
it "returns an immutable string" do
expect(js_base_url).to be_frozen
end
end
describe "brand_config_account" do
it "handles not having @domain_root_account set" do
expect(helper.send(:brand_config_account)).to be_nil

16
spec/lib/brandable_css_spec.rb
View File

@ -144,20 +144,4 @@ describe BrandableCSS do
end
end
end
describe "font_path_cache" do
it "creates the cache" do
BrandableCSS.font_path_cache
expect(BrandableCSS.instance_variable_get(:@decorated_font_paths)).not_to be_nil
end
it "maps font paths" do
cache = BrandableCSS.font_path_cache
cache.each do |key, val|
expect(key).to start_with("/fonts")
expect(val).to start_with("/dist/fonts")
expect(val).to match(/-[a-z0-9]+\.woff2$/)
end
end
end
end

98
spec/lib/canvas/cdn/registry_spec.rb Normal file
View File

@ -0,0 +1,98 @@
# frozen_string_literal: true
#
# Copyright (C) 2011 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
require File.expand_path(File.dirname(__FILE__) + "/../../../spec_helper.rb")
describe ::Canvas::Cdn::Registry do
subject do
described_class.new(
cache: ::Canvas::Cdn::Registry::StaticCache.new(
gulp: @gulp_manifest || {},
webpack: @webpack_manifest || {}
)
)
end
describe ".statics_available?" do
it "is true when the gulp manifest is available" do
@gulp_manifest = { "foo" => "bar" }
expect(subject.statics_available?).to eq(true)
end
it "is false otherwise" do
expect(subject.statics_available?).to eq(false)
end
end
describe ".scripts_available?" do
it "is true when the webpack manifest is available" do
@webpack_manifest = { "foo" => "bar" }
expect(subject.scripts_available?).to eq(true)
end
it "is false otherwise" do
expect(subject.scripts_available?).to eq(false)
end
end
describe ".url_for" do
it "works for gulp assets" do
@gulp_manifest = { "images/foo.png" => "images/foo-1234.png" }
expect(subject.url_for("/images/foo.png")).to eq(
"/dist/images/foo-1234.png"
)
end
end
describe ".include?" do
it "is true given the path to an asset processed by gulp" do
@gulp_manifest = { "images/foo.png" => "images/foo-1234.png" }
expect(subject.include?("/dist/images/foo-1234.png")).to eq(true)
expect(subject.include?("images/foo-1234.png")).to eq(false)
expect(subject.include?("images/foo.png")).to eq(false)
end
it "is true given the path to a javascript produced by webpack" do
@webpack_manifest = { "main" => ["a-1234.js", "b-1234.js"] }
expect(subject.include?("/dist/webpack-dev/a-1234.js")).to eq(true)
expect(subject.include?("/dist/webpack-dev/b-1234.js")).to eq(true)
expect(subject.include?("a-1234.js")).to eq(false)
expect(subject.include?("main")).to eq(false)
end
end
describe ".scripts_for" do
it "returns realpaths to files within the bundle" do
@webpack_manifest = { "main" => ["a-1234.js", "b-1234.js"] }
expect(subject.scripts_for("main")).to eq(
[
"/dist/webpack-dev/a-1234.js",
"/dist/webpack-dev/b-1234.js",
]
)
end
end
end

8
spec/selenium/browser_operations/cdn_spec.rb
View File

@ -76,7 +76,7 @@ describe "Stuff related to how we load stuff from CDN and use brandable_css" do
def check_asset(tag, asset_path, skip_rev = false)
unless skip_rev
asset_path = Canvas::Cdn::RevManifest.url_for(asset_path)
asset_path = Canvas::Cdn.registry.url_for(asset_path)
expect(asset_path).to be_present
end
attribute = (tag == "link") ? "href" : "src"
@ -90,12 +90,10 @@ describe "Stuff related to how we load stuff from CDN and use brandable_css" do
["bundles/common", "bundles/login"].each { |bundle| check_css(bundle) }
["images/favicon-yellow.ico", "images/apple-touch-icon.png"].each { |i| check_asset("link", i) }
optimized_js_flag = ENV["USE_OPTIMIZED_JS"] == "true" || ENV["USE_OPTIMIZED_JS"] == "True"
js_base_url = optimized_js_flag ? "/dist/webpack-production" : "/dist/webpack-dev"
check_asset("script", "/timezone/Etc/UTC.js")
check_asset("script", "/timezone/en_US.js")
Canvas::Cdn::RevManifest.all_webpack_chunks_for("main").each { |c| check_asset("script", "#{js_base_url}/#{c}", true) }
Canvas::Cdn::RevManifest.all_webpack_chunks_for("login").each { |c| check_asset("link", "#{js_base_url}/#{c}", true) }
Canvas::Cdn.registry.scripts_for("main").each { |c| check_asset("script", c, true) }
Canvas::Cdn.registry.scripts_for("login").each { |c| check_asset("link", c, true) }
end
end

37
spec/support/cdn_registry_stubs.rb Normal file
View File

@ -0,0 +1,37 @@
# frozen_string_literal: true
#
# Copyright (C) 2018 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
RSpec.shared_context "cdn registry stubs" do
before do
allow(::Canvas::Cdn).to receive(:registry).and_return(
::Canvas::Cdn::Registry.new(
cache: ::Canvas::Cdn::Registry::StaticCache.new(
gulp: {
"fonts/lato/extended/Lato-Regular.woff2" => "mock_revved_url",
"images/apple-touch-icon.png" => "images/apple-touch-icon-1234.png"
},
webpack: {
"main" => ["main-1234.js"]
}
)
)
)
end
end