ylqjgm
/
canvas-lms
mirror of https://github.com/instructure/canvas-lms.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

keep lockfiles in sync as part of `bundle` commands 

closes AE-283

this eliminates script/sync_lockfiles.rb and integrates its
functionality directly into `bundle install`, `bundle check`, etc.
it also generalizes a few pieces so that the same approach is used
for all use cases:
 * syncing versions between the main Gemfile and gems in gems/
 * maintaining separate lockfiles for no plugins/including
   private plugins
 * maintaining separate lockfiles for multiple Rails versions
   (crossed with the previous bullet)

The differences between them are just small variations on how strict
versions must match between lockfiles, and requiring pinning of
versions not in the default lockfile.

For full details, checks the docs on BundlerLockfileExtensions

This does change the strategy for filtering private plugin dependencies
out of the committed lockfile(s) - instead of filtering based on hash
of source, simply don't even include private plugin gems in the gemfile
when building the filtered lockfile (i.e. dynamic Gemfile, rather than
monkeypatching bundler to filter out -- semi-succesfully -- private
plugins from the Definition).

It also changes the "default" lockfile for Canvas that gets checked
in to be Gemfile.lock, so that other tools that are not
multi-lockfile aware can find it (such as rubocop, dependabot, and
others). This will be the lockfile corresponding to the current
default rails version for Canvas, and without private plugins.

Change-Id: I7ba398381974acbc4445f34fa3b788e8a07c5ce6
Reviewed-on: https://gerrit.instructure.com/c/canvas-lms/+/317888
Tested-by: Service Cloud Jenkins <svc.cloudjenkins@instructure.com>
Reviewed-by: Jacob Burroughs <jburroughs@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
Build-Review: Cody Cutrer <cody@instructure.com>
This commit is contained in:
Cody Cutrer 2023-05-10 12:12:39 -06:00
parent 53012299d0
commit 1c15214a63
37 changed files with 1088 additions and 627 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.dockerignore
View File

@ -36,5 +36,4 @@ vendor/bundle/
vendor/*/.git
Dockerfile
*.Dockerfile
Gemfile.lock
mkmf.log

3
.gitignore vendored
View File

@ -30,8 +30,7 @@
/db/*sql
docker-compose.override.yml
/exports/
/Gemfile.lock
/Gemfile.*.lock
/Gemfile.*.plugins.lock
/log/*
!/log/.keep
!/log/parallel-runtime-rspec.log

3
Dockerfile.jenkins-cache
View File

@ -10,9 +10,10 @@ RUN --mount=target=/tmp/src \
tar --sort=name --mtime='1970-01-01' --owner=0 --group=0 --numeric-owner --mode='a+rwX' -cf - \
config/canvas_rails_switcher.rb \
Gemfile \
Gemfile*.lock.partial \
Gemfile*.lock \
Gemfile.d \
gems/**/Gemfile \
gems/**/Gemfile.lock \
gems/**/*.gemspec \
gems/**/gem_version.rb \
gems/**/version.rb \

1
Dockerfile.jenkins.ruby-runner
View File

@ -12,4 +12,5 @@ RUN set -eux; \
/home/docker/.bundle \
# TODO: --without development \
&& { bundle install --jobs $(nproc) || bundle install; } \
&& bundle config --global frozen true \
&& rm -rf $GEM_HOME/cache

82
Gemfile
View File

@ -8,12 +8,6 @@
# list of gems for development and debuggery, without affecting our ability to
# merge with canvas-lms
#
# NOTE: some files in Gemfile.d/ will have certain required gems indented.
# While this may seem arbitrary, it actually has semantic significance. An
# indented gem required in Gemfile is a gem that is NOT directly used by
# Canvas, but required by a gem that is used by Canvas. We lock into specific
# versions of these gems to prevent regression, and the indentation serves to
# alert us to the relationship between the gem and canvas-lms
source "https://rubygems.org/"
@ -21,47 +15,48 @@ plugin "bundler_lockfile_extensions", path: "gems/bundler_lockfile_extensions"
require File.expand_path("config/canvas_rails_switcher", __dir__)
# Bundler evaluates this from a non-global context for plugins, so we have
# to explicitly pop up to set global constants
# rubocop:disable Style/RedundantConstantBase
# will already be defined during the second Gemfile evaluation
::CANVAS_INCLUDE_PLUGINS = true unless defined?(::CANVAS_INCLUDE_PLUGINS)
if Plugin.installed?("bundler_lockfile_extensions")
Plugin.send(:load_plugin, "bundler_lockfile_extensions") unless defined?(BundlerLockfileExtensions)
# Specifically exclude private plugins + private sources so that we can share a Gemfile.lock
# with OSS users without needing to encrypt / put it in a different repo. In order to actually
# pin any plugin-specific dependencies, the following constraints are introduced:
#
# 1. All dependencies under a private source must be pinned in the private plugin gemspec
# 2. All sub-dependencies of (1) must be pinned in plugins.rb
# 3. All additional public dependencies of private plugins must be pinned in plugins.rb
#
install_filter = lambda do |_lockfile, source|
return false if
source.to_s.match?(%r{plugins/(?!academic_benchmark|account_reports|moodle_importer|qti_exporter|respondus_soap_endpoint|simply_versioned)})
unless BundlerLockfileExtensions.enabled?
default = true
SUPPORTED_RAILS_VERSIONS.product([nil, true]).each do |rails_version, include_plugins|
prepare = lambda do
Object.send(:remove_const, :CANVAS_RAILS)
::CANVAS_RAILS = rails_version
Object.send(:remove_const, :CANVAS_INCLUDE_PLUGINS)
::CANVAS_INCLUDE_PLUGINS = include_plugins
end
source_md5 = ::Digest::MD5.hexdigest(source.to_s) # rubocop:disable Style/RedundantConstantBase
lockfile = ["Gemfile", "rails#{rails_version.delete(".")}", include_plugins && "plugins", "lock"].compact.join(".")
lockfile = nil if default
# only the first lockfile is the default
default = false
return false if
source_md5 == "52288aac483aed012b58e6707e1660a5" || # rubygems repository <redacted>
source_md5 == "252f6aa6a56f69f01f8a19275e91f0d8" # rubygems repository <redacted> or installed locally
current = rails_version == CANVAS_RAILS && include_plugins
true
end
base_gemfile = ENV.fetch("BUNDLE_GEMFILE", "Gemfile")
lockfile_defs = SUPPORTED_VERSIONS.to_h do |x|
prepare_environment = lambda do
Object.send(:remove_const, :CANVAS_RAILS)
::CANVAS_RAILS = x # rubocop:disable Style/RedundantConstantBase
add_lockfile(lockfile,
current: current,
prepare: prepare,
allow_mismatched_dependencies: rails_version != SUPPORTED_RAILS_VERSIONS.first,
enforce_pinned_additional_dependencies: include_plugins)
end
["#{base_gemfile}.rails#{x.delete(".")}.lock",
{
default: x == CANVAS_RAILS,
install_filter: install_filter,
prepare_environment: prepare_environment,
}]
Dir["Gemfile.d/*.lock", "gems/*/Gemfile.lock"].each do |gem_lockfile_name|
return unless add_lockfile(gem_lockfile_name,
gemfile: gem_lockfile_name.sub(/\.lock$/, ""),
allow_mismatched_dependencies: false)
end
end
BundlerLockfileExtensions.enable(lockfile_defs)
end
# rubocop:enable Style/RedundantConstantBase
# Bundler's first pass parses the entire Gemfile and calls to additional sources
# makes it actually go and retrieve metadata from them even though the plugin will
@ -71,6 +66,11 @@ return if method(:source).owner == Bundler::Plugin::DSL
module GemOverride
def gem(name, *version, path: nil, **kwargs)
# Bundler calls `gem` internally by passing a splat with a hash as the
# last argument, instead of properly using kwargs. Detect that.
if version.last.is_a?(Hash) && kwargs.empty?
kwargs = version.pop
end
if File.directory?("vendor/#{name}")
super(name, path: "vendor/#{name}", **kwargs)
else
@ -80,10 +80,12 @@ module GemOverride
end
Bundler::Dsl.prepend(GemOverride)
Dir[File.join(File.dirname(__FILE__), "gems/plugins/*/Gemfile.d/_before.rb")].each do |file|
eval(File.read(file), nil, file) # rubocop:disable Security/Eval
if CANVAS_INCLUDE_PLUGINS
Dir[File.join(File.dirname(__FILE__), "gems/plugins/*/Gemfile.d/_before.rb")].each do |file|
eval_gemfile(file)
end
end
Dir.glob(File.join(File.dirname(__FILE__), "Gemfile.d", "*.rb")).sort.each do |file|
eval(File.read(file), nil, file) # rubocop:disable Security/Eval
eval_gemfile(file)
end

2
Gemfile.d/app.rb
View File

@ -142,7 +142,7 @@ gem "will_paginate", "3.3.0", require: false # required for folio-pagination
gem "faraday", "0.17.4"
path "gems" do
path "../gems" do
gem "activesupport-suspend_callbacks"
gem "acts_as_list"
gem "adheres_to_policy"

2
Gemfile.d/cassandra.rb
View File

@ -27,5 +27,5 @@ group :cassandra do
require: false,
github: "twitter/thrift_client",
ref: "5c10d59881825cb8e26ab1aa8f1d2738e88c0e83"
gem "canvas_cassandra", path: "gems/canvas_cassandra"
gem "canvas_cassandra", path: "../gems/canvas_cassandra"
end

4
Gemfile.d/i18n_tools_and_rake_tasks.rb
View File

@ -18,9 +18,9 @@
# with this program. If not, see <http://www.gnu.org/licenses/>.
group :i18n_tools do
gem "i18n_extraction", path: "gems/i18n_extraction", require: false
gem "i18n_extraction", path: "../gems/i18n_extraction", require: false
end
group :i18n_tools, :development do
gem "i18n_tasks", path: "gems/i18n_tasks"
gem "i18n_tasks", path: "../gems/i18n_tasks"
end

33
Gemfile.d/plugins.rb
View File

@ -17,27 +17,20 @@
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
CANVAS_INLINE_PLUGINS = %w[
academic_benchmark
account_reports
moodle_importer
qti_exporter
respondus_soap_endpoint
simply_versioned
].freeze
Dir[File.join(File.dirname(__FILE__), "../gems/plugins/*")].each do |plugin_dir|
next unless File.directory?(plugin_dir)
gem(File.basename(plugin_dir), path: plugin_dir)
gem_name = File.basename(plugin_dir)
next unless CANVAS_INCLUDE_PLUGINS || CANVAS_INLINE_PLUGINS.include?(gem_name)
gem(gem_name, path: plugin_dir)
end
# Private Plugin Alignment
gem "activeresource", "6.0.0"
gem "colorize", "0.8.1", require: false
gem "crypt", ">= 2.2.0"
gem "dynect4r", "0.2.4"
gem "maxminddb", "0.1.22"
gem "mechanize", "2.7.7"
gem "restforce", "5.0.3"
gem "sshkey", "2.0.0"
gem "xml-simple", "1.1.5"
gem "zendesk_api", "1.28.0"
group :test do
gem "vcr", "6.1.0"
end
# Private Dependency Sub-Dependencies
gem "typhoeus", "~> 1.3"

7
Gemfile.d/rubocop.rb
View File

@ -26,12 +26,7 @@ group :test do
gem "gergich", "2.1.1", require: false
gem "mime-types-data", "~> 3.2023", require: false
rubocop_canvas_path = "gems/rubocop-canvas"
if File.dirname(@gemfile) == __dir__
rubocop_canvas_path = "../#{rubocop_canvas_path}"
end
gem "rubocop-canvas", require: false, path: rubocop_canvas_path
gem "rubocop-canvas", require: false, path: "../gems/rubocop-canvas"
gem "rubocop-inst", "~> 1", require: false
gem "rubocop-graphql", "1.1.1", require: false
gem "rubocop-rails", "2.19.1", require: false

9
Gemfile.d/~after.rb
View File

@ -19,8 +19,11 @@
# Non-standard Canvas extension to Bundler behavior -- load the Gemfiles from
# plugins.
Dir[File.join(File.dirname(__FILE__), "../gems/plugins/*/Gemfile.d/*")].each do |g|
next if g.end_with?("/_before.rb")
eval(File.read(g), nil, g) # rubocop:disable Security/Eval
if CANVAS_INCLUDE_PLUGINS
Dir[File.join(File.dirname(__FILE__), "../gems/plugins/*/Gemfile.d/*")].each do |g|
next if g.end_with?("/_before.rb")
eval_gemfile(g)
end
end

61
Gemfile.rails70.lock.partial → Gemfile.lock
View File

@ -374,20 +374,12 @@ GEM
globalid (>= 0.3.6)
activemodel (7.0.4.3)
activesupport (= 7.0.4.3)
activemodel-serializers-xml (1.0.2)
activemodel (> 5.x)
activesupport (> 5.x)
builder (~> 3.1)
activerecord (7.0.4.3)
activemodel (= 7.0.4.3)
activesupport (= 7.0.4.3)
activerecord-pg-extensions (0.4.4)
activerecord (>= 6.0, < 7.1)
railties (>= 6.0, < 7.1)
activeresource (6.0.0)
activemodel (>= 6.0)
activemodel-serializers-xml (~> 1.0)
activesupport (>= 6.0)
activestorage (7.0.4.3)
actionpack (= 7.0.4.3)
activejob (= 7.0.4.3)
@ -512,15 +504,12 @@ GEM
coercible (1.0.0)
descendants_tracker (~> 0.0.1)
colored (1.2)
colorize (0.8.1)
concurrent-ruby (1.2.2)
connection_pool (2.3.0)
crack (0.4.5)
rexml
crass (1.0.6)
crocodoc-ruby (0.0.1)
json
crypt (2.2.1)
crystalball (0.7.0)
git
database_cleaner (2.0.2)
@ -559,9 +548,6 @@ GEM
pygments.rb
redcarpet
dumb_delegator (1.0.0)
dynect4r (0.2.4)
json
rest-client
ecma-re-validator (0.4.0)
regexp_parser (~> 2.2)
encrypted_cookie_store-instructure (1.2.12)
@ -570,8 +556,6 @@ GEM
escape_code (0.2)
et-orbi (1.2.7)
tzinfo
ethon (0.12.0)
ffi (>= 1.3.0)
expgen (0.1.1)
parslet
extlib (0.9.16)
@ -654,7 +638,6 @@ GEM
json-jwt (~> 1.7)
rexml
simple_oauth (~> 0.3.1)
inflection (1.0.0)
inst-jobs (3.1.10)
activerecord (>= 6.0)
activerecord-pg-extensions (~> 0.4.4)
@ -713,17 +696,6 @@ GEM
actionpack (>= 5.2)
activerecord (>= 5.2)
matrix (0.4.2)
maxminddb (0.1.22)
mechanize (2.7.7)
domain_name (~> 0.5, >= 0.5.1)
http-cookie (~> 1.0)
mime-types (>= 1.17.2)
net-http-digest_auth (~> 1.1, >= 1.1.1)
net-http-persistent (>= 2.5.2)
nokogiri (~> 1.6)
ntlm-http (~> 0.1, >= 0.1.1)
webrick (~> 1.7)
webrobots (>= 0.0.9, < 0.2)
method_source (1.0.0)
mime-types (3.3.1)
mime-types-data (~> 3.2015)
@ -743,9 +715,6 @@ GEM
multi_xml (0.6.0)
multipart-post (2.1.1)
mustache (1.1.1)
net-http-digest_auth (1.4.1)
net-http-persistent (4.0.1)
connection_pool (~> 2.2)
net-imap (0.2.3)
digest
net-protocol
@ -773,7 +742,6 @@ GEM
racc (~> 1.4)
nokogiri-xmlsec-instructure (0.10.2)
nokogiri (>= 1.11.2)
ntlm-http (0.1.1)
oauth (1.1.0)
oauth-tty (~> 1.0, >= 1.0.1)
snaky_hash (~> 2.0)
@ -925,11 +893,6 @@ GEM
http-cookie (>= 1.0.2, < 2.0)
mime-types (>= 1.16, < 4.0)
netrc (~> 0.8)
restforce (5.0.3)
faraday (>= 0.9.0, <= 2.0)
faraday_middleware (>= 0.8.8, <= 2.0)
hashie (>= 1.2.0, < 5.0)
jwt (>= 1.5.6)
retriable (1.4.1)
rexml (3.2.5)
ritex (1.0.1)
@ -1076,7 +1039,6 @@ GEM
sqlite3 (1.6.2-arm64-darwin)
sqlite3 (1.6.2-x86_64-darwin)
sqlite3 (1.6.2-x86_64-linux)
sshkey (2.0.0)
stackprof (0.2.24)
statsd-ruby (1.4.0)
stormbreaker (0.0.8)
@ -1120,8 +1082,6 @@ GEM
twitter-text (3.1.0)
idn-ruby
unf (~> 0.1.0)
typhoeus (1.4.0)
ethon (>= 0.9.0)
tzinfo (2.0.4)
concurrent-ruby (~> 1.0)
unf (0.1.4)
@ -1134,7 +1094,6 @@ GEM
uri_template (0.7.0)
vault (0.15.0)
aws-sigv4
vcr (6.1.0)
vericite_api (1.5.3)
json (>= 1.4.6)
version_gem (1.1.2)
@ -1155,12 +1114,10 @@ GEM
crack (>= 0.3.2)
hashdiff (>= 0.4.0, < 2.0.0)
webrick (1.8.1)
webrobots (0.1.2)
websocket-driver (0.7.5)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)
will_paginate (3.3.0)
xml-simple (1.1.5)
xpath (3.2.0)
nokogiri (~> 1.8)
ya2yaml (0.31)
@ -1168,12 +1125,6 @@ GEM
yard-appendix (0.1.8)
yard (>= 0.8.0)
zeitwerk (2.6.8)
zendesk_api (1.28.0)
faraday (>= 0.9.0, < 2.0.0)
hashie (>= 3.5.2, < 5.0.0)
inflection
mime-types
multipart-post (~> 2.0)
PLATFORMS
aarch64-linux
@ -1189,7 +1140,6 @@ DEPENDENCIES
active_model_serializers (= 0.9.0alpha1)!
active_record_query_trace (= 1.8)
activerecord-pg-extensions (= 0.4.4)
activeresource (= 6.0.0)
activesupport-suspend_callbacks!
acts_as_list!
addressable (~> 2.8)
@ -1247,10 +1197,8 @@ DEPENDENCIES
canvas_webex (= 0.18.2)
cassandra-cql (= 1.2.3)!
chunky_png (= 1.4.0)
colorize (= 0.8.1)
config_file!
crocodoc-ruby (= 0.0.1)
crypt (>= 2.2.0)
crystalball (= 0.7.0)
csv_diff!
database_cleaner (~> 2.0)
@ -1263,7 +1211,6 @@ DEPENDENCIES
dotenv (~> 2.8)
dress_code (= 1.2.1)
dynamic_settings!
dynect4r (= 0.2.4)
encrypted_cookie_store-instructure (= 1.2.12)
escape_code (= 0.2)
event_stream!
@ -1313,8 +1260,6 @@ DEPENDENCIES
mail (= 2.7.1)
marginalia (= 1.11.1)
matrix (= 0.4.2)
maxminddb (= 0.1.22)
mechanize (= 2.7.7)
mime-types (= 3.3.1)
mime-types-data (~> 3.2023)
mini_magick (= 4.11.0)
@ -1360,7 +1305,6 @@ DEPENDENCIES
regexp_parser (= 2.7.0)
request_context!
respondus_soap_endpoint!
restforce (= 5.0.3)
retriable (= 1.4.1)
ritex (= 1.0.1)
rotp (= 6.2.0)
@ -1401,7 +1345,6 @@ DEPENDENCIES
spring-commands-parallel-rspec (= 1.1.0)
spring-commands-rspec (= 1.0.4)
spring-commands-rubocop (= 0.3.0)!
sshkey (= 2.0.0)
stackprof (~> 0.2)
stormbreaker (= 0.0.8)
stringify_ids!
@ -1413,11 +1356,9 @@ DEPENDENCIES
turnitin_api!
twilio-ruby (= 5.36.0)
twitter!
typhoeus (~> 1.3)
tzinfo (= 2.0.4)
utf8_cleaner!
vault (= 0.15.0)
vcr (= 6.1.0)
vericite_api (= 1.5.3)
wcag_color_contrast (= 0.1.0)
webdrivers (= 5.2.0)
@ -1425,10 +1366,8 @@ DEPENDENCIES
week_of_month (= 1.2.5)!
will_paginate (= 3.3.0)
workflow!
xml-simple (= 1.1.5)
yard (~> 0.9)
yard-appendix (= 0.1.8)
zendesk_api (= 1.28.0)
RUBY VERSION
ruby 2.7.5p203

27
bin/brakeman Executable file
View File

@ -0,0 +1,27 @@
#!/usr/bin/env ruby
# frozen_string_literal: true
#
# This file was generated by Bundler.
#
# The application 'brakeman' is installed as part of a gem, and
# this file is here to facilitate running it.
#
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__)
bundle_binstub = File.expand_path("bundle", __dir__)
if File.file?(bundle_binstub)
if File.read(bundle_binstub, 300).include?("This file was generated by Bundler")
load(bundle_binstub)
else
abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run.
Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.")
end
end
require "rubygems"
require "bundler/setup"
load Gem.bin_path("brakeman", "brakeman")

52
build/new-jenkins/linters/run-gergich-bundle.sh
View File

@ -2,57 +2,19 @@
set -ex
# Check that the partial lockfile is as expected with all private plugins installed.
# If this step fails - you probably just didn't commit Gemfile.lock or committed a bad
# version somehow.
# Check that lockfiles haven't changed. If they did, you probably forgot to run
# `bundle install` or to commit the changed lockfiles.
bundle config --global unset frozen
bundle install
for f in Gemfile.rails*.lock.partial; do
if ! git diff --exit-code $f; then
export SKIP_OSS_CHECK=1
git --no-pager diff $f
diff="$(git diff $f)"
diff="$(git diff 'Gemfile*.lock')"
if [ -n "$diff" ]; then
diff="\n\n\`\`\`\n$diff\n\`\`\`"
diff=${diff//$'\n'/'\n'}
diff=${diff//$'"'/'\"'}
message="$f changes were detected when private plugins are installed. Make sure you run 'bundle install'."
gergich comment "{\"path\":\"$f\",\"position\":1,\"severity\":\"error\",\"message\":\"$message$diff\"}"
fi
done
# If this is a plugin build and the change would require Gemfile.lock, the above
# check would catch the issue and the corresponding canvas-lms build would catch
# OSS issues.
if [[ "$SKIP_OSS_CHECK" != "1" && "$GERRIT_PROJECT" == "canvas-lms" ]]; then
read -r -a PLUGINS_LIST_ARR <<< "$PLUGINS_LIST"
rm -rf $(printf 'gems/plugins/%s ' "${PLUGINS_LIST_ARR[@]}")
# Check that the partial lockfile is as expected with no private plugins installed.
# If this step fails - it's likely that one of the constraints is being violated:
#
# 1. All dependencies under a private source must be pinned in the private plugin gemspec
# 2. All sub-dependencies of (1) must be pinned in plugins.rb
# 3. All additional public dependencies of private plugins must be pinned in plugins.rb
bundle install
for f in Gemfile.rails*.lock.partial; do
if ! git diff --exit-code $f; then
git --no-pager diff $f
diff="$(git diff $f)"
diff="\n\n\`\`\`\n$diff\n\`\`\`"
diff=${diff//$'\n'/'\n'}
diff=${diff//$'"'/'\"'}
message="$f changes were detected when private plugins are not installed. Make sure you adhere to the version pin constraints."
gergich comment "{\"path\":\"$f\",\"position\":1,\"severity\":\"error\",\"message\":\"$message$diff\"}"
fi
done
else
echo "skipping OSS check due to previous failure"
message="Lockfile changes were detected. Make sure you run 'bundle install'."
gergich comment "{\"path\":\"/COMMIT_MSG\",\"position\":1,\"severity\":\"error\",\"message\":\"$message$diff\"}"
fi
gergich status

4
build/new-jenkins/linters/run-gergich-linters.sh
View File

@ -47,10 +47,6 @@ ruby script/stylelint
ruby script/rlint --no-fail-on-offense
[ "${SKIP_ESLINT-}" != "true" ] && ruby script/eslint
ruby script/lint_commit_message
if ! ruby script/sync_lockfiles.rb; then
message="The lockfiles for all sub-gems are not in sync. Please run script/sync_lockfiles.rb for details.\\n"
gergich comment "{\"path\":\"/COMMIT_MSG\",\"position\":1,\"severity\":\"error\",\"message\":\"$message\"}"
fi
node script/yarn-validate-workspace-deps.js 2>/dev/null < <(yarn --silent workspaces info --json)
node ui-build/tools/component-info.mjs -i -v -g

2
config/application.rb
View File

@ -209,7 +209,7 @@ module CanvasRails
super
else
# Any is eager, so we must map first or we won't run on all keys
SUPPORTED_VERSIONS.map do |version|
SUPPORTED_RAILS_VERSIONS.map do |version|
super(key, (options || {}).merge(explicit_version: version.delete(".")))
end.any?
end

21
config/brakeman.ignore
View File

@ -975,6 +975,25 @@
],
"note": ""
},
{
"warning_type": "Unmaintained Dependency",
"warning_code": 121,
"fingerprint": "9a3951031616a07c8e02c86652f537e92c08685da97f5ec2b12d5d3602b55bb8",
"check_name": "EOLRuby",
"message": "Support for Ruby 2.7.5 ended on 2023-03-31",
"file": "Gemfile.lock",
"line": 1373,
"link": "https://brakemanscanner.org/docs/warning_types/unmaintained_dependency/",
"code": null,
"render_path": null,
"location": null,
"user_input": null,
"confidence": "High",
"cwe_id": [
1104
],
"note": ""
},
{
"warning_type": "SQL Injection",
"warning_code": 0,
@ -1771,6 +1790,6 @@
"note": ""
}
],
"updated": "2023-04-10 13:22:04 -0500",
"updated": "2023-05-12 16:03:58 -0600",
"brakeman_version": "5.4.1"
}

10
config/canvas_rails_switcher.rb
View File

@ -26,8 +26,8 @@
# 2. Create a file RAILS_VERSION with <supported version> as the contents
# 3. Create a Consul setting private/canvas/rails_version with <supported version> as the contents
DEFAULT_VERSION = "7.0"
SUPPORTED_VERSIONS = %w[7.0].freeze
# the default version (corresponding to the bare Gemfile.lock) must be listed first
SUPPORTED_RAILS_VERSIONS = %w[7.0].freeze
unless defined?(CANVAS_RAILS)
file_path = File.expand_path("RAILS_VERSION", __dir__)
@ -61,13 +61,13 @@ unless defined?(CANVAS_RAILS)
result = nil unless result.is_a?(Net::HTTPSuccess)
break if result
end
CANVAS_RAILS = result ? Base64.decode64(JSON.parse(result.body).first["Value"]).strip : DEFAULT_VERSION
CANVAS_RAILS = result ? Base64.decode64(JSON.parse(result.body).first["Value"]).strip : SUPPORTED_RAILS_VERSIONS.first
rescue
CANVAS_RAILS = DEFAULT_VERSION
CANVAS_RAILS = SUPPORTED_RAILS_VERSIONS.first
end
end
end
unless SUPPORTED_VERSIONS.any?(CANVAS_RAILS)
unless SUPPORTED_RAILS_VERSIONS.any?(CANVAS_RAILS)
raise "unsupported Rails version specified #{CANVAS_RAILS}"
end

2
gems/bundler_lockfile_extensions/Gemfile.lock
View File

@ -6,6 +6,7 @@ PATH
GEM
remote: https://rubygems.org/
specs:
byebug (11.1.3)
diff-lcs (1.5.0)
rspec (3.12.0)
rspec-core (~> 3.12.0)
@ -29,6 +30,7 @@ PLATFORMS
DEPENDENCIES
bundler_lockfile_extensions!
byebug
rspec (~> 3.12)
BUNDLED WITH

1
gems/bundler_lockfile_extensions/bundler_lockfile_extensions.gemspec
View File

@ -9,5 +9,6 @@ Gem::Specification.new do |spec|
spec.files = Dir.glob("{lib,spec}/**/*") + %w[plugins.rb]
spec.require_paths = ["lib"]
spec.add_development_dependency "byebug"
spec.add_development_dependency "rspec", "~> 3.12"
end

440
gems/bundler_lockfile_extensions/lib/bundler_lockfile_extensions.rb
View File

@ -18,187 +18,315 @@
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
module BundlerDefinitionFilterableSources
def dependencies
return super unless BundlerLockfileExtensions.lockfile_writes_enabled && BundlerLockfileExtensions.lockfile_filter
super.filter { |x| source_included?(x.instance_variable_get(:@source)) }
end
def sources
return super unless BundlerLockfileExtensions.lockfile_writes_enabled && BundlerLockfileExtensions.lockfile_filter
res = super.clone
res.instance_variable_get(:@path_sources).filter! { |x| source_included?(x) }
res.instance_variable_get(:@rubygems_sources).filter! { |x| source_included?(x) }
res
end
def lock(...)
return unless BundlerLockfileExtensions.lockfile_writes_enabled
super(...)
end
def nothing_changed?
locked_specs = instance_variable_get(:@locked_specs).to_hash.keys
actual_specs = converge_locked_specs.to_hash.keys
super && (locked_specs - actual_specs).empty?
end
def ensure_filtered_dependencies_pinned
return unless BundlerLockfileExtensions.lockfile_filter
check_dependencies = []
@sources.instance_variable_get(:@rubygems_sources).each do |x|
next if source_included?(x)
specs = resolve.select { |s| x.can_lock?(s) }
specs.each do |s|
check_dependencies << s.name
end
end
proven_pinned = check_dependencies.to_h { |x| [x, false] } # rubocop:disable Rails/IndexWith
valid_sources = []
valid_sources.push(*@sources.instance_variable_get(:@path_sources))
valid_sources.push(*@sources.instance_variable_get(:@rubygems_sources))
valid_sources.each do |x|
next if source_included?(x)
specs = resolve.select { |s| x.can_lock?(s) }
specs.each do |s|
s.dependencies.each do |d|
next unless proven_pinned.key?(d.name)
d.requirement.requirements.each do |r|
proven_pinned[d.name] = true if r[0] == "="
end
end
end
end
proven_pinned.each do |k, v|
raise BundlerLockfileExtensions::Error, "unable to prove that private gem #{k} was pinned - make sure it is pinned to only one resolveable version in the gemspec" unless v
end
end
private
def source_included?(source)
BundlerLockfileExtensions.lockfile_filter.call(BundlerLockfileExtensions.lockfile_path, source)
end
end
require "bundler_lockfile_extensions/bundler"
require "bundler_lockfile_extensions/bundler/definition"
require "bundler_lockfile_extensions/bundler/dsl"
require "bundler_lockfile_extensions/bundler/source_list"
# Extends Bundler to allow arbitrarily many lockfiles (and Gemfiles!)
# for variations of the Gemfile, while keeping all of the lockfiles in sync.
#
# `bundle install`, `bundle lock`, and `bundle update` will operate only on
# the default lockfile (Gemfile.lock), afterwhich all other lockfiles will
# be re-created based on this default lockfile. Additional lockfiles can be
# based on the same Gemfile, but vary at runtime based on something like an
# environment variable, global variable, or constant. When defining such a
# lockfile, you should use a prepare callback that sets up the proper
# environment for that variation, even if that's not what would otherwise
# be selected by the launching environment.
#
# Alternately (or in addition!), you can define a lockfile to use a completely
# different Gemfile. This will have the effect that common dependencies between
# the two Gemfiles will stay locked to the same version in each lockfile.
#
# A lockfile definition can opt in to requiring explicit pinning for
# any dependency that exists in that variation, but does not exist in the default
# lockfile. This is especially useful if for some reason a given
# lockfile will _not_ be committed to version control (such as a variation
# that will include private plugins).
#
# Finally, `bundle check` will enforce additional checks to compare the final
# locked versions of dependencies between the various lockfiles to ensure
# they end up the same. This check might be tripped if Gemfile variations
# (accidentally!) have conflicting version constraints on a dependency, that
# are still self-consistent with that single Gemfile variation.
# `bundle install`, `bundle lock`, and `bundle update` will also verify these
# additional checks. You can additionally explicitly allow version variations
# between explicit dependencies (and their sub-dependencies), for cases where
# the lockfile variation is specifically to transition to a new version of
# a dependency (like a Rails upgrade).
#
module BundlerLockfileExtensions
class Error < Bundler::BundlerError; status_code(99); end
class << self
attr_accessor :lockfile_default, :lockfile_defs, :lockfile_filter, :lockfile_path, :lockfile_writes_enabled
attr_reader :lockfile_definitions
def enabled?
!!@lockfile_defs
@lockfile_definitions
end
def enable(lockfile_defs)
@lockfile_default = lockfile_defs.find { |x| !!x[1][:default] }[0]
@lockfile_defs = lockfile_defs
# @param lockfile [String] The lockfile path
# @param Builder [::Bundler::DSL] The Bundler DSL
# @param gemfile [String, nil]
# The Gemfile for this lockfile (defaults to Gemfile)
# @param current [true, false] If this is the currently active combination
# @param prepare [Proc, nil]
# The callback to set up the environment so your Gemfile knows this is
# the intended lockfile, and to select dependencies appropriately.
# @param allow_mismatched_dependencies [true, false]
# Allows version differences in dependencies between this lockfile and
# the default lockfile. Note that even with this option, only top-level
# dependencies that differ from the default lockfile, and their transitive
# depedencies, are allowed to mismatch.
# @param enforce_pinned_additional_dependencies [true, false]
# If dependencies are present in this lockfile that are not present in the
# default lockfile, enforce that they are pinned.
def add_lockfile(lockfile = nil,
builder:,
gemfile: nil,
current: false,
prepare: nil,
allow_mismatched_dependencies: true,
enforce_pinned_additional_dependencies: false)
enable unless enabled?
@lockfile_path =
if defined?(Bundler::CLI::Cache) || defined?(Bundler::CLI::Lock)
@lockfile_writes_enabled = true
lockfile_default.to_s
elsif (!Bundler.settings[:deployment] && defined?(Bundler::CLI::Install)) || defined?(Bundler::CLI::Update)
# Sadly, this is the only place where the lockfile_path can be set correctly for the installation-like paths.
# Ideally, it would go into before-install-all, but that is called after the lockfile is already loaded.
install_lockfile_name(lockfile_default)
else
lockfile_default.to_s
end
Bundler::Dsl.class_eval do
def to_definition(_lockfile, unlock)
@sources << @rubygems_source if @sources.respond_to?(:include?) && !@sources.include?(@rubygems_source)
Bundler::Definition.new(Bundler.default_lockfile, @dependencies, @sources, unlock, @ruby_version)
end
default = gemfile.nil? && lockfile.nil?
if default && default_lockfile_definition
raise ArgumentError, "Only one default lockfile (gemfile and lockfile unspecified) is allowed"
end
if current && @lockfile_definitions.any? { |definition| definition[:current] }
raise ArgumentError, "Only one lockfile can be flagged as the current lockfile"
end
Bundler::SharedHelpers.class_eval do
class << self
def default_lockfile
Pathname.new(BundlerLockfileExtensions.lockfile_path).expand_path
@lockfile_definitions << (lockfile_def = {
gemfile: (gemfile && Pathname.new(gemfile).expand_path) || ::Bundler.default_gemfile,
lockfile: (lockfile && Pathname.new(lockfile).expand_path) || ::Bundler.default_lockfile,
default: default,
current: current,
prepare: prepare,
allow_mismatched_dependencies: allow_mismatched_dependencies,
enforce_pinned_additional_dependencies: enforce_pinned_additional_dependencies
}.freeze)
# always use the default lockfile for `bundle check`, `bundle install`,
# `bundle lock`, and `bundle update`. `bundle cache` delegates to
# `bundle install`, but we want that to run as-normal.
set_lockfile = if (defined?(::Bundler::CLI::Check) ||
defined?(::Bundler::CLI::Install) ||
defined?(::Bundler::CLI::Lock) ||
defined?(::Bundler::CLI::Update)) &&
!defined?(::Bundler::CLI::Cache)
prepare&.call if default
default
else
current
end
# if BUNDLE_LOCKFILE is specified, explicitly use only that lockfile, regardless of the command
if ENV["BUNDLE_LOCKFILE"] && File.expand_path(ENV["BUNDLE_LOCKFILE"]) == lockfile_def[:lockfile].to_s
prepare&.call
set_lockfile = true
# we started evaluating the project's primary gemfile, but got told to use a lockfile
# associated with a different Gemfile. so we need to evaluate that Gemfile instead
if lockfile_def[:gemfile] != ::Bundler.default_gemfile
# share a cache between all lockfiles
::Bundler.cache_root = ::Bundler.root
ENV["BUNDLE_GEMFILE"] = lockfile_def[:gemfile].to_s
::Bundler.root = ::Bundler.default_gemfile.dirname
::Bundler.default_lockfile = lockfile_def[:lockfile]
builder.eval_gemfile(::Bundler.default_gemfile)
return false
end
end
::Bundler.default_lockfile = lockfile_def[:lockfile] if set_lockfile
true
end
# @!visibility private
def after_install_all(install: true)
previous_recursive = @recursive
return unless enabled?
return if ENV["BUNDLE_LOCKFILE"] # explicitly working against a single lockfile
# must be running `bundle cache`
return unless ::Bundler.default_lockfile == default_lockfile_definition[:lockfile]
if ::Bundler.frozen_bundle? && !install
# only do the checks if we're frozen
require "bundler_lockfile_extensions/check"
exit 1 unless Check.run
return
end
# this hook will be called recursively when it has to install gems
# for a secondary lockfile. defend against that
return if @recursive
@recursive = true
require "tempfile"
require "bundler_lockfile_extensions/lockfile_generator"
::Bundler.ui.info ""
default_lockfile_contents = ::Bundler.default_lockfile.read.freeze
default_specs = ::Bundler::LockfileParser.new(default_lockfile_contents).specs.to_h do |spec| # rubocop:disable Rails/IndexBy
[[spec.name, spec.platform], spec]
end
default_root = ::Bundler.root
::Bundler.settings.temporary(cache_all_platforms: true, suppress_install_using_messages: true) do
@lockfile_definitions.each do |lockfile_definition|
# we already wrote the default lockfile
next if lockfile_definition[:default]
# root needs to be set so that paths are output relative to the correct root in the lockfile
::Bundler.root = lockfile_definition[:gemfile].dirname
relative_lockfile = lockfile_definition[:lockfile].relative_path_from(Dir.pwd)
if ::Bundler.frozen_bundle?
# if we're frozen, you have to use the pre-existing lockfile
unless lockfile_definition[:lockfile].exist?
::Bundler.ui.error("The bundle is locked, but #{relative_lockfile} is missing. Please make sure you have checked #{relative_lockfile} into version control before deploying.")
exit 1
end
::Bundler.ui.info("Installing gems for #{relative_lockfile}...")
write_lockfile(lockfile_definition, lockfile_definition[:lockfile], install: install)
else
::Bundler.ui.info("Syncing to #{relative_lockfile}...")
# adjust locked paths from the default lockfile to be relative to _this_ gemfile
adjusted_default_lockfile_contents = default_lockfile_contents.gsub(/PATH\n remote: ([^\n]+)\n/) do |remote|
remote_path = Pathname.new($1)
next remote if remote_path.absolute?
relative_remote_path = remote_path.expand_path(default_root).relative_path_from(::Bundler.root).to_s
remote.sub($1, relative_remote_path)
end
# add a source for the current gem
gem_spec = default_specs[[File.basename(::Bundler.root), "ruby"]]
if gem_spec
adjusted_default_lockfile_contents += <<~TEXT
PATH
remote: .
specs:
#{gem_spec.to_lock}
TEXT
end
if lockfile_definition[:lockfile].exist?
# if the lockfile already exists, "merge" it together
default_lockfile = ::Bundler::LockfileParser.new(adjusted_default_lockfile_contents)
lockfile = ::Bundler::LockfileParser.new(lockfile_definition[:lockfile].read)
# replace any duplicate specs with what's in the default lockfile
lockfile.specs.map! do |spec|
default_specs[[spec.name, spec.platform]] || spec
end
lockfile.specs.replace(default_lockfile.specs + lockfile.specs).uniq!
lockfile.sources.replace(default_lockfile.sources + lockfile.sources).uniq!
lockfile.platforms.concat(default_lockfile.platforms).uniq!
lockfile.instance_variable_set(:@ruby_version, default_lockfile.ruby_version)
lockfile.instance_variable_set(:@bundler_version, default_lockfile.bundler_version)
new_contents = LockfileGenerator.generate(lockfile)
else
# no lockfile? just start out with the default lockfile's contents to inherit its
# locked gems
new_contents = adjusted_default_lockfile_contents
end
# Now build a definition based on the given Gemfile, with the combined lockfile
Tempfile.create do |temp_lockfile|
temp_lockfile.write(new_contents)
temp_lockfile.flush
write_lockfile(lockfile_definition, temp_lockfile.path, install: install)
end
end
end
end
Bundler::Definition.prepend(BundlerDefinitionFilterableSources)
require "bundler_lockfile_extensions/check"
@lockfile_defs[lockfile_default][:prepare_environment]&.call
exit 1 unless Check.run
ensure
@recursive = previous_recursive
end
def each_lockfile_for_writing(lock)
lock_def = @lockfile_defs[lock]
private
@lockfile_writes_enabled = true
def enable
@lockfile_definitions ||= []
@lockfile_path = lock.to_s
yield @lockfile_path
::Bundler.singleton_class.prepend(Bundler::ClassMethods)
::Bundler::Definition.prepend(Bundler::Definition)
::Bundler::SourceList.prepend(Bundler::SourceList)
end
if lock_def[:install_filter]
@lockfile_filter = lock_def[:install_filter]
@lockfile_path = install_filter_lockfile_name(lock).to_s
yield @lockfile_path
def default_lockfile_definition
@default_lockfile_definition ||= @lockfile_definitions.find { |d| d[:default] }
end
@lockfile_filter = nil
def write_lockfile(lockfile_definition, lockfile, install:)
lockfile_definition[:prepare]&.call
definition = ::Bundler::Definition.build(lockfile_definition[:gemfile], lockfile, false)
resolved_remotely = false
begin
previous_ui_level = ::Bundler.ui.level
::Bundler.ui.level = "warn"
begin
definition.resolve_with_cache!
rescue ::Bundler::GemNotFound
definition = ::Bundler::Definition.build(lockfile_definition[:gemfile], lockfile, false)
definition.resolve_remotely!
resolved_remotely = true
end
definition.lock(lockfile_definition[:lockfile], true)
ensure
::Bundler.ui.level = previous_ui_level
end
@lockfile_writes_enabled = false
end
def install_filter_lockfile_name(lock)
"#{lock}.partial"
end
def install_lockfile_name(lock)
if @lockfile_defs[lock][:install_filter]
install_filter_lockfile_name(lock)
else
lock.to_s
end
end
def write_all_lockfiles
current_definition = Bundler.definition
unlock = current_definition.instance_variable_get(:@unlock)
# Always prepare the default lockfiles first so that we don't re-resolve dependencies remotely
each_lockfile_for_writing(lockfile_default) do |x|
current_definition.ensure_filtered_dependencies_pinned
current_definition.lock(x)
end
lockfile_defs.each do |lock, opts|
next if lock == lockfile_default
@lockfile_path = install_lockfile_name(lock)
opts[:prepare_environment]&.call
definition = Bundler::Definition.build(Bundler.default_gemfile, @lockfile_path, unlock)
definition.resolve_remotely!
definition.specs
each_lockfile_for_writing(lock) do |x|
definition.ensure_filtered_dependencies_pinned
definition.lock(x)
# if we're running `bundle install` or `bundle update`, and something is missing from
# the secondary lockfile, install it.
if install && (definition.missing_specs.any? || resolved_remotely)
::Bundler.with_default_lockfile(lockfile_definition[:lockfile]) do
::Bundler::Installer.install(lockfile_definition[:gemfile].dirname, definition, {})
end
end
end
end
@recursive = false
end
Bundler::Dsl.include(BundlerLockfileExtensions::Bundler::Dsl)
# this is terrible, but we can't prepend into any module because we only load
# _inside_ of the CLI commands already running
if defined?(Bundler::CLI::Check)
require "bundler_lockfile_extensions/check"
at_exit do
next unless $!.nil?
next if $!.is_a?(SystemExit) && !$!.success?
next if BundlerLockfileExtensions::Check.run
Bundler.ui.warn("You can attempt to fix by running `bundle install`")
exit 1
end
end
if defined?(Bundler::CLI::Lock)
at_exit do
next unless $!.nil?
next if $!.is_a?(SystemExit) && !$!.success?
BundlerLockfileExtensions.after_install_all(install: false)
end
end

48
gems/bundler_lockfile_extensions/lib/bundler_lockfile_extensions/bundler.rb Normal file
View File

@ -0,0 +1,48 @@
# frozen_string_literal: true
#
# Copyright (C) 2023 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
module BundlerLockfileExtensions
module Bundler
module ClassMethods
def self.prepended(klass)
super
klass.attr_writer :cache_root, :default_lockfile, :root
end
def app_cache(custom_path = nil)
super(custom_path || @cache_root)
end
def default_lockfile(force_original: false)
return @default_lockfile if @default_lockfile && !force_original
super()
end
def with_default_lockfile(lockfile)
previous_default_lockfile, @default_lockfile = @default_lockfile, lockfile
yield
ensure
@default_lockfile = previous_default_lockfile
end
end
end
end

34
gems/bundler_lockfile_extensions/lib/bundler_lockfile_extensions/bundler/definition.rb Normal file
View File

@ -0,0 +1,34 @@
# frozen_string_literal: true
#
# Copyright (C) 2023 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
module BundlerLockfileExtensions
module Bundler
module Definition
def initialize(lockfile, *args)
# we changed the default lockfile in BundlerLockfileExtensions.add_lockfile
# since DSL.evaluate was called (re-entrantly); sub the proper value in
if !lockfile.equal?(::Bundler.default_lockfile) && ::Bundler.default_lockfile(force_original: true) == lockfile
lockfile = ::Bundler.default_lockfile
end
super
end
end
end
end

29
gems/bundler_lockfile_extensions/lib/bundler_lockfile_extensions/bundler/dsl.rb Normal file
View File

@ -0,0 +1,29 @@
# frozen_string_literal: true
#
# Copyright (C) 2023 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
module BundlerLockfileExtensions
module Bundler
module Dsl
def add_lockfile(*args, **kwargs)
BundlerLockfileExtensions.add_lockfile(*args, builder: self, **kwargs)
end
end
end
end

30
gems/bundler_lockfile_extensions/lib/bundler_lockfile_extensions/bundler/source_list.rb Normal file
View File

@ -0,0 +1,30 @@
# frozen_string_literal: true
#
# Copyright (C) 2023 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
module BundlerLockfileExtensions
module Bundler
module SourceList
# consider them equivalent if the replacements just have a bunch of dups
def equivalent_sources?(lock_sources, replacement_sources)
super(lock_sources, replacement_sources.uniq)
end
end
end
end

130
gems/bundler_lockfile_extensions/lib/bundler_lockfile_extensions/check.rb Normal file
View File

@ -0,0 +1,130 @@
# frozen_string_literal: true
#
# Copyright (C) 2023 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
require "set"
module BundlerLockfileExtensions
module Check
class << self
def run
return true unless ::Bundler.default_lockfile.exist?
default_lockfile_contents = ::Bundler.default_lockfile.read
default_lockfile = ::Bundler::LockfileParser.new(default_lockfile_contents)
default_specs = default_lockfile.specs.to_h do |spec| # rubocop:disable Rails/IndexBy
[[spec.name, spec.platform], spec]
end
success = true
BundlerLockfileExtensions.lockfile_definitions.each do |lockfile_definition|
next unless lockfile_definition[:lockfile].exist?
proven_pinned = Set.new
needs_pin_check = []
lockfile = ::Bundler::LockfileParser.new(lockfile_definition[:lockfile].read)
specs = lockfile.specs.group_by(&:name)
# build list of top-level dependencies that differ from the default lockfile,
# and all _their_ transitive dependencies
if lockfile_definition[:allow_mismatched_dependencies]
transitive_dependencies = Set.new
# only dependencies that differ from the default lockfile
pending_transitive_dependencies = lockfile.dependencies.reject do |name, dep|
default_lockfile.dependencies[name] == dep
end.map(&:first)
until pending_transitive_dependencies.empty?
dep = pending_transitive_dependencies.shift
next if transitive_dependencies.include?(dep)
transitive_dependencies << dep
platform_specs = specs[dep]
unless platform_specs
# should only be bundler that's missing a spec
raise "Could not find spec for dependency #{dep}" unless dep == "bundler"
next
end
pending_transitive_dependencies.concat(platform_specs.flat_map(&:dependencies).map(&:name).uniq)
end
end
# look through top-level explicit dependencies for pinned requirements
if lockfile_definition[:enforce_pinned_additional_dependencies]
find_pinned_dependencies(proven_pinned, lockfile.dependencies.each_value)
end
# check for conflicting requirements (and build list of pins, in the same loop)
specs.values.flatten.each do |spec|
default_spec = default_specs[[spec.name, spec.platform]]
if lockfile_definition[:enforce_pinned_additional_dependencies]
# look through what this spec depends on, and keep track of all pinned requirements
find_pinned_dependencies(proven_pinned, spec.dependencies)
needs_pin_check << spec unless default_spec
end
next unless default_spec
next if default_spec.version == spec.version
next if lockfile_definition[:allow_mismatched_dependencies] && transitive_dependencies.include?(spec.name)
::Bundler.ui.error("#{spec} in #{lockfile_definition[:lockfile].relative_path_from(Dir.pwd)} does not match the default lockfile's version (@#{default_spec.version}); this may be due to a conflicting requirement, which would require manual resolution.")
success = false
end
# now that we have built a list of every gem that is pinned, go through
# the gems that were in this lockfile, but not the default lockfile, and
# ensure it's pinned _somehow_
needs_pin_check.each do |spec|
pinned = case spec.source
when ::Bundler::Source::Git
spec.source.ref == spec.source.revision
when ::Bundler::Source::Path
true
when ::Bundler::Source::Rubygems
proven_pinned.include?(spec.name)
else
false
end
unless pinned
::Bundler.ui.error("#{spec} in #{lockfile_definition[:lockfile].relative_path_from(Dir.pwd)} has not been pinned to a specific version, which is required since it is not part of the default lockfile.")
success = false
end
end
end
success
end
private
def find_pinned_dependencies(proven_pinned, dependencies)
dependencies.each do |dependency|
dependency.requirement.requirements.each do |requirement|
proven_pinned << dependency.name if requirement.first == "="
end
end
end
end
end
end

60
gems/bundler_lockfile_extensions/lib/bundler_lockfile_extensions/lockfile_generator.rb Normal file
View File

@ -0,0 +1,60 @@
# frozen_string_literal: true
#
# Copyright (C) 2023 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
require "bundler/lockfile_generator"
module BundlerLockfileExtensions
# generates a lockfile based on another LockfileParser
class LockfileGenerator < ::Bundler::LockfileGenerator
def self.generate(lockfile)
new(LockfileAdapter.new(lockfile)).generate!
end
private
class LockfileAdapter < SimpleDelegator
def sources
self
end
def lock_sources
__getobj__.sources
end
def resolve
specs
end
def dependencies
super.values
end
def locked_ruby_version
ruby_version
end
end
private_constant :LockfileAdapter
def add_bundled_with
add_section("BUNDLED WITH", definition.bundler_version.to_s)
end
end
end

4
gems/bundler_lockfile_extensions/plugins.rb
View File

@ -18,8 +18,8 @@
# with this program. If not, see <http://www.gnu.org/licenses/>.
#
require_relative "lib/bundler_lockfile_extensions"
require "bundler_lockfile_extensions"
Bundler::Plugin.add_hook(Bundler::Plugin::Events::GEM_AFTER_INSTALL_ALL) do |_|
BundlerLockfileExtensions.write_all_lockfiles unless BundlerLockfileExtensions.lockfile_defs.nil? || defined?(Bundler::CLI::Cache) || defined?(Bundler::CLI::Lock) || Bundler.settings[:deployment]
BundlerLockfileExtensions.after_install_all
end

496
gems/bundler_lockfile_extensions/spec/bundler_lockfile_extensions_spec.rb
View File

@ -24,206 +24,328 @@ require "fileutils"
require_relative "spec_helper"
describe "BundlerLockfileExtensions" do
# the definition section for a gemfile with two lockfiles; one
# that will have more gems than the default.
let(:all_gems_definitions) do
<<~RUBY
add_lockfile(
prepare: -> { ::INCLUDE_ALL_GEMS = false },
current: false
)
add_lockfile(
"Gemfile.full.lock",
prepare: -> { ::INCLUDE_ALL_GEMS = true },
current: true
)
RUBY
end
let(:all_gems_preamble) do
"::INCLUDE_ALL_GEMS = true unless defined?(::INCLUDE_ALL_GEMS)"
end
it "generates a default Gemfile.lock when loaded, but not configured" do
contents = <<-RUBY
gem "concurrent-ruby", "1.2.0"
contents = <<~RUBY
gem "concurrent-ruby", "1.2.2"
RUBY
with_gemfile(contents) do |file|
output = invoke_bundler("install", file.path)
with_gemfile("", contents) do
output = invoke_bundler("install")
expect(output).to include("1.2.0")
expect(File.read("#{file.path}.lock")).to include("1.2.0")
expect(output).to include("1.2.2")
expect(File.read("Gemfile.lock")).to include("1.2.2")
end
end
it "generates custom lockfiles with varying versions and excluded gems" do
contents = <<-RUBY
unless BundlerLockfileExtensions.enabled?
BundlerLockfileExtensions.enable({
"\#{__FILE__}.old.lock": {
default: true,
prepare_environment: -> { ::GEM_VERSION = "1.1.10" },
},
"\#{__FILE__}.new.lock": {
default: false,
prepare_environment: -> { ::GEM_VERSION = "1.2.0" },
},
})
end
it "disallows multiple default lockfiles" do
with_gemfile(<<~RUBY) do
add_lockfile()
add_lockfile()
RUBY
expect { invoke_bundler("install") }.to raise_error(/Only one default lockfile/)
end
end
it "disallows multiple current lockfiles" do
with_gemfile(<<~RUBY) do
add_lockfile(current: true)
add_lockfile("Gemfile.new.lock", current: true)
RUBY
expect { invoke_bundler("install") }.to raise_error(/Only one lockfile/)
end
end
it "generates custom lockfiles with varying versions" do
definitions = <<~RUBY
add_lockfile(
prepare: -> { ::GEM_VERSION = "1.1.10" }
)
add_lockfile(
"Gemfile.new.lock",
prepare: -> { ::GEM_VERSION = "1.2.2" }
)
RUBY
contents = <<~RUBY
::GEM_VERSION = "1.1.10" unless defined?(::GEM_VERSION)
gem "concurrent-ruby", ::GEM_VERSION
RUBY
with_gemfile(contents) do |file|
output = invoke_bundler("install", file.path)
with_gemfile(definitions, contents) do
invoke_bundler("install")
expect(output).to include("1.1.10")
expect(File.read("#{file.path}.old.lock")).to include("1.1.10")
expect(File.read("#{file.path}.new.lock")).to include("1.2.0")
expect(File.read("Gemfile.lock")).to include("1.1.10")
expect(File.read("Gemfile.lock")).not_to include("1.2.2")
expect(File.read("Gemfile.new.lock")).not_to include("1.1.10")
expect(File.read("Gemfile.new.lock")).to include("1.2.2")
end
end
it "generates lockfiles with a subset of gems" do
contents = <<-RUBY
unless BundlerLockfileExtensions.enabled?
BundlerLockfileExtensions.enable({
"\#{__FILE__}.lock": {
default: true,
install_filter: lambda { |_, x| !x.to_s.include?("test_local") },
},
})
end
gem "test_local", path: "test_local"
gem "concurrent-ruby", "1.2.0"
RUBY
with_gemfile(contents) do |file, dir|
create_local_gem(dir, "test_local", "")
invoke_bundler("install", file.path)
expect(File.read("#{file.path}.lock")).to include("test_local")
expect(File.read("#{file.path}.lock.partial")).not_to include("test_local")
expect(File.read("#{file.path}.lock")).to include("concurrent-ruby")
expect(File.read("#{file.path}.lock.partial")).to include("concurrent-ruby")
end
end
it "regenerates the lockfile when a source is completely removed and its dependencies no longer exist" do
contents = <<-RUBY
unless BundlerLockfileExtensions.enabled?
BundlerLockfileExtensions.enable({
"\#{__FILE__}.lock": {
default: true,
install_filter: lambda { |_, x| !x.to_s.include?("test_local") },
},
})
end
if ENV["USE_TEST_LOCAL_GEM"] == "1"
contents = <<~RUBY
if ::INCLUDE_ALL_GEMS
gem "test_local", path: "test_local"
end
gem "concurrent-ruby", "1.2.0"
gem "concurrent-ruby", "1.2.2"
RUBY
with_gemfile(contents) do |file, dir|
create_local_gem(dir, "test_local", <<-RUBY)
spec.add_dependency "dummy", "0.9.6"
RUBY
with_gemfile(all_gems_definitions, contents, all_gems_preamble) do
create_local_gem("test_local", "")
invoke_bundler("install", file.path, env: { "USE_TEST_LOCAL_GEM" => "1" })
invoke_bundler("install")
expect(File.read("#{file.path}.lock")).to include("dummy")
expect(File.read("#{file.path}.lock")).to include("test_local")
expect(File.read("#{file.path}.lock.partial")).to include("dummy")
expect(File.read("#{file.path}.lock.partial")).not_to include("test_local")
expect(File.read("Gemfile.lock")).not_to include("test_local")
expect(File.read("Gemfile.full.lock")).to include("test_local")
invoke_bundler("install", file.path, env: { "USE_TEST_LOCAL_GEM" => "0" })
expect(File.read("#{file.path}.lock")).not_to include("dummy")
expect(File.read("#{file.path}.lock")).not_to include("test_local")
expect(File.read("#{file.path}.lock.partial")).not_to include("dummy")
expect(File.read("#{file.path}.lock.partial")).not_to include("test_local")
expect(File.read("Gemfile.lock")).to include("concurrent-ruby")
expect(File.read("Gemfile.full.lock")).to include("concurrent-ruby")
end
end
it "fails if an additional lockfile contains an invalid gem" do
contents = <<-RUBY
unless BundlerLockfileExtensions.enabled?
BundlerLockfileExtensions.enable({
"\#{__FILE__}.old.lock": {
default: true,
},
"\#{__FILE__}.new.lock": {
default: false,
},
})
end
gem "concurrent-ruby", ">= 1.2.0"
definitions = <<~RUBY
add_lockfile()
add_lockfile(
"Gemfile.new.lock"
)
RUBY
with_gemfile(contents) do |file|
invoke_bundler("install", file.path)
replace_gemfile_lock_pin("#{file.path}.new.lock", "concurrent-ruby", "9.9.9")
contents = <<~RUBY
gem "concurrent-ruby", ">= 1.2.2"
RUBY
expect { invoke_bundler("install", file.path) }.to raise_error(/concurrent-ruby\s+\(9\.9\.9\)\s+has\s+removed\s+it/m)
with_gemfile(definitions, contents) do
invoke_bundler("install")
replace_lockfile_pin("Gemfile.new.lock", "concurrent-ruby", "9.9.9")
expect { invoke_bundler("check") }.to raise_error(/concurrent-ruby.*does not match/m)
end
end
# This section tests that the following constraint is adhered to:
# 1. All dependencies under a private source must be pinned in the private plugin gemspec
context "filtered dependency pins" do
let(:gemfile_contents) do
<<-RUBY
unless BundlerLockfileExtensions.enabled?
BundlerLockfileExtensions.enable({
"\#{__FILE__}.v1.lock": {
default: true,
install_filter: lambda { |_, x| !x.to_s.include?("packagecloud") && !x.to_s.include?("test_local") },
prepare_environment: -> { ::VERSION = "1" },
},
"\#{__FILE__}.v2.lock": {
default: false,
install_filter: lambda { |_, x| !x.to_s.include?("packagecloud") && !x.to_s.include?("test_local") },
prepare_environment: -> { ::VERSION = "2" },
},
})
it "preserves the locked version of a gem in an alternate lockfile when updating a different gem in common" do
contents = <<~RUBY
gem "net-ldap", "0.17.0"
if ::INCLUDE_ALL_GEMS
gem "net-smtp", "0.3.2"
end
RUBY
with_gemfile(all_gems_definitions, contents, all_gems_preamble) do
invoke_bundler("install")
expect(invoke_bundler("info net-ldap")).to include("0.17.0")
expect(invoke_bundler("info net-smtp")).to include("0.3.2")
# loosen the requirement on both gems
write_gemfile(all_gems_definitions, <<~RUBY, all_gems_preamble)
gem "net-ldap", "~> 0.17"
if ::INCLUDE_ALL_GEMS
gem "net-smtp", "~> 0.3"
end
RUBY
gem "test_local", path: "test_local"
gem "concurrent-ruby", "1.2.0"
# but only update net-ldap
invoke_bundler("update net-ldap")
eval(File.read("\#{__dir__}/test_local/Gemfile"))
# net-smtp should be untouched, even though it's no longer pinned
expect(invoke_bundler("info net-ldap")).not_to include("0.17.0")
expect(invoke_bundler("info net-smtp")).to include("0.3.2")
end
end
it "maintains consistency across multiple Gemfiles" do
definitions = <<~RUBY
add_lockfile()
add_lockfile(
"local_test/Gemfile.lock",
gemfile: "local_test/Gemfile")
RUBY
contents = <<~RUBY
gem "net-smtp", "0.3.2"
RUBY
with_gemfile(definitions, contents) do
create_local_gem("local_test", <<~RUBY)
spec.add_dependency "net-smtp", "~> 0.3"
RUBY
invoke_bundler("install")
# locks to 0.3.2 in the local gem's lockfile, even though the local
# gem itself would allow newer
expect(File.read("local_test/Gemfile.lock")).to include("0.3.2")
end
end
it "whines about non-pinned dependencies in flagged gemfiles" do
definitions = <<~RUBY
add_lockfile(
prepare: -> { ::INCLUDE_ALL_GEMS = false },
current: false
)
add_lockfile(
"Gemfile.full.lock",
prepare: -> { ::INCLUDE_ALL_GEMS = true },
current: true,
enforce_pinned_additional_dependencies: true
)
RUBY
contents = <<~RUBY
gem "net-ldap", "0.17.0"
if ::INCLUDE_ALL_GEMS
gem "net-smtp", "~> 0.3"
end
RUBY
with_gemfile(definitions, contents, all_gems_preamble) do
expect { invoke_bundler("install") }.to raise_error(/net-smtp \([0-9.]+\) in Gemfile.full.lock has not been pinned/)
# not only have to pin net-smtp, but also its transitive dependencies
write_gemfile(definitions, <<~RUBY, all_gems_preamble)
gem "net-ldap", "0.17.0"
if ::INCLUDE_ALL_GEMS
gem "net-smtp", "0.3.2"
gem "net-protocol", "0.2.1"
gem "timeout", "0.3.2"
end
RUBY
invoke_bundler("install") # no error, because it's now pinned
end
end
context "with mismatched dependencies disallowed" do
let(:all_gems_definitions) do
<<~RUBY
add_lockfile(
prepare: -> { ::INCLUDE_ALL_GEMS = false },
current: false
)
add_lockfile(
"Gemfile.full.lock",
prepare: -> { ::INCLUDE_ALL_GEMS = true },
allow_mismatched_dependencies: false,
current: true
)
RUBY
end
let(:private_gemfile_contents) do
<<-RUBY
if ::VERSION == ENV["USE_VERSION"]
source "https://packagecloud.io/instructure/rubygems-public/" do
gem "hola", ">= 0.1.3"
end
it "notifies about mismatched versions between different lockfiles" do
contents = <<~RUBY
if ::INCLUDE_ALL_GEMS
gem "activesupport", "7.0.4.3"
else
gem "activesupport", "~> 6.1.0"
end
RUBY
end
it "fails if a filtered dependency isn't included in the gemspec" do
with_gemfile(gemfile_contents) do |file, dir|
create_local_gem(dir, "test_local", "")
File.write("#{dir}/test_local/Gemfile", private_gemfile_contents)
expect { invoke_bundler("install", file.path, env: { "USE_VERSION" => "1" }) }.to raise_error(/unable to prove that private gem hola was pinned/)
expect { invoke_bundler("install", file.path, env: { "USE_VERSION" => "2" }) }.to raise_error(/unable to prove that private gem hola was pinned/)
with_gemfile(all_gems_definitions, contents, all_gems_preamble) do
expect { invoke_bundler("install") }.to raise_error(/activesupport \(7.0.4.3\) in Gemfile.full.lock does not match the default lockfile's version/)
end
end
it "fails if a filtered dependency isn't pinned to an exact version in the gemspec" do
with_gemfile(gemfile_contents) do |file, dir|
create_local_gem(dir, "test_local", <<-RUBY)
spec.add_dependency "dummy", ">= 0.9.6"
RUBY
it "notifies about mismatched versions between different lockfiles for sub-dependencies" do
definitions = <<~RUBY
add_lockfile(
prepare: -> { ::INCLUDE_ALL_GEMS = false },
current: false
)
add_lockfile(
"Gemfile.full.lock",
prepare: -> { ::INCLUDE_ALL_GEMS = true },
allow_mismatched_dependencies: false,
current: true
)
RUBY
File.write("#{dir}/test_local/Gemfile", private_gemfile_contents)
contents = <<~RUBY
gem "activesupport", "7.0.4.3" # depends on tzinfo ~> 2.0, so will get >= 2.0.6
expect { invoke_bundler("install", file.path, env: { "USE_VERSION" => "1" }) }.to raise_error(/unable to prove that private gem hola was pinned/)
expect { invoke_bundler("install", file.path, env: { "USE_VERSION" => "2" }) }.to raise_error(/unable to prove that private gem hola was pinned/)
if ::INCLUDE_ALL_GEMS
gem "tzinfo", "2.0.5"
end
RUBY
with_gemfile(definitions, contents, all_gems_preamble) do
expect { invoke_bundler("install") }.to raise_error(/tzinfo \(2.0.5\) in Gemfile.full.lock does not match the default lockfile's version/)
end
end
end
it "allows mismatched explicit dependencies by default" do
contents = <<~RUBY
if ::INCLUDE_ALL_GEMS
gem "activesupport", "7.0.4.3"
else
gem "activesupport", "~> 6.1.0"
end
RUBY
with_gemfile(all_gems_definitions, contents, all_gems_preamble) do
invoke_bundler("install") # no error
expect(File.read("Gemfile.lock")).to include("6.1.")
expect(File.read("Gemfile.lock")).not_to include("7.0.4.3")
expect(File.read("Gemfile.full.lock")).not_to include("6.1.")
expect(File.read("Gemfile.full.lock")).to include("7.0.4.3")
end
end
it "disallows mismatched implicit dependencies" do
definitions = <<~RUBY
add_lockfile()
add_lockfile(
"local_test/Gemfile.lock",
allow_mismatched_dependencies: false,
gemfile: "local_test/Gemfile")
RUBY
contents = <<~RUBY
gem "activesupport", "7.0.4.3"
gem "concurrent-ruby", "1.0.2"
RUBY
with_gemfile(definitions, contents) do
create_local_gem("local_test", <<~RUBY)
spec.add_dependency "activesupport", "7.0.4.3"
RUBY
expect { invoke_bundler("install") }.to raise_error(%r{concurrent-ruby \([0-9.]+\) in local_test/Gemfile.lock does not match the default lockfile's version \(@1.0.2\)})
end
end
private
def create_local_gem(dir, name, content)
FileUtils.mkdir_p("#{dir}/#{name}")
File.write("#{dir}/#{name}/#{name}.gemspec", <<-RUBY)
def create_local_gem(name, content)
FileUtils.mkdir_p(name)
File.write("#{name}/#{name}.gemspec", <<~RUBY)
Gem::Specification.new do |spec|
spec.name = "#{name}"
spec.name = #{name.inspect}
spec.version = "0.0.1"
spec.authors = ["Instructure"]
spec.summary = "for testing only"
@ -231,40 +353,84 @@ describe "BundlerLockfileExtensions" do
#{content}
end
RUBY
File.write("#{name}/Gemfile", <<~RUBY)
source "https://rubygems.org"
gemspec
RUBY
end
def with_gemfile(content)
dir = Dir.mktmpdir
file = Tempfile.new("Gemfile", dir).tap do |f|
f.write(<<-RUBY)
source "https://rubygems.org"
plugin "bundler_lockfile_extensions", path: "#{File.dirname(__FILE__)}/.."
Plugin.send(:load_plugin, 'bundler_lockfile_extensions') if Plugin.installed?('bundler_lockfile_extensions') && !defined?(BundlerLockfileExtensions)
#{content}
RUBY
f.rewind
# creates a new temporary directory, writes the gemfile to it, and yields
#
# @param (see #write_gemfile)
# @yield
def with_gemfile(definitions, content = nil, preamble = nil)
Dir.mktmpdir do |dir|
Dir.chdir(dir) do
write_gemfile(definitions, content, preamble)
invoke_bundler("config frozen false")
yield
end
end
yield(file, dir)
ensure
FileUtils.remove_dir(dir, true)
end
def invoke_bundler(subcommand, gemfile_path, env: {})
# @param definitions [String]
# Ruby code to set up lockfiles by calling add_lockfile. Called inside a
# conditional for when BundlerLockfileExtensions is loaded the first time.
# @param content [String]
# Additional Ruby code for adding gem requirements to the Gemfile
# @param preamble [String]
# Additional Ruby code to execute prior to installing the plugin.
def write_gemfile(definitions, content = nil, preamble = nil)
raise ArgumentError, "Did you mean to use `with_gemfile`?" if block_given?
File.write("Gemfile", <<~RUBY)
source "https://rubygems.org"
#{preamble}
plugin "bundler_lockfile_extensions", path: #{File.dirname(__dir__).inspect}
if Plugin.installed?("bundler_lockfile_extensions")
Plugin.send(:load_plugin, "bundler_lockfile_extensions") unless defined?(BundlerLockfileExtensions)
unless BundlerLockfileExtensions.enabled?
#{definitions}
end
end
#{content}
RUBY
end
# Shells out to a new instance of bundler, with a clean bundler env
#
# @param subcommand [String] Args to pass to bundler
# @raise [RuntimeError] if the bundle command fails
def invoke_bundler(subcommand, env: {})
output = nil
bundler_version = ENV.fetch("BUNDLER_VERSION")
command = "bundle _#{bundler_version}_ #{subcommand}"
Bundler.with_unbundled_env do
output, status = Open3.capture2e({ "BUNDLE_GEMFILE" => gemfile_path }.merge(env), command)
output, status = Open3.capture2e(env, command)
raise "bundle #{subcommand} failed: #{output}" unless status.success?
end
output
end
def replace_gemfile_lock_pin(path, name, version)
new_contents = File.read(path).gsub(%r{#{name} \([0-9.]+\)}, "#{name} (#{version})")
# Directly modifies a lockfile to adjust the version of a gem
#
# Useful for simulating certain unusual situations that can arise.
#
# @param lockfile [String] The lockfile's location
# @param gem [String] The gem's name
# @param version [String] The new version to "pin" the gem to
def replace_lockfile_pin(lockfile, gem, version)
new_contents = File.read(lockfile).gsub(%r{#{gem} \([0-9.]+\)}, "#{gem} (#{version})")
File.write(path, new_contents)
File.write(lockfile, new_contents)
end
end

2
gems/canvas_color/Gemfile.lock
View File

@ -20,4 +20,4 @@ DEPENDENCIES
rake
BUNDLED WITH
2.4.10
2.3.26

2
gems/canvas_panda_pub/Gemfile.lock
View File

@ -92,4 +92,4 @@ DEPENDENCIES
webmock (~> 3.0)
BUNDLED WITH
2.4.10
2.3.26

3
gems/dr_diff/Gemfile.lock
View File

@ -30,7 +30,10 @@ GEM
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.12.0)
rspec-support (3.12.0)
sqlite3 (1.6.2-aarch64-linux)
sqlite3 (1.6.2-arm64-darwin)
sqlite3 (1.6.2-x86_64-darwin)
sqlite3 (1.6.2-x86_64-linux)
PLATFORMS
aarch64-linux

1
gems/google_drive/test.sh
View File

@ -1,6 +1,5 @@
#!/bin/bash
set -e
rm -f Gemfile.lock
bundle check || bundle install
bundle exec rspec spec

1
gems/twitter/test.sh
View File

@ -1,6 +1,5 @@
#!/bin/bash
set -e
rm -f Gemfile.lock
bundle check || bundle install
bundle exec rspec spec

2
hooks/pre-commit
View File

@ -27,7 +27,7 @@ if [ $(pwd -P) = $CANVAS ]; then
if git diff --cached --name-only | grep -q 'Gemfile\S*.lock'; then
echo "Checking lockfiles..."
script/sync_lockfiles.rb
bundle check
fi
fi

104
script/sync_lockfiles.rb
View File

@ -1,104 +0,0 @@
#!/usr/bin/env ruby
# frozen_string_literal: true
# Copyright (C) 2023 - present Instructure, Inc.
#
# This file is part of Canvas.
#
# Canvas is free software: you can redistribute it and/or modify it under
# the terms of the GNU Affero General Public License as published by the Free
# Software Foundation, version 3 of the License.
#
# Canvas is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
require "bundler"
require "tempfile"
do_sync = ARGV.include?("--sync")
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__)
canvas_lockfile_name = Dir.glob(Bundler.default_lockfile.dirname + "Gemfile.rails*.lock*").first
canvas_lockfile_contents = File.read(canvas_lockfile_name)
canvas_specs = Bundler::LockfileParser.new(canvas_lockfile_contents).specs.to_h do |spec| # rubocop:disable Rails/IndexBy
[[spec.name, spec.platform], spec]
end
canvas_root = File.dirname(canvas_lockfile_name)
success = true
Bundler.settings.temporary(cache_all_platforms: true) do
previous_ui_level = Bundler.ui.level
Bundler.ui.level = "silent"
Dir["Gemfile.d/*.lock", "gems/*/Gemfile.lock"].each do |gem_lockfile_name|
if do_sync
gem_gemfile_name = gem_lockfile_name.sub(/\.lock$/, "")
# root needs to be set so that paths are output relative to the correct root in the lockfile
Bundler.instance_variable_set(:@root, Pathname.new(gem_lockfile_name).dirname.expand_path)
# adjust locked paths from the Canvas lockfile to be relative to _this_ gemfile
new_contents = canvas_lockfile_contents.gsub(/PATH\n remote: ([^\n]+)\n/) do |remote|
remote_path = Pathname.new($1)
next remote if remote_path.absolute?
relative_remote_path = remote_path.expand_path(canvas_root).relative_path_from(Bundler.root).to_s
remote.sub($1, relative_remote_path)
end
# add a source for the current gem
gem_spec = canvas_specs[[File.basename(Bundler.root), "ruby"]]
if gem_spec
new_contents += <<~TEXT
PATH
remote: .
specs:
#{gem_spec.to_lock}
TEXT
end
definition = nil
puts "Syncing #{gem_gemfile_name}..."
# Now build a definition based on the gem's Gemfile, but *Canvas* (tweaked) lockfile
Tempfile.create do |temp_lockfile|
temp_lockfile.write(new_contents)
definition = Bundler::Definition.build(gem_gemfile_name, temp_lockfile.path, false)
end
changed = !definition.send(:lockfiles_equal?, File.read(gem_lockfile_name), definition.to_lock, true)
success = false if changed
if changed
definition.lock(gem_lockfile_name, true)
end
end
# now do a double check for conflicting requirements
Bundler::LockfileParser.new(File.read(gem_lockfile_name)).specs.each do |spec|
next unless (canvas_spec = canvas_specs[[spec.name, spec.platform]])
platform = (spec.platform == "ruby") ? "" : "-#{spec.platform}"
next if canvas_spec.version == spec.version
warn "#{spec.name}#{platform}@#{spec.version} in #{gem_lockfile_name} does not match Canvas (@#{canvas_spec.version}); this is may be due to a conflicting requirement, which would require manual resolution."
success = false
end
end
ensure
Bundler.ui.level = previous_ui_level
end
if !success && !do_sync
warn "\nYou can attempt to fix by running script/sync_lockfiles.rb --sync"
end
exit(success ? 0 : 1)

2
spec/lib/ha_store_spec.rb
View File

@ -29,7 +29,7 @@ describe ActiveSupport::Cache::HaStore do
describe "#delete" do
it "triggers a consul event when configured" do
# will get called twice; once with rails52: prefix, once without
expect(Diplomat::Event).to receive(:fire).with("invalidate", match(/mykey$/), nil, nil, nil, nil).exactly(SUPPORTED_VERSIONS.count).times
expect(Diplomat::Event).to receive(:fire).with("invalidate", match(/mykey$/), nil, nil, nil, nil).exactly(SUPPORTED_RAILS_VERSIONS.count).times
store.delete("mykey")
end
end