The memoization of Scheme#key_provider was removed completely in
https://github.com/rails/rails/pull/51019 because it prevented
overriding the `key_provider` via `with_encryption_context`. However,
this also made our tests for the HEY app about 5 times slower. We traced
this down to all attributes where we either provide a key or declare them
as deterministic and have to derive a key to instantiate the provider
every time we load them. This might happen hundreds of times per test,
and ultimately, we call
```
ActiveSupport::KeyGenerator#generate_key
```
and
```
OpenSSL::KDF.pbkdf2_hmac
```
hundreds of times. This adds significant overhead per test.
In reality, what's overridden bv `with_encryption_context` is the value
used as default provider, that is, `ActiveRecord::Encryption.key_provider`.
This is only used in `Scheme#key_provider` if the scheme doesn't already have
either a `key_provider` passed directly, or a `key`, or is `deterministic`.
The `key_provider` passed directly is already memoized simply by having it
stored as is. Let's memoize the other two, in order, so we save all those
extra calls to derive the same keys again and again.
This commit addresses the following errors against MySQL 8.0.18 or lower version of MySQL 8.0.
- Steps to reproduce
```ruby
git clone https://github.com/rails/rails
cd rails
git clone https://github.com/rails/buildkite-config .buildkite/
RUBY_IMAGE=ruby:3.3 docker-compose -f .buildkite/docker-compose.yml build base &&
CI=1 MYSQL_IMAGE=mysql:8.0.18 docker-compose -f .buildkite/docker-compose.yml run mysqldb runner activerecord 'rake db:mysql:rebuild test:mysql2'
```
- Actual behavior
```ruby
... snip ...
Error:
InsertAllTest#test_upsert_all_implicitly_sets_timestamps_on_create_when_model_record_timestamps_is_false_but_overridden:
ActiveRecord::StatementInvalid: Mysql2::Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AS `ships_values` ON DUPLICATE KEY UPDATE updated_at=(CASE WHEN (`ships`.`name`<' at line 1
/usr/local/bundle/gems/mysql2-0.5.6/lib/mysql2/client.rb:151:in `_query'
/usr/local/bundle/gems/mysql2-0.5.6/lib/mysql2/client.rb:151:in `block in query'
/usr/local/bundle/gems/mysql2-0.5.6/lib/mysql2/client.rb:150:in `handle_interrupt'
/usr/local/bundle/gems/mysql2-0.5.6/lib/mysql2/client.rb:150:in `query'
lib/active_record/connection_adapters/mysql2/database_statements.rb:104:in `block (2 levels) in raw_execute'
lib/active_record/connection_adapters/abstract_adapter.rb:997:in `block in with_raw_connection'
/rails/activesupport/lib/active_support/concurrency/load_interlock_aware_monitor.rb:23:in `handle_interrupt'
/rails/activesupport/lib/active_support/concurrency/load_interlock_aware_monitor.rb:23:in `block in synchronize'
/rails/activesupport/lib/active_support/concurrency/load_interlock_aware_monitor.rb:19:in `handle_interrupt'
/rails/activesupport/lib/active_support/concurrency/load_interlock_aware_monitor.rb:19:in `synchronize'
lib/active_record/connection_adapters/abstract_adapter.rb:969:in `with_raw_connection'
lib/active_record/connection_adapters/mysql2/database_statements.rb:102:in `block in raw_execute'
/rails/activesupport/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
lib/active_record/connection_adapters/abstract_adapter.rb:1112:in `log'
lib/active_record/connection_adapters/mysql2/database_statements.rb:101:in `raw_execute'
lib/active_record/connection_adapters/abstract_mysql_adapter.rb:237:in `execute_and_free'
lib/active_record/connection_adapters/mysql2/database_statements.rb:23:in `internal_exec_query'
lib/active_record/connection_adapters/abstract/database_statements.rb:171:in `exec_insert_all'
lib/active_record/connection_adapters/abstract/query_cache.rb:26:in `exec_insert_all'
lib/active_record/insert_all.rb:55:in `execute'
lib/active_record/insert_all.rb:13:in `block in execute'
lib/active_record/connection_adapters/abstract/connection_pool.rb:384:in `with_connection'
lib/active_record/connection_handling.rb:270:in `with_connection'
lib/active_record/insert_all.rb:12:in `execute'
lib/active_record/persistence.rb:363:in `upsert_all'
test/cases/insert_all_test.rb:561:in `block in test_upsert_all_implicitly_sets_timestamps_on_create_when_model_record_timestamps_is_false_but_overridden'
test/cases/insert_all_test.rb:809:in `with_record_timestamps'
test/cases/insert_all_test.rb:560:in `test_upsert_all_implicitly_sets_timestamps_on_create_when_model_record_timestamps_is_false_but_overridden'
bin/rails test /rails/activerecord/test/cases/insert_all_test.rb:557
E
... snip ...
8856 runs, 25842 assertions, 1 failures, 52 errors, 41 skips
```
Follow up #51274
Refer to these release notes, WL and commits for MySQL 8.0.19 and 8.0.20.
- MySQL 8.0.19 supports aliases in the VALUES and SET clauses of INSERT INTO ... ON DUPLICATE KEY UPDATE statement
https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-19.html
> MySQL now supports aliases in the VALUES and SET clauses of INSERT INTO ... ON DUPLICATE KEY UPDATE statement
> for the row to be inserted and its columns. Consider a statement such as this one:
https://dev.mysql.com/worklog/task/?id=6312c39355e9e6
- MySQL 8.0.20 deprecates the old `VALUES()` syntax in INSERT ... ON DUPLICATE KEY UPDATE statements
https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-20.html
> The use of VALUES() to access new row values in INSERT ... ON DUPLICATE KEY UPDATE statements
> is now deprecated, and is subject to removal in a future MySQL release.
> Instead, you should use aliases for the new row and its columns as implemented in MySQL 8.0.19 and later.
https://dev.mysql.com/worklog/task/?id=133256f3b9df50b
The API documentation for `ActiveRecord::Migration` mentions controlling
the level of log output through a `.verbose` class attribute. The line
isn't wrapped in `<tt>`, `+`, or backticks, so isn't emphasized as if it
were code.
This commit visually emphasizes that line so that it's more obviously
code.
This adds a `dirties` option to `ActiveRecord::Base.uncached` and
`ActiveRecord::ConnectionAdapters::ConnectionPool#uncached`.
Setting `dirties` to `false`, means database writes to the connection
pool will not mark any query caches as dirty.
The option defaults to `true` which retains the existing behaviour and
clears query caches on all connection pools used by the current thread.
Co-authored-by: Jeremy Daer <jeremy@rubyonrails.org>
Replaced by `#lease_connection` to better reflect what it does.
`ActiveRecord::Base#connection` is deprecated in the same way
but without a removal timeline nor a deprecation warning.
Inside the Active Record test suite, we do remove `Base.connection`
to ensure it's not used internally.
Some callsites have been converted to use `with_connection`,
some other have been more simply migrated to `lease_connection`
and will serve as a list of callsites to convert for
https://github.com/rails/rails/pull/50793
Ref: https://github.com/rails/rails/pull/51083
The introduction of `ActiveRecord::Base.with_connection` somewhat broke
some expectations, namely that calling `.connection` would cause the
connection to be permenently leased, hence that future calls to it
would return the same connection, with all it's possible environmental
changes.
So any call to `.connection`, even inside `.with_connection` should
cause the lease to be sticky, and persist beyond the `with_connection`
block.
Also rename `.connection` into `.lease_connection`, as to make it
more explicit.
```ruby
assert_equal "Ruby on Rails", web_sites(:rubyonrails).name
assert_equal "Ruby on Rails", fixture(:web_sites, :rubyonrails).name
```
This was brought to me by someone with a `Metadata` model. The fixtures
accessor being `metadata` which conflicts with the `metadata` method
recently added in `Minitest`.
Extracted from: https://github.com/rails/rails/pull/50793
Right now quoting table or column names requires a leased Adapter
instance, even though none of the implementations actually requires
an active connection.
The idea here is to move these methods to the class so that the quoting
can be done without leasing a connection or even needing the connection
to ever have been established.
I also checked `activerecord-sqlserver-adapter` and `oracle-enhanced`
gems, and neither need an active connection.
Extracted from: https://github.com/rails/rails/pull/50793
Similar to the recent refactoring of schema caches, rather than to directly
hold a connection, they now hold a pool and checkout a connection when needed.
* relation#order supports hash like relation#where
This allows for an ActiveRecord::Relation to take a hash such as
`Topic.includes(:posts).order(posts: { created_at: :desc })`
* use is_a? to support subclasses of each
Co-authored-by: Rafael Mendonça França <rafael@rubyonrails.org>
Ref: https://github.com/rails/rails/pull/50793
To make not caching connection checkout viable, we need to reduced
the amount of places where we need a connection.
Once big source of this is query/relation building, where in many
cases it eagerly quote and interpolation bound values in SQL fragments.
Doing this requires an active connection because both MySQL and Postgres
may quote values differently based on the connection settings.
Instead of eagerly doing all this, we can instead just insert these
as bound values in the Arel AST. For adapters with prepared statements
this is better anyway as it will avoid leaking statements, and for those
that don't support it, it will simply delay the quoting to just
before the query is executed.
However, the `%` API (`where("title = %s", something)`) can't realistically
be fixed this way, but I don't see much value in it and it probably should
be deprecated and removed.
Followup: https://github.com/rails/rails/pull/50938
The behavior changed to always clear the query cache as soon
as it's disabled, on the assumption that once queries have been
performed without it, all bets are off.
However that isn't quite true, we can apply the same clearing
logic than when it's enabled. If you perform read only queries
without the cache, there is no reason to clear it.
Jean Boussier