* Support encrypting binary columns
ActiveRecord Encryption doesn't prevent you from encrypting binary
columns but it doesn't have proper support for it either.
When the data is fed through encrypt/decrypt it is converted to a
String. This means that the the encryption layer is not transparent
to binary data - which should be passed as Type::Binary::Data.
As a result the data is not properly escaped in the SQL queries or
deserialized correctly after decryption.
However it just happens to work fine for MySQL and SQLite because the
MessageSerializer doesn't use any characters that need to be encoded.
However if you try to use a custom serializer that does then it breaks.
PostgreSQL on the other hand does not work - because the Bytea type
is passed a String rather than a Type::Binary::Data to deserialize, it
attempts to unescape the data and either mangles it or raises an error
if it contains null bytes.
The commit fixes the issue, by reserializing the data after
encryption and decryption. For text data that's a no-op, but for binary
data we'll convert it back to a Type::Binary::Data.
* Extract decrypt_as_text/encrypt_as_text
* Handle serialized binary data in encrypted columns
Calling `serialize` is not always possible, because the column type
might not expect to be serializing a String, for example when declared
as serialzed or store attribute.
With binary data the encryptor was passed an
`ActiveModel::Type::Binary::Data`` and returned a `String``. In order to
remain transparent we need to turn the data back into a
`ActiveModel::Type::Binary::Data` before passing it on.
We'll also rename `serialize`` to `text_to_database_type` to be a bit
more descriptive.
Extracted from https://github.com/rails/rails/pull/50793
The leased connection is yielded, and for the duration of the block,
any call to `ActiveRecord::Base.connection` will yield that same connection.
This is useful to perform a few database operations without causing a
connection to be leased for the entire duration of the request or job.
* Deprecate config.active_record.warn_on_records_fetched_greater_than
* Review changes
Co-authored-by: Rafael Mendonça França <rafael@franca.dev>
---------
Co-authored-by: Jason Nochlin <hundredwatt@users.noreply.github.com>
Co-authored-by: Rafael Mendonça França <rafael@franca.dev>
Configure ActiveRecord::Encryption (ARE) on ActiveRecord::Base (AR)
loading, so that ARE configs are ready before AR models start using
`encrypts` to declare encrypted attributes.
This means that you can add ARE configurations in initializers, as long
as you don't trigger the loading of ActiveRecord::Base or your AR models
in prior initializers.
Enums have historically been defined using keyword arguments:
```ruby
class Function > ApplicationRecord
enum color: [:red, :blue],
type: [:instance, :class],
_scopes: false
```
This has the advantage of being able to define multiple enums at once
with the same options. However, it also has a downside that enum options
must be prefixed with an underscore to separate them from the enum
definitions (to enable models to have enums with the same name as an
option).
In Rails 7, a new syntax was [introduced][1] to instead define enums with
positional arguments:
```ruby
class Function > ApplicationRecord
enum :color, [:red, :blue], scopes: false
enum :type, [:instance, :class], scopes: false
```
This new syntax eliminates the need to prefix options with an underscore,
and the docs were updated to recommend this new syntax.
However, both versions of the API have been supported since, and it has
started to cause some problems:
The first issue is that the available options have drifted. In Rails
7.1, an option was added to make assigning an invalid enum value use
validation errors instead of runtime errors. However, the equivalent
underscored prefix option was not added for the original enum syntax
Articles have been created that describe the new option in Rails 7.1,
but the examples in the articles use un-prefixed options with the old
syntax. This confusion has also lead to issues opened asking why that
incorrect syntax is not working.
Additionally, the presence of underscored options is just generally
confusing because it tends to imply an option is for internal use.
This commit aims to fix all of these issues by deprecating the old enum
syntax. With only one way to define enums, options cannot drift and
there will be less confusion around how enums should be defined.
[1]: 0618d2d84a
When set, an `ActiveRecord::InvalidMigrationTimestampError` will be raised if the timestamp
prefix for a migration is more than a day ahead of the timestamp associated with the current time.
This is done to prevent forward-dating of migration files, which can impact migration generation
and other migration commands.
It is turned off by default, but will be turned on for applications starting in Rails 7.2.
As well as `disconnect!` and `verify!`.
This generally isn't a big problem as connections must not be shared between
threads, but is required when running transactional tests or system tests
and could lead to a SEGV.
Add a `compress` option to ActiveRecord::Encryption::Encryptor, which
defaults to `true`. When set to `false`, the encryptor will never
compress the data.
This is useful for cases where the data is already compressed.
This can be used with the `encryptor` option in the model:
```ruby
class Record < ApplicationRecord
encrypts :field, encryptor: ActiveRecord::Encryption::Encryptor.new(compress: false)
end
```
This field returns the amount of rows returned by the query that emitted the notification.
This metric is useful in cases where one wants to detect queries with big result sets.
MySQL 5.7.5+ supports generated columns, which can be used to create a column that is computed from an expression. This commit fixes the escaping of the default value for such expressions if a single quote is included.
See the following for more: https://dev.mysql.com/blog-archive/generated-columns-in-mysql-5-7-5/
Option validation was [added][1] for 7.1+ Migration classes, and a
compatibility layer was added to ensure that previous Migration versions
do not have their options validated. However, the `t.references` method
was missing in the compatibility layer which results in pre 7.1
Migrations validating options passed to `t.references`.
This commit fixes the issue by adding t.references to the compatibility
layer.
See also a [similar fix][2] for `add_reference`
[1]: e6da3ebd6c
[2]: 71b4e22301
Option validation was [added][1] for 7.1+ Migration classes, and
a compatibility layer was added to ensure that previous Migration
versions do not have their options validated. However, the add_reference
method was missing in the compatibility layer which results in pre 7.1
Migrations validating options passed to add_reference.
This commit fixes the issue by adding add_reference to the compatibility
layer. In addition to adding add_reference to the "no validation"
compatibility test, the test is refactored to run against each previous
migration version to ensure that they all behave consistently.
[1]: e6da3ebd6c
Fix: https://github.com/rails/rails/pull/50745
I went a bit farther and handled all the boolean configs, not just `schema_cache`.
Co-Authored-By: Mike Coutermarsh <coutermarsh.mike@gmail.com>
It is the role of the underlying serializer to accept or reject the data
to decrypt depending on its type. This behavior mirrors what is done at
encryption, where the serializer asserts that the input is an
ActiveRecord::Encryption::Message.
This change allows for a wider variety of custom serializers, but does
not change the behavior when using the default MessageSerializer class.
Indeed, the default message serializer will raise a TypeError when
invoking JSON.parse on any non-String input. This error will subsequently
be translated into an ActiveRecord::Encryption::Errors::Encoding error
by the encryptor, which does not change the current behavior at the
encryptor level.
A new test asserts that the default MessageSerializer is able to reject
unexpected data types on its own at decryption time, just as it does at
encryption time (test already present). The test also asserts that an
exception is translated into an ActiveRecord::Encryption::Error::Encoding
error at the encryptor level.
Also fix implementation of TestEncryptor#encrypted?
The assertions fail because encrypted_attribute? delegates a decryption
attempt to the default encryptor (instead of the one configured) to
check if the value is actually encrypted.
Adds missing punctuation, revises grammar, updates typography, etc.
This follows our documentation guidelines, and makes the CHANGELOG
consistent in style.
`explain` can be called on a relation to explain how the database would
execute a query. Currently `explain` doesn't work for queries using
`last`, `pluck` or `count`, as these return the actual result instead of
a relation. This makes it difficult to optimize these queries.
By letting `explain` return a proxy instead, we can support these
methods.
```ruby
User.all.explain.count
\# => "EXPLAIN SELECT COUNT(*) FROM `users`"
User.all.explain.maximum(:id)
\# => "EXPLAIN SELECT MAX(`users`.`id`) FROM `users`"
```
This breaks the existing behaviour in that it requires calling inspect
after explain.
```ruby
User.all.explain.inspect
\# => "EXPLAIN SELECT `users`.* FROM `users`"
```
However, as `explain` is mostly used from the commandline, this won't be a
problem as inspect is called automatically in IRB.
Co-authored-by: Rafael Mendonça França <rafael@rubyonrails.org>
The docs imply that when specified the `on` option of
`validates_associated` should be respected by the validated association.
Prior to this change the validation context used will simply be the
default behavior as if no `on` option were specified. With this change,
the validation_context of the parent record is passed to the `valid?`
check.
When associations are created or updated allow them to use their own
default context according to their own persistence status rather than
overriding with the context of their associated object.
Fixes#46239
Co-authored-by: Alex Ghiculescu <alex@tanda.co>
Co-authored-by: Rafał Brize <rafal@buyapowa.com>
* Allow overriding SQLite defaults from `database.yml`
Any PRAGMA configuration set under the `pragmas` key in the configuration file take precedence over Rails' defaults, and additional PRAGMAs can be set as well.
```yaml
database: storage/development.sqlite3
timeout: 5000
pragmas:
synchronous: full
temp_store: memory
```
* Style
* Allow overriding SQLite defaults from `database.yml`
Any PRAGMA configuration set under the `pragmas` key in the configuration file take precedence over Rails' defaults, and additional PRAGMAs can be set as well.
```yaml
database: storage/development.sqlite3
timeout: 5000
pragmas:
synchronous: full
temp_store: memory
```
---------
Co-authored-by: David Heinemeier Hansson <david@hey.com>
There are valid use cases for running SQLite in production, however it must be done
with care, so instead of a warning most users won't see anyway, it's preferable to
leave the configuration commented out to force them to think about having the database
on a persistent volume etc.
Co-Authored-By: Jacopo Beschi <beschi.jacopo@gmail.com>
Generated columns (both stored and dynamic) are supported since version 3.31.0 of SQLite.
This adds support for those to the SQLite3 adapter.
```ruby
create_table :users do |t|
t.string :name
t.virtual :name_upper, type: :string, as: 'UPPER(name)'
t.virtual :name_lower, type: :string, as: 'LOWER(name)', stored: true
end
```
Contrary to mysql2, trilogy will ignore the `socket` config
if `host` is set.
This makes it impossible to configure a UNIX domain socket connection
via `DATABASE_URL`, as the URL must include a host.
To assert the expected number of queries are made, Rails internally uses
`assert_queries` and `assert_no_queries`. These assertions can be
useful in applications as well.
By extracting these assertions to a module, the assertions can be
included where required.
These assertions are added to `ActiveSupport::TestCase` when
ActiveRecord is defined.
ActiveStorage, ActionView and ActionText are using this module now as
well, instead of duplicating the implementation.
The internal ActiveRecord::TestCase, used for testing ActiveRecord,
implements these assertions as well. However, these are slighlty more
advanced/complex and use the SQLCounter class. To keep things simple,
for now this implementation isn't used.
Follow-up to #49146
The original behavior of `has_secure_token` was to use the
`send("#{attribute}=", some_value)` method so that the setter method, if
defined, was called. PR #49146 replaced the `send` method with
`write_attribute` which doesn't call the setter method and breaks
existing applications.
Rafael Mendonça França