Foreign keys could be created to the same table.
So `remove_foreign_key :from_table, :to_table` is sometimes ambiguous.
This allows `remove_foreign_key` to remove the select one on the same
table with giving both `to_table` and `options`.
This change adds a new method that loads the YAML for the database
config without parsing the ERB. This may seem odd but bear with me:
When we added the ability to have rake tasks for multiple databases we
started looping through the configurations to collect the namespaces so
we could do `rake db:create:my_second_db`. See #32274.
This caused a problem where if you had `Rails.config.max_threads` set in
your database.yml it will blow up because the environment that defines
`max_threads` isn't loaded during `rake -T`. See #35468.
We tried to fix this by adding the ability to just load the YAML and
ignore ERB all together but that caused a bug in GitHub's YAML loading
where if you used multi-line ERB the YAML was invalid. That led us to
reverting some changes in #33748.
After trying to resolve this a bunch of ways `@tenderlove` came up with
replacing the ERB values so that we don't need to load the environment
but we also can load the YAML.
This change adds a DummyCompiler for ERB that will replace all the
values so we can load the database yaml and create the rake tasks.
Nothing else uses this method so it's "safe".
DO NOT use this method in your application.
Fixes#35468
Since #23461, all adapters supports prepared statements, so that clears
the prepared statements cache is no longer database specific.
Actually, I struggled to identify the cause of random CI failure in
#23461, that was missing `@statements.clear` in `clear_cache!`.
This extracts `clear_cache!` to ensure the common concerns in the
abstract adapter.
Adds a method to ActiveRecord allowing records to be inserted in bulk without instantiating ActiveRecord models. This method supports options for handling uniqueness violations by skipping duplicate records or overwriting them in an UPSERT operation.
ActiveRecord already supports bulk-update and bulk-destroy actions that execute SQL UPDATE and DELETE commands directly. It also supports bulk-read actions through `pluck`. It makes sense for it also to support bulk-creation.
* Add `ActiveRecord::Base.connection.truncate` for SQLite3 adapter.
SQLite doesn't support `TRUNCATE TABLE`, but SQLite3 adapter can support
`ActiveRecord::Base.connection.truncate` by using `DELETE FROM`.
`DELETE` without `WHERE` uses "The Truncate Optimization",
see https://www.sqlite.org/lang_delete.html.
* Add `rails db:seed:replant` that truncates database tables and loads the seeds
Closes#34765
#35360 allows table name qualified if `from` has original table name.
But that is still too strict. We have a valid use case that `from` with
INDEX hint (e.g. `from("comments USE INDEX (PRIMARY)")`).
So I've relaxed the table name detection in `from` to allow any
extension like INDEX hint.
Fixes#35359.
It's mentioned everywhere as `ActiveRecord::RecordNotFound` so to be
coherent with the rest of the documentation I've applied it here.
Also doc was saying if the parameter is integer it coerces it which is
other way around.
When the middle association doesn't have any records and the inner
association is not an empty scope the owner will be `nil` so we can't
try to reset the inverse association.
Caching `find_by` statements on STI subclasses is unsafe, since
`type IN (?,?,?,?)` part is dynamic, and we don't have SQL statements
cache invalidation when a STI subclass is created or removed for now.
When using `select` with `'DISTINCT( ... )'` if you use method `size` on a non loaded relation it overrides the column selected by passing `:all` so it returns different value than count.
This fixes#35214
Related cbcdecd, 2a56b2d.
This is a regression caused by cbcdecd.
If query caching is enabled, prepared statement handles are never
re-used, since we missed that a query is preprocessed when query caching
is enabled, but doesn't keep the `preparable` flag.
We should care about that case.
* The READ_QUERY regex would consider reads to be writes if they started with
spaces or parens. For example, a UNION query might have parens around each
SELECT - (SELECT ...) UNION (SELECT ...).
* It will now correctly treat these queries as reads.
Incrementing the lock version invalidates any other process's optimistic
lock, which is the desired outcome: the record no longer looks the same
as it did when they loaded it.
This is caused by 0ee96d1.
Since #18744, `select` columns doesn't be qualified by table name if
using `from`. 0ee96d1 follows that for `pluck` as well.
But people depends that `pluck` columns are qualified even if using
`from`.
So I've fixed that to be qualified if `from` has the original table name
to keep the behavior as much as before.
Fixes#35359.
Revert "Fix lint `ShadowingOuterLocalVariable`"
This reverts commit 38bd45a48992b500478a82d56d31468a322937a8.
Change of variable name
Fix lint `ShadowingOuterLocalVariable`
Using the subscript method `#[]` on a string has several overloads and
rather complex implementation. One of the overloads is the capability to
accept a regular expression and then run a match, then return the
receiver (if it matched) or one of the groups from the MatchData.
The function of the `UUID#cast` method is to cast a UUID to a type and
format acceptable by postgres. Naturally UUIDs are supposed to be
string and of a certain format, but it had been determined that it was
not ideal for the framework to send just any old string to Postgres and
allow the engine to complain when "foobar" or "" was sent, being
obviously of the wrong format for a valid UUID. Therefore this code was
written to facilitate the checking, and if it were not of the correct
format, a `nil` would be returned as is conventional in Rails.
Now, the subscript method will allocate one or more strings on a match
and return one of them, based on the index parameter. However, there
is no need for a new string, as a UUID of the correct format is already
such, and so long as the format was verified then the string supplied is
adequate for consumption by the database.
The subscript method also creates a MatchData object which will never be
used, and so must eventually be garbage collected.
Garbage collection indeed. This innocuous method tends to be called
quite a lot, for example if the primary key of a table is a uuid, then
this method will be called. If the foreign key of a relation is a UUID,
once again this method is called. If that foreign key is belonging to
a has_many relationship with dozens of objects, then again dozens of
UUIDs shall be cast to a dup of themselves, and spawn dozens of
MatchData objects, and so on.
So, for users that:
* Use UUIDs as primary keys
* Use Postgres
* Operate on collections of objects
This accomplishes a significant savings in total allocations, and may
save many garbage collections.
The special case happens when through association has a custom scope
that is applied to the source association when loading.
In this case, the soucre association would need to be reset after
main association is loaded. See tests.
The special case exists when a through association has
That is considered as silently leaking information.
If type casting doesn't return any actual value, it should not be
matched to any record.
Fixes#33624.
Closes#33946.
This fixes a bug that the `foreign_key` and the `foreign_type` are
separated as different table conditions if a polymorphic association has
a scope that joins another tables.
The CI failure for `test_errors_for_bigint_fks_on_integer_pk_table` is
due to the poor regex that extract all ``` `(\w+)` ``` like parts from
the message (`:foreign_key` should be `"old_car_id"`, but `"engines"`):
https://travis-ci.org/rails/rails/jobs/494123455#L1703
I've improved the regex more strictly and have more exercised mismatched
foreign key tests.
Fixes#35294
This deprecates using class level querying methods if the receiver scope
regarded as leaked, since #32380 and #35186 may cause that silently
leaking information when people upgrade the app.
We need deprecation first before making those.
This reverts commit b67d5c6ded, reversing
changes made to 2e018361c7.
Reason: #35186 may cause that silently leaking information when people
upgrade the app.
We need deprecation first before making this.
In chat Sam Saffron asked how to use the setter now that configurations
is no longer a hash and you need to do AR::Base.configurations["test"]=.
Technically you can do `ActiveRecord::Base.configurations = { the hash
}` but I realized the old way throws an error and is unintuitive.
To aid in the transition from hashes to objects this PR makes a few
changes:
1) Re-adds a deprecated hash setter `[]=` that will add a new hash
to the configurations list OR replace an existing hash if that
environment is already present. This won't be supported in future Rails
versions but a good error is important.
2) Changed to throw deprecation warnings on the methods we decided to support
for hash conversion and raise on the methods we don't support.
3) Refactored the setter/getter hash deprecation warnings messages and
rewrote them.
Getters message:
```
DEPRECATION WARNING: `ActiveRecord::Base.configurations` no longer
returns a hash. Methods that act on the hash like `values` are
deprecated and will be removed in Rails 6.1. Use the `configs_for`
method to collect and iterate over the database configurations.
```
Setter message:
```
DEPRECATION WARNING: Setting `ActiveRecord::Base.configurations` with
`[]=` is deprecated. Use `ActiveRecord::Base.configurations=` directly
to set the configurations instead.
```
4) Rewrote the legacy configurations test file to test all the public
methods in the DatabaseConfigurations class.
This reverts commit eec3e28a1a, reversing
changes made to 5588fb4802.
Reason: Marking as loaded without actual loading is too greedy optimization.
See more context #35239.
Closes#35239.
[Edouard CHIN & Ryuta Kamizono]
Currently custom attributes are always qualified by the table name in
the generated SQL wrongly even if the table doesn't have the named
column, it would cause an invalid SQL error.
Custom attributes should only be qualified if the table has the same
named column.
The implicit delegation in the migration class is to be logged. It is
not intended in the migration compatibility, so it should be avoided.
Fixes#35224.
I implemented Foreign key create in `create_table` for SQLite3 at
#24743. This follows #24743 to implement `add_foreign_key` and
`remove_foreign_key`.
Unfortunately SQLite3 has one limitation that
`PRAGMA foreign_key_list(table-name)` doesn't have constraint name.
So we couldn't implement find/remove foreign key by name for now.
Fixes#35207.
Closes#31343.
The docs for `ActiveRecord::Associations::CollectionProxy` describe
`ActiveRecord::Associations::Association`. I've moved them to
association and rewrote collection proxy's docs to be more applicable to
what the class actually does.`
[ci skip]
`Association#target=` invokes `loaded!`, so we no longer need to call
the `loaded!` explicitly.
Since Preloader is private API, we don't guarantee that it behaves like
older version as long as using Preloader directly. But this refactoring
fortunately also fix the Preloader compatibility issue #35195.
Closes#35195.
I've found a few places in Rails code base where I think it makes sense
to calculate elapsed time more precisely by using
`Concurrent.monotonic_time`:
- Fix calculation of elapsed time in `ActiveSupport::Cache::MemoryStore#prune`
- Fix calculation of elapsed time in
`ActiveRecord::ConnectionAdapters::ConnectionPool::Queue#wait_poll`
- Fix calculation of elapsed time in
`ActiveRecord::ConnectionAdapters::ConnectionPool#attempt_to_checkout_all_existing_connections`
- Fix calculation of elapsed time in `ActiveRecord::ConnectionAdapters::Mysql2Adapter#explain`
See
https://docs.ruby-lang.org/en/2.5.0/Process.html#method-c-clock_gettimehttps://blog.dnsimple.com/2018/03/elapsed-time-with-ruby-the-right-way
Related to 7c4542146f
The `distinct` affects (reduces) rows of the result, so it is important
part when both `distinct` and `offset` are given.
Replacing SELECT clause to `1 AS one` and removing `distinct` and
`order` is just optimization for the `exists?`, we should not apply the
optimization for that case.
Fixes#35191.
`relation.create` populates scope attributes to new record by `scoping`,
it is necessary to assign the scope attributes to the record and to find
STI subclass from the scope attributes.
But the effect of `scoping` is class global, it was caused undesired
behavior that pollute all class level querying methods in initialization
block and callbacks (`after_initialize`, `before_validation`,
`before_save`, etc), which are user provided code.
To avoid the leaking scope issue, restore the original current scope
before initialization block and callbacks are invoked.
Fixes#9894.
Fixes#17577.
Closes#31526.
This is more consistent with Resolver, which has build called. This
allows using a Proc instead of a class, which could be nice if you need
to vary switching logic based on the request in a more ad-hoc way (ie.
check if it is an API request).
This commit speeds up rendering partials by caching the variable name
calculation on the template. The variable name is based on the "virtual
path" used for looking up the template. The same virtual path
information lives on the template, so we can just ask the cached
template object for the variable.
This benchmark takes a couple files, so I'll cat them below:
```
[aaron@TC ~/g/r/actionview (speed-up-partials)]$ cat render_benchmark.rb
require "benchmark/ips"
require "action_view"
require "action_pack"
require "action_controller"
class TestController < ActionController::Base
end
TestController.view_paths = [File.expand_path("test/benchmarks")]
controller_view = TestController.new.view_context
result = Benchmark.ips do |x|
x.report("render") do
controller_view.render("many_partials")
end
end
[aaron@TC ~/g/r/actionview (speed-up-partials)]$ cat test/benchmarks/test/_many_partials.html.erb
Looping:
<ul>
<% 100.times do |i| %>
<%= render partial: "list_item", locals: { i: i } %>
<% end %>
</ul>
[aaron@TC ~/g/r/actionview (speed-up-partials)]$ cat test/benchmarks/test/_list_item.html.erb
<li>Number: <%= i %></li>
```
Benchmark results (master):
```
[aaron@TC ~/g/r/actionview (master)]$ be ruby render_benchmark.rb
Warming up --------------------------------------
render 41.000 i/100ms
Calculating -------------------------------------
render 424.269 (± 3.5%) i/s - 2.132k in 5.031455s
```
Benchmark results (this branch):
```
[aaron@TC ~/g/r/actionview (speed-up-partials)]$ be ruby render_benchmark.rb
Warming up --------------------------------------
render 50.000 i/100ms
Calculating -------------------------------------
render 521.862 (± 3.8%) i/s - 2.650k in 5.085885s
```
Active Record uses `scoping` to delegate to named scopes from relations
for propagating the chaining source scope. It was needed to restore the
source scope in named scopes, but it was caused undesired behavior that
pollute all class level querying methods.
Example:
```ruby
class Topic < ActiveRecord::Base
scope :toplevel, -> { where(parent_id: nil) }
scope :children, -> { where.not(parent_id: nil) }
scope :has_children, -> { where(id: Topic.children.select(:parent_id)) }
end
# Works as expected.
Topic.toplevel.where(id: Topic.children.select(:parent_id))
# Doesn't work due to leaking `toplevel` to `Topic.children`.
Topic.toplevel.has_children
```
Since #29301, the receiver in named scopes has changed from the model
class to the chaining source scope, so the polluting class level
querying methods is no longer required for that purpose.
Fixes#14003.
With this benchmark:
require "bundler/setup"
require "active_record"
require "benchmark/ips"
# This connection will do for database-independent bug reports.
ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")
ActiveRecord::Schema.define do
create_table :posts, force: true do |t|
end
end
class Post < ActiveRecord::Base
end
new_post = Post.new
Benchmark.ips do |b|
b.report("present?") do
new_post.present?
end
b.report("blank?") do
new_post.blank?
end
end
Before:
Warming up --------------------------------------
present? 52.147k i/100ms
blank? 53.077k i/100ms
Calculating -------------------------------------
present? 580.184k (±21.8%) i/s - 2.555M in 5.427085s
blank? 601.537k (± 9.2%) i/s - 2.972M in 5.003503s
After:
Warming up --------------------------------------
present? 378.235k i/100ms
blank? 375.476k i/100ms
Calculating -------------------------------------
present? 17.381M (± 7.5%) i/s - 86.238M in 5.001815s
blank? 17.877M (± 6.4%) i/s - 88.988M in 5.004634s
This improvement is mostly because those methods were hitting
`method_missing` on a lot of levels to be able to return the value.
To avoid all this stack walking we are short-circuiting those methods.
Closes#35059.
This change ensures that all query cahces are cleared across all
connections per handler for the current thread so if you write on one
connection the read will have the query cache cleared.
When I wrote the `connected_to` and `connects_to` API's I wrote them
with the idea in mind that it didn't really matter what the
handlers/roles were called as long as those connecting to the roles knew
which one wrote and which one read.
With the introduction of the middleware Rails begins to assume it's
`writing` and `reading` and there's no room for other roles. At GitHub
we've been using this method for a long time so we have a ton of legacy
code that uses different handler names `default` and `readonly`. We
could rename all our code but I think this is better for a few reasons:
- Legacy apps that have been using multiple databases for a long time
can have an eaiser time switching.
- If we later find this to cause more issues than it's worth we can
easily deprecate.
- We won't force old apps to rewrite the resolver middleware just to use
a different handler.
Adding the writing_role/reading_role required that I move the code that
creates the first handler for writing to the railtie. If I didn't move
this the core class would assign the handler before I was able to assign
a new one in my configuration and I'd end up with 3 handlers instead of
2.
Right now we only have one option that's supported, the delay. However I
can see us supporting other options in the future.
This PR refactors the options to get passed into the resolver so whether
you're using middleware or using the config options you can pass options
to the resolver. This will also make it easy to add new options in the
future.
We sometimes display simple examples of additional parameters that can be
supplied to table-wise methods like these and I found it particularly difficult
to figure out which options `t.foreign_key` accepts without drilling very deep
into the specific SchemaStatements docs.
Since it's relatively common to create foreign keys with custom column names or
primary keys, it seems like this should help quite a few people.
[ci skip]
We need to update using the timestamp from the end of the request, not
the start. For example, if a request spends 5+ seconds writing, we still
want to wait another 5 seconds for replication lag.
Since we now run the update after we yield, we need to use ensure to
make sure we update the timestamp even if there is an exception.
The following PR adds behavior to Rails to allow an application to
automatically switch it's connection from the primary to the replica.
A request will be sent to the replica if:
* The request is a read request (`GET` or `HEAD`)
* AND It's been 2 seconds since the last write to the database (because
we don't want to send a user to a replica if the write hasn't made it
to the replica yet)
A request will be sent to the primary if:
* It's not a GET/HEAD request (ie is a POST, PATCH, etc)
* Has been less than 2 seconds since the last write to the database
The implementation that decides when to switch reads (the 2 seconds) is
"safe" to use in production but not recommended without adequate testing
with your infrastructure. At GitHub in addition to the a 5 second delay
we have a curcuit breaker that checks the replication delay
and will send the query to a replica before the 5 seconds has passed.
This is specific to our application and therefore not something Rails
should be doing for you. You'll need to test and implement more robust
handling of when to switch based on your infrastructure. The auto
switcher in Rails is meant to be a basic implementation / API that acts
as a guide for how to implement autoswitching.
The impementation here is meant to be strict enough that you know how to
implement your own resolver and operations classes but flexible enough
that we're not telling you how to do it.
The middleware is not included automatically and can be installed in
your application with the classes you want to use for the resolver and
operations passed in. If you don't pass any classes into the middleware
the Rails default Resolver and Session classes will be used.
The Resolver decides what parameters define when to
switch, Operations sets timestamps for the Resolver to read from. For
example you may want to use cookies instead of a session so you'd
implement a Resolver::Cookies class and pass that into the middleware
via configuration options.
```
config.active_record.database_selector = { delay: 2.seconds }
config.active_record.database_resolver = MyResolver
config.active_record.database_operations = MyResolver::MyCookies
```
Your classes can inherit from the existing classes and reimplment the
methods (or implement more methods) that you need to do the switching.
You only need to implement methods that you want to change. For example
if you wanted to set the session token for the last read from a replica
you would reimplement the `read_from_replica` method in your resolver
class and implement a method that updates a new timestamp in your
operations class.
Previously if the `url` key in a config hash was nil we'd ignore the
configuration as invalid. This can happen when you're relying on a
`DATABASE_URL` in the env and that is not set in the environment.
```
production:
<<: *default
url: ENV['DATABASE_URL']
```
This PR fixes that case by checking if there is a `url` key in the
config instead of checking if the `url` is not nil in the config.
In addition to changing the conditional we then need to build a url hash
to merge with the original hash in the `UrlConfig` object.
Fixes#35091
The transaction used to restore fixtures is an implementation detail
that should be abstracted away. Idealy a test should behave the same
wether or not transactional fixtures are enabled.
However since transactions have been made lazy, the fixture
transaction started leaking into tests case. e.g. consider the
following (oversimplified) test:
```ruby
class SQLSubscriber
attr_accessor :sql
def initialize
@sql = []
end
def call(*, event)
sql << event[:sql]
end
end
subscriber = SQLSubscriber.new
ActiveSupport::Notifications.subscribe("sql.active_record", subscriber)
User.connection.execute('SELECT 1', 'Generic name')
assert_equal ['SELECT 1'], subscriber.sql
```
On Rails 6 it starts to break because the `sql` array will be `['BEGIN', 'SELECT 1']`.
Several things are wrong here:
- That transaction is not generated by the tested code, so it shouldn't be visible.
- The transaction is not even closed yet, which again doesn't reflect the reality.
In MySQL, the text column size is 65,535 bytes by default (1 GiB in
PostgreSQL). It is sometimes too short when people want to use a text
column, so they sometimes change the text size to mediumtext (16 MiB) or
longtext (4 GiB) by giving the `limit` option.
Unlike MySQL, PostgreSQL doesn't allow the `limit` option for a text
column (raises ERROR: type modifier is not allowed for type "text").
So `limit: 4294967295` (longtext) couldn't be used in Action Text.
I've allowed changing text and blob size without giving the `limit`
option, it prevents that migration failure on PostgreSQL.
While working on another feature for multiple databases (auto-switching)
I observed that in development the first request won't autoload the
application record connection for the primary database and may not yet
know about the replica connection.
In my test application this caused the application to thrown an error if
I tried to send the first request to the replica before the replica was
connected. This wouldn't be an issue in production because the
application is preloaded.
In order to fix this I decided to leave the original error message and
delete the new error message. I updated the original error message to
include the `role` to make it a bit clearer that the connection isn't
established for that particular role.
The error now reads:
```
No connection pool with 'primary' found for the 'reading' role.
```
A single database application will continue uisng the original error
message:
```
No connection pool with 'primary' found.
```
Allows aliasing, predications, ordering, and various other functions on `And` and `Case` nodes. This brings them in line with other nodes like `Binary` and `Unary`.
Currently `conn.column_exists?("testings", "created_at", "datetime")`
returns false even if the table has the `created_at` column.
That reason is that `column.type` is a symbol but passed `type` is not
normalized to symbol unlike `column_name`, it is surprising behavior to
me.
I've improved that to normalize a value before comparison.
Since #31230, `change_column` is executed as a bulk statement.
That caused incorrect type casting column default by looking up the
before changed type, not the after changed type.
In a bulk statement, we can't use `change_column_default_for_alter` if
the statement changes the column type.
This fixes the type casting to use the constructed target sql_type.
Fixes#34938.
Since 31ffbf8d, finder methods no longer raise `RangeError`. So
`StatementCache#execute` is the only place to raise the exception for
finder queries.
`StatementCache` is used for simple equality queries in the codebase.
This means that if `StatementCache#execute` raises `RangeError`, the
result could always be regarded as empty.
So `StatementCache#execute` just return nil in that range error case,
and treat that as empty in the caller side, then we can avoid catching
the exception in much places.
Ryuta Kamizono