A fairly common mistake with Rails is to enqueue a job from inside a
transaction, with a record as argumemnt, which then lead to a RecordNotFound
error when picked up by the queue.
This is even one of the arguments advanced for job runners backed by the
database such as `solid_queue`, `delayed_job` or `good_job`.
But relying on this is undesirable in my opinion as it makes the Active Job
abstraction leaky, and if in the future you need to migrate to another backend
or even just move the queue to a separate database, you may experience a lot of
race conditions of the sort.
To resolve this problem globally, we can make Active Job optionally transaction
aware, and automatically defer job queueing to `after_commit`.
Co-Authored-By: Cristian Bica <cristian.bica@gmail.com>
Durations that were round-tripped through ActiveJob::Arguments.serialize
would appear fine at a first glance, but trying to perform
duration-math on them would fail:
```
irb(main):001:0> d = ActiveJob::Arguments.deserialize(ActiveJob::Arguments.serialize([1.year]))[0]
=> 1 year
irb(main):002:0> d + 1.day
activesupport-6.1.4.4/lib/active_support/duration.rb:242:in `+': undefined method `merge' for [[:years, 1]]:Array (NoMethodError)
```
This has utility for gems/modules included on jobs, which can tie into this behaviour and run something when a job fails.
after_discard respects the existing retry behaviour, but will run even if a retried exception is handled in a block.
This gives queue adapters more freedom to name and organize their code.
For example, if `FancyQueue` wants to have their adapter at
`FancyQueue::ActiveJobAdapter`, the name would be `active_job` before this
change. After this change, they can implement `queue_adapter_name` to return
`fancy_queue`.
Sidekiq has a useful optimisation called `push_bulk` that enqueues many jobs at
once, eliminating the repeated Redis roundtrips. However, this feature is not
exposed through Active Job, so it only works for `Sidekiq::Worker` jobs. This
adds a barrier to Active Job adoption for apps that rely on this feature. It
also makes it harder for other queue adapters to implement similar
functionality, as they then have to take care of serialization, callbacks, etc.
themselves.
This commit adds `ActiveJob.perform_all_later(<job1>, <job2>)`, backed by
Sidekiq's `push_bulk` and with a fallback to enqueuing serially if the queue
adapter does not support bulk enqueue.
The performance benefit for 1000 jobs can be more than an order of magnitude:
| Enqueue type | Serial time (ms) | Bulk time (ms) | Speedup |
| ------------------ | ---------------- | -------------- | ------- |
| Raw Sidekiq | 2661 | 119 | 22x |
| Active Job Sidekiq | 2853 | 208 | 14x |
(Measured in a simple test app in our production environment.)
Instrumentation for perform_all_later uses a new event `enqueue_all.active_job`
Fixes https://github.com/rails/rails/issues/46103
An issue exists if you set `config.active_record.query_log_tags` to an array that includes `:controller`, `:action`, or `:job`; the relevant item will get duplicated in the log line. This occured because the relevant railties would add the item to `config.active_record.query_log_tags` again during setup. This PR fixes that by only adding those items to the config if they aren't already set.
The issue proposed more documentation to work around this, but I think it's a bug and should be fixed directly.
Previously, BigDecimal was listed as not needing a serializer. However,
when used with an adapter storing the job arguments as JSON, it would get
serialized as a simple String, resulting in deserialization also producing
a String (instead of a BigDecimal).
By using a serializer, we ensure the round trip is safe.
During upgrade deployments of applications with multiple replicas making use of
BigDecimal job arguments with a queue adapter serializing to JSON, there exists
a possible race condition, whereby a "new" replica enqueues a job with an
argument serialized using `BigDecimalSerializer`, and an "old" replica fails to
deserialize it (as it does not have `BigDecimalSerializer`).
Therefore, to ensure safe upgrades, serialization will not use
`BigDecimalSerializer` until `config.active_job.use_big_decimal_serializer` is
enabled, which can be done safely after successful deployment of Rails 7.1.
This option will be removed in Rails 7.2, when it will become the default.
This adds `:db_runtime` to `perform.active_job` notification payloads,
which is the total time taken by database queries while performing a
job. This value can be used to better understand how a job's time is
spent.
This is similar to the `:db_runtime` value added to
`process_action.action_controller` notification payloads by
`ActiveRecord::Railties::ControllerRuntime`.
Closes#35354.
Co-authored-by: Cory Gwin <gwincr11@github.com>
The ModuleSerializer does not support serializing anonymous classes
because when we try to deserialize the anonymous class, it wouldn't
know which class to use (since class name is nil).
For this reason, ModuleSerialzier now raises an error if the class
name is nil. Previously, ModuleSerializer would raise an `undefined
method `constantize' for nil:NilClass` error during deserialization.
It's not clear why the deserialization failed from the error.
In this commit, we raise an explicit error when trying to serialize
an anonymous class indicating this behaviour is not supported.
Previously in `perform_enqueued_jobs`, `deserialize_arguments_if_needed`
was called before calling `perform_now`. When a record no longer exists
and is serialized using GlobalID this led to raising
an `ActiveJob::DeserializationError` before reaching `perform_now` call.
This behaviour makes difficult testing the job `discard_on/retry_on` logic.
Now `deserialize_arguments_if_needed` call is postponed to when `perform_now`
is called.
Example:
```ruby
class UpdateUserJob < ActiveJob::Base
discard_on ActiveJob::DeserializationError
def perform(user)
# ...
end
end
User.destroy_all
assert_nothing_raised do
perform_enqueued_jobs only: UpdateUserJob
end
assert_no_enqueued_jobs
```
Before this changes the test will fail, now it passes.
In some applications, some classes of errors may be raised during the
execution of a job which the developer would want to retry forever.
These classes of errors would most likely be infrastructure problems that
the developer knows will be resolved eventually but may take a variable
amount of time or errors where due to application business logic, there
could be something temporarily blocking the job from executing, like a
resource that is needed for the job being locked for a lengthy amount of
time.
While an arbitrarily large number of attempts could previously be passed,
this is inexpressive as sometimes the developer may just need the job to
continue to be retried until it eventually succeeds. Without this,
developers would need to include additional code to handle the situation
where the job eventually fails its attempts limit and has to be re-enqueued
manually.
As with many things this should be used with caution and only for errors
that the developer knows will definitely eventually be resolved, allowing
the job to continue.
[Daniel Morton + Rafael Mendonça França]
Rafael Mendonça França