class SensitiveJob < ApplicationJob
self.log_arguments = false
def perform(my_sensitive_argument)
end
end
When dealing with sensitive arugments as password and tokens it is
now possible to configure the job to not put the sensitive argument
in the logs.
Closes#34438.
Using `self` works by essentially pointing to the included class, in
this case `ActiveJob::Base`, so that'd end up delegating to
`ActiveJob::Base.queue_adapter` instead.
Currently `wrapped` is a String. Instead pass Class so that Sidekiq can get access to the ActiveJob and its configuration without having to constantize. I verified that `JobClass.to_s` == `"JobClass"` so serialization to JSON should not be affected.
```ruby
> JSON.dump({"wrapped" => SomeJob})
=> "{\"wrapped\":\"SomeJob\"}"
```
By having access to the ActiveJob class, Sidekiq can get access to any `sidekiq_options` which have been set on that ActiveJob type and serialize those options into Redis.
https://github.com/mperham/sidekiq/blob/master/Changes.md#60
This commit allows ActiveJob to serialize classes and modules without a custom serializer. This allows for workflows like:
```
class EmailJob < ApplicationJob
queue_as :default
def perform(template_class, *arguments)
template_class.new(*arguments).send!
end
end
module Email
class FooTemplate ... end
class BarTemplate ... end
end
EmailJob.perform_later(Email::FooTemplate, ...)
EmailJob.perform_later(Email::BarTemplate, ...)
```
Currently this is only achieveable through a custom serializer or through constantizing in each instance.
The `Sidekiq::Logging` module was removed in Sidekiq 6.0 in favour of
`Sidekiq.logger=`.
https://github.com/mperham/sidekiq/wiki/Logging#api-changes
> The old Sidekiq::Logging class was removed in Sidekiq 6.0. You should
> not be accessing any Sidekiq internal classes to use the logger. Use
> `logger` directly in your Worker or use `Sidekiq.logger` elsewhere.
In #36864 we made a change to an ActiveJob test helper to add a new
parameter for testing jobs delayed into the future.
This commit adds a CHANGELOG entry for that change as well as fixing a
circular argument error.
Currently, the `perform_enqueued_jobs` helpers will also immediately
perform jobs that are scheduled via `set(wait:)` to run in the future.
This commit adds a new argument to `perform_enqueued_jobs` to make it
only run jobs scheduled at or before the passed in `Time`. This allows
testing the side effects of immediate job execution separate of jobs
delayed in the future.
`assert_enqueued_with` and `assert_performed_with` were not able to
properly test jobs with relative delay. `:at` option was asserted for
equality and test will always fail cause small fraction of time will
pass between job call and assertion.
This commit fixes that by droping microseconds from `:at` argument
assertions.
Rails 6 introduces retries per-exception, instead of a global count of
retries. Because ActiveJob 5.2 doesn't serialize the execution count
per-exception, when ActiveJob 6.0 picks up an "old" job it can't know
the exception count in the new format.
This can also be an issue if AJ 6.0 serializes a new job with
exception_executions which is later picked up by AJ 5.2, which would
clear exception_executions (since it has no knowledge of it).
Previously we handled this by resetting exception_executions, if it
wasn't defined on a job, which could result in the worst case retrying
the job 2x the times we should.
This commit changes how we handle loading a legacy job: instead of
resetting exception_executions, we instead will always use the global
executions count.
This way, jobs which only have one retry_on (and didn't have a behaviour
change in AJ 6) are backwards-and-forwards-compatible with counts
respected exactly.
Jobs with multiple retry_on will revert to the AJ5.2 behaviour if they
were ever run under AJ5.2.
Individual execution counters were introduced in #34352. However
`#determine_delay` which is used to calculate retry delay still uses the
global counter. This commit fixes it.
Motivation:
- Currently we have 2 seperate monkey patches in place for tracking
enqueded time for 2 seperate workers. It seems that activejob could be
a source of truth for how long an item has been enqued so that we can
easily use it for consistent monitoring across workers/apps to ensure
that jobs are running at an acceptable speed.
Changes:
- Add an enqueded at attribute and serilization tooling.
- Add a method to get how long a job has been enqueded for.
- Add a logging item to show how long a job was enqued prior to the
perform method firing.
Currently, we only document the use case for ActiveJob's `queue_as` block
arguments in [Rails Guides][1]. It seems necessary to document them in the API
docs as well considering how useful and powerful this option is.
[1]: https://edgeguides.rubyonrails.org/active_job_basics.html#queues
[ci skip]
We had a discussion on the Core team and we don't want to expose this information
as a JSON endpoint and not by default.
It doesn't make sense to expose this JSON locally and this controller is only
accessible in dev, so the proposed access from a production app seems off.
This reverts commit 8eaffe7e89, reversing
changes made to b6e4305c3b.
Some adapters like Resque that use Redis, convert the Ruby hash with a
default value, Hash.new(0), into a regular hash without a default value
after serializing, storing and deserializing. This raises an error when
we try to access a missing exception key. A simple solution is not to
rely on the hash's default value, and provide a default as alternative
when accessing it instead.
Previously, by extending ActiveJob::TestCase, the test adapter provided
for tests was being used always, in all executions where supposedly
different adapters were being used. As a consequence, some bugs visible
only for some adapters might have gone undetected. This commit changes
that, skipping queue adapters for which we can't test scheduling jobs
with a delay.
Also, make tests and examples for individual execution counters
clearer, as it wasn't entierly clear what would happen in this case:
```
retry_on CustomException, OtherException, attempts: 3
```
The job would be retried at most 3 times in total, for both
CustomException and OtherException. To have the job retry 3 times at
most for each exception individually, the following retry_on
declarations are necessary:
```
retry_on CustomException, attempts: 3
retry_on OtherException, attempts: 3
```
Generally followed the pattern for https://github.com/rails/rails/pull/32034
* Removes needless CI configs for 2.4
* Targets 2.5 in rubocop
* Updates existing CHANGELOG entries for fewer merge conflicts
* Removes Hash#slice extension as that's inlined on Ruby 2.5.
* Removes the need for send on define_method in MethodCallAssertions.
Trusting any GlobaID object when deserializing jobs can allow
attackers to access information that should not be accessible to them.
Fix CVE-2018-16476.
* Keep executions for each specific declaration
Fixes#34337
ActiveJob used the global executions counter to control the number of
times a job should be retried. The problem with this approach was that
in case a job raised different exceptions during its executions they
weren't retried the number of times defined by their `attemps` number.
**Example:**
Having the following job:
```ruby
class BuggyJob < ActiveJob::Base
retry_on CustomException, attemps: 3
retry_on OtherException, attempts: 3
end
```
If the job raised `CustomException` in the first two executions and then
it raised `OtherException`, the job wasn't retried anymore because the
global executions counter was already indicating 3 attempts.
With this patch each `retry_on` declaration has its specific counter so
that the first two executions that raise `CustomException` don't affect
the retries count that future exceptions may have.
* Revert "clarifies documentation around the attempts arugment to retry_on"
This reverts commit 86aa8f8c56.
- That feature is useful to enqueue or assert that jobs got enqueued
or performed based on dynamic conditions.
We will be able to leverage that feature to fix all ActionMailer
assertion helper issue when a Mailer define a custom delivery job
(see next commit).
My spellchecker flagged this as an incorrect spelling, upon further
research it appears to be a point of contention in English. Either way
might work.
After further examination queuing is much more common in the Rails
codebase so making this change will serve to standardize the spelling.
The constant `PERMITTED_TYPES` is only used by the private method
`serialize_argument` and it already has the `# :nodoc:` annotation as
the other constants in the class.
Complements e899e22
Fixes#34366
Currently setting queue_name_prefix will not combine a prefix with the
default_queue_name; it will only affect queue names set with `queue_as`.
With this PR the prefix will affect the default_queue_name as well.
Closes#19831
Currently setting default_queue_name doesn't actually affect the
queue_name default (although default_queue_name does get used if you
pass a falsey `part_name` to `queue_as`). This PR would get
default_queue_name working as expected as well.
Because the queue_name default is now a lambda wrapping the
default_queue_name, rather than the default_queue_name itself, I had to
update one test to use the instance method `#queue_name` (which
`instance_exec`s the value) instead of the class method. I think this
change is OK, since only the instance method is documented.
There was a question about whether we want a `default_queue_name`
configuration. If we want to get rid of it, I would also be happy to
open a PR for that instead. It has been around for a while now, but it
also hasn't really worked for a while now.
r? @matthewd
since you had an opinion about this before
Since `@mutex.synchronize` is enforced in the `ensure_connection!` method,
there is no need to do so on the caller side.
c1b47f9c5d/lib/sneakers/publisher.rb (L22-L26)
Due to this, `ThreadError(deadlock; recursive locking)` has occurred.
`assert_enqueued_with` and `assert_performed_with` return a instantiated
instance of the matching job for further assertion (#21010).
Before this commit the `arguments` method on the returned instance
returns a serialized version of the arguments.
Since Rails 6.0 will support Ruby 2.4.1 or higher
`# frozen_string_literal: true` magic comment is enough to make string object frozen.
This magic comment is enabled by `Style/FrozenStringLiteralComment` cop.
* Exclude these files not to auto correct false positive `Regexp#freeze`
- 'actionpack/lib/action_dispatch/journey/router/utils.rb'
- 'activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb'
It has been fixed by https://github.com/rubocop-hq/rubocop/pull/6333
Once the newer version of RuboCop released and available at Code Climate these exclude entries should be removed.
* Replace `String#freeze` with `String#-@` manually if explicit frozen string objects are required
- 'actionpack/test/controller/test_case_test.rb'
- 'activemodel/test/cases/type/string_test.rb'
- 'activesupport/lib/active_support/core_ext/string/strip.rb'
- 'activesupport/test/core_ext/string_ext_test.rb'
- 'railties/test/generators/actions_test.rb'
- When calling `assert_performed_with`/`assert_enqueued_with`, the
+args+ needs to match exactly what the job get passed.
Some jobs can have lot of arguments, or even a simple hash argument
has many key. This is not convenient to test as most tests doesn't
need to check if the arguments matches perfectly.
This PR make it possible to only check if a subset of arguments were
passed to the job.
Since #33751 was added `enqueue_retry.active_job` instrumentation to
the `retry_on` method, then #33897 moved the instrumentation to `retry_job`
method in order to ensure that this method publish `enqueue_retry.active_job`
notification too.
See related discussion https://github.com/rails/rails/pull/33751#discussion_r214140008
Since `enqueue_retry.active_job` moved to `retry_job`, there is no guarantee
that payload of `enqueue_retry.active_job` would have `:error`.
See test `LoggingTest#test_enqueue_retry_logging_on_retry_job` as
example of that case.
Related to https://github.com/rails/rails/pull/33897#discussion_r219707024
I think we can improve notification of `enqueue_retry.active_job`:
- If there is no `event.payload[:error]`, then publish like
"Retrying RescueJob in 3 seconds." only.
- If `event.payload[:wait]` is `nil`, then publish
"Retrying RescueJob in 0 seconds." instead of "Retrying RescueJob in nil seconds."
- If there is `event.payload[:error]`, then publish
"Retrying RescueJob in 3 seconds, due to a DefaultsError.".
- Change the type of the message from `error` to `info.`
Also, this commit removes part of messages - "The original exception was #{ex.cause.inspect}."
of `enqueue_retry.active_job`, `retry_stopped.active_job`, and `discard.active_job` since I
haven't found it useful. Please let me know whether you agree with that?
psqlrc files can affect the execution of commands in ways that can hold
up execution by blocking or otherwise cause unexpected side effects and
should best be ignored when using psql programmatically.
Currently when executing `bundle exec rake test:integration` under `activejob/`
derectory, it prints helpful info like:
```
(snip)
*** rake aj:integration:async ***
(snip)
*** rake aj:integration:delayed_job ***
(snip)
```
but there is no defined `:aj` scope in `activejob/Rakefile`,
so I think output should be like:
```
(snip)
*** rake test:integration:async ***
(snip)
*** rake test:integration:delayed_job ***
(snip)
```
By the way `rake test:integration` doesn't work if execute it without
prepending `bundle exec` to that command. It is probably what we should
fix too.
As @dhh brings up, the point of `ActionDispatch::IntegrationTest` is to
allow users to test the integration of all the pieces called by a
controller. Asserting about the emails and jobs queued is part of that
task.
This commit includes the `ActionMailer::TestHelper` and
`ActiveJob::TestHelper` modules when the ActionMailer and ActiveJob
railties are initialized respectively.
The priority wasn't being passed from ActiveJob to Backburner, despite
priority being supported. This also brings it inline with the docs,
which mark Backburner as supporting priorities in the "Backend Features"
table: https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters.html
If the `:queue` option is specified, then only the job(s) enqueued to a specific
queue will not be performed.
Example:
```
def test_assert_no_performed_jobs_with_queue_option
assert_no_performed_jobs queue: :some_queue do
HelloJob.set(queue: :other_queue).perform_later("jeremy")
end
end
```
If the `:queue` option is specified, then only the job(s) enqueued to a specific
queue will be performed.
Example:
```
def test_assert_performed_jobs_with_queue_option
assert_performed_jobs 1, queue: :some_queue do
HelloJob.set(queue: :some_queue).perform_later("jeremy")
HelloJob.set(queue: :other_queue).perform_later("bogdan")
end
end
```
If the `:queue` option is specified, then only the job(s) enqueued to
a specific queue will be performed.
Example:
```
def test_perform_enqueued_jobs_with_queue
perform_enqueued_jobs queue: :some_queue do
MyJob.set(queue: :some_queue).perform_later(1, 2, 3) # will be performed
HelloJob.set(queue: :other_queue).perform_later(1, 2, 3) # will not be performed
end
assert_performed_jobs 1
end
```
Follow up #33265
[bogdanvlviv & Jeremy Daer]
Currently, the execution count increments after deserializes arguments.
Therefore, if an error occurs with deserialize, it retries indefinitely.
In order to prevent this, the count is moved before deserialize.
Fixes#33344.
It can be asserted that no jobs are enqueued to a specific queue:
```ruby
def test_no_logging
assert_no_enqueued_jobs queue: 'default' do
LoggingJob.set(queue: :some_queue).perform_later
end
end
```
Rename `in_block_job` to `enqueued_job` since this variable can refer not only
to jobs that were created in the block.
See #33258.
Return back accidentally removed test to activejob/test/cases/test_helper_test.rb
See #33258.
Fix name of tests.
Example of `assert_enqueued_with` with no block
```ruby
def test_assert_enqueued_with
MyJob.perform_later(1,2,3)
assert_enqueued_with(job: MyJob, args: [1,2,3], queue: 'low')
MyJob.set(wait_until: Date.tomorrow.noon).perform_later
assert_enqueued_with(job: MyJob, at: Date.tomorrow.noon)
end
```
Example of `assert_enqueued_email_with` with no block:
```ruby
def test_email
ContactMailer.welcome.deliver_later
assert_enqueued_email_with ContactMailer, :welcome
end
def test_email_with_arguments
ContactMailer.welcome("Hello", "Goodbye").deliver_later
assert_enqueued_email_with ContactMailer, :welcome, args: ["Hello", "Goodbye"]
end
```
Related to #33243
The documentation for assert_enqueued_email_with states that it's
supposed to work without a block yet it calls assert_enqueued_with
which doesn't check whether a block was passed before calling `yield`
When a job was just deserialized `arguments` is `nil` and the serialized
arguments are in the `@serialized_arguments` variable. If we try to
serialize this job again the arguments are going to be `nil` instead of
what was serialized.
The test we had was not checking this case because it was deserializing
the job in the same object that had the arguments.
To fix this, when the `@serialized_arguments` are present we return it
instead of the result of the `arguments` serialized.
Reasons are that the Qu gem wasn't compatible since Rails 5.1,
gem development was stopped in 2014 and maintainers have
confirmed its demise. See issue #32273
Record what was the current timezone in effect when the job was
enqueued and then restore when the job is executed in same way
that the current locale is recorded and restored.
Add `:nodoc:` to `ActiveJob::Serializers`
Add `:doc:` to `ActiveJob::Serializers::ObjectSerializer#klass`
Express `ActiveJob::Serializers::ObjectSerializer#klass` as private method
The serializer serializes an instance of `ActiveSupport::TimeWithZone`.
The serializer deserializes value to `ActiveSupport::TimeWithZone` if possible.
The serializer should be set up in `after_initialize` so that it work
properly even if the user specifies serializer with initializers.
Also, since `custom_serializers` is `Array`, it needs to be flattened
before setting the value.
We can speed up things for the supported types by keeping the code in the
way it was.
We can also avoid to loop trough all serializers in the deserialization by
trying to access the class already in the Hash.
We could also speed up the custom serialization if we define the class
that is going to be serialized when registering the serializers, but
that will remove the possibility of defining a serialzer for a
superclass and have the subclass serialized using it.
Now custom serialziers can register itself in the serialized hash using
the "_aj_serialized" key that constains the serializer name.
This way we can avoid poluting the hash with many reserved keys.
Currently, sidekiq integration test + Ruby 2.5.0-rc1 show exception as follows.
```
#<Thread:0x000000000670bec0@/home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:23 run> terminated with exception (report_on_exception is true):
/home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:53:in `block in load_missing_constant': uninitialized constant Sidekiq::CLI (NameError)
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:8:in `without_bootsnap_cache'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:53:in `rescue in load_missing_constant'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:42:in `load_missing_constant'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/launcher.rb:65:in `heartbeat'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/launcher.rb:123:in `start_heartbeat'
```
https://travis-ci.org/rails/rails/jobs/317187279#L2152
The reason for this is that `Sidekiq::CLI` has not been loaded.
Sidekiq integration test launches a Sidekiq instance within
another Ruby process. In such a case, need to require 'sidekiq/cli'
in that launch code.
Ref: https://github.com/mperham/sidekiq/pull/3692#issuecomment-352032251
`BigDecimal.new` has been deprecated in BigDecimal 1.3.3
which will be a default for Ruby 2.5.
Refer
533737338d
* This commit has been made as follows:
```
cd rails
git grep -l BigDecimal.new | grep -v guides/source/5_0_release_notes.md | grep -v activesupport/test/xml_mini_test.rb | xargs sed -i -e "s/BigDecimal.new/BigDecimal/g"
```
- `activesupport/test/xml_mini_test.rb`
Editmanually to remove `.new` and `::`
- guides/source/5_0_release_notes.md
This is a Rails 5.0 release notes.
1) It seems that it raise error on example code in `ActiveJob::Core`.
Before:
```ruby
class DeliverWebhookJob < ActiveJob::Base
def serialize
super.merge('attempt_number' => (@attempt_number || 0) + 1)
end
def deserialize(job_data)
super
@attempt_number = job_data['attempt_number']
end
rescue_from(Timeout::Error) do |exception|
raise exception if @attempt_number > 5
retry_job(wait: 10)
end
def perform
raise Timeout::Error
end
end
```
Then it run `DeliverWebhookJob.perform_now` in `rails console`. And raise error:
NoMethodError: undefined method `>' for nil:NilClass
from /app/jobs/deliver_webhook_job.rb:12:in `block in <class:DeliverWebhookJob>'
So I thought it's necessary to fix it.
After:
```ruby
class DeliverWebhookJob < ActiveJob::Base
attr_writer :attempt_number
def attempt_number
@attempt_number ||= 0
end
def serialize
super.merge('attempt_number' => attempt_number + 1)
end
def deserialize(job_data)
super
self.attempt_number = job_data['attempt_number']
end
rescue_from(Timeout::Error) do |exception|
raise exception if attempt_number > 5
retry_job(wait: 10)
end
def perform
raise Timeout::Error
end
end
```
Then it run `DeliverWebhookJob.perform_now` in `rails console`. And it does'nt raise error NoMethodError.
2) Use `Timeout::Error` instead of `TimeoutError` (`TimeoutError` is deprecated).
ActiveJob wraps every adapter into its own class, that is later passed
into DelayedJob which is responsible for displaying all the logs.
This change improves the logging so we can easily trace executed
jobs and see meaningful information in the logs.
Omit `rails activestorage:install` for jdbcmysql, jdbc and shebang tests
AppGeneratorTest#test_config_jdbcmysql_database
rails aborted!
LoadError: Could not load 'active_record/connection_adapters/mysql_adapter'.
Make sure that the adapter in config/database.yml is valid.
If you use an adapter other than 'mysql2', 'postgresql' or 'sqlite3' add
the necessary adapter gem to the Gemfile.
(compressed)
bin/rails:4:in `<main>'
Tasks: TOP => activestorage:install => environment
(See full trace by running task with --trace)
AppGeneratorTest#test_config_jdbc_database
rails aborted!
LoadError: Could not load 'active_record/connection_adapters/jdbc_adapter'.
Make sure that the adapter in config/database.yml is valid.
If you use an adapter other than 'mysql2', 'postgresql' or 'sqlite3' add
the necessary adapter gem to the Gemfile.
(compressed)
bin/rails:4:in `<main>'
Tasks: TOP => activestorage:install => environment
(See full trace by running task with --trace)
AppGeneratorTest#test_shebang_is_added_to_rails_file
/home/ubuntu/.rbenv/versions/2.4.1/bin/ruby: no Ruby script found in input (LoadError)
Prevent PendingMigrationError in tests
* Run `bin/rails db:migrate RAILS_ENV=test` in test_cases before start tests to prevent PendingMigrationError
* FileUtils.rm_r("db/migrate")
* --skip-active-storage
Fix failed tests in `railties/test/railties/engine_test.rb`
Related to #30111
Imporve `SharedGeneratorTests#test_default_frameworks_are_required_when_others_are_removed`
- Explicitly skip active_storage
- Ensure that skipped frameworks are commented
- Ensure that default frameworks are not commented
Fix error `Errno::ENOSPC: No space left on device - sendfile`
Since `rails new` runs `rails active_storage:install`
that boots an app.
Since adding Bootsnap 0312a5c67e
during booting an app, it creates the cache:
264K tmp/cache/bootsnap-load-path-cache
27M tmp/cache/bootsnap-compile-cache
* teardown_app must remove app
* Use `gem 'redis', '~> 4.0'` for new app Gemfiles
* Loosen Action Cable redis-rb dep to `>= 3.3, < 5`
* Bump redis-namespace for looser Redis version dep
* Avoid using the underlying `redis.client` directly
* Use `Redis.new` instead of `Redis.connect`
Since f55ecc6, the integration test of sidekiq and resque is not working
in CI.
https://travis-ci.org/rails/rails/jobs/251783876
Because f55ecc6 required a password to access redis.
Therefore, handling by passing passwords when connecting to redis.
* Allow a default value to be declared for class_attribute
* Convert to using class_attribute default rather than explicit setter
* Removed instance_accessor option by mistake
* False is a valid default value
* Documentation
Active Job logging instrumentation is changed to log errors (with
backtrace) when a job raises an exception in #perform. This improves
debugging during development and test with the default configuration.
Prior to Rails 5, the default development configuration ran jobs with
InlineAdapter, which would raise exceptions to the caller and be
shown in the development log. In Rails 5, the default adapter was
changed to AsyncAdapter, which would silently swallow exceptions
and log a "Performed SomeJob from Async..." info message. This could
be confusing to a developer, as it would seem that the job was
performed successfully.
This patch removes the "Performed..." info message from the log
and adds an error-level "Error performing SomeJob..." log message
which includes the exception backtrace for jobs that raise an
exception within the #perform method. It provides this behavior for
all adapters.
Currently we provide the Job ID in logs only related to enqueuing a job.
This adds the job id to the remaining ActiveJob logs when:
- a job started performing
- a job ended performing
Providing the job id in those logs will ease searching logs by job id.
lrns