mirrors
/
rails
mirror of https://github.com/rails/rails
8
0
Fork 0
Code Issues Projects Releases Wiki Activity

Support replacing cache compressor 

This commit adds support for replacing the compressor used for
serialized cache entries.  Custom compressors must respond to `deflate`
and `inflate`.  For example:

  ```ruby
  module MyCompressor
    def self.deflate(string)
      # compression logic...
    end

    def self.inflate(compressed)
      # decompression logic...
    end
  end

  config.cache_store = :redis_cache_store, { compressor: MyCompressor }
  ```

As part of this work, cache stores now also support a `:serializer`
option.  Similar to the `:coder` option, serializers must respond to
`dump` and `load`. However, serializers are only responsible for
serializing a cached value, whereas coders are responsible for
serializing the entire `ActiveSupport::Cache::Entry` instance.
Additionally, the output from serializers can be automatically
compressed, whereas coders are responsible for their own compression.

Specifying a serializer instead of a coder also enables performance
optimizations, including the bare string optimization introduced by cache
format version 7.1.
This commit is contained in:
Jonathan Hefner 2023-06-10 13:39:28 -05:00
parent 3bdd57fba6
commit 3efb84486e
19 changed files with 546 additions and 223 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

54
activesupport/CHANGELOG.md
View File

@ -1,3 +1,39 @@
* Active Support cache stores now support replacing the default compressor via
a `:compressor` option. The specified compressor must respond to `deflate`
and `inflate`. For example:
```ruby
module MyCompressor
def self.deflate(string)
# compression logic...
end
def self.inflate(compressed)
# decompression logic...
end
end
config.cache_store = :redis_cache_store, { compressor: MyCompressor }
```
*Jonathan Hefner*
* Active Support cache stores now support a `:serializer` option. Similar to
the `:coder` option, serializers must respond to `dump` and `load`. However,
serializers are only responsible for serializing a cached value, whereas
coders are responsible for serializing the entire `ActiveSupport::Cache::Entry`
instance. Additionally, the output from serializers can be automatically
compressed, whereas coders are responsible for their own compression.
Specifying a serializer instead of a coder also enables performance
optimizations, including the bare string optimization introduced by cache
format version 7.1.
The `:serializer` and `:coder` options are mutually exclusive. Specifying
both will raise an `ArgumentError`.
*Jonathan Hefner*
* Fix `ActiveSupport::Inflector.humanize(nil)` raising ``NoMethodError: undefined method `end_with?' for nil:NilClass``.
*James Robinson*
@ -164,25 +200,23 @@
read caches from upgraded servers, leave the cache format unchanged on the
first deploy, then enable the `7.1` cache format on a subsequent deploy.
The new `:message_pack` cache coder also includes this optimization.
*Jonathan Hefner*
* The `:coder` option for Active Support cache stores now supports a
`:message_pack` value:
* Active Support cache stores can now use a preconfigured serializer based on
`ActiveSupport::MessagePack` via the `:serializer` option:
```ruby
config.cache_store = :redis_cache_store, { coder: :message_pack }
config.cache_store = :redis_cache_store, { serializer: :message_pack }
```
The `:message_pack` coder can reduce cache entry sizes and improve
The `:message_pack` serializer can reduce cache entry sizes and improve
performance, but requires the [`msgpack` gem](https://rubygems.org/gems/msgpack)
(>= 1.7.0).
The `:message_pack` coder can read cache entries written by the default
coder, and the default coder can now read entries written by the
`:message_pack` coder. These behaviors make it easy to migrate between
coders without invalidating the entire cache.
The `:message_pack` serializer can read cache entries written by the default
serializer, and the default serializer can now read entries written by the
`:message_pack` serializer. These behaviors make it easy to migrate between
serializer without invalidating the entire cache.
*Jonathan Hefner*

99
activesupport/lib/active_support/cache.rb
View File

@ -1,5 +1,6 @@
# frozen_string_literal: true
require "zlib"
require "active_support/core_ext/array/extract_options"
require "active_support/core_ext/enumerable"
require "active_support/core_ext/module/attribute_accessors"
@ -7,6 +8,7 @@ require "active_support/core_ext/numeric/bytes"
require "active_support/core_ext/object/to_param"
require "active_support/core_ext/object/try"
require "active_support/core_ext/string/inflections"
require_relative "cache/coder"
require_relative "cache/entry"
require_relative "cache/serializer_with_fallback"
@ -25,11 +27,13 @@ module ActiveSupport
:coder,
:compress,
:compress_threshold,
:compressor,
:expire_in,
:expired_in,
:expires_in,
:namespace,
:race_condition_ttl,
:serializer,
:skip_nil,
]
@ -249,27 +253,79 @@ module ActiveSupport
# Sets the namespace for the cache. This option is especially useful if
# your application shares a cache with other applications.
#
# [+:coder+]
# Replaces the default serializer for cache entries. +coder+ must
# respond to +dump+ and +load+. Using a custom coder disables automatic
# compression.
# [+:serializer+]
# The serializer for cached values. Must respond to +dump+ and +load+.
#
# Alternatively, you can specify <tt>coder: :message_pack</tt> to use a
# preconfigured coder based on ActiveSupport::MessagePack that supports
# automatic compression and includes a fallback mechanism to load old
# cache entries from the default coder. However, this option requires
# the +msgpack+ gem.
# The default serializer depends on the cache format version (set via
# +config.active_support.cache_format_version+ when using Rails). The
# default serializer for each format version includes a fallback
# mechanism to deserialize values from any format version. This behavior
# makes it easy to migrate between format versions without invalidating
# the entire cache.
#
# You can also specify <tt>serializer: :message_pack</tt> to use a
# preconfigured serializer based on ActiveSupport::MessagePack. The
# +:message_pack+ serializer includes the same deserialization fallback
# mechanism, allowing easy migration from (or to) the default
# serializer. The +:message_pack+ serializer may improve performance,
# but it requires the +msgpack+ gem.
#
# [+:compressor+]
# The compressor for serialized cache values. Must respond to +deflate+
# and +inflate+.
#
# The default compressor is +Zlib+. To define a new custom compressor
# that also decompresses old cache entries, you can check compressed
# values for Zlib's <tt>"\x78"</tt> signature:
#
# module MyCompressor
# def self.deflate(dumped)
# # compression logic... (make sure result does not start with "\x78"!)
# end
#
# def self.inflate(compressed)
# if compressed.start_with?("\x78")
# Zlib.inflate(compressed)
# else
# # decompression logic...
# end
# end
# end
#
# ActiveSupport::Cache.lookup_store(:redis_cache_store, compressor: MyCompressor)
#
# [+:coder+]
# The coder for serializing and (optionally) compressing cache entries.
# Must respond to +dump+ and +load+.
#
# The default coder composes the serializer and compressor, and includes
# some performance optimizations. If you only need to override the
# serializer or compressor, you should specify the +:serializer+ or
# +:compressor+ options instead.
#
# The +:coder+ option is mutally exclusive with the +:serializer+ and
# +:compressor+ options. Specifying them together will raise an
# +ArgumentError+.
#
# Any other specified options are treated as default options for the
# relevant cache operations, such as #read, #write, and #fetch.
def initialize(options = nil)
@options = options ? normalize_options(options) : {}
@options = options ? validate_options(normalize_options(options)) : {}
@options[:compress] = true unless @options.key?(:compress)
@options[:compress_threshold] ||= DEFAULT_COMPRESS_LIMIT
@coder = @options.delete(:coder) { default_coder } || :passthrough
@coder = Cache::SerializerWithFallback[@coder] if @coder.is_a?(Symbol)
@coder = @options.delete(:coder) do
legacy_serializer = Cache.format_version < 7.1 && !@options[:serializer]
serializer = @options.delete(:serializer) || default_serializer
serializer = Cache::SerializerWithFallback[serializer] if serializer.is_a?(Symbol)
compressor = @options.delete(:compressor) { Zlib }
Cache::Coder.new(serializer, compressor, legacy_serializer: legacy_serializer)
end
@coder ||= Cache::SerializerWithFallback[:passthrough]
@coder_supports_compression = @coder.respond_to?(:dump_compressed)
end
@ -686,7 +742,7 @@ module ActiveSupport
end
private
def default_coder
def default_serializer
case Cache.format_version
when 6.1
ActiveSupport.deprecator.warn <<~EOM
@ -842,6 +898,23 @@ module ActiveSupport
options
end
def validate_options(options)
if options.key?(:coder) && options[:serializer]
raise ArgumentError, "Cannot specify :serializer and :coder options together"
end
if options.key?(:coder) && options[:compressor]
raise ArgumentError, "Cannot specify :compressor and :coder options together"
end
if Cache.format_version < 7.1 && !options[:serializer] && options[:compressor]
raise ArgumentError, "Cannot specify :compressor option when using" \
" default serializer and cache format version is < 7.1"
end
options
end
# Expands and namespaces the cache key.
# Raises an exception when the key is +nil+ or an empty string.
# May be overridden by cache stores to do additional normalization.

123
activesupport/lib/active_support/cache/coder.rb vendored Normal file
View File

@ -0,0 +1,123 @@
# frozen_string_literal: true
require_relative "entry"
module ActiveSupport
module Cache
class Coder # :nodoc:
def initialize(serializer, compressor, legacy_serializer: false)
@serializer = serializer
@compressor = compressor
@legacy_serializer = legacy_serializer
end
def dump(entry)
return @serializer.dump(entry) if @legacy_serializer
dump_compressed(entry, Float::INFINITY)
end
def dump_compressed(entry, threshold)
return @serializer.dump_compressed(entry, threshold) if @legacy_serializer
# If value is a string with a supported encoding, use it as the payload
# instead of passing it through the serializer.
if type = type_for_string(entry.value)
payload = entry.value.b
else
type = OBJECT_DUMP_TYPE
payload = @serializer.dump(entry.value)
end
if compressed = try_compress(payload, threshold)
payload = compressed
type = type | COMPRESSED_FLAG
end
expires_at = entry.expires_at || -1.0
version = dump_version(entry.version) if entry.version
version_length = version&.bytesize || -1
packed = SIGNATURE.b
packed << [type, expires_at, version_length].pack(PACKED_TEMPLATE)
packed << version if version
packed << payload
end
def load(dumped)
return @serializer.load(dumped) if !signature?(dumped)
type = dumped.unpack1(PACKED_TYPE_TEMPLATE)
expires_at = dumped.unpack1(PACKED_EXPIRES_AT_TEMPLATE)
version_length = dumped.unpack1(PACKED_VERSION_LENGTH_TEMPLATE)
expires_at = nil if expires_at < 0
version = load_version(dumped.byteslice(PACKED_VERSION_INDEX, version_length)) if version_length >= 0
payload = dumped.byteslice((PACKED_VERSION_INDEX + [version_length, 0].max)..)
payload = @compressor.inflate(payload) if type & COMPRESSED_FLAG > 0
if string_encoding = STRING_ENCODINGS[type & ~COMPRESSED_FLAG]
value = payload.force_encoding(string_encoding)
else
value = @serializer.load(payload)
end
Cache::Entry.new(value, version: version, expires_at: expires_at)
end
private
SIGNATURE = "\x00\x11".b.freeze
OBJECT_DUMP_TYPE = 0x01
STRING_ENCODINGS = {
0x02 => Encoding::UTF_8,
0x03 => Encoding::BINARY,
0x04 => Encoding::US_ASCII,
}
COMPRESSED_FLAG = 0x80
PACKED_TEMPLATE = "CEl<"
PACKED_TYPE_TEMPLATE = "@#{SIGNATURE.bytesize}C"
PACKED_EXPIRES_AT_TEMPLATE = "@#{[0].pack(PACKED_TYPE_TEMPLATE).bytesize}E"
PACKED_VERSION_LENGTH_TEMPLATE = "@#{[0].pack(PACKED_EXPIRES_AT_TEMPLATE).bytesize}l<"
PACKED_VERSION_INDEX = [0].pack(PACKED_VERSION_LENGTH_TEMPLATE).bytesize
MARSHAL_SIGNATURE = "\x04\x08".b.freeze
def signature?(dumped)
dumped.is_a?(String) && dumped.start_with?(SIGNATURE)
end
def type_for_string(value)
STRING_ENCODINGS.key(value.encoding) if value.instance_of?(String)
end
def try_compress(string, threshold)
if @compressor && string.bytesize >= threshold
compressed = @compressor.deflate(string)
compressed if compressed.bytesize < string.bytesize
end
end
def dump_version(version)
if version.encoding != Encoding::UTF_8 || version.start_with?(MARSHAL_SIGNATURE)
Marshal.dump(version)
else
version.b
end
end
def load_version(dumped_version)
if dumped_version.start_with?(MARSHAL_SIGNATURE)
Marshal.load(dumped_version)
else
dumped_version.force_encoding(Encoding::UTF_8)
end
end
end
end
end

2
activesupport/lib/active_support/cache/mem_cache_store.rb vendored
View File

@ -222,7 +222,7 @@ module ActiveSupport
end
private
def default_coder
def default_serializer
if Cache.format_version == 6.1
ActiveSupport.deprecator.warn <<~EOM
Support for `config.active_support.cache_format_version = 6.1` has been deprecated and will be removed in Rails 7.2.

5
activesupport/lib/active_support/cache/memory_store.rb vendored
View File

@ -72,6 +72,7 @@ module ActiveSupport
def initialize(options = nil)
options ||= {}
options[:coder] = DupCoder unless options.key?(:coder) || options.key?(:serializer)
# Disable compression by default.
options[:compress] ||= false
super(options)
@ -189,10 +190,6 @@ module ActiveSupport
private
PER_ENTRY_OVERHEAD = 240
def default_coder
DupCoder
end
def cached_size(key, payload)
key.to_s.bytesize + payload.bytesize + PER_ENTRY_OVERHEAD
end

132
activesupport/lib/active_support/cache/serializer_with_fallback.rb vendored
View File

@ -14,28 +14,15 @@ module ActiveSupport
SERIALIZERS.fetch(format)
end
def dump(entry)
try_dump_bare_string(entry) || _dump(entry)
end
def dump_compressed(entry, threshold)
dumped = dump(entry)
try_compress(dumped, threshold) || dumped
end
def load(dumped)
if dumped.is_a?(String)
dumped = decompress(dumped) if compressed?(dumped)
case
when loaded = try_load_bare_string(dumped)
loaded
when MessagePackWithFallback.dumped?(dumped)
MessagePackWithFallback._load(dumped)
when Marshal71WithFallback.dumped?(dumped)
Marshal71WithFallback._load(dumped)
when Marshal61WithFallback.dumped?(dumped)
Marshal61WithFallback._load(dumped)
when Marshal70WithFallback.dumped?(dumped)
Marshal70WithFallback._load(dumped)
else
Cache::Store.logger&.warn("Unrecognized payload prefix #{dumped.byteslice(0).inspect}; deserializing as nil")
nil
@ -49,72 +36,12 @@ module ActiveSupport
end
private
BARE_STRING_SIGNATURES = {
255 => Encoding::UTF_8,
254 => Encoding::BINARY,
253 => Encoding::US_ASCII,
}
BARE_STRING_TEMPLATE = "CEl<"
BARE_STRING_EXPIRES_AT_TEMPLATE = "@1E"
BARE_STRING_VERSION_LENGTH_TEMPLATE = "@#{[0].pack(BARE_STRING_EXPIRES_AT_TEMPLATE).bytesize}l<"
BARE_STRING_VERSION_INDEX = [0].pack(BARE_STRING_VERSION_LENGTH_TEMPLATE).bytesize
def marshal_load(payload)
Marshal.load(payload)
rescue ArgumentError => error
raise Cache::DeserializationError, error.message
end
def try_dump_bare_string(entry)
value = entry.value
return if !value.instance_of?(String)
version = entry.version
return if version && version.encoding != Encoding::UTF_8
signature = BARE_STRING_SIGNATURES.key(value.encoding)
return if !signature
packed = [signature, entry.expires_at || -1.0, version&.bytesize || -1].pack(BARE_STRING_TEMPLATE)
packed << version.b if version
packed << value.b
end
def try_load_bare_string(dumped)
encoding = BARE_STRING_SIGNATURES[dumped.getbyte(0)]
return if !encoding
expires_at = dumped.unpack1(BARE_STRING_EXPIRES_AT_TEMPLATE)
version_length = dumped.unpack1(BARE_STRING_VERSION_LENGTH_TEMPLATE)
value_index = BARE_STRING_VERSION_INDEX + [version_length, 0].max
Cache::Entry.new(
dumped.byteslice(value_index..-1).force_encoding(encoding),
version: dumped.byteslice(BARE_STRING_VERSION_INDEX, version_length)&.force_encoding(Encoding::UTF_8),
expires_at: (expires_at unless expires_at < 0),
)
end
ZLIB_HEADER = "\x78".b.freeze
def compressed?(dumped)
dumped.start_with?(ZLIB_HEADER)
end
def compress(dumped)
Zlib::Deflate.deflate(dumped)
end
def try_compress(dumped, threshold)
if dumped.bytesize >= threshold
compressed = compress(dumped)
compressed unless compressed.bytesize >= dumped.bytesize
end
end
def decompress(compressed)
Zlib::Inflate.inflate(compressed)
end
module PassthroughWithFallback
include SerializerWithFallback
extend self
@ -158,38 +85,32 @@ module ActiveSupport
end
end
module Marshal71WithFallback
module Marshal70WithFallback
include SerializerWithFallback
extend self
MARK_UNCOMPRESSED = "\x00".b.freeze
MARK_COMPRESSED = "\x01".b.freeze
def dump(entry, raw = false)
if raw
super(entry)
else
MARK_UNCOMPRESSED + super(entry)
end
end
def _dump(entry)
Marshal.dump(entry.pack)
def dump(entry)
MARK_UNCOMPRESSED + Marshal.dump(entry.pack)
end
def dump_compressed(entry, threshold)
dumped = dump(entry, true)
if compressed = try_compress(dumped, threshold)
MARK_COMPRESSED + compressed
else
MARK_UNCOMPRESSED + dumped
dumped = Marshal.dump(entry.pack)
if dumped.bytesize >= threshold
compressed = Zlib::Deflate.deflate(dumped)
return MARK_COMPRESSED + compressed if compressed.bytesize < dumped.bytesize
end
MARK_UNCOMPRESSED + dumped
end
def _load(marked)
dumped = marked.byteslice(1..-1)
dumped = decompress(dumped) if marked.start_with?(MARK_COMPRESSED)
try_load_bare_string(dumped) || Cache::Entry.unpack(marshal_load(dumped))
dumped = Zlib::Inflate.inflate(dumped) if marked.start_with?(MARK_COMPRESSED)
Cache::Entry.unpack(marshal_load(dumped))
end
def dumped?(dumped)
@ -197,12 +118,22 @@ module ActiveSupport
end
end
module Marshal70WithFallback
include Marshal71WithFallback
module Marshal71WithFallback
include SerializerWithFallback
extend self
def try_dump_bare_string(_entry)
nil # Prevent dumping bare strings.
MARSHAL_SIGNATURE = "\x04\x08".b.freeze
def dump(value)
Marshal.dump(value)
end
def _load(dumped)
marshal_load(dumped)
end
def dumped?(dumped)
dumped.start_with?(MARSHAL_SIGNATURE)
end
end
@ -210,13 +141,12 @@ module ActiveSupport
include SerializerWithFallback
extend self
def _dump(entry)
ActiveSupport::MessagePack::CacheSerializer.dump(entry.pack)
def dump(value)
ActiveSupport::MessagePack::CacheSerializer.dump(value)
end
def _load(dumped)
packed = ActiveSupport::MessagePack::CacheSerializer.load(dumped)
Cache::Entry.unpack(packed) if packed
ActiveSupport::MessagePack::CacheSerializer.load(dumped)
end
def dumped?(dumped)

1
activesupport/test/cache/behaviors.rb vendored
View File

@ -9,6 +9,7 @@ require_relative "behaviors/cache_store_version_behavior"
require_relative "behaviors/cache_store_coder_behavior"
require_relative "behaviors/cache_store_compression_behavior"
require_relative "behaviors/cache_store_format_version_behavior"
require_relative "behaviors/cache_store_serializer_behavior"
require_relative "behaviors/connection_pool_behavior"
require_relative "behaviors/encoded_key_cache_behavior"
require_relative "behaviors/failure_safety_behavior"

48
activesupport/test/cache/behaviors/cache_store_compression_behavior.rb vendored
View File

@ -17,7 +17,7 @@ module CacheStoreCompressionBehavior
assert_compression true
end
test "compression works with cache format version 7.1 (using Marshal71WithFallback)" do
test "compression works with cache format version >= 7.1 (using Cache::Coder)" do
@cache = with_format(7.1) { lookup_store(compress: true) }
assert_compression true
end
@ -27,6 +27,11 @@ module CacheStoreCompressionBehavior
assert_compression false
end
test "compression works with custom serializer" do
@cache = with_format(7.1) { lookup_store(compress: true, serializer: Marshal) }
assert_compression true
end
test "compression by default" do
@cache = lookup_store
assert_compression !compression_always_disabled_by_default?
@ -72,6 +77,47 @@ module CacheStoreCompressionBehavior
assert_not_compress "", with: { compress: true, compress_threshold: 1 }
assert_not_compress [*0..127].pack("C*"), with: { compress: true, compress_threshold: 1 }
end
test "compressor can be specified" do
lossy_compressor = Module.new do
def self.deflate(dumped)
"yolo"
end
def self.inflate(compressed)
Marshal.dump("lossy!") if compressed == "yolo"
end
end
@cache = with_format(7.1) do
lookup_store(compress: true, compressor: lossy_compressor, serializer: Marshal)
end
key = SecureRandom.uuid
@cache.write(key, LARGE_OBJECT)
assert_equal "lossy!", @cache.read(key)
end
test "compressor can be nil" do
@cache = with_format(7.1) { lookup_store(compressor: nil) }
assert_compression false
end
test "specifying a compressor raises when cache format version < 7.1" do
with_format(7.0) do
assert_raises ArgumentError, match: /compressor/i do
lookup_store(compressor: Zlib)
end
end
end
test "specifying a compressor raises when also specifying a coder" do
with_format(7.1) do
assert_raises ArgumentError, match: /compressor/i do
lookup_store(compressor: Zlib, coder: Marshal)
end
end
end
end
private

4
activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb vendored
View File

@ -15,8 +15,8 @@ module CacheStoreFormatVersionBehavior
"\x01\x78".b, # "\x01" + Zlib::Deflate.deflate(...)
],
7.1 => [
"\x00\x04\x08[".b, # "\x00" + Marshal.dump(entry.pack)
"\x01\x78".b, # "\x01" + Zlib::Deflate.deflate(...)
"\x00\x11\x01".b, # ActiveSupport::Cache::Coder#dump
"\x00\x11\x81".b, # ActiveSupport::Cache::Coder#dump_compressed
],
}

54
activesupport/test/cache/behaviors/cache_store_serializer_behavior.rb vendored Normal file
View File

@ -0,0 +1,54 @@
# frozen_string_literal: true
require "active_support/core_ext/object/with"
module CacheStoreSerializerBehavior
extend ActiveSupport::Concern
included do
test "serializer can be specified" do
serializer = Module.new do
def self.dump(value)
value.class.name
end
def self.load(dumped)
Object.const_get(dumped)
end
end
@cache = with_format(7.1) { lookup_store(serializer: serializer) }
key = "key#{rand}"
@cache.write(key, 123)
assert_equal Integer, @cache.read(key)
end
test "serializer can be :message_pack" do
@cache = with_format(7.1) { lookup_store(serializer: :message_pack) }
key = "key#{rand}"
@cache.write(key, 123)
assert_equal 123, @cache.read(key)
assert_raises ActiveSupport::MessagePack::UnserializableObjectError do
@cache.write(key, Object.new)
end
end
test "specifying a serializer raises when also specifying a coder" do
with_format(7.1) do
assert_raises ArgumentError, match: /serializer/i do
lookup_store(serializer: Marshal, coder: Marshal)
end
end
end
end
private
def with_format(format_version, &block)
ActiveSupport.deprecator.silence do
ActiveSupport::Cache.with(format_version: format_version, &block)
end
end
end

126
activesupport/test/cache/cache_coder_test.rb vendored Normal file
View File

@ -0,0 +1,126 @@
# frozen_string_literal: true
require_relative "../abstract_unit"
require "active_support/core_ext/integer/time"
class CacheCoderTest < ActiveSupport::TestCase
setup do
@coder = ActiveSupport::Cache::Coder.new(Serializer, Compressor)
end
test "roundtrips entry" do
ENTRIES.each do |entry|
assert_entry entry, @coder.load(@coder.dump(entry))
end
end
test "roundtrips entry when using compression" do
ENTRIES.each do |entry|
assert_entry entry, @coder.load(@coder.dump_compressed(entry, 1))
end
end
test "compresses values that are larger than the threshold" do
COMPRESSIBLE_ENTRIES.each do |entry|
dumped = @coder.dump(entry)
compressed = @coder.dump_compressed(entry, 1)
assert_operator compressed.bytesize, :<, dumped.bytesize
end
end
test "does not compress values that are smaller than the threshold" do
COMPRESSIBLE_ENTRIES.each do |entry|
dumped = @coder.dump(entry)
not_compressed = @coder.dump_compressed(entry, 1_000_000)
assert_equal dumped, not_compressed
end
end
test "does not apply compression to incompressible values" do
(ENTRIES - COMPRESSIBLE_ENTRIES).each do |entry|
dumped = @coder.dump(entry)
not_compressed = @coder.dump_compressed(entry, 1)
assert_equal dumped, not_compressed
end
end
test "loads dumped entries from original serializer" do
ENTRIES.each do |entry|
assert_entry entry, @coder.load(Serializer.dump(entry))
end
end
test "matches output of original serializer when legacy_serializer: true" do
@coder = ActiveSupport::Cache::Coder.new(Serializer, Compressor, legacy_serializer: true)
ENTRIES.each do |entry|
assert_equal Serializer.dump(entry), @coder.dump(entry)
assert_equal Serializer.dump_compressed(entry, 1), @coder.dump_compressed(entry, 1)
end
end
test "dumps bare strings with reduced overhead when possible" do
unoptimized = @coder.dump(ActiveSupport::Cache::Entry.new("".encode(Encoding::WINDOWS_1252)))
[Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding|
optimized = @coder.dump(ActiveSupport::Cache::Entry.new("".encode(encoding)))
assert_operator optimized.size, :<, unoptimized.size
end
end
private
module Serializer
extend self
def dump(entry)
"SERIALIZED:" + Marshal.dump(entry)
end
def dump_compressed(*)
"via Serializer#dump_compressed"
end
def load(dumped)
Marshal.load(dumped.delete_prefix!("SERIALIZED:"))
end
end
module Compressor
extend self
def deflate(string)
"COMPRESSED:" + Zlib.deflate(string)
end
def inflate(deflated)
Zlib.inflate(deflated.delete_prefix!("COMPRESSED:"))
end
end
STRING = "x" * 100
COMPRESSIBLE_VALUES = [
{ string: STRING },
STRING,
STRING.encode(Encoding::BINARY),
STRING.encode(Encoding::US_ASCII),
STRING.encode(Encoding::WINDOWS_1252),
]
VALUES = [nil, true, 1, "", "ümlaut", [*0..255].pack("C*"), *COMPRESSIBLE_VALUES]
VERSIONS = [nil, "", "ümlaut", [*0..255].pack("C*"), "x" * 256]
EXPIRIES = [nil, 0, 100.years]
ENTRIES = VALUES.product(VERSIONS, EXPIRIES).map do |value, version, expires_in|
ActiveSupport::Cache::Entry.new(value, version: version, expires_in: expires_in).freeze
end
COMPRESSIBLE_ENTRIES = ENTRIES.select { |entry| COMPRESSIBLE_VALUES.include?(entry.value) }
def assert_entry(expected, actual)
assert_equal \
[expected.value, expected.version, expected.expires_at],
[actual.value, actual.version, actual.expires_at]
end
end

101
activesupport/test/cache/serializer_with_fallback_test.rb vendored
View File

@ -5,6 +5,7 @@ require "active_support/core_ext/object/with"
class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
FORMATS = ActiveSupport::Cache::SerializerWithFallback::SERIALIZERS.keys
LEGACY_FORMATS = [:passthrough, :marshal_6_1, :marshal_7_0]
setup do
@entry = ActiveSupport::Cache::Entry.new(
@ -12,28 +13,21 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
)
end
FORMATS.product(FORMATS) do |load_format, dump_format|
test "#{load_format.inspect} serializer can load #{dump_format.inspect} dump" do
dumped = serializer(dump_format).dump(@entry)
assert_entry @entry, serializer(load_format).load(dumped)
FORMATS.product(FORMATS - LEGACY_FORMATS) do |loader, dumper|
test "#{loader.inspect} serializer can load #{dumper.inspect} dump" do
dumped = serializer(dumper).dump(@entry.value)
assert_equal @entry.value, serializer(loader).load(dumped)
end
end
test "#{load_format.inspect} serializer can load #{dump_format.inspect} dump with compression" do
compressed = serializer(dump_format).dump_compressed(@entry, 1)
assert_entry @entry, serializer(load_format).load(compressed)
uncompressed = serializer(dump_format).dump_compressed(@entry, 100_000)
assert_entry @entry, serializer(load_format).load(uncompressed)
FORMATS.product(LEGACY_FORMATS) do |loader, dumper|
test "#{loader.inspect} serializer can load #{dumper.inspect} dump" do
dumped = serializer(dumper).dump(@entry)
assert_entry @entry, serializer(loader).load(dumped)
end
end
FORMATS.each do |format|
test "#{format.inspect} serializer can compress entries" do
compressed = serializer(format).dump_compressed(@entry, 1)
uncompressed = serializer(format).dump_compressed(@entry, 100_000)
assert_operator compressed.bytesize, :<, uncompressed.bytesize
end
test "#{format.inspect} serializer handles unrecognized payloads gracefully" do
assert_nil serializer(format).load(Object.new)
assert_nil serializer(format).load("")
@ -45,71 +39,14 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
end
end
(FORMATS - [:passthrough, :marshal_6_1, :marshal_7_0]).each do |format|
test "#{format.inspect} serializer preserves version with bare string" do
entry = ActiveSupport::Cache::Entry.new("abc", version: "123")
assert_entry entry, roundtrip(format, entry)
end
LEGACY_FORMATS.each do |format|
test "#{format.inspect} serializer can compress entries" do
compressed = serializer(format).dump_compressed(@entry, 1)
uncompressed = serializer(format).dump_compressed(@entry, 100_000)
test "#{format.inspect} serializer preserves expiration with bare string" do
entry = ActiveSupport::Cache::Entry.new("abc", expires_in: 123)
assert_entry entry, roundtrip(format, entry)
end
test "#{format.inspect} serializer preserves encoding of version with bare string" do
[Encoding::UTF_8, Encoding::BINARY].each do |encoding|
version = "123".encode(encoding)
roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new("abc", version: version))
assert_equal version.encoding, roundtripped.version.encoding
end
end
test "#{format.inspect} serializer preserves encoding of bare string" do
[Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding|
string = "abc".encode(encoding)
roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new(string))
assert_equal string.encoding, roundtripped.value.encoding
end
end
test "#{format.inspect} serializer handles non-ASCII-only bare string" do
entry = ActiveSupport::Cache::Entry.new("ümlaut")
assert_entry entry, roundtrip(format, entry)
end
test "#{format.inspect} serializer handles non-ASCII-only version with bare string" do
entry = ActiveSupport::Cache::Entry.new("abc", version: "ümlaut")
assert_entry entry, roundtrip(format, entry)
end
test "#{format.inspect} serializer dumps bare string with reduced overhead when possible" do
string = "abc"
options = { version: "123", expires_in: 123 }
unsupported = string.encode(Encoding::WINDOWS_1252)
unoptimized = serializer(format).dump(ActiveSupport::Cache::Entry.new(unsupported, **options))
[Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding|
supported = string.encode(encoding)
optimized = serializer(format).dump(ActiveSupport::Cache::Entry.new(supported, **options))
assert_operator optimized.size, :<, unoptimized.size
end
end
test "#{format.inspect} serializer can compress bare strings" do
entry = ActiveSupport::Cache::Entry.new("abc" * 100, version: "123", expires_in: 123)
compressed = serializer(format).dump_compressed(entry, 1)
uncompressed = serializer(format).dump_compressed(entry, 100_000)
assert_operator compressed.bytesize, :<, uncompressed.bytesize
end
end
[:passthrough, :marshal_6_1, :marshal_7_0].each do |format|
test "#{format.inspect} serializer dumps bare string in a backward compatible way" do
string = +"abc"
string.instance_variable_set(:@baz, true)
roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new(string))
assert roundtripped.value.instance_variable_get(:@baz)
assert_entry @entry, serializer(format).load(compressed)
assert_entry @entry, serializer(format).load(uncompressed)
end
end
@ -120,7 +57,7 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
def to_msgpack_ext; ""; end
end
dumped = serializer(:message_pack).dump(ActiveSupport::Cache::Entry.new(klass.new))
dumped = serializer(:message_pack).dump(klass.new)
assert_not_nil dumped
assert_nil serializer(:message_pack).load(dumped)
end
@ -136,10 +73,6 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
ActiveSupport::Cache::SerializerWithFallback[format]
end
def roundtrip(format, entry)
serializer(format).load(serializer(format).dump(entry))
end
def assert_entry(expected, actual)
assert_equal \
[expected.value, expected.version, expected.expires_at],

1
activesupport/test/cache/stores/file_store_test.rb vendored
View File

@ -33,6 +33,7 @@ class FileStoreTest < ActiveSupport::TestCase
include CacheStoreVersionBehavior
include CacheStoreCoderBehavior
include CacheStoreCompressionBehavior
include CacheStoreSerializerBehavior
include CacheStoreFormatVersionBehavior
include CacheDeleteMatchedBehavior
include CacheIncrementDecrementBehavior

3
activesupport/test/cache/stores/mem_cache_store_test.rb vendored
View File

@ -78,6 +78,7 @@ class MemCacheStoreTest < ActiveSupport::TestCase
include CacheStoreVersionBehavior
include CacheStoreCoderBehavior
include CacheStoreCompressionBehavior
include CacheStoreSerializerBehavior
include CacheStoreFormatVersionBehavior
include LocalCacheBehavior
include CacheIncrementDecrementBehavior
@ -271,7 +272,7 @@ class MemCacheStoreTest < ActiveSupport::TestCase
compressed = Zlib::Deflate.deflate(val)
assert_called(
Zlib::Deflate,
Zlib,
:deflate,
"Memcached writes should not perform duplicate compression.",
times: 1,

1
activesupport/test/cache/stores/memory_store_test.rb vendored
View File

@ -17,6 +17,7 @@ class MemoryStoreTest < ActiveSupport::TestCase
include CacheStoreVersionBehavior
include CacheStoreCoderBehavior
include CacheStoreCompressionBehavior
include CacheStoreSerializerBehavior
include CacheDeleteMatchedBehavior
include CacheIncrementDecrementBehavior
include CacheInstrumentationBehavior

1
activesupport/test/cache/stores/redis_cache_store_test.rb vendored
View File

@ -161,6 +161,7 @@ module ActiveSupport::Cache::RedisCacheStoreTests
include CacheStoreCoderBehavior
include CacheStoreCompressionBehavior
include CacheStoreFormatVersionBehavior
include CacheStoreSerializerBehavior
include LocalCacheBehavior
include CacheIncrementDecrementBehavior
include CacheInstrumentationBehavior

7
guides/source/configuring.md
View File

@ -2411,9 +2411,10 @@ The default value depends on the `config.load_defaults` target version:
Specifies which serialization format to use for the cache. Possible values are
`6.1`, `7.0`, and `7.1`.
The `6.1`, `7.0`, and `7.1` formats all use `Marshal` for the default coder, but
`7.0` uses a more efficient representation for cache entries, and `7.1` includes
an additional optimization for bare string values such as view fragments.
`7.0` serializes cache entries more efficiently.
`7.1` further improves efficiency, and includes an optimization for bare string
values such as view fragments.
All formats are backward and forward compatible, meaning cache entries written
in one format can be read when using another format. This behavior makes it

5
railties/test/application/configuration_test.rb
View File

@ -4242,9 +4242,10 @@ module ApplicationTests
app "development"
assert_not_nil Rails.cache.instance_variable_get(:@coder)
assert_equal \
ActiveSupport::Cache::NullStore.new.instance_variable_get(:@coder),
Rails.cache.instance_variable_get(:@coder)
Marshal.dump(ActiveSupport::Cache::NullStore.new.instance_variable_get(:@coder)),
Marshal.dump(Rails.cache.instance_variable_get(:@coder))
end
test "ActiveSupport::Cache.format_version 6.1 is deprecated" do

2
railties/test/application/initializers/frameworks_test.rb
View File

@ -404,7 +404,7 @@ module ApplicationTests
rails %w(db:migrate)
add_to_config <<~RUBY
config.cache_store = :file_store, #{app_path("tmp/cache").inspect}, { coder: :message_pack }
config.cache_store = :file_store, #{app_path("tmp/cache").inspect}, { serializer: :message_pack }
RUBY
require "#{app_path}/config/environment"