diff --git a/activesupport/CHANGELOG.md b/activesupport/CHANGELOG.md
index b2206764b1e..b82b8f9fc57 100644
--- a/activesupport/CHANGELOG.md
+++ b/activesupport/CHANGELOG.md
@@ -1,3 +1,39 @@
+* Active Support cache stores now support replacing the default compressor via
+ a `:compressor` option. The specified compressor must respond to `deflate`
+ and `inflate`. For example:
+
+ ```ruby
+ module MyCompressor
+ def self.deflate(string)
+ # compression logic...
+ end
+
+ def self.inflate(compressed)
+ # decompression logic...
+ end
+ end
+
+ config.cache_store = :redis_cache_store, { compressor: MyCompressor }
+ ```
+
+ *Jonathan Hefner*
+
+* Active Support cache stores now support a `:serializer` option. Similar to
+ the `:coder` option, serializers must respond to `dump` and `load`. However,
+ serializers are only responsible for serializing a cached value, whereas
+ coders are responsible for serializing the entire `ActiveSupport::Cache::Entry`
+ instance. Additionally, the output from serializers can be automatically
+ compressed, whereas coders are responsible for their own compression.
+
+ Specifying a serializer instead of a coder also enables performance
+ optimizations, including the bare string optimization introduced by cache
+ format version 7.1.
+
+ The `:serializer` and `:coder` options are mutually exclusive. Specifying
+ both will raise an `ArgumentError`.
+
+ *Jonathan Hefner*
+
* Fix `ActiveSupport::Inflector.humanize(nil)` raising ``NoMethodError: undefined method `end_with?' for nil:NilClass``.
*James Robinson*
@@ -164,25 +200,23 @@
read caches from upgraded servers, leave the cache format unchanged on the
first deploy, then enable the `7.1` cache format on a subsequent deploy.
- The new `:message_pack` cache coder also includes this optimization.
-
*Jonathan Hefner*
-* The `:coder` option for Active Support cache stores now supports a
- `:message_pack` value:
+* Active Support cache stores can now use a preconfigured serializer based on
+ `ActiveSupport::MessagePack` via the `:serializer` option:
```ruby
- config.cache_store = :redis_cache_store, { coder: :message_pack }
+ config.cache_store = :redis_cache_store, { serializer: :message_pack }
```
- The `:message_pack` coder can reduce cache entry sizes and improve
+ The `:message_pack` serializer can reduce cache entry sizes and improve
performance, but requires the [`msgpack` gem](https://rubygems.org/gems/msgpack)
(>= 1.7.0).
- The `:message_pack` coder can read cache entries written by the default
- coder, and the default coder can now read entries written by the
- `:message_pack` coder. These behaviors make it easy to migrate between
- coders without invalidating the entire cache.
+ The `:message_pack` serializer can read cache entries written by the default
+ serializer, and the default serializer can now read entries written by the
+ `:message_pack` serializer. These behaviors make it easy to migrate between
+ serializer without invalidating the entire cache.
*Jonathan Hefner*
diff --git a/activesupport/lib/active_support/cache.rb b/activesupport/lib/active_support/cache.rb
index c245d03f195..6579d5c8cdf 100644
--- a/activesupport/lib/active_support/cache.rb
+++ b/activesupport/lib/active_support/cache.rb
@@ -1,5 +1,6 @@
# frozen_string_literal: true
+require "zlib"
require "active_support/core_ext/array/extract_options"
require "active_support/core_ext/enumerable"
require "active_support/core_ext/module/attribute_accessors"
@@ -7,6 +8,7 @@ require "active_support/core_ext/numeric/bytes"
require "active_support/core_ext/object/to_param"
require "active_support/core_ext/object/try"
require "active_support/core_ext/string/inflections"
+require_relative "cache/coder"
require_relative "cache/entry"
require_relative "cache/serializer_with_fallback"
@@ -25,11 +27,13 @@ module ActiveSupport
:coder,
:compress,
:compress_threshold,
+ :compressor,
:expire_in,
:expired_in,
:expires_in,
:namespace,
:race_condition_ttl,
+ :serializer,
:skip_nil,
]
@@ -249,27 +253,79 @@ module ActiveSupport
# Sets the namespace for the cache. This option is especially useful if
# your application shares a cache with other applications.
#
- # [+:coder+]
- # Replaces the default serializer for cache entries. +coder+ must
- # respond to +dump+ and +load+. Using a custom coder disables automatic
- # compression.
+ # [+:serializer+]
+ # The serializer for cached values. Must respond to +dump+ and +load+.
#
- # Alternatively, you can specify coder: :message_pack to use a
- # preconfigured coder based on ActiveSupport::MessagePack that supports
- # automatic compression and includes a fallback mechanism to load old
- # cache entries from the default coder. However, this option requires
- # the +msgpack+ gem.
+ # The default serializer depends on the cache format version (set via
+ # +config.active_support.cache_format_version+ when using Rails). The
+ # default serializer for each format version includes a fallback
+ # mechanism to deserialize values from any format version. This behavior
+ # makes it easy to migrate between format versions without invalidating
+ # the entire cache.
+ #
+ # You can also specify serializer: :message_pack to use a
+ # preconfigured serializer based on ActiveSupport::MessagePack. The
+ # +:message_pack+ serializer includes the same deserialization fallback
+ # mechanism, allowing easy migration from (or to) the default
+ # serializer. The +:message_pack+ serializer may improve performance,
+ # but it requires the +msgpack+ gem.
+ #
+ # [+:compressor+]
+ # The compressor for serialized cache values. Must respond to +deflate+
+ # and +inflate+.
+ #
+ # The default compressor is +Zlib+. To define a new custom compressor
+ # that also decompresses old cache entries, you can check compressed
+ # values for Zlib's "\x78" signature:
+ #
+ # module MyCompressor
+ # def self.deflate(dumped)
+ # # compression logic... (make sure result does not start with "\x78"!)
+ # end
+ #
+ # def self.inflate(compressed)
+ # if compressed.start_with?("\x78")
+ # Zlib.inflate(compressed)
+ # else
+ # # decompression logic...
+ # end
+ # end
+ # end
+ #
+ # ActiveSupport::Cache.lookup_store(:redis_cache_store, compressor: MyCompressor)
+ #
+ # [+:coder+]
+ # The coder for serializing and (optionally) compressing cache entries.
+ # Must respond to +dump+ and +load+.
+ #
+ # The default coder composes the serializer and compressor, and includes
+ # some performance optimizations. If you only need to override the
+ # serializer or compressor, you should specify the +:serializer+ or
+ # +:compressor+ options instead.
+ #
+ # The +:coder+ option is mutally exclusive with the +:serializer+ and
+ # +:compressor+ options. Specifying them together will raise an
+ # +ArgumentError+.
#
# Any other specified options are treated as default options for the
# relevant cache operations, such as #read, #write, and #fetch.
def initialize(options = nil)
- @options = options ? normalize_options(options) : {}
+ @options = options ? validate_options(normalize_options(options)) : {}
@options[:compress] = true unless @options.key?(:compress)
@options[:compress_threshold] ||= DEFAULT_COMPRESS_LIMIT
- @coder = @options.delete(:coder) { default_coder } || :passthrough
- @coder = Cache::SerializerWithFallback[@coder] if @coder.is_a?(Symbol)
+ @coder = @options.delete(:coder) do
+ legacy_serializer = Cache.format_version < 7.1 && !@options[:serializer]
+ serializer = @options.delete(:serializer) || default_serializer
+ serializer = Cache::SerializerWithFallback[serializer] if serializer.is_a?(Symbol)
+ compressor = @options.delete(:compressor) { Zlib }
+
+ Cache::Coder.new(serializer, compressor, legacy_serializer: legacy_serializer)
+ end
+
+ @coder ||= Cache::SerializerWithFallback[:passthrough]
+
@coder_supports_compression = @coder.respond_to?(:dump_compressed)
end
@@ -686,7 +742,7 @@ module ActiveSupport
end
private
- def default_coder
+ def default_serializer
case Cache.format_version
when 6.1
ActiveSupport.deprecator.warn <<~EOM
@@ -842,6 +898,23 @@ module ActiveSupport
options
end
+ def validate_options(options)
+ if options.key?(:coder) && options[:serializer]
+ raise ArgumentError, "Cannot specify :serializer and :coder options together"
+ end
+
+ if options.key?(:coder) && options[:compressor]
+ raise ArgumentError, "Cannot specify :compressor and :coder options together"
+ end
+
+ if Cache.format_version < 7.1 && !options[:serializer] && options[:compressor]
+ raise ArgumentError, "Cannot specify :compressor option when using" \
+ " default serializer and cache format version is < 7.1"
+ end
+
+ options
+ end
+
# Expands and namespaces the cache key.
# Raises an exception when the key is +nil+ or an empty string.
# May be overridden by cache stores to do additional normalization.
diff --git a/activesupport/lib/active_support/cache/coder.rb b/activesupport/lib/active_support/cache/coder.rb
new file mode 100644
index 00000000000..325596c8ead
--- /dev/null
+++ b/activesupport/lib/active_support/cache/coder.rb
@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+require_relative "entry"
+
+module ActiveSupport
+ module Cache
+ class Coder # :nodoc:
+ def initialize(serializer, compressor, legacy_serializer: false)
+ @serializer = serializer
+ @compressor = compressor
+ @legacy_serializer = legacy_serializer
+ end
+
+ def dump(entry)
+ return @serializer.dump(entry) if @legacy_serializer
+
+ dump_compressed(entry, Float::INFINITY)
+ end
+
+ def dump_compressed(entry, threshold)
+ return @serializer.dump_compressed(entry, threshold) if @legacy_serializer
+
+ # If value is a string with a supported encoding, use it as the payload
+ # instead of passing it through the serializer.
+ if type = type_for_string(entry.value)
+ payload = entry.value.b
+ else
+ type = OBJECT_DUMP_TYPE
+ payload = @serializer.dump(entry.value)
+ end
+
+ if compressed = try_compress(payload, threshold)
+ payload = compressed
+ type = type | COMPRESSED_FLAG
+ end
+
+ expires_at = entry.expires_at || -1.0
+
+ version = dump_version(entry.version) if entry.version
+ version_length = version&.bytesize || -1
+
+ packed = SIGNATURE.b
+ packed << [type, expires_at, version_length].pack(PACKED_TEMPLATE)
+ packed << version if version
+ packed << payload
+ end
+
+ def load(dumped)
+ return @serializer.load(dumped) if !signature?(dumped)
+
+ type = dumped.unpack1(PACKED_TYPE_TEMPLATE)
+ expires_at = dumped.unpack1(PACKED_EXPIRES_AT_TEMPLATE)
+ version_length = dumped.unpack1(PACKED_VERSION_LENGTH_TEMPLATE)
+
+ expires_at = nil if expires_at < 0
+ version = load_version(dumped.byteslice(PACKED_VERSION_INDEX, version_length)) if version_length >= 0
+ payload = dumped.byteslice((PACKED_VERSION_INDEX + [version_length, 0].max)..)
+
+ payload = @compressor.inflate(payload) if type & COMPRESSED_FLAG > 0
+
+ if string_encoding = STRING_ENCODINGS[type & ~COMPRESSED_FLAG]
+ value = payload.force_encoding(string_encoding)
+ else
+ value = @serializer.load(payload)
+ end
+
+ Cache::Entry.new(value, version: version, expires_at: expires_at)
+ end
+
+ private
+ SIGNATURE = "\x00\x11".b.freeze
+
+ OBJECT_DUMP_TYPE = 0x01
+
+ STRING_ENCODINGS = {
+ 0x02 => Encoding::UTF_8,
+ 0x03 => Encoding::BINARY,
+ 0x04 => Encoding::US_ASCII,
+ }
+
+ COMPRESSED_FLAG = 0x80
+
+ PACKED_TEMPLATE = "CEl<"
+ PACKED_TYPE_TEMPLATE = "@#{SIGNATURE.bytesize}C"
+ PACKED_EXPIRES_AT_TEMPLATE = "@#{[0].pack(PACKED_TYPE_TEMPLATE).bytesize}E"
+ PACKED_VERSION_LENGTH_TEMPLATE = "@#{[0].pack(PACKED_EXPIRES_AT_TEMPLATE).bytesize}l<"
+ PACKED_VERSION_INDEX = [0].pack(PACKED_VERSION_LENGTH_TEMPLATE).bytesize
+
+ MARSHAL_SIGNATURE = "\x04\x08".b.freeze
+
+ def signature?(dumped)
+ dumped.is_a?(String) && dumped.start_with?(SIGNATURE)
+ end
+
+ def type_for_string(value)
+ STRING_ENCODINGS.key(value.encoding) if value.instance_of?(String)
+ end
+
+ def try_compress(string, threshold)
+ if @compressor && string.bytesize >= threshold
+ compressed = @compressor.deflate(string)
+ compressed if compressed.bytesize < string.bytesize
+ end
+ end
+
+ def dump_version(version)
+ if version.encoding != Encoding::UTF_8 || version.start_with?(MARSHAL_SIGNATURE)
+ Marshal.dump(version)
+ else
+ version.b
+ end
+ end
+
+ def load_version(dumped_version)
+ if dumped_version.start_with?(MARSHAL_SIGNATURE)
+ Marshal.load(dumped_version)
+ else
+ dumped_version.force_encoding(Encoding::UTF_8)
+ end
+ end
+ end
+ end
+end
diff --git a/activesupport/lib/active_support/cache/mem_cache_store.rb b/activesupport/lib/active_support/cache/mem_cache_store.rb
index 2fb5b89483b..6caf0f91ab0 100644
--- a/activesupport/lib/active_support/cache/mem_cache_store.rb
+++ b/activesupport/lib/active_support/cache/mem_cache_store.rb
@@ -222,7 +222,7 @@ module ActiveSupport
end
private
- def default_coder
+ def default_serializer
if Cache.format_version == 6.1
ActiveSupport.deprecator.warn <<~EOM
Support for `config.active_support.cache_format_version = 6.1` has been deprecated and will be removed in Rails 7.2.
diff --git a/activesupport/lib/active_support/cache/memory_store.rb b/activesupport/lib/active_support/cache/memory_store.rb
index 16651fbea17..6450a910cd4 100644
--- a/activesupport/lib/active_support/cache/memory_store.rb
+++ b/activesupport/lib/active_support/cache/memory_store.rb
@@ -72,6 +72,7 @@ module ActiveSupport
def initialize(options = nil)
options ||= {}
+ options[:coder] = DupCoder unless options.key?(:coder) || options.key?(:serializer)
# Disable compression by default.
options[:compress] ||= false
super(options)
@@ -189,10 +190,6 @@ module ActiveSupport
private
PER_ENTRY_OVERHEAD = 240
- def default_coder
- DupCoder
- end
-
def cached_size(key, payload)
key.to_s.bytesize + payload.bytesize + PER_ENTRY_OVERHEAD
end
diff --git a/activesupport/lib/active_support/cache/serializer_with_fallback.rb b/activesupport/lib/active_support/cache/serializer_with_fallback.rb
index 8f139bb227a..12d0e8f76f5 100644
--- a/activesupport/lib/active_support/cache/serializer_with_fallback.rb
+++ b/activesupport/lib/active_support/cache/serializer_with_fallback.rb
@@ -14,28 +14,15 @@ module ActiveSupport
SERIALIZERS.fetch(format)
end
- def dump(entry)
- try_dump_bare_string(entry) || _dump(entry)
- end
-
- def dump_compressed(entry, threshold)
- dumped = dump(entry)
- try_compress(dumped, threshold) || dumped
- end
-
def load(dumped)
if dumped.is_a?(String)
- dumped = decompress(dumped) if compressed?(dumped)
-
case
- when loaded = try_load_bare_string(dumped)
- loaded
when MessagePackWithFallback.dumped?(dumped)
MessagePackWithFallback._load(dumped)
when Marshal71WithFallback.dumped?(dumped)
Marshal71WithFallback._load(dumped)
- when Marshal61WithFallback.dumped?(dumped)
- Marshal61WithFallback._load(dumped)
+ when Marshal70WithFallback.dumped?(dumped)
+ Marshal70WithFallback._load(dumped)
else
Cache::Store.logger&.warn("Unrecognized payload prefix #{dumped.byteslice(0).inspect}; deserializing as nil")
nil
@@ -49,72 +36,12 @@ module ActiveSupport
end
private
- BARE_STRING_SIGNATURES = {
- 255 => Encoding::UTF_8,
- 254 => Encoding::BINARY,
- 253 => Encoding::US_ASCII,
- }
- BARE_STRING_TEMPLATE = "CEl<"
- BARE_STRING_EXPIRES_AT_TEMPLATE = "@1E"
- BARE_STRING_VERSION_LENGTH_TEMPLATE = "@#{[0].pack(BARE_STRING_EXPIRES_AT_TEMPLATE).bytesize}l<"
- BARE_STRING_VERSION_INDEX = [0].pack(BARE_STRING_VERSION_LENGTH_TEMPLATE).bytesize
-
def marshal_load(payload)
Marshal.load(payload)
rescue ArgumentError => error
raise Cache::DeserializationError, error.message
end
- def try_dump_bare_string(entry)
- value = entry.value
- return if !value.instance_of?(String)
-
- version = entry.version
- return if version && version.encoding != Encoding::UTF_8
-
- signature = BARE_STRING_SIGNATURES.key(value.encoding)
- return if !signature
-
- packed = [signature, entry.expires_at || -1.0, version&.bytesize || -1].pack(BARE_STRING_TEMPLATE)
- packed << version.b if version
- packed << value.b
- end
-
- def try_load_bare_string(dumped)
- encoding = BARE_STRING_SIGNATURES[dumped.getbyte(0)]
- return if !encoding
- expires_at = dumped.unpack1(BARE_STRING_EXPIRES_AT_TEMPLATE)
- version_length = dumped.unpack1(BARE_STRING_VERSION_LENGTH_TEMPLATE)
- value_index = BARE_STRING_VERSION_INDEX + [version_length, 0].max
-
- Cache::Entry.new(
- dumped.byteslice(value_index..-1).force_encoding(encoding),
- version: dumped.byteslice(BARE_STRING_VERSION_INDEX, version_length)&.force_encoding(Encoding::UTF_8),
- expires_at: (expires_at unless expires_at < 0),
- )
- end
-
- ZLIB_HEADER = "\x78".b.freeze
-
- def compressed?(dumped)
- dumped.start_with?(ZLIB_HEADER)
- end
-
- def compress(dumped)
- Zlib::Deflate.deflate(dumped)
- end
-
- def try_compress(dumped, threshold)
- if dumped.bytesize >= threshold
- compressed = compress(dumped)
- compressed unless compressed.bytesize >= dumped.bytesize
- end
- end
-
- def decompress(compressed)
- Zlib::Inflate.inflate(compressed)
- end
-
module PassthroughWithFallback
include SerializerWithFallback
extend self
@@ -158,38 +85,32 @@ module ActiveSupport
end
end
- module Marshal71WithFallback
+ module Marshal70WithFallback
include SerializerWithFallback
extend self
MARK_UNCOMPRESSED = "\x00".b.freeze
MARK_COMPRESSED = "\x01".b.freeze
- def dump(entry, raw = false)
- if raw
- super(entry)
- else
- MARK_UNCOMPRESSED + super(entry)
- end
- end
-
- def _dump(entry)
- Marshal.dump(entry.pack)
+ def dump(entry)
+ MARK_UNCOMPRESSED + Marshal.dump(entry.pack)
end
def dump_compressed(entry, threshold)
- dumped = dump(entry, true)
- if compressed = try_compress(dumped, threshold)
- MARK_COMPRESSED + compressed
- else
- MARK_UNCOMPRESSED + dumped
+ dumped = Marshal.dump(entry.pack)
+
+ if dumped.bytesize >= threshold
+ compressed = Zlib::Deflate.deflate(dumped)
+ return MARK_COMPRESSED + compressed if compressed.bytesize < dumped.bytesize
end
+
+ MARK_UNCOMPRESSED + dumped
end
def _load(marked)
dumped = marked.byteslice(1..-1)
- dumped = decompress(dumped) if marked.start_with?(MARK_COMPRESSED)
- try_load_bare_string(dumped) || Cache::Entry.unpack(marshal_load(dumped))
+ dumped = Zlib::Inflate.inflate(dumped) if marked.start_with?(MARK_COMPRESSED)
+ Cache::Entry.unpack(marshal_load(dumped))
end
def dumped?(dumped)
@@ -197,12 +118,22 @@ module ActiveSupport
end
end
- module Marshal70WithFallback
- include Marshal71WithFallback
+ module Marshal71WithFallback
+ include SerializerWithFallback
extend self
- def try_dump_bare_string(_entry)
- nil # Prevent dumping bare strings.
+ MARSHAL_SIGNATURE = "\x04\x08".b.freeze
+
+ def dump(value)
+ Marshal.dump(value)
+ end
+
+ def _load(dumped)
+ marshal_load(dumped)
+ end
+
+ def dumped?(dumped)
+ dumped.start_with?(MARSHAL_SIGNATURE)
end
end
@@ -210,13 +141,12 @@ module ActiveSupport
include SerializerWithFallback
extend self
- def _dump(entry)
- ActiveSupport::MessagePack::CacheSerializer.dump(entry.pack)
+ def dump(value)
+ ActiveSupport::MessagePack::CacheSerializer.dump(value)
end
def _load(dumped)
- packed = ActiveSupport::MessagePack::CacheSerializer.load(dumped)
- Cache::Entry.unpack(packed) if packed
+ ActiveSupport::MessagePack::CacheSerializer.load(dumped)
end
def dumped?(dumped)
diff --git a/activesupport/test/cache/behaviors.rb b/activesupport/test/cache/behaviors.rb
index d4f68be4d0c..12bb8adc66d 100644
--- a/activesupport/test/cache/behaviors.rb
+++ b/activesupport/test/cache/behaviors.rb
@@ -9,6 +9,7 @@ require_relative "behaviors/cache_store_version_behavior"
require_relative "behaviors/cache_store_coder_behavior"
require_relative "behaviors/cache_store_compression_behavior"
require_relative "behaviors/cache_store_format_version_behavior"
+require_relative "behaviors/cache_store_serializer_behavior"
require_relative "behaviors/connection_pool_behavior"
require_relative "behaviors/encoded_key_cache_behavior"
require_relative "behaviors/failure_safety_behavior"
diff --git a/activesupport/test/cache/behaviors/cache_store_compression_behavior.rb b/activesupport/test/cache/behaviors/cache_store_compression_behavior.rb
index a20737e7d6b..02df1ac6c95 100644
--- a/activesupport/test/cache/behaviors/cache_store_compression_behavior.rb
+++ b/activesupport/test/cache/behaviors/cache_store_compression_behavior.rb
@@ -17,7 +17,7 @@ module CacheStoreCompressionBehavior
assert_compression true
end
- test "compression works with cache format version 7.1 (using Marshal71WithFallback)" do
+ test "compression works with cache format version >= 7.1 (using Cache::Coder)" do
@cache = with_format(7.1) { lookup_store(compress: true) }
assert_compression true
end
@@ -27,6 +27,11 @@ module CacheStoreCompressionBehavior
assert_compression false
end
+ test "compression works with custom serializer" do
+ @cache = with_format(7.1) { lookup_store(compress: true, serializer: Marshal) }
+ assert_compression true
+ end
+
test "compression by default" do
@cache = lookup_store
assert_compression !compression_always_disabled_by_default?
@@ -72,6 +77,47 @@ module CacheStoreCompressionBehavior
assert_not_compress "", with: { compress: true, compress_threshold: 1 }
assert_not_compress [*0..127].pack("C*"), with: { compress: true, compress_threshold: 1 }
end
+
+ test "compressor can be specified" do
+ lossy_compressor = Module.new do
+ def self.deflate(dumped)
+ "yolo"
+ end
+
+ def self.inflate(compressed)
+ Marshal.dump("lossy!") if compressed == "yolo"
+ end
+ end
+
+ @cache = with_format(7.1) do
+ lookup_store(compress: true, compressor: lossy_compressor, serializer: Marshal)
+ end
+ key = SecureRandom.uuid
+
+ @cache.write(key, LARGE_OBJECT)
+ assert_equal "lossy!", @cache.read(key)
+ end
+
+ test "compressor can be nil" do
+ @cache = with_format(7.1) { lookup_store(compressor: nil) }
+ assert_compression false
+ end
+
+ test "specifying a compressor raises when cache format version < 7.1" do
+ with_format(7.0) do
+ assert_raises ArgumentError, match: /compressor/i do
+ lookup_store(compressor: Zlib)
+ end
+ end
+ end
+
+ test "specifying a compressor raises when also specifying a coder" do
+ with_format(7.1) do
+ assert_raises ArgumentError, match: /compressor/i do
+ lookup_store(compressor: Zlib, coder: Marshal)
+ end
+ end
+ end
end
private
diff --git a/activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb b/activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb
index 580f89e6387..6fd75ab088b 100644
--- a/activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb
+++ b/activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb
@@ -15,8 +15,8 @@ module CacheStoreFormatVersionBehavior
"\x01\x78".b, # "\x01" + Zlib::Deflate.deflate(...)
],
7.1 => [
- "\x00\x04\x08[".b, # "\x00" + Marshal.dump(entry.pack)
- "\x01\x78".b, # "\x01" + Zlib::Deflate.deflate(...)
+ "\x00\x11\x01".b, # ActiveSupport::Cache::Coder#dump
+ "\x00\x11\x81".b, # ActiveSupport::Cache::Coder#dump_compressed
],
}
diff --git a/activesupport/test/cache/behaviors/cache_store_serializer_behavior.rb b/activesupport/test/cache/behaviors/cache_store_serializer_behavior.rb
new file mode 100644
index 00000000000..447ef7de28c
--- /dev/null
+++ b/activesupport/test/cache/behaviors/cache_store_serializer_behavior.rb
@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require "active_support/core_ext/object/with"
+
+module CacheStoreSerializerBehavior
+ extend ActiveSupport::Concern
+
+ included do
+ test "serializer can be specified" do
+ serializer = Module.new do
+ def self.dump(value)
+ value.class.name
+ end
+
+ def self.load(dumped)
+ Object.const_get(dumped)
+ end
+ end
+
+ @cache = with_format(7.1) { lookup_store(serializer: serializer) }
+ key = "key#{rand}"
+
+ @cache.write(key, 123)
+ assert_equal Integer, @cache.read(key)
+ end
+
+ test "serializer can be :message_pack" do
+ @cache = with_format(7.1) { lookup_store(serializer: :message_pack) }
+ key = "key#{rand}"
+
+ @cache.write(key, 123)
+ assert_equal 123, @cache.read(key)
+
+ assert_raises ActiveSupport::MessagePack::UnserializableObjectError do
+ @cache.write(key, Object.new)
+ end
+ end
+
+ test "specifying a serializer raises when also specifying a coder" do
+ with_format(7.1) do
+ assert_raises ArgumentError, match: /serializer/i do
+ lookup_store(serializer: Marshal, coder: Marshal)
+ end
+ end
+ end
+ end
+
+ private
+ def with_format(format_version, &block)
+ ActiveSupport.deprecator.silence do
+ ActiveSupport::Cache.with(format_version: format_version, &block)
+ end
+ end
+end
diff --git a/activesupport/test/cache/cache_coder_test.rb b/activesupport/test/cache/cache_coder_test.rb
new file mode 100644
index 00000000000..6a6dcb9c1f9
--- /dev/null
+++ b/activesupport/test/cache/cache_coder_test.rb
@@ -0,0 +1,126 @@
+# frozen_string_literal: true
+
+require_relative "../abstract_unit"
+require "active_support/core_ext/integer/time"
+
+class CacheCoderTest < ActiveSupport::TestCase
+ setup do
+ @coder = ActiveSupport::Cache::Coder.new(Serializer, Compressor)
+ end
+
+ test "roundtrips entry" do
+ ENTRIES.each do |entry|
+ assert_entry entry, @coder.load(@coder.dump(entry))
+ end
+ end
+
+ test "roundtrips entry when using compression" do
+ ENTRIES.each do |entry|
+ assert_entry entry, @coder.load(@coder.dump_compressed(entry, 1))
+ end
+ end
+
+ test "compresses values that are larger than the threshold" do
+ COMPRESSIBLE_ENTRIES.each do |entry|
+ dumped = @coder.dump(entry)
+ compressed = @coder.dump_compressed(entry, 1)
+
+ assert_operator compressed.bytesize, :<, dumped.bytesize
+ end
+ end
+
+ test "does not compress values that are smaller than the threshold" do
+ COMPRESSIBLE_ENTRIES.each do |entry|
+ dumped = @coder.dump(entry)
+ not_compressed = @coder.dump_compressed(entry, 1_000_000)
+
+ assert_equal dumped, not_compressed
+ end
+ end
+
+ test "does not apply compression to incompressible values" do
+ (ENTRIES - COMPRESSIBLE_ENTRIES).each do |entry|
+ dumped = @coder.dump(entry)
+ not_compressed = @coder.dump_compressed(entry, 1)
+
+ assert_equal dumped, not_compressed
+ end
+ end
+
+ test "loads dumped entries from original serializer" do
+ ENTRIES.each do |entry|
+ assert_entry entry, @coder.load(Serializer.dump(entry))
+ end
+ end
+
+ test "matches output of original serializer when legacy_serializer: true" do
+ @coder = ActiveSupport::Cache::Coder.new(Serializer, Compressor, legacy_serializer: true)
+
+ ENTRIES.each do |entry|
+ assert_equal Serializer.dump(entry), @coder.dump(entry)
+ assert_equal Serializer.dump_compressed(entry, 1), @coder.dump_compressed(entry, 1)
+ end
+ end
+
+ test "dumps bare strings with reduced overhead when possible" do
+ unoptimized = @coder.dump(ActiveSupport::Cache::Entry.new("".encode(Encoding::WINDOWS_1252)))
+
+ [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding|
+ optimized = @coder.dump(ActiveSupport::Cache::Entry.new("".encode(encoding)))
+ assert_operator optimized.size, :<, unoptimized.size
+ end
+ end
+
+ private
+ module Serializer
+ extend self
+
+ def dump(entry)
+ "SERIALIZED:" + Marshal.dump(entry)
+ end
+
+ def dump_compressed(*)
+ "via Serializer#dump_compressed"
+ end
+
+ def load(dumped)
+ Marshal.load(dumped.delete_prefix!("SERIALIZED:"))
+ end
+ end
+
+ module Compressor
+ extend self
+
+ def deflate(string)
+ "COMPRESSED:" + Zlib.deflate(string)
+ end
+
+ def inflate(deflated)
+ Zlib.inflate(deflated.delete_prefix!("COMPRESSED:"))
+ end
+ end
+
+ STRING = "x" * 100
+ COMPRESSIBLE_VALUES = [
+ { string: STRING },
+ STRING,
+ STRING.encode(Encoding::BINARY),
+ STRING.encode(Encoding::US_ASCII),
+ STRING.encode(Encoding::WINDOWS_1252),
+ ]
+ VALUES = [nil, true, 1, "", "ümlaut", [*0..255].pack("C*"), *COMPRESSIBLE_VALUES]
+ VERSIONS = [nil, "", "ümlaut", [*0..255].pack("C*"), "x" * 256]
+ EXPIRIES = [nil, 0, 100.years]
+
+ ENTRIES = VALUES.product(VERSIONS, EXPIRIES).map do |value, version, expires_in|
+ ActiveSupport::Cache::Entry.new(value, version: version, expires_in: expires_in).freeze
+ end
+
+ COMPRESSIBLE_ENTRIES = ENTRIES.select { |entry| COMPRESSIBLE_VALUES.include?(entry.value) }
+
+ def assert_entry(expected, actual)
+ assert_equal \
+ [expected.value, expected.version, expected.expires_at],
+ [actual.value, actual.version, actual.expires_at]
+ end
+end
diff --git a/activesupport/test/cache/serializer_with_fallback_test.rb b/activesupport/test/cache/serializer_with_fallback_test.rb
index f7e027eb05d..acef0ca8f1b 100644
--- a/activesupport/test/cache/serializer_with_fallback_test.rb
+++ b/activesupport/test/cache/serializer_with_fallback_test.rb
@@ -5,6 +5,7 @@ require "active_support/core_ext/object/with"
class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
FORMATS = ActiveSupport::Cache::SerializerWithFallback::SERIALIZERS.keys
+ LEGACY_FORMATS = [:passthrough, :marshal_6_1, :marshal_7_0]
setup do
@entry = ActiveSupport::Cache::Entry.new(
@@ -12,28 +13,21 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
)
end
- FORMATS.product(FORMATS) do |load_format, dump_format|
- test "#{load_format.inspect} serializer can load #{dump_format.inspect} dump" do
- dumped = serializer(dump_format).dump(@entry)
- assert_entry @entry, serializer(load_format).load(dumped)
+ FORMATS.product(FORMATS - LEGACY_FORMATS) do |loader, dumper|
+ test "#{loader.inspect} serializer can load #{dumper.inspect} dump" do
+ dumped = serializer(dumper).dump(@entry.value)
+ assert_equal @entry.value, serializer(loader).load(dumped)
end
+ end
- test "#{load_format.inspect} serializer can load #{dump_format.inspect} dump with compression" do
- compressed = serializer(dump_format).dump_compressed(@entry, 1)
- assert_entry @entry, serializer(load_format).load(compressed)
-
- uncompressed = serializer(dump_format).dump_compressed(@entry, 100_000)
- assert_entry @entry, serializer(load_format).load(uncompressed)
+ FORMATS.product(LEGACY_FORMATS) do |loader, dumper|
+ test "#{loader.inspect} serializer can load #{dumper.inspect} dump" do
+ dumped = serializer(dumper).dump(@entry)
+ assert_entry @entry, serializer(loader).load(dumped)
end
end
FORMATS.each do |format|
- test "#{format.inspect} serializer can compress entries" do
- compressed = serializer(format).dump_compressed(@entry, 1)
- uncompressed = serializer(format).dump_compressed(@entry, 100_000)
- assert_operator compressed.bytesize, :<, uncompressed.bytesize
- end
-
test "#{format.inspect} serializer handles unrecognized payloads gracefully" do
assert_nil serializer(format).load(Object.new)
assert_nil serializer(format).load("")
@@ -45,71 +39,14 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
end
end
- (FORMATS - [:passthrough, :marshal_6_1, :marshal_7_0]).each do |format|
- test "#{format.inspect} serializer preserves version with bare string" do
- entry = ActiveSupport::Cache::Entry.new("abc", version: "123")
- assert_entry entry, roundtrip(format, entry)
- end
+ LEGACY_FORMATS.each do |format|
+ test "#{format.inspect} serializer can compress entries" do
+ compressed = serializer(format).dump_compressed(@entry, 1)
+ uncompressed = serializer(format).dump_compressed(@entry, 100_000)
- test "#{format.inspect} serializer preserves expiration with bare string" do
- entry = ActiveSupport::Cache::Entry.new("abc", expires_in: 123)
- assert_entry entry, roundtrip(format, entry)
- end
-
- test "#{format.inspect} serializer preserves encoding of version with bare string" do
- [Encoding::UTF_8, Encoding::BINARY].each do |encoding|
- version = "123".encode(encoding)
- roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new("abc", version: version))
- assert_equal version.encoding, roundtripped.version.encoding
- end
- end
-
- test "#{format.inspect} serializer preserves encoding of bare string" do
- [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding|
- string = "abc".encode(encoding)
- roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new(string))
- assert_equal string.encoding, roundtripped.value.encoding
- end
- end
-
- test "#{format.inspect} serializer handles non-ASCII-only bare string" do
- entry = ActiveSupport::Cache::Entry.new("ümlaut")
- assert_entry entry, roundtrip(format, entry)
- end
-
- test "#{format.inspect} serializer handles non-ASCII-only version with bare string" do
- entry = ActiveSupport::Cache::Entry.new("abc", version: "ümlaut")
- assert_entry entry, roundtrip(format, entry)
- end
-
- test "#{format.inspect} serializer dumps bare string with reduced overhead when possible" do
- string = "abc"
- options = { version: "123", expires_in: 123 }
-
- unsupported = string.encode(Encoding::WINDOWS_1252)
- unoptimized = serializer(format).dump(ActiveSupport::Cache::Entry.new(unsupported, **options))
-
- [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding|
- supported = string.encode(encoding)
- optimized = serializer(format).dump(ActiveSupport::Cache::Entry.new(supported, **options))
- assert_operator optimized.size, :<, unoptimized.size
- end
- end
-
- test "#{format.inspect} serializer can compress bare strings" do
- entry = ActiveSupport::Cache::Entry.new("abc" * 100, version: "123", expires_in: 123)
- compressed = serializer(format).dump_compressed(entry, 1)
- uncompressed = serializer(format).dump_compressed(entry, 100_000)
assert_operator compressed.bytesize, :<, uncompressed.bytesize
- end
- end
-
- [:passthrough, :marshal_6_1, :marshal_7_0].each do |format|
- test "#{format.inspect} serializer dumps bare string in a backward compatible way" do
- string = +"abc"
- string.instance_variable_set(:@baz, true)
- roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new(string))
- assert roundtripped.value.instance_variable_get(:@baz)
+ assert_entry @entry, serializer(format).load(compressed)
+ assert_entry @entry, serializer(format).load(uncompressed)
end
end
@@ -120,7 +57,7 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
def to_msgpack_ext; ""; end
end
- dumped = serializer(:message_pack).dump(ActiveSupport::Cache::Entry.new(klass.new))
+ dumped = serializer(:message_pack).dump(klass.new)
assert_not_nil dumped
assert_nil serializer(:message_pack).load(dumped)
end
@@ -136,10 +73,6 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase
ActiveSupport::Cache::SerializerWithFallback[format]
end
- def roundtrip(format, entry)
- serializer(format).load(serializer(format).dump(entry))
- end
-
def assert_entry(expected, actual)
assert_equal \
[expected.value, expected.version, expected.expires_at],
diff --git a/activesupport/test/cache/stores/file_store_test.rb b/activesupport/test/cache/stores/file_store_test.rb
index 7a1217a67eb..68130e8d3d2 100644
--- a/activesupport/test/cache/stores/file_store_test.rb
+++ b/activesupport/test/cache/stores/file_store_test.rb
@@ -33,6 +33,7 @@ class FileStoreTest < ActiveSupport::TestCase
include CacheStoreVersionBehavior
include CacheStoreCoderBehavior
include CacheStoreCompressionBehavior
+ include CacheStoreSerializerBehavior
include CacheStoreFormatVersionBehavior
include CacheDeleteMatchedBehavior
include CacheIncrementDecrementBehavior
diff --git a/activesupport/test/cache/stores/mem_cache_store_test.rb b/activesupport/test/cache/stores/mem_cache_store_test.rb
index 40b0dd6c8d8..3789a0e2a8e 100644
--- a/activesupport/test/cache/stores/mem_cache_store_test.rb
+++ b/activesupport/test/cache/stores/mem_cache_store_test.rb
@@ -78,6 +78,7 @@ class MemCacheStoreTest < ActiveSupport::TestCase
include CacheStoreVersionBehavior
include CacheStoreCoderBehavior
include CacheStoreCompressionBehavior
+ include CacheStoreSerializerBehavior
include CacheStoreFormatVersionBehavior
include LocalCacheBehavior
include CacheIncrementDecrementBehavior
@@ -271,7 +272,7 @@ class MemCacheStoreTest < ActiveSupport::TestCase
compressed = Zlib::Deflate.deflate(val)
assert_called(
- Zlib::Deflate,
+ Zlib,
:deflate,
"Memcached writes should not perform duplicate compression.",
times: 1,
diff --git a/activesupport/test/cache/stores/memory_store_test.rb b/activesupport/test/cache/stores/memory_store_test.rb
index e2f530fbea4..2941e4c4d17 100644
--- a/activesupport/test/cache/stores/memory_store_test.rb
+++ b/activesupport/test/cache/stores/memory_store_test.rb
@@ -17,6 +17,7 @@ class MemoryStoreTest < ActiveSupport::TestCase
include CacheStoreVersionBehavior
include CacheStoreCoderBehavior
include CacheStoreCompressionBehavior
+ include CacheStoreSerializerBehavior
include CacheDeleteMatchedBehavior
include CacheIncrementDecrementBehavior
include CacheInstrumentationBehavior
diff --git a/activesupport/test/cache/stores/redis_cache_store_test.rb b/activesupport/test/cache/stores/redis_cache_store_test.rb
index da938c34c45..b566eac2089 100644
--- a/activesupport/test/cache/stores/redis_cache_store_test.rb
+++ b/activesupport/test/cache/stores/redis_cache_store_test.rb
@@ -161,6 +161,7 @@ module ActiveSupport::Cache::RedisCacheStoreTests
include CacheStoreCoderBehavior
include CacheStoreCompressionBehavior
include CacheStoreFormatVersionBehavior
+ include CacheStoreSerializerBehavior
include LocalCacheBehavior
include CacheIncrementDecrementBehavior
include CacheInstrumentationBehavior
diff --git a/guides/source/configuring.md b/guides/source/configuring.md
index c7299ee56a8..107e62c4a72 100644
--- a/guides/source/configuring.md
+++ b/guides/source/configuring.md
@@ -2411,9 +2411,10 @@ The default value depends on the `config.load_defaults` target version:
Specifies which serialization format to use for the cache. Possible values are
`6.1`, `7.0`, and `7.1`.
-The `6.1`, `7.0`, and `7.1` formats all use `Marshal` for the default coder, but
-`7.0` uses a more efficient representation for cache entries, and `7.1` includes
-an additional optimization for bare string values such as view fragments.
+`7.0` serializes cache entries more efficiently.
+
+`7.1` further improves efficiency, and includes an optimization for bare string
+values such as view fragments.
All formats are backward and forward compatible, meaning cache entries written
in one format can be read when using another format. This behavior makes it
diff --git a/railties/test/application/configuration_test.rb b/railties/test/application/configuration_test.rb
index c252145bd7c..eeb48c2d236 100644
--- a/railties/test/application/configuration_test.rb
+++ b/railties/test/application/configuration_test.rb
@@ -4242,9 +4242,10 @@ module ApplicationTests
app "development"
+ assert_not_nil Rails.cache.instance_variable_get(:@coder)
assert_equal \
- ActiveSupport::Cache::NullStore.new.instance_variable_get(:@coder),
- Rails.cache.instance_variable_get(:@coder)
+ Marshal.dump(ActiveSupport::Cache::NullStore.new.instance_variable_get(:@coder)),
+ Marshal.dump(Rails.cache.instance_variable_get(:@coder))
end
test "ActiveSupport::Cache.format_version 6.1 is deprecated" do
diff --git a/railties/test/application/initializers/frameworks_test.rb b/railties/test/application/initializers/frameworks_test.rb
index 80729b67614..1fcdca6127b 100644
--- a/railties/test/application/initializers/frameworks_test.rb
+++ b/railties/test/application/initializers/frameworks_test.rb
@@ -404,7 +404,7 @@ module ApplicationTests
rails %w(db:migrate)
add_to_config <<~RUBY
- config.cache_store = :file_store, #{app_path("tmp/cache").inspect}, { coder: :message_pack }
+ config.cache_store = :file_store, #{app_path("tmp/cache").inspect}, { serializer: :message_pack }
RUBY
require "#{app_path}/config/environment"