diff --git a/activesupport/CHANGELOG.md b/activesupport/CHANGELOG.md index b2206764b1e..b82b8f9fc57 100644 --- a/activesupport/CHANGELOG.md +++ b/activesupport/CHANGELOG.md @@ -1,3 +1,39 @@ +* Active Support cache stores now support replacing the default compressor via + a `:compressor` option. The specified compressor must respond to `deflate` + and `inflate`. For example: + + ```ruby + module MyCompressor + def self.deflate(string) + # compression logic... + end + + def self.inflate(compressed) + # decompression logic... + end + end + + config.cache_store = :redis_cache_store, { compressor: MyCompressor } + ``` + + *Jonathan Hefner* + +* Active Support cache stores now support a `:serializer` option. Similar to + the `:coder` option, serializers must respond to `dump` and `load`. However, + serializers are only responsible for serializing a cached value, whereas + coders are responsible for serializing the entire `ActiveSupport::Cache::Entry` + instance. Additionally, the output from serializers can be automatically + compressed, whereas coders are responsible for their own compression. + + Specifying a serializer instead of a coder also enables performance + optimizations, including the bare string optimization introduced by cache + format version 7.1. + + The `:serializer` and `:coder` options are mutually exclusive. Specifying + both will raise an `ArgumentError`. + + *Jonathan Hefner* + * Fix `ActiveSupport::Inflector.humanize(nil)` raising ``NoMethodError: undefined method `end_with?' for nil:NilClass``. *James Robinson* @@ -164,25 +200,23 @@ read caches from upgraded servers, leave the cache format unchanged on the first deploy, then enable the `7.1` cache format on a subsequent deploy. - The new `:message_pack` cache coder also includes this optimization. - *Jonathan Hefner* -* The `:coder` option for Active Support cache stores now supports a - `:message_pack` value: +* Active Support cache stores can now use a preconfigured serializer based on + `ActiveSupport::MessagePack` via the `:serializer` option: ```ruby - config.cache_store = :redis_cache_store, { coder: :message_pack } + config.cache_store = :redis_cache_store, { serializer: :message_pack } ``` - The `:message_pack` coder can reduce cache entry sizes and improve + The `:message_pack` serializer can reduce cache entry sizes and improve performance, but requires the [`msgpack` gem](https://rubygems.org/gems/msgpack) (>= 1.7.0). - The `:message_pack` coder can read cache entries written by the default - coder, and the default coder can now read entries written by the - `:message_pack` coder. These behaviors make it easy to migrate between - coders without invalidating the entire cache. + The `:message_pack` serializer can read cache entries written by the default + serializer, and the default serializer can now read entries written by the + `:message_pack` serializer. These behaviors make it easy to migrate between + serializer without invalidating the entire cache. *Jonathan Hefner* diff --git a/activesupport/lib/active_support/cache.rb b/activesupport/lib/active_support/cache.rb index c245d03f195..6579d5c8cdf 100644 --- a/activesupport/lib/active_support/cache.rb +++ b/activesupport/lib/active_support/cache.rb @@ -1,5 +1,6 @@ # frozen_string_literal: true +require "zlib" require "active_support/core_ext/array/extract_options" require "active_support/core_ext/enumerable" require "active_support/core_ext/module/attribute_accessors" @@ -7,6 +8,7 @@ require "active_support/core_ext/numeric/bytes" require "active_support/core_ext/object/to_param" require "active_support/core_ext/object/try" require "active_support/core_ext/string/inflections" +require_relative "cache/coder" require_relative "cache/entry" require_relative "cache/serializer_with_fallback" @@ -25,11 +27,13 @@ module ActiveSupport :coder, :compress, :compress_threshold, + :compressor, :expire_in, :expired_in, :expires_in, :namespace, :race_condition_ttl, + :serializer, :skip_nil, ] @@ -249,27 +253,79 @@ module ActiveSupport # Sets the namespace for the cache. This option is especially useful if # your application shares a cache with other applications. # - # [+:coder+] - # Replaces the default serializer for cache entries. +coder+ must - # respond to +dump+ and +load+. Using a custom coder disables automatic - # compression. + # [+:serializer+] + # The serializer for cached values. Must respond to +dump+ and +load+. # - # Alternatively, you can specify coder: :message_pack to use a - # preconfigured coder based on ActiveSupport::MessagePack that supports - # automatic compression and includes a fallback mechanism to load old - # cache entries from the default coder. However, this option requires - # the +msgpack+ gem. + # The default serializer depends on the cache format version (set via + # +config.active_support.cache_format_version+ when using Rails). The + # default serializer for each format version includes a fallback + # mechanism to deserialize values from any format version. This behavior + # makes it easy to migrate between format versions without invalidating + # the entire cache. + # + # You can also specify serializer: :message_pack to use a + # preconfigured serializer based on ActiveSupport::MessagePack. The + # +:message_pack+ serializer includes the same deserialization fallback + # mechanism, allowing easy migration from (or to) the default + # serializer. The +:message_pack+ serializer may improve performance, + # but it requires the +msgpack+ gem. + # + # [+:compressor+] + # The compressor for serialized cache values. Must respond to +deflate+ + # and +inflate+. + # + # The default compressor is +Zlib+. To define a new custom compressor + # that also decompresses old cache entries, you can check compressed + # values for Zlib's "\x78" signature: + # + # module MyCompressor + # def self.deflate(dumped) + # # compression logic... (make sure result does not start with "\x78"!) + # end + # + # def self.inflate(compressed) + # if compressed.start_with?("\x78") + # Zlib.inflate(compressed) + # else + # # decompression logic... + # end + # end + # end + # + # ActiveSupport::Cache.lookup_store(:redis_cache_store, compressor: MyCompressor) + # + # [+:coder+] + # The coder for serializing and (optionally) compressing cache entries. + # Must respond to +dump+ and +load+. + # + # The default coder composes the serializer and compressor, and includes + # some performance optimizations. If you only need to override the + # serializer or compressor, you should specify the +:serializer+ or + # +:compressor+ options instead. + # + # The +:coder+ option is mutally exclusive with the +:serializer+ and + # +:compressor+ options. Specifying them together will raise an + # +ArgumentError+. # # Any other specified options are treated as default options for the # relevant cache operations, such as #read, #write, and #fetch. def initialize(options = nil) - @options = options ? normalize_options(options) : {} + @options = options ? validate_options(normalize_options(options)) : {} @options[:compress] = true unless @options.key?(:compress) @options[:compress_threshold] ||= DEFAULT_COMPRESS_LIMIT - @coder = @options.delete(:coder) { default_coder } || :passthrough - @coder = Cache::SerializerWithFallback[@coder] if @coder.is_a?(Symbol) + @coder = @options.delete(:coder) do + legacy_serializer = Cache.format_version < 7.1 && !@options[:serializer] + serializer = @options.delete(:serializer) || default_serializer + serializer = Cache::SerializerWithFallback[serializer] if serializer.is_a?(Symbol) + compressor = @options.delete(:compressor) { Zlib } + + Cache::Coder.new(serializer, compressor, legacy_serializer: legacy_serializer) + end + + @coder ||= Cache::SerializerWithFallback[:passthrough] + @coder_supports_compression = @coder.respond_to?(:dump_compressed) end @@ -686,7 +742,7 @@ module ActiveSupport end private - def default_coder + def default_serializer case Cache.format_version when 6.1 ActiveSupport.deprecator.warn <<~EOM @@ -842,6 +898,23 @@ module ActiveSupport options end + def validate_options(options) + if options.key?(:coder) && options[:serializer] + raise ArgumentError, "Cannot specify :serializer and :coder options together" + end + + if options.key?(:coder) && options[:compressor] + raise ArgumentError, "Cannot specify :compressor and :coder options together" + end + + if Cache.format_version < 7.1 && !options[:serializer] && options[:compressor] + raise ArgumentError, "Cannot specify :compressor option when using" \ + " default serializer and cache format version is < 7.1" + end + + options + end + # Expands and namespaces the cache key. # Raises an exception when the key is +nil+ or an empty string. # May be overridden by cache stores to do additional normalization. diff --git a/activesupport/lib/active_support/cache/coder.rb b/activesupport/lib/active_support/cache/coder.rb new file mode 100644 index 00000000000..325596c8ead --- /dev/null +++ b/activesupport/lib/active_support/cache/coder.rb @@ -0,0 +1,123 @@ +# frozen_string_literal: true + +require_relative "entry" + +module ActiveSupport + module Cache + class Coder # :nodoc: + def initialize(serializer, compressor, legacy_serializer: false) + @serializer = serializer + @compressor = compressor + @legacy_serializer = legacy_serializer + end + + def dump(entry) + return @serializer.dump(entry) if @legacy_serializer + + dump_compressed(entry, Float::INFINITY) + end + + def dump_compressed(entry, threshold) + return @serializer.dump_compressed(entry, threshold) if @legacy_serializer + + # If value is a string with a supported encoding, use it as the payload + # instead of passing it through the serializer. + if type = type_for_string(entry.value) + payload = entry.value.b + else + type = OBJECT_DUMP_TYPE + payload = @serializer.dump(entry.value) + end + + if compressed = try_compress(payload, threshold) + payload = compressed + type = type | COMPRESSED_FLAG + end + + expires_at = entry.expires_at || -1.0 + + version = dump_version(entry.version) if entry.version + version_length = version&.bytesize || -1 + + packed = SIGNATURE.b + packed << [type, expires_at, version_length].pack(PACKED_TEMPLATE) + packed << version if version + packed << payload + end + + def load(dumped) + return @serializer.load(dumped) if !signature?(dumped) + + type = dumped.unpack1(PACKED_TYPE_TEMPLATE) + expires_at = dumped.unpack1(PACKED_EXPIRES_AT_TEMPLATE) + version_length = dumped.unpack1(PACKED_VERSION_LENGTH_TEMPLATE) + + expires_at = nil if expires_at < 0 + version = load_version(dumped.byteslice(PACKED_VERSION_INDEX, version_length)) if version_length >= 0 + payload = dumped.byteslice((PACKED_VERSION_INDEX + [version_length, 0].max)..) + + payload = @compressor.inflate(payload) if type & COMPRESSED_FLAG > 0 + + if string_encoding = STRING_ENCODINGS[type & ~COMPRESSED_FLAG] + value = payload.force_encoding(string_encoding) + else + value = @serializer.load(payload) + end + + Cache::Entry.new(value, version: version, expires_at: expires_at) + end + + private + SIGNATURE = "\x00\x11".b.freeze + + OBJECT_DUMP_TYPE = 0x01 + + STRING_ENCODINGS = { + 0x02 => Encoding::UTF_8, + 0x03 => Encoding::BINARY, + 0x04 => Encoding::US_ASCII, + } + + COMPRESSED_FLAG = 0x80 + + PACKED_TEMPLATE = "CEl<" + PACKED_TYPE_TEMPLATE = "@#{SIGNATURE.bytesize}C" + PACKED_EXPIRES_AT_TEMPLATE = "@#{[0].pack(PACKED_TYPE_TEMPLATE).bytesize}E" + PACKED_VERSION_LENGTH_TEMPLATE = "@#{[0].pack(PACKED_EXPIRES_AT_TEMPLATE).bytesize}l<" + PACKED_VERSION_INDEX = [0].pack(PACKED_VERSION_LENGTH_TEMPLATE).bytesize + + MARSHAL_SIGNATURE = "\x04\x08".b.freeze + + def signature?(dumped) + dumped.is_a?(String) && dumped.start_with?(SIGNATURE) + end + + def type_for_string(value) + STRING_ENCODINGS.key(value.encoding) if value.instance_of?(String) + end + + def try_compress(string, threshold) + if @compressor && string.bytesize >= threshold + compressed = @compressor.deflate(string) + compressed if compressed.bytesize < string.bytesize + end + end + + def dump_version(version) + if version.encoding != Encoding::UTF_8 || version.start_with?(MARSHAL_SIGNATURE) + Marshal.dump(version) + else + version.b + end + end + + def load_version(dumped_version) + if dumped_version.start_with?(MARSHAL_SIGNATURE) + Marshal.load(dumped_version) + else + dumped_version.force_encoding(Encoding::UTF_8) + end + end + end + end +end diff --git a/activesupport/lib/active_support/cache/mem_cache_store.rb b/activesupport/lib/active_support/cache/mem_cache_store.rb index 2fb5b89483b..6caf0f91ab0 100644 --- a/activesupport/lib/active_support/cache/mem_cache_store.rb +++ b/activesupport/lib/active_support/cache/mem_cache_store.rb @@ -222,7 +222,7 @@ module ActiveSupport end private - def default_coder + def default_serializer if Cache.format_version == 6.1 ActiveSupport.deprecator.warn <<~EOM Support for `config.active_support.cache_format_version = 6.1` has been deprecated and will be removed in Rails 7.2. diff --git a/activesupport/lib/active_support/cache/memory_store.rb b/activesupport/lib/active_support/cache/memory_store.rb index 16651fbea17..6450a910cd4 100644 --- a/activesupport/lib/active_support/cache/memory_store.rb +++ b/activesupport/lib/active_support/cache/memory_store.rb @@ -72,6 +72,7 @@ module ActiveSupport def initialize(options = nil) options ||= {} + options[:coder] = DupCoder unless options.key?(:coder) || options.key?(:serializer) # Disable compression by default. options[:compress] ||= false super(options) @@ -189,10 +190,6 @@ module ActiveSupport private PER_ENTRY_OVERHEAD = 240 - def default_coder - DupCoder - end - def cached_size(key, payload) key.to_s.bytesize + payload.bytesize + PER_ENTRY_OVERHEAD end diff --git a/activesupport/lib/active_support/cache/serializer_with_fallback.rb b/activesupport/lib/active_support/cache/serializer_with_fallback.rb index 8f139bb227a..12d0e8f76f5 100644 --- a/activesupport/lib/active_support/cache/serializer_with_fallback.rb +++ b/activesupport/lib/active_support/cache/serializer_with_fallback.rb @@ -14,28 +14,15 @@ module ActiveSupport SERIALIZERS.fetch(format) end - def dump(entry) - try_dump_bare_string(entry) || _dump(entry) - end - - def dump_compressed(entry, threshold) - dumped = dump(entry) - try_compress(dumped, threshold) || dumped - end - def load(dumped) if dumped.is_a?(String) - dumped = decompress(dumped) if compressed?(dumped) - case - when loaded = try_load_bare_string(dumped) - loaded when MessagePackWithFallback.dumped?(dumped) MessagePackWithFallback._load(dumped) when Marshal71WithFallback.dumped?(dumped) Marshal71WithFallback._load(dumped) - when Marshal61WithFallback.dumped?(dumped) - Marshal61WithFallback._load(dumped) + when Marshal70WithFallback.dumped?(dumped) + Marshal70WithFallback._load(dumped) else Cache::Store.logger&.warn("Unrecognized payload prefix #{dumped.byteslice(0).inspect}; deserializing as nil") nil @@ -49,72 +36,12 @@ module ActiveSupport end private - BARE_STRING_SIGNATURES = { - 255 => Encoding::UTF_8, - 254 => Encoding::BINARY, - 253 => Encoding::US_ASCII, - } - BARE_STRING_TEMPLATE = "CEl<" - BARE_STRING_EXPIRES_AT_TEMPLATE = "@1E" - BARE_STRING_VERSION_LENGTH_TEMPLATE = "@#{[0].pack(BARE_STRING_EXPIRES_AT_TEMPLATE).bytesize}l<" - BARE_STRING_VERSION_INDEX = [0].pack(BARE_STRING_VERSION_LENGTH_TEMPLATE).bytesize - def marshal_load(payload) Marshal.load(payload) rescue ArgumentError => error raise Cache::DeserializationError, error.message end - def try_dump_bare_string(entry) - value = entry.value - return if !value.instance_of?(String) - - version = entry.version - return if version && version.encoding != Encoding::UTF_8 - - signature = BARE_STRING_SIGNATURES.key(value.encoding) - return if !signature - - packed = [signature, entry.expires_at || -1.0, version&.bytesize || -1].pack(BARE_STRING_TEMPLATE) - packed << version.b if version - packed << value.b - end - - def try_load_bare_string(dumped) - encoding = BARE_STRING_SIGNATURES[dumped.getbyte(0)] - return if !encoding - expires_at = dumped.unpack1(BARE_STRING_EXPIRES_AT_TEMPLATE) - version_length = dumped.unpack1(BARE_STRING_VERSION_LENGTH_TEMPLATE) - value_index = BARE_STRING_VERSION_INDEX + [version_length, 0].max - - Cache::Entry.new( - dumped.byteslice(value_index..-1).force_encoding(encoding), - version: dumped.byteslice(BARE_STRING_VERSION_INDEX, version_length)&.force_encoding(Encoding::UTF_8), - expires_at: (expires_at unless expires_at < 0), - ) - end - - ZLIB_HEADER = "\x78".b.freeze - - def compressed?(dumped) - dumped.start_with?(ZLIB_HEADER) - end - - def compress(dumped) - Zlib::Deflate.deflate(dumped) - end - - def try_compress(dumped, threshold) - if dumped.bytesize >= threshold - compressed = compress(dumped) - compressed unless compressed.bytesize >= dumped.bytesize - end - end - - def decompress(compressed) - Zlib::Inflate.inflate(compressed) - end - module PassthroughWithFallback include SerializerWithFallback extend self @@ -158,38 +85,32 @@ module ActiveSupport end end - module Marshal71WithFallback + module Marshal70WithFallback include SerializerWithFallback extend self MARK_UNCOMPRESSED = "\x00".b.freeze MARK_COMPRESSED = "\x01".b.freeze - def dump(entry, raw = false) - if raw - super(entry) - else - MARK_UNCOMPRESSED + super(entry) - end - end - - def _dump(entry) - Marshal.dump(entry.pack) + def dump(entry) + MARK_UNCOMPRESSED + Marshal.dump(entry.pack) end def dump_compressed(entry, threshold) - dumped = dump(entry, true) - if compressed = try_compress(dumped, threshold) - MARK_COMPRESSED + compressed - else - MARK_UNCOMPRESSED + dumped + dumped = Marshal.dump(entry.pack) + + if dumped.bytesize >= threshold + compressed = Zlib::Deflate.deflate(dumped) + return MARK_COMPRESSED + compressed if compressed.bytesize < dumped.bytesize end + + MARK_UNCOMPRESSED + dumped end def _load(marked) dumped = marked.byteslice(1..-1) - dumped = decompress(dumped) if marked.start_with?(MARK_COMPRESSED) - try_load_bare_string(dumped) || Cache::Entry.unpack(marshal_load(dumped)) + dumped = Zlib::Inflate.inflate(dumped) if marked.start_with?(MARK_COMPRESSED) + Cache::Entry.unpack(marshal_load(dumped)) end def dumped?(dumped) @@ -197,12 +118,22 @@ module ActiveSupport end end - module Marshal70WithFallback - include Marshal71WithFallback + module Marshal71WithFallback + include SerializerWithFallback extend self - def try_dump_bare_string(_entry) - nil # Prevent dumping bare strings. + MARSHAL_SIGNATURE = "\x04\x08".b.freeze + + def dump(value) + Marshal.dump(value) + end + + def _load(dumped) + marshal_load(dumped) + end + + def dumped?(dumped) + dumped.start_with?(MARSHAL_SIGNATURE) end end @@ -210,13 +141,12 @@ module ActiveSupport include SerializerWithFallback extend self - def _dump(entry) - ActiveSupport::MessagePack::CacheSerializer.dump(entry.pack) + def dump(value) + ActiveSupport::MessagePack::CacheSerializer.dump(value) end def _load(dumped) - packed = ActiveSupport::MessagePack::CacheSerializer.load(dumped) - Cache::Entry.unpack(packed) if packed + ActiveSupport::MessagePack::CacheSerializer.load(dumped) end def dumped?(dumped) diff --git a/activesupport/test/cache/behaviors.rb b/activesupport/test/cache/behaviors.rb index d4f68be4d0c..12bb8adc66d 100644 --- a/activesupport/test/cache/behaviors.rb +++ b/activesupport/test/cache/behaviors.rb @@ -9,6 +9,7 @@ require_relative "behaviors/cache_store_version_behavior" require_relative "behaviors/cache_store_coder_behavior" require_relative "behaviors/cache_store_compression_behavior" require_relative "behaviors/cache_store_format_version_behavior" +require_relative "behaviors/cache_store_serializer_behavior" require_relative "behaviors/connection_pool_behavior" require_relative "behaviors/encoded_key_cache_behavior" require_relative "behaviors/failure_safety_behavior" diff --git a/activesupport/test/cache/behaviors/cache_store_compression_behavior.rb b/activesupport/test/cache/behaviors/cache_store_compression_behavior.rb index a20737e7d6b..02df1ac6c95 100644 --- a/activesupport/test/cache/behaviors/cache_store_compression_behavior.rb +++ b/activesupport/test/cache/behaviors/cache_store_compression_behavior.rb @@ -17,7 +17,7 @@ module CacheStoreCompressionBehavior assert_compression true end - test "compression works with cache format version 7.1 (using Marshal71WithFallback)" do + test "compression works with cache format version >= 7.1 (using Cache::Coder)" do @cache = with_format(7.1) { lookup_store(compress: true) } assert_compression true end @@ -27,6 +27,11 @@ module CacheStoreCompressionBehavior assert_compression false end + test "compression works with custom serializer" do + @cache = with_format(7.1) { lookup_store(compress: true, serializer: Marshal) } + assert_compression true + end + test "compression by default" do @cache = lookup_store assert_compression !compression_always_disabled_by_default? @@ -72,6 +77,47 @@ module CacheStoreCompressionBehavior assert_not_compress "", with: { compress: true, compress_threshold: 1 } assert_not_compress [*0..127].pack("C*"), with: { compress: true, compress_threshold: 1 } end + + test "compressor can be specified" do + lossy_compressor = Module.new do + def self.deflate(dumped) + "yolo" + end + + def self.inflate(compressed) + Marshal.dump("lossy!") if compressed == "yolo" + end + end + + @cache = with_format(7.1) do + lookup_store(compress: true, compressor: lossy_compressor, serializer: Marshal) + end + key = SecureRandom.uuid + + @cache.write(key, LARGE_OBJECT) + assert_equal "lossy!", @cache.read(key) + end + + test "compressor can be nil" do + @cache = with_format(7.1) { lookup_store(compressor: nil) } + assert_compression false + end + + test "specifying a compressor raises when cache format version < 7.1" do + with_format(7.0) do + assert_raises ArgumentError, match: /compressor/i do + lookup_store(compressor: Zlib) + end + end + end + + test "specifying a compressor raises when also specifying a coder" do + with_format(7.1) do + assert_raises ArgumentError, match: /compressor/i do + lookup_store(compressor: Zlib, coder: Marshal) + end + end + end end private diff --git a/activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb b/activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb index 580f89e6387..6fd75ab088b 100644 --- a/activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb +++ b/activesupport/test/cache/behaviors/cache_store_format_version_behavior.rb @@ -15,8 +15,8 @@ module CacheStoreFormatVersionBehavior "\x01\x78".b, # "\x01" + Zlib::Deflate.deflate(...) ], 7.1 => [ - "\x00\x04\x08[".b, # "\x00" + Marshal.dump(entry.pack) - "\x01\x78".b, # "\x01" + Zlib::Deflate.deflate(...) + "\x00\x11\x01".b, # ActiveSupport::Cache::Coder#dump + "\x00\x11\x81".b, # ActiveSupport::Cache::Coder#dump_compressed ], } diff --git a/activesupport/test/cache/behaviors/cache_store_serializer_behavior.rb b/activesupport/test/cache/behaviors/cache_store_serializer_behavior.rb new file mode 100644 index 00000000000..447ef7de28c --- /dev/null +++ b/activesupport/test/cache/behaviors/cache_store_serializer_behavior.rb @@ -0,0 +1,54 @@ +# frozen_string_literal: true + +require "active_support/core_ext/object/with" + +module CacheStoreSerializerBehavior + extend ActiveSupport::Concern + + included do + test "serializer can be specified" do + serializer = Module.new do + def self.dump(value) + value.class.name + end + + def self.load(dumped) + Object.const_get(dumped) + end + end + + @cache = with_format(7.1) { lookup_store(serializer: serializer) } + key = "key#{rand}" + + @cache.write(key, 123) + assert_equal Integer, @cache.read(key) + end + + test "serializer can be :message_pack" do + @cache = with_format(7.1) { lookup_store(serializer: :message_pack) } + key = "key#{rand}" + + @cache.write(key, 123) + assert_equal 123, @cache.read(key) + + assert_raises ActiveSupport::MessagePack::UnserializableObjectError do + @cache.write(key, Object.new) + end + end + + test "specifying a serializer raises when also specifying a coder" do + with_format(7.1) do + assert_raises ArgumentError, match: /serializer/i do + lookup_store(serializer: Marshal, coder: Marshal) + end + end + end + end + + private + def with_format(format_version, &block) + ActiveSupport.deprecator.silence do + ActiveSupport::Cache.with(format_version: format_version, &block) + end + end +end diff --git a/activesupport/test/cache/cache_coder_test.rb b/activesupport/test/cache/cache_coder_test.rb new file mode 100644 index 00000000000..6a6dcb9c1f9 --- /dev/null +++ b/activesupport/test/cache/cache_coder_test.rb @@ -0,0 +1,126 @@ +# frozen_string_literal: true + +require_relative "../abstract_unit" +require "active_support/core_ext/integer/time" + +class CacheCoderTest < ActiveSupport::TestCase + setup do + @coder = ActiveSupport::Cache::Coder.new(Serializer, Compressor) + end + + test "roundtrips entry" do + ENTRIES.each do |entry| + assert_entry entry, @coder.load(@coder.dump(entry)) + end + end + + test "roundtrips entry when using compression" do + ENTRIES.each do |entry| + assert_entry entry, @coder.load(@coder.dump_compressed(entry, 1)) + end + end + + test "compresses values that are larger than the threshold" do + COMPRESSIBLE_ENTRIES.each do |entry| + dumped = @coder.dump(entry) + compressed = @coder.dump_compressed(entry, 1) + + assert_operator compressed.bytesize, :<, dumped.bytesize + end + end + + test "does not compress values that are smaller than the threshold" do + COMPRESSIBLE_ENTRIES.each do |entry| + dumped = @coder.dump(entry) + not_compressed = @coder.dump_compressed(entry, 1_000_000) + + assert_equal dumped, not_compressed + end + end + + test "does not apply compression to incompressible values" do + (ENTRIES - COMPRESSIBLE_ENTRIES).each do |entry| + dumped = @coder.dump(entry) + not_compressed = @coder.dump_compressed(entry, 1) + + assert_equal dumped, not_compressed + end + end + + test "loads dumped entries from original serializer" do + ENTRIES.each do |entry| + assert_entry entry, @coder.load(Serializer.dump(entry)) + end + end + + test "matches output of original serializer when legacy_serializer: true" do + @coder = ActiveSupport::Cache::Coder.new(Serializer, Compressor, legacy_serializer: true) + + ENTRIES.each do |entry| + assert_equal Serializer.dump(entry), @coder.dump(entry) + assert_equal Serializer.dump_compressed(entry, 1), @coder.dump_compressed(entry, 1) + end + end + + test "dumps bare strings with reduced overhead when possible" do + unoptimized = @coder.dump(ActiveSupport::Cache::Entry.new("".encode(Encoding::WINDOWS_1252))) + + [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding| + optimized = @coder.dump(ActiveSupport::Cache::Entry.new("".encode(encoding))) + assert_operator optimized.size, :<, unoptimized.size + end + end + + private + module Serializer + extend self + + def dump(entry) + "SERIALIZED:" + Marshal.dump(entry) + end + + def dump_compressed(*) + "via Serializer#dump_compressed" + end + + def load(dumped) + Marshal.load(dumped.delete_prefix!("SERIALIZED:")) + end + end + + module Compressor + extend self + + def deflate(string) + "COMPRESSED:" + Zlib.deflate(string) + end + + def inflate(deflated) + Zlib.inflate(deflated.delete_prefix!("COMPRESSED:")) + end + end + + STRING = "x" * 100 + COMPRESSIBLE_VALUES = [ + { string: STRING }, + STRING, + STRING.encode(Encoding::BINARY), + STRING.encode(Encoding::US_ASCII), + STRING.encode(Encoding::WINDOWS_1252), + ] + VALUES = [nil, true, 1, "", "ümlaut", [*0..255].pack("C*"), *COMPRESSIBLE_VALUES] + VERSIONS = [nil, "", "ümlaut", [*0..255].pack("C*"), "x" * 256] + EXPIRIES = [nil, 0, 100.years] + + ENTRIES = VALUES.product(VERSIONS, EXPIRIES).map do |value, version, expires_in| + ActiveSupport::Cache::Entry.new(value, version: version, expires_in: expires_in).freeze + end + + COMPRESSIBLE_ENTRIES = ENTRIES.select { |entry| COMPRESSIBLE_VALUES.include?(entry.value) } + + def assert_entry(expected, actual) + assert_equal \ + [expected.value, expected.version, expected.expires_at], + [actual.value, actual.version, actual.expires_at] + end +end diff --git a/activesupport/test/cache/serializer_with_fallback_test.rb b/activesupport/test/cache/serializer_with_fallback_test.rb index f7e027eb05d..acef0ca8f1b 100644 --- a/activesupport/test/cache/serializer_with_fallback_test.rb +++ b/activesupport/test/cache/serializer_with_fallback_test.rb @@ -5,6 +5,7 @@ require "active_support/core_ext/object/with" class CacheSerializerWithFallbackTest < ActiveSupport::TestCase FORMATS = ActiveSupport::Cache::SerializerWithFallback::SERIALIZERS.keys + LEGACY_FORMATS = [:passthrough, :marshal_6_1, :marshal_7_0] setup do @entry = ActiveSupport::Cache::Entry.new( @@ -12,28 +13,21 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase ) end - FORMATS.product(FORMATS) do |load_format, dump_format| - test "#{load_format.inspect} serializer can load #{dump_format.inspect} dump" do - dumped = serializer(dump_format).dump(@entry) - assert_entry @entry, serializer(load_format).load(dumped) + FORMATS.product(FORMATS - LEGACY_FORMATS) do |loader, dumper| + test "#{loader.inspect} serializer can load #{dumper.inspect} dump" do + dumped = serializer(dumper).dump(@entry.value) + assert_equal @entry.value, serializer(loader).load(dumped) end + end - test "#{load_format.inspect} serializer can load #{dump_format.inspect} dump with compression" do - compressed = serializer(dump_format).dump_compressed(@entry, 1) - assert_entry @entry, serializer(load_format).load(compressed) - - uncompressed = serializer(dump_format).dump_compressed(@entry, 100_000) - assert_entry @entry, serializer(load_format).load(uncompressed) + FORMATS.product(LEGACY_FORMATS) do |loader, dumper| + test "#{loader.inspect} serializer can load #{dumper.inspect} dump" do + dumped = serializer(dumper).dump(@entry) + assert_entry @entry, serializer(loader).load(dumped) end end FORMATS.each do |format| - test "#{format.inspect} serializer can compress entries" do - compressed = serializer(format).dump_compressed(@entry, 1) - uncompressed = serializer(format).dump_compressed(@entry, 100_000) - assert_operator compressed.bytesize, :<, uncompressed.bytesize - end - test "#{format.inspect} serializer handles unrecognized payloads gracefully" do assert_nil serializer(format).load(Object.new) assert_nil serializer(format).load("") @@ -45,71 +39,14 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase end end - (FORMATS - [:passthrough, :marshal_6_1, :marshal_7_0]).each do |format| - test "#{format.inspect} serializer preserves version with bare string" do - entry = ActiveSupport::Cache::Entry.new("abc", version: "123") - assert_entry entry, roundtrip(format, entry) - end + LEGACY_FORMATS.each do |format| + test "#{format.inspect} serializer can compress entries" do + compressed = serializer(format).dump_compressed(@entry, 1) + uncompressed = serializer(format).dump_compressed(@entry, 100_000) - test "#{format.inspect} serializer preserves expiration with bare string" do - entry = ActiveSupport::Cache::Entry.new("abc", expires_in: 123) - assert_entry entry, roundtrip(format, entry) - end - - test "#{format.inspect} serializer preserves encoding of version with bare string" do - [Encoding::UTF_8, Encoding::BINARY].each do |encoding| - version = "123".encode(encoding) - roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new("abc", version: version)) - assert_equal version.encoding, roundtripped.version.encoding - end - end - - test "#{format.inspect} serializer preserves encoding of bare string" do - [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding| - string = "abc".encode(encoding) - roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new(string)) - assert_equal string.encoding, roundtripped.value.encoding - end - end - - test "#{format.inspect} serializer handles non-ASCII-only bare string" do - entry = ActiveSupport::Cache::Entry.new("ümlaut") - assert_entry entry, roundtrip(format, entry) - end - - test "#{format.inspect} serializer handles non-ASCII-only version with bare string" do - entry = ActiveSupport::Cache::Entry.new("abc", version: "ümlaut") - assert_entry entry, roundtrip(format, entry) - end - - test "#{format.inspect} serializer dumps bare string with reduced overhead when possible" do - string = "abc" - options = { version: "123", expires_in: 123 } - - unsupported = string.encode(Encoding::WINDOWS_1252) - unoptimized = serializer(format).dump(ActiveSupport::Cache::Entry.new(unsupported, **options)) - - [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].each do |encoding| - supported = string.encode(encoding) - optimized = serializer(format).dump(ActiveSupport::Cache::Entry.new(supported, **options)) - assert_operator optimized.size, :<, unoptimized.size - end - end - - test "#{format.inspect} serializer can compress bare strings" do - entry = ActiveSupport::Cache::Entry.new("abc" * 100, version: "123", expires_in: 123) - compressed = serializer(format).dump_compressed(entry, 1) - uncompressed = serializer(format).dump_compressed(entry, 100_000) assert_operator compressed.bytesize, :<, uncompressed.bytesize - end - end - - [:passthrough, :marshal_6_1, :marshal_7_0].each do |format| - test "#{format.inspect} serializer dumps bare string in a backward compatible way" do - string = +"abc" - string.instance_variable_set(:@baz, true) - roundtripped = roundtrip(format, ActiveSupport::Cache::Entry.new(string)) - assert roundtripped.value.instance_variable_get(:@baz) + assert_entry @entry, serializer(format).load(compressed) + assert_entry @entry, serializer(format).load(uncompressed) end end @@ -120,7 +57,7 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase def to_msgpack_ext; ""; end end - dumped = serializer(:message_pack).dump(ActiveSupport::Cache::Entry.new(klass.new)) + dumped = serializer(:message_pack).dump(klass.new) assert_not_nil dumped assert_nil serializer(:message_pack).load(dumped) end @@ -136,10 +73,6 @@ class CacheSerializerWithFallbackTest < ActiveSupport::TestCase ActiveSupport::Cache::SerializerWithFallback[format] end - def roundtrip(format, entry) - serializer(format).load(serializer(format).dump(entry)) - end - def assert_entry(expected, actual) assert_equal \ [expected.value, expected.version, expected.expires_at], diff --git a/activesupport/test/cache/stores/file_store_test.rb b/activesupport/test/cache/stores/file_store_test.rb index 7a1217a67eb..68130e8d3d2 100644 --- a/activesupport/test/cache/stores/file_store_test.rb +++ b/activesupport/test/cache/stores/file_store_test.rb @@ -33,6 +33,7 @@ class FileStoreTest < ActiveSupport::TestCase include CacheStoreVersionBehavior include CacheStoreCoderBehavior include CacheStoreCompressionBehavior + include CacheStoreSerializerBehavior include CacheStoreFormatVersionBehavior include CacheDeleteMatchedBehavior include CacheIncrementDecrementBehavior diff --git a/activesupport/test/cache/stores/mem_cache_store_test.rb b/activesupport/test/cache/stores/mem_cache_store_test.rb index 40b0dd6c8d8..3789a0e2a8e 100644 --- a/activesupport/test/cache/stores/mem_cache_store_test.rb +++ b/activesupport/test/cache/stores/mem_cache_store_test.rb @@ -78,6 +78,7 @@ class MemCacheStoreTest < ActiveSupport::TestCase include CacheStoreVersionBehavior include CacheStoreCoderBehavior include CacheStoreCompressionBehavior + include CacheStoreSerializerBehavior include CacheStoreFormatVersionBehavior include LocalCacheBehavior include CacheIncrementDecrementBehavior @@ -271,7 +272,7 @@ class MemCacheStoreTest < ActiveSupport::TestCase compressed = Zlib::Deflate.deflate(val) assert_called( - Zlib::Deflate, + Zlib, :deflate, "Memcached writes should not perform duplicate compression.", times: 1, diff --git a/activesupport/test/cache/stores/memory_store_test.rb b/activesupport/test/cache/stores/memory_store_test.rb index e2f530fbea4..2941e4c4d17 100644 --- a/activesupport/test/cache/stores/memory_store_test.rb +++ b/activesupport/test/cache/stores/memory_store_test.rb @@ -17,6 +17,7 @@ class MemoryStoreTest < ActiveSupport::TestCase include CacheStoreVersionBehavior include CacheStoreCoderBehavior include CacheStoreCompressionBehavior + include CacheStoreSerializerBehavior include CacheDeleteMatchedBehavior include CacheIncrementDecrementBehavior include CacheInstrumentationBehavior diff --git a/activesupport/test/cache/stores/redis_cache_store_test.rb b/activesupport/test/cache/stores/redis_cache_store_test.rb index da938c34c45..b566eac2089 100644 --- a/activesupport/test/cache/stores/redis_cache_store_test.rb +++ b/activesupport/test/cache/stores/redis_cache_store_test.rb @@ -161,6 +161,7 @@ module ActiveSupport::Cache::RedisCacheStoreTests include CacheStoreCoderBehavior include CacheStoreCompressionBehavior include CacheStoreFormatVersionBehavior + include CacheStoreSerializerBehavior include LocalCacheBehavior include CacheIncrementDecrementBehavior include CacheInstrumentationBehavior diff --git a/guides/source/configuring.md b/guides/source/configuring.md index c7299ee56a8..107e62c4a72 100644 --- a/guides/source/configuring.md +++ b/guides/source/configuring.md @@ -2411,9 +2411,10 @@ The default value depends on the `config.load_defaults` target version: Specifies which serialization format to use for the cache. Possible values are `6.1`, `7.0`, and `7.1`. -The `6.1`, `7.0`, and `7.1` formats all use `Marshal` for the default coder, but -`7.0` uses a more efficient representation for cache entries, and `7.1` includes -an additional optimization for bare string values such as view fragments. +`7.0` serializes cache entries more efficiently. + +`7.1` further improves efficiency, and includes an optimization for bare string +values such as view fragments. All formats are backward and forward compatible, meaning cache entries written in one format can be read when using another format. This behavior makes it diff --git a/railties/test/application/configuration_test.rb b/railties/test/application/configuration_test.rb index c252145bd7c..eeb48c2d236 100644 --- a/railties/test/application/configuration_test.rb +++ b/railties/test/application/configuration_test.rb @@ -4242,9 +4242,10 @@ module ApplicationTests app "development" + assert_not_nil Rails.cache.instance_variable_get(:@coder) assert_equal \ - ActiveSupport::Cache::NullStore.new.instance_variable_get(:@coder), - Rails.cache.instance_variable_get(:@coder) + Marshal.dump(ActiveSupport::Cache::NullStore.new.instance_variable_get(:@coder)), + Marshal.dump(Rails.cache.instance_variable_get(:@coder)) end test "ActiveSupport::Cache.format_version 6.1 is deprecated" do diff --git a/railties/test/application/initializers/frameworks_test.rb b/railties/test/application/initializers/frameworks_test.rb index 80729b67614..1fcdca6127b 100644 --- a/railties/test/application/initializers/frameworks_test.rb +++ b/railties/test/application/initializers/frameworks_test.rb @@ -404,7 +404,7 @@ module ApplicationTests rails %w(db:migrate) add_to_config <<~RUBY - config.cache_store = :file_store, #{app_path("tmp/cache").inspect}, { coder: :message_pack } + config.cache_store = :file_store, #{app_path("tmp/cache").inspect}, { serializer: :message_pack } RUBY require "#{app_path}/config/environment"