maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity

Reland: [Remarks] Add an LLVM-bitstream-based remark serializer 

Add a new serializer, using a binary format based on the LLVM bitstream
format.

This format provides a way to serialize the remarks in two modes:

1) Separate mode: the metadata is separate from the remark entries.
2) Standalone mode: the metadata and the remark entries are in the same
file.

The format contains:

* a meta block: container version, container type, string table,
external file path, remark version
* a remark block: type, remark name, pass name, function name, debug
file, debug line, debug column, hotness, arguments (key, value, debug
file, debug line, debug column)

A string table is required for this format, which will be dumped in the
meta block to be consumed before parsing the remark blocks.

On clang itself, we noticed a size reduction of 13.4x compared to YAML,
and a compile-time reduction of between 1.7% and 3.5% on CTMark.

Differential Revision: https://reviews.llvm.org/D63466

Original llvm-svn: 367364
Revert llvm-svn: 367370

llvm-svn: 367372
This commit is contained in:
Francis Visoiu Mistrih 2019-07-31 00:13:51 +00:00
parent 50044763f6
commit 84e80979b5
20 changed files with 1362 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

245
llvm/docs/Remarks.rst
View File

@ -113,6 +113,7 @@ following options:
* :ref:`yaml <yamlremarks>` (default)
* :ref:`yaml-strtab <yamlstrtabremarks>`
* :ref:`bitstream <bitstreamremarks>`
``Content configuration``
@ -260,6 +261,250 @@ should be present and point to the file where the remarks are serialized to.
In case the metadata only acts as a header to the remarks, the file path can be
omitted.
.. _bitstreamremarks:
LLVM bitstream remarks
======================
This format is using :doc:`LLVM bitstream <BitCodeFormat>` to serialize remarks
and their associated metadata.
A bitstream remark stream can be identified by the magic number ``"RMRK"`` that
is placed at the very beginning.
The format for serializing remarks is composed of two different block types:
.. _bitstreamremarksmetablock:
META_BLOCK
----------
The block providing information about the rest of the content in the stream.
Exactly one block is expected. Having multiple metadata blocks is an error.
This block can contain the following records:
.. _bitstreamremarksrecordmetacontainerinfo:
``RECORD_META_CONTAINER_INFO``
The container version and type.
Version: u32
Type: u2
.. _bitstreamremarksrecordmetaremarkversion:
``RECORD_META_REMARK_VERSION``
The version of the remark entries. This can change independently from the
container version.
Version: u32
.. _bitstreamremarksrecordmetastrtab:
``RECORD_META_STRTAB``
The string table used by the remark entries. The format of the string table
is a sequence of strings separated by ``\0``.
.. _bitstreamremarksrecordmetaexternalfile:
``RECORD_META_EXTERNAL_FILE``
The external remark file path that contains the remark blocks associated
with this metadata. This is an absolute path.
.. _bitstreamremarksremarkblock:
REMARK_BLOCK
------------
The block describing a remark entry.
0 or more blocks per file are allowed. Each block will depend on the
:ref:`META_BLOCK <bitstreamremarksmetablock>` in order to be parsed correctly.
This block can contain the following records:
``RECORD_REMARK_HEADER``
The header of the remark. This contains all the mandatory information about
a remark.
+---------------+---------------------------+
| Type | u3 |
+---------------+---------------------------+
| Remark name | VBR6 (string table index) |
+---------------+---------------------------+
| Pass name | VBR6 (string table index) |
+---------------+---------------------------+
| Function name | VBR6 (string table index) |
+---------------+---------------------------+
``RECORD_REMARK_DEBUG_LOC``
The source location for the corresponding remark. This record is optional.
+--------+---------------------------+
| File | VBR7 (string table index) |
+--------+---------------------------+
| Line | u32 |
+--------+---------------------------+
| Column | u32 |
+--------+---------------------------+
``RECORD_REMARK_HOTNESS``
The hotness of the remark. This record is optional.
+---------------+---------------------+
| Hotness | VBR8 (string table index) |
+---------------+---------------------+
``RECORD_REMARK_ARG_WITH_DEBUGLOC``
A remark argument with an associated debug location.
+--------+---------------------------+
| Key | VBR7 (string table index) |
+--------+---------------------------+
| Value | VBR7 (string table index) |
+--------+---------------------------+
| File | VBR7 (string table index) |
+--------+---------------------------+
| Line | u32 |
+--------+---------------------------+
| Column | u32 |
+--------+---------------------------+
``RECORD_REMARK_ARG_WITHOUT_DEBUGLOC``
A remark argument with an associated debug location.
+--------+---------------------------+
| Key | VBR7 (string table index) |
+--------+---------------------------+
| Value | VBR7 (string table index) |
+--------+---------------------------+
The remark container
--------------------
Bitstream remarks are designed to be used in two different modes:
``The separate mode``
The separate mode is the mode that is typically used during compilation. It
provides a way to serialize the remark entries to a stream while some
metadata is kept in memory to be emitted in the product of the compilation
(typically, an object file).
``The standalone mode``
The standalone mode is typically stored and used after the distribution of
a program. It contains all the information that allows the parsing of all
the remarks without having any external dependencies.
In order to support multiple modes, the format introduces the concept of a
bitstream remark container type.
.. _bitstreamremarksseparateremarksmeta:
``SeparateRemarksMeta: the metadata emitted separately``
This container type expects only a :ref:`META_BLOCK <bitstreamremarksmetablock>` containing only:
* :ref:`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
* :ref:`RECORD_META_STRTAB <bitstreamremarksrecordmetastrtab>`
* :ref:`RECORD_META_EXTERNAL_FILE <bitstreamremarksrecordmetaexternalfile>`
Typically, this is emitted in a section in the object files, allowing
clients to retrieve remarks and their associated metadata directly from
intermediate products.
``SeparateRemarksFile: the remark entries emitted separately``
This container type expects only a :ref:`META_BLOCK <bitstreamremarksmetablock>` containing only:
* :ref:`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
* :ref:`RECORD_META_REMARK_VERSION <bitstreamremarksrecordmetaremarkversion>`
This container type expects 0 or more :ref:`REMARK_BLOCK <bitstreamremarksremarkblock>`.
Typically, this is emitted in a side-file alongside an object file, and is
made to be able to stream to without increasing the memory consumption of
the compiler. This is referenced by the :ref:`RECORD_META_EXTERNAL_FILE
<bitstreamremarksrecordmetaexternalfile>` entry in the
:ref:`SeparateRemarksMeta <bitstreamremarksseparateremarksmeta>` container.
When the parser tries to parse a container that contains the metadata for the
separate remarks, it should parse the version and type, then keep the string
table in memory while opening the external file, validating its metadata and
parsing the remark entries.
The container versions from the separate container should match in order to
have a well-formed file.
``Standalone: the metadata and the remark entries emitted together``
This container type expects only a :ref:`META_BLOCK <bitstreamremarksmetablock>` containing only:
* :ref:`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
* :ref:`RECORD_META_REMARK_VERSION <bitstreamremarksrecordmetaremarkversion>`
* :ref:`RECORD_META_STRTAB <bitstreamremarksrecordmetastrtab>`
This container type expects 0 or more :ref:`REMARK_BLOCK <bitstreamremarksremarkblock>`.
A complete output of :program:`llvm-bcanalyzer` on the different container types:
``SeparateRemarksMeta``
.. code-block:: none
<BLOCKINFO_BLOCK/>
<Meta BlockID=8 NumWords=13 BlockCodeSize=3>
<Container info codeid=1 abbrevid=4 op0=5 op1=0/>
<String table codeid=3 abbrevid=5/> blob data = 'pass\\x00key\\x00value\\x00'
<External File codeid=4 abbrevid=6/> blob data = '/path/to/file/name'
</Meta>
``SeparateRemarksFile``
.. code-block:: none
<BLOCKINFO_BLOCK/>
<Meta BlockID=8 NumWords=3 BlockCodeSize=3>
<Container info codeid=1 abbrevid=4 op0=0 op1=1/>
<Remark version codeid=2 abbrevid=5 op0=0/>
</Meta>
<Remark BlockID=9 NumWords=8 BlockCodeSize=4>
<Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>
<Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>
<Remark hotness codeid=7 abbrevid=6 op0=999999999/>
<Argument with debug location codeid=8 abbrevid=7 op0=4 op1=5 op2=6 op3=11 op4=66/>
</Remark>
``Standalone``
.. code-block:: none
<BLOCKINFO_BLOCK/>
<Meta BlockID=8 NumWords=15 BlockCodeSize=3>
<Container info codeid=1 abbrevid=4 op0=5 op1=2/>
<Remark version codeid=2 abbrevid=5 op0=30/>
<String table codeid=3 abbrevid=6/> blob data = 'pass\\x00remark\\x00function\\x00path\\x00key\\x00value\\x00argpath\\x00'
</Meta>
<Remark BlockID=9 NumWords=8 BlockCodeSize=4>
<Remark header codeid=5 abbrevid=4 op0=2 op1=1 op2=0 op3=2/>
<Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>
<Remark hotness codeid=7 abbrevid=6 op0=999999999/>
<Argument with debug location codeid=8 abbrevid=7 op0=4 op1=5 op2=6 op3=11 op4=66/>
</Remark>
opt-viewer
==========

1
llvm/include/llvm/Bitcode/BitcodeAnalyzer.h
View File

@ -30,6 +30,7 @@ enum CurStreamTypeType {
LLVMIRBitstream,
ClangSerializedASTBitstream,
ClangSerializedDiagnosticsBitstream,
LLVMBitstreamRemarks
};
struct BCDumpOptions {

106
llvm/include/llvm/Remarks/BitstreamRemarkContainer.h Normal file
View File

@ -0,0 +1,106 @@
//===-- BitstreamRemarkContainer.h - Container for remarks --------------*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file provides declarations for things used in the various types of
// remark containers.
//
//===----------------------------------------------------------------------===//
#ifndef LLVM_REMARKS_REMARK_CONTAINER_H
#define LLVM_REMARKS_REMARK_CONTAINER_H
#include "llvm/ADT/StringRef.h"
#include "llvm/Bitstream/BitCodes.h"
#include <cstdint>
namespace llvm {
namespace remarks {
/// The current version of the remark container.
/// Note: this is different from the version of the remark entry.
constexpr uint64_t CurrentContainerVersion = 0;
/// The magic number used for identifying remark blocks.
constexpr StringRef ContainerMagic("RMRK", 4);
/// Type of the remark container.
/// The remark container has two modes:
/// * separate: the metadata is separate from the remarks and points to the
/// auxiliary file that contains the remarks.
/// * standalone: the metadata and the remarks are emitted together.
enum class BitstreamRemarkContainerType {
/// The metadata emitted separately.
/// This will contain the following:
/// * Container version and type
/// * String table
/// * External file
SeparateRemarksMeta,
/// The remarks emitted separately.
/// This will contain the following:
/// * Container version and type
/// * Remark version
SeparateRemarksFile,
/// Everything is emitted together.
/// This will contain the following:
/// * Container version and type
/// * Remark version
/// * String table
Standalone,
First = SeparateRemarksMeta,
Last = Standalone,
};
/// The possible blocks that will be encountered in a bitstream remark
/// container.
enum BlockIDs {
/// The metadata block is mandatory. It should always come after the
/// BLOCKINFO_BLOCK, and contains metadata that should be used when parsing
/// REMARK_BLOCKs.
/// There should always be only one META_BLOCK.
META_BLOCK_ID = bitc::FIRST_APPLICATION_BLOCKID,
/// One remark entry is represented using a REMARK_BLOCK. There can be
/// multiple REMARK_BLOCKs in the same file.
REMARK_BLOCK_ID
};
constexpr StringRef MetaBlockName = StringRef("Meta", 4);
constexpr StringRef RemarkBlockName = StringRef("Remark", 6);
/// The possible records that can be encountered in the previously described
/// blocks.
enum RecordIDs {
// Meta block records.
RECORD_META_CONTAINER_INFO = 1,
RECORD_META_REMARK_VERSION,
RECORD_META_STRTAB,
RECORD_META_EXTERNAL_FILE,
// Remark block records.
RECORD_REMARK_HEADER,
RECORD_REMARK_DEBUG_LOC,
RECORD_REMARK_HOTNESS,
RECORD_REMARK_ARG_WITH_DEBUGLOC,
RECORD_REMARK_ARG_WITHOUT_DEBUGLOC,
// Helpers.
RECORD_FIRST = RECORD_META_CONTAINER_INFO,
RECORD_LAST = RECORD_REMARK_ARG_WITHOUT_DEBUGLOC
};
constexpr StringRef MetaContainerInfoName = StringRef("Container info", 14);
constexpr StringRef MetaRemarkVersionName = StringRef("Remark version", 14);
constexpr StringRef MetaStrTabName = StringRef("String table", 12);
constexpr StringRef MetaExternalFileName = StringRef("External File", 13);
constexpr StringRef RemarkHeaderName = StringRef("Remark header", 13);
constexpr StringRef RemarkDebugLocName = StringRef("Remark debug location", 21);
constexpr StringRef RemarkHotnessName = StringRef("Remark hotness", 14);
constexpr StringRef RemarkArgWithDebugLocName =
StringRef("Argument with debug location", 28);
constexpr StringRef RemarkArgWithoutDebugLocName = StringRef("Argument", 8);
} // end namespace remarks
} // end namespace llvm
#endif /* LLVM_REMARKS_REMARK_CONTAINER_H */

192
llvm/include/llvm/Remarks/BitstreamRemarkSerializer.h Normal file
View File

@ -0,0 +1,192 @@
//===-- BitstreamRemarkSerializer.h - Bitstream serializer ------*- C++ -*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file provides an implementation of the serializer using the LLVM
// Bitstream format.
//
//===----------------------------------------------------------------------===//
#ifndef LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H
#define LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H
#include "llvm/Bitstream/BitstreamWriter.h"
#include "llvm/Remarks/BitstreamRemarkContainer.h"
#include "llvm/Remarks/RemarkSerializer.h"
#include "llvm/Support/raw_ostream.h"
namespace llvm {
namespace remarks {
/// Serialize the remarks to LLVM bitstream.
/// This class provides ways to emit remarks in the LLVM bitstream format and
/// its associated metadata.
///
/// * The separate model:
/// Separate meta: | Container info
/// | String table
/// | External file
///
/// Separate remarks: | Container info
/// | Remark version
/// | Remark0
/// | Remark1
/// | Remark2
/// | ...
///
/// * The standalone model: | Container info
/// | String table
/// | Remark version
/// | Remark0
/// | Remark1
/// | Remark2
/// | ...
///
struct BitstreamRemarkSerializerHelper {
/// Buffer used for encoding the bitstream before writing it to the final
/// stream.
SmallVector<char, 1024> Encoded;
/// Buffer used to construct records and pass to the bitstream writer.
SmallVector<uint64_t, 64> R;
/// The Bitstream writer.
BitstreamWriter Bitstream;
/// The type of the container we are serializing.
BitstreamRemarkContainerType ContainerType;
/// Abbrev IDs initialized in the block info block.
/// Note: depending on the container type, some IDs might be uninitialized.
/// Warning: When adding more abbrev IDs, make sure to update the
/// BlockCodeSize (in the call to EnterSubblock).
uint64_t RecordMetaContainerInfoAbbrevID = 0;
uint64_t RecordMetaRemarkVersionAbbrevID = 0;
uint64_t RecordMetaStrTabAbbrevID = 0;
uint64_t RecordMetaExternalFileAbbrevID = 0;
uint64_t RecordRemarkHeaderAbbrevID = 0;
uint64_t RecordRemarkDebugLocAbbrevID = 0;
uint64_t RecordRemarkHotnessAbbrevID = 0;
uint64_t RecordRemarkArgWithDebugLocAbbrevID = 0;
uint64_t RecordRemarkArgWithoutDebugLocAbbrevID = 0;
BitstreamRemarkSerializerHelper(BitstreamRemarkContainerType ContainerType);
// Disable copy and move: Bitstream points to Encoded, which needs special
// handling during copy/move, but moving the vectors is probably useless
// anyway.
BitstreamRemarkSerializerHelper(const BitstreamRemarkSerializerHelper &) =
delete;
BitstreamRemarkSerializerHelper &
operator=(const BitstreamRemarkSerializerHelper &) = delete;
BitstreamRemarkSerializerHelper(BitstreamRemarkSerializerHelper &&) = delete;
BitstreamRemarkSerializerHelper &
operator=(BitstreamRemarkSerializerHelper &&) = delete;
/// Set up the necessary block info entries according to the container type.
void setupBlockInfo();
/// Set up the block info for the metadata block.
void setupMetaBlockInfo();
/// The remark version in the metadata block.
void setupMetaRemarkVersion();
void emitMetaRemarkVersion(uint64_t RemarkVersion);
/// The strtab in the metadata block.
void setupMetaStrTab();
void emitMetaStrTab(const StringTable &StrTab);
/// The external file in the metadata block.
void setupMetaExternalFile();
void emitMetaExternalFile(StringRef Filename);
/// The block info for the remarks block.
void setupRemarkBlockInfo();
/// Emit the metadata for the remarks.
void emitMetaBlock(uint64_t ContainerVersion,
Optional<uint64_t> RemarkVersion,
Optional<const StringTable *> StrTab = None,
Optional<StringRef> Filename = None);
/// Emit a remark block. The string table is required.
void emitRemarkBlock(const Remark &Remark, StringTable &StrTab);
/// Finalize the writing to \p OS.
void flushToStream(raw_ostream &OS);
/// Finalize the writing to a buffer.
/// The contents of the buffer remain valid for the lifetime of the object.
/// Any call to any other function in this class will invalidate the buffer.
StringRef getBuffer();
};
/// Implementation of the remark serializer using LLVM bitstream.
struct BitstreamRemarkSerializer : public RemarkSerializer {
/// The file should contain:
/// 1) The block info block that describes how to read the blocks.
/// 2) The metadata block that contains various information about the remarks
/// in the file.
/// 3) A number of remark blocks.
/// We need to set up 1) and 2) first, so that we can emit 3) after. This flag
/// is used to emit the first two blocks only once.
bool DidSetUp = false;
/// The helper to emit bitstream.
BitstreamRemarkSerializerHelper Helper;
/// Construct a serializer that will create its own string table.
BitstreamRemarkSerializer(raw_ostream &OS, SerializerMode Mode);
/// Construct a serializer with a pre-filled string table.
BitstreamRemarkSerializer(raw_ostream &OS, SerializerMode Mode,
StringTable StrTab);
/// Emit a remark to the stream. This also emits the metadata associated to
/// the remarks based on the SerializerMode specified at construction.
/// This writes the serialized output to the provided stream.
void emit(const Remark &Remark) override;
/// The metadata serializer associated to this remark serializer. Based on the
/// container type of the current serializer, the container type of the
/// metadata serializer will change.
std::unique_ptr<MetaSerializer>
metaSerializer(raw_ostream &OS,
Optional<StringRef> ExternalFilename = None) override;
};
/// Serializer of metadata for bitstream remarks.
struct BitstreamMetaSerializer : public MetaSerializer {
/// This class can be used with [1] a pre-constructed
/// BitstreamRemarkSerializerHelper, or with [2] one that is owned by the meta
/// serializer. In case of [1], we need to be able to store a reference to the
/// object, while in case of [2] we need to store the whole object.
Optional<BitstreamRemarkSerializerHelper> TmpHelper;
/// The actual helper, that can point to \p TmpHelper or to an external helper
/// object.
BitstreamRemarkSerializerHelper *Helper = nullptr;
Optional<const StringTable *> StrTab;
Optional<StringRef> ExternalFilename;
/// Create a new meta serializer based on \p ContainerType.
BitstreamMetaSerializer(raw_ostream &OS,
BitstreamRemarkContainerType ContainerType,
Optional<const StringTable *> StrTab = None,
Optional<StringRef> ExternalFilename = None)
: MetaSerializer(OS), TmpHelper(None), Helper(nullptr), StrTab(StrTab),
ExternalFilename(ExternalFilename) {
TmpHelper.emplace(ContainerType);
Helper = &*TmpHelper;
}
/// Create a new meta serializer based on a previously built \p Helper.
BitstreamMetaSerializer(raw_ostream &OS,
BitstreamRemarkSerializerHelper &Helper,
Optional<const StringTable *> StrTab = None,
Optional<StringRef> ExternalFilename = None)
: MetaSerializer(OS), TmpHelper(None), Helper(&Helper), StrTab(StrTab),
ExternalFilename(ExternalFilename) {}
void emit() override;
};
} // end namespace remarks
} // end namespace llvm
#endif /* LLVM_REMARKS_BITSTREAM_REMARK_SERIALIZER_H */

6
llvm/include/llvm/Remarks/Remark.h
View File

@ -23,7 +23,8 @@
namespace llvm {
namespace remarks {
constexpr uint64_t Version = 0;
/// The current version of the remark entry.
constexpr uint64_t CurrentRemarkVersion = 0;
/// The debug location used to track a remark back to the source file.
struct RemarkLocation {
@ -58,7 +59,8 @@ enum class Type {
AnalysisFPCommute,
AnalysisAliasing,
Failure,
LastTypeValue = Failure
First = Unknown,
Last = Failure
};
/// A remark type used for both emission and parsing.

2
llvm/include/llvm/Remarks/RemarkFormat.h
View File

@ -22,7 +22,7 @@ namespace remarks {
constexpr StringRef Magic("REMARKS", 7);
/// The format used for serializing/deserializing remarks.
enum class Format { Unknown, YAML, YAMLStrTab };
enum class Format { Unknown, YAML, YAMLStrTab, Bitstream };
/// Parse and validate a string for the remark format.
Expected<Format> parseFormat(StringRef FormatStr);

10
llvm/lib/Bitcode/Reader/BitcodeAnalyzer.cpp
View File

@ -434,6 +434,13 @@ static Expected<CurStreamTypeType> ReadSignature(BitstreamCursor &Stream) {
return std::move(Err);
if (Signature[2] == 'A' && Signature[3] == 'G')
return ClangSerializedDiagnosticsBitstream;
} else if (Signature[0] == 'R' && Signature[1] == 'M') {
if (Error Err = tryRead(Signature[2], 8))
return std::move(Err);
if (Error Err = tryRead(Signature[3], 8))
return std::move(Err);
if (Signature[2] == 'R' && Signature[3] == 'K')
return LLVMBitstreamRemarks;
} else {
if (Error Err = tryRead(Signature[2], 4))
return std::move(Err);
@ -627,6 +634,9 @@ void BitcodeAnalyzer::printStats(BCDumpOptions O,
case ClangSerializedDiagnosticsBitstream:
O.OS << "Clang Serialized Diagnostics\n";
break;
case LLVMBitstreamRemarks:
O.OS << "LLVM Remarks\n";
break;
}
O.OS << " # Toplevel Blocks: " << NumTopBlocks << "\n";
O.OS << "\n";

1
llvm/lib/IR/RemarkStreamer.cpp
View File

@ -15,6 +15,7 @@
#include "llvm/IR/DiagnosticInfo.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/GlobalValue.h"
#include "llvm/Remarks/BitstreamRemarkSerializer.h"
#include "llvm/Remarks/RemarkFormat.h"
#include "llvm/Remarks/RemarkSerializer.h"

386
llvm/lib/Remarks/BitstreamRemarkSerializer.cpp Normal file
View File

@ -0,0 +1,386 @@
//===- BitstreamRemarkSerializer.cpp --------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file provides the implementation of the LLVM bitstream remark serializer
// using LLVM's bitstream writer.
//
//===----------------------------------------------------------------------===//
#include "llvm/Remarks/BitstreamRemarkSerializer.h"
using namespace llvm;
using namespace llvm::remarks;
BitstreamRemarkSerializerHelper::BitstreamRemarkSerializerHelper(
BitstreamRemarkContainerType ContainerType)
: Encoded(), R(), Bitstream(Encoded), ContainerType(ContainerType) {}
static void push(SmallVectorImpl<uint64_t> &R, StringRef Str) {
for (const char C : Str)
R.push_back(C);
}
static void setRecordName(unsigned RecordID, BitstreamWriter &Bitstream,
SmallVectorImpl<uint64_t> &R, StringRef Str) {
R.clear();
R.push_back(RecordID);
push(R, Str);
Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_SETRECORDNAME, R);
}
static void initBlock(unsigned BlockID, BitstreamWriter &Bitstream,
SmallVectorImpl<uint64_t> &R, StringRef Str) {
R.clear();
R.push_back(BlockID);
Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_SETBID, R);
R.clear();
push(R, Str);
Bitstream.EmitRecord(bitc::BLOCKINFO_CODE_BLOCKNAME, R);
}
void BitstreamRemarkSerializerHelper::setupMetaBlockInfo() {
// Setup the metadata block.
initBlock(META_BLOCK_ID, Bitstream, R, MetaBlockName);
// The container information.
setRecordName(RECORD_META_CONTAINER_INFO, Bitstream, R,
MetaContainerInfoName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_META_CONTAINER_INFO));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Version.
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 2)); // Type.
RecordMetaContainerInfoAbbrevID =
Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev);
}
void BitstreamRemarkSerializerHelper::setupMetaRemarkVersion() {
setRecordName(RECORD_META_REMARK_VERSION, Bitstream, R,
MetaRemarkVersionName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_META_REMARK_VERSION));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Version.
RecordMetaRemarkVersionAbbrevID =
Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev);
}
void BitstreamRemarkSerializerHelper::emitMetaRemarkVersion(
uint64_t RemarkVersion) {
// The remark version is emitted only if we emit remarks.
R.clear();
R.push_back(RECORD_META_REMARK_VERSION);
R.push_back(RemarkVersion);
Bitstream.EmitRecordWithAbbrev(RecordMetaRemarkVersionAbbrevID, R);
}
void BitstreamRemarkSerializerHelper::setupMetaStrTab() {
setRecordName(RECORD_META_STRTAB, Bitstream, R, MetaStrTabName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_META_STRTAB));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Raw table.
RecordMetaStrTabAbbrevID =
Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev);
}
void BitstreamRemarkSerializerHelper::emitMetaStrTab(
const StringTable &StrTab) {
// The string table is not emitted if we emit remarks separately.
R.clear();
R.push_back(RECORD_META_STRTAB);
// Serialize to a blob.
std::string Buf;
raw_string_ostream OS(Buf);
StrTab.serialize(OS);
StringRef Blob = OS.str();
Bitstream.EmitRecordWithBlob(RecordMetaStrTabAbbrevID, R, Blob);
}
void BitstreamRemarkSerializerHelper::setupMetaExternalFile() {
setRecordName(RECORD_META_EXTERNAL_FILE, Bitstream, R, MetaExternalFileName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_META_EXTERNAL_FILE));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Filename.
RecordMetaExternalFileAbbrevID =
Bitstream.EmitBlockInfoAbbrev(META_BLOCK_ID, Abbrev);
}
void BitstreamRemarkSerializerHelper::emitMetaExternalFile(StringRef Filename) {
// The external file is emitted only if we emit the separate metadata.
R.clear();
R.push_back(RECORD_META_EXTERNAL_FILE);
Bitstream.EmitRecordWithBlob(RecordMetaExternalFileAbbrevID, R, Filename);
}
void BitstreamRemarkSerializerHelper::setupRemarkBlockInfo() {
// Setup the remark block.
initBlock(REMARK_BLOCK_ID, Bitstream, R, RemarkBlockName);
// The header of a remark.
{
setRecordName(RECORD_REMARK_HEADER, Bitstream, R, RemarkHeaderName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_HEADER));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 3)); // Type
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Remark Name
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Pass name
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // Function name
RecordRemarkHeaderAbbrevID =
Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
}
// The location of a remark.
{
setRecordName(RECORD_REMARK_DEBUG_LOC, Bitstream, R, RemarkDebugLocName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_DEBUG_LOC));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // File
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Line
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Column
RecordRemarkDebugLocAbbrevID =
Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
}
// The hotness of a remark.
{
setRecordName(RECORD_REMARK_HOTNESS, Bitstream, R, RemarkHotnessName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_HOTNESS));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // Hotness
RecordRemarkHotnessAbbrevID =
Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
}
// An argument entry with a debug location attached.
{
setRecordName(RECORD_REMARK_ARG_WITH_DEBUGLOC, Bitstream, R,
RemarkArgWithDebugLocName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_ARG_WITH_DEBUGLOC));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Key
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Value
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // File
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Line
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Column
RecordRemarkArgWithDebugLocAbbrevID =
Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
}
// An argument entry with no debug location attached.
{
setRecordName(RECORD_REMARK_ARG_WITHOUT_DEBUGLOC, Bitstream, R,
RemarkArgWithoutDebugLocName);
auto Abbrev = std::make_shared<BitCodeAbbrev>();
Abbrev->Add(BitCodeAbbrevOp(RECORD_REMARK_ARG_WITHOUT_DEBUGLOC));
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Key
Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 7)); // Value
RecordRemarkArgWithoutDebugLocAbbrevID =
Bitstream.EmitBlockInfoAbbrev(REMARK_BLOCK_ID, Abbrev);
}
}
void BitstreamRemarkSerializerHelper::setupBlockInfo() {
// Emit magic number.
for (const char C : ContainerMagic)
Bitstream.Emit(static_cast<unsigned>(C), 8);
Bitstream.EnterBlockInfoBlock();
// Setup the main metadata. Depending on the container type, we'll setup the
// required records next.
setupMetaBlockInfo();
switch (ContainerType) {
case BitstreamRemarkContainerType::SeparateRemarksMeta:
// Needs a string table that the separate remark file is using.
setupMetaStrTab();
// Needs to know where the external remarks file is.
setupMetaExternalFile();
break;
case BitstreamRemarkContainerType::SeparateRemarksFile:
// Contains remarks: emit the version.
setupMetaRemarkVersion();
// Contains remarks: emit the remark abbrevs.
setupRemarkBlockInfo();
break;
case BitstreamRemarkContainerType::Standalone:
// Contains remarks: emit the version.
setupMetaRemarkVersion();
// Needs a string table.
setupMetaStrTab();
// Contains remarks: emit the remark abbrevs.
setupRemarkBlockInfo();
break;
}
Bitstream.ExitBlock();
}
void BitstreamRemarkSerializerHelper::emitMetaBlock(
uint64_t ContainerVersion, Optional<uint64_t> RemarkVersion,
Optional<const StringTable *> StrTab, Optional<StringRef> Filename) {
// Emit the meta block
Bitstream.EnterSubblock(META_BLOCK_ID, 3);
// The container version and type.
R.clear();
R.push_back(RECORD_META_CONTAINER_INFO);
R.push_back(ContainerVersion);
R.push_back(static_cast<uint64_t>(ContainerType));
Bitstream.EmitRecordWithAbbrev(RecordMetaContainerInfoAbbrevID, R);
switch (ContainerType) {
case BitstreamRemarkContainerType::SeparateRemarksMeta:
assert(StrTab != None && *StrTab != nullptr);
emitMetaStrTab(**StrTab);
assert(Filename != None);
emitMetaExternalFile(*Filename);
break;
case BitstreamRemarkContainerType::SeparateRemarksFile:
assert(RemarkVersion != None);
emitMetaRemarkVersion(*RemarkVersion);
break;
case BitstreamRemarkContainerType::Standalone:
assert(RemarkVersion != None);
emitMetaRemarkVersion(*RemarkVersion);
assert(StrTab != None && *StrTab != nullptr);
emitMetaStrTab(**StrTab);
break;
}
Bitstream.ExitBlock();
}
void BitstreamRemarkSerializerHelper::emitRemarkBlock(const Remark &Remark,
StringTable &StrTab) {
Bitstream.EnterSubblock(REMARK_BLOCK_ID, 4);
R.clear();
R.push_back(RECORD_REMARK_HEADER);
R.push_back(static_cast<uint64_t>(Remark.RemarkType));
R.push_back(StrTab.add(Remark.RemarkName).first);
R.push_back(StrTab.add(Remark.PassName).first);
R.push_back(StrTab.add(Remark.FunctionName).first);
Bitstream.EmitRecordWithAbbrev(RecordRemarkHeaderAbbrevID, R);
if (const Optional<RemarkLocation> &Loc = Remark.Loc) {
R.clear();
R.push_back(RECORD_REMARK_DEBUG_LOC);
R.push_back(StrTab.add(Loc->SourceFilePath).first);
R.push_back(Loc->SourceLine);
R.push_back(Loc->SourceColumn);
Bitstream.EmitRecordWithAbbrev(RecordRemarkDebugLocAbbrevID, R);
}
if (Optional<uint64_t> Hotness = Remark.Hotness) {
R.clear();
R.push_back(RECORD_REMARK_HOTNESS);
R.push_back(*Hotness);
Bitstream.EmitRecordWithAbbrev(RecordRemarkHotnessAbbrevID, R);
}
for (const Argument &Arg : Remark.Args) {
R.clear();
unsigned Key = StrTab.add(Arg.Key).first;
unsigned Val = StrTab.add(Arg.Val).first;
bool HasDebugLoc = Arg.Loc != None;
R.push_back(HasDebugLoc ? RECORD_REMARK_ARG_WITH_DEBUGLOC
: RECORD_REMARK_ARG_WITHOUT_DEBUGLOC);
R.push_back(Key);
R.push_back(Val);
if (HasDebugLoc) {
R.push_back(StrTab.add(Arg.Loc->SourceFilePath).first);
R.push_back(Arg.Loc->SourceLine);
R.push_back(Arg.Loc->SourceColumn);
}
Bitstream.EmitRecordWithAbbrev(HasDebugLoc
? RecordRemarkArgWithDebugLocAbbrevID
: RecordRemarkArgWithoutDebugLocAbbrevID,
R);
}
Bitstream.ExitBlock();
}
void BitstreamRemarkSerializerHelper::flushToStream(raw_ostream &OS) {
OS.write(Encoded.data(), Encoded.size());
Encoded.clear();
}
StringRef BitstreamRemarkSerializerHelper::getBuffer() {
return StringRef(Encoded.data(), Encoded.size());
}
BitstreamRemarkSerializer::BitstreamRemarkSerializer(raw_ostream &OS,
SerializerMode Mode)
: RemarkSerializer(OS, Mode),
Helper(BitstreamRemarkContainerType::SeparateRemarksFile) {
assert(Mode == SerializerMode::Separate &&
"For SerializerMode::Standalone, a pre-filled string table needs to "
"be provided.");
// We always use a string table with bitstream.
StrTab.emplace();
}
BitstreamRemarkSerializer::BitstreamRemarkSerializer(raw_ostream &OS,
SerializerMode Mode,
StringTable StrTabIn)
: RemarkSerializer(OS, Mode),
Helper(Mode == SerializerMode::Separate
? BitstreamRemarkContainerType::SeparateRemarksFile
: BitstreamRemarkContainerType::Standalone) {
StrTab = std::move(StrTabIn);
}
void BitstreamRemarkSerializer::emit(const Remark &Remark) {
if (!DidSetUp) {
// Emit the metadata that is embedded in the remark file.
// If we're in standalone mode, serialize the string table as well.
bool IsStandalone =
Helper.ContainerType == BitstreamRemarkContainerType::Standalone;
BitstreamMetaSerializer MetaSerializer(
OS, Helper,
IsStandalone ? &*StrTab : Optional<const StringTable *>(None));
MetaSerializer.emit();
DidSetUp = true;
}
assert(DidSetUp &&
"The Block info block and the meta block were not emitted yet.");
Helper.emitRemarkBlock(Remark, *StrTab);
Helper.flushToStream(OS);
}
std::unique_ptr<MetaSerializer> BitstreamRemarkSerializer::metaSerializer(
raw_ostream &OS, Optional<StringRef> ExternalFilename) {
assert(Helper.ContainerType !=
BitstreamRemarkContainerType::SeparateRemarksMeta);
bool IsStandalone =
Helper.ContainerType == BitstreamRemarkContainerType::Standalone;
return llvm::make_unique<BitstreamMetaSerializer>(
OS,
IsStandalone ? BitstreamRemarkContainerType::Standalone
: BitstreamRemarkContainerType::SeparateRemarksMeta,
&*StrTab, ExternalFilename);
}
void BitstreamMetaSerializer::emit() {
Helper->setupBlockInfo();
Helper->emitMetaBlock(CurrentContainerVersion, CurrentRemarkVersion, StrTab,
ExternalFilename);
Helper->flushToStream(OS);
}

1
llvm/lib/Remarks/CMakeLists.txt
View File

@ -1,4 +1,5 @@
add_llvm_library(LLVMRemarks
BitstreamRemarkSerializer.cpp
Remark.cpp
RemarkFormat.cpp
RemarkParser.cpp

3
llvm/lib/Remarks/RemarkFormat.cpp
View File

@ -19,7 +19,8 @@ using namespace llvm::remarks;
Expected<Format> llvm::remarks::parseFormat(StringRef FormatStr) {
auto Result = StringSwitch<Format>(FormatStr)
.Cases("", "yaml", Format::YAML)
.Cases("", "yaml-strtab", Format::YAMLStrTab)
.Case("yaml-strtab", Format::YAMLStrTab)
.Case("bitstream", Format::Bitstream)
.Default(Format::Unknown);
if (Result == Format::Unknown)

9
llvm/lib/Remarks/RemarkParser.cpp
View File

@ -56,6 +56,9 @@ llvm::remarks::createRemarkParser(Format ParserFormat, StringRef Buf) {
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"The YAML with string table format requires a parsed string table.");
case Format::Bitstream:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Parsing bitstream remarks is not supported.");
case Format::Unknown:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Unknown remark parser format.");
@ -73,6 +76,9 @@ llvm::remarks::createRemarkParser(Format ParserFormat, StringRef Buf,
"table. Use yaml-strtab instead.");
case Format::YAMLStrTab:
return llvm::make_unique<YAMLStrTabRemarkParser>(Buf, std::move(StrTab));
case Format::Bitstream:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Parsing bitstream remarks is not supported.");
case Format::Unknown:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Unknown remark parser format.");
@ -89,6 +95,9 @@ llvm::remarks::createRemarkParserFromMeta(Format ParserFormat, StringRef Buf,
case Format::YAML:
case Format::YAMLStrTab:
return createYAMLParserFromMeta(Buf, std::move(StrTab));
case Format::Bitstream:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Parsing bitstream remarks is not supported.");
case Format::Unknown:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Unknown remark parser format.");

6
llvm/lib/Remarks/RemarkSerializer.cpp
View File

@ -11,6 +11,7 @@
//===----------------------------------------------------------------------===//
#include "llvm/Remarks/RemarkSerializer.h"
#include "llvm/Remarks/BitstreamRemarkSerializer.h"
#include "llvm/Remarks/YAMLRemarkSerializer.h"
using namespace llvm;
@ -27,6 +28,8 @@ remarks::createRemarkSerializer(Format RemarksFormat, SerializerMode Mode,
return llvm::make_unique<YAMLRemarkSerializer>(OS, Mode);
case Format::YAMLStrTab:
return llvm::make_unique<YAMLStrTabRemarkSerializer>(OS, Mode);
case Format::Bitstream:
return llvm::make_unique<BitstreamRemarkSerializer>(OS, Mode);
}
llvm_unreachable("Unknown remarks::Format enum");
}
@ -45,6 +48,9 @@ remarks::createRemarkSerializer(Format RemarksFormat, SerializerMode Mode,
case Format::YAMLStrTab:
return llvm::make_unique<YAMLStrTabRemarkSerializer>(OS, Mode,
std::move(StrTab));
case Format::Bitstream:
return llvm::make_unique<BitstreamRemarkSerializer>(OS, Mode,
std::move(StrTab));
}
llvm_unreachable("Unknown remarks::Format enum");
}

4
llvm/lib/Remarks/YAMLRemarkParser.cpp
View File

@ -75,11 +75,11 @@ static Expected<uint64_t> parseVersion(StringRef &Buf) {
uint64_t Version =
support::endian::read<uint64_t, support::little, support::unaligned>(
Buf.data());
if (Version != remarks::Version)
if (Version != remarks::CurrentRemarkVersion)
return createStringError(std::errc::illegal_byte_sequence,
"Mismatching remark version. Got %" PRId64
", expected %" PRId64 ".",
Version, remarks::Version);
Version, remarks::CurrentRemarkVersion);
Buf = Buf.drop_front(sizeof(uint64_t));
return Version;
}

2
llvm/lib/Remarks/YAMLRemarkSerializer.cpp
View File

@ -191,7 +191,7 @@ static void emitMagic(raw_ostream &OS) {
static void emitVersion(raw_ostream &OS) {
// Emit the version number: little-endian uint64_t.
std::array<char, 8> Version;
support::endian::write64le(Version.data(), remarks::Version);
support::endian::write64le(Version.data(), remarks::CurrentRemarkVersion);
OS.write(Version.data(), Version.size());
}

3
llvm/test/Bitcode/stream-types.c
View File

@ -10,3 +10,6 @@
// RUN: not llvm-bcanalyzer -dump %s.ast.incomplete 2>&1 | FileCheck %s -check-prefix=CHECK-INCOMPLETE
// RUN: not llvm-bcanalyzer -dump %s.dia.incomplete 2>&1 | FileCheck %s -check-prefix=CHECK-INCOMPLETE
// CHECK-INCOMPLETE: Bitcode stream should be a multiple of 4 bytes in length
// RUN: llvm-bcanalyzer -dump %s.opt.bitstream | FileCheck %s -check-prefix=CHECK-REMARKS
// CHECK-REMARKS: Stream type: LLVM Remarks

BIN
llvm/test/Bitcode/stream-types.c.opt.bitstream Normal file
View File

Binary file not shown.

48
llvm/unittests/Remarks/BitstreamRemarksFormatTest.cpp Normal file
View File

@ -0,0 +1,48 @@
//===- unittest/Support/BitstreamRemarksFormatTest.cpp - BitCodes tests ---===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
#include "llvm/Remarks/BitstreamRemarkContainer.h"
#include "gtest/gtest.h"
using namespace llvm;
// The goal for this test is to observe test failures and carefully update the
// constants when they change.
// This should not change over time.
TEST(BitstreamRemarksFormat, Magic) {
EXPECT_EQ(remarks::ContainerMagic, "RMRK");
}
// This should be updated whenever any of the tests below are modified.
TEST(BitstreamRemarksFormat, ContainerVersion) {
EXPECT_EQ(remarks::CurrentContainerVersion, 0UL);
}
// The values of the current blocks should not change over time.
// When adding new blocks, make sure to append them to the enum.
TEST(BitstreamRemarksFormat, BlockIDs) {
EXPECT_EQ(remarks::META_BLOCK_ID, 8);
EXPECT_EQ(remarks::REMARK_BLOCK_ID, 9);
}
// The values of the current records should not change over time.
// When adding new records, make sure to append them to the enum.
TEST(BitstreamRemarksFormat, RecordIDs) {
EXPECT_EQ(remarks::RECORD_FIRST, 1);
EXPECT_EQ(remarks::RECORD_META_CONTAINER_INFO, 1);
EXPECT_EQ(remarks::RECORD_META_REMARK_VERSION, 2);
EXPECT_EQ(remarks::RECORD_META_STRTAB, 3);
EXPECT_EQ(remarks::RECORD_META_EXTERNAL_FILE, 4);
EXPECT_EQ(remarks::RECORD_REMARK_HEADER, 5);
EXPECT_EQ(remarks::RECORD_REMARK_DEBUG_LOC, 6);
EXPECT_EQ(remarks::RECORD_REMARK_HOTNESS, 7);
EXPECT_EQ(remarks::RECORD_REMARK_ARG_WITH_DEBUGLOC, 8);
EXPECT_EQ(remarks::RECORD_REMARK_ARG_WITHOUT_DEBUGLOC, 9);
EXPECT_EQ(remarks::RECORD_LAST, 9);
}

341
llvm/unittests/Remarks/BitstreamRemarksSerializerTest.cpp Normal file
View File

@ -0,0 +1,341 @@
//===- unittest/Support/BitstreamRemarksSerializerTest.cpp ----------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
#include "llvm/Bitcode/BitcodeAnalyzer.h"
#include "llvm/Remarks/BitstreamRemarkSerializer.h"
#include "llvm/Support/raw_ostream.h"
#include "gtest/gtest.h"
#include <string>
// We need to supprt Windows paths as well. In order to have paths with the same
// length, use a different path according to the platform.
#ifdef _WIN32
#define EXTERNALFILETESTPATH "C:/externalfi"
#else
#define EXTERNALFILETESTPATH "/externalfile"
#endif
using namespace llvm;
static void checkAnalyze(StringRef Input, StringRef Expected) {
std::string OutputBuf;
raw_string_ostream OutputOS(OutputBuf);
BCDumpOptions O(OutputOS);
O.ShowBinaryBlobs = true;
BitcodeAnalyzer BA(Input);
EXPECT_FALSE(BA.analyze(O)); // Expect no errors.
EXPECT_EQ(OutputOS.str(), Expected);
}
static void check(remarks::SerializerMode Mode, const remarks::Remark &R,
StringRef ExpectedR, Optional<StringRef> ExpectedMeta,
Optional<remarks::StringTable> StrTab) {
// Emit the remark.
std::string InputBuf;
raw_string_ostream InputOS(InputBuf);
Expected<std::unique_ptr<remarks::RemarkSerializer>> MaybeSerializer = [&] {
if (StrTab)
return createRemarkSerializer(remarks::Format::Bitstream, Mode, InputOS,
std::move(*StrTab));
else
return createRemarkSerializer(remarks::Format::Bitstream, Mode, InputOS);
}();
EXPECT_FALSE(errorToBool(MaybeSerializer.takeError()));
std::unique_ptr<remarks::RemarkSerializer> Serializer =
std::move(*MaybeSerializer);
Serializer->emit(R);
// Analyze the serialized remark.
checkAnalyze(InputOS.str(), ExpectedR);
// Analyze the serialized metadata if it's not in standalone mode.
if (ExpectedMeta) {
std::string MetaBuf;
raw_string_ostream MetaOS(MetaBuf);
std::unique_ptr<remarks::MetaSerializer> MetaSerializer =
Serializer->metaSerializer(MetaOS, StringRef(EXTERNALFILETESTPATH));
MetaSerializer->emit();
checkAnalyze(MetaOS.str(), *ExpectedMeta);
}
}
static void check(const remarks::Remark &R, StringRef ExpectedR,
StringRef ExpectedMeta,
Optional<remarks::StringTable> StrTab = None) {
return check(remarks::SerializerMode::Separate, R, ExpectedR, ExpectedMeta,
std::move(StrTab));
}
static void checkStandalone(const remarks::Remark &R, StringRef ExpectedR,
Optional<remarks::StringTable> StrTab = None) {
return check(remarks::SerializerMode::Standalone, R, ExpectedR,
/*ExpectedMeta=*/None, std::move(StrTab));
}
TEST(BitstreamRemarkSerializer, SeparateRemarkFileNoOptionals) {
remarks::Remark R;
R.RemarkType = remarks::Type::Missed;
R.PassName = "pass";
R.RemarkName = "remark";
R.FunctionName = "function";
check(R,
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
" <Remark version codeid=2 abbrevid=5 op0=0/>\n"
"</Meta>\n"
"<Remark BlockID=9 NumWords=1 BlockCodeSize=4>\n"
" <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
"</Remark>\n",
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=14 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
" <String table codeid=3 abbrevid=5/> blob data = "
"'remark\\x00pass\\x00function\\x00'\n"
" <External File codeid=4 abbrevid=6/> blob data = "
"'" EXTERNALFILETESTPATH"'\n"
"</Meta>\n");
}
TEST(BitstreamRemarkSerializer, SeparateRemarkFileNoOptionalsSeparateStrTab) {
remarks::StringTable StrTab;
StrTab.add("function");
StrTab.add("pass");
StrTab.add("remark");
remarks::Remark R;
R.RemarkType = remarks::Type::Missed;
R.PassName = "pass";
R.RemarkName = "remark";
R.FunctionName = "function";
check(R,
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
" <Remark version codeid=2 abbrevid=5 op0=0/>\n"
"</Meta>\n"
"<Remark BlockID=9 NumWords=1 BlockCodeSize=4>\n"
" <Remark header codeid=5 abbrevid=4 op0=2 op1=2 op2=1 op3=0/>\n"
"</Remark>\n",
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=14 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
" <String table codeid=3 abbrevid=5/> blob data = "
"'function\\x00pass\\x00remark\\x00'\n"
" <External File codeid=4 abbrevid=6/> blob data = "
"'" EXTERNALFILETESTPATH"'\n"
"</Meta>\n",
std::move(StrTab));
}
TEST(BitstreamRemarkSerializer, SeparateRemarkFileDebugLoc) {
remarks::Remark R;
R.RemarkType = remarks::Type::Missed;
R.PassName = "pass";
R.RemarkName = "remark";
R.FunctionName = "function";
R.Loc.emplace();
R.Loc->SourceFilePath = "path";
R.Loc->SourceLine = 99;
R.Loc->SourceColumn = 55;
check(R,
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
" <Remark version codeid=2 abbrevid=5 op0=0/>\n"
"</Meta>\n"
"<Remark BlockID=9 NumWords=4 BlockCodeSize=4>\n"
" <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
" <Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>\n"
"</Remark>\n",
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=15 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
" <String table codeid=3 abbrevid=5/> blob data = "
"'remark\\x00pass\\x00function\\x00path\\x00'\n"
" <External File codeid=4 abbrevid=6/> blob data = "
"'" EXTERNALFILETESTPATH"'\n"
"</Meta>\n");
}
TEST(BitstreamRemarkSerializer, SeparateRemarkFileHotness) {
remarks::Remark R;
R.RemarkType = remarks::Type::Missed;
R.PassName = "pass";
R.RemarkName = "remark";
R.FunctionName = "function";
R.Hotness.emplace(999999999);
check(R,
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
" <Remark version codeid=2 abbrevid=5 op0=0/>\n"
"</Meta>\n"
"<Remark BlockID=9 NumWords=3 BlockCodeSize=4>\n"
" <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
" <Remark hotness codeid=7 abbrevid=6 op0=999999999/>\n"
"</Remark>\n",
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=14 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
" <String table codeid=3 abbrevid=5/> blob data = "
"'remark\\x00pass\\x00function\\x00'\n"
" <External File codeid=4 abbrevid=6/> blob data = "
"'" EXTERNALFILETESTPATH"'\n"
"</Meta>\n");
}
TEST(BitstreamRemarkSerializer, SeparateRemarkFileArgNoDebugLoc) {
remarks::Remark R;
R.RemarkType = remarks::Type::Missed;
R.PassName = "pass";
R.RemarkName = "remark";
R.FunctionName = "function";
R.Args.emplace_back();
R.Args.back().Key = "key";
R.Args.back().Val = "value";
check(R,
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
" <Remark version codeid=2 abbrevid=5 op0=0/>\n"
"</Meta>\n"
"<Remark BlockID=9 NumWords=2 BlockCodeSize=4>\n"
" <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
" <Argument codeid=9 abbrevid=8 op0=3 op1=4/>\n"
"</Remark>\n",
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=16 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
" <String table codeid=3 abbrevid=5/> blob data = "
"'remark\\x00pass\\x00function\\x00key\\x00value\\x00'\n"
" <External File codeid=4 abbrevid=6/> blob data = "
"'" EXTERNALFILETESTPATH"'\n"
"</Meta>\n");
}
TEST(BitstreamRemarkSerializer, SeparateRemarkFileArgDebugLoc) {
remarks::Remark R;
R.RemarkType = remarks::Type::Missed;
R.PassName = "pass";
R.RemarkName = "remark";
R.FunctionName = "function";
R.Args.emplace_back();
R.Args.back().Key = "key";
R.Args.back().Val = "value";
R.Args.back().Loc.emplace();
R.Args.back().Loc->SourceFilePath = "path";
R.Args.back().Loc->SourceLine = 99;
R.Args.back().Loc->SourceColumn = 55;
check(R,
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
" <Remark version codeid=2 abbrevid=5 op0=0/>\n"
"</Meta>\n"
"<Remark BlockID=9 NumWords=4 BlockCodeSize=4>\n"
" <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
" <Argument with debug location codeid=8 abbrevid=7 op0=3 op1=4 op2=5 "
"op3=99 op4=55/>\n"
"</Remark>\n",
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=17 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
" <String table codeid=3 abbrevid=5/> blob data = "
"'remark\\x00pass\\x00function\\x00key\\x00value\\x00path\\x00'\n"
" <External File codeid=4 abbrevid=6/> blob data = "
"'" EXTERNALFILETESTPATH"'\n"
"</Meta>\n");
}
TEST(BitstreamRemarkSerializer, SeparateRemarkFileAll) {
remarks::Remark R;
R.RemarkType = remarks::Type::Missed;
R.PassName = "pass";
R.RemarkName = "remark";
R.FunctionName = "function";
R.Loc.emplace();
R.Loc->SourceFilePath = "path";
R.Loc->SourceLine = 99;
R.Loc->SourceColumn = 55;
R.Hotness.emplace(999999999);
R.Args.emplace_back();
R.Args.back().Key = "key";
R.Args.back().Val = "value";
R.Args.back().Loc.emplace();
R.Args.back().Loc->SourceFilePath = "argpath";
R.Args.back().Loc->SourceLine = 11;
R.Args.back().Loc->SourceColumn = 66;
check(R,
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=3 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=1/>\n"
" <Remark version codeid=2 abbrevid=5 op0=0/>\n"
"</Meta>\n"
"<Remark BlockID=9 NumWords=8 BlockCodeSize=4>\n"
" <Remark header codeid=5 abbrevid=4 op0=2 op1=0 op2=1 op3=2/>\n"
" <Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>\n"
" <Remark hotness codeid=7 abbrevid=6 op0=999999999/>\n"
" <Argument with debug location codeid=8 abbrevid=7 op0=4 op1=5 op2=6 "
"op3=11 op4=66/>\n"
"</Remark>\n",
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=19 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=0/>\n"
" <String table codeid=3 abbrevid=5/> blob data = "
"'remark\\x00pass\\x00function\\x00path\\x00key\\x00value\\x00argpa"
"th\\x00'\n <External File codeid=4 abbrevid=6/> blob data = "
"'" EXTERNALFILETESTPATH"'\n"
"</Meta>\n");
}
TEST(BitstreamRemarkSerializer, Standalone) {
// Pre-populate the string table.
remarks::StringTable StrTab;
StrTab.add("pass");
StrTab.add("remark");
StrTab.add("function");
StrTab.add("path");
StrTab.add("key");
StrTab.add("value");
StrTab.add("argpath");
remarks::Remark R;
R.RemarkType = remarks::Type::Missed;
R.PassName = "pass";
R.RemarkName = "remark";
R.FunctionName = "function";
R.Loc.emplace();
R.Loc->SourceFilePath = "path";
R.Loc->SourceLine = 99;
R.Loc->SourceColumn = 55;
R.Hotness.emplace(999999999);
R.Args.emplace_back();
R.Args.back().Key = "key";
R.Args.back().Val = "value";
R.Args.back().Loc.emplace();
R.Args.back().Loc->SourceFilePath = "argpath";
R.Args.back().Loc->SourceLine = 11;
R.Args.back().Loc->SourceColumn = 66;
checkStandalone(
R,
"<BLOCKINFO_BLOCK/>\n"
"<Meta BlockID=8 NumWords=15 BlockCodeSize=3>\n"
" <Container info codeid=1 abbrevid=4 op0=0 op1=2/>\n"
" <Remark version codeid=2 abbrevid=5 op0=0/>\n"
" <String table codeid=3 abbrevid=6/> blob data = "
"'pass\\x00remark\\x00function\\x00path\\x00key\\x00value\\x00argpath\\x0"
"0'\n"
"</Meta>\n"
"<Remark BlockID=9 NumWords=8 BlockCodeSize=4>\n"
" <Remark header codeid=5 abbrevid=4 op0=2 op1=1 op2=0 op3=2/>\n"
" <Remark debug location codeid=6 abbrevid=5 op0=3 op1=99 op2=55/>\n"
" <Remark hotness codeid=7 abbrevid=6 op0=999999999/>\n"
" <Argument with debug location codeid=8 abbrevid=7 op0=4 op1=5 op2=6 "
"op3=11 op4=66/>\n"
"</Remark>\n",
std::move(StrTab));
}

3
llvm/unittests/Remarks/CMakeLists.txt
View File

@ -1,9 +1,12 @@
set(LLVM_LINK_COMPONENTS
BitReader
Remarks
Support
)
add_llvm_unittest(RemarksTests
BitstreamRemarksFormatTest.cpp
BitstreamRemarksSerializerTest.cpp
RemarksStrTabParsingTest.cpp
YAMLRemarksParsingTest.cpp
YAMLRemarksSerializerTest.cpp