2021-12-22 02:21:41 +08:00
|
|
|
//===- bolt/runtime/hugify.cpp --------------------------------------------===//
|
Adding automatic huge page support
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
2020-05-03 02:14:38 +08:00
|
|
|
//
|
2021-03-18 06:04:19 +08:00
|
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
Adding automatic huge page support
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
2020-05-03 02:14:38 +08:00
|
|
|
//
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
|
2021-10-16 22:35:29 +08:00
|
|
|
#if defined (__x86_64__)
|
2020-10-15 18:51:56 +08:00
|
|
|
#if !defined(__APPLE__)
|
|
|
|
|
Adding automatic huge page support
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
2020-05-03 02:14:38 +08:00
|
|
|
#include "common.h"
|
|
|
|
#include <sys/mman.h>
|
|
|
|
|
|
|
|
// Enables a very verbose logging to stderr useful when debugging
|
|
|
|
//#define ENABLE_DEBUG
|
|
|
|
|
|
|
|
// Function pointers to init routines in the binary, so we can resume
|
|
|
|
// regular execution of the function that we hooked.
|
|
|
|
extern void (*__bolt_hugify_init_ptr)();
|
|
|
|
|
|
|
|
// The __hot_start and __hot_end symbols set by Bolt. We use them to figure
|
|
|
|
// out the rage for marking huge pages.
|
|
|
|
extern uint64_t __hot_start;
|
|
|
|
extern uint64_t __hot_end;
|
|
|
|
|
|
|
|
#ifdef MADV_HUGEPAGE
|
2021-06-03 10:11:52 +08:00
|
|
|
/// Check whether the kernel supports THP via corresponding sysfs entry.
|
2020-07-03 05:28:13 +08:00
|
|
|
static bool has_pagecache_thp_support() {
|
2021-06-03 10:11:52 +08:00
|
|
|
char buf[256] = {0};
|
|
|
|
const char *madviseStr = "always [madvise] never";
|
|
|
|
|
|
|
|
int fd = __open("/sys/kernel/mm/transparent_hugepage/enabled",
|
|
|
|
0 /* O_RDONLY */, 0);
|
|
|
|
if (fd < 0)
|
Adding automatic huge page support
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
2020-05-03 02:14:38 +08:00
|
|
|
return false;
|
|
|
|
|
2021-06-03 10:11:52 +08:00
|
|
|
size_t res = __read(fd, buf, 256);
|
|
|
|
if (res < 0)
|
Adding automatic huge page support
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
2020-05-03 02:14:38 +08:00
|
|
|
return false;
|
2021-06-03 10:11:52 +08:00
|
|
|
|
|
|
|
int cmp = strnCmp(buf, madviseStr, strLen(madviseStr));
|
|
|
|
return cmp == 0;
|
Adding automatic huge page support
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
2020-05-03 02:14:38 +08:00
|
|
|
}
|
|
|
|
|
2020-07-03 05:28:13 +08:00
|
|
|
static void hugify_for_old_kernel(uint8_t *from, uint8_t *to) {
|
Adding automatic huge page support
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
2020-05-03 02:14:38 +08:00
|
|
|
size_t size = to - from;
|
|
|
|
|
|
|
|
uint8_t *mem = reinterpret_cast<uint8_t *>(
|
|
|
|
__mmap(0, size, 0x3 /* PROT_READ | PROT_WRITE*/,
|
|
|
|
0x22 /* MAP_PRIVATE | MAP_ANONYMOUS*/, -1, 0));
|
|
|
|
|
|
|
|
if (mem == (void *)MAP_FAILED) {
|
|
|
|
char msg[] = "Could not allocate memory for text move\n";
|
|
|
|
reportError(msg, sizeof(msg));
|
|
|
|
}
|
|
|
|
#ifdef ENABLE_DEBUG
|
|
|
|
reportNumber("Allocated temporary space: ", (uint64_t)mem, 16);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
// Copy the hot code to a temproary location.
|
|
|
|
memCpy(mem, from, size);
|
|
|
|
|
|
|
|
// Maps out the existing hot code.
|
|
|
|
if (__mmap(reinterpret_cast<uint64_t>(from), size,
|
|
|
|
PROT_READ | PROT_WRITE | PROT_EXEC,
|
|
|
|
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1,
|
|
|
|
0) == (void *)MAP_FAILED) {
|
|
|
|
char msg[] = "failed to mmap memory for large page move terminating\n";
|
|
|
|
reportError(msg, sizeof(msg));
|
|
|
|
}
|
|
|
|
|
|
|
|
// Mark the hot code page to be huge page.
|
|
|
|
if (__madvise(from, size, MADV_HUGEPAGE) == -1) {
|
|
|
|
char msg[] = "failed to allocate large page\n";
|
|
|
|
reportError(msg, sizeof(msg));
|
|
|
|
}
|
|
|
|
|
|
|
|
// Copy the hot code back.
|
|
|
|
memCpy(from, mem, size);
|
|
|
|
|
|
|
|
// Change permission back to read-only, ignore failure
|
|
|
|
__mprotect(from, size, PROT_READ | PROT_EXEC);
|
|
|
|
|
|
|
|
__munmap(mem, size);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
extern "C" void __bolt_hugify_self_impl() {
|
|
|
|
#ifdef MADV_HUGEPAGE
|
|
|
|
uint8_t *hotStart = (uint8_t *)&__hot_start;
|
|
|
|
uint8_t *hotEnd = (uint8_t *)&__hot_end;
|
|
|
|
// Make sure the start and end are aligned with huge page address
|
|
|
|
const size_t hugePageBytes = 2L * 1024 * 1024;
|
|
|
|
uint8_t *from = hotStart - ((intptr_t)hotStart & (hugePageBytes - 1));
|
|
|
|
uint8_t *to = hotEnd + (hugePageBytes - 1);
|
|
|
|
to -= (intptr_t)to & (hugePageBytes - 1);
|
|
|
|
|
|
|
|
#ifdef ENABLE_DEBUG
|
|
|
|
reportNumber("[hugify] hot start: ", (uint64_t)hotStart, 16);
|
|
|
|
reportNumber("[hugify] hot end: ", (uint64_t)hotEnd, 16);
|
|
|
|
reportNumber("[hugify] aligned huge page from: ", (uint64_t)from, 16);
|
|
|
|
reportNumber("[hugify] aligned huge page to: ", (uint64_t)to, 16);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
if (!has_pagecache_thp_support()) {
|
|
|
|
hugify_for_old_kernel(from, to);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (__madvise(from, (to - from), MADV_HUGEPAGE) == -1) {
|
|
|
|
char msg[] = "failed to allocate large page\n";
|
|
|
|
// TODO: allow user to control the failure behavior.
|
|
|
|
reportError(msg, sizeof(msg));
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
/// This is hooking ELF's entry, it needs to save all machine state.
|
|
|
|
extern "C" __attribute((naked)) void __bolt_hugify_self() {
|
2020-07-28 07:52:51 +08:00
|
|
|
__asm__ __volatile__(SAVE_ALL
|
|
|
|
"call __bolt_hugify_self_impl\n"
|
|
|
|
RESTORE_ALL
|
|
|
|
"jmp *__bolt_hugify_init_ptr(%%rip)\n"
|
|
|
|
:::);
|
Adding automatic huge page support
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
2020-05-03 02:14:38 +08:00
|
|
|
}
|
2020-10-15 18:51:56 +08:00
|
|
|
|
|
|
|
#endif
|
2021-10-16 22:35:29 +08:00
|
|
|
#endif
|