2019-08-16 13:45:43 +08:00
|
|
|
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
|
selftests/bpf: add bpf-gcc support
Now that binutils and gcc support for BPF is upstream, make use of it in
BPF selftests using alu32-like approach. Share as much as possible of
CFLAGS calculation with clang.
Fixes only obvious issues, leaving more complex ones for later:
- Use gcc-provided bpf-helpers.h instead of manually defining the
helpers, change bpf_helpers.h include guard to avoid conflict.
- Include <linux/stddef.h> for __always_inline.
- Add $(OUTPUT)/../usr/include to include path in order to use local
kernel headers instead of system kernel headers when building with O=.
In order to activate the bpf-gcc support, one needs to configure
binutils and gcc with --target=bpf and make them available in $PATH. In
particular, gcc must be installed as `bpf-gcc`, which is the default.
Right now with binutils 25a2915e8dba and gcc r275589 only a handful of
tests work:
# ./test_progs_bpf_gcc
# Summary: 7/39 PASSED, 1 SKIPPED, 98 FAILED
The reason for those failures are as follows:
- Build errors:
- `error: too many function arguments for eBPF` for __always_inline
functions read_str_var and read_map_var - must be inlining issue,
and for process_l3_headers_v6, which relies on optimizing away
function arguments.
- `error: indirect call in function, which are not supported by eBPF`
where there are no obvious indirect calls in the source calls, e.g.
in __encap_ipip_none.
- `error: field 'lock' has incomplete type` for fields of `struct
bpf_spin_lock` type - bpf_spin_lock is re#defined by bpf-helpers.h,
so its usage is sensitive to order of #includes.
- `error: eBPF stack limit exceeded` in sysctl_tcp_mem.
- Load errors:
- Missing object files due to above build errors.
- `libbpf: failed to create map (name: 'test_ver.bss')`.
- `libbpf: object file doesn't contain bpf program`.
- `libbpf: Program '.text' contains unrecognized relo data pointing to
section 0`.
- `libbpf: BTF is required, but is missing or corrupted` - no BTF
support in gcc yet.
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Cc: Jose E. Marchesi <jose.marchesi@oracle.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2019-09-13 00:05:43 +08:00
|
|
|
#ifndef __BPF_HELPERS__
|
|
|
|
|
#define __BPF_HELPERS__
|
2014-12-02 07:06:37 +08:00
|
|
|
|
2020-04-21 08:05:27 +08:00
|
|
|
/*
|
|
|
|
|
* Note that bpf programs need to include either
|
|
|
|
|
* vmlinux.h (auto-generated from BTF) or linux/types.h
|
|
|
|
|
* in advance since bpf_helper_defs.h uses such types
|
|
|
|
|
* as __u64.
|
|
|
|
|
*/
|
2019-10-07 11:07:38 +08:00
|
|
|
#include "bpf_helper_defs.h"
|
|
|
|
|
|
2019-07-05 23:50:10 +08:00
|
|
|
#define __uint(name, val) int (*name)[val]
|
2019-10-04 12:02:11 +08:00
|
|
|
#define __type(name, val) typeof(val) *name
|
libbpf: Add BTF-defined map-in-map support
As discussed at LPC 2019 ([0]), this patch brings (a quite belated) support
for declarative BTF-defined map-in-map support in libbpf. It allows to define
ARRAY_OF_MAPS and HASH_OF_MAPS BPF maps without any user-space initialization
code involved.
Additionally, it allows to initialize outer map's slots with references to
respective inner maps at load time, also completely declaratively.
Despite a weak type system of C, the way BTF-defined map-in-map definition
works, it's actually quite hard to accidentally initialize outer map with
incompatible inner maps. This being C, of course, it's still possible, but
even that would be caught at load time and error returned with helpful debug
log pointing exactly to the slot that failed to be initialized.
As an example, here's a rather advanced HASH_OF_MAPS declaration and
initialization example, filling slots #0 and #4 with two inner maps:
#include <bpf/bpf_helpers.h>
struct inner_map {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, 1);
__type(key, int);
__type(value, int);
} inner_map1 SEC(".maps"),
inner_map2 SEC(".maps");
struct outer_hash {
__uint(type, BPF_MAP_TYPE_HASH_OF_MAPS);
__uint(max_entries, 5);
__uint(key_size, sizeof(int));
__array(values, struct inner_map);
} outer_hash SEC(".maps") = {
.values = {
[0] = &inner_map2,
[4] = &inner_map1,
},
};
Here's the relevant part of libbpf debug log showing pretty clearly of what's
going on with map-in-map initialization:
libbpf: .maps relo #0: for 6 value 0 rel.r_offset 96 name 260 ('inner_map1')
libbpf: .maps relo #0: map 'outer_arr' slot [0] points to map 'inner_map1'
libbpf: .maps relo #1: for 7 value 32 rel.r_offset 112 name 249 ('inner_map2')
libbpf: .maps relo #1: map 'outer_arr' slot [2] points to map 'inner_map2'
libbpf: .maps relo #2: for 7 value 32 rel.r_offset 144 name 249 ('inner_map2')
libbpf: .maps relo #2: map 'outer_hash' slot [0] points to map 'inner_map2'
libbpf: .maps relo #3: for 6 value 0 rel.r_offset 176 name 260 ('inner_map1')
libbpf: .maps relo #3: map 'outer_hash' slot [4] points to map 'inner_map1'
libbpf: map 'inner_map1': created successfully, fd=4
libbpf: map 'inner_map2': created successfully, fd=5
libbpf: map 'outer_hash': created successfully, fd=7
libbpf: map 'outer_hash': slot [0] set to map 'inner_map2' fd=5
libbpf: map 'outer_hash': slot [4] set to map 'inner_map1' fd=4
Notice from the log above that fd=6 (not logged explicitly) is used for inner
"prototype" map, necessary for creation of outer map. It is destroyed
immediately after outer map is created.
See also included selftest with some extra comments explaining extra details
of usage. Additionally, similar initialization syntax and libbpf functionality
can be used to do initialization of BPF_PROG_ARRAY with references to BPF
sub-programs. This can be done in follow up patches, if there will be a demand
for this.
[0] https://linuxplumbersconf.org/event/4/contributions/448/
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Toke Høiland-Jørgensen <toke@redhat.com>
Link: https://lore.kernel.org/bpf/20200429002739.48006-4-andriin@fb.com
2020-04-29 08:27:39 +08:00
|
|
|
#define __array(name, val) typeof(val) *name[]
|
2019-07-05 23:50:10 +08:00
|
|
|
|
2019-10-09 01:59:41 +08:00
|
|
|
/*
|
|
|
|
|
* Helper macro to place programs, maps, license in
|
selftests/bpf: add bpf-gcc support
Now that binutils and gcc support for BPF is upstream, make use of it in
BPF selftests using alu32-like approach. Share as much as possible of
CFLAGS calculation with clang.
Fixes only obvious issues, leaving more complex ones for later:
- Use gcc-provided bpf-helpers.h instead of manually defining the
helpers, change bpf_helpers.h include guard to avoid conflict.
- Include <linux/stddef.h> for __always_inline.
- Add $(OUTPUT)/../usr/include to include path in order to use local
kernel headers instead of system kernel headers when building with O=.
In order to activate the bpf-gcc support, one needs to configure
binutils and gcc with --target=bpf and make them available in $PATH. In
particular, gcc must be installed as `bpf-gcc`, which is the default.
Right now with binutils 25a2915e8dba and gcc r275589 only a handful of
tests work:
# ./test_progs_bpf_gcc
# Summary: 7/39 PASSED, 1 SKIPPED, 98 FAILED
The reason for those failures are as follows:
- Build errors:
- `error: too many function arguments for eBPF` for __always_inline
functions read_str_var and read_map_var - must be inlining issue,
and for process_l3_headers_v6, which relies on optimizing away
function arguments.
- `error: indirect call in function, which are not supported by eBPF`
where there are no obvious indirect calls in the source calls, e.g.
in __encap_ipip_none.
- `error: field 'lock' has incomplete type` for fields of `struct
bpf_spin_lock` type - bpf_spin_lock is re#defined by bpf-helpers.h,
so its usage is sensitive to order of #includes.
- `error: eBPF stack limit exceeded` in sysctl_tcp_mem.
- Load errors:
- Missing object files due to above build errors.
- `libbpf: failed to create map (name: 'test_ver.bss')`.
- `libbpf: object file doesn't contain bpf program`.
- `libbpf: Program '.text' contains unrecognized relo data pointing to
section 0`.
- `libbpf: BTF is required, but is missing or corrupted` - no BTF
support in gcc yet.
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Cc: Jose E. Marchesi <jose.marchesi@oracle.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2019-09-13 00:05:43 +08:00
|
|
|
* different sections in elf_bpf file. Section names
|
2021-04-24 02:13:33 +08:00
|
|
|
* are interpreted by libbpf depending on the context (BPF programs, BPF maps,
|
|
|
|
|
* extern variables, etc).
|
|
|
|
|
* To allow use of SEC() with externs (e.g., for extern .maps declarations),
|
|
|
|
|
* make sure __attribute__((unused)) doesn't trigger compilation warning.
|
selftests/bpf: add bpf-gcc support
Now that binutils and gcc support for BPF is upstream, make use of it in
BPF selftests using alu32-like approach. Share as much as possible of
CFLAGS calculation with clang.
Fixes only obvious issues, leaving more complex ones for later:
- Use gcc-provided bpf-helpers.h instead of manually defining the
helpers, change bpf_helpers.h include guard to avoid conflict.
- Include <linux/stddef.h> for __always_inline.
- Add $(OUTPUT)/../usr/include to include path in order to use local
kernel headers instead of system kernel headers when building with O=.
In order to activate the bpf-gcc support, one needs to configure
binutils and gcc with --target=bpf and make them available in $PATH. In
particular, gcc must be installed as `bpf-gcc`, which is the default.
Right now with binutils 25a2915e8dba and gcc r275589 only a handful of
tests work:
# ./test_progs_bpf_gcc
# Summary: 7/39 PASSED, 1 SKIPPED, 98 FAILED
The reason for those failures are as follows:
- Build errors:
- `error: too many function arguments for eBPF` for __always_inline
functions read_str_var and read_map_var - must be inlining issue,
and for process_l3_headers_v6, which relies on optimizing away
function arguments.
- `error: indirect call in function, which are not supported by eBPF`
where there are no obvious indirect calls in the source calls, e.g.
in __encap_ipip_none.
- `error: field 'lock' has incomplete type` for fields of `struct
bpf_spin_lock` type - bpf_spin_lock is re#defined by bpf-helpers.h,
so its usage is sensitive to order of #includes.
- `error: eBPF stack limit exceeded` in sysctl_tcp_mem.
- Load errors:
- Missing object files due to above build errors.
- `libbpf: failed to create map (name: 'test_ver.bss')`.
- `libbpf: object file doesn't contain bpf program`.
- `libbpf: Program '.text' contains unrecognized relo data pointing to
section 0`.
- `libbpf: BTF is required, but is missing or corrupted` - no BTF
support in gcc yet.
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Cc: Jose E. Marchesi <jose.marchesi@oracle.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2019-09-13 00:05:43 +08:00
|
|
|
*/
|
2021-04-24 02:13:33 +08:00
|
|
|
#define SEC(name) \
|
|
|
|
|
_Pragma("GCC diagnostic push") \
|
|
|
|
|
_Pragma("GCC diagnostic ignored \"-Wignored-attributes\"") \
|
|
|
|
|
__attribute__((section(name), used)) \
|
|
|
|
|
_Pragma("GCC diagnostic pop") \
|
selftests/bpf: add bpf-gcc support
Now that binutils and gcc support for BPF is upstream, make use of it in
BPF selftests using alu32-like approach. Share as much as possible of
CFLAGS calculation with clang.
Fixes only obvious issues, leaving more complex ones for later:
- Use gcc-provided bpf-helpers.h instead of manually defining the
helpers, change bpf_helpers.h include guard to avoid conflict.
- Include <linux/stddef.h> for __always_inline.
- Add $(OUTPUT)/../usr/include to include path in order to use local
kernel headers instead of system kernel headers when building with O=.
In order to activate the bpf-gcc support, one needs to configure
binutils and gcc with --target=bpf and make them available in $PATH. In
particular, gcc must be installed as `bpf-gcc`, which is the default.
Right now with binutils 25a2915e8dba and gcc r275589 only a handful of
tests work:
# ./test_progs_bpf_gcc
# Summary: 7/39 PASSED, 1 SKIPPED, 98 FAILED
The reason for those failures are as follows:
- Build errors:
- `error: too many function arguments for eBPF` for __always_inline
functions read_str_var and read_map_var - must be inlining issue,
and for process_l3_headers_v6, which relies on optimizing away
function arguments.
- `error: indirect call in function, which are not supported by eBPF`
where there are no obvious indirect calls in the source calls, e.g.
in __encap_ipip_none.
- `error: field 'lock' has incomplete type` for fields of `struct
bpf_spin_lock` type - bpf_spin_lock is re#defined by bpf-helpers.h,
so its usage is sensitive to order of #includes.
- `error: eBPF stack limit exceeded` in sysctl_tcp_mem.
- Load errors:
- Missing object files due to above build errors.
- `libbpf: failed to create map (name: 'test_ver.bss')`.
- `libbpf: object file doesn't contain bpf program`.
- `libbpf: Program '.text' contains unrecognized relo data pointing to
section 0`.
- `libbpf: BTF is required, but is missing or corrupted` - no BTF
support in gcc yet.
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Cc: Jose E. Marchesi <jose.marchesi@oracle.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2019-09-13 00:05:43 +08:00
|
|
|
|
2021-03-15 01:38:38 +08:00
|
|
|
/* Avoid 'linux/stddef.h' definition of '__always_inline'. */
|
|
|
|
|
#undef __always_inline
|
2021-01-14 06:36:08 +08:00
|
|
|
#define __always_inline inline __attribute__((always_inline))
|
2021-03-15 01:38:38 +08:00
|
|
|
|
2020-08-21 07:12:37 +08:00
|
|
|
#ifndef __noinline
|
|
|
|
|
#define __noinline __attribute__((noinline))
|
|
|
|
|
#endif
|
libbpf: Support libbpf-provided extern variables
Add support for extern variables, provided to BPF program by libbpf. Currently
the following extern variables are supported:
- LINUX_KERNEL_VERSION; version of a kernel in which BPF program is
executing, follows KERNEL_VERSION() macro convention, can be 4- and 8-byte
long;
- CONFIG_xxx values; a set of values of actual kernel config. Tristate,
boolean, strings, and integer values are supported.
Set of possible values is determined by declared type of extern variable.
Supported types of variables are:
- Tristate values. Are represented as `enum libbpf_tristate`. Accepted values
are **strictly** 'y', 'n', or 'm', which are represented as TRI_YES, TRI_NO,
or TRI_MODULE, respectively.
- Boolean values. Are represented as bool (_Bool) types. Accepted values are
'y' and 'n' only, turning into true/false values, respectively.
- Single-character values. Can be used both as a substritute for
bool/tristate, or as a small-range integer:
- 'y'/'n'/'m' are represented as is, as characters 'y', 'n', or 'm';
- integers in a range [-128, 127] or [0, 255] (depending on signedness of
char in target architecture) are recognized and represented with
respective values of char type.
- Strings. String values are declared as fixed-length char arrays. String of
up to that length will be accepted and put in first N bytes of char array,
with the rest of bytes zeroed out. If config string value is longer than
space alloted, it will be truncated and warning message emitted. Char array
is always zero terminated. String literals in config have to be enclosed in
double quotes, just like C-style string literals.
- Integers. 8-, 16-, 32-, and 64-bit integers are supported, both signed and
unsigned variants. Libbpf enforces parsed config value to be in the
supported range of corresponding integer type. Integers values in config can
be:
- decimal integers, with optional + and - signs;
- hexadecimal integers, prefixed with 0x or 0X;
- octal integers, starting with 0.
Config file itself is searched in /boot/config-$(uname -r) location with
fallback to /proc/config.gz, unless config path is specified explicitly
through bpf_object_open_opts' kernel_config_path option. Both gzipped and
plain text formats are supported. Libbpf adds explicit dependency on zlib
because of this, but this shouldn't be a problem, given libelf already depends
on zlib.
All detected extern variables, are put into a separate .extern internal map.
It, similarly to .rodata map, is marked as read-only from BPF program side, as
well as is frozen on load. This allows BPF verifier to track extern values as
constants and perform enhanced branch prediction and dead code elimination.
This can be relied upon for doing kernel version/feature detection and using
potentially unsupported field relocations or BPF helpers in a CO-RE-based BPF
program, while still having a single version of BPF program running on old and
new kernels. Selftests are validating this explicitly for unexisting BPF
helper.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20191214014710.3449601-3-andriin@fb.com
2019-12-14 09:47:08 +08:00
|
|
|
#ifndef __weak
|
|
|
|
|
#define __weak __attribute__((weak))
|
|
|
|
|
#endif
|
2019-10-09 01:59:41 +08:00
|
|
|
|
2021-04-24 02:13:34 +08:00
|
|
|
/*
|
|
|
|
|
* Use __hidden attribute to mark a non-static BPF subprogram effectively
|
|
|
|
|
* static for BPF verifier's verification algorithm purposes, allowing more
|
|
|
|
|
* extensive and permissive BPF verification process, taking into account
|
|
|
|
|
* subprogram's caller context.
|
|
|
|
|
*/
|
|
|
|
|
#define __hidden __attribute__((visibility("hidden")))
|
|
|
|
|
|
libbpf: provide NULL and KERNEL_VERSION macros in bpf_helpers.h
Given that vmlinux.h is not compatible with headers like stddef.h, NULL poses
an annoying problem: it is defined as #define, so is not captured in BTF, so
is not emitted into vmlinux.h. This leads to users either sticking to explicit
0, or defining their own NULL (as progs/skb_pkt_end.c does).
But it's easy for bpf_helpers.h to provide (conditionally) NULL definition.
Similarly, KERNEL_VERSION is another commonly missed macro that came up
multiple times. So this patch adds both of them, along with offsetof(), that
also is typically defined in stddef.h, just like NULL.
This might cause compilation warning for existing BPF applications defining
their own NULL and/or KERNEL_VERSION already:
progs/skb_pkt_end.c:7:9: warning: 'NULL' macro redefined [-Wmacro-redefined]
#define NULL 0
^
/tmp/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h:4:9: note: previous definition is here
#define NULL ((void *)0)
^
It is trivial to fix, though, so long-term benefits outweight temporary
inconveniences.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/r/20210317200510.1354627-2-andrii@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2021-03-18 04:05:09 +08:00
|
|
|
/* When utilizing vmlinux.h with BPF CO-RE, user BPF programs can't include
|
|
|
|
|
* any system-level headers (such as stddef.h, linux/version.h, etc), and
|
|
|
|
|
* commonly-used macros like NULL and KERNEL_VERSION aren't available through
|
|
|
|
|
* vmlinux.h. This just adds unnecessary hurdles and forces users to re-define
|
|
|
|
|
* them on their own. So as a convenience, provide such definitions here.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef NULL
|
|
|
|
|
#define NULL ((void *)0)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef KERNEL_VERSION
|
2021-04-05 12:01:19 +08:00
|
|
|
#define KERNEL_VERSION(a, b, c) (((a) << 16) + ((b) << 8) + ((c) > 255 ? 255 : (c)))
|
libbpf: provide NULL and KERNEL_VERSION macros in bpf_helpers.h
Given that vmlinux.h is not compatible with headers like stddef.h, NULL poses
an annoying problem: it is defined as #define, so is not captured in BTF, so
is not emitted into vmlinux.h. This leads to users either sticking to explicit
0, or defining their own NULL (as progs/skb_pkt_end.c does).
But it's easy for bpf_helpers.h to provide (conditionally) NULL definition.
Similarly, KERNEL_VERSION is another commonly missed macro that came up
multiple times. So this patch adds both of them, along with offsetof(), that
also is typically defined in stddef.h, just like NULL.
This might cause compilation warning for existing BPF applications defining
their own NULL and/or KERNEL_VERSION already:
progs/skb_pkt_end.c:7:9: warning: 'NULL' macro redefined [-Wmacro-redefined]
#define NULL 0
^
/tmp/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h:4:9: note: previous definition is here
#define NULL ((void *)0)
^
It is trivial to fix, though, so long-term benefits outweight temporary
inconveniences.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/r/20210317200510.1354627-2-andrii@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2021-03-18 04:05:09 +08:00
|
|
|
#endif
|
|
|
|
|
|
2020-05-10 01:59:19 +08:00
|
|
|
/*
|
libbpf: provide NULL and KERNEL_VERSION macros in bpf_helpers.h
Given that vmlinux.h is not compatible with headers like stddef.h, NULL poses
an annoying problem: it is defined as #define, so is not captured in BTF, so
is not emitted into vmlinux.h. This leads to users either sticking to explicit
0, or defining their own NULL (as progs/skb_pkt_end.c does).
But it's easy for bpf_helpers.h to provide (conditionally) NULL definition.
Similarly, KERNEL_VERSION is another commonly missed macro that came up
multiple times. So this patch adds both of them, along with offsetof(), that
also is typically defined in stddef.h, just like NULL.
This might cause compilation warning for existing BPF applications defining
their own NULL and/or KERNEL_VERSION already:
progs/skb_pkt_end.c:7:9: warning: 'NULL' macro redefined [-Wmacro-redefined]
#define NULL 0
^
/tmp/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h:4:9: note: previous definition is here
#define NULL ((void *)0)
^
It is trivial to fix, though, so long-term benefits outweight temporary
inconveniences.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/r/20210317200510.1354627-2-andrii@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2021-03-18 04:05:09 +08:00
|
|
|
* Helper macros to manipulate data structures
|
2020-05-10 01:59:19 +08:00
|
|
|
*/
|
|
|
|
|
#ifndef offsetof
|
2020-08-11 11:08:52 +08:00
|
|
|
#define offsetof(TYPE, MEMBER) ((unsigned long)&((TYPE *)0)->MEMBER)
|
2020-05-10 01:59:19 +08:00
|
|
|
#endif
|
|
|
|
|
#ifndef container_of
|
|
|
|
|
#define container_of(ptr, type, member) \
|
|
|
|
|
({ \
|
|
|
|
|
void *__mptr = (void *)(ptr); \
|
|
|
|
|
((type *)(__mptr - offsetof(type, member))); \
|
|
|
|
|
})
|
|
|
|
|
#endif
|
|
|
|
|
|
2020-09-30 23:18:18 +08:00
|
|
|
/*
|
|
|
|
|
* Helper macro to throw a compilation error if __bpf_unreachable() gets
|
|
|
|
|
* built into the resulting code. This works given BPF back end does not
|
|
|
|
|
* implement __builtin_trap(). This is useful to assert that certain paths
|
|
|
|
|
* of the program code are never used and hence eliminated by the compiler.
|
|
|
|
|
*
|
|
|
|
|
* For example, consider a switch statement that covers known cases used by
|
|
|
|
|
* the program. __bpf_unreachable() can then reside in the default case. If
|
|
|
|
|
* the program gets extended such that a case is not covered in the switch
|
|
|
|
|
* statement, then it will throw a build error due to the default case not
|
|
|
|
|
* being compiled out.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef __bpf_unreachable
|
|
|
|
|
# define __bpf_unreachable() __builtin_trap()
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Helper function to perform a tail call with a constant/immediate map slot.
|
|
|
|
|
*/
|
bpf, libbpf: Guard bpf inline asm from bpf_tail_call_static
Yaniv reported a compilation error after pulling latest libbpf:
[...]
../libbpf/src/root/usr/include/bpf/bpf_helpers.h:99:10: error:
unknown register name 'r0' in asm
: "r0", "r1", "r2", "r3", "r4", "r5");
[...]
The issue got triggered given Yaniv was compiling tracing programs with native
target (e.g. x86) instead of BPF target, hence no BTF generated vmlinux.h nor
CO-RE used, and later llc with -march=bpf was invoked to compile from LLVM IR
to BPF object file. Given that clang was expecting x86 inline asm and not BPF
one the error complained that these regs don't exist on the former.
Guard bpf_tail_call_static() with defined(__bpf__) where BPF inline asm is valid
to use. BPF tracing programs on more modern kernels use BPF target anyway and
thus the bpf_tail_call_static() function will be available for them. BPF inline
asm is supported since clang 7 (clang <= 6 otherwise throws same above error),
and __bpf_unreachable() since clang 8, therefore include the latter condition
in order to prevent compilation errors for older clang versions. Given even an
old Ubuntu 18.04 LTS has official LLVM packages all the way up to llvm-10, I did
not bother to special case the __bpf_unreachable() inside bpf_tail_call_static()
further.
Also, undo the sockex3_kern's use of bpf_tail_call_static() sample given they
still have the old hacky way to even compile networking progs with native instead
of BPF target so bpf_tail_call_static() won't be defined there anymore.
Fixes: 0e9f6841f664 ("bpf, libbpf: Add bpf_tail_call_static helper for bpf programs")
Reported-by: Yaniv Agman <yanivagman@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Yonghong Song <yhs@fb.com>
Tested-by: Yaniv Agman <yanivagman@gmail.com>
Link: https://lore.kernel.org/bpf/CAMy7=ZUk08w5Gc2Z-EKi4JFtuUCaZYmE4yzhJjrExXpYKR4L8w@mail.gmail.com
Link: https://lore.kernel.org/bpf/20201021203257.26223-1-daniel@iogearbox.net
2020-10-22 04:32:57 +08:00
|
|
|
#if __clang_major__ >= 8 && defined(__bpf__)
|
2020-09-30 23:18:18 +08:00
|
|
|
static __always_inline void
|
|
|
|
|
bpf_tail_call_static(void *ctx, const void *map, const __u32 slot)
|
|
|
|
|
{
|
|
|
|
|
if (!__builtin_constant_p(slot))
|
|
|
|
|
__bpf_unreachable();
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Provide a hard guarantee that LLVM won't optimize setting r2 (map
|
|
|
|
|
* pointer) and r3 (constant map index) from _different paths_ ending
|
|
|
|
|
* up at the _same_ call insn as otherwise we won't be able to use the
|
|
|
|
|
* jmpq/nopl retpoline-free patching by the x86-64 JIT in the kernel
|
|
|
|
|
* given they mismatch. See also d2e4c1e6c294 ("bpf: Constant map key
|
|
|
|
|
* tracking for prog array pokes") for details on verifier tracking.
|
|
|
|
|
*
|
|
|
|
|
* Note on clobber list: we need to stay in-line with BPF calling
|
|
|
|
|
* convention, so even if we don't end up using r0, r4, r5, we need
|
|
|
|
|
* to mark them as clobber so that LLVM doesn't end up using them
|
|
|
|
|
* before / after the call.
|
|
|
|
|
*/
|
|
|
|
|
asm volatile("r1 = %[ctx]\n\t"
|
|
|
|
|
"r2 = %[map]\n\t"
|
|
|
|
|
"r3 = %[slot]\n\t"
|
|
|
|
|
"call 12"
|
|
|
|
|
:: [ctx]"r"(ctx), [map]"r"(map), [slot]"i"(slot)
|
|
|
|
|
: "r0", "r1", "r2", "r3", "r4", "r5");
|
|
|
|
|
}
|
bpf, libbpf: Guard bpf inline asm from bpf_tail_call_static
Yaniv reported a compilation error after pulling latest libbpf:
[...]
../libbpf/src/root/usr/include/bpf/bpf_helpers.h:99:10: error:
unknown register name 'r0' in asm
: "r0", "r1", "r2", "r3", "r4", "r5");
[...]
The issue got triggered given Yaniv was compiling tracing programs with native
target (e.g. x86) instead of BPF target, hence no BTF generated vmlinux.h nor
CO-RE used, and later llc with -march=bpf was invoked to compile from LLVM IR
to BPF object file. Given that clang was expecting x86 inline asm and not BPF
one the error complained that these regs don't exist on the former.
Guard bpf_tail_call_static() with defined(__bpf__) where BPF inline asm is valid
to use. BPF tracing programs on more modern kernels use BPF target anyway and
thus the bpf_tail_call_static() function will be available for them. BPF inline
asm is supported since clang 7 (clang <= 6 otherwise throws same above error),
and __bpf_unreachable() since clang 8, therefore include the latter condition
in order to prevent compilation errors for older clang versions. Given even an
old Ubuntu 18.04 LTS has official LLVM packages all the way up to llvm-10, I did
not bother to special case the __bpf_unreachable() inside bpf_tail_call_static()
further.
Also, undo the sockex3_kern's use of bpf_tail_call_static() sample given they
still have the old hacky way to even compile networking progs with native instead
of BPF target so bpf_tail_call_static() won't be defined there anymore.
Fixes: 0e9f6841f664 ("bpf, libbpf: Add bpf_tail_call_static helper for bpf programs")
Reported-by: Yaniv Agman <yanivagman@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Yonghong Song <yhs@fb.com>
Tested-by: Yaniv Agman <yanivagman@gmail.com>
Link: https://lore.kernel.org/bpf/CAMy7=ZUk08w5Gc2Z-EKi4JFtuUCaZYmE4yzhJjrExXpYKR4L8w@mail.gmail.com
Link: https://lore.kernel.org/bpf/20201021203257.26223-1-daniel@iogearbox.net
2020-10-22 04:32:57 +08:00
|
|
|
#endif
|
2020-09-30 23:18:18 +08:00
|
|
|
|
2019-10-09 01:59:41 +08:00
|
|
|
/*
|
|
|
|
|
* Helper structure used by eBPF C program
|
2019-10-09 01:59:37 +08:00
|
|
|
* to describe BPF map attributes to libbpf loader
|
2014-12-02 07:06:37 +08:00
|
|
|
*/
|
|
|
|
|
struct bpf_map_def {
|
|
|
|
|
unsigned int type;
|
|
|
|
|
unsigned int key_size;
|
|
|
|
|
unsigned int value_size;
|
|
|
|
|
unsigned int max_entries;
|
2016-03-08 13:57:20 +08:00
|
|
|
unsigned int map_flags;
|
2014-12-02 07:06:37 +08:00
|
|
|
};
|
|
|
|
|
|
2019-11-02 19:09:41 +08:00
|
|
|
enum libbpf_pin_type {
|
|
|
|
|
LIBBPF_PIN_NONE,
|
|
|
|
|
/* PIN_BY_NAME: pin maps by name (in /sys/fs/bpf by default) */
|
|
|
|
|
LIBBPF_PIN_BY_NAME,
|
|
|
|
|
};
|
|
|
|
|
|
libbpf: Support libbpf-provided extern variables
Add support for extern variables, provided to BPF program by libbpf. Currently
the following extern variables are supported:
- LINUX_KERNEL_VERSION; version of a kernel in which BPF program is
executing, follows KERNEL_VERSION() macro convention, can be 4- and 8-byte
long;
- CONFIG_xxx values; a set of values of actual kernel config. Tristate,
boolean, strings, and integer values are supported.
Set of possible values is determined by declared type of extern variable.
Supported types of variables are:
- Tristate values. Are represented as `enum libbpf_tristate`. Accepted values
are **strictly** 'y', 'n', or 'm', which are represented as TRI_YES, TRI_NO,
or TRI_MODULE, respectively.
- Boolean values. Are represented as bool (_Bool) types. Accepted values are
'y' and 'n' only, turning into true/false values, respectively.
- Single-character values. Can be used both as a substritute for
bool/tristate, or as a small-range integer:
- 'y'/'n'/'m' are represented as is, as characters 'y', 'n', or 'm';
- integers in a range [-128, 127] or [0, 255] (depending on signedness of
char in target architecture) are recognized and represented with
respective values of char type.
- Strings. String values are declared as fixed-length char arrays. String of
up to that length will be accepted and put in first N bytes of char array,
with the rest of bytes zeroed out. If config string value is longer than
space alloted, it will be truncated and warning message emitted. Char array
is always zero terminated. String literals in config have to be enclosed in
double quotes, just like C-style string literals.
- Integers. 8-, 16-, 32-, and 64-bit integers are supported, both signed and
unsigned variants. Libbpf enforces parsed config value to be in the
supported range of corresponding integer type. Integers values in config can
be:
- decimal integers, with optional + and - signs;
- hexadecimal integers, prefixed with 0x or 0X;
- octal integers, starting with 0.
Config file itself is searched in /boot/config-$(uname -r) location with
fallback to /proc/config.gz, unless config path is specified explicitly
through bpf_object_open_opts' kernel_config_path option. Both gzipped and
plain text formats are supported. Libbpf adds explicit dependency on zlib
because of this, but this shouldn't be a problem, given libelf already depends
on zlib.
All detected extern variables, are put into a separate .extern internal map.
It, similarly to .rodata map, is marked as read-only from BPF program side, as
well as is frozen on load. This allows BPF verifier to track extern values as
constants and perform enhanced branch prediction and dead code elimination.
This can be relied upon for doing kernel version/feature detection and using
potentially unsupported field relocations or BPF helpers in a CO-RE-based BPF
program, while still having a single version of BPF program running on old and
new kernels. Selftests are validating this explicitly for unexisting BPF
helper.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20191214014710.3449601-3-andriin@fb.com
2019-12-14 09:47:08 +08:00
|
|
|
enum libbpf_tristate {
|
|
|
|
|
TRI_NO = 0,
|
|
|
|
|
TRI_YES = 1,
|
|
|
|
|
TRI_MODULE = 2,
|
|
|
|
|
};
|
|
|
|
|
|
2019-12-19 08:28:34 +08:00
|
|
|
#define __kconfig __attribute__((section(".kconfig")))
|
libbpf: Add support for extracting kernel symbol addresses
Add support for another (in addition to existing Kconfig) special kind of
externs in BPF code, kernel symbol externs. Such externs allow BPF code to
"know" kernel symbol address and either use it for comparisons with kernel
data structures (e.g., struct file's f_op pointer, to distinguish different
kinds of file), or, with the help of bpf_probe_user_kernel(), to follow
pointers and read data from global variables. Kernel symbol addresses are
found through /proc/kallsyms, which should be present in the system.
Currently, such kernel symbol variables are typeless: they have to be defined
as `extern const void <symbol>` and the only operation you can do (in C code)
with them is to take its address. Such extern should reside in a special
section '.ksyms'. bpf_helpers.h header provides __ksym macro for this. Strong
vs weak semantics stays the same as with Kconfig externs. If symbol is not
found in /proc/kallsyms, this will be a failure for strong (non-weak) extern,
but will be defaulted to 0 for weak externs.
If the same symbol is defined multiple times in /proc/kallsyms, then it will
be error if any of the associated addresses differs. In that case, address is
ambiguous, so libbpf falls on the side of caution, rather than confusing user
with randomly chosen address.
In the future, once kernel is extended with variables BTF information, such
ksym externs will be supported in a typed version, which will allow BPF
program to read variable's contents directly, similarly to how it's done for
fentry/fexit input arguments.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Hao Luo <haoluo@google.com>
Link: https://lore.kernel.org/bpf/20200619231703.738941-3-andriin@fb.com
2020-06-20 07:16:56 +08:00
|
|
|
#define __ksym __attribute__((section(".ksyms")))
|
2019-12-19 08:28:34 +08:00
|
|
|
|
2021-05-27 00:46:43 +08:00
|
|
|
#ifndef ___bpf_concat
|
|
|
|
|
#define ___bpf_concat(a, b) a ## b
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef ___bpf_apply
|
|
|
|
|
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef ___bpf_nth
|
|
|
|
|
#define ___bpf_nth(_, _1, _2, _3, _4, _5, _6, _7, _8, _9, _a, _b, _c, N, ...) N
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef ___bpf_narg
|
|
|
|
|
#define ___bpf_narg(...) \
|
|
|
|
|
___bpf_nth(_, ##__VA_ARGS__, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#define ___bpf_fill0(arr, p, x) do {} while (0)
|
|
|
|
|
#define ___bpf_fill1(arr, p, x) arr[p] = x
|
|
|
|
|
#define ___bpf_fill2(arr, p, x, args...) arr[p] = x; ___bpf_fill1(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill3(arr, p, x, args...) arr[p] = x; ___bpf_fill2(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill4(arr, p, x, args...) arr[p] = x; ___bpf_fill3(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill5(arr, p, x, args...) arr[p] = x; ___bpf_fill4(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill6(arr, p, x, args...) arr[p] = x; ___bpf_fill5(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill7(arr, p, x, args...) arr[p] = x; ___bpf_fill6(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill8(arr, p, x, args...) arr[p] = x; ___bpf_fill7(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill9(arr, p, x, args...) arr[p] = x; ___bpf_fill8(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill10(arr, p, x, args...) arr[p] = x; ___bpf_fill9(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill11(arr, p, x, args...) arr[p] = x; ___bpf_fill10(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill12(arr, p, x, args...) arr[p] = x; ___bpf_fill11(arr, p + 1, args)
|
|
|
|
|
#define ___bpf_fill(arr, args...) \
|
|
|
|
|
___bpf_apply(___bpf_fill, ___bpf_narg(args))(arr, 0, args)
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BPF_SEQ_PRINTF to wrap bpf_seq_printf to-be-printed values
|
|
|
|
|
* in a structure.
|
|
|
|
|
*/
|
|
|
|
|
#define BPF_SEQ_PRINTF(seq, fmt, args...) \
|
|
|
|
|
({ \
|
|
|
|
|
static const char ___fmt[] = fmt; \
|
|
|
|
|
unsigned long long ___param[___bpf_narg(args)]; \
|
|
|
|
|
\
|
|
|
|
|
_Pragma("GCC diagnostic push") \
|
|
|
|
|
_Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
|
|
|
|
|
___bpf_fill(___param, args); \
|
|
|
|
|
_Pragma("GCC diagnostic pop") \
|
|
|
|
|
\
|
|
|
|
|
bpf_seq_printf(seq, ___fmt, sizeof(___fmt), \
|
|
|
|
|
___param, sizeof(___param)); \
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BPF_SNPRINTF wraps the bpf_snprintf helper with variadic arguments instead of
|
|
|
|
|
* an array of u64.
|
|
|
|
|
*/
|
|
|
|
|
#define BPF_SNPRINTF(out, out_size, fmt, args...) \
|
|
|
|
|
({ \
|
|
|
|
|
static const char ___fmt[] = fmt; \
|
|
|
|
|
unsigned long long ___param[___bpf_narg(args)]; \
|
|
|
|
|
\
|
|
|
|
|
_Pragma("GCC diagnostic push") \
|
|
|
|
|
_Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
|
|
|
|
|
___bpf_fill(___param, args); \
|
|
|
|
|
_Pragma("GCC diagnostic pop") \
|
|
|
|
|
\
|
|
|
|
|
bpf_snprintf(out, out_size, ___fmt, \
|
|
|
|
|
___param, sizeof(___param)); \
|
|
|
|
|
})
|
|
|
|
|
|
2021-09-18 02:29:07 +08:00
|
|
|
#ifdef BPF_NO_GLOBAL_DATA
|
|
|
|
|
#define BPF_PRINTK_FMT_MOD
|
|
|
|
|
#else
|
|
|
|
|
#define BPF_PRINTK_FMT_MOD static const
|
|
|
|
|
#endif
|
|
|
|
|
|
2021-09-18 02:29:06 +08:00
|
|
|
#define __bpf_printk(fmt, ...) \
|
|
|
|
|
({ \
|
2021-09-18 02:29:07 +08:00
|
|
|
BPF_PRINTK_FMT_MOD char ____fmt[] = fmt; \
|
2021-09-18 02:29:06 +08:00
|
|
|
bpf_trace_printk(____fmt, sizeof(____fmt), \
|
|
|
|
|
##__VA_ARGS__); \
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* __bpf_vprintk wraps the bpf_trace_vprintk helper with variadic arguments
|
|
|
|
|
* instead of an array of u64.
|
|
|
|
|
*/
|
|
|
|
|
#define __bpf_vprintk(fmt, args...) \
|
|
|
|
|
({ \
|
|
|
|
|
static const char ___fmt[] = fmt; \
|
|
|
|
|
unsigned long long ___param[___bpf_narg(args)]; \
|
|
|
|
|
\
|
|
|
|
|
_Pragma("GCC diagnostic push") \
|
|
|
|
|
_Pragma("GCC diagnostic ignored \"-Wint-conversion\"") \
|
|
|
|
|
___bpf_fill(___param, args); \
|
|
|
|
|
_Pragma("GCC diagnostic pop") \
|
|
|
|
|
\
|
|
|
|
|
bpf_trace_vprintk(___fmt, sizeof(___fmt), \
|
|
|
|
|
___param, sizeof(___param)); \
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
/* Use __bpf_printk when bpf_printk call has 3 or fewer fmt args
|
|
|
|
|
* Otherwise use __bpf_vprintk
|
|
|
|
|
*/
|
|
|
|
|
#define ___bpf_pick_printk(...) \
|
|
|
|
|
___bpf_nth(_, ##__VA_ARGS__, __bpf_vprintk, __bpf_vprintk, __bpf_vprintk, \
|
|
|
|
|
__bpf_vprintk, __bpf_vprintk, __bpf_vprintk, __bpf_vprintk, \
|
|
|
|
|
__bpf_vprintk, __bpf_vprintk, __bpf_printk /*3*/, __bpf_printk /*2*/,\
|
|
|
|
|
__bpf_printk /*1*/, __bpf_printk /*0*/)
|
|
|
|
|
|
|
|
|
|
/* Helper macro to print out debug messages */
|
|
|
|
|
#define bpf_printk(fmt, args...) ___bpf_pick_printk(args)(fmt, ##args)
|
|
|
|
|
|
2014-12-02 07:06:37 +08:00
|
|
|
#endif
|