UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge branch 'Allocated objects, BPF linked lists' 

Kumar Kartikeya Dwivedi says:

====================

This series introduces user defined BPF objects of a type in program
BTF. This allows BPF programs to allocate their own objects, build their
own object hierarchies, and use the basic building blocks provided by
BPF runtime to build their own data structures flexibly.

Then, we introduce the support for single ownership BPF linked lists,
which can be put inside BPF maps, or allocated objects, and hold such
allocated objects as elements. It works as an instrusive collection,
which is done to allow making allocated objects part of multiple data
structures at the same time in the future.

The eventual goal of this and future patches is to allow one to do some
limited form of kernel style programming in BPF C, and allow programmers
to build their own complex data structures flexibly out of basic
building blocks.

The key difference will be that such programs are verified to be safe,
preserve runtime integrity of the system, and are proven to be bug free
as far as the invariants of BPF specific APIs are concerned.

One immediate use case that will be using the entire infrastructure this
series is introducing will be managing percpu NMI safe linked lists
inside BPF programs.

The other use case this will serve in the near future will be linking
kernel structures like XDP frame and sk_buff directly into user data
structures (rbtree, pifomap, etc.) for packet queueing. This will follow
single ownership concept included in this series.

The user has complete control of the internal locking, and hence also
the batching of operations for each critical section.

The features are:
- Allocated objects.
- bpf_obj_new, bpf_obj_drop to allocate and free them.
- Single ownership BPF linked lists.
  - Support for them in BPF maps.
  - Support for them in allocated objects.
- Global spin locks.
- Spin locks inside allocated objects.

Some other notable things:
- Completely static verification of locking.
- Kfunc argument handling has been completely reworked.
- Argument rewriting support for kfuncs.
- A new bpf_experimental.h header as a dumping ground for these APIs.

Any functionality exposed in this series is NOT part of UAPI. It is only
available through use of kfuncs, and structs that can be added to map
value may also change their size or name in the future. Hence, every
feature in this series must be considered experimental.

Follow-ups:
-----------
 * Support for kptrs (local and kernel) in local storage and percpu maps + kptr tests
 * Fixes for helper access checks rebasing on top of this series

Next steps:
-----------
 * NMI safe percpu single ownership linked lists (using local_t protection).
 * Lockless linked lists.
 * Allow RCU protected BPF allocated objects. This then allows RCU
   protected list lookups, since spinlock protection for readers does
   not scale.
 * Introduce bpf_refcount for local kptrs, shared ownership.
 * Introduce shared ownership linked lists.
 * Documentation.

Changelog:
----------
 v9 -> v10
 v9: https://lore.kernel.org/bpf/20221117225510.1676785-1-memxor@gmail.com

  * Deduplicate code to find btf_record of reg (Alexei)
  * Add linked_list test to DENYLIST.aarch64 (Alexei)
  * Disable some linked list tests for now so that they compile with
    clang nightly (Alexei)

 v8 -> v9
 v8: https://lore.kernel.org/bpf/20221117162430.1213770-1-memxor@gmail.com

  * Fix up commit log of patch 2, Simplify patch 3
  * Explain the implicit requirement of bpf_list_head requiring map BTF
    to match in btf_record_equal in a separate patch.

 v7 -> v8
 v7: https://lore.kernel.org/bpf/20221114191547.1694267-1-memxor@gmail.com

  * Fix early return in map_check_btf (Dan Carpenter)
  * Fix two memory leak bugs in local storage maps, outer maps
  * Address comments from Alexei and Dave
   * More local kptr -> allocated object renaming
   * Use krealloc with NULL instead kmalloc + krealloc
   * Drop WARN_ON_ONCE for field_offs parsing
   * Combine kfunc add + remove patches into one
   * Drop STRONG suffix from KF_ARG_PTR_TO_KPTR
   * Rename is_kfunc_arg_ret_buf_size to is_kfunc_arg_scalar_with_name
   * Remove redundant check for reg->type and arg type in it
   * Drop void * ret type check
   * Remove code duplication in checks for NULL pointer with offset != 0
   * Fix two bpf_list_node typos
   * Improve log message for bpf_list_head operations
   * Improve comments for active_lock struct
   * Improve comments for Implementation details of process_spin_lock
  * Add Dave's acks

 v6 -> v7
 v6: https://lore.kernel.org/bpf/20221111193224.876706-1-memxor@gmail.com

  * Fix uninitialized variable warning (Dan Carpenter, Kernel Test Robot)
  * One more local_kptr renaming

 v5 -> v6
 v5: https://lore.kernel.org/bpf/20221107230950.7117-1-memxor@gmail.com

  * Replace (i && !off) check with next_off, include test (Andrii)
  * Drop local kptrs naming (Andrii, Alexei)
  * Drop reg->precise == EXACT patch (Andrii)
  * Add comment about ptr member of struct active_lock (Andrii)
  * Use btf__new_empty + btf__add_xxx APIs (Andrii)
  * Address other misc nits from Andrii

 v4 -> v5
 v4: https://lore.kernel.org/bpf/20221103191013.1236066-1-memxor@gmail.com

  * Add a lot more selftests (failure, success, runtime, BTF)
  * Make sure series is bisect friendly
  * Move list draining out of spin lock
    * This exposed an issue where bpf_mem_free can now be called in
      map_free path without migrate_disable, also fixed that.
  * Rename MEM_ALLOC -> MEM_RINGBUF, MEM_TYPE_LOCAL -> MEM_ALLOC (Alexei)
  * Group lock identity into a struct active_lock { ptr, id } (Dave)
  * Split set_release_on_unlock logic into separate patch (Alexei)

 v3 -> v4
 v3: https://lore.kernel.org/bpf/20221102202658.963008-1-memxor@gmail.com

  * Fix compiler error for !CONFIG_BPF_SYSCALL (Kernel Test Robot)
  * Fix error due to BUILD_BUG_ON on 32-bit platforms (Kernel Test Robot)

 v2 -> v3
 v2: https://lore.kernel.org/bpf/20221013062303.896469-1-memxor@gmail.com

  * Add ack from Dave for patch 5
  * Rename btf_type_fields -> btf_record, btf_type_fields_off ->
    btf_field_offs, rename functions similarly (Alexei)
  * Remove 'kind' component from contains declaration tag (Alexei)
  * Move bpf_list_head, bpf_list_node definitions to UAPI bpf.h (Alexei)
  * Add note in commit log about modifying btf_struct_access API (Dave)
  * Downgrade WARN_ON_ONCE to verbose(env, "...") and return -EFAULT (Dave)
  * Add type_is_local_kptr wrapper to avoid noisy checks (Dave)
  * Remove unused flags parameter from bpf_kptr_new (Alexei)
  * Rename bpf_kptr_new -> bpf_obj_new, bpf_kptr_drop -> bpf_obj_drop (Alexei)
  * Reword comment in ref_obj_id_set_release_on_unlock (Dave)
  * Fix return type of ref_obj_id_set_release_on_unlock (Dave)
  * Introduce is_bpf_list_api_kfunc to dedup checks (Dave)
  * Disallow BPF_WRITE to untrusted local kptrs
  * Add details about soundness of check_reg_allocation_locked logic
  * List untrusted local kptrs for PROBE_MEM handling

 v1 -> v2
 v1: https://lore.kernel.org/bpf/20221011012240.3149-1-memxor@gmail.com

  * Rebase on bpf-next to resolve merge conflict in DENYLIST.s390x
  * Fix a couple of mental lapses in bpf_list_head_free

 RFC v1 -> v1
 RFC v1: https://lore.kernel.org/bpf/20220904204145.3089-1-memxor@gmail.com

  * Mostly a complete rewrite of BTF parsing, refactor existing code (Kartikeya)
  * Rebase kfunc rewrite for bpf-next, add support for more changes
  * Cache type metadata in BTF to avoid recomputation inside verifier (Kartikeya)
  * Remove __kernel tag, make things similar to map values, reserve bpf_ prefix
  * bpf_kptr_new, bpf_kptr_drop
  * Rename precision state enum values (Alexei)
  * Drop explicit constructor/destructor support (Alexei)
  * Rewrite code for constructing/destructing objects and offload to runtime
  * Minimize duplication in bpf_map_value_off_desc handling (Alexei)
  * Expose global memory allocator (Alexei)
  * Address other nits from Alexei
  * Split out local kptrs in maps, more kptrs in maps support into a follow up

Links:
------
 * Dave's BPF RB-Tree RFC series
   v1 (Discussion thread)
     https://lore.kernel.org/bpf/20220722183438.3319790-1-davemarchevsky@fb.com
   v2 (With support for static locks)
     https://lore.kernel.org/bpf/20220830172759.4069786-1-davemarchevsky@fb.com
 * BPF Linked Lists Discussion
   https://lore.kernel.org/bpf/CAP01T74U30+yeBHEgmgzTJ-XYxZ0zj71kqCDJtTH9YQNfTK+Xw@mail.gmail.com
 * BPF Memory Allocator from Alexei
   https://lore.kernel.org/bpf/20220902211058.60789-1-alexei.starovoitov@gmail.com
 * BPF Memory Allocator UAPI Discussion
   https://lore.kernel.org/bpf/d3f76b27f4e55ec9e400ae8dcaecbb702a4932e8.camel@fb.com
====================

Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This commit is contained in:
Alexei Starovoitov 2022-11-17 19:11:34 -08:00
parent 98b2afc8a6 0a2f85a1be
commit db6bf99954
26 changed files with 3931 additions and 567 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
Documentation/bpf/kfuncs.rst
View File

@ -72,6 +72,30 @@ argument as its size. By default, without __sz annotation, the size of the type
of the pointer is used. Without __sz annotation, a kfunc cannot accept a void
pointer.
2.2.2 __k Annotation
--------------------
This annotation is only understood for scalar arguments, where it indicates that
the verifier must check the scalar argument to be a known constant, which does
not indicate a size parameter, and the value of the constant is relevant to the
safety of the program.
An example is given below::
void *bpf_obj_new(u32 local_type_id__k, ...)
{
...
}
Here, bpf_obj_new uses local_type_id argument to find out the size of that type
ID in program's BTF and return a sized pointer to it. Each type ID will have a
distinct size, hence it is crucial to treat each such call as distinct when
values don't match during verifier state pruning checks.
Hence, whenever a constant scalar argument is accepted by a kfunc which is not a
size parameter, and the value of the constant matters for program safety, __k
suffix should be used.
.. _BPF_kfunc_nodef:
2.3 Using an existing kernel function

51
include/linux/bpf.h
View File

@ -54,6 +54,8 @@ struct cgroup;
extern struct idr btf_idr;
extern spinlock_t btf_idr_lock;
extern struct kobject *btf_kobj;
extern struct bpf_mem_alloc bpf_global_ma;
extern bool bpf_global_ma_set;
typedef u64 (*bpf_callback_t)(u64, u64, u64, u64, u64);
typedef int (*bpf_iter_init_seq_priv_t)(void *private_data,
@ -177,6 +179,7 @@ enum btf_field_type {
BPF_KPTR_REF = (1 << 3),
BPF_KPTR = BPF_KPTR_UNREF | BPF_KPTR_REF,
BPF_LIST_HEAD = (1 << 4),
BPF_LIST_NODE = (1 << 5),
};
struct btf_field_kptr {
@ -190,6 +193,7 @@ struct btf_field_list_head {
struct btf *btf;
u32 value_btf_id;
u32 node_offset;
struct btf_record *value_rec;
};
struct btf_field {
@ -277,6 +281,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type)
return "kptr";
case BPF_LIST_HEAD:
return "bpf_list_head";
case BPF_LIST_NODE:
return "bpf_list_node";
default:
WARN_ON_ONCE(1);
return "unknown";
@ -295,6 +301,8 @@ static inline u32 btf_field_type_size(enum btf_field_type type)
return sizeof(u64);
case BPF_LIST_HEAD:
return sizeof(struct bpf_list_head);
case BPF_LIST_NODE:
return sizeof(struct bpf_list_node);
default:
WARN_ON_ONCE(1);
return 0;
@ -313,6 +321,8 @@ static inline u32 btf_field_type_align(enum btf_field_type type)
return __alignof__(u64);
case BPF_LIST_HEAD:
return __alignof__(struct bpf_list_head);
case BPF_LIST_NODE:
return __alignof__(struct bpf_list_node);
default:
WARN_ON_ONCE(1);
return 0;
@ -326,16 +336,19 @@ static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_f
return rec->field_mask & type;
}
static inline void bpf_obj_init(const struct btf_field_offs *foffs, void *obj)
{
int i;
if (!foffs)
return;
for (i = 0; i < foffs->cnt; i++)
memset(obj + foffs->field_off[i], 0, foffs->field_sz[i]);
}
static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
{
if (!IS_ERR_OR_NULL(map->record)) {
struct btf_field *fields = map->record->fields;
u32 cnt = map->record->cnt;
int i;
for (i = 0; i < cnt; i++)
memset(dst + fields[i].offset, 0, btf_field_type_size(fields[i].type));
}
bpf_obj_init(map->field_offs, dst);
}
/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
@ -525,6 +538,11 @@ enum bpf_type_flag {
/* Size is known at compile time. */
MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),
/* MEM is of an allocated object of type in program BTF. This is used to
* tag PTR_TO_BTF_ID allocated using bpf_obj_new.
*/
MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS),
__BPF_TYPE_FLAG_MAX,
__BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1,
};
@ -2096,22 +2114,11 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
const char *func_name,
struct btf_func_model *m);
struct bpf_kfunc_arg_meta {
u64 r0_size;
bool r0_rdonly;
int ref_obj_id;
u32 flags;
};
struct bpf_reg_state;
int btf_check_subprog_arg_match(struct bpf_verifier_env *env, int subprog,
struct bpf_reg_state *regs);
int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog,
struct bpf_reg_state *regs);
int btf_check_kfunc_arg_match(struct bpf_verifier_env *env,
const struct btf *btf, u32 func_id,
struct bpf_reg_state *regs,
struct bpf_kfunc_arg_meta *meta);
int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog,
struct bpf_reg_state *reg);
int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,
@ -2792,4 +2799,10 @@ struct bpf_key {
bool has_ref;
};
#endif /* CONFIG_KEYS */
static inline bool type_is_alloc(u32 type)
{
return type & MEM_ALLOC;
}
#endif /* _LINUX_BPF_H */

25
include/linux/bpf_verifier.h
View File

@ -223,6 +223,11 @@ struct bpf_reference_state {
* exiting a callback function.
*/
int callback_ref;
/* Mark the reference state to release the registers sharing the same id
* on bpf_spin_unlock (for nodes that we will lose ownership to but are
* safe to access inside the critical section).
*/
bool release_on_unlock;
};
/* state of the program:
@ -323,7 +328,21 @@ struct bpf_verifier_state {
u32 branches;
u32 insn_idx;
u32 curframe;
u32 active_spin_lock;
/* For every reg representing a map value or allocated object pointer,
* we consider the tuple of (ptr, id) for them to be unique in verifier
* context and conside them to not alias each other for the purposes of
* tracking lock state.
*/
struct {
/* This can either be reg->map_ptr or reg->btf. If ptr is NULL,
* there's no active lock held, and other fields have no
* meaning. If non-NULL, it indicates that a lock is held and
* id member has the reg->id of the register which can be >= 0.
*/
void *ptr;
/* This will be reg->id */
u32 id;
} active_lock;
bool speculative;
/* first and last insn idx of this verifier state */
@ -419,6 +438,8 @@ struct bpf_insn_aux_data {
*/
struct bpf_loop_inline_state loop_inline_state;
};
u64 obj_new_size; /* remember the size of type passed to bpf_obj_new to rewrite R1 */
struct btf_struct_meta *kptr_struct_meta;
u64 map_key_state; /* constant (32 bit) key tracking for maps */
int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
@ -589,8 +610,6 @@ int check_ptr_off_reg(struct bpf_verifier_env *env,
int check_func_arg_reg_off(struct bpf_verifier_env *env,
const struct bpf_reg_state *reg, int regno,
enum bpf_arg_type arg_type);
int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
u32 regno);
int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
u32 regno, u32 mem_size);
bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env,

69
include/linux/btf.h
View File

@ -6,6 +6,8 @@
#include <linux/types.h>
#include <linux/bpfptr.h>
#include <linux/bsearch.h>
#include <linux/btf_ids.h>
#include <uapi/linux/btf.h>
#include <uapi/linux/bpf.h>
@ -78,6 +80,17 @@ struct btf_id_dtor_kfunc {
u32 kfunc_btf_id;
};
struct btf_struct_meta {
u32 btf_id;
struct btf_record *record;
struct btf_field_offs *field_offs;
};
struct btf_struct_metas {
u32 cnt;
struct btf_struct_meta types[];
};
typedef void (*btf_dtor_kfunc_t)(void *);
extern const struct file_operations btf_fops;
@ -165,6 +178,7 @@ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t);
int btf_find_timer(const struct btf *btf, const struct btf_type *t);
struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
u32 field_mask, u32 value_size);
int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec);
struct btf_field_offs *btf_parse_field_offs(struct btf_record *rec);
bool btf_type_is_void(const struct btf_type *t);
s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind);
@ -324,6 +338,16 @@ static inline bool btf_type_is_struct(const struct btf_type *t)
return kind == BTF_KIND_STRUCT || kind == BTF_KIND_UNION;
}
static inline bool __btf_type_is_struct(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT;
}
static inline bool btf_type_is_array(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
}
static inline u16 btf_type_vlen(const struct btf_type *t)
{
return BTF_INFO_VLEN(t->info);
@ -408,9 +432,27 @@ static inline struct btf_param *btf_params(const struct btf_type *t)
return (struct btf_param *)(t + 1);
}
#ifdef CONFIG_BPF_SYSCALL
struct bpf_prog;
static inline int btf_id_cmp_func(const void *a, const void *b)
{
const int *pa = a, *pb = b;
return *pa - *pb;
}
static inline bool btf_id_set_contains(const struct btf_id_set *set, u32 id)
{
return bsearch(&id, set->ids, set->cnt, sizeof(u32), btf_id_cmp_func) != NULL;
}
static inline void *btf_id_set8_contains(const struct btf_id_set8 *set, u32 id)
{
return bsearch(&id, set->pairs, set->cnt, sizeof(set->pairs[0]), btf_id_cmp_func);
}
struct bpf_prog;
struct bpf_verifier_log;
#ifdef CONFIG_BPF_SYSCALL
const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
const char *btf_name_by_offset(const struct btf *btf, u32 offset);
struct btf *btf_parse_vmlinux(void);
@ -423,6 +465,13 @@ int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id);
int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
struct module *owner);
struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id);
const struct btf_member *
btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg);
bool btf_types_are_same(const struct btf *btf1, u32 id1,
const struct btf *btf2, u32 id2);
#else
static inline const struct btf_type *btf_type_by_id(const struct btf *btf,
u32 type_id)
@ -454,6 +503,22 @@ static inline int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dt
{
return 0;
}
static inline struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id)
{
return NULL;
}
static inline const struct btf_member *
btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg)
{
return NULL;
}
static inline bool btf_types_are_same(const struct btf *btf1, u32 id1,
const struct btf *btf2, u32 id2)
{
return false;
}
#endif
static inline bool btf_type_is_struct_ptr(struct btf *btf, const struct btf_type *t)

1
kernel/bpf/arraymap.c
View File

@ -430,7 +430,6 @@ static void array_map_free(struct bpf_map *map)
for (i = 0; i < array->map.max_entries; i++)
bpf_obj_free_fields(map->record, array_map_elem_ptr(array, i));
}
bpf_map_free_record(map);
}
if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY)

656
kernel/bpf/btf.c
View File

@ -237,6 +237,7 @@ struct btf {
struct rcu_head rcu;
struct btf_kfunc_set_tab *kfunc_set_tab;
struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
struct btf_struct_metas *struct_meta_tab;
/* split BTF support */
struct btf *base_btf;
@ -477,16 +478,6 @@ static bool btf_type_nosize_or_null(const struct btf_type *t)
return !t || btf_type_nosize(t);
}
static bool __btf_type_is_struct(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT;
}
static bool btf_type_is_array(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
}
static bool btf_type_is_datasec(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
@ -1642,8 +1633,30 @@ static void btf_free_dtor_kfunc_tab(struct btf *btf)
btf->dtor_kfunc_tab = NULL;
}
static void btf_struct_metas_free(struct btf_struct_metas *tab)
{
int i;
if (!tab)
return;
for (i = 0; i < tab->cnt; i++) {
btf_record_free(tab->types[i].record);
kfree(tab->types[i].field_offs);
}
kfree(tab);
}
static void btf_free_struct_meta_tab(struct btf *btf)
{
struct btf_struct_metas *tab = btf->struct_meta_tab;
btf_struct_metas_free(tab);
btf->struct_meta_tab = NULL;
}
static void btf_free(struct btf *btf)
{
btf_free_struct_meta_tab(btf);
btf_free_dtor_kfunc_tab(btf);
btf_free_kfunc_set_tab(btf);
kvfree(btf->types);
@ -3353,6 +3366,12 @@ static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask,
goto end;
}
}
if (field_mask & BPF_LIST_NODE) {
if (!strcmp(name, "bpf_list_node")) {
type = BPF_LIST_NODE;
goto end;
}
}
/* Only return BPF_KPTR when all other types with matchable names fail */
if (field_mask & BPF_KPTR) {
type = BPF_KPTR_REF;
@ -3396,6 +3415,7 @@ static int btf_find_struct_field(const struct btf *btf,
switch (field_type) {
case BPF_SPIN_LOCK:
case BPF_TIMER:
case BPF_LIST_NODE:
ret = btf_find_struct(btf, member_type, off, sz, field_type,
idx < info_cnt ? &info[idx] : &tmp);
if (ret < 0)
@ -3456,6 +3476,7 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
switch (field_type) {
case BPF_SPIN_LOCK:
case BPF_TIMER:
case BPF_LIST_NODE:
ret = btf_find_struct(btf, var_type, off, sz, field_type,
idx < info_cnt ? &info[idx] : &tmp);
if (ret < 0)
@ -3627,6 +3648,9 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
return NULL;
cnt = ret;
/* This needs to be kzalloc to zero out padding and unused fields, see
* comment in btf_record_equal.
*/
rec = kzalloc(offsetof(struct btf_record, fields[cnt]), GFP_KERNEL | __GFP_NOWARN);
if (!rec)
return ERR_PTR(-ENOMEM);
@ -3671,6 +3695,8 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
if (ret < 0)
goto end;
break;
case BPF_LIST_NODE:
break;
default:
ret = -EFAULT;
goto end;
@ -3690,6 +3716,67 @@ end:
return ERR_PTR(ret);
}
int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)
{
int i;
/* There are two owning types, kptr_ref and bpf_list_head. The former
* only supports storing kernel types, which can never store references
* to program allocated local types, atleast not yet. Hence we only need
* to ensure that bpf_list_head ownership does not form cycles.
*/
if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & BPF_LIST_HEAD))
return 0;
for (i = 0; i < rec->cnt; i++) {
struct btf_struct_meta *meta;
u32 btf_id;
if (!(rec->fields[i].type & BPF_LIST_HEAD))
continue;
btf_id = rec->fields[i].list_head.value_btf_id;
meta = btf_find_struct_meta(btf, btf_id);
if (!meta)
return -EFAULT;
rec->fields[i].list_head.value_rec = meta->record;
if (!(rec->field_mask & BPF_LIST_NODE))
continue;
/* We need to ensure ownership acyclicity among all types. The
* proper way to do it would be to topologically sort all BTF
* IDs based on the ownership edges, since there can be multiple
* bpf_list_head in a type. Instead, we use the following
* reasoning:
*
* - A type can only be owned by another type in user BTF if it
* has a bpf_list_node.
* - A type can only _own_ another type in user BTF if it has a
* bpf_list_head.
*
* We ensure that if a type has both bpf_list_head and
* bpf_list_node, its element types cannot be owning types.
*
* To ensure acyclicity:
*
* When A only has bpf_list_head, ownership chain can be:
* A -> B -> C
* Where:
* - B has both bpf_list_head and bpf_list_node.
* - C only has bpf_list_node.
*
* When A has both bpf_list_head and bpf_list_node, some other
* type already owns it in the BTF domain, hence it can not own
* another owning type through any of the bpf_list_head edges.
* A -> B
* Where:
* - B only has bpf_list_node.
*/
if (meta->record->field_mask & BPF_LIST_HEAD)
return -ELOOP;
}
return 0;
}
static int btf_field_offs_cmp(const void *_a, const void *_b, const void *priv)
{
const u32 a = *(const u32 *)_a;
@ -5141,6 +5228,119 @@ static int btf_parse_hdr(struct btf_verifier_env *env)
return btf_check_sec_info(env, btf_data_size);
}
static const char *alloc_obj_fields[] = {
"bpf_spin_lock",
"bpf_list_head",
"bpf_list_node",
};
static struct btf_struct_metas *
btf_parse_struct_metas(struct bpf_verifier_log *log, struct btf *btf)
{
union {
struct btf_id_set set;
struct {
u32 _cnt;
u32 _ids[ARRAY_SIZE(alloc_obj_fields)];
} _arr;
} aof;
struct btf_struct_metas *tab = NULL;
int i, n, id, ret;
BUILD_BUG_ON(offsetof(struct btf_id_set, cnt) != 0);
BUILD_BUG_ON(sizeof(struct btf_id_set) != sizeof(u32));
memset(&aof, 0, sizeof(aof));
for (i = 0; i < ARRAY_SIZE(alloc_obj_fields); i++) {
/* Try to find whether this special type exists in user BTF, and
* if so remember its ID so we can easily find it among members
* of structs that we iterate in the next loop.
*/
id = btf_find_by_name_kind(btf, alloc_obj_fields[i], BTF_KIND_STRUCT);
if (id < 0)
continue;
aof.set.ids[aof.set.cnt++] = id;
}
if (!aof.set.cnt)
return NULL;
sort(&aof.set.ids, aof.set.cnt, sizeof(aof.set.ids[0]), btf_id_cmp_func, NULL);
n = btf_nr_types(btf);
for (i = 1; i < n; i++) {
struct btf_struct_metas *new_tab;
const struct btf_member *member;
struct btf_field_offs *foffs;
struct btf_struct_meta *type;
struct btf_record *record;
const struct btf_type *t;
int j, tab_cnt;
t = btf_type_by_id(btf, i);
if (!t) {
ret = -EINVAL;
goto free;
}
if (!__btf_type_is_struct(t))
continue;
cond_resched();
for_each_member(j, t, member) {
if (btf_id_set_contains(&aof.set, member->type))
goto parse;
}
continue;
parse:
tab_cnt = tab ? tab->cnt : 0;
new_tab = krealloc(tab, offsetof(struct btf_struct_metas, types[tab_cnt + 1]),
GFP_KERNEL | __GFP_NOWARN);
if (!new_tab) {
ret = -ENOMEM;
goto free;
}
if (!tab)
new_tab->cnt = 0;
tab = new_tab;
type = &tab->types[tab->cnt];
type->btf_id = i;
record = btf_parse_fields(btf, t, BPF_SPIN_LOCK | BPF_LIST_HEAD | BPF_LIST_NODE, t->size);
/* The record cannot be unset, treat it as an error if so */
if (IS_ERR_OR_NULL(record)) {
ret = PTR_ERR_OR_ZERO(record) ?: -EFAULT;
goto free;
}
foffs = btf_parse_field_offs(record);
/* We need the field_offs to be valid for a valid record,
* either both should be set or both should be unset.
*/
if (IS_ERR_OR_NULL(foffs)) {
btf_record_free(record);
ret = -EFAULT;
goto free;
}
type->record = record;
type->field_offs = foffs;
tab->cnt++;
}
return tab;
free:
btf_struct_metas_free(tab);
return ERR_PTR(ret);
}
struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id)
{
struct btf_struct_metas *tab;
BUILD_BUG_ON(offsetof(struct btf_struct_meta, btf_id) != 0);
tab = btf->struct_meta_tab;
if (!tab)
return NULL;
return bsearch(&btf_id, tab->types, tab->cnt, sizeof(tab->types[0]), btf_id_cmp_func);
}
static int btf_check_type_tags(struct btf_verifier_env *env,
struct btf *btf, int start_id)
{
@ -5191,6 +5391,7 @@ static int btf_check_type_tags(struct btf_verifier_env *env,
static struct btf *btf_parse(bpfptr_t btf_data, u32 btf_data_size,
u32 log_level, char __user *log_ubuf, u32 log_size)
{
struct btf_struct_metas *struct_meta_tab;
struct btf_verifier_env *env = NULL;
struct bpf_verifier_log *log;
struct btf *btf = NULL;
@ -5259,15 +5460,34 @@ static struct btf *btf_parse(bpfptr_t btf_data, u32 btf_data_size,
if (err)
goto errout;
struct_meta_tab = btf_parse_struct_metas(log, btf);
if (IS_ERR(struct_meta_tab)) {
err = PTR_ERR(struct_meta_tab);
goto errout;
}
btf->struct_meta_tab = struct_meta_tab;
if (struct_meta_tab) {
int i;
for (i = 0; i < struct_meta_tab->cnt; i++) {
err = btf_check_and_fixup_fields(btf, struct_meta_tab->types[i].record);
if (err < 0)
goto errout_meta;
}
}
if (log->level && bpf_verifier_log_full(log)) {
err = -ENOSPC;
goto errout;
goto errout_meta;
}
btf_verifier_env_free(env);
refcount_set(&btf->refcnt, 1);
return btf;
errout_meta:
btf_free_struct_meta_tab(btf);
errout:
btf_verifier_env_free(env);
if (btf)
@ -5309,7 +5529,7 @@ static u8 bpf_ctx_convert_map[] = {
#undef BPF_MAP_TYPE
#undef BPF_LINK_TYPE
static const struct btf_member *
const struct btf_member *
btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg)
@ -6028,12 +6248,39 @@ int btf_struct_access(struct bpf_verifier_log *log,
u32 id = reg->btf_id;
int err;
while (type_is_alloc(reg->type)) {
struct btf_struct_meta *meta;
struct btf_record *rec;
int i;
meta = btf_find_struct_meta(btf, id);
if (!meta)
break;
rec = meta->record;
for (i = 0; i < rec->cnt; i++) {
struct btf_field *field = &rec->fields[i];
u32 offset = field->offset;
if (off < offset + btf_field_type_size(field->type) && offset < off + size) {
bpf_log(log,
"direct access to %s is disallowed\n",
btf_field_type_name(field->type));
return -EACCES;
}
}
break;
}
t = btf_type_by_id(btf, id);
do {
err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag);
switch (err) {
case WALK_PTR:
/* For local types, the destination register cannot
* become a pointer again.
*/
if (type_is_alloc(reg->type))
return SCALAR_VALUE;
/* If we found the pointer or scalar on t+off,
* we're done.
*/
@ -6068,8 +6315,8 @@ int btf_struct_access(struct bpf_verifier_log *log,
* end up with two different module BTFs, but IDs point to the common type in
* vmlinux BTF.
*/
static bool btf_types_are_same(const struct btf *btf1, u32 id1,
const struct btf *btf2, u32 id2)
bool btf_types_are_same(const struct btf *btf1, u32 id1,
const struct btf *btf2, u32 id2)
{
if (id1 != id2)
return false;
@ -6351,122 +6598,19 @@ int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *pr
return btf_check_func_type_match(log, btf1, t1, btf2, t2);
}
static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = {
#ifdef CONFIG_NET
[PTR_TO_SOCKET] = &btf_sock_ids[BTF_SOCK_TYPE_SOCK],
[PTR_TO_SOCK_COMMON] = &btf_sock_ids[BTF_SOCK_TYPE_SOCK_COMMON],
[PTR_TO_TCP_SOCK] = &btf_sock_ids[BTF_SOCK_TYPE_TCP],
#endif
};
/* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
static bool __btf_type_is_scalar_struct(struct bpf_verifier_log *log,
const struct btf *btf,
const struct btf_type *t, int rec)
{
const struct btf_type *member_type;
const struct btf_member *member;
u32 i;
if (!btf_type_is_struct(t))
return false;
for_each_member(i, t, member) {
const struct btf_array *array;
member_type = btf_type_skip_modifiers(btf, member->type, NULL);
if (btf_type_is_struct(member_type)) {
if (rec >= 3) {
bpf_log(log, "max struct nesting depth exceeded\n");
return false;
}
if (!__btf_type_is_scalar_struct(log, btf, member_type, rec + 1))
return false;
continue;
}
if (btf_type_is_array(member_type)) {
array = btf_type_array(member_type);
if (!array->nelems)
return false;
member_type = btf_type_skip_modifiers(btf, array->type, NULL);
if (!btf_type_is_scalar(member_type))
return false;
continue;
}
if (!btf_type_is_scalar(member_type))
return false;
}
return true;
}
static bool is_kfunc_arg_mem_size(const struct btf *btf,
const struct btf_param *arg,
const struct bpf_reg_state *reg)
{
int len, sfx_len = sizeof("__sz") - 1;
const struct btf_type *t;
const char *param_name;
t = btf_type_skip_modifiers(btf, arg->type, NULL);
if (!btf_type_is_scalar(t) || reg->type != SCALAR_VALUE)
return false;
/* In the future, this can be ported to use BTF tagging */
param_name = btf_name_by_offset(btf, arg->name_off);
if (str_is_empty(param_name))
return false;
len = strlen(param_name);
if (len < sfx_len)
return false;
param_name += len - sfx_len;
if (strncmp(param_name, "__sz", sfx_len))
return false;
return true;
}
static bool btf_is_kfunc_arg_mem_size(const struct btf *btf,
const struct btf_param *arg,
const struct bpf_reg_state *reg,
const char *name)
{
int len, target_len = strlen(name);
const struct btf_type *t;
const char *param_name;
t = btf_type_skip_modifiers(btf, arg->type, NULL);
if (!btf_type_is_scalar(t) || reg->type != SCALAR_VALUE)
return false;
param_name = btf_name_by_offset(btf, arg->name_off);
if (str_is_empty(param_name))
return false;
len = strlen(param_name);
if (len != target_len)
return false;
if (strcmp(param_name, name))
return false;
return true;
}
static int btf_check_func_arg_match(struct bpf_verifier_env *env,
const struct btf *btf, u32 func_id,
struct bpf_reg_state *regs,
bool ptr_to_mem_ok,
struct bpf_kfunc_arg_meta *kfunc_meta,
bool processing_call)
{
enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
bool rel = false, kptr_get = false, trusted_args = false;
bool sleepable = false;
struct bpf_verifier_log *log = &env->log;
u32 i, nargs, ref_id, ref_obj_id = 0;
bool is_kfunc = btf_is_kernel(btf);
const char *func_name, *ref_tname;
const struct btf_type *t, *ref_t;
const struct btf_param *args;
int ref_regno = 0, ret;
u32 i, nargs, ref_id;
int ret;
t = btf_type_by_id(btf, func_id);
if (!t || !btf_type_is_func(t)) {
@ -6492,14 +6636,6 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,
return -EINVAL;
}
if (is_kfunc && kfunc_meta) {
/* Only kfunc can be release func */
rel = kfunc_meta->flags & KF_RELEASE;
kptr_get = kfunc_meta->flags & KF_KPTR_GET;
trusted_args = kfunc_meta->flags & KF_TRUSTED_ARGS;
sleepable = kfunc_meta->flags & KF_SLEEPABLE;
}
/* check that BTF function arguments match actual types that the
* verifier sees.
*/
@ -6507,42 +6643,9 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,
enum bpf_arg_type arg_type = ARG_DONTCARE;
u32 regno = i + 1;
struct bpf_reg_state *reg = &regs[regno];
bool obj_ptr = false;
t = btf_type_skip_modifiers(btf, args[i].type, NULL);
if (btf_type_is_scalar(t)) {
if (is_kfunc && kfunc_meta) {
bool is_buf_size = false;
/* check for any const scalar parameter of name "rdonly_buf_size"
* or "rdwr_buf_size"
*/
if (btf_is_kfunc_arg_mem_size(btf, &args[i], reg,
"rdonly_buf_size")) {
kfunc_meta->r0_rdonly = true;
is_buf_size = true;
} else if (btf_is_kfunc_arg_mem_size(btf, &args[i], reg,
"rdwr_buf_size"))
is_buf_size = true;
if (is_buf_size) {
if (kfunc_meta->r0_size) {
bpf_log(log, "2 or more rdonly/rdwr_buf_size parameters for kfunc");
return -EINVAL;
}
if (!tnum_is_const(reg->var_off)) {
bpf_log(log, "R%d is not a const\n", regno);
return -EINVAL;
}
kfunc_meta->r0_size = reg->var_off.value;
ret = mark_chain_precision(env, regno);
if (ret)
return ret;
}
}
if (reg->type == SCALAR_VALUE)
continue;
bpf_log(log, "R%d is not a scalar\n", regno);
@ -6555,88 +6658,14 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,
return -EINVAL;
}
/* These register types have special constraints wrt ref_obj_id
* and offset checks. The rest of trusted args don't.
*/
obj_ptr = reg->type == PTR_TO_CTX || reg->type == PTR_TO_BTF_ID ||
reg2btf_ids[base_type(reg->type)];
/* Check if argument must be a referenced pointer, args + i has
* been verified to be a pointer (after skipping modifiers).
* PTR_TO_CTX is ok without having non-zero ref_obj_id.
*/
if (is_kfunc && trusted_args && (obj_ptr && reg->type != PTR_TO_CTX) && !reg->ref_obj_id) {
bpf_log(log, "R%d must be referenced\n", regno);
return -EINVAL;
}
ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
ref_tname = btf_name_by_offset(btf, ref_t->name_off);
/* Trusted args have the same offset checks as release arguments */
if ((trusted_args && obj_ptr) || (rel && reg->ref_obj_id))
arg_type |= OBJ_RELEASE;
ret = check_func_arg_reg_off(env, reg, regno, arg_type);
if (ret < 0)
return ret;
if (is_kfunc && reg->ref_obj_id) {
/* Ensure only one argument is referenced PTR_TO_BTF_ID */
if (ref_obj_id) {
bpf_log(log, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
regno, reg->ref_obj_id, ref_obj_id);
return -EFAULT;
}
ref_regno = regno;
ref_obj_id = reg->ref_obj_id;
}
/* kptr_get is only true for kfunc */
if (i == 0 && kptr_get) {
struct btf_field *kptr_field;
if (reg->type != PTR_TO_MAP_VALUE) {
bpf_log(log, "arg#0 expected pointer to map value\n");
return -EINVAL;
}
/* check_func_arg_reg_off allows var_off for
* PTR_TO_MAP_VALUE, but we need fixed offset to find
* off_desc.
*/
if (!tnum_is_const(reg->var_off)) {
bpf_log(log, "arg#0 must have constant offset\n");
return -EINVAL;
}
kptr_field = btf_record_find(reg->map_ptr->record, reg->off + reg->var_off.value, BPF_KPTR);
if (!kptr_field || kptr_field->type != BPF_KPTR_REF) {
bpf_log(log, "arg#0 no referenced kptr at map value offset=%llu\n",
reg->off + reg->var_off.value);
return -EINVAL;
}
if (!btf_type_is_ptr(ref_t)) {
bpf_log(log, "arg#0 BTF type must be a double pointer\n");
return -EINVAL;
}
ref_t = btf_type_skip_modifiers(btf, ref_t->type, &ref_id);
ref_tname = btf_name_by_offset(btf, ref_t->name_off);
if (!btf_type_is_struct(ref_t)) {
bpf_log(log, "kernel function %s args#%d pointer type %s %s is not supported\n",
func_name, i, btf_type_str(ref_t), ref_tname);
return -EINVAL;
}
if (!btf_struct_ids_match(log, btf, ref_id, 0, kptr_field->kptr.btf,
kptr_field->kptr.btf_id, true)) {
bpf_log(log, "kernel function %s args#%d expected pointer to %s %s\n",
func_name, i, btf_type_str(ref_t), ref_tname);
return -EINVAL;
}
/* rest of the arguments can be anything, like normal kfunc */
} else if (btf_get_prog_ctx_type(log, btf, t, prog_type, i)) {
if (btf_get_prog_ctx_type(log, btf, t, prog_type, i)) {
/* If function expects ctx type in BTF check that caller
* is passing PTR_TO_CTX.
*/
@ -6646,109 +6675,10 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,
i, btf_type_str(t));
return -EINVAL;
}
} else if (is_kfunc && (reg->type == PTR_TO_BTF_ID ||
(reg2btf_ids[base_type(reg->type)] && !type_flag(reg->type)))) {
const struct btf_type *reg_ref_t;
const struct btf *reg_btf;
const char *reg_ref_tname;
u32 reg_ref_id;
if (!btf_type_is_struct(ref_t)) {
bpf_log(log, "kernel function %s args#%d pointer type %s %s is not supported\n",
func_name, i, btf_type_str(ref_t),
ref_tname);
return -EINVAL;
}
if (reg->type == PTR_TO_BTF_ID) {
reg_btf = reg->btf;
reg_ref_id = reg->btf_id;
} else {
reg_btf = btf_vmlinux;
reg_ref_id = *reg2btf_ids[base_type(reg->type)];
}
reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id,
&reg_ref_id);
reg_ref_tname = btf_name_by_offset(reg_btf,
reg_ref_t->name_off);
if (!btf_struct_ids_match(log, reg_btf, reg_ref_id,
reg->off, btf, ref_id,
trusted_args || (rel && reg->ref_obj_id))) {
bpf_log(log, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n",
func_name, i,
btf_type_str(ref_t), ref_tname,
regno, btf_type_str(reg_ref_t),
reg_ref_tname);
return -EINVAL;
}
} else if (ptr_to_mem_ok && processing_call) {
const struct btf_type *resolve_ret;
u32 type_size;
if (is_kfunc) {
bool arg_mem_size = i + 1 < nargs && is_kfunc_arg_mem_size(btf, &args[i + 1], &regs[regno + 1]);
bool arg_dynptr = btf_type_is_struct(ref_t) &&
!strcmp(ref_tname,
stringify_struct(bpf_dynptr_kern));
/* Permit pointer to mem, but only when argument
* type is pointer to scalar, or struct composed
* (recursively) of scalars.
* When arg_mem_size is true, the pointer can be
* void *.
* Also permit initialized local dynamic pointers.
*/
if (!btf_type_is_scalar(ref_t) &&
!__btf_type_is_scalar_struct(log, btf, ref_t, 0) &&
!arg_dynptr &&
(arg_mem_size ? !btf_type_is_void(ref_t) : 1)) {
bpf_log(log,
"arg#%d pointer type %s %s must point to %sscalar, or struct with scalar\n",
i, btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : "");
return -EINVAL;
}
if (arg_dynptr) {
if (reg->type != PTR_TO_STACK) {
bpf_log(log, "arg#%d pointer type %s %s not to stack\n",
i, btf_type_str(ref_t),
ref_tname);
return -EINVAL;
}
if (!is_dynptr_reg_valid_init(env, reg)) {
bpf_log(log,
"arg#%d pointer type %s %s must be valid and initialized\n",
i, btf_type_str(ref_t),
ref_tname);
return -EINVAL;
}
if (!is_dynptr_type_expected(env, reg,
ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL)) {
bpf_log(log,
"arg#%d pointer type %s %s points to unsupported dynamic pointer type\n",
i, btf_type_str(ref_t),
ref_tname);
return -EINVAL;
}
continue;
}
/* Check for mem, len pair */
if (arg_mem_size) {
if (check_kfunc_mem_size_reg(env, &regs[regno + 1], regno + 1)) {
bpf_log(log, "arg#%d arg#%d memory, len pair leads to invalid memory access\n",
i, i + 1);
return -EINVAL;
}
i++;
continue;
}
}
resolve_ret = btf_resolve_size(btf, ref_t, &type_size);
if (IS_ERR(resolve_ret)) {
bpf_log(log,
@ -6761,36 +6691,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,
if (check_mem_reg(env, reg, regno, type_size))
return -EINVAL;
} else {
bpf_log(log, "reg type unsupported for arg#%d %sfunction %s#%d\n", i,
is_kfunc ? "kernel " : "", func_name, func_id);
bpf_log(log, "reg type unsupported for arg#%d function %s#%d\n", i,
func_name, func_id);
return -EINVAL;
}
}
/* Either both are set, or neither */
WARN_ON_ONCE((ref_obj_id && !ref_regno) || (!ref_obj_id && ref_regno));
/* We already made sure ref_obj_id is set only for one argument. We do
* allow (!rel && ref_obj_id), so that passing such referenced
* PTR_TO_BTF_ID to other kfuncs works. Note that rel is only true when
* is_kfunc is true.
*/
if (rel && !ref_obj_id) {
bpf_log(log, "release kernel function %s expects refcounted PTR_TO_BTF_ID\n",
func_name);
return -EINVAL;
}
if (sleepable && !env->prog->aux->sleepable) {
bpf_log(log, "kernel function %s is sleepable but the program is not\n",
func_name);
return -EINVAL;
}
if (kfunc_meta && ref_obj_id)
kfunc_meta->ref_obj_id = ref_obj_id;
/* returns argument register number > 0 in case of reference release kfunc */
return rel ? ref_regno : 0;
return 0;
}
/* Compare BTF of a function declaration with given bpf_reg_state.
@ -6820,7 +6727,7 @@ int btf_check_subprog_arg_match(struct bpf_verifier_env *env, int subprog,
return -EINVAL;
is_global = prog->aux->func_info_aux[subprog].linkage == BTF_FUNC_GLOBAL;
err = btf_check_func_arg_match(env, btf, btf_id, regs, is_global, NULL, false);
err = btf_check_func_arg_match(env, btf, btf_id, regs, is_global, false);
/* Compiler optimizations can remove arguments from static functions
* or mismatched type can be passed into a global function.
@ -6863,7 +6770,7 @@ int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog,
return -EINVAL;
is_global = prog->aux->func_info_aux[subprog].linkage == BTF_FUNC_GLOBAL;
err = btf_check_func_arg_match(env, btf, btf_id, regs, is_global, NULL, true);
err = btf_check_func_arg_match(env, btf, btf_id, regs, is_global, true);
/* Compiler optimizations can remove arguments from static functions
* or mismatched type can be passed into a global function.
@ -6874,14 +6781,6 @@ int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog,
return err;
}
int btf_check_kfunc_arg_match(struct bpf_verifier_env *env,
const struct btf *btf, u32 func_id,
struct bpf_reg_state *regs,
struct bpf_kfunc_arg_meta *meta)
{
return btf_check_func_arg_match(env, btf, func_id, regs, true, meta, true);
}
/* Convert BTF of a function into bpf_reg_state if possible
* Returns:
* EFAULT - there is a verifier bug. Abort verification.
@ -7264,23 +7163,6 @@ bool btf_is_module(const struct btf *btf)
return btf->kernel_btf && strcmp(btf->name, "vmlinux") != 0;
}
static int btf_id_cmp_func(const void *a, const void *b)
{
const int *pa = a, *pb = b;
return *pa - *pb;
}
bool btf_id_set_contains(const struct btf_id_set *set, u32 id)
{
return bsearch(&id, set->ids, set->cnt, sizeof(u32), btf_id_cmp_func) != NULL;
}
static void *btf_id_set8_contains(const struct btf_id_set8 *set, u32 id)
{
return bsearch(&id, set->pairs, set->cnt, sizeof(set->pairs[0]), btf_id_cmp_func);
}
enum {
BTF_MODULE_F_LIVE = (1 << 0),
};

16
kernel/bpf/core.c
View File

@ -34,6 +34,7 @@
#include <linux/log2.h>
#include <linux/bpf_verifier.h>
#include <linux/nodemask.h>
#include <linux/bpf_mem_alloc.h>
#include <asm/barrier.h>
#include <asm/unaligned.h>
@ -60,6 +61,9 @@
#define CTX regs[BPF_REG_CTX]
#define IMM insn->imm
struct bpf_mem_alloc bpf_global_ma;
bool bpf_global_ma_set;
/* No hurry in this branch
*
* Exported for the bpf jit load helper.
@ -2746,6 +2750,18 @@ int __weak bpf_arch_text_invalidate(void *dst, size_t len)
return -ENOTSUPP;
}
#ifdef CONFIG_BPF_SYSCALL
static int __init bpf_global_ma_init(void)
{
int ret;
ret = bpf_mem_alloc_init(&bpf_global_ma, 0, false);
bpf_global_ma_set = !ret;
return ret;
}
late_initcall(bpf_global_ma_init);
#endif
DEFINE_STATIC_KEY_FALSE(bpf_stats_enabled_key);
EXPORT_SYMBOL(bpf_stats_enabled_key);

1
kernel/bpf/hashtab.c
View File

@ -1511,7 +1511,6 @@ static void htab_map_free(struct bpf_map *map)
prealloc_destroy(htab);
}
bpf_map_free_record(map);
free_percpu(htab->extra_elems);
bpf_map_area_free(htab->buckets);
bpf_mem_alloc_destroy(&htab->pcpu_ma);

113
kernel/bpf/helpers.c
View File

@ -19,6 +19,7 @@
#include <linux/proc_ns.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include "../../lib/kstrtox.h"
@ -336,6 +337,7 @@ const struct bpf_func_proto bpf_spin_lock_proto = {
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
@ -358,6 +360,7 @@ const struct bpf_func_proto bpf_spin_unlock_proto = {
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
@ -1733,25 +1736,121 @@ unlock:
obj -= field->list_head.node_offset;
head = head->next;
/* TODO: Rework later */
kfree(obj);
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
bpf_obj_free_fields(field->list_head.value_rec, obj);
/* bpf_mem_free requires migrate_disable(), since we can be
* called from map free path as well apart from BPF program (as
* part of map ops doing bpf_obj_free_fields).
*/
migrate_disable();
bpf_mem_free(&bpf_global_ma, obj);
migrate_enable();
}
}
BTF_SET8_START(tracing_btf_ids)
__diag_push();
__diag_ignore_all("-Wmissing-prototypes",
"Global functions as their definitions will be in vmlinux BTF");
void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
struct btf_struct_meta *meta = meta__ign;
u64 size = local_type_id__k;
void *p;
if (unlikely(!bpf_global_ma_set))
return NULL;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->field_offs, p);
return p;
}
void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
struct btf_struct_meta *meta = meta__ign;
void *p = p__alloc;
if (meta)
bpf_obj_free_fields(meta->record, p);
bpf_mem_free(&bpf_global_ma, p);
}
static void __bpf_list_add(struct bpf_list_node *node, struct bpf_list_head *head, bool tail)
{
struct list_head *n = (void *)node, *h = (void *)head;
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (unlikely(!n->next))
INIT_LIST_HEAD(n);
tail ? list_add_tail(n, h) : list_add(n, h);
}
void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node)
{
return __bpf_list_add(node, head, false);
}
void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node)
{
return __bpf_list_add(node, head, true);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
list_del_init(n);
return (struct bpf_list_node *)n;
}
struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__diag_pop();
BTF_SET8_START(generic_btf_ids)
#ifdef CONFIG_KEXEC_CORE
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_SET8_END(tracing_btf_ids)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_list_push_front)
BTF_ID_FLAGS(func, bpf_list_push_back)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_SET8_END(generic_btf_ids)
static const struct btf_kfunc_id_set tracing_kfunc_set = {
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &tracing_btf_ids,
.set = &generic_btf_ids,
};
static int __init kfunc_init(void)
{
return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &tracing_kfunc_set);
int ret;
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
if (ret)
return ret;
return register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
}
late_initcall(kfunc_init);

48
kernel/bpf/map_in_map.c
View File

@ -12,6 +12,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
struct bpf_map *inner_map, *inner_map_meta;
u32 inner_map_meta_size;
struct fd f;
int ret;
f = fdget(inner_map_ufd);
inner_map = __bpf_map_get(f);
@ -20,18 +21,13 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
/* Does not support >1 level map-in-map */
if (inner_map->inner_map_meta) {
fdput(f);
return ERR_PTR(-EINVAL);
ret = -EINVAL;
goto put;
}
if (!inner_map->ops->map_meta_equal) {
fdput(f);
return ERR_PTR(-ENOTSUPP);
}
if (btf_record_has_field(inner_map->record, BPF_SPIN_LOCK)) {
fdput(f);
return ERR_PTR(-ENOTSUPP);
ret = -ENOTSUPP;
goto put;
}
inner_map_meta_size = sizeof(*inner_map_meta);
@ -41,8 +37,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
inner_map_meta = kzalloc(inner_map_meta_size, GFP_USER);
if (!inner_map_meta) {
fdput(f);
return ERR_PTR(-ENOMEM);
ret = -ENOMEM;
goto put;
}
inner_map_meta->map_type = inner_map->map_type;
@ -50,15 +46,33 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
inner_map_meta->value_size = inner_map->value_size;
inner_map_meta->map_flags = inner_map->map_flags;
inner_map_meta->max_entries = inner_map->max_entries;
inner_map_meta->record = btf_record_dup(inner_map->record);
if (IS_ERR(inner_map_meta->record)) {
/* btf_record_dup returns NULL or valid pointer in case of
* invalid/empty/valid, but ERR_PTR in case of errors. During
* equality NULL or IS_ERR is equivalent.
*/
fdput(f);
return ERR_CAST(inner_map_meta->record);
ret = PTR_ERR(inner_map_meta->record);
goto free;
}
if (inner_map_meta->record) {
struct btf_field_offs *field_offs;
/* If btf_record is !IS_ERR_OR_NULL, then field_offs is always
* valid.
*/
field_offs = kmemdup(inner_map->field_offs, sizeof(*inner_map->field_offs), GFP_KERNEL | __GFP_NOWARN);
if (!field_offs) {
ret = -ENOMEM;
goto free_rec;
}
inner_map_meta->field_offs = field_offs;
}
/* Note: We must use the same BTF, as we also used btf_record_dup above
* which relies on BTF being same for both maps, as some members like
* record->fields.list_head have pointers like value_rec pointing into
* inner_map->btf.
*/
if (inner_map->btf) {
btf_get(inner_map->btf);
inner_map_meta->btf = inner_map->btf;
@ -74,10 +88,18 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
fdput(f);
return inner_map_meta;
free_rec:
btf_record_free(inner_map_meta->record);
free:
kfree(inner_map_meta);
put:
fdput(f);
return ERR_PTR(ret);
}
void bpf_map_meta_free(struct bpf_map *map_meta)
{
kfree(map_meta->field_offs);
bpf_map_free_record(map_meta);
btf_put(map_meta->btf);
kfree(map_meta);

42
kernel/bpf/syscall.c
View File

@ -537,6 +537,7 @@ void btf_record_free(struct btf_record *rec)
btf_put(rec->fields[i].kptr.btf);
break;
case BPF_LIST_HEAD:
case BPF_LIST_NODE:
/* Nothing to release for bpf_list_head */
break;
default:
@ -582,6 +583,7 @@ struct btf_record *btf_record_dup(const struct btf_record *rec)
}
break;
case BPF_LIST_HEAD:
case BPF_LIST_NODE:
/* Nothing to acquire for bpf_list_head */
break;
default:
@ -609,6 +611,20 @@ bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *r
if (rec_a->cnt != rec_b->cnt)
return false;
size = offsetof(struct btf_record, fields[rec_a->cnt]);
/* btf_parse_fields uses kzalloc to allocate a btf_record, so unused
* members are zeroed out. So memcmp is safe to do without worrying
* about padding/unused fields.
*
* While spin_lock, timer, and kptr have no relation to map BTF,
* list_head metadata is specific to map BTF, the btf and value_rec
* members in particular. btf is the map BTF, while value_rec points to
* btf_record in that map BTF.
*
* So while by default, we don't rely on the map BTF (which the records
* were parsed from) matching for both records, which is not backwards
* compatible, in case list_head is part of it, we implicitly rely on
* that by way of depending on memcmp succeeding for it.
*/
return !memcmp(rec_a, rec_b, size);
}
@ -648,6 +664,8 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj)
continue;
bpf_list_head_free(field, field_ptr, obj + rec->spin_lock_off);
break;
case BPF_LIST_NODE:
break;
default:
WARN_ON_ONCE(1);
continue;
@ -659,14 +677,24 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj)
static void bpf_map_free_deferred(struct work_struct *work)
{
struct bpf_map *map = container_of(work, struct bpf_map, work);
struct btf_field_offs *foffs = map->field_offs;
struct btf_record *rec = map->record;
security_bpf_map_free(map);
kfree(map->field_offs);
bpf_map_release_memcg(map);
/* implementation dependent freeing, map_free callback also does
* bpf_map_free_record, if needed.
*/
/* implementation dependent freeing */
map->ops->map_free(map);
/* Delay freeing of field_offs and btf_record for maps, as map_free
* callback usually needs access to them. It is better to do it here
* than require each callback to do the free itself manually.
*
* Note that the btf_record stashed in map->inner_map_meta->record was
* already freed using the map_free callback for map in map case which
* eventually calls bpf_map_free_meta, since inner_map_meta is only a
* template bpf_map struct used during verification.
*/
kfree(foffs);
btf_record_free(rec);
}
static void bpf_map_put_uref(struct bpf_map *map)
@ -1010,7 +1038,7 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
if (map->map_type != BPF_MAP_TYPE_HASH &&
map->map_type != BPF_MAP_TYPE_LRU_HASH &&
map->map_type != BPF_MAP_TYPE_ARRAY) {
return -EOPNOTSUPP;
ret = -EOPNOTSUPP;
goto free_map_tab;
}
break;
@ -1040,6 +1068,10 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
}
}
ret = btf_check_and_fixup_fields(btf, map->record);
if (ret < 0)
goto free_map_tab;
if (map->ops->map_check_btf) {
ret = map->ops->map_check_btf(map, btf, key_type, value_type);
if (ret < 0)

1220
kernel/bpf/verifier.c
View File

File diff suppressed because it is too large Load Diff

1
tools/testing/selftests/bpf/DENYLIST.aarch64
View File

@ -38,6 +38,7 @@ kprobe_multi_test/skel_api # kprobe_multi__attach unexpect
ksyms_module/libbpf # 'bpf_testmod_ksym_percpu': not found in kernel BTF
ksyms_module/lskel # test_ksyms_module_lskel__open_and_load unexpected error: -2
libbpf_get_fd_by_id_opts # test_libbpf_get_fd_by_id_opts__attach unexpected error: -524 (errno 524)
linked_list
lookup_key # test_lookup_key__attach unexpected error: -524 (errno 524)
lru_bug # lru_bug__attach unexpected error: -524 (errno 524)
modify_return # modify_return__attach failed unexpected error: -524 (errno 524)

1
tools/testing/selftests/bpf/DENYLIST.s390x
View File

@ -33,6 +33,7 @@ ksyms_module # test_ksyms_module__open_and_load unex
ksyms_module_libbpf # JIT does not support calling kernel function (kfunc)
ksyms_module_lskel # test_ksyms_module_lskel__open_and_load unexpected error: -9 (?)
libbpf_get_fd_by_id_opts # failed to attach: ERROR: strerror_r(-524)=22 (trampoline)
linked_list # JIT does not support calling kernel function (kfunc)
lookup_key # JIT does not support calling kernel function (kfunc)
lru_bug # prog 'printk': failed to auto-attach: -524
map_kptr # failed to open_and_load program: -524 (trampoline)

68
tools/testing/selftests/bpf/bpf_experimental.h Normal file
View File

@ -0,0 +1,68 @@
#ifndef __BPF_EXPERIMENTAL__
#define __BPF_EXPERIMENTAL__
#include <vmlinux.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#define __contains(name, node) __attribute__((btf_decl_tag("contains:" #name ":" #node)))
/* Description
* Allocates an object of the type represented by 'local_type_id' in
* program BTF. User may use the bpf_core_type_id_local macro to pass the
* type ID of a struct in program BTF.
*
* The 'local_type_id' parameter must be a known constant.
* The 'meta' parameter is a hidden argument that is ignored.
* Returns
* A pointer to an object of the type corresponding to the passed in
* 'local_type_id', or NULL on failure.
*/
extern void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym;
/* Convenience macro to wrap over bpf_obj_new_impl */
#define bpf_obj_new(type) ((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL))
/* Description
* Free an allocated object. All fields of the object that require
* destruction will be destructed before the storage is freed.
*
* The 'meta' parameter is a hidden argument that is ignored.
* Returns
* Void.
*/
extern void bpf_obj_drop_impl(void *kptr, void *meta) __ksym;
/* Convenience macro to wrap over bpf_obj_drop_impl */
#define bpf_obj_drop(kptr) bpf_obj_drop_impl(kptr, NULL)
/* Description
* Add a new entry to the beginning of the BPF linked list.
* Returns
* Void.
*/
extern void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node) __ksym;
/* Description
* Add a new entry to the end of the BPF linked list.
* Returns
* Void.
*/
extern void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node) __ksym;
/* Description
* Remove the entry at the beginning of the BPF linked list.
* Returns
* Pointer to bpf_list_node of deleted entry, or NULL if list is empty.
*/
extern struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) __ksym;
/* Description
* Remove the entry at the end of the BPF linked list.
* Returns
* Pointer to bpf_list_node of deleted entry, or NULL if list is empty.
*/
extern struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) __ksym;
#endif

2
tools/testing/selftests/bpf/prog_tests/kfunc_dynptr_param.c
View File

@ -22,7 +22,7 @@ static struct {
"arg#0 pointer type STRUCT bpf_dynptr_kern points to unsupported dynamic pointer type", 0},
{"not_valid_dynptr",
"arg#0 pointer type STRUCT bpf_dynptr_kern must be valid and initialized", 0},
{"not_ptr_to_stack", "arg#0 pointer type STRUCT bpf_dynptr_kern not to stack", 0},
{"not_ptr_to_stack", "arg#0 expected pointer to stack", 0},
{"dynptr_data_null", NULL, -EBADMSG},
};

747
tools/testing/selftests/bpf/prog_tests/linked_list.c Normal file
View File

@ -0,0 +1,747 @@
// SPDX-License-Identifier: GPL-2.0
#include <bpf/btf.h>
#include <test_btf.h>
#include <linux/btf.h>
#include <test_progs.h>
#include <network_helpers.h>
#include "linked_list.skel.h"
#include "linked_list_fail.skel.h"
static char log_buf[1024 * 1024];
static struct {
const char *prog_name;
const char *err_msg;
} linked_list_fail_tests[] = {
#define TEST(test, off) \
{ #test "_missing_lock_push_front", \
"bpf_spin_lock at off=" #off " must be held for bpf_list_head" }, \
{ #test "_missing_lock_push_back", \
"bpf_spin_lock at off=" #off " must be held for bpf_list_head" }, \
{ #test "_missing_lock_pop_front", \
"bpf_spin_lock at off=" #off " must be held for bpf_list_head" }, \
{ #test "_missing_lock_pop_back", \
"bpf_spin_lock at off=" #off " must be held for bpf_list_head" },
TEST(kptr, 32)
/* FIXME
TEST(global, 16)
*/
TEST(map, 0)
TEST(inner_map, 0)
#undef TEST
#define TEST(test, op) \
{ #test "_kptr_incorrect_lock_" #op, \
"held lock and object are not in the same allocation\n" \
"bpf_spin_lock at off=32 must be held for bpf_list_head" }, \
{ #test "_map_incorrect_lock_" #op, \
"held lock and object are not in the same allocation\n" \
"bpf_spin_lock at off=0 must be held for bpf_list_head" }, \
{ #test "_inner_map_incorrect_lock_" #op, \
"held lock and object are not in the same allocation\n" \
"bpf_spin_lock at off=0 must be held for bpf_list_head" },
TEST(kptr, push_front)
TEST(kptr, push_back)
TEST(kptr, pop_front)
TEST(kptr, pop_back)
TEST(map, push_front)
TEST(map, push_back)
TEST(map, pop_front)
TEST(map, pop_back)
TEST(inner_map, push_front)
TEST(inner_map, push_back)
TEST(inner_map, pop_front)
TEST(inner_map, pop_back)
#undef TEST
/* FIXME
{ "map_compat_kprobe", "tracing progs cannot use bpf_list_head yet" },
{ "map_compat_kretprobe", "tracing progs cannot use bpf_list_head yet" },
{ "map_compat_tp", "tracing progs cannot use bpf_list_head yet" },
{ "map_compat_perf", "tracing progs cannot use bpf_list_head yet" },
{ "map_compat_raw_tp", "tracing progs cannot use bpf_list_head yet" },
{ "map_compat_raw_tp_w", "tracing progs cannot use bpf_list_head yet" },
*/
{ "obj_type_id_oor", "local type ID argument must be in range [0, U32_MAX]" },
{ "obj_new_no_composite", "bpf_obj_new type ID argument must be of a struct" },
{ "obj_new_no_struct", "bpf_obj_new type ID argument must be of a struct" },
{ "obj_drop_non_zero_off", "R1 must have zero offset when passed to release func" },
{ "new_null_ret", "R0 invalid mem access 'ptr_or_null_'" },
{ "obj_new_acq", "Unreleased reference id=" },
{ "use_after_drop", "invalid mem access 'scalar'" },
{ "ptr_walk_scalar", "type=scalar expected=percpu_ptr_" },
{ "direct_read_lock", "direct access to bpf_spin_lock is disallowed" },
{ "direct_write_lock", "direct access to bpf_spin_lock is disallowed" },
{ "direct_read_head", "direct access to bpf_list_head is disallowed" },
{ "direct_write_head", "direct access to bpf_list_head is disallowed" },
{ "direct_read_node", "direct access to bpf_list_node is disallowed" },
{ "direct_write_node", "direct access to bpf_list_node is disallowed" },
/* FIXME
{ "write_after_push_front", "only read is supported" },
{ "write_after_push_back", "only read is supported" },
{ "use_after_unlock_push_front", "invalid mem access 'scalar'" },
{ "use_after_unlock_push_back", "invalid mem access 'scalar'" },
{ "double_push_front", "arg#1 expected pointer to allocated object" },
{ "double_push_back", "arg#1 expected pointer to allocated object" },
{ "no_node_value_type", "bpf_list_node not found at offset=0" },
{ "incorrect_value_type",
"operation on bpf_list_head expects arg#1 bpf_list_node at offset=0 in struct foo, "
"but arg is at offset=0 in struct bar" },
{ "incorrect_node_var_off", "variable ptr_ access var_off=(0x0; 0xffffffff) disallowed" },
{ "incorrect_node_off1", "bpf_list_node not found at offset=1" },
{ "incorrect_node_off2", "arg#1 offset=40, but expected bpf_list_node at offset=0 in struct foo" },
{ "no_head_type", "bpf_list_head not found at offset=0" },
{ "incorrect_head_var_off1", "R1 doesn't have constant offset" },
{ "incorrect_head_var_off2", "variable ptr_ access var_off=(0x0; 0xffffffff) disallowed" },
*/
{ "incorrect_head_off1", "bpf_list_head not found at offset=17" },
/* FIXME
{ "incorrect_head_off2", "bpf_list_head not found at offset=1" },
*/
{ "pop_front_off",
"15: (bf) r1 = r6 ; R1_w=ptr_or_null_foo(id=4,ref_obj_id=4,off=40,imm=0) "
"R6_w=ptr_or_null_foo(id=4,ref_obj_id=4,off=40,imm=0) refs=2,4\n"
"16: (85) call bpf_this_cpu_ptr#154\nR1 type=ptr_or_null_ expected=percpu_ptr_" },
{ "pop_back_off",
"15: (bf) r1 = r6 ; R1_w=ptr_or_null_foo(id=4,ref_obj_id=4,off=40,imm=0) "
"R6_w=ptr_or_null_foo(id=4,ref_obj_id=4,off=40,imm=0) refs=2,4\n"
"16: (85) call bpf_this_cpu_ptr#154\nR1 type=ptr_or_null_ expected=percpu_ptr_" },
};
static void test_linked_list_fail_prog(const char *prog_name, const char *err_msg)
{
LIBBPF_OPTS(bpf_object_open_opts, opts, .kernel_log_buf = log_buf,
.kernel_log_size = sizeof(log_buf),
.kernel_log_level = 1);
struct linked_list_fail *skel;
struct bpf_program *prog;
int ret;
skel = linked_list_fail__open_opts(&opts);
if (!ASSERT_OK_PTR(skel, "linked_list_fail__open_opts"))
return;
prog = bpf_object__find_program_by_name(skel->obj, prog_name);
if (!ASSERT_OK_PTR(prog, "bpf_object__find_program_by_name"))
goto end;
bpf_program__set_autoload(prog, true);
ret = linked_list_fail__load(skel);
if (!ASSERT_ERR(ret, "linked_list_fail__load must fail"))
goto end;
if (!ASSERT_OK_PTR(strstr(log_buf, err_msg), "expected error message")) {
fprintf(stderr, "Expected: %s\n", err_msg);
fprintf(stderr, "Verifier: %s\n", log_buf);
}
end:
linked_list_fail__destroy(skel);
}
static void clear_fields(struct bpf_map *map)
{
char buf[24];
int key = 0;
memset(buf, 0xff, sizeof(buf));
ASSERT_OK(bpf_map__update_elem(map, &key, sizeof(key), buf, sizeof(buf), 0), "check_and_free_fields");
}
enum {
TEST_ALL,
PUSH_POP,
PUSH_POP_MULT,
LIST_IN_LIST,
};
static void test_linked_list_success(int mode, bool leave_in_map)
{
LIBBPF_OPTS(bpf_test_run_opts, opts,
.data_in = &pkt_v4,
.data_size_in = sizeof(pkt_v4),
.repeat = 1,
);
struct linked_list *skel;
int ret;
skel = linked_list__open_and_load();
if (!ASSERT_OK_PTR(skel, "linked_list__open_and_load"))
return;
if (mode == LIST_IN_LIST)
goto lil;
if (mode == PUSH_POP_MULT)
goto ppm;
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.map_list_push_pop), &opts);
ASSERT_OK(ret, "map_list_push_pop");
ASSERT_OK(opts.retval, "map_list_push_pop retval");
if (!leave_in_map)
clear_fields(skel->maps.array_map);
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.inner_map_list_push_pop), &opts);
ASSERT_OK(ret, "inner_map_list_push_pop");
ASSERT_OK(opts.retval, "inner_map_list_push_pop retval");
if (!leave_in_map)
clear_fields(skel->maps.inner_map);
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.global_list_push_pop), &opts);
ASSERT_OK(ret, "global_list_push_pop");
ASSERT_OK(opts.retval, "global_list_push_pop retval");
/* FIXME:
if (!leave_in_map)
clear_fields(skel->maps.data_A);
*/
if (mode == PUSH_POP)
goto end;
ppm:
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.map_list_push_pop_multiple), &opts);
ASSERT_OK(ret, "map_list_push_pop_multiple");
ASSERT_OK(opts.retval, "map_list_push_pop_multiple retval");
if (!leave_in_map)
clear_fields(skel->maps.array_map);
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.inner_map_list_push_pop_multiple), &opts);
ASSERT_OK(ret, "inner_map_list_push_pop_multiple");
ASSERT_OK(opts.retval, "inner_map_list_push_pop_multiple retval");
if (!leave_in_map)
clear_fields(skel->maps.inner_map);
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.global_list_push_pop_multiple), &opts);
ASSERT_OK(ret, "global_list_push_pop_multiple");
ASSERT_OK(opts.retval, "global_list_push_pop_multiple retval");
/* FIXME:
if (!leave_in_map)
clear_fields(skel->maps.data_A);
*/
if (mode == PUSH_POP_MULT)
goto end;
lil:
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.map_list_in_list), &opts);
ASSERT_OK(ret, "map_list_in_list");
ASSERT_OK(opts.retval, "map_list_in_list retval");
if (!leave_in_map)
clear_fields(skel->maps.array_map);
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.inner_map_list_in_list), &opts);
ASSERT_OK(ret, "inner_map_list_in_list");
ASSERT_OK(opts.retval, "inner_map_list_in_list retval");
if (!leave_in_map)
clear_fields(skel->maps.inner_map);
ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.global_list_in_list), &opts);
ASSERT_OK(ret, "global_list_in_list");
ASSERT_OK(opts.retval, "global_list_in_list retval");
/* FIXME:
if (!leave_in_map)
clear_fields(skel->maps.data_A);
*/
end:
linked_list__destroy(skel);
}
#define SPIN_LOCK 2
#define LIST_HEAD 3
#define LIST_NODE 4
static struct btf *init_btf(void)
{
int id, lid, hid, nid;
struct btf *btf;
btf = btf__new_empty();
if (!ASSERT_OK_PTR(btf, "btf__new_empty"))
return NULL;
id = btf__add_int(btf, "int", 4, BTF_INT_SIGNED);
if (!ASSERT_EQ(id, 1, "btf__add_int"))
goto end;
lid = btf__add_struct(btf, "bpf_spin_lock", 4);
if (!ASSERT_EQ(lid, SPIN_LOCK, "btf__add_struct bpf_spin_lock"))
goto end;
hid = btf__add_struct(btf, "bpf_list_head", 16);
if (!ASSERT_EQ(hid, LIST_HEAD, "btf__add_struct bpf_list_head"))
goto end;
nid = btf__add_struct(btf, "bpf_list_node", 16);
if (!ASSERT_EQ(nid, LIST_NODE, "btf__add_struct bpf_list_node"))
goto end;
return btf;
end:
btf__free(btf);
return NULL;
}
static void test_btf(void)
{
struct btf *btf = NULL;
int id, err;
while (test__start_subtest("btf: too many locks")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 24);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", SPIN_LOCK, 0, 0);
if (!ASSERT_OK(err, "btf__add_struct foo::a"))
break;
err = btf__add_field(btf, "b", SPIN_LOCK, 32, 0);
if (!ASSERT_OK(err, "btf__add_struct foo::a"))
break;
err = btf__add_field(btf, "c", LIST_HEAD, 64, 0);
if (!ASSERT_OK(err, "btf__add_struct foo::a"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -E2BIG, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: missing lock")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 16);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_struct foo::a"))
break;
id = btf__add_decl_tag(btf, "contains:baz:a", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:baz:a"))
break;
id = btf__add_struct(btf, "baz", 16);
if (!ASSERT_EQ(id, 7, "btf__add_struct baz"))
break;
err = btf__add_field(btf, "a", LIST_NODE, 0, 0);
if (!ASSERT_OK(err, "btf__add_field baz::a"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -EINVAL, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: bad offset")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 36);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::c"))
break;
id = btf__add_decl_tag(btf, "contains:foo:b", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:foo:b"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -EEXIST, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: missing contains:")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 24);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", SPIN_LOCK, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", LIST_HEAD, 64, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -EINVAL, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: missing struct")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 24);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", SPIN_LOCK, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", LIST_HEAD, 64, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
id = btf__add_decl_tag(btf, "contains:bar:bar", 5, 1);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:bar:bar"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -ENOENT, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: missing node")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 24);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", SPIN_LOCK, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", LIST_HEAD, 64, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
id = btf__add_decl_tag(btf, "contains:foo:c", 5, 1);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:foo:c"))
break;
err = btf__load_into_kernel(btf);
btf__free(btf);
ASSERT_EQ(err, -ENOENT, "check btf");
break;
}
while (test__start_subtest("btf: node incorrect type")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 20);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", SPIN_LOCK, 128, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
id = btf__add_decl_tag(btf, "contains:bar:a", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:bar:a"))
break;
id = btf__add_struct(btf, "bar", 4);
if (!ASSERT_EQ(id, 7, "btf__add_struct bar"))
break;
err = btf__add_field(btf, "a", SPIN_LOCK, 0, 0);
if (!ASSERT_OK(err, "btf__add_field bar::a"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -EINVAL, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: multiple bpf_list_node with name b")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 52);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 256, 0);
if (!ASSERT_OK(err, "btf__add_field foo::c"))
break;
err = btf__add_field(btf, "d", SPIN_LOCK, 384, 0);
if (!ASSERT_OK(err, "btf__add_field foo::d"))
break;
id = btf__add_decl_tag(btf, "contains:foo:b", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:foo:b"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -EINVAL, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: owning | owned AA cycle")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 36);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 256, 0);
if (!ASSERT_OK(err, "btf__add_field foo::c"))
break;
id = btf__add_decl_tag(btf, "contains:foo:b", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:foo:b"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -ELOOP, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: owning | owned ABA cycle")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 36);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 256, 0);
if (!ASSERT_OK(err, "btf__add_field foo::c"))
break;
id = btf__add_decl_tag(btf, "contains:bar:b", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:bar:b"))
break;
id = btf__add_struct(btf, "bar", 36);
if (!ASSERT_EQ(id, 7, "btf__add_struct bar"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field bar::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field bar::b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 256, 0);
if (!ASSERT_OK(err, "btf__add_field bar::c"))
break;
id = btf__add_decl_tag(btf, "contains:foo:b", 7, 0);
if (!ASSERT_EQ(id, 8, "btf__add_decl_tag contains:foo:b"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -ELOOP, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: owning -> owned")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 20);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", SPIN_LOCK, 128, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
id = btf__add_decl_tag(btf, "contains:bar:a", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:bar:a"))
break;
id = btf__add_struct(btf, "bar", 16);
if (!ASSERT_EQ(id, 7, "btf__add_struct bar"))
break;
err = btf__add_field(btf, "a", LIST_NODE, 0, 0);
if (!ASSERT_OK(err, "btf__add_field bar::a"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, 0, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: owning -> owning | owned -> owned")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 20);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", SPIN_LOCK, 128, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
id = btf__add_decl_tag(btf, "contains:bar:b", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:bar:b"))
break;
id = btf__add_struct(btf, "bar", 36);
if (!ASSERT_EQ(id, 7, "btf__add_struct bar"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field bar::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field bar::b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 256, 0);
if (!ASSERT_OK(err, "btf__add_field bar::c"))
break;
id = btf__add_decl_tag(btf, "contains:baz:a", 7, 0);
if (!ASSERT_EQ(id, 8, "btf__add_decl_tag contains:baz:a"))
break;
id = btf__add_struct(btf, "baz", 16);
if (!ASSERT_EQ(id, 9, "btf__add_struct baz"))
break;
err = btf__add_field(btf, "a", LIST_NODE, 0, 0);
if (!ASSERT_OK(err, "btf__add_field baz:a"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, 0, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: owning | owned -> owning | owned -> owned")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 36);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 256, 0);
if (!ASSERT_OK(err, "btf__add_field foo::c"))
break;
id = btf__add_decl_tag(btf, "contains:bar:b", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:bar:b"))
break;
id = btf__add_struct(btf, "bar", 36);
if (!ASSERT_EQ(id, 7, "btf__add_struct bar"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field bar:a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field bar:b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 256, 0);
if (!ASSERT_OK(err, "btf__add_field bar:c"))
break;
id = btf__add_decl_tag(btf, "contains:baz:a", 7, 0);
if (!ASSERT_EQ(id, 8, "btf__add_decl_tag contains:baz:a"))
break;
id = btf__add_struct(btf, "baz", 16);
if (!ASSERT_EQ(id, 9, "btf__add_struct baz"))
break;
err = btf__add_field(btf, "a", LIST_NODE, 0, 0);
if (!ASSERT_OK(err, "btf__add_field baz:a"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -ELOOP, "check btf");
btf__free(btf);
break;
}
while (test__start_subtest("btf: owning -> owning | owned -> owning | owned -> owned")) {
btf = init_btf();
if (!ASSERT_OK_PTR(btf, "init_btf"))
break;
id = btf__add_struct(btf, "foo", 20);
if (!ASSERT_EQ(id, 5, "btf__add_struct foo"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field foo::a"))
break;
err = btf__add_field(btf, "b", SPIN_LOCK, 128, 0);
if (!ASSERT_OK(err, "btf__add_field foo::b"))
break;
id = btf__add_decl_tag(btf, "contains:bar:b", 5, 0);
if (!ASSERT_EQ(id, 6, "btf__add_decl_tag contains:bar:b"))
break;
id = btf__add_struct(btf, "bar", 36);
if (!ASSERT_EQ(id, 7, "btf__add_struct bar"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field bar::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field bar::b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 256, 0);
if (!ASSERT_OK(err, "btf__add_field bar::c"))
break;
id = btf__add_decl_tag(btf, "contains:baz:b", 7, 0);
if (!ASSERT_EQ(id, 8, "btf__add_decl_tag"))
break;
id = btf__add_struct(btf, "baz", 36);
if (!ASSERT_EQ(id, 9, "btf__add_struct baz"))
break;
err = btf__add_field(btf, "a", LIST_HEAD, 0, 0);
if (!ASSERT_OK(err, "btf__add_field bar::a"))
break;
err = btf__add_field(btf, "b", LIST_NODE, 128, 0);
if (!ASSERT_OK(err, "btf__add_field bar::b"))
break;
err = btf__add_field(btf, "c", SPIN_LOCK, 256, 0);
if (!ASSERT_OK(err, "btf__add_field bar::c"))
break;
id = btf__add_decl_tag(btf, "contains:bam:a", 9, 0);
if (!ASSERT_EQ(id, 10, "btf__add_decl_tag contains:bam:a"))
break;
id = btf__add_struct(btf, "bam", 16);
if (!ASSERT_EQ(id, 11, "btf__add_struct bam"))
break;
err = btf__add_field(btf, "a", LIST_NODE, 0, 0);
if (!ASSERT_OK(err, "btf__add_field bam::a"))
break;
err = btf__load_into_kernel(btf);
ASSERT_EQ(err, -ELOOP, "check btf");
btf__free(btf);
break;
}
}
void test_linked_list(void)
{
int i;
for (i = 0; i < ARRAY_SIZE(linked_list_fail_tests); i++) {
if (!test__start_subtest(linked_list_fail_tests[i].prog_name))
continue;
test_linked_list_fail_prog(linked_list_fail_tests[i].prog_name,
linked_list_fail_tests[i].err_msg);
}
test_btf();
test_linked_list_success(PUSH_POP, false);
test_linked_list_success(PUSH_POP, true);
test_linked_list_success(PUSH_POP_MULT, false);
test_linked_list_success(PUSH_POP_MULT, true);
test_linked_list_success(LIST_IN_LIST, false);
test_linked_list_success(LIST_IN_LIST, true);
test_linked_list_success(TEST_ALL, false);
}

136
tools/testing/selftests/bpf/prog_tests/spin_lock.c Normal file
View File

@ -0,0 +1,136 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
#include <network_helpers.h>
#include "test_spin_lock.skel.h"
#include "test_spin_lock_fail.skel.h"
static char log_buf[1024 * 1024];
static struct {
const char *prog_name;
const char *err_msg;
} spin_lock_fail_tests[] = {
{ "lock_id_kptr_preserve",
"5: (bf) r1 = r0 ; R0_w=ptr_foo(id=2,ref_obj_id=2,off=0,imm=0) "
"R1_w=ptr_foo(id=2,ref_obj_id=2,off=0,imm=0) refs=2\n6: (85) call bpf_this_cpu_ptr#154\n"
"R1 type=ptr_ expected=percpu_ptr_" },
{ "lock_id_global_zero",
"; R1_w=map_value(off=0,ks=4,vs=4,imm=0)\n2: (85) call bpf_this_cpu_ptr#154\n"
"R1 type=map_value expected=percpu_ptr_" },
{ "lock_id_mapval_preserve",
"8: (bf) r1 = r0 ; R0_w=map_value(id=1,off=0,ks=4,vs=8,imm=0) "
"R1_w=map_value(id=1,off=0,ks=4,vs=8,imm=0)\n9: (85) call bpf_this_cpu_ptr#154\n"
"R1 type=map_value expected=percpu_ptr_" },
{ "lock_id_innermapval_preserve",
"13: (bf) r1 = r0 ; R0=map_value(id=2,off=0,ks=4,vs=8,imm=0) "
"R1_w=map_value(id=2,off=0,ks=4,vs=8,imm=0)\n14: (85) call bpf_this_cpu_ptr#154\n"
"R1 type=map_value expected=percpu_ptr_" },
{ "lock_id_mismatch_kptr_kptr", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_kptr_global", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_kptr_mapval", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_kptr_innermapval", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_global_global", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_global_kptr", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_global_mapval", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_global_innermapval", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_mapval_mapval", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_mapval_kptr", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_mapval_global", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_mapval_innermapval", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_innermapval_innermapval1", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_innermapval_innermapval2", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_innermapval_kptr", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_innermapval_global", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_innermapval_mapval", "bpf_spin_unlock of different lock" },
};
static void test_spin_lock_fail_prog(const char *prog_name, const char *err_msg)
{
LIBBPF_OPTS(bpf_object_open_opts, opts, .kernel_log_buf = log_buf,
.kernel_log_size = sizeof(log_buf),
.kernel_log_level = 1);
struct test_spin_lock_fail *skel;
struct bpf_program *prog;
int ret;
skel = test_spin_lock_fail__open_opts(&opts);
if (!ASSERT_OK_PTR(skel, "test_spin_lock_fail__open_opts"))
return;
prog = bpf_object__find_program_by_name(skel->obj, prog_name);
if (!ASSERT_OK_PTR(prog, "bpf_object__find_program_by_name"))
goto end;
bpf_program__set_autoload(prog, true);
ret = test_spin_lock_fail__load(skel);
if (!ASSERT_ERR(ret, "test_spin_lock_fail__load must fail"))
goto end;
if (!ASSERT_OK_PTR(strstr(log_buf, err_msg), "expected error message")) {
fprintf(stderr, "Expected: %s\n", err_msg);
fprintf(stderr, "Verifier: %s\n", log_buf);
}
end:
test_spin_lock_fail__destroy(skel);
}
static void *spin_lock_thread(void *arg)
{
int err, prog_fd = *(u32 *) arg;
LIBBPF_OPTS(bpf_test_run_opts, topts,
.data_in = &pkt_v4,
.data_size_in = sizeof(pkt_v4),
.repeat = 10000,
);
err = bpf_prog_test_run_opts(prog_fd, &topts);
ASSERT_OK(err, "test_run");
ASSERT_OK(topts.retval, "test_run retval");
pthread_exit(arg);
}
void test_spin_lock_success(void)
{
struct test_spin_lock *skel;
pthread_t thread_id[4];
int prog_fd, i;
void *ret;
skel = test_spin_lock__open_and_load();
if (!ASSERT_OK_PTR(skel, "test_spin_lock__open_and_load"))
return;
prog_fd = bpf_program__fd(skel->progs.bpf_spin_lock_test);
for (i = 0; i < 4; i++) {
int err;
err = pthread_create(&thread_id[i], NULL, &spin_lock_thread, &prog_fd);
if (!ASSERT_OK(err, "pthread_create"))
goto end;
}
for (i = 0; i < 4; i++) {
if (!ASSERT_OK(pthread_join(thread_id[i], &ret), "pthread_join"))
goto end;
if (!ASSERT_EQ(ret, &prog_fd, "ret == prog_fd"))
goto end;
}
end:
test_spin_lock__destroy(skel);
}
void test_spin_lock(void)
{
int i;
test_spin_lock_success();
for (i = 0; i < ARRAY_SIZE(spin_lock_fail_tests); i++) {
if (!test__start_subtest(spin_lock_fail_tests[i].prog_name))
continue;
test_spin_lock_fail_prog(spin_lock_fail_tests[i].prog_name,
spin_lock_fail_tests[i].err_msg);
}
}

45
tools/testing/selftests/bpf/prog_tests/spinlock.c
View File

@ -1,45 +0,0 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
#include <network_helpers.h>
static void *spin_lock_thread(void *arg)
{
int err, prog_fd = *(u32 *) arg;
LIBBPF_OPTS(bpf_test_run_opts, topts,
.data_in = &pkt_v4,
.data_size_in = sizeof(pkt_v4),
.repeat = 10000,
);
err = bpf_prog_test_run_opts(prog_fd, &topts);
ASSERT_OK(err, "test_run");
ASSERT_OK(topts.retval, "test_run retval");
pthread_exit(arg);
}
void test_spinlock(void)
{
const char *file = "./test_spin_lock.bpf.o";
pthread_t thread_id[4];
struct bpf_object *obj = NULL;
int prog_fd;
int err = 0, i;
void *ret;
err = bpf_prog_test_load(file, BPF_PROG_TYPE_CGROUP_SKB, &obj, &prog_fd);
if (CHECK_FAIL(err)) {
printf("test_spin_lock:bpf_prog_test_load errno %d\n", errno);
goto close_prog;
}
for (i = 0; i < 4; i++)
if (CHECK_FAIL(pthread_create(&thread_id[i], NULL,
&spin_lock_thread, &prog_fd)))
goto close_prog;
for (i = 0; i < 4; i++)
if (CHECK_FAIL(pthread_join(thread_id[i], &ret) ||
ret != (void *)&prog_fd))
goto close_prog;
close_prog:
bpf_object__close(obj);
}

379
tools/testing/selftests/bpf/progs/linked_list.c Normal file
View File

@ -0,0 +1,379 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#include "bpf_experimental.h"
#ifndef ARRAY_SIZE
#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
#endif
#include "linked_list.h"
static __always_inline
int list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool leave_in_map)
{
struct bpf_list_node *n;
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 2;
bpf_spin_lock(lock);
n = bpf_list_pop_front(head);
bpf_spin_unlock(lock);
if (n) {
bpf_obj_drop(container_of(n, struct foo, node));
bpf_obj_drop(f);
return 3;
}
bpf_spin_lock(lock);
n = bpf_list_pop_back(head);
bpf_spin_unlock(lock);
if (n) {
bpf_obj_drop(container_of(n, struct foo, node));
bpf_obj_drop(f);
return 4;
}
bpf_spin_lock(lock);
f->data = 42;
bpf_list_push_front(head, &f->node);
bpf_spin_unlock(lock);
if (leave_in_map)
return 0;
bpf_spin_lock(lock);
n = bpf_list_pop_back(head);
bpf_spin_unlock(lock);
if (!n)
return 5;
f = container_of(n, struct foo, node);
if (f->data != 42) {
bpf_obj_drop(f);
return 6;
}
bpf_spin_lock(lock);
f->data = 13;
bpf_list_push_front(head, &f->node);
bpf_spin_unlock(lock);
bpf_spin_lock(lock);
n = bpf_list_pop_front(head);
bpf_spin_unlock(lock);
if (!n)
return 7;
f = container_of(n, struct foo, node);
if (f->data != 13) {
bpf_obj_drop(f);
return 8;
}
bpf_obj_drop(f);
bpf_spin_lock(lock);
n = bpf_list_pop_front(head);
bpf_spin_unlock(lock);
if (n) {
bpf_obj_drop(container_of(n, struct foo, node));
return 9;
}
bpf_spin_lock(lock);
n = bpf_list_pop_back(head);
bpf_spin_unlock(lock);
if (n) {
bpf_obj_drop(container_of(n, struct foo, node));
return 10;
}
return 0;
}
static __always_inline
int list_push_pop_multiple(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool leave_in_map)
{
struct bpf_list_node *n;
struct foo *f[8], *pf;
int i;
for (i = 0; i < ARRAY_SIZE(f); i++) {
f[i] = bpf_obj_new(typeof(**f));
if (!f[i])
return 2;
f[i]->data = i;
bpf_spin_lock(lock);
bpf_list_push_front(head, &f[i]->node);
bpf_spin_unlock(lock);
}
for (i = 0; i < ARRAY_SIZE(f); i++) {
bpf_spin_lock(lock);
n = bpf_list_pop_front(head);
bpf_spin_unlock(lock);
if (!n)
return 3;
pf = container_of(n, struct foo, node);
if (pf->data != (ARRAY_SIZE(f) - i - 1)) {
bpf_obj_drop(pf);
return 4;
}
bpf_spin_lock(lock);
bpf_list_push_back(head, &pf->node);
bpf_spin_unlock(lock);
}
if (leave_in_map)
return 0;
for (i = 0; i < ARRAY_SIZE(f); i++) {
bpf_spin_lock(lock);
n = bpf_list_pop_back(head);
bpf_spin_unlock(lock);
if (!n)
return 5;
pf = container_of(n, struct foo, node);
if (pf->data != i) {
bpf_obj_drop(pf);
return 6;
}
bpf_obj_drop(pf);
}
bpf_spin_lock(lock);
n = bpf_list_pop_back(head);
bpf_spin_unlock(lock);
if (n) {
bpf_obj_drop(container_of(n, struct foo, node));
return 7;
}
bpf_spin_lock(lock);
n = bpf_list_pop_front(head);
bpf_spin_unlock(lock);
if (n) {
bpf_obj_drop(container_of(n, struct foo, node));
return 8;
}
return 0;
}
static __always_inline
int list_in_list(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool leave_in_map)
{
struct bpf_list_node *n;
struct bar *ba[8], *b;
struct foo *f;
int i;
f = bpf_obj_new(typeof(*f));
if (!f)
return 2;
for (i = 0; i < ARRAY_SIZE(ba); i++) {
b = bpf_obj_new(typeof(*b));
if (!b) {
bpf_obj_drop(f);
return 3;
}
b->data = i;
bpf_spin_lock(&f->lock);
bpf_list_push_back(&f->head, &b->node);
bpf_spin_unlock(&f->lock);
}
bpf_spin_lock(lock);
f->data = 42;
bpf_list_push_front(head, &f->node);
bpf_spin_unlock(lock);
if (leave_in_map)
return 0;
bpf_spin_lock(lock);
n = bpf_list_pop_front(head);
bpf_spin_unlock(lock);
if (!n)
return 4;
f = container_of(n, struct foo, node);
if (f->data != 42) {
bpf_obj_drop(f);
return 5;
}
for (i = 0; i < ARRAY_SIZE(ba); i++) {
bpf_spin_lock(&f->lock);
n = bpf_list_pop_front(&f->head);
bpf_spin_unlock(&f->lock);
if (!n) {
bpf_obj_drop(f);
return 6;
}
b = container_of(n, struct bar, node);
if (b->data != i) {
bpf_obj_drop(f);
bpf_obj_drop(b);
return 7;
}
bpf_obj_drop(b);
}
bpf_spin_lock(&f->lock);
n = bpf_list_pop_front(&f->head);
bpf_spin_unlock(&f->lock);
if (n) {
bpf_obj_drop(f);
bpf_obj_drop(container_of(n, struct bar, node));
return 8;
}
bpf_obj_drop(f);
return 0;
}
static __always_inline
int test_list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head)
{
int ret;
ret = list_push_pop(lock, head, false);
if (ret)
return ret;
return list_push_pop(lock, head, true);
}
static __always_inline
int test_list_push_pop_multiple(struct bpf_spin_lock *lock, struct bpf_list_head *head)
{
int ret;
ret = list_push_pop_multiple(lock ,head, false);
if (ret)
return ret;
return list_push_pop_multiple(lock, head, true);
}
static __always_inline
int test_list_in_list(struct bpf_spin_lock *lock, struct bpf_list_head *head)
{
int ret;
ret = list_in_list(lock, head, false);
if (ret)
return ret;
return list_in_list(lock, head, true);
}
SEC("tc")
int map_list_push_pop(void *ctx)
{
struct map_value *v;
v = bpf_map_lookup_elem(&array_map, &(int){0});
if (!v)
return 1;
return test_list_push_pop(&v->lock, &v->head);
}
SEC("tc")
int inner_map_list_push_pop(void *ctx)
{
struct map_value *v;
void *map;
map = bpf_map_lookup_elem(&map_of_maps, &(int){0});
if (!map)
return 1;
v = bpf_map_lookup_elem(map, &(int){0});
if (!v)
return 1;
return test_list_push_pop(&v->lock, &v->head);
}
SEC("tc")
int global_list_push_pop(void *ctx)
{
/* FIXME:
* return test_list_push_pop(&glock, &ghead);
*/
return 0;
}
SEC("tc")
int map_list_push_pop_multiple(void *ctx)
{
struct map_value *v;
int ret;
v = bpf_map_lookup_elem(&array_map, &(int){0});
if (!v)
return 1;
return test_list_push_pop_multiple(&v->lock, &v->head);
}
SEC("tc")
int inner_map_list_push_pop_multiple(void *ctx)
{
struct map_value *v;
void *map;
int ret;
map = bpf_map_lookup_elem(&map_of_maps, &(int){0});
if (!map)
return 1;
v = bpf_map_lookup_elem(map, &(int){0});
if (!v)
return 1;
return test_list_push_pop_multiple(&v->lock, &v->head);
}
SEC("tc")
int global_list_push_pop_multiple(void *ctx)
{
int ret;
/* FIXME:
ret = list_push_pop_multiple(&glock, &ghead, false);
if (ret)
return ret;
return list_push_pop_multiple(&glock, &ghead, true);
*/
return 0;
}
SEC("tc")
int map_list_in_list(void *ctx)
{
struct map_value *v;
int ret;
v = bpf_map_lookup_elem(&array_map, &(int){0});
if (!v)
return 1;
return test_list_in_list(&v->lock, &v->head);
}
SEC("tc")
int inner_map_list_in_list(void *ctx)
{
struct map_value *v;
void *map;
int ret;
map = bpf_map_lookup_elem(&map_of_maps, &(int){0});
if (!map)
return 1;
v = bpf_map_lookup_elem(map, &(int){0});
if (!v)
return 1;
return test_list_in_list(&v->lock, &v->head);
}
SEC("tc")
int global_list_in_list(void *ctx)
{
/* FIXME
return test_list_in_list(&glock, &ghead);
*/
return 0;
}
char _license[] SEC("license") = "GPL";

58
tools/testing/selftests/bpf/progs/linked_list.h Normal file
View File

@ -0,0 +1,58 @@
// SPDX-License-Identifier: GPL-2.0
#ifndef LINKED_LIST_H
#define LINKED_LIST_H
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include "bpf_experimental.h"
struct bar {
struct bpf_list_node node;
int data;
};
struct foo {
struct bpf_list_node node;
struct bpf_list_head head __contains(bar, node);
struct bpf_spin_lock lock;
int data;
struct bpf_list_node node2;
};
struct map_value {
struct bpf_spin_lock lock;
int data;
struct bpf_list_head head __contains(foo, node);
};
struct array_map {
__uint(type, BPF_MAP_TYPE_ARRAY);
__type(key, int);
__type(value, struct map_value);
__uint(max_entries, 1);
};
struct array_map array_map SEC(".maps");
struct array_map inner_map SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
__uint(max_entries, 1);
__type(key, int);
__type(value, int);
__array(values, struct array_map);
} map_of_maps SEC(".maps") = {
.values = {
[0] = &inner_map,
},
};
/* FIXME
#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))
private(A) struct bpf_spin_lock glock;
private(A) struct bpf_list_head ghead __contains(foo, node);
private(B) struct bpf_spin_lock glock2;
*/
#endif

581
tools/testing/selftests/bpf/progs/linked_list_fail.c Normal file
View File

@ -0,0 +1,581 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#include "bpf_experimental.h"
#include "linked_list.h"
#define INIT \
struct map_value *v, *v2, *iv, *iv2; \
struct foo *f, *f1, *f2; \
struct bar *b; \
void *map; \
\
map = bpf_map_lookup_elem(&map_of_maps, &(int){ 0 }); \
if (!map) \
return 0; \
v = bpf_map_lookup_elem(&array_map, &(int){ 0 }); \
if (!v) \
return 0; \
v2 = bpf_map_lookup_elem(&array_map, &(int){ 0 }); \
if (!v2) \
return 0; \
iv = bpf_map_lookup_elem(map, &(int){ 0 }); \
if (!iv) \
return 0; \
iv2 = bpf_map_lookup_elem(map, &(int){ 0 }); \
if (!iv2) \
return 0; \
f = bpf_obj_new(typeof(*f)); \
if (!f) \
return 0; \
f1 = f; \
f2 = bpf_obj_new(typeof(*f2)); \
if (!f2) { \
bpf_obj_drop(f1); \
return 0; \
} \
b = bpf_obj_new(typeof(*b)); \
if (!b) { \
bpf_obj_drop(f2); \
bpf_obj_drop(f1); \
return 0; \
}
#define CHECK(test, op, hexpr) \
SEC("?tc") \
int test##_missing_lock_##op(void *ctx) \
{ \
INIT; \
void (*p)(void *) = (void *)&bpf_list_##op; \
p(hexpr); \
return 0; \
}
CHECK(kptr, push_front, &f->head);
CHECK(kptr, push_back, &f->head);
CHECK(kptr, pop_front, &f->head);
CHECK(kptr, pop_back, &f->head);
/* FIXME
CHECK(global, push_front, &ghead);
CHECK(global, push_back, &ghead);
CHECK(global, pop_front, &ghead);
CHECK(global, pop_back, &ghead);
*/
CHECK(map, push_front, &v->head);
CHECK(map, push_back, &v->head);
CHECK(map, pop_front, &v->head);
CHECK(map, pop_back, &v->head);
CHECK(inner_map, push_front, &iv->head);
CHECK(inner_map, push_back, &iv->head);
CHECK(inner_map, pop_front, &iv->head);
CHECK(inner_map, pop_back, &iv->head);
#undef CHECK
#define CHECK(test, op, lexpr, hexpr) \
SEC("?tc") \
int test##_incorrect_lock_##op(void *ctx) \
{ \
INIT; \
void (*p)(void *) = (void *)&bpf_list_##op; \
bpf_spin_lock(lexpr); \
p(hexpr); \
return 0; \
}
#define CHECK_OP(op) \
CHECK(kptr_kptr, op, &f1->lock, &f2->head); \
CHECK(kptr_map, op, &f1->lock, &v->head); \
CHECK(kptr_inner_map, op, &f1->lock, &iv->head); \
\
CHECK(map_map, op, &v->lock, &v2->head); \
CHECK(map_kptr, op, &v->lock, &f2->head); \
CHECK(map_inner_map, op, &v->lock, &iv->head); \
\
CHECK(inner_map_inner_map, op, &iv->lock, &iv2->head); \
CHECK(inner_map_kptr, op, &iv->lock, &f2->head); \
CHECK(inner_map_map, op, &iv->lock, &v->head);
CHECK_OP(push_front);
CHECK_OP(push_back);
CHECK_OP(pop_front);
CHECK_OP(pop_back);
#undef CHECK
#undef CHECK_OP
#undef INIT
/* FIXME
SEC("?kprobe/xyz")
int map_compat_kprobe(void *ctx)
{
bpf_list_push_front(&ghead, NULL);
return 0;
}
SEC("?kretprobe/xyz")
int map_compat_kretprobe(void *ctx)
{
bpf_list_push_front(&ghead, NULL);
return 0;
}
SEC("?tracepoint/xyz")
int map_compat_tp(void *ctx)
{
bpf_list_push_front(&ghead, NULL);
return 0;
}
SEC("?perf_event")
int map_compat_perf(void *ctx)
{
bpf_list_push_front(&ghead, NULL);
return 0;
}
SEC("?raw_tp/xyz")
int map_compat_raw_tp(void *ctx)
{
bpf_list_push_front(&ghead, NULL);
return 0;
}
SEC("?raw_tp.w/xyz")
int map_compat_raw_tp_w(void *ctx)
{
bpf_list_push_front(&ghead, NULL);
return 0;
}
*/
SEC("?tc")
int obj_type_id_oor(void *ctx)
{
bpf_obj_new_impl(~0UL, NULL);
return 0;
}
SEC("?tc")
int obj_new_no_composite(void *ctx)
{
bpf_obj_new_impl(bpf_core_type_id_local(int), (void *)42);
return 0;
}
SEC("?tc")
int obj_new_no_struct(void *ctx)
{
bpf_obj_new(union { int data; unsigned udata; });
return 0;
}
SEC("?tc")
int obj_drop_non_zero_off(void *ctx)
{
void *f;
f = bpf_obj_new(struct foo);
if (!f)
return 0;
bpf_obj_drop(f+1);
return 0;
}
SEC("?tc")
int new_null_ret(void *ctx)
{
return bpf_obj_new(struct foo)->data;
}
SEC("?tc")
int obj_new_acq(void *ctx)
{
bpf_obj_new(struct foo);
return 0;
}
SEC("?tc")
int use_after_drop(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_obj_drop(f);
return f->data;
}
SEC("?tc")
int ptr_walk_scalar(void *ctx)
{
struct test1 {
struct test2 {
struct test2 *next;
} *ptr;
} *p;
p = bpf_obj_new(typeof(*p));
if (!p)
return 0;
bpf_this_cpu_ptr(p->ptr);
return 0;
}
SEC("?tc")
int direct_read_lock(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
return *(int *)&f->lock;
}
SEC("?tc")
int direct_write_lock(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
*(int *)&f->lock = 0;
return 0;
}
SEC("?tc")
int direct_read_head(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
return *(int *)&f->head;
}
SEC("?tc")
int direct_write_head(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
*(int *)&f->head = 0;
return 0;
}
SEC("?tc")
int direct_read_node(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
return *(int *)&f->node;
}
SEC("?tc")
int direct_write_node(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
*(int *)&f->node = 0;
return 0;
}
/* FIXME
static __always_inline
int write_after_op(void (*push_op)(void *head, void *node))
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
push_op(&ghead, &f->node);
f->data = 42;
bpf_spin_unlock(&glock);
return 0;
}
SEC("?tc")
int write_after_push_front(void *ctx)
{
return write_after_op((void *)bpf_list_push_front);
}
SEC("?tc")
int write_after_push_back(void *ctx)
{
return write_after_op((void *)bpf_list_push_back);
}
static __always_inline
int use_after_unlock(void (*op)(void *head, void *node))
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
f->data = 42;
op(&ghead, &f->node);
bpf_spin_unlock(&glock);
return f->data;
}
SEC("?tc")
int use_after_unlock_push_front(void *ctx)
{
return use_after_unlock((void *)bpf_list_push_front);
}
SEC("?tc")
int use_after_unlock_push_back(void *ctx)
{
return use_after_unlock((void *)bpf_list_push_back);
}
static __always_inline
int list_double_add(void (*op)(void *head, void *node))
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
op(&ghead, &f->node);
op(&ghead, &f->node);
bpf_spin_unlock(&glock);
return 0;
}
SEC("?tc")
int double_push_front(void *ctx)
{
return list_double_add((void *)bpf_list_push_front);
}
SEC("?tc")
int double_push_back(void *ctx)
{
return list_double_add((void *)bpf_list_push_back);
}
SEC("?tc")
int no_node_value_type(void *ctx)
{
void *p;
p = bpf_obj_new(struct { int data; });
if (!p)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front(&ghead, p);
bpf_spin_unlock(&glock);
return 0;
}
SEC("?tc")
int incorrect_value_type(void *ctx)
{
struct bar *b;
b = bpf_obj_new(typeof(*b));
if (!b)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front(&ghead, &b->node);
bpf_spin_unlock(&glock);
return 0;
}
SEC("?tc")
int incorrect_node_var_off(struct __sk_buff *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front(&ghead, (void *)&f->node + ctx->protocol);
bpf_spin_unlock(&glock);
return 0;
}
SEC("?tc")
int incorrect_node_off1(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front(&ghead, (void *)&f->node + 1);
bpf_spin_unlock(&glock);
return 0;
}
SEC("?tc")
int incorrect_node_off2(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front(&ghead, &f->node2);
bpf_spin_unlock(&glock);
return 0;
}
SEC("?tc")
int no_head_type(void *ctx)
{
void *p;
p = bpf_obj_new(typeof(struct { int data; }));
if (!p)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front(p, NULL);
bpf_spin_lock(&glock);
return 0;
}
SEC("?tc")
int incorrect_head_var_off1(struct __sk_buff *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front((void *)&ghead + ctx->protocol, &f->node);
bpf_spin_unlock(&glock);
return 0;
}
SEC("?tc")
int incorrect_head_var_off2(struct __sk_buff *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front((void *)&f->head + ctx->protocol, &f->node);
bpf_spin_unlock(&glock);
return 0;
}
*/
SEC("?tc")
int incorrect_head_off1(void *ctx)
{
struct foo *f;
struct bar *b;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
b = bpf_obj_new(typeof(*b));
if (!b) {
bpf_obj_drop(f);
return 0;
}
bpf_spin_lock(&f->lock);
bpf_list_push_front((void *)&f->head + 1, &b->node);
bpf_spin_unlock(&f->lock);
return 0;
}
/* FIXME
SEC("?tc")
int incorrect_head_off2(void *ctx)
{
struct foo *f;
struct bar *b;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_spin_lock(&glock);
bpf_list_push_front((void *)&ghead + 1, &f->node);
bpf_spin_unlock(&glock);
return 0;
}
*/
static __always_inline
int pop_ptr_off(void *(*op)(void *head))
{
struct {
struct bpf_list_head head __contains(foo, node2);
struct bpf_spin_lock lock;
} *p;
struct bpf_list_node *n;
p = bpf_obj_new(typeof(*p));
if (!p)
return 0;
bpf_spin_lock(&p->lock);
n = op(&p->head);
bpf_spin_unlock(&p->lock);
bpf_this_cpu_ptr(n);
return 0;
}
SEC("?tc")
int pop_front_off(void *ctx)
{
return pop_ptr_off((void *)bpf_list_pop_front);
}
SEC("?tc")
int pop_back_off(void *ctx)
{
return pop_ptr_off((void *)bpf_list_pop_back);
}
char _license[] SEC("license") = "GPL";

4
tools/testing/selftests/bpf/progs/test_spin_lock.c
View File

@ -45,8 +45,8 @@ struct {
#define CREDIT_PER_NS(delta, rate) (((delta) * rate) >> 20)
SEC("tc")
int bpf_sping_lock_test(struct __sk_buff *skb)
SEC("cgroup_skb/ingress")
int bpf_spin_lock_test(struct __sk_buff *skb)
{
volatile int credit = 0, max_credit = 100, pkt_len = 64;
struct hmap_elem zero = {}, *val;

204
tools/testing/selftests/bpf/progs/test_spin_lock_fail.c Normal file
View File

@ -0,0 +1,204 @@
// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_helpers.h>
#include "bpf_experimental.h"
struct foo {
struct bpf_spin_lock lock;
int data;
};
struct array_map {
__uint(type, BPF_MAP_TYPE_ARRAY);
__type(key, int);
__type(value, struct foo);
__uint(max_entries, 1);
} array_map SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
__uint(max_entries, 1);
__type(key, int);
__type(value, int);
__array(values, struct array_map);
} map_of_maps SEC(".maps") = {
.values = {
[0] = &array_map,
},
};
SEC(".data.A") struct bpf_spin_lock lockA;
SEC(".data.B") struct bpf_spin_lock lockB;
SEC("?tc")
int lock_id_kptr_preserve(void *ctx)
{
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return 0;
bpf_this_cpu_ptr(f);
return 0;
}
SEC("?tc")
int lock_id_global_zero(void *ctx)
{
bpf_this_cpu_ptr(&lockA);
return 0;
}
SEC("?tc")
int lock_id_mapval_preserve(void *ctx)
{
struct foo *f;
int key = 0;
f = bpf_map_lookup_elem(&array_map, &key);
if (!f)
return 0;
bpf_this_cpu_ptr(f);
return 0;
}
SEC("?tc")
int lock_id_innermapval_preserve(void *ctx)
{
struct foo *f;
int key = 0;
void *map;
map = bpf_map_lookup_elem(&map_of_maps, &key);
if (!map)
return 0;
f = bpf_map_lookup_elem(map, &key);
if (!f)
return 0;
bpf_this_cpu_ptr(f);
return 0;
}
#define CHECK(test, A, B) \
SEC("?tc") \
int lock_id_mismatch_##test(void *ctx) \
{ \
struct foo *f1, *f2, *v, *iv; \
int key = 0; \
void *map; \
\
map = bpf_map_lookup_elem(&map_of_maps, &key); \
if (!map) \
return 0; \
iv = bpf_map_lookup_elem(map, &key); \
if (!iv) \
return 0; \
v = bpf_map_lookup_elem(&array_map, &key); \
if (!v) \
return 0; \
f1 = bpf_obj_new(typeof(*f1)); \
if (!f1) \
return 0; \
f2 = bpf_obj_new(typeof(*f2)); \
if (!f2) { \
bpf_obj_drop(f1); \
return 0; \
} \
bpf_spin_lock(A); \
bpf_spin_unlock(B); \
return 0; \
}
CHECK(kptr_kptr, &f1->lock, &f2->lock);
CHECK(kptr_global, &f1->lock, &lockA);
CHECK(kptr_mapval, &f1->lock, &v->lock);
CHECK(kptr_innermapval, &f1->lock, &iv->lock);
CHECK(global_global, &lockA, &lockB);
CHECK(global_kptr, &lockA, &f1->lock);
CHECK(global_mapval, &lockA, &v->lock);
CHECK(global_innermapval, &lockA, &iv->lock);
SEC("?tc")
int lock_id_mismatch_mapval_mapval(void *ctx)
{
struct foo *f1, *f2;
int key = 0;
f1 = bpf_map_lookup_elem(&array_map, &key);
if (!f1)
return 0;
f2 = bpf_map_lookup_elem(&array_map, &key);
if (!f2)
return 0;
bpf_spin_lock(&f1->lock);
f1->data = 42;
bpf_spin_unlock(&f2->lock);
return 0;
}
CHECK(mapval_kptr, &v->lock, &f1->lock);
CHECK(mapval_global, &v->lock, &lockB);
CHECK(mapval_innermapval, &v->lock, &iv->lock);
SEC("?tc")
int lock_id_mismatch_innermapval_innermapval1(void *ctx)
{
struct foo *f1, *f2;
int key = 0;
void *map;
map = bpf_map_lookup_elem(&map_of_maps, &key);
if (!map)
return 0;
f1 = bpf_map_lookup_elem(map, &key);
if (!f1)
return 0;
f2 = bpf_map_lookup_elem(map, &key);
if (!f2)
return 0;
bpf_spin_lock(&f1->lock);
f1->data = 42;
bpf_spin_unlock(&f2->lock);
return 0;
}
SEC("?tc")
int lock_id_mismatch_innermapval_innermapval2(void *ctx)
{
struct foo *f1, *f2;
int key = 0;
void *map;
map = bpf_map_lookup_elem(&map_of_maps, &key);
if (!map)
return 0;
f1 = bpf_map_lookup_elem(map, &key);
if (!f1)
return 0;
map = bpf_map_lookup_elem(&map_of_maps, &key);
if (!map)
return 0;
f2 = bpf_map_lookup_elem(map, &key);
if (!f2)
return 0;
bpf_spin_lock(&f1->lock);
f1->data = 42;
bpf_spin_unlock(&f2->lock);
return 0;
}
CHECK(innermapval_kptr, &iv->lock, &f1->lock);
CHECK(innermapval_global, &iv->lock, &lockA);
CHECK(innermapval_mapval, &iv->lock, &v->lock);
#undef CHECK
char _license[] SEC("license") = "GPL";

2
tools/testing/selftests/bpf/verifier/calls.c
View File

@ -109,7 +109,7 @@
},
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
.result = REJECT,
.errstr = "arg#0 pointer type STRUCT prog_test_ref_kfunc must point",
.errstr = "arg#0 expected pointer to btf or socket",
.fixup_kfunc_btf_id = {
{ "bpf_kfunc_call_test_acquire", 3 },
{ "bpf_kfunc_call_test_release", 5 },

4
tools/testing/selftests/bpf/verifier/ref_tracking.c
View File

@ -142,7 +142,7 @@
.kfunc = "bpf",
.expected_attach_type = BPF_LSM_MAC,
.flags = BPF_F_SLEEPABLE,
.errstr = "arg#0 pointer type STRUCT bpf_key must point to scalar, or struct with scalar",
.errstr = "arg#0 expected pointer to btf or socket",
.fixup_kfunc_btf_id = {
{ "bpf_lookup_user_key", 2 },
{ "bpf_key_put", 4 },
@ -163,7 +163,7 @@
.kfunc = "bpf",
.expected_attach_type = BPF_LSM_MAC,
.flags = BPF_F_SLEEPABLE,
.errstr = "arg#0 pointer type STRUCT bpf_key must point to scalar, or struct with scalar",
.errstr = "arg#0 expected pointer to btf or socket",
.fixup_kfunc_btf_id = {
{ "bpf_lookup_system_key", 1 },
{ "bpf_key_put", 3 },