2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
|
Red Black Trees
|
|
|
|
|
(C) 1999 Andrea Arcangeli <andrea@suse.de>
|
|
|
|
|
|
|
|
|
|
This program is free software; you can redistribute it and/or modify
|
|
|
|
|
it under the terms of the GNU General Public License as published by
|
|
|
|
|
the Free Software Foundation; either version 2 of the License, or
|
|
|
|
|
(at your option) any later version.
|
|
|
|
|
|
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
GNU General Public License for more details.
|
|
|
|
|
|
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
|
|
|
along with this program; if not, write to the Free Software
|
|
|
|
|
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
|
|
|
|
|
|
|
|
|
linux/include/linux/rbtree.h
|
|
|
|
|
|
|
|
|
|
To use rbtrees you'll have to implement your own insert and search cores.
|
|
|
|
|
This will avoid us to use callbacks and to drop drammatically performances.
|
|
|
|
|
I know it's not the cleaner way, but in C (not in C++) to get
|
|
|
|
|
performances and genericity...
|
|
|
|
|
|
rbtree: reference Documentation/rbtree.txt for usage instructions
I recently started looking at the rbtree code (with an eye towards
improving the augmented rbtree support, but I haven't gotten there yet).
I noticed a lot of possible speed improvements, which I am now proposing
in this patch set.
Patches 1-4 are preparatory: remove internal functions from rbtree.h so
that users won't be tempted to use them instead of the documented APIs,
clean up some incorrect usages I've noticed (in particular, with the
recently added fs/proc/proc_sysctl.c rbtree usage), reference the
documentation so that people have one less excuse to miss it, etc.
Patch 5 is a small module I wrote to check the rbtree performance. It
creates 100 nodes with random keys and repeatedly inserts and erases them
from an rbtree. Additionally, it has code to check for rbtree invariants
after each insert or erase operation.
Patches 6-12 is where the rbtree optimizations are done, and they touch
only that one file, lib/rbtree.c . I am getting good results out of these
- in my small benchmark doing rbtree insertion (including search) and
erase, I'm seeing a 30% runtime reduction on Sandybridge E5, which is more
than I initially thought would be possible. (the results aren't as
impressive on my two other test hosts though, AMD barcelona and Intel
Westmere, where I am seeing 14% runtime reduction only). The code size -
both source (ommiting comments) and compiled - is also shorter after these
changes. However, I do admit that the updated code is more arduous to
read - one big reason for that is the removal of the tree rotation
helpers, which added some overhead but also made it easier to reason about
things locally. Overall, I believe this is an acceptable compromise,
given that this code doesn't get modified very often, and that I have good
tests for it.
Upon Peter's suggestion, I added comments showing the rtree configuration
before every rotation. I think they help; however it's still best to have
a copy of the cormen/leiserson/rivest book when digging into this code.
This patch: reference Documentation/rbtree.txt for usage instructions
include/linux/rbtree.h included some basic usage instructions, while
Documentation/rbtree.txt had some more complete and easier to follow
instructions. Replacing the former with a reference to the latter.
Signed-off-by: Michel Lespinasse <walken@google.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Acked-by: David Woodhouse <David.Woodhouse@intel.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Daniel Santos <daniel.santos@pobox.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-10-09 07:30:28 +08:00
|
|
|
See Documentation/rbtree.txt for documentation and samples.
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef _LINUX_RBTREE_H
|
|
|
|
|
#define _LINUX_RBTREE_H
|
|
|
|
|
|
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
|
#include <linux/stddef.h>
|
2015-05-27 09:39:36 +08:00
|
|
|
#include <linux/rcupdate.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-10-09 07:30:37 +08:00
|
|
|
struct rb_node {
|
|
|
|
|
unsigned long __rb_parent_color;
|
2005-04-17 06:20:36 +08:00
|
|
|
struct rb_node *rb_right;
|
|
|
|
|
struct rb_node *rb_left;
|
2006-04-22 06:15:39 +08:00
|
|
|
} __attribute__((aligned(sizeof(long))));
|
|
|
|
|
/* The alignment might seem pointless, but allegedly CRIS needs it */
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-10-09 07:30:37 +08:00
|
|
|
struct rb_root {
|
2005-04-17 06:20:36 +08:00
|
|
|
struct rb_node *rb_node;
|
|
|
|
|
};
|
|
|
|
|
|
rbtree: cache leftmost node internally
Patch series "rbtree: Cache leftmost node internally", v4.
A series to extending rbtrees to internally cache the leftmost node such
that we can have fast overlap check optimization for all interval tree
users[1]. The benefits of this series are that:
(i) Unify users that do internal leftmost node caching.
(ii) Optimize all interval tree users.
(iii) Convert at least two new users (epoll and procfs) to the new interface.
This patch (of 16):
Red-black tree semantics imply that nodes with smaller or greater (or
equal for duplicates) keys always be to the left and right,
respectively. For the kernel this is extremely evident when considering
our rb_first() semantics. Enabling lookups for the smallest node in the
tree in O(1) can save a good chunk of cycles in not having to walk down
the tree each time. To this end there are a few core users that
explicitly do this, such as the scheduler and rtmutexes. There is also
the desire for interval trees to have this optimization allowing faster
overlap checking.
This patch introduces a new 'struct rb_root_cached' which is just the
root with a cached pointer to the leftmost node. The reason why the
regular rb_root was not extended instead of adding a new structure was
that this allows the user to have the choice between memory footprint
and actual tree performance. The new wrappers on top of the regular
rb_root calls are:
- rb_first_cached(cached_root) -- which is a fast replacement
for rb_first.
- rb_insert_color_cached(node, cached_root, new)
- rb_erase_cached(node, cached_root)
In addition, augmented cached interfaces are also added for basic
insertion and deletion operations; which becomes important for the
interval tree changes.
With the exception of the inserts, which adds a bool for updating the
new leftmost, the interfaces are kept the same. To this end, porting rb
users to the cached version becomes really trivial, and keeping current
rbtree semantics for users that don't care about the optimization
requires zero overhead.
Link: http://lkml.kernel.org/r/20170719014603.19029-2-dave@stgolabs.net
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-09-09 07:14:36 +08:00
|
|
|
/*
|
|
|
|
|
* Leftmost-cached rbtrees.
|
|
|
|
|
*
|
|
|
|
|
* We do not cache the rightmost node based on footprint
|
|
|
|
|
* size vs number of potential users that could benefit
|
|
|
|
|
* from O(1) rb_last(). Just not worth it, users that want
|
|
|
|
|
* this feature can always implement the logic explicitly.
|
|
|
|
|
* Furthermore, users that want to cache both pointers may
|
|
|
|
|
* find it a bit asymmetric, but that's ok.
|
|
|
|
|
*/
|
|
|
|
|
struct rb_root_cached {
|
|
|
|
|
struct rb_root rb_root;
|
|
|
|
|
struct rb_node *rb_leftmost;
|
|
|
|
|
};
|
2006-04-21 20:35:51 +08:00
|
|
|
|
2012-10-09 07:30:37 +08:00
|
|
|
#define rb_parent(r) ((struct rb_node *)((r)->__rb_parent_color & ~3))
|
2006-04-21 20:35:51 +08:00
|
|
|
|
2010-05-29 21:31:43 +08:00
|
|
|
#define RB_ROOT (struct rb_root) { NULL, }
|
rbtree: cache leftmost node internally
Patch series "rbtree: Cache leftmost node internally", v4.
A series to extending rbtrees to internally cache the leftmost node such
that we can have fast overlap check optimization for all interval tree
users[1]. The benefits of this series are that:
(i) Unify users that do internal leftmost node caching.
(ii) Optimize all interval tree users.
(iii) Convert at least two new users (epoll and procfs) to the new interface.
This patch (of 16):
Red-black tree semantics imply that nodes with smaller or greater (or
equal for duplicates) keys always be to the left and right,
respectively. For the kernel this is extremely evident when considering
our rb_first() semantics. Enabling lookups for the smallest node in the
tree in O(1) can save a good chunk of cycles in not having to walk down
the tree each time. To this end there are a few core users that
explicitly do this, such as the scheduler and rtmutexes. There is also
the desire for interval trees to have this optimization allowing faster
overlap checking.
This patch introduces a new 'struct rb_root_cached' which is just the
root with a cached pointer to the leftmost node. The reason why the
regular rb_root was not extended instead of adding a new structure was
that this allows the user to have the choice between memory footprint
and actual tree performance. The new wrappers on top of the regular
rb_root calls are:
- rb_first_cached(cached_root) -- which is a fast replacement
for rb_first.
- rb_insert_color_cached(node, cached_root, new)
- rb_erase_cached(node, cached_root)
In addition, augmented cached interfaces are also added for basic
insertion and deletion operations; which becomes important for the
interval tree changes.
With the exception of the inserts, which adds a bool for updating the
new leftmost, the interfaces are kept the same. To this end, porting rb
users to the cached version becomes really trivial, and keeping current
rbtree semantics for users that don't care about the optimization
requires zero overhead.
Link: http://lkml.kernel.org/r/20170719014603.19029-2-dave@stgolabs.net
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-09-09 07:14:36 +08:00
|
|
|
#define RB_ROOT_CACHED (struct rb_root_cached) { {NULL, }, NULL }
|
2005-04-17 06:20:36 +08:00
|
|
|
#define rb_entry(ptr, type, member) container_of(ptr, type, member)
|
|
|
|
|
|
2016-01-21 07:00:42 +08:00
|
|
|
#define RB_EMPTY_ROOT(root) (READ_ONCE((root)->rb_node) == NULL)
|
2012-10-09 07:30:32 +08:00
|
|
|
|
2015-02-18 05:46:04 +08:00
|
|
|
/* 'empty' nodes are nodes that are known not to be inserted in an rbtree */
|
2012-10-09 07:30:37 +08:00
|
|
|
#define RB_EMPTY_NODE(node) \
|
|
|
|
|
((node)->__rb_parent_color == (unsigned long)(node))
|
|
|
|
|
#define RB_CLEAR_NODE(node) \
|
|
|
|
|
((node)->__rb_parent_color = (unsigned long)(node))
|
2006-06-21 15:36:18 +08:00
|
|
|
|
2011-01-04 10:59:43 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
extern void rb_insert_color(struct rb_node *, struct rb_root *);
|
|
|
|
|
extern void rb_erase(struct rb_node *, struct rb_root *);
|
|
|
|
|
|
2012-10-09 07:31:17 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/* Find logical next and previous nodes in a tree */
|
2009-01-10 19:12:09 +08:00
|
|
|
extern struct rb_node *rb_next(const struct rb_node *);
|
|
|
|
|
extern struct rb_node *rb_prev(const struct rb_node *);
|
|
|
|
|
extern struct rb_node *rb_first(const struct rb_root *);
|
|
|
|
|
extern struct rb_node *rb_last(const struct rb_root *);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
rbtree: cache leftmost node internally
Patch series "rbtree: Cache leftmost node internally", v4.
A series to extending rbtrees to internally cache the leftmost node such
that we can have fast overlap check optimization for all interval tree
users[1]. The benefits of this series are that:
(i) Unify users that do internal leftmost node caching.
(ii) Optimize all interval tree users.
(iii) Convert at least two new users (epoll and procfs) to the new interface.
This patch (of 16):
Red-black tree semantics imply that nodes with smaller or greater (or
equal for duplicates) keys always be to the left and right,
respectively. For the kernel this is extremely evident when considering
our rb_first() semantics. Enabling lookups for the smallest node in the
tree in O(1) can save a good chunk of cycles in not having to walk down
the tree each time. To this end there are a few core users that
explicitly do this, such as the scheduler and rtmutexes. There is also
the desire for interval trees to have this optimization allowing faster
overlap checking.
This patch introduces a new 'struct rb_root_cached' which is just the
root with a cached pointer to the leftmost node. The reason why the
regular rb_root was not extended instead of adding a new structure was
that this allows the user to have the choice between memory footprint
and actual tree performance. The new wrappers on top of the regular
rb_root calls are:
- rb_first_cached(cached_root) -- which is a fast replacement
for rb_first.
- rb_insert_color_cached(node, cached_root, new)
- rb_erase_cached(node, cached_root)
In addition, augmented cached interfaces are also added for basic
insertion and deletion operations; which becomes important for the
interval tree changes.
With the exception of the inserts, which adds a bool for updating the
new leftmost, the interfaces are kept the same. To this end, porting rb
users to the cached version becomes really trivial, and keeping current
rbtree semantics for users that don't care about the optimization
requires zero overhead.
Link: http://lkml.kernel.org/r/20170719014603.19029-2-dave@stgolabs.net
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-09-09 07:14:36 +08:00
|
|
|
extern void rb_insert_color_cached(struct rb_node *,
|
|
|
|
|
struct rb_root_cached *, bool);
|
|
|
|
|
extern void rb_erase_cached(struct rb_node *node, struct rb_root_cached *);
|
|
|
|
|
/* Same as rb_first(), but O(1) */
|
|
|
|
|
#define rb_first_cached(root) (root)->rb_leftmost
|
|
|
|
|
|
2013-09-12 05:25:10 +08:00
|
|
|
/* Postorder iteration - always visit the parent after its children */
|
|
|
|
|
extern struct rb_node *rb_first_postorder(const struct rb_root *);
|
|
|
|
|
extern struct rb_node *rb_next_postorder(const struct rb_node *);
|
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/* Fast replacement of a single node without remove/rebalance/add/rebalance */
|
2015-05-27 09:39:36 +08:00
|
|
|
extern void rb_replace_node(struct rb_node *victim, struct rb_node *new,
|
2005-04-17 06:20:36 +08:00
|
|
|
struct rb_root *root);
|
2016-07-01 14:53:51 +08:00
|
|
|
extern void rb_replace_node_rcu(struct rb_node *victim, struct rb_node *new,
|
|
|
|
|
struct rb_root *root);
|
2017-12-15 07:32:28 +08:00
|
|
|
extern void rb_replace_node_cached(struct rb_node *victim, struct rb_node *new,
|
|
|
|
|
struct rb_root_cached *root);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2015-05-27 09:39:36 +08:00
|
|
|
static inline void rb_link_node(struct rb_node *node, struct rb_node *parent,
|
|
|
|
|
struct rb_node **rb_link)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2012-10-09 07:30:37 +08:00
|
|
|
node->__rb_parent_color = (unsigned long)parent;
|
2005-04-17 06:20:36 +08:00
|
|
|
node->rb_left = node->rb_right = NULL;
|
|
|
|
|
|
|
|
|
|
*rb_link = node;
|
|
|
|
|
}
|
|
|
|
|
|
2015-05-27 09:39:36 +08:00
|
|
|
static inline void rb_link_node_rcu(struct rb_node *node, struct rb_node *parent,
|
|
|
|
|
struct rb_node **rb_link)
|
|
|
|
|
{
|
|
|
|
|
node->__rb_parent_color = (unsigned long)parent;
|
|
|
|
|
node->rb_left = node->rb_right = NULL;
|
|
|
|
|
|
|
|
|
|
rcu_assign_pointer(*rb_link, node);
|
|
|
|
|
}
|
|
|
|
|
|
2013-11-13 07:11:19 +08:00
|
|
|
#define rb_entry_safe(ptr, type, member) \
|
|
|
|
|
({ typeof(ptr) ____ptr = (ptr); \
|
|
|
|
|
____ptr ? rb_entry(____ptr, type, member) : NULL; \
|
|
|
|
|
})
|
|
|
|
|
|
2013-09-12 05:25:11 +08:00
|
|
|
/**
|
2015-11-07 08:31:28 +08:00
|
|
|
* rbtree_postorder_for_each_entry_safe - iterate in post-order over rb_root of
|
|
|
|
|
* given type allowing the backing memory of @pos to be invalidated
|
2013-09-12 05:25:11 +08:00
|
|
|
*
|
|
|
|
|
* @pos: the 'type *' to use as a loop cursor.
|
|
|
|
|
* @n: another 'type *' to use as temporary storage
|
|
|
|
|
* @root: 'rb_root *' of the rbtree.
|
|
|
|
|
* @field: the name of the rb_node field within 'type'.
|
2015-11-07 08:31:28 +08:00
|
|
|
*
|
|
|
|
|
* rbtree_postorder_for_each_entry_safe() provides a similar guarantee as
|
|
|
|
|
* list_for_each_entry_safe() and allows the iteration to continue independent
|
|
|
|
|
* of changes to @pos by the body of the loop.
|
|
|
|
|
*
|
|
|
|
|
* Note, however, that it cannot handle other modifications that re-order the
|
|
|
|
|
* rbtree it is iterating over. This includes calling rb_erase() on @pos, as
|
|
|
|
|
* rb_erase() may rebalance the tree, causing us to miss some nodes.
|
2013-09-12 05:25:11 +08:00
|
|
|
*/
|
|
|
|
|
#define rbtree_postorder_for_each_entry_safe(pos, n, root, field) \
|
2013-11-13 07:11:19 +08:00
|
|
|
for (pos = rb_entry_safe(rb_first_postorder(root), typeof(*pos), field); \
|
|
|
|
|
pos && ({ n = rb_entry_safe(rb_next_postorder(&pos->field), \
|
|
|
|
|
typeof(*pos), field); 1; }); \
|
|
|
|
|
pos = n)
|
2013-09-12 05:25:11 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
#endif /* _LINUX_RBTREE_H */
|