2017-06-20 07:28:30 +08:00
|
|
|
#ifndef _MM_PERCPU_INTERNAL_H
|
|
|
|
|
#define _MM_PERCPU_INTERNAL_H
|
|
|
|
|
|
|
|
|
|
#include <linux/types.h>
|
|
|
|
|
#include <linux/percpu.h>
|
|
|
|
|
|
|
|
|
|
struct pcpu_chunk {
|
2017-06-20 07:28:31 +08:00
|
|
|
#ifdef CONFIG_PERCPU_STATS
|
|
|
|
|
int nr_alloc; /* # of allocations */
|
|
|
|
|
size_t max_alloc_size; /* largest allocation size */
|
|
|
|
|
#endif
|
|
|
|
|
|
2017-06-20 07:28:30 +08:00
|
|
|
struct list_head list; /* linked to pcpu_slot lists */
|
percpu: replace area map allocator with bitmap
The percpu memory allocator is experiencing scalability issues when
allocating and freeing large numbers of counters as in BPF.
Additionally, there is a corner case where iteration is triggered over
all chunks if the contig_hint is the right size, but wrong alignment.
This patch replaces the area map allocator with a basic bitmap allocator
implementation. Each subsequent patch will introduce new features and
replace full scanning functions with faster non-scanning options when
possible.
Implementation:
This patchset removes the area map allocator in favor of a bitmap
allocator backed by metadata blocks. The primary goal is to provide
consistency in performance and memory footprint with a focus on small
allocations (< 64 bytes). The bitmap removes the heavy memmove from the
freeing critical path and provides a consistent memory footprint. The
metadata blocks provide a bound on the amount of scanning required by
maintaining a set of hints.
In an effort to make freeing fast, the metadata is updated on the free
path if the new free area makes a page free, a block free, or spans
across blocks. This causes the chunk's contig hint to potentially be
smaller than what it could allocate by up to the smaller of a page or a
block. If the chunk's contig hint is contained within a block, a check
occurs and the hint is kept accurate. Metadata is always kept accurate
on allocation, so there will not be a situation where a chunk has a
later contig hint than available.
Evaluation:
I have primarily done testing against a simple workload of allocation of
1 million objects (2^20) of varying size. Deallocation was done by in
order, alternating, and in reverse. These numbers were collected after
rebasing ontop of a80099a152. I present the worst-case numbers here:
Area Map Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 310 | 4770
16B | 557 | 1325
64B | 436 | 273
256B | 776 | 131
1024B | 3280 | 122
Bitmap Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 490 | 70
16B | 515 | 75
64B | 610 | 80
256B | 950 | 100
1024B | 3520 | 200
This data demonstrates the inability for the area map allocator to
handle less than ideal situations. In the best case of reverse
deallocation, the area map allocator was able to perform within range
of the bitmap allocator. In the worst case situation, freeing took
nearly 5 seconds for 1 million 4-byte objects. The bitmap allocator
dramatically improves the consistency of the free path. The small
allocations performed nearly identical regardless of the freeing
pattern.
While it does add to the allocation latency, the allocation scenario
here is optimal for the area map allocator. The area map allocator runs
into trouble when it is allocating in chunks where the latter half is
full. It is difficult to replicate this, so I present a variant where
the pages are second half filled. Freeing was done sequentially. Below
are the numbers for this scenario:
Area Map Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 4118 | 4892
16B | 1651 | 1163
64B | 598 | 285
256B | 771 | 158
1024B | 3034 | 160
Bitmap Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 481 | 67
16B | 506 | 69
64B | 636 | 75
256B | 892 | 90
1024B | 3262 | 147
The data shows a parabolic curve of performance for the area map
allocator. This is due to the memmove operation being the dominant cost
with the lower object sizes as more objects are packed in a chunk and at
higher object sizes, the traversal of the chunk slots is the dominating
cost. The bitmap allocator suffers this problem as well. The above data
shows the inability to scale for the allocation path with the area map
allocator and that the bitmap allocator demonstrates consistent
performance in general.
The second problem of additional scanning can result in the area map
allocator completing in 52 minutes when trying to allocate 1 million
4-byte objects with 8-byte alignment. The same workload takes
approximately 16 seconds to complete for the bitmap allocator.
V2:
Fixed a bug in pcpu_alloc_first_chunk end_offset was setting the bitmap
using bytes instead of bits.
Added a comment to pcpu_cnt_pop_pages to explain bitmap_weight.
Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
Reviewed-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
2017-07-13 02:27:32 +08:00
|
|
|
int free_bytes; /* free bytes in the chunk */
|
|
|
|
|
int contig_bits; /* max contiguous size hint */
|
2017-06-20 07:28:30 +08:00
|
|
|
void *base_addr; /* base address of this chunk */
|
|
|
|
|
|
percpu: replace area map allocator with bitmap
The percpu memory allocator is experiencing scalability issues when
allocating and freeing large numbers of counters as in BPF.
Additionally, there is a corner case where iteration is triggered over
all chunks if the contig_hint is the right size, but wrong alignment.
This patch replaces the area map allocator with a basic bitmap allocator
implementation. Each subsequent patch will introduce new features and
replace full scanning functions with faster non-scanning options when
possible.
Implementation:
This patchset removes the area map allocator in favor of a bitmap
allocator backed by metadata blocks. The primary goal is to provide
consistency in performance and memory footprint with a focus on small
allocations (< 64 bytes). The bitmap removes the heavy memmove from the
freeing critical path and provides a consistent memory footprint. The
metadata blocks provide a bound on the amount of scanning required by
maintaining a set of hints.
In an effort to make freeing fast, the metadata is updated on the free
path if the new free area makes a page free, a block free, or spans
across blocks. This causes the chunk's contig hint to potentially be
smaller than what it could allocate by up to the smaller of a page or a
block. If the chunk's contig hint is contained within a block, a check
occurs and the hint is kept accurate. Metadata is always kept accurate
on allocation, so there will not be a situation where a chunk has a
later contig hint than available.
Evaluation:
I have primarily done testing against a simple workload of allocation of
1 million objects (2^20) of varying size. Deallocation was done by in
order, alternating, and in reverse. These numbers were collected after
rebasing ontop of a80099a152. I present the worst-case numbers here:
Area Map Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 310 | 4770
16B | 557 | 1325
64B | 436 | 273
256B | 776 | 131
1024B | 3280 | 122
Bitmap Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 490 | 70
16B | 515 | 75
64B | 610 | 80
256B | 950 | 100
1024B | 3520 | 200
This data demonstrates the inability for the area map allocator to
handle less than ideal situations. In the best case of reverse
deallocation, the area map allocator was able to perform within range
of the bitmap allocator. In the worst case situation, freeing took
nearly 5 seconds for 1 million 4-byte objects. The bitmap allocator
dramatically improves the consistency of the free path. The small
allocations performed nearly identical regardless of the freeing
pattern.
While it does add to the allocation latency, the allocation scenario
here is optimal for the area map allocator. The area map allocator runs
into trouble when it is allocating in chunks where the latter half is
full. It is difficult to replicate this, so I present a variant where
the pages are second half filled. Freeing was done sequentially. Below
are the numbers for this scenario:
Area Map Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 4118 | 4892
16B | 1651 | 1163
64B | 598 | 285
256B | 771 | 158
1024B | 3034 | 160
Bitmap Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 481 | 67
16B | 506 | 69
64B | 636 | 75
256B | 892 | 90
1024B | 3262 | 147
The data shows a parabolic curve of performance for the area map
allocator. This is due to the memmove operation being the dominant cost
with the lower object sizes as more objects are packed in a chunk and at
higher object sizes, the traversal of the chunk slots is the dominating
cost. The bitmap allocator suffers this problem as well. The above data
shows the inability to scale for the allocation path with the area map
allocator and that the bitmap allocator demonstrates consistent
performance in general.
The second problem of additional scanning can result in the area map
allocator completing in 52 minutes when trying to allocate 1 million
4-byte objects with 8-byte alignment. The same workload takes
approximately 16 seconds to complete for the bitmap allocator.
V2:
Fixed a bug in pcpu_alloc_first_chunk end_offset was setting the bitmap
using bytes instead of bits.
Added a comment to pcpu_cnt_pop_pages to explain bitmap_weight.
Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
Reviewed-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
2017-07-13 02:27:32 +08:00
|
|
|
unsigned long *alloc_map; /* allocation map */
|
|
|
|
|
unsigned long *bound_map; /* boundary map */
|
2017-06-20 07:28:30 +08:00
|
|
|
|
|
|
|
|
void *data; /* chunk data */
|
|
|
|
|
int first_free; /* no free below this */
|
|
|
|
|
bool immutable; /* no [de]population allowed */
|
2017-07-25 07:01:59 +08:00
|
|
|
int start_offset; /* the overlap with the previous
|
|
|
|
|
region to have a page aligned
|
|
|
|
|
base_addr */
|
2017-07-25 07:02:03 +08:00
|
|
|
int end_offset; /* additional area required to
|
|
|
|
|
have the region end page
|
|
|
|
|
aligned */
|
2017-07-25 07:02:05 +08:00
|
|
|
|
|
|
|
|
int nr_pages; /* # of pages served by this chunk */
|
2017-06-20 07:28:30 +08:00
|
|
|
int nr_populated; /* # of populated pages */
|
2017-07-25 07:02:08 +08:00
|
|
|
int nr_empty_pop_pages; /* # of empty populated pages */
|
2017-06-20 07:28:30 +08:00
|
|
|
unsigned long populated[]; /* populated bitmap */
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
extern spinlock_t pcpu_lock;
|
|
|
|
|
|
|
|
|
|
extern struct list_head *pcpu_slot;
|
|
|
|
|
extern int pcpu_nr_slots;
|
2017-07-16 10:23:08 +08:00
|
|
|
extern int pcpu_nr_empty_pop_pages;
|
2017-06-20 07:28:30 +08:00
|
|
|
|
|
|
|
|
extern struct pcpu_chunk *pcpu_first_chunk;
|
|
|
|
|
extern struct pcpu_chunk *pcpu_reserved_chunk;
|
|
|
|
|
|
percpu: replace area map allocator with bitmap
The percpu memory allocator is experiencing scalability issues when
allocating and freeing large numbers of counters as in BPF.
Additionally, there is a corner case where iteration is triggered over
all chunks if the contig_hint is the right size, but wrong alignment.
This patch replaces the area map allocator with a basic bitmap allocator
implementation. Each subsequent patch will introduce new features and
replace full scanning functions with faster non-scanning options when
possible.
Implementation:
This patchset removes the area map allocator in favor of a bitmap
allocator backed by metadata blocks. The primary goal is to provide
consistency in performance and memory footprint with a focus on small
allocations (< 64 bytes). The bitmap removes the heavy memmove from the
freeing critical path and provides a consistent memory footprint. The
metadata blocks provide a bound on the amount of scanning required by
maintaining a set of hints.
In an effort to make freeing fast, the metadata is updated on the free
path if the new free area makes a page free, a block free, or spans
across blocks. This causes the chunk's contig hint to potentially be
smaller than what it could allocate by up to the smaller of a page or a
block. If the chunk's contig hint is contained within a block, a check
occurs and the hint is kept accurate. Metadata is always kept accurate
on allocation, so there will not be a situation where a chunk has a
later contig hint than available.
Evaluation:
I have primarily done testing against a simple workload of allocation of
1 million objects (2^20) of varying size. Deallocation was done by in
order, alternating, and in reverse. These numbers were collected after
rebasing ontop of a80099a152. I present the worst-case numbers here:
Area Map Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 310 | 4770
16B | 557 | 1325
64B | 436 | 273
256B | 776 | 131
1024B | 3280 | 122
Bitmap Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 490 | 70
16B | 515 | 75
64B | 610 | 80
256B | 950 | 100
1024B | 3520 | 200
This data demonstrates the inability for the area map allocator to
handle less than ideal situations. In the best case of reverse
deallocation, the area map allocator was able to perform within range
of the bitmap allocator. In the worst case situation, freeing took
nearly 5 seconds for 1 million 4-byte objects. The bitmap allocator
dramatically improves the consistency of the free path. The small
allocations performed nearly identical regardless of the freeing
pattern.
While it does add to the allocation latency, the allocation scenario
here is optimal for the area map allocator. The area map allocator runs
into trouble when it is allocating in chunks where the latter half is
full. It is difficult to replicate this, so I present a variant where
the pages are second half filled. Freeing was done sequentially. Below
are the numbers for this scenario:
Area Map Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 4118 | 4892
16B | 1651 | 1163
64B | 598 | 285
256B | 771 | 158
1024B | 3034 | 160
Bitmap Allocator:
Object Size | Alloc Time (ms) | Free Time (ms)
----------------------------------------------
4B | 481 | 67
16B | 506 | 69
64B | 636 | 75
256B | 892 | 90
1024B | 3262 | 147
The data shows a parabolic curve of performance for the area map
allocator. This is due to the memmove operation being the dominant cost
with the lower object sizes as more objects are packed in a chunk and at
higher object sizes, the traversal of the chunk slots is the dominating
cost. The bitmap allocator suffers this problem as well. The above data
shows the inability to scale for the allocation path with the area map
allocator and that the bitmap allocator demonstrates consistent
performance in general.
The second problem of additional scanning can result in the area map
allocator completing in 52 minutes when trying to allocate 1 million
4-byte objects with 8-byte alignment. The same workload takes
approximately 16 seconds to complete for the bitmap allocator.
V2:
Fixed a bug in pcpu_alloc_first_chunk end_offset was setting the bitmap
using bytes instead of bits.
Added a comment to pcpu_cnt_pop_pages to explain bitmap_weight.
Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
Reviewed-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
2017-07-13 02:27:32 +08:00
|
|
|
/**
|
|
|
|
|
* pcpu_nr_pages_to_map_bits - converts the pages to size of bitmap
|
|
|
|
|
* @pages: number of physical pages
|
|
|
|
|
*
|
|
|
|
|
* This conversion is from physical pages to the number of bits
|
|
|
|
|
* required in the bitmap.
|
|
|
|
|
*/
|
|
|
|
|
static inline int pcpu_nr_pages_to_map_bits(int pages)
|
|
|
|
|
{
|
|
|
|
|
return pages * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* pcpu_chunk_map_bits - helper to convert nr_pages to size of bitmap
|
|
|
|
|
* @chunk: chunk of interest
|
|
|
|
|
*
|
|
|
|
|
* This conversion is from the number of physical pages that the chunk
|
|
|
|
|
* serves to the number of bits in the bitmap.
|
|
|
|
|
*/
|
|
|
|
|
static inline int pcpu_chunk_map_bits(struct pcpu_chunk *chunk)
|
|
|
|
|
{
|
|
|
|
|
return pcpu_nr_pages_to_map_bits(chunk->nr_pages);
|
|
|
|
|
}
|
|
|
|
|
|
2017-06-20 07:28:31 +08:00
|
|
|
#ifdef CONFIG_PERCPU_STATS
|
|
|
|
|
|
|
|
|
|
#include <linux/spinlock.h>
|
|
|
|
|
|
|
|
|
|
struct percpu_stats {
|
|
|
|
|
u64 nr_alloc; /* lifetime # of allocations */
|
|
|
|
|
u64 nr_dealloc; /* lifetime # of deallocations */
|
|
|
|
|
u64 nr_cur_alloc; /* current # of allocations */
|
|
|
|
|
u64 nr_max_alloc; /* max # of live allocations */
|
|
|
|
|
u32 nr_chunks; /* current # of live chunks */
|
|
|
|
|
u32 nr_max_chunks; /* max # of live chunks */
|
|
|
|
|
size_t min_alloc_size; /* min allocaiton size */
|
|
|
|
|
size_t max_alloc_size; /* max allocation size */
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
extern struct percpu_stats pcpu_stats;
|
|
|
|
|
extern struct pcpu_alloc_info pcpu_stats_ai;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* For debug purposes. We don't care about the flexible array.
|
|
|
|
|
*/
|
|
|
|
|
static inline void pcpu_stats_save_ai(const struct pcpu_alloc_info *ai)
|
|
|
|
|
{
|
|
|
|
|
memcpy(&pcpu_stats_ai, ai, sizeof(struct pcpu_alloc_info));
|
|
|
|
|
|
|
|
|
|
/* initialize min_alloc_size to unit_size */
|
|
|
|
|
pcpu_stats.min_alloc_size = pcpu_stats_ai.unit_size;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* pcpu_stats_area_alloc - increment area allocation stats
|
|
|
|
|
* @chunk: the location of the area being allocated
|
|
|
|
|
* @size: size of area to allocate in bytes
|
|
|
|
|
*
|
|
|
|
|
* CONTEXT:
|
|
|
|
|
* pcpu_lock.
|
|
|
|
|
*/
|
|
|
|
|
static inline void pcpu_stats_area_alloc(struct pcpu_chunk *chunk, size_t size)
|
|
|
|
|
{
|
|
|
|
|
lockdep_assert_held(&pcpu_lock);
|
|
|
|
|
|
|
|
|
|
pcpu_stats.nr_alloc++;
|
|
|
|
|
pcpu_stats.nr_cur_alloc++;
|
|
|
|
|
pcpu_stats.nr_max_alloc =
|
|
|
|
|
max(pcpu_stats.nr_max_alloc, pcpu_stats.nr_cur_alloc);
|
|
|
|
|
pcpu_stats.min_alloc_size =
|
|
|
|
|
min(pcpu_stats.min_alloc_size, size);
|
|
|
|
|
pcpu_stats.max_alloc_size =
|
|
|
|
|
max(pcpu_stats.max_alloc_size, size);
|
|
|
|
|
|
|
|
|
|
chunk->nr_alloc++;
|
|
|
|
|
chunk->max_alloc_size = max(chunk->max_alloc_size, size);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* pcpu_stats_area_dealloc - decrement allocation stats
|
|
|
|
|
* @chunk: the location of the area being deallocated
|
|
|
|
|
*
|
|
|
|
|
* CONTEXT:
|
|
|
|
|
* pcpu_lock.
|
|
|
|
|
*/
|
|
|
|
|
static inline void pcpu_stats_area_dealloc(struct pcpu_chunk *chunk)
|
|
|
|
|
{
|
|
|
|
|
lockdep_assert_held(&pcpu_lock);
|
|
|
|
|
|
|
|
|
|
pcpu_stats.nr_dealloc++;
|
|
|
|
|
pcpu_stats.nr_cur_alloc--;
|
|
|
|
|
|
|
|
|
|
chunk->nr_alloc--;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* pcpu_stats_chunk_alloc - increment chunk stats
|
|
|
|
|
*/
|
|
|
|
|
static inline void pcpu_stats_chunk_alloc(void)
|
|
|
|
|
{
|
2017-06-22 01:52:46 +08:00
|
|
|
unsigned long flags;
|
|
|
|
|
spin_lock_irqsave(&pcpu_lock, flags);
|
2017-06-20 07:28:31 +08:00
|
|
|
|
|
|
|
|
pcpu_stats.nr_chunks++;
|
|
|
|
|
pcpu_stats.nr_max_chunks =
|
|
|
|
|
max(pcpu_stats.nr_max_chunks, pcpu_stats.nr_chunks);
|
|
|
|
|
|
2017-06-22 01:52:46 +08:00
|
|
|
spin_unlock_irqrestore(&pcpu_lock, flags);
|
2017-06-20 07:28:31 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* pcpu_stats_chunk_dealloc - decrement chunk stats
|
|
|
|
|
*/
|
|
|
|
|
static inline void pcpu_stats_chunk_dealloc(void)
|
|
|
|
|
{
|
2017-06-22 01:52:46 +08:00
|
|
|
unsigned long flags;
|
|
|
|
|
spin_lock_irqsave(&pcpu_lock, flags);
|
2017-06-20 07:28:31 +08:00
|
|
|
|
|
|
|
|
pcpu_stats.nr_chunks--;
|
|
|
|
|
|
2017-06-22 01:52:46 +08:00
|
|
|
spin_unlock_irqrestore(&pcpu_lock, flags);
|
2017-06-20 07:28:31 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#else
|
|
|
|
|
|
|
|
|
|
static inline void pcpu_stats_save_ai(const struct pcpu_alloc_info *ai)
|
|
|
|
|
{
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline void pcpu_stats_area_alloc(struct pcpu_chunk *chunk, size_t size)
|
|
|
|
|
{
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline void pcpu_stats_area_dealloc(struct pcpu_chunk *chunk)
|
|
|
|
|
{
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline void pcpu_stats_chunk_alloc(void)
|
|
|
|
|
{
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline void pcpu_stats_chunk_dealloc(void)
|
|
|
|
|
{
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif /* !CONFIG_PERCPU_STATS */
|
|
|
|
|
|
2017-06-20 07:28:30 +08:00
|
|
|
#endif
|