mm/util: Add folio_mapping() and folio_file_mapping()
These are the folio equivalent of page_mapping() and page_file_mapping().
Add an out-of-line page_mapping() wrapper around folio_mapping()
in order to prevent the page_folio() call from bloating every caller
of page_mapping(). Adjust page_file_mapping() and page_mapping_file()
to use folios internally. Rename __page_file_mapping() to
swapcache_mapping() and change it to take a folio.
This ends up saving 122 bytes of text overall. folio_mapping() is
45 bytes shorter than page_mapping() was, but the new page_mapping()
wrapper is 30 bytes. The major reduction is a few bytes less in dozens
of nfs functions (which call page_file_mapping()). Most of these appear
to be a slight change in gcc's register allocation decisions, which allow:
48 8b 56 08 mov 0x8(%rsi),%rdx
48 8d 42 ff lea -0x1(%rdx),%rax
83 e2 01 and $0x1,%edx
48 0f 44 c6 cmove %rsi,%rax
to become:
48 8b 46 08 mov 0x8(%rsi),%rax
48 8d 78 ff lea -0x1(%rax),%rdi
a8 01 test $0x1,%al
48 0f 44 fe cmove %rsi,%rdi
for a reduction of a single byte. Once the NFS client is converted to
use folios, this entire sequence will disappear.
Also add folio_mapping() documentation.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Jeff Layton <jlayton@kernel.org>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Reviewed-by: David Howells <dhowells@redhat.com>
2020-12-10 23:55:05 +08:00
|
|
|
/*
|
|
|
|
* Compatibility functions which bloat the callers too much to make inline.
|
|
|
|
* All of the callers of these functions should be converted to use folios
|
|
|
|
* eventually.
|
|
|
|
*/
|
|
|
|
|
2021-05-07 19:28:40 +08:00
|
|
|
#include <linux/migrate.h>
|
mm/util: Add folio_mapping() and folio_file_mapping()
These are the folio equivalent of page_mapping() and page_file_mapping().
Add an out-of-line page_mapping() wrapper around folio_mapping()
in order to prevent the page_folio() call from bloating every caller
of page_mapping(). Adjust page_file_mapping() and page_mapping_file()
to use folios internally. Rename __page_file_mapping() to
swapcache_mapping() and change it to take a folio.
This ends up saving 122 bytes of text overall. folio_mapping() is
45 bytes shorter than page_mapping() was, but the new page_mapping()
wrapper is 30 bytes. The major reduction is a few bytes less in dozens
of nfs functions (which call page_file_mapping()). Most of these appear
to be a slight change in gcc's register allocation decisions, which allow:
48 8b 56 08 mov 0x8(%rsi),%rdx
48 8d 42 ff lea -0x1(%rdx),%rax
83 e2 01 and $0x1,%edx
48 0f 44 c6 cmove %rsi,%rax
to become:
48 8b 46 08 mov 0x8(%rsi),%rax
48 8d 78 ff lea -0x1(%rax),%rdi
a8 01 test $0x1,%al
48 0f 44 fe cmove %rsi,%rdi
for a reduction of a single byte. Once the NFS client is converted to
use folios, this entire sequence will disappear.
Also add folio_mapping() documentation.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Jeff Layton <jlayton@kernel.org>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Reviewed-by: David Howells <dhowells@redhat.com>
2020-12-10 23:55:05 +08:00
|
|
|
#include <linux/pagemap.h>
|
2023-01-11 22:28:54 +08:00
|
|
|
#include <linux/rmap.h>
|
2021-04-27 22:47:39 +08:00
|
|
|
#include <linux/swap.h>
|
2021-12-25 02:26:22 +08:00
|
|
|
#include "internal.h"
|
mm/util: Add folio_mapping() and folio_file_mapping()
These are the folio equivalent of page_mapping() and page_file_mapping().
Add an out-of-line page_mapping() wrapper around folio_mapping()
in order to prevent the page_folio() call from bloating every caller
of page_mapping(). Adjust page_file_mapping() and page_mapping_file()
to use folios internally. Rename __page_file_mapping() to
swapcache_mapping() and change it to take a folio.
This ends up saving 122 bytes of text overall. folio_mapping() is
45 bytes shorter than page_mapping() was, but the new page_mapping()
wrapper is 30 bytes. The major reduction is a few bytes less in dozens
of nfs functions (which call page_file_mapping()). Most of these appear
to be a slight change in gcc's register allocation decisions, which allow:
48 8b 56 08 mov 0x8(%rsi),%rdx
48 8d 42 ff lea -0x1(%rdx),%rax
83 e2 01 and $0x1,%edx
48 0f 44 c6 cmove %rsi,%rax
to become:
48 8b 46 08 mov 0x8(%rsi),%rax
48 8d 78 ff lea -0x1(%rax),%rdi
a8 01 test $0x1,%al
48 0f 44 fe cmove %rsi,%rdi
for a reduction of a single byte. Once the NFS client is converted to
use folios, this entire sequence will disappear.
Also add folio_mapping() documentation.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Jeff Layton <jlayton@kernel.org>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Reviewed-by: David Howells <dhowells@redhat.com>
2020-12-10 23:55:05 +08:00
|
|
|
|
|
|
|
struct address_space *page_mapping(struct page *page)
|
|
|
|
{
|
|
|
|
return folio_mapping(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(page_mapping);
|
2020-12-08 04:44:35 +08:00
|
|
|
|
|
|
|
void unlock_page(struct page *page)
|
|
|
|
{
|
|
|
|
return folio_unlock(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(unlock_page);
|
2021-03-04 04:21:55 +08:00
|
|
|
|
|
|
|
void end_page_writeback(struct page *page)
|
|
|
|
{
|
|
|
|
return folio_end_writeback(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(end_page_writeback);
|
2021-03-05 00:09:17 +08:00
|
|
|
|
|
|
|
void wait_on_page_writeback(struct page *page)
|
|
|
|
{
|
|
|
|
return folio_wait_writeback(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(wait_on_page_writeback);
|
2021-03-05 00:25:25 +08:00
|
|
|
|
|
|
|
void wait_for_stable_page(struct page *page)
|
|
|
|
{
|
|
|
|
return folio_wait_stable(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(wait_for_stable_page);
|
2021-04-13 04:45:17 +08:00
|
|
|
|
2021-04-27 22:47:39 +08:00
|
|
|
void mark_page_accessed(struct page *page)
|
|
|
|
{
|
|
|
|
folio_mark_accessed(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(mark_page_accessed);
|
2021-05-07 19:28:40 +08:00
|
|
|
|
2021-04-25 00:00:48 +08:00
|
|
|
bool set_page_writeback(struct page *page)
|
|
|
|
{
|
|
|
|
return folio_start_writeback(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(set_page_writeback);
|
2021-04-27 11:53:10 +08:00
|
|
|
|
|
|
|
bool set_page_dirty(struct page *page)
|
|
|
|
{
|
|
|
|
return folio_mark_dirty(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(set_page_dirty);
|
2021-05-04 11:30:44 +08:00
|
|
|
|
|
|
|
int __set_page_dirty_nobuffers(struct page *page)
|
|
|
|
{
|
|
|
|
return filemap_dirty_folio(page_mapping(page), page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__set_page_dirty_nobuffers);
|
2021-03-01 05:21:20 +08:00
|
|
|
|
|
|
|
bool clear_page_dirty_for_io(struct page *page)
|
|
|
|
{
|
|
|
|
return folio_clear_dirty_for_io(page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(clear_page_dirty_for_io);
|
2021-05-03 11:22:52 +08:00
|
|
|
|
|
|
|
bool redirty_page_for_writepage(struct writeback_control *wbc,
|
|
|
|
struct page *page)
|
|
|
|
{
|
|
|
|
return folio_redirty_for_writepage(wbc, page_folio(page));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(redirty_page_for_writepage);
|
2021-04-29 23:09:31 +08:00
|
|
|
|
2022-09-03 03:46:01 +08:00
|
|
|
void lru_cache_add_inactive_or_unevictable(struct page *page,
|
|
|
|
struct vm_area_struct *vma)
|
|
|
|
{
|
|
|
|
folio_add_lru_vma(page_folio(page), vma);
|
|
|
|
}
|
|
|
|
|
2020-12-08 21:56:28 +08:00
|
|
|
int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
|
|
|
|
pgoff_t index, gfp_t gfp)
|
|
|
|
{
|
|
|
|
return filemap_add_folio(mapping, page_folio(page), index, gfp);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(add_to_page_cache_lru);
|
2021-03-09 00:45:35 +08:00
|
|
|
|
2020-12-25 01:55:56 +08:00
|
|
|
noinline
|
2021-03-09 00:45:35 +08:00
|
|
|
struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
|
|
|
|
int fgp_flags, gfp_t gfp)
|
|
|
|
{
|
|
|
|
struct folio *folio;
|
|
|
|
|
|
|
|
folio = __filemap_get_folio(mapping, index, fgp_flags, gfp);
|
2022-10-20 02:33:32 +08:00
|
|
|
if (!folio || xa_is_value(folio))
|
2021-03-09 00:45:35 +08:00
|
|
|
return &folio->page;
|
|
|
|
return folio_file_page(folio, index);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(pagecache_get_page);
|
2020-12-25 01:55:56 +08:00
|
|
|
|
|
|
|
struct page *grab_cache_page_write_begin(struct address_space *mapping,
|
2022-02-23 00:25:12 +08:00
|
|
|
pgoff_t index)
|
2020-12-25 01:55:56 +08:00
|
|
|
{
|
|
|
|
unsigned fgp_flags = FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE;
|
|
|
|
|
|
|
|
return pagecache_get_page(mapping, index, fgp_flags,
|
|
|
|
mapping_gfp_mask(mapping));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(grab_cache_page_write_begin);
|
2021-05-09 21:33:42 +08:00
|
|
|
|
2021-12-25 02:26:22 +08:00
|
|
|
int isolate_lru_page(struct page *page)
|
|
|
|
{
|
mm: change to return bool for folio_isolate_lru()
Patch series "Change the return value for page isolation functions", v3.
Now the page isolation functions did not return a boolean to indicate
success or not, instead it will return a negative error when failed
to isolate a page. So below code used in most places seem a boolean
success/failure thing, which can confuse people whether the isolation
is successful.
if (folio_isolate_lru(folio))
continue;
Moreover the page isolation functions only return 0 or -EBUSY, and
most users did not care about the negative error except for few users,
thus we can convert all page isolation functions to return a boolean
value, which can remove the confusion to make code more clear.
No functional changes intended in this patch series.
This patch (of 4):
Now the folio_isolate_lru() did not return a boolean value to indicate
isolation success or not, however below code checking the return value can
make people think that it was a boolean success/failure thing, which makes
people easy to make mistakes (see the fix patch[1]).
if (folio_isolate_lru(folio))
continue;
Thus it's better to check the negative error value expilictly returned by
folio_isolate_lru(), which makes code more clear per Linus's
suggestion[2]. Moreover Matthew suggested we can convert the isolation
functions to return a boolean[3], since most users did not care about the
negative error value, and can also remove the confusing of checking return
value.
So this patch converts the folio_isolate_lru() to return a boolean value,
which means return 'true' to indicate the folio isolation is successful,
and 'false' means a failure to isolation. Meanwhile changing all users'
logic of checking the isolation state.
No functional changes intended.
[1] https://lore.kernel.org/all/20230131063206.28820-1-Kuan-Ying.Lee@mediatek.com/T/#u
[2] https://lore.kernel.org/all/CAHk-=wiBrY+O-4=2mrbVyxR+hOqfdJ=Do6xoucfJ9_5az01L4Q@mail.gmail.com/
[3] https://lore.kernel.org/all/Y+sTFqwMNAjDvxw3@casper.infradead.org/
Link: https://lkml.kernel.org/r/cover.1676424378.git.baolin.wang@linux.alibaba.com
Link: https://lkml.kernel.org/r/8a4e3679ed4196168efadf7ea36c038f2f7d5aa9.1676424378.git.baolin.wang@linux.alibaba.com
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: SeongJae Park <sj@kernel.org>
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Naoya Horiguchi <naoya.horiguchi@nec.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeelb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-02-15 18:39:34 +08:00
|
|
|
bool ret;
|
|
|
|
|
2021-12-25 02:26:22 +08:00
|
|
|
if (WARN_RATELIMIT(PageTail(page), "trying to isolate tail page"))
|
|
|
|
return -EBUSY;
|
mm: change to return bool for folio_isolate_lru()
Patch series "Change the return value for page isolation functions", v3.
Now the page isolation functions did not return a boolean to indicate
success or not, instead it will return a negative error when failed
to isolate a page. So below code used in most places seem a boolean
success/failure thing, which can confuse people whether the isolation
is successful.
if (folio_isolate_lru(folio))
continue;
Moreover the page isolation functions only return 0 or -EBUSY, and
most users did not care about the negative error except for few users,
thus we can convert all page isolation functions to return a boolean
value, which can remove the confusion to make code more clear.
No functional changes intended in this patch series.
This patch (of 4):
Now the folio_isolate_lru() did not return a boolean value to indicate
isolation success or not, however below code checking the return value can
make people think that it was a boolean success/failure thing, which makes
people easy to make mistakes (see the fix patch[1]).
if (folio_isolate_lru(folio))
continue;
Thus it's better to check the negative error value expilictly returned by
folio_isolate_lru(), which makes code more clear per Linus's
suggestion[2]. Moreover Matthew suggested we can convert the isolation
functions to return a boolean[3], since most users did not care about the
negative error value, and can also remove the confusing of checking return
value.
So this patch converts the folio_isolate_lru() to return a boolean value,
which means return 'true' to indicate the folio isolation is successful,
and 'false' means a failure to isolation. Meanwhile changing all users'
logic of checking the isolation state.
No functional changes intended.
[1] https://lore.kernel.org/all/20230131063206.28820-1-Kuan-Ying.Lee@mediatek.com/T/#u
[2] https://lore.kernel.org/all/CAHk-=wiBrY+O-4=2mrbVyxR+hOqfdJ=Do6xoucfJ9_5az01L4Q@mail.gmail.com/
[3] https://lore.kernel.org/all/Y+sTFqwMNAjDvxw3@casper.infradead.org/
Link: https://lkml.kernel.org/r/cover.1676424378.git.baolin.wang@linux.alibaba.com
Link: https://lkml.kernel.org/r/8a4e3679ed4196168efadf7ea36c038f2f7d5aa9.1676424378.git.baolin.wang@linux.alibaba.com
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: SeongJae Park <sj@kernel.org>
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Naoya Horiguchi <naoya.horiguchi@nec.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeelb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-02-15 18:39:34 +08:00
|
|
|
ret = folio_isolate_lru((struct folio *)page);
|
|
|
|
if (ret)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
return -EBUSY;
|
2021-12-25 02:26:22 +08:00
|
|
|
}
|
2022-01-21 21:41:46 +08:00
|
|
|
|
|
|
|
void putback_lru_page(struct page *page)
|
|
|
|
{
|
|
|
|
folio_putback_lru(page_folio(page));
|
|
|
|
}
|
2023-01-11 22:28:54 +08:00
|
|
|
|
|
|
|
#ifdef CONFIG_MMU
|
|
|
|
void page_add_new_anon_rmap(struct page *page, struct vm_area_struct *vma,
|
|
|
|
unsigned long address)
|
|
|
|
{
|
|
|
|
VM_BUG_ON_PAGE(PageTail(page), page);
|
|
|
|
|
|
|
|
return folio_add_new_anon_rmap((struct folio *)page, vma, address);
|
|
|
|
}
|
|
|
|
#endif
|