OpenCloudOS-Kernel/net/smc
Dust Li 349d43127d net/smc: fix kernel panic caused by race of smc_sock
A crash occurs when smc_cdc_tx_handler() tries to access smc_sock
but smc_release() has already freed it.

[ 4570.695099] BUG: unable to handle page fault for address: 000000002eae9e88
[ 4570.696048] #PF: supervisor write access in kernel mode
[ 4570.696728] #PF: error_code(0x0002) - not-present page
[ 4570.697401] PGD 0 P4D 0
[ 4570.697716] Oops: 0002 [#1] PREEMPT SMP NOPTI
[ 4570.698228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc4+ #111
[ 4570.699013] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 8c24b4c 04/0
[ 4570.699933] RIP: 0010:_raw_spin_lock+0x1a/0x30
<...>
[ 4570.711446] Call Trace:
[ 4570.711746]  <IRQ>
[ 4570.711992]  smc_cdc_tx_handler+0x41/0xc0
[ 4570.712470]  smc_wr_tx_tasklet_fn+0x213/0x560
[ 4570.712981]  ? smc_cdc_tx_dismisser+0x10/0x10
[ 4570.713489]  tasklet_action_common.isra.17+0x66/0x140
[ 4570.714083]  __do_softirq+0x123/0x2f4
[ 4570.714521]  irq_exit_rcu+0xc4/0xf0
[ 4570.714934]  common_interrupt+0xba/0xe0

Though smc_cdc_tx_handler() checked the existence of smc connection,
smc_release() may have already dismissed and released the smc socket
before smc_cdc_tx_handler() further visits it.

smc_cdc_tx_handler()           |smc_release()
if (!conn)                     |
                               |
                               |smc_cdc_tx_dismiss_slots()
                               |      smc_cdc_tx_dismisser()
                               |
                               |sock_put(&smc->sk) <- last sock_put,
                               |                      smc_sock freed
bh_lock_sock(&smc->sk) (panic) |

To make sure we won't receive any CDC messages after we free the
smc_sock, add a refcount on the smc_connection for inflight CDC
message(posted to the QP but haven't received related CQE), and
don't release the smc_connection until all the inflight CDC messages
haven been done, for both success or failed ones.

Using refcount on CDC messages brings another problem: when the link
is going to be destroyed, smcr_link_clear() will reset the QP, which
then remove all the pending CQEs related to the QP in the CQ. To make
sure all the CQEs will always come back so the refcount on the
smc_connection can always reach 0, smc_ib_modify_qp_reset() was replaced
by smc_ib_modify_qp_error().
And remove the timeout in smc_wr_tx_wait_no_pending_sends() since we
need to wait for all pending WQEs done, or we may encounter use-after-
free when handling CQEs.

For IB device removal routine, we need to wait for all the QPs on that
device been destroyed before we can destroy CQs on the device, or
the refcount on smc_connection won't reach 0 and smc_sock cannot be
released.

Fixes: 5f08318f61 ("smc: connection data control (CDC)")
Reported-by: Wen Gu <guwen@linux.alibaba.com>
Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-12-28 12:42:45 +00:00
..
Kconfig treewide: replace '---help---' in Kconfig files with 'help' 2020-06-14 01:57:21 +09:00
Makefile net/smc: Introduce tracepoint for fallback 2021-11-01 13:39:14 +00:00
af_smc.c net/smc: Prevent smc_release() from long blocking 2021-12-16 08:11:05 -08:00
smc.h net/smc: fix kernel panic caused by race of smc_sock 2021-12-28 12:42:45 +00:00
smc_cdc.c net/smc: fix kernel panic caused by race of smc_sock 2021-12-28 12:42:45 +00:00
smc_cdc.h net/smc: fix kernel panic caused by race of smc_sock 2021-12-28 12:42:45 +00:00
smc_clc.c net/smc: add v2 format of CLC decline message 2021-10-16 14:58:13 +01:00
smc_clc.h net/smc: add v2 format of CLC decline message 2021-10-16 14:58:13 +01:00
smc_close.c net/smc: Keep smc_close_final rc during active close 2021-12-02 12:14:36 +00:00
smc_close.h net/smc: remove close abort worker 2019-10-22 11:23:44 -07:00
smc_core.c net/smc: fix kernel panic caused by race of smc_sock 2021-12-28 12:42:45 +00:00
smc_core.h net/smc: don't send CDC/LLC message if link not ready 2021-12-28 12:42:45 +00:00
smc_diag.c net/smc: Introduce SMCR get link command 2020-12-01 17:56:13 -08:00
smc_ib.c net/smc: fix kernel panic caused by race of smc_sock 2021-12-28 12:42:45 +00:00
smc_ib.h net/smc: fix kernel panic caused by race of smc_sock 2021-12-28 12:42:45 +00:00
smc_ism.c net/smc: keep static copy of system EID 2021-09-14 12:49:10 +01:00
smc_ism.h net/smc: keep static copy of system EID 2021-09-14 12:49:10 +01:00
smc_llc.c net/smc: don't send CDC/LLC message if link not ready 2021-12-28 12:42:45 +00:00
smc_llc.h net/smc: extend LLC layer for SMC-Rv2 2021-10-16 14:58:13 +01:00
smc_netlink.c net/smc: add generic netlink support for system EID 2021-09-14 12:49:10 +01:00
smc_netlink.h net/smc: add support for user defined EIDs 2021-09-14 12:49:10 +01:00
smc_netns.h net/smc: introduce list of pnetids for Ethernet devices 2020-09-28 15:19:03 -07:00
smc_pnet.c net/smc: retrieve v2 gid from IB device 2021-10-16 14:58:13 +01:00
smc_pnet.h net/smc: determine proposed ISM devices 2020-09-28 15:19:03 -07:00
smc_rx.c net/smc: Introduce tracepoints for tx and rx msg 2021-11-01 13:39:14 +00:00
smc_rx.h smc: add support for splice() 2018-05-04 11:45:06 -04:00
smc_stats.c net/smc: Fix ENODATA tests in smc_nl_get_fback_stats() 2021-06-21 12:16:58 -07:00
smc_stats.h net/smc: Make SMC statistics network namespace aware 2021-06-16 12:54:02 -07:00
smc_tracepoint.c net/smc: Introduce tracepoint for smcr link down 2021-11-01 13:39:14 +00:00
smc_tracepoint.h net/smc: Print function name in smcr_link_down tracepoint 2021-11-05 10:14:38 +00:00
smc_tx.c net/smc: Introduce tracepoints for tx and rx msg 2021-11-01 13:39:14 +00:00
smc_tx.h net/smc: eliminate cursor read and write calls 2018-07-23 10:57:14 -07:00
smc_wr.c net/smc: fix kernel panic caused by race of smc_sock 2021-12-28 12:42:45 +00:00
smc_wr.h net/smc: fix kernel panic caused by race of smc_sock 2021-12-28 12:42:45 +00:00