2005-07-08 08:57:13 +08:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2005 Topspin Communications. All rights reserved.
|
2007-03-05 08:15:11 +08:00
|
|
|
* Copyright (c) 2005, 2006, 2007 Cisco Systems. All rights reserved.
|
2005-09-28 06:07:25 +08:00
|
|
|
* Copyright (c) 2005 PathScale, Inc. All rights reserved.
|
2006-02-14 08:31:57 +08:00
|
|
|
* Copyright (c) 2006 Mellanox Technologies. All rights reserved.
|
2005-07-08 08:57:13 +08:00
|
|
|
*
|
|
|
|
* This software is available to you under a choice of one of two
|
|
|
|
* licenses. You may choose to be licensed under the terms of the GNU
|
|
|
|
* General Public License (GPL) Version 2, available from the file
|
|
|
|
* COPYING in the main directory of this source tree, or the
|
|
|
|
* OpenIB.org BSD license below:
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or
|
|
|
|
* without modification, are permitted provided that the following
|
|
|
|
* conditions are met:
|
|
|
|
*
|
|
|
|
* - Redistributions of source code must retain the above
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
* disclaimer.
|
|
|
|
*
|
|
|
|
* - Redistributions in binary form must reproduce the above
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
* disclaimer in the documentation and/or other materials
|
|
|
|
* provided with the distribution.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
|
|
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
|
|
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
|
|
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
|
|
|
|
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
|
|
|
|
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
|
|
|
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
|
|
* SOFTWARE.
|
|
|
|
*/
|
|
|
|
|
2005-09-27 04:53:25 +08:00
|
|
|
#include <linux/file.h>
|
2005-10-29 06:38:26 +08:00
|
|
|
#include <linux/fs.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/slab.h>
|
2014-12-11 23:04:17 +08:00
|
|
|
#include <linux/sched.h>
|
2005-09-27 04:53:25 +08:00
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
#include <asm/uaccess.h>
|
|
|
|
|
|
|
|
#include "uverbs.h"
|
2013-12-13 00:03:17 +08:00
|
|
|
#include "core_priv.h"
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2012-05-01 01:27:26 +08:00
|
|
|
struct uverbs_lock_class {
|
|
|
|
struct lock_class_key key;
|
|
|
|
char name[16];
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct uverbs_lock_class pd_lock_class = { .name = "PD-uobj" };
|
|
|
|
static struct uverbs_lock_class mr_lock_class = { .name = "MR-uobj" };
|
2013-02-07 00:19:13 +08:00
|
|
|
static struct uverbs_lock_class mw_lock_class = { .name = "MW-uobj" };
|
2012-05-01 01:27:26 +08:00
|
|
|
static struct uverbs_lock_class cq_lock_class = { .name = "CQ-uobj" };
|
|
|
|
static struct uverbs_lock_class qp_lock_class = { .name = "QP-uobj" };
|
|
|
|
static struct uverbs_lock_class ah_lock_class = { .name = "AH-uobj" };
|
|
|
|
static struct uverbs_lock_class srq_lock_class = { .name = "SRQ-uobj" };
|
|
|
|
static struct uverbs_lock_class xrcd_lock_class = { .name = "XRCD-uobj" };
|
2013-08-14 18:58:30 +08:00
|
|
|
static struct uverbs_lock_class rule_lock_class = { .name = "RULE-uobj" };
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
/*
|
|
|
|
* The ib_uobject locking scheme is as follows:
|
|
|
|
*
|
|
|
|
* - ib_uverbs_idr_lock protects the uverbs idrs themselves, so it
|
2015-11-03 01:13:25 +08:00
|
|
|
* needs to be held during all idr write operations. When an object is
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
* looked up, a reference must be taken on the object's kref before
|
2015-11-03 01:13:25 +08:00
|
|
|
* dropping this lock. For read operations, the rcu_read_lock()
|
|
|
|
* and rcu_write_lock() but similarly the kref reference is grabbed
|
|
|
|
* before the rcu_read_unlock().
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
*
|
|
|
|
* - Each object also has an rwsem. This rwsem must be held for
|
|
|
|
* reading while an operation that uses the object is performed.
|
|
|
|
* For example, while registering an MR, the associated PD's
|
|
|
|
* uobject.mutex must be held for reading. The rwsem must be held
|
|
|
|
* for writing while initializing or destroying an object.
|
|
|
|
*
|
|
|
|
* - In addition, each object has a "live" flag. If this flag is not
|
|
|
|
* set, then lookups of the object will fail even if it is found in
|
|
|
|
* the idr. This handles a reader that blocks and does not acquire
|
|
|
|
* the rwsem until after the object is destroyed. The destroy
|
|
|
|
* operation will set the live flag to 0 and then drop the rwsem;
|
|
|
|
* this will allow the reader to acquire the rwsem, see that the
|
|
|
|
* live flag is 0, and then drop the rwsem and its reference to
|
|
|
|
* object. The underlying storage will not be freed until the last
|
|
|
|
* reference to the object is dropped.
|
|
|
|
*/
|
|
|
|
|
|
|
|
static void init_uobj(struct ib_uobject *uobj, u64 user_handle,
|
2012-05-01 01:27:26 +08:00
|
|
|
struct ib_ucontext *context, struct uverbs_lock_class *c)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
{
|
|
|
|
uobj->user_handle = user_handle;
|
|
|
|
uobj->context = context;
|
|
|
|
kref_init(&uobj->ref);
|
|
|
|
init_rwsem(&uobj->mutex);
|
2012-05-01 01:27:26 +08:00
|
|
|
lockdep_set_class_and_name(&uobj->mutex, &c->key, c->name);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj->live = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void release_uobj(struct kref *kref)
|
|
|
|
{
|
2015-11-03 01:13:25 +08:00
|
|
|
kfree_rcu(container_of(kref, struct ib_uobject, ref), rcu);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void put_uobj(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
kref_put(&uobj->ref, release_uobj);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void put_uobj_read(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
up_read(&uobj->mutex);
|
|
|
|
put_uobj(uobj);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void put_uobj_write(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
up_write(&uobj->mutex);
|
|
|
|
put_uobj(uobj);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int idr_add_uobj(struct idr *idr, struct ib_uobject *uobj)
|
2006-06-18 11:37:40 +08:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2013-02-28 09:04:16 +08:00
|
|
|
idr_preload(GFP_KERNEL);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
spin_lock(&ib_uverbs_idr_lock);
|
2006-06-18 11:37:40 +08:00
|
|
|
|
2013-02-28 09:04:16 +08:00
|
|
|
ret = idr_alloc(idr, uobj, 0, 0, GFP_NOWAIT);
|
|
|
|
if (ret >= 0)
|
|
|
|
uobj->id = ret;
|
|
|
|
|
|
|
|
spin_unlock(&ib_uverbs_idr_lock);
|
|
|
|
idr_preload_end();
|
2006-06-18 11:37:40 +08:00
|
|
|
|
2013-02-28 09:04:16 +08:00
|
|
|
return ret < 0 ? ret : 0;
|
2006-06-18 11:37:40 +08:00
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
void idr_remove_uobj(struct idr *idr, struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
spin_lock(&ib_uverbs_idr_lock);
|
|
|
|
idr_remove(idr, uobj->id);
|
|
|
|
spin_unlock(&ib_uverbs_idr_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct ib_uobject *__idr_get_uobj(struct idr *idr, int id,
|
|
|
|
struct ib_ucontext *context)
|
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
2015-11-03 01:13:25 +08:00
|
|
|
rcu_read_lock();
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj = idr_find(idr, id);
|
2007-10-20 11:01:43 +08:00
|
|
|
if (uobj) {
|
|
|
|
if (uobj->context == context)
|
|
|
|
kref_get(&uobj->ref);
|
|
|
|
else
|
|
|
|
uobj = NULL;
|
|
|
|
}
|
2015-11-03 01:13:25 +08:00
|
|
|
rcu_read_unlock();
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
|
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct ib_uobject *idr_read_uobj(struct idr *idr, int id,
|
2006-09-23 06:17:20 +08:00
|
|
|
struct ib_ucontext *context, int nested)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
|
|
|
uobj = __idr_get_uobj(idr, id, context);
|
|
|
|
if (!uobj)
|
|
|
|
return NULL;
|
|
|
|
|
2006-09-23 06:17:20 +08:00
|
|
|
if (nested)
|
|
|
|
down_read_nested(&uobj->mutex, SINGLE_DEPTH_NESTING);
|
|
|
|
else
|
|
|
|
down_read(&uobj->mutex);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!uobj->live) {
|
|
|
|
put_uobj_read(uobj);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct ib_uobject *idr_write_uobj(struct idr *idr, int id,
|
|
|
|
struct ib_ucontext *context)
|
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
|
|
|
uobj = __idr_get_uobj(idr, id, context);
|
|
|
|
if (!uobj)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
down_write(&uobj->mutex);
|
|
|
|
if (!uobj->live) {
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
2006-09-23 06:17:20 +08:00
|
|
|
static void *idr_read_obj(struct idr *idr, int id, struct ib_ucontext *context,
|
|
|
|
int nested)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
2006-09-23 06:17:20 +08:00
|
|
|
uobj = idr_read_uobj(idr, id, context, nested);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return uobj ? uobj->object : NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct ib_pd *idr_read_pd(int pd_handle, struct ib_ucontext *context)
|
|
|
|
{
|
2006-09-23 06:17:20 +08:00
|
|
|
return idr_read_obj(&ib_uverbs_pd_idr, pd_handle, context, 0);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void put_pd_read(struct ib_pd *pd)
|
|
|
|
{
|
|
|
|
put_uobj_read(pd->uobject);
|
|
|
|
}
|
|
|
|
|
2006-09-23 06:17:20 +08:00
|
|
|
static struct ib_cq *idr_read_cq(int cq_handle, struct ib_ucontext *context, int nested)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
{
|
2006-09-23 06:17:20 +08:00
|
|
|
return idr_read_obj(&ib_uverbs_cq_idr, cq_handle, context, nested);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void put_cq_read(struct ib_cq *cq)
|
|
|
|
{
|
|
|
|
put_uobj_read(cq->uobject);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct ib_ah *idr_read_ah(int ah_handle, struct ib_ucontext *context)
|
|
|
|
{
|
2006-09-23 06:17:20 +08:00
|
|
|
return idr_read_obj(&ib_uverbs_ah_idr, ah_handle, context, 0);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void put_ah_read(struct ib_ah *ah)
|
|
|
|
{
|
|
|
|
put_uobj_read(ah->uobject);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct ib_qp *idr_read_qp(int qp_handle, struct ib_ucontext *context)
|
|
|
|
{
|
2006-09-23 06:17:20 +08:00
|
|
|
return idr_read_obj(&ib_uverbs_qp_idr, qp_handle, context, 0);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
}
|
|
|
|
|
2012-01-04 12:36:48 +08:00
|
|
|
static struct ib_qp *idr_write_qp(int qp_handle, struct ib_ucontext *context)
|
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
|
|
|
uobj = idr_write_uobj(&ib_uverbs_qp_idr, qp_handle, context);
|
|
|
|
return uobj ? uobj->object : NULL;
|
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
static void put_qp_read(struct ib_qp *qp)
|
|
|
|
{
|
|
|
|
put_uobj_read(qp->uobject);
|
|
|
|
}
|
|
|
|
|
2012-01-04 12:36:48 +08:00
|
|
|
static void put_qp_write(struct ib_qp *qp)
|
|
|
|
{
|
|
|
|
put_uobj_write(qp->uobject);
|
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
static struct ib_srq *idr_read_srq(int srq_handle, struct ib_ucontext *context)
|
|
|
|
{
|
2006-09-23 06:17:20 +08:00
|
|
|
return idr_read_obj(&ib_uverbs_srq_idr, srq_handle, context, 0);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void put_srq_read(struct ib_srq *srq)
|
|
|
|
{
|
|
|
|
put_uobj_read(srq->uobject);
|
|
|
|
}
|
|
|
|
|
2011-05-24 23:33:46 +08:00
|
|
|
static struct ib_xrcd *idr_read_xrcd(int xrcd_handle, struct ib_ucontext *context,
|
|
|
|
struct ib_uobject **uobj)
|
|
|
|
{
|
|
|
|
*uobj = idr_read_uobj(&ib_uverbs_xrcd_idr, xrcd_handle, context, 0);
|
|
|
|
return *uobj ? (*uobj)->object : NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void put_xrcd_read(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
put_uobj_read(uobj);
|
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf,
|
|
|
|
int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_get_context cmd;
|
|
|
|
struct ib_uverbs_get_context_resp resp;
|
|
|
|
struct ib_udata udata;
|
2005-09-27 04:01:03 +08:00
|
|
|
struct ib_ucontext *ucontext;
|
2005-09-27 04:53:25 +08:00
|
|
|
struct file *filp;
|
2005-09-27 04:01:03 +08:00
|
|
|
int ret;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
2005-09-27 04:01:03 +08:00
|
|
|
|
|
|
|
if (file->ucontext) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd,
|
|
|
|
(unsigned long) cmd.response + sizeof resp,
|
|
|
|
in_len - sizeof cmd, out_len - sizeof resp);
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
ucontext = ib_dev->alloc_ucontext(ib_dev, &udata);
|
2006-06-18 11:37:40 +08:00
|
|
|
if (IS_ERR(ucontext)) {
|
2009-12-10 06:30:44 +08:00
|
|
|
ret = PTR_ERR(ucontext);
|
2006-06-18 11:37:40 +08:00
|
|
|
goto err;
|
|
|
|
}
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
ucontext->device = ib_dev;
|
2005-09-27 04:01:03 +08:00
|
|
|
INIT_LIST_HEAD(&ucontext->pd_list);
|
|
|
|
INIT_LIST_HEAD(&ucontext->mr_list);
|
|
|
|
INIT_LIST_HEAD(&ucontext->mw_list);
|
|
|
|
INIT_LIST_HEAD(&ucontext->cq_list);
|
|
|
|
INIT_LIST_HEAD(&ucontext->qp_list);
|
|
|
|
INIT_LIST_HEAD(&ucontext->srq_list);
|
|
|
|
INIT_LIST_HEAD(&ucontext->ah_list);
|
2011-05-24 23:33:46 +08:00
|
|
|
INIT_LIST_HEAD(&ucontext->xrcd_list);
|
2013-08-14 18:58:30 +08:00
|
|
|
INIT_LIST_HEAD(&ucontext->rule_list);
|
2014-12-11 23:04:17 +08:00
|
|
|
rcu_read_lock();
|
|
|
|
ucontext->tgid = get_task_pid(current->group_leader, PIDTYPE_PID);
|
|
|
|
rcu_read_unlock();
|
2007-03-05 08:15:11 +08:00
|
|
|
ucontext->closing = 0;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2014-12-11 23:04:18 +08:00
|
|
|
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
|
|
|
|
ucontext->umem_tree = RB_ROOT;
|
|
|
|
init_rwsem(&ucontext->umem_rwsem);
|
|
|
|
ucontext->odp_mrs_count = 0;
|
|
|
|
INIT_LIST_HEAD(&ucontext->no_private_counters);
|
|
|
|
|
2015-12-18 16:59:45 +08:00
|
|
|
if (!(ib_dev->attrs.device_cap_flags & IB_DEVICE_ON_DEMAND_PAGING))
|
2014-12-11 23:04:18 +08:00
|
|
|
ucontext->invalidate_range = NULL;
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
2005-09-27 04:53:25 +08:00
|
|
|
resp.num_comp_vectors = file->device->num_comp_vectors;
|
|
|
|
|
2013-07-09 02:15:45 +08:00
|
|
|
ret = get_unused_fd_flags(O_CLOEXEC);
|
2010-01-18 14:38:00 +08:00
|
|
|
if (ret < 0)
|
|
|
|
goto err_free;
|
|
|
|
resp.async_fd = ret;
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
filp = ib_uverbs_alloc_event_file(file, ib_dev, 1);
|
2005-09-27 04:53:25 +08:00
|
|
|
if (IS_ERR(filp)) {
|
|
|
|
ret = PTR_ERR(filp);
|
2010-01-18 14:38:00 +08:00
|
|
|
goto err_fd;
|
2005-09-27 04:53:25 +08:00
|
|
|
}
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
2005-09-27 04:01:03 +08:00
|
|
|
&resp, sizeof resp)) {
|
|
|
|
ret = -EFAULT;
|
2005-09-27 04:53:25 +08:00
|
|
|
goto err_file;
|
2005-09-27 04:01:03 +08:00
|
|
|
}
|
|
|
|
|
2005-10-29 06:38:26 +08:00
|
|
|
file->ucontext = ucontext;
|
2005-09-27 04:53:25 +08:00
|
|
|
|
|
|
|
fd_install(resp.async_fd, filp);
|
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
return in_len;
|
|
|
|
|
2005-09-27 04:53:25 +08:00
|
|
|
err_file:
|
2015-08-13 23:32:02 +08:00
|
|
|
ib_uverbs_free_async_event_file(file);
|
2005-09-27 04:53:25 +08:00
|
|
|
fput(filp);
|
|
|
|
|
2010-01-18 14:38:00 +08:00
|
|
|
err_fd:
|
|
|
|
put_unused_fd(resp.async_fd);
|
|
|
|
|
2005-09-27 04:01:03 +08:00
|
|
|
err_free:
|
2014-12-11 23:04:17 +08:00
|
|
|
put_pid(ucontext->tgid);
|
2015-08-13 23:32:04 +08:00
|
|
|
ib_dev->dealloc_ucontext(ucontext);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2005-09-27 04:01:03 +08:00
|
|
|
err:
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-09-27 04:01:03 +08:00
|
|
|
return ret;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2015-02-08 19:28:50 +08:00
|
|
|
static void copy_query_dev_fields(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2015-02-08 19:28:50 +08:00
|
|
|
struct ib_uverbs_query_device_resp *resp,
|
|
|
|
struct ib_device_attr *attr)
|
|
|
|
{
|
|
|
|
resp->fw_ver = attr->fw_ver;
|
2015-08-13 23:32:04 +08:00
|
|
|
resp->node_guid = ib_dev->node_guid;
|
2015-02-08 19:28:50 +08:00
|
|
|
resp->sys_image_guid = attr->sys_image_guid;
|
|
|
|
resp->max_mr_size = attr->max_mr_size;
|
|
|
|
resp->page_size_cap = attr->page_size_cap;
|
|
|
|
resp->vendor_id = attr->vendor_id;
|
|
|
|
resp->vendor_part_id = attr->vendor_part_id;
|
|
|
|
resp->hw_ver = attr->hw_ver;
|
|
|
|
resp->max_qp = attr->max_qp;
|
|
|
|
resp->max_qp_wr = attr->max_qp_wr;
|
|
|
|
resp->device_cap_flags = attr->device_cap_flags;
|
|
|
|
resp->max_sge = attr->max_sge;
|
|
|
|
resp->max_sge_rd = attr->max_sge_rd;
|
|
|
|
resp->max_cq = attr->max_cq;
|
|
|
|
resp->max_cqe = attr->max_cqe;
|
|
|
|
resp->max_mr = attr->max_mr;
|
|
|
|
resp->max_pd = attr->max_pd;
|
|
|
|
resp->max_qp_rd_atom = attr->max_qp_rd_atom;
|
|
|
|
resp->max_ee_rd_atom = attr->max_ee_rd_atom;
|
|
|
|
resp->max_res_rd_atom = attr->max_res_rd_atom;
|
|
|
|
resp->max_qp_init_rd_atom = attr->max_qp_init_rd_atom;
|
|
|
|
resp->max_ee_init_rd_atom = attr->max_ee_init_rd_atom;
|
|
|
|
resp->atomic_cap = attr->atomic_cap;
|
|
|
|
resp->max_ee = attr->max_ee;
|
|
|
|
resp->max_rdd = attr->max_rdd;
|
|
|
|
resp->max_mw = attr->max_mw;
|
|
|
|
resp->max_raw_ipv6_qp = attr->max_raw_ipv6_qp;
|
|
|
|
resp->max_raw_ethy_qp = attr->max_raw_ethy_qp;
|
|
|
|
resp->max_mcast_grp = attr->max_mcast_grp;
|
|
|
|
resp->max_mcast_qp_attach = attr->max_mcast_qp_attach;
|
|
|
|
resp->max_total_mcast_qp_attach = attr->max_total_mcast_qp_attach;
|
|
|
|
resp->max_ah = attr->max_ah;
|
|
|
|
resp->max_fmr = attr->max_fmr;
|
|
|
|
resp->max_map_per_fmr = attr->max_map_per_fmr;
|
|
|
|
resp->max_srq = attr->max_srq;
|
|
|
|
resp->max_srq_wr = attr->max_srq_wr;
|
|
|
|
resp->max_srq_sge = attr->max_srq_sge;
|
|
|
|
resp->max_pkeys = attr->max_pkeys;
|
|
|
|
resp->local_ca_ack_delay = attr->local_ca_ack_delay;
|
2015-08-13 23:32:04 +08:00
|
|
|
resp->phys_port_cnt = ib_dev->phys_port_cnt;
|
2015-02-08 19:28:50 +08:00
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
ssize_t ib_uverbs_query_device(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf,
|
|
|
|
int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_query_device cmd;
|
|
|
|
struct ib_uverbs_query_device_resp resp;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
memset(&resp, 0, sizeof resp);
|
2015-12-18 16:59:45 +08:00
|
|
|
copy_query_dev_fields(file, ib_dev, &resp, &ib_dev->attrs);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_query_port(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf,
|
|
|
|
int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_query_port cmd;
|
|
|
|
struct ib_uverbs_query_port_resp resp;
|
|
|
|
struct ib_port_attr attr;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
ret = ib_query_port(ib_dev, cmd.port_num, &attr);
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
|
|
|
|
resp.state = attr.state;
|
|
|
|
resp.max_mtu = attr.max_mtu;
|
|
|
|
resp.active_mtu = attr.active_mtu;
|
|
|
|
resp.gid_tbl_len = attr.gid_tbl_len;
|
|
|
|
resp.port_cap_flags = attr.port_cap_flags;
|
|
|
|
resp.max_msg_sz = attr.max_msg_sz;
|
|
|
|
resp.bad_pkey_cntr = attr.bad_pkey_cntr;
|
|
|
|
resp.qkey_viol_cntr = attr.qkey_viol_cntr;
|
|
|
|
resp.pkey_tbl_len = attr.pkey_tbl_len;
|
|
|
|
resp.lid = attr.lid;
|
|
|
|
resp.sm_lid = attr.sm_lid;
|
|
|
|
resp.lmc = attr.lmc;
|
|
|
|
resp.max_vl_num = attr.max_vl_num;
|
|
|
|
resp.sm_sl = attr.sm_sl;
|
|
|
|
resp.subnet_timeout = attr.subnet_timeout;
|
|
|
|
resp.init_type_reply = attr.init_type_reply;
|
|
|
|
resp.active_width = attr.active_width;
|
|
|
|
resp.active_speed = attr.active_speed;
|
|
|
|
resp.phys_state = attr.phys_state;
|
2015-08-13 23:32:04 +08:00
|
|
|
resp.link_layer = rdma_port_get_link_layer(ib_dev,
|
2010-10-19 05:45:20 +08:00
|
|
|
cmd.port_num);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf,
|
|
|
|
int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_alloc_pd cmd;
|
|
|
|
struct ib_uverbs_alloc_pd_resp resp;
|
|
|
|
struct ib_udata udata;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
struct ib_pd *pd;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd,
|
|
|
|
(unsigned long) cmd.response + sizeof resp,
|
|
|
|
in_len - sizeof cmd, out_len - sizeof resp);
|
|
|
|
|
|
|
|
uobj = kmalloc(sizeof *uobj, GFP_KERNEL);
|
|
|
|
if (!uobj)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2012-05-01 01:27:26 +08:00
|
|
|
init_uobj(uobj, 0, file->ucontext, &pd_lock_class);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
down_write(&uobj->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
pd = ib_dev->alloc_pd(ib_dev, file->ucontext, &udata);
|
2005-07-08 08:57:13 +08:00
|
|
|
if (IS_ERR(pd)) {
|
|
|
|
ret = PTR_ERR(pd);
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
pd->device = ib_dev;
|
2005-07-08 08:57:13 +08:00
|
|
|
pd->uobject = uobj;
|
2015-08-06 04:14:45 +08:00
|
|
|
pd->local_mr = NULL;
|
2005-07-08 08:57:13 +08:00
|
|
|
atomic_set(&pd->usecnt, 0);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj->object = pd;
|
|
|
|
ret = idr_add_uobj(&ib_uverbs_pd_idr, uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_idr;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
resp.pd_handle = uobj->id;
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp)) {
|
|
|
|
ret = -EFAULT;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_copy;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
2005-09-28 06:07:25 +08:00
|
|
|
list_add_tail(&uobj->list, &file->ucontext->pd_list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj->live = 1;
|
|
|
|
|
|
|
|
up_write(&uobj->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2005-09-28 06:07:25 +08:00
|
|
|
return in_len;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_copy:
|
|
|
|
idr_remove_uobj(&ib_uverbs_pd_idr, uobj);
|
2005-09-28 06:07:25 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_idr:
|
2005-07-08 08:57:13 +08:00
|
|
|
ib_dealloc_pd(pd);
|
|
|
|
|
|
|
|
err:
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj_write(uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf,
|
|
|
|
int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_dealloc_pd cmd;
|
|
|
|
struct ib_uobject *uobj;
|
2015-08-06 04:34:31 +08:00
|
|
|
struct ib_pd *pd;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
int ret;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj = idr_write_uobj(&ib_uverbs_pd_idr, cmd.pd_handle, file->ucontext);
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
2015-08-06 04:34:31 +08:00
|
|
|
pd = uobj->object;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-08-06 04:34:31 +08:00
|
|
|
if (atomic_read(&pd->usecnt)) {
|
|
|
|
ret = -EBUSY;
|
|
|
|
goto err_put;
|
|
|
|
}
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-08-06 04:34:31 +08:00
|
|
|
ret = pd->device->dealloc_pd(uobj->object);
|
|
|
|
WARN_ONCE(ret, "Infiniband HW driver failed dealloc_pd");
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
2015-08-06 04:34:31 +08:00
|
|
|
goto err_put;
|
|
|
|
|
|
|
|
uobj->live = 0;
|
|
|
|
put_uobj_write(uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_pd_idr, uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
list_del(&uobj->list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj(uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return in_len;
|
2015-08-06 04:34:31 +08:00
|
|
|
|
|
|
|
err_put:
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
return ret;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2011-05-24 23:33:46 +08:00
|
|
|
struct xrcd_table_entry {
|
|
|
|
struct rb_node node;
|
|
|
|
struct ib_xrcd *xrcd;
|
|
|
|
struct inode *inode;
|
|
|
|
};
|
|
|
|
|
|
|
|
static int xrcd_table_insert(struct ib_uverbs_device *dev,
|
|
|
|
struct inode *inode,
|
|
|
|
struct ib_xrcd *xrcd)
|
|
|
|
{
|
|
|
|
struct xrcd_table_entry *entry, *scan;
|
|
|
|
struct rb_node **p = &dev->xrcd_tree.rb_node;
|
|
|
|
struct rb_node *parent = NULL;
|
|
|
|
|
|
|
|
entry = kmalloc(sizeof *entry, GFP_KERNEL);
|
|
|
|
if (!entry)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
entry->xrcd = xrcd;
|
|
|
|
entry->inode = inode;
|
|
|
|
|
|
|
|
while (*p) {
|
|
|
|
parent = *p;
|
|
|
|
scan = rb_entry(parent, struct xrcd_table_entry, node);
|
|
|
|
|
|
|
|
if (inode < scan->inode) {
|
|
|
|
p = &(*p)->rb_left;
|
|
|
|
} else if (inode > scan->inode) {
|
|
|
|
p = &(*p)->rb_right;
|
|
|
|
} else {
|
|
|
|
kfree(entry);
|
|
|
|
return -EEXIST;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
rb_link_node(&entry->node, parent, p);
|
|
|
|
rb_insert_color(&entry->node, &dev->xrcd_tree);
|
|
|
|
igrab(inode);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct xrcd_table_entry *xrcd_table_search(struct ib_uverbs_device *dev,
|
|
|
|
struct inode *inode)
|
|
|
|
{
|
|
|
|
struct xrcd_table_entry *entry;
|
|
|
|
struct rb_node *p = dev->xrcd_tree.rb_node;
|
|
|
|
|
|
|
|
while (p) {
|
|
|
|
entry = rb_entry(p, struct xrcd_table_entry, node);
|
|
|
|
|
|
|
|
if (inode < entry->inode)
|
|
|
|
p = p->rb_left;
|
|
|
|
else if (inode > entry->inode)
|
|
|
|
p = p->rb_right;
|
|
|
|
else
|
|
|
|
return entry;
|
|
|
|
}
|
|
|
|
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct ib_xrcd *find_xrcd(struct ib_uverbs_device *dev, struct inode *inode)
|
|
|
|
{
|
|
|
|
struct xrcd_table_entry *entry;
|
|
|
|
|
|
|
|
entry = xrcd_table_search(dev, inode);
|
|
|
|
if (!entry)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
return entry->xrcd;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void xrcd_table_delete(struct ib_uverbs_device *dev,
|
|
|
|
struct inode *inode)
|
|
|
|
{
|
|
|
|
struct xrcd_table_entry *entry;
|
|
|
|
|
|
|
|
entry = xrcd_table_search(dev, inode);
|
|
|
|
if (entry) {
|
|
|
|
iput(inode);
|
|
|
|
rb_erase(&entry->node, &dev->xrcd_tree);
|
|
|
|
kfree(entry);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2011-05-24 23:33:46 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_open_xrcd cmd;
|
|
|
|
struct ib_uverbs_open_xrcd_resp resp;
|
|
|
|
struct ib_udata udata;
|
|
|
|
struct ib_uxrcd_object *obj;
|
|
|
|
struct ib_xrcd *xrcd = NULL;
|
2012-08-29 00:52:22 +08:00
|
|
|
struct fd f = {NULL, 0};
|
2011-05-24 23:33:46 +08:00
|
|
|
struct inode *inode = NULL;
|
2012-08-29 00:52:22 +08:00
|
|
|
int ret = 0;
|
2011-05-24 23:33:46 +08:00
|
|
|
int new_xrcd = 0;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd,
|
|
|
|
(unsigned long) cmd.response + sizeof resp,
|
|
|
|
in_len - sizeof cmd, out_len - sizeof resp);
|
|
|
|
|
|
|
|
mutex_lock(&file->device->xrcd_tree_mutex);
|
|
|
|
|
|
|
|
if (cmd.fd != -1) {
|
|
|
|
/* search for file descriptor */
|
2012-08-29 00:52:22 +08:00
|
|
|
f = fdget(cmd.fd);
|
|
|
|
if (!f.file) {
|
2011-05-24 23:33:46 +08:00
|
|
|
ret = -EBADF;
|
|
|
|
goto err_tree_mutex_unlock;
|
|
|
|
}
|
|
|
|
|
2013-01-24 06:07:38 +08:00
|
|
|
inode = file_inode(f.file);
|
2011-05-24 23:33:46 +08:00
|
|
|
xrcd = find_xrcd(file->device, inode);
|
|
|
|
if (!xrcd && !(cmd.oflags & O_CREAT)) {
|
|
|
|
/* no file descriptor. Need CREATE flag */
|
|
|
|
ret = -EAGAIN;
|
|
|
|
goto err_tree_mutex_unlock;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (xrcd && cmd.oflags & O_EXCL) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_tree_mutex_unlock;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
obj = kmalloc(sizeof *obj, GFP_KERNEL);
|
|
|
|
if (!obj) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto err_tree_mutex_unlock;
|
|
|
|
}
|
|
|
|
|
2012-05-01 01:27:26 +08:00
|
|
|
init_uobj(&obj->uobject, 0, file->ucontext, &xrcd_lock_class);
|
2011-05-24 23:33:46 +08:00
|
|
|
|
|
|
|
down_write(&obj->uobject.mutex);
|
|
|
|
|
|
|
|
if (!xrcd) {
|
2015-08-13 23:32:04 +08:00
|
|
|
xrcd = ib_dev->alloc_xrcd(ib_dev, file->ucontext, &udata);
|
2011-05-24 23:33:46 +08:00
|
|
|
if (IS_ERR(xrcd)) {
|
|
|
|
ret = PTR_ERR(xrcd);
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
|
|
|
xrcd->inode = inode;
|
2015-08-13 23:32:04 +08:00
|
|
|
xrcd->device = ib_dev;
|
2011-05-24 23:33:46 +08:00
|
|
|
atomic_set(&xrcd->usecnt, 0);
|
|
|
|
mutex_init(&xrcd->tgt_qp_mutex);
|
|
|
|
INIT_LIST_HEAD(&xrcd->tgt_qp_list);
|
|
|
|
new_xrcd = 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
atomic_set(&obj->refcnt, 0);
|
|
|
|
obj->uobject.object = xrcd;
|
|
|
|
ret = idr_add_uobj(&ib_uverbs_xrcd_idr, &obj->uobject);
|
|
|
|
if (ret)
|
|
|
|
goto err_idr;
|
|
|
|
|
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
resp.xrcd_handle = obj->uobject.id;
|
|
|
|
|
|
|
|
if (inode) {
|
|
|
|
if (new_xrcd) {
|
|
|
|
/* create new inode/xrcd table entry */
|
|
|
|
ret = xrcd_table_insert(file->device, inode, xrcd);
|
|
|
|
if (ret)
|
|
|
|
goto err_insert_xrcd;
|
|
|
|
}
|
|
|
|
atomic_inc(&xrcd->usecnt);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp)) {
|
|
|
|
ret = -EFAULT;
|
|
|
|
goto err_copy;
|
|
|
|
}
|
|
|
|
|
2012-08-29 00:52:22 +08:00
|
|
|
if (f.file)
|
|
|
|
fdput(f);
|
2011-05-24 23:33:46 +08:00
|
|
|
|
|
|
|
mutex_lock(&file->mutex);
|
|
|
|
list_add_tail(&obj->uobject.list, &file->ucontext->xrcd_list);
|
|
|
|
mutex_unlock(&file->mutex);
|
|
|
|
|
|
|
|
obj->uobject.live = 1;
|
|
|
|
up_write(&obj->uobject.mutex);
|
|
|
|
|
|
|
|
mutex_unlock(&file->device->xrcd_tree_mutex);
|
|
|
|
return in_len;
|
|
|
|
|
|
|
|
err_copy:
|
|
|
|
if (inode) {
|
|
|
|
if (new_xrcd)
|
|
|
|
xrcd_table_delete(file->device, inode);
|
|
|
|
atomic_dec(&xrcd->usecnt);
|
|
|
|
}
|
|
|
|
|
|
|
|
err_insert_xrcd:
|
|
|
|
idr_remove_uobj(&ib_uverbs_xrcd_idr, &obj->uobject);
|
|
|
|
|
|
|
|
err_idr:
|
|
|
|
ib_dealloc_xrcd(xrcd);
|
|
|
|
|
|
|
|
err:
|
|
|
|
put_uobj_write(&obj->uobject);
|
|
|
|
|
|
|
|
err_tree_mutex_unlock:
|
2012-08-29 00:52:22 +08:00
|
|
|
if (f.file)
|
|
|
|
fdput(f);
|
2011-05-24 23:33:46 +08:00
|
|
|
|
|
|
|
mutex_unlock(&file->device->xrcd_tree_mutex);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_close_xrcd(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2011-05-24 23:33:46 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_close_xrcd cmd;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
struct ib_xrcd *xrcd = NULL;
|
|
|
|
struct inode *inode = NULL;
|
|
|
|
struct ib_uxrcd_object *obj;
|
|
|
|
int live;
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
mutex_lock(&file->device->xrcd_tree_mutex);
|
|
|
|
uobj = idr_write_uobj(&ib_uverbs_xrcd_idr, cmd.xrcd_handle, file->ucontext);
|
|
|
|
if (!uobj) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
xrcd = uobj->object;
|
|
|
|
inode = xrcd->inode;
|
|
|
|
obj = container_of(uobj, struct ib_uxrcd_object, uobject);
|
|
|
|
if (atomic_read(&obj->refcnt)) {
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
ret = -EBUSY;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!inode || atomic_dec_and_test(&xrcd->usecnt)) {
|
|
|
|
ret = ib_dealloc_xrcd(uobj->object);
|
|
|
|
if (!ret)
|
|
|
|
uobj->live = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
live = uobj->live;
|
|
|
|
if (inode && ret)
|
|
|
|
atomic_inc(&xrcd->usecnt);
|
|
|
|
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
|
|
|
|
if (ret)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (inode && !live)
|
|
|
|
xrcd_table_delete(file->device, inode);
|
|
|
|
|
|
|
|
idr_remove_uobj(&ib_uverbs_xrcd_idr, uobj);
|
|
|
|
mutex_lock(&file->mutex);
|
|
|
|
list_del(&uobj->list);
|
|
|
|
mutex_unlock(&file->mutex);
|
|
|
|
|
|
|
|
put_uobj(uobj);
|
|
|
|
ret = in_len;
|
|
|
|
|
|
|
|
out:
|
|
|
|
mutex_unlock(&file->device->xrcd_tree_mutex);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
void ib_uverbs_dealloc_xrcd(struct ib_uverbs_device *dev,
|
|
|
|
struct ib_xrcd *xrcd)
|
|
|
|
{
|
|
|
|
struct inode *inode;
|
|
|
|
|
|
|
|
inode = xrcd->inode;
|
|
|
|
if (inode && !atomic_dec_and_test(&xrcd->usecnt))
|
|
|
|
return;
|
|
|
|
|
|
|
|
ib_dealloc_xrcd(xrcd);
|
|
|
|
|
|
|
|
if (inode)
|
|
|
|
xrcd_table_delete(dev, inode);
|
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
ssize_t ib_uverbs_reg_mr(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_reg_mr cmd;
|
|
|
|
struct ib_uverbs_reg_mr_resp resp;
|
|
|
|
struct ib_udata udata;
|
2007-03-05 08:15:11 +08:00
|
|
|
struct ib_uobject *uobj;
|
2005-07-08 08:57:13 +08:00
|
|
|
struct ib_pd *pd;
|
|
|
|
struct ib_mr *mr;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd,
|
|
|
|
(unsigned long) cmd.response + sizeof resp,
|
|
|
|
in_len - sizeof cmd, out_len - sizeof resp);
|
|
|
|
|
|
|
|
if ((cmd.start & ~PAGE_MASK) != (cmd.hca_va & ~PAGE_MASK))
|
|
|
|
return -EINVAL;
|
|
|
|
|
2013-10-31 21:26:32 +08:00
|
|
|
ret = ib_check_mr_access(cmd.access_flags);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
2005-10-04 00:18:02 +08:00
|
|
|
|
2007-03-05 08:15:11 +08:00
|
|
|
uobj = kmalloc(sizeof *uobj, GFP_KERNEL);
|
|
|
|
if (!uobj)
|
2005-07-08 08:57:13 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
|
2012-05-01 01:27:26 +08:00
|
|
|
init_uobj(uobj, 0, file->ucontext, &mr_lock_class);
|
2007-03-05 08:15:11 +08:00
|
|
|
down_write(&uobj->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
pd = idr_read_pd(cmd.pd_handle, file->ucontext);
|
2007-02-23 05:16:51 +08:00
|
|
|
if (!pd) {
|
|
|
|
ret = -EINVAL;
|
2007-03-05 08:15:11 +08:00
|
|
|
goto err_free;
|
2007-02-23 05:16:51 +08:00
|
|
|
}
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2014-12-11 23:04:16 +08:00
|
|
|
if (cmd.access_flags & IB_ACCESS_ON_DEMAND) {
|
2015-12-18 16:59:45 +08:00
|
|
|
if (!(pd->device->attrs.device_cap_flags &
|
|
|
|
IB_DEVICE_ON_DEMAND_PAGING)) {
|
2014-12-11 23:04:16 +08:00
|
|
|
pr_debug("ODP support not available\n");
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-03-05 08:15:11 +08:00
|
|
|
mr = pd->device->reg_user_mr(pd, cmd.start, cmd.length, cmd.hca_va,
|
|
|
|
cmd.access_flags, &udata);
|
2005-07-08 08:57:13 +08:00
|
|
|
if (IS_ERR(mr)) {
|
|
|
|
ret = PTR_ERR(mr);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_put;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
mr->device = pd->device;
|
|
|
|
mr->pd = pd;
|
2007-03-05 08:15:11 +08:00
|
|
|
mr->uobject = uobj;
|
2005-07-08 08:57:13 +08:00
|
|
|
atomic_inc(&pd->usecnt);
|
|
|
|
|
2007-03-05 08:15:11 +08:00
|
|
|
uobj->object = mr;
|
|
|
|
ret = idr_add_uobj(&ib_uverbs_mr_idr, uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
|
|
|
goto err_unreg;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
resp.lkey = mr->lkey;
|
|
|
|
resp.rkey = mr->rkey;
|
2007-03-05 08:15:11 +08:00
|
|
|
resp.mr_handle = uobj->id;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp)) {
|
|
|
|
ret = -EFAULT;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_copy;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_pd_read(pd);
|
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
2007-03-05 08:15:11 +08:00
|
|
|
list_add_tail(&uobj->list, &file->ucontext->mr_list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-09-28 06:07:25 +08:00
|
|
|
|
2007-03-05 08:15:11 +08:00
|
|
|
uobj->live = 1;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
|
2007-03-05 08:15:11 +08:00
|
|
|
up_write(&uobj->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
return in_len;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_copy:
|
2007-03-05 08:15:11 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_mr_idr, uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
err_unreg:
|
|
|
|
ib_dereg_mr(mr);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_put:
|
|
|
|
put_pd_read(pd);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
err_free:
|
2007-03-05 08:15:11 +08:00
|
|
|
put_uobj_write(uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2014-07-31 16:01:28 +08:00
|
|
|
ssize_t ib_uverbs_rereg_mr(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2014-07-31 16:01:28 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_rereg_mr cmd;
|
|
|
|
struct ib_uverbs_rereg_mr_resp resp;
|
|
|
|
struct ib_udata udata;
|
|
|
|
struct ib_pd *pd = NULL;
|
|
|
|
struct ib_mr *mr;
|
|
|
|
struct ib_pd *old_pd;
|
|
|
|
int ret;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
|
|
|
if (out_len < sizeof(resp))
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof(cmd)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
INIT_UDATA(&udata, buf + sizeof(cmd),
|
|
|
|
(unsigned long) cmd.response + sizeof(resp),
|
|
|
|
in_len - sizeof(cmd), out_len - sizeof(resp));
|
|
|
|
|
|
|
|
if (cmd.flags & ~IB_MR_REREG_SUPPORTED || !cmd.flags)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if ((cmd.flags & IB_MR_REREG_TRANS) &&
|
|
|
|
(!cmd.start || !cmd.hca_va || 0 >= cmd.length ||
|
|
|
|
(cmd.start & ~PAGE_MASK) != (cmd.hca_va & ~PAGE_MASK)))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
uobj = idr_write_uobj(&ib_uverbs_mr_idr, cmd.mr_handle,
|
|
|
|
file->ucontext);
|
|
|
|
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
mr = uobj->object;
|
|
|
|
|
|
|
|
if (cmd.flags & IB_MR_REREG_ACCESS) {
|
|
|
|
ret = ib_check_mr_access(cmd.access_flags);
|
|
|
|
if (ret)
|
|
|
|
goto put_uobjs;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (cmd.flags & IB_MR_REREG_PD) {
|
|
|
|
pd = idr_read_pd(cmd.pd_handle, file->ucontext);
|
|
|
|
if (!pd) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto put_uobjs;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
old_pd = mr->pd;
|
|
|
|
ret = mr->device->rereg_user_mr(mr, cmd.flags, cmd.start,
|
|
|
|
cmd.length, cmd.hca_va,
|
|
|
|
cmd.access_flags, pd, &udata);
|
|
|
|
if (!ret) {
|
|
|
|
if (cmd.flags & IB_MR_REREG_PD) {
|
|
|
|
atomic_inc(&pd->usecnt);
|
|
|
|
mr->pd = pd;
|
|
|
|
atomic_dec(&old_pd->usecnt);
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
goto put_uobj_pd;
|
|
|
|
}
|
|
|
|
|
|
|
|
memset(&resp, 0, sizeof(resp));
|
|
|
|
resp.lkey = mr->lkey;
|
|
|
|
resp.rkey = mr->rkey;
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *)(unsigned long)cmd.response,
|
|
|
|
&resp, sizeof(resp)))
|
|
|
|
ret = -EFAULT;
|
|
|
|
else
|
|
|
|
ret = in_len;
|
|
|
|
|
|
|
|
put_uobj_pd:
|
|
|
|
if (cmd.flags & IB_MR_REREG_PD)
|
|
|
|
put_pd_read(pd);
|
|
|
|
|
|
|
|
put_uobjs:
|
|
|
|
|
|
|
|
put_uobj_write(mr->uobject);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
ssize_t ib_uverbs_dereg_mr(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_dereg_mr cmd;
|
|
|
|
struct ib_mr *mr;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_uobject *uobj;
|
2005-07-08 08:57:13 +08:00
|
|
|
int ret = -EINVAL;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj = idr_write_uobj(&ib_uverbs_mr_idr, cmd.mr_handle, file->ucontext);
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2007-03-05 08:15:11 +08:00
|
|
|
mr = uobj->object;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
ret = ib_dereg_mr(mr);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!ret)
|
|
|
|
uobj->live = 0;
|
|
|
|
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return ret;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_mr_idr, uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
list_del(&uobj->list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj(uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return in_len;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2013-02-07 00:19:13 +08:00
|
|
|
ssize_t ib_uverbs_alloc_mw(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
2013-02-07 00:19:13 +08:00
|
|
|
{
|
|
|
|
struct ib_uverbs_alloc_mw cmd;
|
|
|
|
struct ib_uverbs_alloc_mw_resp resp;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
struct ib_pd *pd;
|
|
|
|
struct ib_mw *mw;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof(resp))
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof(cmd)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
uobj = kmalloc(sizeof(*uobj), GFP_KERNEL);
|
|
|
|
if (!uobj)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
init_uobj(uobj, 0, file->ucontext, &mw_lock_class);
|
|
|
|
down_write(&uobj->mutex);
|
|
|
|
|
|
|
|
pd = idr_read_pd(cmd.pd_handle, file->ucontext);
|
|
|
|
if (!pd) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_free;
|
|
|
|
}
|
|
|
|
|
|
|
|
mw = pd->device->alloc_mw(pd, cmd.mw_type);
|
|
|
|
if (IS_ERR(mw)) {
|
|
|
|
ret = PTR_ERR(mw);
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
mw->device = pd->device;
|
|
|
|
mw->pd = pd;
|
|
|
|
mw->uobject = uobj;
|
|
|
|
atomic_inc(&pd->usecnt);
|
|
|
|
|
|
|
|
uobj->object = mw;
|
|
|
|
ret = idr_add_uobj(&ib_uverbs_mw_idr, uobj);
|
|
|
|
if (ret)
|
|
|
|
goto err_unalloc;
|
|
|
|
|
|
|
|
memset(&resp, 0, sizeof(resp));
|
|
|
|
resp.rkey = mw->rkey;
|
|
|
|
resp.mw_handle = uobj->id;
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *)(unsigned long)cmd.response,
|
|
|
|
&resp, sizeof(resp))) {
|
|
|
|
ret = -EFAULT;
|
|
|
|
goto err_copy;
|
|
|
|
}
|
|
|
|
|
|
|
|
put_pd_read(pd);
|
|
|
|
|
|
|
|
mutex_lock(&file->mutex);
|
|
|
|
list_add_tail(&uobj->list, &file->ucontext->mw_list);
|
|
|
|
mutex_unlock(&file->mutex);
|
|
|
|
|
|
|
|
uobj->live = 1;
|
|
|
|
|
|
|
|
up_write(&uobj->mutex);
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
|
|
|
|
err_copy:
|
|
|
|
idr_remove_uobj(&ib_uverbs_mw_idr, uobj);
|
|
|
|
|
|
|
|
err_unalloc:
|
2015-12-24 02:12:48 +08:00
|
|
|
uverbs_dealloc_mw(mw);
|
2013-02-07 00:19:13 +08:00
|
|
|
|
|
|
|
err_put:
|
|
|
|
put_pd_read(pd);
|
|
|
|
|
|
|
|
err_free:
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_dealloc_mw(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
2013-02-07 00:19:13 +08:00
|
|
|
{
|
|
|
|
struct ib_uverbs_dealloc_mw cmd;
|
|
|
|
struct ib_mw *mw;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
int ret = -EINVAL;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof(cmd)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
uobj = idr_write_uobj(&ib_uverbs_mw_idr, cmd.mw_handle, file->ucontext);
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
mw = uobj->object;
|
|
|
|
|
2015-12-24 02:12:48 +08:00
|
|
|
ret = uverbs_dealloc_mw(mw);
|
2013-02-07 00:19:13 +08:00
|
|
|
if (!ret)
|
|
|
|
uobj->live = 0;
|
|
|
|
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
idr_remove_uobj(&ib_uverbs_mw_idr, uobj);
|
|
|
|
|
|
|
|
mutex_lock(&file->mutex);
|
|
|
|
list_del(&uobj->list);
|
|
|
|
mutex_unlock(&file->mutex);
|
|
|
|
|
|
|
|
put_uobj(uobj);
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
}
|
|
|
|
|
2005-09-27 04:53:25 +08:00
|
|
|
ssize_t ib_uverbs_create_comp_channel(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-09-27 04:53:25 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_create_comp_channel cmd;
|
|
|
|
struct ib_uverbs_create_comp_channel_resp resp;
|
|
|
|
struct file *filp;
|
2010-01-18 14:38:00 +08:00
|
|
|
int ret;
|
2005-09-27 04:53:25 +08:00
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2013-07-09 02:15:45 +08:00
|
|
|
ret = get_unused_fd_flags(O_CLOEXEC);
|
2010-01-18 14:38:00 +08:00
|
|
|
if (ret < 0)
|
|
|
|
return ret;
|
|
|
|
resp.fd = ret;
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
filp = ib_uverbs_alloc_event_file(file, ib_dev, 0);
|
2010-01-18 14:38:00 +08:00
|
|
|
if (IS_ERR(filp)) {
|
|
|
|
put_unused_fd(resp.fd);
|
2005-09-27 04:53:25 +08:00
|
|
|
return PTR_ERR(filp);
|
2010-01-18 14:38:00 +08:00
|
|
|
}
|
2005-09-27 04:53:25 +08:00
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp)) {
|
|
|
|
put_unused_fd(resp.fd);
|
|
|
|
fput(filp);
|
|
|
|
return -EFAULT;
|
|
|
|
}
|
|
|
|
|
|
|
|
fd_install(resp.fd, filp);
|
|
|
|
return in_len;
|
|
|
|
}
|
|
|
|
|
2015-06-11 21:35:23 +08:00
|
|
|
static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2015-06-11 21:35:23 +08:00
|
|
|
struct ib_udata *ucore,
|
|
|
|
struct ib_udata *uhw,
|
|
|
|
struct ib_uverbs_ex_create_cq *cmd,
|
|
|
|
size_t cmd_sz,
|
|
|
|
int (*cb)(struct ib_uverbs_file *file,
|
|
|
|
struct ib_ucq_object *obj,
|
|
|
|
struct ib_uverbs_ex_create_cq_resp *resp,
|
|
|
|
struct ib_udata *udata,
|
|
|
|
void *context),
|
|
|
|
void *context)
|
2005-07-08 08:57:13 +08:00
|
|
|
{
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_ucq_object *obj;
|
2005-09-27 04:53:25 +08:00
|
|
|
struct ib_uverbs_event_file *ev_file = NULL;
|
2005-07-08 08:57:13 +08:00
|
|
|
struct ib_cq *cq;
|
|
|
|
int ret;
|
2015-06-11 21:35:23 +08:00
|
|
|
struct ib_uverbs_ex_create_cq_resp resp;
|
2015-06-11 21:35:20 +08:00
|
|
|
struct ib_cq_init_attr attr = {};
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-06-11 21:35:23 +08:00
|
|
|
if (cmd->comp_vector >= file->device->num_comp_vectors)
|
|
|
|
return ERR_PTR(-EINVAL);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj = kmalloc(sizeof *obj, GFP_KERNEL);
|
|
|
|
if (!obj)
|
2015-06-11 21:35:23 +08:00
|
|
|
return ERR_PTR(-ENOMEM);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-06-11 21:35:23 +08:00
|
|
|
init_uobj(&obj->uobject, cmd->user_handle, file->ucontext, &cq_lock_class);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
down_write(&obj->uobject.mutex);
|
|
|
|
|
2015-06-11 21:35:23 +08:00
|
|
|
if (cmd->comp_channel >= 0) {
|
|
|
|
ev_file = ib_uverbs_lookup_comp_file(cmd->comp_channel);
|
2006-01-07 08:43:14 +08:00
|
|
|
if (!ev_file) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj->uverbs_file = file;
|
|
|
|
obj->comp_events_reported = 0;
|
|
|
|
obj->async_events_reported = 0;
|
|
|
|
INIT_LIST_HEAD(&obj->comp_list);
|
|
|
|
INIT_LIST_HEAD(&obj->async_list);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-06-11 21:35:23 +08:00
|
|
|
attr.cqe = cmd->cqe;
|
|
|
|
attr.comp_vector = cmd->comp_vector;
|
|
|
|
|
|
|
|
if (cmd_sz > offsetof(typeof(*cmd), flags) + sizeof(cmd->flags))
|
|
|
|
attr.flags = cmd->flags;
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
cq = ib_dev->create_cq(ib_dev, &attr,
|
2015-06-11 21:35:23 +08:00
|
|
|
file->ucontext, uhw);
|
2005-07-08 08:57:13 +08:00
|
|
|
if (IS_ERR(cq)) {
|
|
|
|
ret = PTR_ERR(cq);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_file;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
cq->device = ib_dev;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
cq->uobject = &obj->uobject;
|
2005-07-08 08:57:13 +08:00
|
|
|
cq->comp_handler = ib_uverbs_comp_handler;
|
|
|
|
cq->event_handler = ib_uverbs_cq_event_handler;
|
2005-09-27 04:53:25 +08:00
|
|
|
cq->cq_context = ev_file;
|
2005-07-08 08:57:13 +08:00
|
|
|
atomic_set(&cq->usecnt, 0);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj->uobject.object = cq;
|
|
|
|
ret = idr_add_uobj(&ib_uverbs_cq_idr, &obj->uobject);
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_free;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
memset(&resp, 0, sizeof resp);
|
2015-06-11 21:35:23 +08:00
|
|
|
resp.base.cq_handle = obj->uobject.id;
|
|
|
|
resp.base.cqe = cq->cqe;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-06-11 21:35:23 +08:00
|
|
|
resp.response_length = offsetof(typeof(resp), response_length) +
|
|
|
|
sizeof(resp.response_length);
|
|
|
|
|
|
|
|
ret = cb(file, obj, &resp, ucore, context);
|
|
|
|
if (ret)
|
|
|
|
goto err_cb;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
list_add_tail(&obj->uobject.list, &file->ucontext->cq_list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj->uobject.live = 1;
|
|
|
|
|
|
|
|
up_write(&obj->uobject.mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-06-11 21:35:23 +08:00
|
|
|
return obj;
|
2005-09-28 06:07:25 +08:00
|
|
|
|
2015-06-11 21:35:23 +08:00
|
|
|
err_cb:
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_cq_idr, &obj->uobject);
|
2005-09-28 06:07:25 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_free:
|
2005-07-08 08:57:13 +08:00
|
|
|
ib_destroy_cq(cq);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_file:
|
2006-01-07 08:43:14 +08:00
|
|
|
if (ev_file)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ib_uverbs_release_ucq(file, ev_file, obj);
|
|
|
|
|
|
|
|
err:
|
|
|
|
put_uobj_write(&obj->uobject);
|
2015-06-11 21:35:23 +08:00
|
|
|
|
|
|
|
return ERR_PTR(ret);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int ib_uverbs_create_cq_cb(struct ib_uverbs_file *file,
|
|
|
|
struct ib_ucq_object *obj,
|
|
|
|
struct ib_uverbs_ex_create_cq_resp *resp,
|
|
|
|
struct ib_udata *ucore, void *context)
|
|
|
|
{
|
|
|
|
if (ib_copy_to_udata(ucore, &resp->base, sizeof(resp->base)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_create_cq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2015-06-11 21:35:23 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_create_cq cmd;
|
|
|
|
struct ib_uverbs_ex_create_cq cmd_ex;
|
|
|
|
struct ib_uverbs_create_cq_resp resp;
|
|
|
|
struct ib_udata ucore;
|
|
|
|
struct ib_udata uhw;
|
|
|
|
struct ib_ucq_object *obj;
|
|
|
|
|
|
|
|
if (out_len < sizeof(resp))
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof(cmd)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2015-10-07 20:29:51 +08:00
|
|
|
INIT_UDATA(&ucore, buf, (unsigned long)cmd.response, sizeof(cmd), sizeof(resp));
|
2015-06-11 21:35:23 +08:00
|
|
|
|
|
|
|
INIT_UDATA(&uhw, buf + sizeof(cmd),
|
|
|
|
(unsigned long)cmd.response + sizeof(resp),
|
|
|
|
in_len - sizeof(cmd), out_len - sizeof(resp));
|
|
|
|
|
|
|
|
memset(&cmd_ex, 0, sizeof(cmd_ex));
|
|
|
|
cmd_ex.user_handle = cmd.user_handle;
|
|
|
|
cmd_ex.cqe = cmd.cqe;
|
|
|
|
cmd_ex.comp_vector = cmd.comp_vector;
|
|
|
|
cmd_ex.comp_channel = cmd.comp_channel;
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
obj = create_cq(file, ib_dev, &ucore, &uhw, &cmd_ex,
|
2015-06-11 21:35:23 +08:00
|
|
|
offsetof(typeof(cmd_ex), comp_channel) +
|
|
|
|
sizeof(cmd.comp_channel), ib_uverbs_create_cq_cb,
|
|
|
|
NULL);
|
|
|
|
|
|
|
|
if (IS_ERR(obj))
|
|
|
|
return PTR_ERR(obj);
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int ib_uverbs_ex_create_cq_cb(struct ib_uverbs_file *file,
|
|
|
|
struct ib_ucq_object *obj,
|
|
|
|
struct ib_uverbs_ex_create_cq_resp *resp,
|
|
|
|
struct ib_udata *ucore, void *context)
|
|
|
|
{
|
|
|
|
if (ib_copy_to_udata(ucore, resp, resp->response_length))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int ib_uverbs_ex_create_cq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2015-06-11 21:35:23 +08:00
|
|
|
struct ib_udata *ucore,
|
|
|
|
struct ib_udata *uhw)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_ex_create_cq_resp resp;
|
|
|
|
struct ib_uverbs_ex_create_cq cmd;
|
|
|
|
struct ib_ucq_object *obj;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (ucore->inlen < sizeof(cmd))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
err = ib_copy_from_udata(&cmd, ucore, sizeof(cmd));
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
if (cmd.comp_mask)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (cmd.reserved)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (ucore->outlen < (offsetof(typeof(resp), response_length) +
|
|
|
|
sizeof(resp.response_length)))
|
|
|
|
return -ENOSPC;
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
obj = create_cq(file, ib_dev, ucore, uhw, &cmd,
|
2015-06-11 21:35:23 +08:00
|
|
|
min(ucore->inlen, sizeof(cmd)),
|
|
|
|
ib_uverbs_ex_create_cq_cb, NULL);
|
|
|
|
|
|
|
|
if (IS_ERR(obj))
|
|
|
|
return PTR_ERR(obj);
|
|
|
|
|
|
|
|
return 0;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2006-01-31 06:29:21 +08:00
|
|
|
ssize_t ib_uverbs_resize_cq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2006-01-31 06:29:21 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_resize_cq cmd;
|
|
|
|
struct ib_uverbs_resize_cq_resp resp;
|
|
|
|
struct ib_udata udata;
|
|
|
|
struct ib_cq *cq;
|
|
|
|
int ret = -EINVAL;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd,
|
|
|
|
(unsigned long) cmd.response + sizeof resp,
|
|
|
|
in_len - sizeof cmd, out_len - sizeof resp);
|
|
|
|
|
2006-09-23 06:17:20 +08:00
|
|
|
cq = idr_read_cq(cmd.cq_handle, file->ucontext, 0);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!cq)
|
|
|
|
return -EINVAL;
|
2006-01-31 06:29:21 +08:00
|
|
|
|
|
|
|
ret = cq->device->resize_cq(cq, cmd.cqe, &udata);
|
|
|
|
if (ret)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
resp.cqe = cq->cqe;
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
2006-09-23 06:22:24 +08:00
|
|
|
&resp, sizeof resp.cqe))
|
2006-01-31 06:29:21 +08:00
|
|
|
ret = -EFAULT;
|
|
|
|
|
|
|
|
out:
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_cq_read(cq);
|
2006-01-31 06:29:21 +08:00
|
|
|
|
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
|
|
|
|
2010-10-13 17:13:12 +08:00
|
|
|
static int copy_wc_to_user(void __user *dest, struct ib_wc *wc)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_wc tmp;
|
|
|
|
|
|
|
|
tmp.wr_id = wc->wr_id;
|
|
|
|
tmp.status = wc->status;
|
|
|
|
tmp.opcode = wc->opcode;
|
|
|
|
tmp.vendor_err = wc->vendor_err;
|
|
|
|
tmp.byte_len = wc->byte_len;
|
|
|
|
tmp.ex.imm_data = (__u32 __force) wc->ex.imm_data;
|
|
|
|
tmp.qp_num = wc->qp->qp_num;
|
|
|
|
tmp.src_qp = wc->src_qp;
|
|
|
|
tmp.wc_flags = wc->wc_flags;
|
|
|
|
tmp.pkey_index = wc->pkey_index;
|
|
|
|
tmp.slid = wc->slid;
|
|
|
|
tmp.sl = wc->sl;
|
|
|
|
tmp.dlid_path_bits = wc->dlid_path_bits;
|
|
|
|
tmp.port_num = wc->port_num;
|
|
|
|
tmp.reserved = 0;
|
|
|
|
|
|
|
|
if (copy_to_user(dest, &tmp, sizeof tmp))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2005-10-15 06:26:04 +08:00
|
|
|
ssize_t ib_uverbs_poll_cq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-10-15 06:26:04 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_poll_cq cmd;
|
2010-10-13 17:13:12 +08:00
|
|
|
struct ib_uverbs_poll_cq_resp resp;
|
|
|
|
u8 __user *header_ptr;
|
|
|
|
u8 __user *data_ptr;
|
2005-10-15 06:26:04 +08:00
|
|
|
struct ib_cq *cq;
|
2010-10-13 17:13:12 +08:00
|
|
|
struct ib_wc wc;
|
|
|
|
int ret;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2006-09-23 06:17:20 +08:00
|
|
|
cq = idr_read_cq(cmd.cq_handle, file->ucontext, 0);
|
2010-10-13 17:13:12 +08:00
|
|
|
if (!cq)
|
|
|
|
return -EINVAL;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
2010-10-13 17:13:12 +08:00
|
|
|
/* we copy a struct ib_uverbs_poll_cq_resp to user space */
|
|
|
|
header_ptr = (void __user *)(unsigned long) cmd.response;
|
|
|
|
data_ptr = header_ptr + sizeof resp;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
|
2010-10-13 17:13:12 +08:00
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
while (resp.count < cmd.ne) {
|
|
|
|
ret = ib_poll_cq(cq, 1, &wc);
|
|
|
|
if (ret < 0)
|
|
|
|
goto out_put;
|
|
|
|
if (!ret)
|
|
|
|
break;
|
|
|
|
|
|
|
|
ret = copy_wc_to_user(data_ptr, &wc);
|
|
|
|
if (ret)
|
|
|
|
goto out_put;
|
|
|
|
|
|
|
|
data_ptr += sizeof(struct ib_uverbs_wc);
|
|
|
|
++resp.count;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
2010-10-13 17:13:12 +08:00
|
|
|
if (copy_to_user(header_ptr, &resp, sizeof resp)) {
|
2005-10-15 06:26:04 +08:00
|
|
|
ret = -EFAULT;
|
2010-10-13 17:13:12 +08:00
|
|
|
goto out_put;
|
|
|
|
}
|
2005-10-15 06:26:04 +08:00
|
|
|
|
2010-10-13 17:13:12 +08:00
|
|
|
ret = in_len;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
2010-10-13 17:13:12 +08:00
|
|
|
out_put:
|
|
|
|
put_cq_read(cq);
|
|
|
|
return ret;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_req_notify_cq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-10-15 06:26:04 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_req_notify_cq cmd;
|
|
|
|
struct ib_cq *cq;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2006-09-23 06:17:20 +08:00
|
|
|
cq = idr_read_cq(cmd.cq_handle, file->ucontext, 0);
|
2006-09-23 06:17:19 +08:00
|
|
|
if (!cq)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return -EINVAL;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ib_req_notify_cq(cq, cmd.solicited_only ?
|
|
|
|
IB_CQ_SOLICITED : IB_CQ_NEXT_COMP);
|
|
|
|
|
2006-09-23 06:17:19 +08:00
|
|
|
put_cq_read(cq);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
|
|
|
|
return in_len;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
ssize_t ib_uverbs_destroy_cq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
2005-09-10 06:55:08 +08:00
|
|
|
struct ib_uverbs_destroy_cq cmd;
|
|
|
|
struct ib_uverbs_destroy_cq_resp resp;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_uobject *uobj;
|
2005-09-10 06:55:08 +08:00
|
|
|
struct ib_cq *cq;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_ucq_object *obj;
|
2005-09-27 04:53:25 +08:00
|
|
|
struct ib_uverbs_event_file *ev_file;
|
2005-09-10 06:55:08 +08:00
|
|
|
int ret = -EINVAL;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj = idr_write_uobj(&ib_uverbs_cq_idr, cmd.cq_handle, file->ucontext);
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
|
|
|
cq = uobj->object;
|
|
|
|
ev_file = cq->cq_context;
|
|
|
|
obj = container_of(cq->uobject, struct ib_ucq_object, uobject);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ret = ib_destroy_cq(cq);
|
|
|
|
if (!ret)
|
|
|
|
uobj->live = 0;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj_write(uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return ret;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_cq_idr, uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
list_del(&uobj->list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ib_uverbs_release_ucq(file, ev_file, obj);
|
2005-09-10 06:55:08 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
resp.comp_events_reported = obj->comp_events_reported;
|
|
|
|
resp.async_events_reported = obj->async_events_reported;
|
2005-09-10 06:55:08 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj(uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2005-09-10 06:55:08 +08:00
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return -EFAULT;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return in_len;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
static int create_qp(struct ib_uverbs_file *file,
|
|
|
|
struct ib_udata *ucore,
|
|
|
|
struct ib_udata *uhw,
|
|
|
|
struct ib_uverbs_ex_create_qp *cmd,
|
|
|
|
size_t cmd_sz,
|
|
|
|
int (*cb)(struct ib_uverbs_file *file,
|
|
|
|
struct ib_uverbs_ex_create_qp_resp *resp,
|
|
|
|
struct ib_udata *udata),
|
|
|
|
void *context)
|
2005-07-08 08:57:13 +08:00
|
|
|
{
|
2015-10-21 22:00:42 +08:00
|
|
|
struct ib_uqp_object *obj;
|
|
|
|
struct ib_device *device;
|
|
|
|
struct ib_pd *pd = NULL;
|
|
|
|
struct ib_xrcd *xrcd = NULL;
|
|
|
|
struct ib_uobject *uninitialized_var(xrcd_uobj);
|
|
|
|
struct ib_cq *scq = NULL, *rcq = NULL;
|
|
|
|
struct ib_srq *srq = NULL;
|
|
|
|
struct ib_qp *qp;
|
|
|
|
char *buf;
|
|
|
|
struct ib_qp_init_attr attr;
|
|
|
|
struct ib_uverbs_ex_create_qp_resp resp;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (cmd->qp_type == IB_QPT_RAW_PACKET && !capable(CAP_NET_RAW))
|
2012-03-01 18:17:51 +08:00
|
|
|
return -EPERM;
|
|
|
|
|
2013-08-01 23:49:54 +08:00
|
|
|
obj = kzalloc(sizeof *obj, GFP_KERNEL);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!obj)
|
2005-07-08 08:57:13 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
init_uobj(&obj->uevent.uobject, cmd->user_handle, file->ucontext,
|
|
|
|
&qp_lock_class);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
down_write(&obj->uevent.uobject.mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
if (cmd->qp_type == IB_QPT_XRC_TGT) {
|
|
|
|
xrcd = idr_read_xrcd(cmd->pd_handle, file->ucontext,
|
|
|
|
&xrcd_uobj);
|
2011-05-27 15:00:12 +08:00
|
|
|
if (!xrcd) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
device = xrcd->device;
|
2011-05-26 23:17:04 +08:00
|
|
|
} else {
|
2015-10-21 22:00:42 +08:00
|
|
|
if (cmd->qp_type == IB_QPT_XRC_INI) {
|
|
|
|
cmd->max_recv_wr = 0;
|
|
|
|
cmd->max_recv_sge = 0;
|
2011-05-27 15:00:12 +08:00
|
|
|
} else {
|
2015-10-21 22:00:42 +08:00
|
|
|
if (cmd->is_srq) {
|
|
|
|
srq = idr_read_srq(cmd->srq_handle,
|
|
|
|
file->ucontext);
|
2011-05-27 15:00:12 +08:00
|
|
|
if (!srq || srq->srq_type != IB_SRQT_BASIC) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
}
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
if (cmd->recv_cq_handle != cmd->send_cq_handle) {
|
|
|
|
rcq = idr_read_cq(cmd->recv_cq_handle,
|
|
|
|
file->ucontext, 0);
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
if (!rcq) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put;
|
|
|
|
}
|
2011-05-26 23:17:04 +08:00
|
|
|
}
|
|
|
|
}
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
scq = idr_read_cq(cmd->send_cq_handle, file->ucontext, !!rcq);
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
rcq = rcq ?: scq;
|
2015-10-21 22:00:42 +08:00
|
|
|
pd = idr_read_pd(cmd->pd_handle, file->ucontext);
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
if (!pd || !scq) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
|
2011-05-27 15:00:12 +08:00
|
|
|
device = pd->device;
|
2011-05-26 23:17:04 +08:00
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
attr.event_handler = ib_uverbs_qp_event_handler;
|
|
|
|
attr.qp_context = file;
|
|
|
|
attr.send_cq = scq;
|
|
|
|
attr.recv_cq = rcq;
|
2005-08-19 03:24:13 +08:00
|
|
|
attr.srq = srq;
|
2011-05-27 15:00:12 +08:00
|
|
|
attr.xrcd = xrcd;
|
2015-10-21 22:00:42 +08:00
|
|
|
attr.sq_sig_type = cmd->sq_sig_all ? IB_SIGNAL_ALL_WR :
|
|
|
|
IB_SIGNAL_REQ_WR;
|
|
|
|
attr.qp_type = cmd->qp_type;
|
2008-04-17 12:09:27 +08:00
|
|
|
attr.create_flags = 0;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
attr.cap.max_send_wr = cmd->max_send_wr;
|
|
|
|
attr.cap.max_recv_wr = cmd->max_recv_wr;
|
|
|
|
attr.cap.max_send_sge = cmd->max_send_sge;
|
|
|
|
attr.cap.max_recv_sge = cmd->max_recv_sge;
|
|
|
|
attr.cap.max_inline_data = cmd->max_inline_data;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj->uevent.events_reported = 0;
|
|
|
|
INIT_LIST_HEAD(&obj->uevent.event_list);
|
|
|
|
INIT_LIST_HEAD(&obj->mcast_list);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
if (cmd_sz >= offsetof(typeof(*cmd), create_flags) +
|
|
|
|
sizeof(cmd->create_flags))
|
|
|
|
attr.create_flags = cmd->create_flags;
|
|
|
|
|
2015-12-20 18:16:10 +08:00
|
|
|
if (attr.create_flags & ~(IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK |
|
|
|
|
IB_QP_CREATE_CROSS_CHANNEL |
|
|
|
|
IB_QP_CREATE_MANAGED_SEND |
|
|
|
|
IB_QP_CREATE_MANAGED_RECV)) {
|
2015-10-21 22:00:42 +08:00
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
buf = (void *)cmd + sizeof(*cmd);
|
|
|
|
if (cmd_sz > sizeof(*cmd))
|
|
|
|
if (!(buf[0] == 0 && !memcmp(buf, buf + 1,
|
|
|
|
cmd_sz - sizeof(*cmd) - 1))) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (cmd->qp_type == IB_QPT_XRC_TGT)
|
2011-05-27 15:00:12 +08:00
|
|
|
qp = ib_create_qp(pd, &attr);
|
|
|
|
else
|
2015-10-21 22:00:42 +08:00
|
|
|
qp = device->create_qp(pd, &attr, uhw);
|
2011-05-27 15:00:12 +08:00
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
if (IS_ERR(qp)) {
|
|
|
|
ret = PTR_ERR(qp);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_put;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
if (cmd->qp_type != IB_QPT_XRC_TGT) {
|
2011-08-09 06:31:51 +08:00
|
|
|
qp->real_qp = qp;
|
2011-05-27 15:00:12 +08:00
|
|
|
qp->device = device;
|
|
|
|
qp->pd = pd;
|
|
|
|
qp->send_cq = attr.send_cq;
|
|
|
|
qp->recv_cq = attr.recv_cq;
|
|
|
|
qp->srq = attr.srq;
|
|
|
|
qp->event_handler = attr.event_handler;
|
|
|
|
qp->qp_context = attr.qp_context;
|
|
|
|
qp->qp_type = attr.qp_type;
|
2012-01-21 02:43:54 +08:00
|
|
|
atomic_set(&qp->usecnt, 0);
|
2011-05-27 15:00:12 +08:00
|
|
|
atomic_inc(&pd->usecnt);
|
|
|
|
atomic_inc(&attr.send_cq->usecnt);
|
|
|
|
if (attr.recv_cq)
|
|
|
|
atomic_inc(&attr.recv_cq->usecnt);
|
|
|
|
if (attr.srq)
|
|
|
|
atomic_inc(&attr.srq->usecnt);
|
|
|
|
}
|
|
|
|
qp->uobject = &obj->uevent.uobject;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj->uevent.uobject.object = qp;
|
|
|
|
ret = idr_add_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject);
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
|
|
|
goto err_destroy;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
memset(&resp, 0, sizeof resp);
|
2015-10-21 22:00:42 +08:00
|
|
|
resp.base.qpn = qp->qp_num;
|
|
|
|
resp.base.qp_handle = obj->uevent.uobject.id;
|
|
|
|
resp.base.max_recv_sge = attr.cap.max_recv_sge;
|
|
|
|
resp.base.max_send_sge = attr.cap.max_send_sge;
|
|
|
|
resp.base.max_recv_wr = attr.cap.max_recv_wr;
|
|
|
|
resp.base.max_send_wr = attr.cap.max_send_wr;
|
|
|
|
resp.base.max_inline_data = attr.cap.max_inline_data;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
resp.response_length = offsetof(typeof(resp), response_length) +
|
|
|
|
sizeof(resp.response_length);
|
|
|
|
|
|
|
|
ret = cb(file, &resp, ucore);
|
|
|
|
if (ret)
|
|
|
|
goto err_cb;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2013-08-01 23:49:54 +08:00
|
|
|
if (xrcd) {
|
|
|
|
obj->uxrcd = container_of(xrcd_uobj, struct ib_uxrcd_object,
|
|
|
|
uobject);
|
|
|
|
atomic_inc(&obj->uxrcd->refcnt);
|
2011-05-27 15:00:12 +08:00
|
|
|
put_xrcd_read(xrcd_uobj);
|
2013-08-01 23:49:54 +08:00
|
|
|
}
|
|
|
|
|
2011-05-27 15:00:12 +08:00
|
|
|
if (pd)
|
|
|
|
put_pd_read(pd);
|
|
|
|
if (scq)
|
|
|
|
put_cq_read(scq);
|
2011-05-26 23:17:04 +08:00
|
|
|
if (rcq && rcq != scq)
|
2006-07-24 06:16:04 +08:00
|
|
|
put_cq_read(rcq);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (srq)
|
|
|
|
put_srq_read(srq);
|
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
list_add_tail(&obj->uevent.uobject.list, &file->ucontext->qp_list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-09-28 06:07:25 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj->uevent.uobject.live = 1;
|
|
|
|
|
|
|
|
up_write(&obj->uevent.uobject.mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
return 0;
|
|
|
|
err_cb:
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
err_destroy:
|
|
|
|
ib_destroy_qp(qp);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_put:
|
2011-05-27 15:00:12 +08:00
|
|
|
if (xrcd)
|
|
|
|
put_xrcd_read(xrcd_uobj);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (pd)
|
|
|
|
put_pd_read(pd);
|
|
|
|
if (scq)
|
|
|
|
put_cq_read(scq);
|
2006-07-24 06:16:04 +08:00
|
|
|
if (rcq && rcq != scq)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_cq_read(rcq);
|
|
|
|
if (srq)
|
|
|
|
put_srq_read(srq);
|
|
|
|
|
|
|
|
put_uobj_write(&obj->uevent.uobject);
|
2005-07-08 08:57:13 +08:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2015-10-21 22:00:42 +08:00
|
|
|
static int ib_uverbs_create_qp_cb(struct ib_uverbs_file *file,
|
|
|
|
struct ib_uverbs_ex_create_qp_resp *resp,
|
|
|
|
struct ib_udata *ucore)
|
|
|
|
{
|
|
|
|
if (ib_copy_to_udata(ucore, &resp->base, sizeof(resp->base)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_create_qp(struct ib_uverbs_file *file,
|
|
|
|
struct ib_device *ib_dev,
|
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_create_qp cmd;
|
|
|
|
struct ib_uverbs_ex_create_qp cmd_ex;
|
|
|
|
struct ib_udata ucore;
|
|
|
|
struct ib_udata uhw;
|
|
|
|
ssize_t resp_size = sizeof(struct ib_uverbs_create_qp_resp);
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (out_len < resp_size)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof(cmd)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
INIT_UDATA(&ucore, buf, (unsigned long)cmd.response, sizeof(cmd),
|
|
|
|
resp_size);
|
|
|
|
INIT_UDATA(&uhw, buf + sizeof(cmd),
|
|
|
|
(unsigned long)cmd.response + resp_size,
|
|
|
|
in_len - sizeof(cmd), out_len - resp_size);
|
|
|
|
|
|
|
|
memset(&cmd_ex, 0, sizeof(cmd_ex));
|
|
|
|
cmd_ex.user_handle = cmd.user_handle;
|
|
|
|
cmd_ex.pd_handle = cmd.pd_handle;
|
|
|
|
cmd_ex.send_cq_handle = cmd.send_cq_handle;
|
|
|
|
cmd_ex.recv_cq_handle = cmd.recv_cq_handle;
|
|
|
|
cmd_ex.srq_handle = cmd.srq_handle;
|
|
|
|
cmd_ex.max_send_wr = cmd.max_send_wr;
|
|
|
|
cmd_ex.max_recv_wr = cmd.max_recv_wr;
|
|
|
|
cmd_ex.max_send_sge = cmd.max_send_sge;
|
|
|
|
cmd_ex.max_recv_sge = cmd.max_recv_sge;
|
|
|
|
cmd_ex.max_inline_data = cmd.max_inline_data;
|
|
|
|
cmd_ex.sq_sig_all = cmd.sq_sig_all;
|
|
|
|
cmd_ex.qp_type = cmd.qp_type;
|
|
|
|
cmd_ex.is_srq = cmd.is_srq;
|
|
|
|
|
|
|
|
err = create_qp(file, &ucore, &uhw, &cmd_ex,
|
|
|
|
offsetof(typeof(cmd_ex), is_srq) +
|
|
|
|
sizeof(cmd.is_srq), ib_uverbs_create_qp_cb,
|
|
|
|
NULL);
|
|
|
|
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int ib_uverbs_ex_create_qp_cb(struct ib_uverbs_file *file,
|
|
|
|
struct ib_uverbs_ex_create_qp_resp *resp,
|
|
|
|
struct ib_udata *ucore)
|
|
|
|
{
|
|
|
|
if (ib_copy_to_udata(ucore, resp, resp->response_length))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int ib_uverbs_ex_create_qp(struct ib_uverbs_file *file,
|
|
|
|
struct ib_device *ib_dev,
|
|
|
|
struct ib_udata *ucore,
|
|
|
|
struct ib_udata *uhw)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_ex_create_qp_resp resp;
|
|
|
|
struct ib_uverbs_ex_create_qp cmd = {0};
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (ucore->inlen < (offsetof(typeof(cmd), comp_mask) +
|
|
|
|
sizeof(cmd.comp_mask)))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
err = ib_copy_from_udata(&cmd, ucore, min(sizeof(cmd), ucore->inlen));
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
if (cmd.comp_mask)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (cmd.reserved)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (ucore->outlen < (offsetof(typeof(resp), response_length) +
|
|
|
|
sizeof(resp.response_length)))
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
err = create_qp(file, ucore, uhw, &cmd,
|
|
|
|
min(ucore->inlen, sizeof(cmd)),
|
|
|
|
ib_uverbs_ex_create_qp_cb, NULL);
|
|
|
|
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-08-12 04:57:43 +08:00
|
|
|
ssize_t ib_uverbs_open_qp(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2011-08-12 04:57:43 +08:00
|
|
|
const char __user *buf, int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_open_qp cmd;
|
|
|
|
struct ib_uverbs_create_qp_resp resp;
|
|
|
|
struct ib_udata udata;
|
|
|
|
struct ib_uqp_object *obj;
|
|
|
|
struct ib_xrcd *xrcd;
|
|
|
|
struct ib_uobject *uninitialized_var(xrcd_uobj);
|
|
|
|
struct ib_qp *qp;
|
|
|
|
struct ib_qp_open_attr attr;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd,
|
|
|
|
(unsigned long) cmd.response + sizeof resp,
|
|
|
|
in_len - sizeof cmd, out_len - sizeof resp);
|
|
|
|
|
|
|
|
obj = kmalloc(sizeof *obj, GFP_KERNEL);
|
|
|
|
if (!obj)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2012-05-01 01:27:26 +08:00
|
|
|
init_uobj(&obj->uevent.uobject, cmd.user_handle, file->ucontext, &qp_lock_class);
|
2011-08-12 04:57:43 +08:00
|
|
|
down_write(&obj->uevent.uobject.mutex);
|
|
|
|
|
|
|
|
xrcd = idr_read_xrcd(cmd.pd_handle, file->ucontext, &xrcd_uobj);
|
|
|
|
if (!xrcd) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
attr.event_handler = ib_uverbs_qp_event_handler;
|
|
|
|
attr.qp_context = file;
|
|
|
|
attr.qp_num = cmd.qpn;
|
|
|
|
attr.qp_type = cmd.qp_type;
|
|
|
|
|
|
|
|
obj->uevent.events_reported = 0;
|
|
|
|
INIT_LIST_HEAD(&obj->uevent.event_list);
|
|
|
|
INIT_LIST_HEAD(&obj->mcast_list);
|
|
|
|
|
|
|
|
qp = ib_open_qp(xrcd, &attr);
|
|
|
|
if (IS_ERR(qp)) {
|
|
|
|
ret = PTR_ERR(qp);
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
qp->uobject = &obj->uevent.uobject;
|
|
|
|
|
|
|
|
obj->uevent.uobject.object = qp;
|
|
|
|
ret = idr_add_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject);
|
|
|
|
if (ret)
|
|
|
|
goto err_destroy;
|
|
|
|
|
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
resp.qpn = qp->qp_num;
|
|
|
|
resp.qp_handle = obj->uevent.uobject.id;
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp)) {
|
|
|
|
ret = -EFAULT;
|
|
|
|
goto err_remove;
|
|
|
|
}
|
|
|
|
|
2013-08-01 23:49:54 +08:00
|
|
|
obj->uxrcd = container_of(xrcd_uobj, struct ib_uxrcd_object, uobject);
|
|
|
|
atomic_inc(&obj->uxrcd->refcnt);
|
2011-08-12 04:57:43 +08:00
|
|
|
put_xrcd_read(xrcd_uobj);
|
|
|
|
|
|
|
|
mutex_lock(&file->mutex);
|
|
|
|
list_add_tail(&obj->uevent.uobject.list, &file->ucontext->qp_list);
|
|
|
|
mutex_unlock(&file->mutex);
|
|
|
|
|
|
|
|
obj->uevent.uobject.live = 1;
|
|
|
|
|
|
|
|
up_write(&obj->uevent.uobject.mutex);
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
|
|
|
|
err_remove:
|
|
|
|
idr_remove_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject);
|
|
|
|
|
|
|
|
err_destroy:
|
|
|
|
ib_destroy_qp(qp);
|
|
|
|
|
|
|
|
err_put:
|
|
|
|
put_xrcd_read(xrcd_uobj);
|
|
|
|
put_uobj_write(&obj->uevent.uobject);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2006-02-14 08:31:25 +08:00
|
|
|
ssize_t ib_uverbs_query_qp(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2006-02-14 08:31:25 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_query_qp cmd;
|
|
|
|
struct ib_uverbs_query_qp_resp resp;
|
|
|
|
struct ib_qp *qp;
|
|
|
|
struct ib_qp_attr *attr;
|
|
|
|
struct ib_qp_init_attr *init_attr;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
attr = kmalloc(sizeof *attr, GFP_KERNEL);
|
|
|
|
init_attr = kmalloc(sizeof *init_attr, GFP_KERNEL);
|
|
|
|
if (!attr || !init_attr) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
qp = idr_read_qp(cmd.qp_handle, file->ucontext);
|
|
|
|
if (!qp) {
|
2006-02-14 08:31:25 +08:00
|
|
|
ret = -EINVAL;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = ib_query_qp(qp, attr, cmd.attr_mask, init_attr);
|
2006-02-14 08:31:25 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_qp_read(qp);
|
2006-02-14 08:31:25 +08:00
|
|
|
|
|
|
|
if (ret)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
|
|
|
|
resp.qp_state = attr->qp_state;
|
|
|
|
resp.cur_qp_state = attr->cur_qp_state;
|
|
|
|
resp.path_mtu = attr->path_mtu;
|
|
|
|
resp.path_mig_state = attr->path_mig_state;
|
|
|
|
resp.qkey = attr->qkey;
|
|
|
|
resp.rq_psn = attr->rq_psn;
|
|
|
|
resp.sq_psn = attr->sq_psn;
|
|
|
|
resp.dest_qp_num = attr->dest_qp_num;
|
|
|
|
resp.qp_access_flags = attr->qp_access_flags;
|
|
|
|
resp.pkey_index = attr->pkey_index;
|
|
|
|
resp.alt_pkey_index = attr->alt_pkey_index;
|
2006-10-25 18:54:20 +08:00
|
|
|
resp.sq_draining = attr->sq_draining;
|
2006-02-14 08:31:25 +08:00
|
|
|
resp.max_rd_atomic = attr->max_rd_atomic;
|
|
|
|
resp.max_dest_rd_atomic = attr->max_dest_rd_atomic;
|
|
|
|
resp.min_rnr_timer = attr->min_rnr_timer;
|
|
|
|
resp.port_num = attr->port_num;
|
|
|
|
resp.timeout = attr->timeout;
|
|
|
|
resp.retry_cnt = attr->retry_cnt;
|
|
|
|
resp.rnr_retry = attr->rnr_retry;
|
|
|
|
resp.alt_port_num = attr->alt_port_num;
|
|
|
|
resp.alt_timeout = attr->alt_timeout;
|
|
|
|
|
|
|
|
memcpy(resp.dest.dgid, attr->ah_attr.grh.dgid.raw, 16);
|
|
|
|
resp.dest.flow_label = attr->ah_attr.grh.flow_label;
|
|
|
|
resp.dest.sgid_index = attr->ah_attr.grh.sgid_index;
|
|
|
|
resp.dest.hop_limit = attr->ah_attr.grh.hop_limit;
|
|
|
|
resp.dest.traffic_class = attr->ah_attr.grh.traffic_class;
|
|
|
|
resp.dest.dlid = attr->ah_attr.dlid;
|
|
|
|
resp.dest.sl = attr->ah_attr.sl;
|
|
|
|
resp.dest.src_path_bits = attr->ah_attr.src_path_bits;
|
|
|
|
resp.dest.static_rate = attr->ah_attr.static_rate;
|
|
|
|
resp.dest.is_global = !!(attr->ah_attr.ah_flags & IB_AH_GRH);
|
|
|
|
resp.dest.port_num = attr->ah_attr.port_num;
|
|
|
|
|
|
|
|
memcpy(resp.alt_dest.dgid, attr->alt_ah_attr.grh.dgid.raw, 16);
|
|
|
|
resp.alt_dest.flow_label = attr->alt_ah_attr.grh.flow_label;
|
|
|
|
resp.alt_dest.sgid_index = attr->alt_ah_attr.grh.sgid_index;
|
|
|
|
resp.alt_dest.hop_limit = attr->alt_ah_attr.grh.hop_limit;
|
|
|
|
resp.alt_dest.traffic_class = attr->alt_ah_attr.grh.traffic_class;
|
|
|
|
resp.alt_dest.dlid = attr->alt_ah_attr.dlid;
|
|
|
|
resp.alt_dest.sl = attr->alt_ah_attr.sl;
|
|
|
|
resp.alt_dest.src_path_bits = attr->alt_ah_attr.src_path_bits;
|
|
|
|
resp.alt_dest.static_rate = attr->alt_ah_attr.static_rate;
|
|
|
|
resp.alt_dest.is_global = !!(attr->alt_ah_attr.ah_flags & IB_AH_GRH);
|
|
|
|
resp.alt_dest.port_num = attr->alt_ah_attr.port_num;
|
|
|
|
|
|
|
|
resp.max_send_wr = init_attr->cap.max_send_wr;
|
|
|
|
resp.max_recv_wr = init_attr->cap.max_recv_wr;
|
|
|
|
resp.max_send_sge = init_attr->cap.max_send_sge;
|
|
|
|
resp.max_recv_sge = init_attr->cap.max_recv_sge;
|
|
|
|
resp.max_inline_data = init_attr->cap.max_inline_data;
|
2006-03-03 03:25:27 +08:00
|
|
|
resp.sq_sig_all = init_attr->sq_sig_type == IB_SIGNAL_ALL_WR;
|
2006-02-14 08:31:25 +08:00
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
|
|
|
ret = -EFAULT;
|
|
|
|
|
|
|
|
out:
|
|
|
|
kfree(attr);
|
|
|
|
kfree(init_attr);
|
|
|
|
|
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
|
|
|
|
2011-05-26 23:17:04 +08:00
|
|
|
/* Remove ignored fields set in the attribute mask */
|
|
|
|
static int modify_qp_mask(enum ib_qp_type qp_type, int mask)
|
|
|
|
{
|
|
|
|
switch (qp_type) {
|
|
|
|
case IB_QPT_XRC_INI:
|
|
|
|
return mask & ~(IB_QP_MAX_DEST_RD_ATOMIC | IB_QP_MIN_RNR_TIMER);
|
2011-05-27 15:00:12 +08:00
|
|
|
case IB_QPT_XRC_TGT:
|
|
|
|
return mask & ~(IB_QP_MAX_QP_RD_ATOMIC | IB_QP_RETRY_CNT |
|
|
|
|
IB_QP_RNR_RETRY);
|
2011-05-26 23:17:04 +08:00
|
|
|
default:
|
|
|
|
return mask;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
ssize_t ib_uverbs_modify_qp(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_modify_qp cmd;
|
2006-08-12 05:58:09 +08:00
|
|
|
struct ib_udata udata;
|
2005-07-08 08:57:13 +08:00
|
|
|
struct ib_qp *qp;
|
|
|
|
struct ib_qp_attr *attr;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2006-08-12 05:58:09 +08:00
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd, NULL, in_len - sizeof cmd,
|
|
|
|
out_len);
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
attr = kmalloc(sizeof *attr, GFP_KERNEL);
|
|
|
|
if (!attr)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
qp = idr_read_qp(cmd.qp_handle, file->ucontext);
|
|
|
|
if (!qp) {
|
2005-07-08 08:57:13 +08:00
|
|
|
ret = -EINVAL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
attr->qp_state = cmd.qp_state;
|
|
|
|
attr->cur_qp_state = cmd.cur_qp_state;
|
|
|
|
attr->path_mtu = cmd.path_mtu;
|
|
|
|
attr->path_mig_state = cmd.path_mig_state;
|
|
|
|
attr->qkey = cmd.qkey;
|
|
|
|
attr->rq_psn = cmd.rq_psn;
|
|
|
|
attr->sq_psn = cmd.sq_psn;
|
|
|
|
attr->dest_qp_num = cmd.dest_qp_num;
|
|
|
|
attr->qp_access_flags = cmd.qp_access_flags;
|
|
|
|
attr->pkey_index = cmd.pkey_index;
|
2006-03-21 02:08:24 +08:00
|
|
|
attr->alt_pkey_index = cmd.alt_pkey_index;
|
2005-07-08 08:57:13 +08:00
|
|
|
attr->en_sqd_async_notify = cmd.en_sqd_async_notify;
|
|
|
|
attr->max_rd_atomic = cmd.max_rd_atomic;
|
|
|
|
attr->max_dest_rd_atomic = cmd.max_dest_rd_atomic;
|
|
|
|
attr->min_rnr_timer = cmd.min_rnr_timer;
|
|
|
|
attr->port_num = cmd.port_num;
|
|
|
|
attr->timeout = cmd.timeout;
|
|
|
|
attr->retry_cnt = cmd.retry_cnt;
|
|
|
|
attr->rnr_retry = cmd.rnr_retry;
|
|
|
|
attr->alt_port_num = cmd.alt_port_num;
|
|
|
|
attr->alt_timeout = cmd.alt_timeout;
|
|
|
|
|
|
|
|
memcpy(attr->ah_attr.grh.dgid.raw, cmd.dest.dgid, 16);
|
|
|
|
attr->ah_attr.grh.flow_label = cmd.dest.flow_label;
|
|
|
|
attr->ah_attr.grh.sgid_index = cmd.dest.sgid_index;
|
|
|
|
attr->ah_attr.grh.hop_limit = cmd.dest.hop_limit;
|
|
|
|
attr->ah_attr.grh.traffic_class = cmd.dest.traffic_class;
|
|
|
|
attr->ah_attr.dlid = cmd.dest.dlid;
|
|
|
|
attr->ah_attr.sl = cmd.dest.sl;
|
|
|
|
attr->ah_attr.src_path_bits = cmd.dest.src_path_bits;
|
|
|
|
attr->ah_attr.static_rate = cmd.dest.static_rate;
|
|
|
|
attr->ah_attr.ah_flags = cmd.dest.is_global ? IB_AH_GRH : 0;
|
|
|
|
attr->ah_attr.port_num = cmd.dest.port_num;
|
|
|
|
|
|
|
|
memcpy(attr->alt_ah_attr.grh.dgid.raw, cmd.alt_dest.dgid, 16);
|
|
|
|
attr->alt_ah_attr.grh.flow_label = cmd.alt_dest.flow_label;
|
|
|
|
attr->alt_ah_attr.grh.sgid_index = cmd.alt_dest.sgid_index;
|
|
|
|
attr->alt_ah_attr.grh.hop_limit = cmd.alt_dest.hop_limit;
|
|
|
|
attr->alt_ah_attr.grh.traffic_class = cmd.alt_dest.traffic_class;
|
|
|
|
attr->alt_ah_attr.dlid = cmd.alt_dest.dlid;
|
|
|
|
attr->alt_ah_attr.sl = cmd.alt_dest.sl;
|
|
|
|
attr->alt_ah_attr.src_path_bits = cmd.alt_dest.src_path_bits;
|
|
|
|
attr->alt_ah_attr.static_rate = cmd.alt_dest.static_rate;
|
|
|
|
attr->alt_ah_attr.ah_flags = cmd.alt_dest.is_global ? IB_AH_GRH : 0;
|
|
|
|
attr->alt_ah_attr.port_num = cmd.alt_dest.port_num;
|
|
|
|
|
2011-08-09 06:31:51 +08:00
|
|
|
if (qp->real_qp == qp) {
|
2015-10-15 23:38:51 +08:00
|
|
|
ret = ib_resolve_eth_dmac(qp, attr, &cmd.attr_mask);
|
2013-12-13 00:03:17 +08:00
|
|
|
if (ret)
|
2015-02-05 19:53:52 +08:00
|
|
|
goto release_qp;
|
2011-08-09 06:31:51 +08:00
|
|
|
ret = qp->device->modify_qp(qp, attr,
|
|
|
|
modify_qp_mask(qp->qp_type, cmd.attr_mask), &udata);
|
|
|
|
} else {
|
|
|
|
ret = ib_modify_qp(qp, attr, modify_qp_mask(qp->qp_type, cmd.attr_mask));
|
|
|
|
}
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
2015-02-05 19:53:52 +08:00
|
|
|
goto release_qp;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
ret = in_len;
|
|
|
|
|
2015-02-05 19:53:52 +08:00
|
|
|
release_qp:
|
|
|
|
put_qp_read(qp);
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
out:
|
|
|
|
kfree(attr);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_destroy_qp(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
2005-09-10 06:55:08 +08:00
|
|
|
struct ib_uverbs_destroy_qp cmd;
|
|
|
|
struct ib_uverbs_destroy_qp_resp resp;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_uobject *uobj;
|
2005-09-10 06:55:08 +08:00
|
|
|
struct ib_qp *qp;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_uqp_object *obj;
|
2005-09-10 06:55:08 +08:00
|
|
|
int ret = -EINVAL;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2005-09-10 06:55:08 +08:00
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj = idr_write_uobj(&ib_uverbs_qp_idr, cmd.qp_handle, file->ucontext);
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
|
|
|
qp = uobj->object;
|
|
|
|
obj = container_of(uobj, struct ib_uqp_object, uevent.uobject);
|
2005-11-30 08:57:01 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!list_empty(&obj->mcast_list)) {
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
return -EBUSY;
|
2005-11-30 08:57:01 +08:00
|
|
|
}
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
ret = ib_destroy_qp(qp);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!ret)
|
|
|
|
uobj->live = 0;
|
|
|
|
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return ret;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2013-08-01 23:49:54 +08:00
|
|
|
if (obj->uxrcd)
|
|
|
|
atomic_dec(&obj->uxrcd->refcnt);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_qp_idr, uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
list_del(&uobj->list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ib_uverbs_release_uevent(file, &obj->uevent);
|
2005-09-10 06:55:08 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
resp.events_reported = obj->uevent.events_reported;
|
2005-09-10 06:55:08 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj(uobj);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2005-09-10 06:55:08 +08:00
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return -EFAULT;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return in_len;
|
2005-07-08 08:57:13 +08:00
|
|
|
}
|
|
|
|
|
2015-10-08 16:16:33 +08:00
|
|
|
static void *alloc_wr(size_t wr_size, __u32 num_sge)
|
|
|
|
{
|
|
|
|
return kmalloc(ALIGN(wr_size, sizeof (struct ib_sge)) +
|
|
|
|
num_sge * sizeof (struct ib_sge), GFP_KERNEL);
|
|
|
|
};
|
|
|
|
|
2005-10-15 06:26:04 +08:00
|
|
|
ssize_t ib_uverbs_post_send(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2006-02-14 08:30:49 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
2005-10-15 06:26:04 +08:00
|
|
|
{
|
|
|
|
struct ib_uverbs_post_send cmd;
|
|
|
|
struct ib_uverbs_post_send_resp resp;
|
|
|
|
struct ib_uverbs_send_wr *user_wr;
|
|
|
|
struct ib_send_wr *wr = NULL, *last, *next, *bad_wr;
|
|
|
|
struct ib_qp *qp;
|
|
|
|
int i, sg_ind;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
int is_ud;
|
2005-10-15 06:26:04 +08:00
|
|
|
ssize_t ret = -EINVAL;
|
2015-12-01 23:13:51 +08:00
|
|
|
size_t next_size;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
if (in_len < sizeof cmd + cmd.wqe_size * cmd.wr_count +
|
|
|
|
cmd.sge_count * sizeof (struct ib_uverbs_sge))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (cmd.wqe_size < sizeof (struct ib_uverbs_send_wr))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
user_wr = kmalloc(cmd.wqe_size, GFP_KERNEL);
|
|
|
|
if (!user_wr)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
qp = idr_read_qp(cmd.qp_handle, file->ucontext);
|
|
|
|
if (!qp)
|
2005-10-15 06:26:04 +08:00
|
|
|
goto out;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
is_ud = qp->qp_type == IB_QPT_UD;
|
2005-10-15 06:26:04 +08:00
|
|
|
sg_ind = 0;
|
|
|
|
last = NULL;
|
|
|
|
for (i = 0; i < cmd.wr_count; ++i) {
|
|
|
|
if (copy_from_user(user_wr,
|
|
|
|
buf + sizeof cmd + i * cmd.wqe_size,
|
|
|
|
cmd.wqe_size)) {
|
|
|
|
ret = -EFAULT;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto out_put;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
if (user_wr->num_sge + sg_ind > cmd.sge_count) {
|
|
|
|
ret = -EINVAL;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto out_put;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
2015-10-08 16:16:33 +08:00
|
|
|
if (is_ud) {
|
|
|
|
struct ib_ud_wr *ud;
|
|
|
|
|
|
|
|
if (user_wr->opcode != IB_WR_SEND &&
|
|
|
|
user_wr->opcode != IB_WR_SEND_WITH_IMM) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto out_put;
|
|
|
|
}
|
|
|
|
|
2015-12-01 23:13:51 +08:00
|
|
|
next_size = sizeof(*ud);
|
|
|
|
ud = alloc_wr(next_size, user_wr->num_sge);
|
2015-10-08 16:16:33 +08:00
|
|
|
if (!ud) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto out_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
ud->ah = idr_read_ah(user_wr->wr.ud.ah, file->ucontext);
|
|
|
|
if (!ud->ah) {
|
|
|
|
kfree(ud);
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto out_put;
|
|
|
|
}
|
|
|
|
ud->remote_qpn = user_wr->wr.ud.remote_qpn;
|
|
|
|
ud->remote_qkey = user_wr->wr.ud.remote_qkey;
|
|
|
|
|
|
|
|
next = &ud->wr;
|
|
|
|
} else if (user_wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM ||
|
|
|
|
user_wr->opcode == IB_WR_RDMA_WRITE ||
|
|
|
|
user_wr->opcode == IB_WR_RDMA_READ) {
|
|
|
|
struct ib_rdma_wr *rdma;
|
|
|
|
|
2015-12-01 23:13:51 +08:00
|
|
|
next_size = sizeof(*rdma);
|
|
|
|
rdma = alloc_wr(next_size, user_wr->num_sge);
|
2015-10-08 16:16:33 +08:00
|
|
|
if (!rdma) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto out_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
rdma->remote_addr = user_wr->wr.rdma.remote_addr;
|
|
|
|
rdma->rkey = user_wr->wr.rdma.rkey;
|
|
|
|
|
|
|
|
next = &rdma->wr;
|
|
|
|
} else if (user_wr->opcode == IB_WR_ATOMIC_CMP_AND_SWP ||
|
|
|
|
user_wr->opcode == IB_WR_ATOMIC_FETCH_AND_ADD) {
|
|
|
|
struct ib_atomic_wr *atomic;
|
|
|
|
|
2015-12-01 23:13:51 +08:00
|
|
|
next_size = sizeof(*atomic);
|
|
|
|
atomic = alloc_wr(next_size, user_wr->num_sge);
|
2015-10-08 16:16:33 +08:00
|
|
|
if (!atomic) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto out_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
atomic->remote_addr = user_wr->wr.atomic.remote_addr;
|
|
|
|
atomic->compare_add = user_wr->wr.atomic.compare_add;
|
|
|
|
atomic->swap = user_wr->wr.atomic.swap;
|
|
|
|
atomic->rkey = user_wr->wr.atomic.rkey;
|
|
|
|
|
|
|
|
next = &atomic->wr;
|
|
|
|
} else if (user_wr->opcode == IB_WR_SEND ||
|
|
|
|
user_wr->opcode == IB_WR_SEND_WITH_IMM ||
|
|
|
|
user_wr->opcode == IB_WR_SEND_WITH_INV) {
|
2015-12-01 23:13:51 +08:00
|
|
|
next_size = sizeof(*next);
|
|
|
|
next = alloc_wr(next_size, user_wr->num_sge);
|
2015-10-08 16:16:33 +08:00
|
|
|
if (!next) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto out_put;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
ret = -EINVAL;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto out_put;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
2015-10-08 16:16:33 +08:00
|
|
|
if (user_wr->opcode == IB_WR_SEND_WITH_IMM ||
|
|
|
|
user_wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM) {
|
|
|
|
next->ex.imm_data =
|
|
|
|
(__be32 __force) user_wr->ex.imm_data;
|
|
|
|
} else if (user_wr->opcode == IB_WR_SEND_WITH_INV) {
|
|
|
|
next->ex.invalidate_rkey = user_wr->ex.invalidate_rkey;
|
|
|
|
}
|
|
|
|
|
2005-10-15 06:26:04 +08:00
|
|
|
if (!last)
|
|
|
|
wr = next;
|
|
|
|
else
|
|
|
|
last->next = next;
|
|
|
|
last = next;
|
|
|
|
|
|
|
|
next->next = NULL;
|
|
|
|
next->wr_id = user_wr->wr_id;
|
|
|
|
next->num_sge = user_wr->num_sge;
|
|
|
|
next->opcode = user_wr->opcode;
|
|
|
|
next->send_flags = user_wr->send_flags;
|
|
|
|
|
|
|
|
if (next->num_sge) {
|
|
|
|
next->sg_list = (void *) next +
|
2015-12-01 23:13:51 +08:00
|
|
|
ALIGN(next_size, sizeof(struct ib_sge));
|
2005-10-15 06:26:04 +08:00
|
|
|
if (copy_from_user(next->sg_list,
|
|
|
|
buf + sizeof cmd +
|
|
|
|
cmd.wr_count * cmd.wqe_size +
|
|
|
|
sg_ind * sizeof (struct ib_sge),
|
|
|
|
next->num_sge * sizeof (struct ib_sge))) {
|
|
|
|
ret = -EFAULT;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto out_put;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
sg_ind += next->num_sge;
|
|
|
|
} else
|
|
|
|
next->sg_list = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
resp.bad_wr = 0;
|
2011-08-09 06:31:51 +08:00
|
|
|
ret = qp->device->post_send(qp->real_qp, wr, &bad_wr);
|
2005-10-15 06:26:04 +08:00
|
|
|
if (ret)
|
|
|
|
for (next = wr; next; next = next->next) {
|
|
|
|
++resp.bad_wr;
|
|
|
|
if (next == bad_wr)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
|
|
|
ret = -EFAULT;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
out_put:
|
|
|
|
put_qp_read(qp);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
|
|
|
while (wr) {
|
2015-10-08 16:16:33 +08:00
|
|
|
if (is_ud && ud_wr(wr)->ah)
|
|
|
|
put_ah_read(ud_wr(wr)->ah);
|
2005-10-15 06:26:04 +08:00
|
|
|
next = wr->next;
|
|
|
|
kfree(wr);
|
|
|
|
wr = next;
|
|
|
|
}
|
|
|
|
|
2006-06-22 22:47:27 +08:00
|
|
|
out:
|
2005-10-15 06:26:04 +08:00
|
|
|
kfree(user_wr);
|
|
|
|
|
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct ib_recv_wr *ib_uverbs_unmarshall_recv(const char __user *buf,
|
|
|
|
int in_len,
|
|
|
|
u32 wr_count,
|
|
|
|
u32 sge_count,
|
|
|
|
u32 wqe_size)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_recv_wr *user_wr;
|
|
|
|
struct ib_recv_wr *wr = NULL, *last, *next;
|
|
|
|
int sg_ind;
|
|
|
|
int i;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (in_len < wqe_size * wr_count +
|
|
|
|
sge_count * sizeof (struct ib_uverbs_sge))
|
|
|
|
return ERR_PTR(-EINVAL);
|
|
|
|
|
|
|
|
if (wqe_size < sizeof (struct ib_uverbs_recv_wr))
|
|
|
|
return ERR_PTR(-EINVAL);
|
|
|
|
|
|
|
|
user_wr = kmalloc(wqe_size, GFP_KERNEL);
|
|
|
|
if (!user_wr)
|
|
|
|
return ERR_PTR(-ENOMEM);
|
|
|
|
|
|
|
|
sg_ind = 0;
|
|
|
|
last = NULL;
|
|
|
|
for (i = 0; i < wr_count; ++i) {
|
|
|
|
if (copy_from_user(user_wr, buf + i * wqe_size,
|
|
|
|
wqe_size)) {
|
|
|
|
ret = -EFAULT;
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (user_wr->num_sge + sg_ind > sge_count) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
|
|
|
next = kmalloc(ALIGN(sizeof *next, sizeof (struct ib_sge)) +
|
|
|
|
user_wr->num_sge * sizeof (struct ib_sge),
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (!next) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!last)
|
|
|
|
wr = next;
|
|
|
|
else
|
|
|
|
last->next = next;
|
|
|
|
last = next;
|
|
|
|
|
|
|
|
next->next = NULL;
|
|
|
|
next->wr_id = user_wr->wr_id;
|
|
|
|
next->num_sge = user_wr->num_sge;
|
|
|
|
|
|
|
|
if (next->num_sge) {
|
|
|
|
next->sg_list = (void *) next +
|
|
|
|
ALIGN(sizeof *next, sizeof (struct ib_sge));
|
|
|
|
if (copy_from_user(next->sg_list,
|
|
|
|
buf + wr_count * wqe_size +
|
|
|
|
sg_ind * sizeof (struct ib_sge),
|
|
|
|
next->num_sge * sizeof (struct ib_sge))) {
|
|
|
|
ret = -EFAULT;
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
sg_ind += next->num_sge;
|
|
|
|
} else
|
|
|
|
next->sg_list = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
kfree(user_wr);
|
|
|
|
return wr;
|
|
|
|
|
|
|
|
err:
|
|
|
|
kfree(user_wr);
|
|
|
|
|
|
|
|
while (wr) {
|
|
|
|
next = wr->next;
|
|
|
|
kfree(wr);
|
|
|
|
wr = next;
|
|
|
|
}
|
|
|
|
|
|
|
|
return ERR_PTR(ret);
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_post_recv(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2006-02-14 08:30:49 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
2005-10-15 06:26:04 +08:00
|
|
|
{
|
|
|
|
struct ib_uverbs_post_recv cmd;
|
|
|
|
struct ib_uverbs_post_recv_resp resp;
|
|
|
|
struct ib_recv_wr *wr, *next, *bad_wr;
|
|
|
|
struct ib_qp *qp;
|
|
|
|
ssize_t ret = -EINVAL;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
wr = ib_uverbs_unmarshall_recv(buf + sizeof cmd,
|
|
|
|
in_len - sizeof cmd, cmd.wr_count,
|
|
|
|
cmd.sge_count, cmd.wqe_size);
|
|
|
|
if (IS_ERR(wr))
|
|
|
|
return PTR_ERR(wr);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
qp = idr_read_qp(cmd.qp_handle, file->ucontext);
|
|
|
|
if (!qp)
|
2005-10-15 06:26:04 +08:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
resp.bad_wr = 0;
|
2011-08-09 06:31:51 +08:00
|
|
|
ret = qp->device->post_recv(qp->real_qp, wr, &bad_wr);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
|
|
|
|
put_qp_read(qp);
|
|
|
|
|
2005-10-15 06:26:04 +08:00
|
|
|
if (ret)
|
|
|
|
for (next = wr; next; next = next->next) {
|
|
|
|
++resp.bad_wr;
|
|
|
|
if (next == bad_wr)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
|
|
|
ret = -EFAULT;
|
|
|
|
|
|
|
|
out:
|
|
|
|
while (wr) {
|
|
|
|
next = wr->next;
|
|
|
|
kfree(wr);
|
|
|
|
wr = next;
|
|
|
|
}
|
|
|
|
|
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_post_srq_recv(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2006-02-14 08:30:49 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
2005-10-15 06:26:04 +08:00
|
|
|
{
|
|
|
|
struct ib_uverbs_post_srq_recv cmd;
|
|
|
|
struct ib_uverbs_post_srq_recv_resp resp;
|
|
|
|
struct ib_recv_wr *wr, *next, *bad_wr;
|
|
|
|
struct ib_srq *srq;
|
|
|
|
ssize_t ret = -EINVAL;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
wr = ib_uverbs_unmarshall_recv(buf + sizeof cmd,
|
|
|
|
in_len - sizeof cmd, cmd.wr_count,
|
|
|
|
cmd.sge_count, cmd.wqe_size);
|
|
|
|
if (IS_ERR(wr))
|
|
|
|
return PTR_ERR(wr);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
srq = idr_read_srq(cmd.srq_handle, file->ucontext);
|
|
|
|
if (!srq)
|
2005-10-15 06:26:04 +08:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
resp.bad_wr = 0;
|
|
|
|
ret = srq->device->post_srq_recv(srq, wr, &bad_wr);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
|
|
|
|
put_srq_read(srq);
|
|
|
|
|
2005-10-15 06:26:04 +08:00
|
|
|
if (ret)
|
|
|
|
for (next = wr; next; next = next->next) {
|
|
|
|
++resp.bad_wr;
|
|
|
|
if (next == bad_wr)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
|
|
|
ret = -EFAULT;
|
|
|
|
|
|
|
|
out:
|
|
|
|
while (wr) {
|
|
|
|
next = wr->next;
|
|
|
|
kfree(wr);
|
|
|
|
wr = next;
|
|
|
|
}
|
|
|
|
|
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-10-15 06:26:04 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_create_ah cmd;
|
|
|
|
struct ib_uverbs_create_ah_resp resp;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
struct ib_pd *pd;
|
|
|
|
struct ib_ah *ah;
|
|
|
|
struct ib_ah_attr attr;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
uobj = kmalloc(sizeof *uobj, GFP_KERNEL);
|
|
|
|
if (!uobj)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2012-05-01 01:27:26 +08:00
|
|
|
init_uobj(uobj, cmd.user_handle, file->ucontext, &ah_lock_class);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
down_write(&uobj->mutex);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
pd = idr_read_pd(cmd.pd_handle, file->ucontext);
|
|
|
|
if (!pd) {
|
2005-10-15 06:26:04 +08:00
|
|
|
ret = -EINVAL;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
attr.dlid = cmd.attr.dlid;
|
|
|
|
attr.sl = cmd.attr.sl;
|
|
|
|
attr.src_path_bits = cmd.attr.src_path_bits;
|
|
|
|
attr.static_rate = cmd.attr.static_rate;
|
2006-01-07 08:24:45 +08:00
|
|
|
attr.ah_flags = cmd.attr.is_global ? IB_AH_GRH : 0;
|
2005-10-15 06:26:04 +08:00
|
|
|
attr.port_num = cmd.attr.port_num;
|
|
|
|
attr.grh.flow_label = cmd.attr.grh.flow_label;
|
|
|
|
attr.grh.sgid_index = cmd.attr.grh.sgid_index;
|
|
|
|
attr.grh.hop_limit = cmd.attr.grh.hop_limit;
|
|
|
|
attr.grh.traffic_class = cmd.attr.grh.traffic_class;
|
2014-09-26 23:15:32 +08:00
|
|
|
memset(&attr.dmac, 0, sizeof(attr.dmac));
|
2005-10-15 06:26:04 +08:00
|
|
|
memcpy(attr.grh.dgid.raw, cmd.attr.grh.dgid, 16);
|
|
|
|
|
|
|
|
ah = ib_create_ah(pd, &attr);
|
|
|
|
if (IS_ERR(ah)) {
|
|
|
|
ret = PTR_ERR(ah);
|
2006-07-17 23:20:51 +08:00
|
|
|
goto err_put;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ah->uobject = uobj;
|
|
|
|
uobj->object = ah;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ret = idr_add_uobj(&ib_uverbs_ah_idr, uobj);
|
2005-10-15 06:26:04 +08:00
|
|
|
if (ret)
|
|
|
|
goto err_destroy;
|
|
|
|
|
|
|
|
resp.ah_handle = uobj->id;
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp)) {
|
|
|
|
ret = -EFAULT;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_copy;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_pd_read(pd);
|
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
2005-10-15 06:26:04 +08:00
|
|
|
list_add_tail(&uobj->list, &file->ucontext->ah_list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj->live = 1;
|
|
|
|
|
|
|
|
up_write(&uobj->mutex);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
|
|
|
return in_len;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_copy:
|
|
|
|
idr_remove_uobj(&ib_uverbs_ah_idr, uobj);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
|
|
|
err_destroy:
|
|
|
|
ib_destroy_ah(ah);
|
|
|
|
|
2006-07-17 23:20:51 +08:00
|
|
|
err_put:
|
|
|
|
put_pd_read(pd);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err:
|
|
|
|
put_uobj_write(uobj);
|
2005-10-15 06:26:04 +08:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_destroy_ah(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-10-15 06:26:04 +08:00
|
|
|
const char __user *buf, int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_destroy_ah cmd;
|
|
|
|
struct ib_ah *ah;
|
|
|
|
struct ib_uobject *uobj;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
int ret;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj = idr_write_uobj(&ib_uverbs_ah_idr, cmd.ah_handle, file->ucontext);
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
|
|
|
ah = uobj->object;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ret = ib_destroy_ah(ah);
|
|
|
|
if (!ret)
|
|
|
|
uobj->live = 0;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj_write(uobj);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return ret;
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_ah_idr, uobj);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
2005-10-15 06:26:04 +08:00
|
|
|
list_del(&uobj->list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj(uobj);
|
2005-10-15 06:26:04 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return in_len;
|
2005-10-15 06:26:04 +08:00
|
|
|
}
|
|
|
|
|
2005-07-08 08:57:13 +08:00
|
|
|
ssize_t ib_uverbs_attach_mcast(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_attach_mcast cmd;
|
|
|
|
struct ib_qp *qp;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_uqp_object *obj;
|
2005-11-30 08:57:01 +08:00
|
|
|
struct ib_uverbs_mcast_entry *mcast;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
int ret;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2012-01-04 12:36:48 +08:00
|
|
|
qp = idr_write_qp(cmd.qp_handle, file->ucontext);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!qp)
|
|
|
|
return -EINVAL;
|
2005-11-30 08:57:01 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj = container_of(qp->uobject, struct ib_uqp_object, uevent.uobject);
|
2005-11-30 08:57:01 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
list_for_each_entry(mcast, &obj->mcast_list, list)
|
2005-11-30 08:57:01 +08:00
|
|
|
if (cmd.mlid == mcast->lid &&
|
|
|
|
!memcmp(cmd.gid, mcast->gid.raw, sizeof mcast->gid.raw)) {
|
|
|
|
ret = 0;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto out_put;
|
2005-11-30 08:57:01 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
mcast = kmalloc(sizeof *mcast, GFP_KERNEL);
|
|
|
|
if (!mcast) {
|
|
|
|
ret = -ENOMEM;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto out_put;
|
2005-11-30 08:57:01 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
mcast->lid = cmd.mlid;
|
|
|
|
memcpy(mcast->gid.raw, cmd.gid, sizeof mcast->gid.raw);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2005-11-30 08:57:01 +08:00
|
|
|
ret = ib_attach_mcast(qp, &mcast->gid, cmd.mlid);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!ret)
|
|
|
|
list_add_tail(&mcast->list, &obj->mcast_list);
|
|
|
|
else
|
2005-11-30 08:57:01 +08:00
|
|
|
kfree(mcast);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
out_put:
|
2012-01-04 12:36:48 +08:00
|
|
|
put_qp_write(qp);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_detach_mcast(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-07-08 08:57:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_detach_mcast cmd;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_uqp_object *obj;
|
2005-07-08 08:57:13 +08:00
|
|
|
struct ib_qp *qp;
|
2005-11-30 08:57:01 +08:00
|
|
|
struct ib_uverbs_mcast_entry *mcast;
|
2005-07-08 08:57:13 +08:00
|
|
|
int ret = -EINVAL;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2012-01-04 12:36:48 +08:00
|
|
|
qp = idr_write_qp(cmd.qp_handle, file->ucontext);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
if (!qp)
|
|
|
|
return -EINVAL;
|
2005-07-08 08:57:13 +08:00
|
|
|
|
2005-11-30 08:57:01 +08:00
|
|
|
ret = ib_detach_mcast(qp, (union ib_gid *) cmd.gid, cmd.mlid);
|
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto out_put;
|
2005-11-30 08:57:01 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj = container_of(qp->uobject, struct ib_uqp_object, uevent.uobject);
|
2005-11-30 08:57:01 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
list_for_each_entry(mcast, &obj->mcast_list, list)
|
2005-11-30 08:57:01 +08:00
|
|
|
if (cmd.mlid == mcast->lid &&
|
|
|
|
!memcmp(cmd.gid, mcast->gid.raw, sizeof mcast->gid.raw)) {
|
|
|
|
list_del(&mcast->list);
|
|
|
|
kfree(mcast);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
out_put:
|
2012-01-04 12:36:48 +08:00
|
|
|
put_qp_write(qp);
|
2005-07-08 08:57:13 +08:00
|
|
|
|
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2013-11-07 06:21:46 +08:00
|
|
|
static int kern_spec_to_ib_spec(struct ib_uverbs_flow_spec *kern_spec,
|
2013-08-14 18:58:30 +08:00
|
|
|
union ib_flow_spec *ib_spec)
|
|
|
|
{
|
2013-12-12 06:01:49 +08:00
|
|
|
if (kern_spec->reserved)
|
|
|
|
return -EINVAL;
|
|
|
|
|
2013-08-14 18:58:30 +08:00
|
|
|
ib_spec->type = kern_spec->type;
|
|
|
|
|
|
|
|
switch (ib_spec->type) {
|
|
|
|
case IB_FLOW_SPEC_ETH:
|
|
|
|
ib_spec->eth.size = sizeof(struct ib_flow_spec_eth);
|
|
|
|
if (ib_spec->eth.size != kern_spec->eth.size)
|
|
|
|
return -EINVAL;
|
|
|
|
memcpy(&ib_spec->eth.val, &kern_spec->eth.val,
|
|
|
|
sizeof(struct ib_flow_eth_filter));
|
|
|
|
memcpy(&ib_spec->eth.mask, &kern_spec->eth.mask,
|
|
|
|
sizeof(struct ib_flow_eth_filter));
|
|
|
|
break;
|
|
|
|
case IB_FLOW_SPEC_IPV4:
|
|
|
|
ib_spec->ipv4.size = sizeof(struct ib_flow_spec_ipv4);
|
|
|
|
if (ib_spec->ipv4.size != kern_spec->ipv4.size)
|
|
|
|
return -EINVAL;
|
|
|
|
memcpy(&ib_spec->ipv4.val, &kern_spec->ipv4.val,
|
|
|
|
sizeof(struct ib_flow_ipv4_filter));
|
|
|
|
memcpy(&ib_spec->ipv4.mask, &kern_spec->ipv4.mask,
|
|
|
|
sizeof(struct ib_flow_ipv4_filter));
|
|
|
|
break;
|
|
|
|
case IB_FLOW_SPEC_TCP:
|
|
|
|
case IB_FLOW_SPEC_UDP:
|
|
|
|
ib_spec->tcp_udp.size = sizeof(struct ib_flow_spec_tcp_udp);
|
|
|
|
if (ib_spec->tcp_udp.size != kern_spec->tcp_udp.size)
|
|
|
|
return -EINVAL;
|
|
|
|
memcpy(&ib_spec->tcp_udp.val, &kern_spec->tcp_udp.val,
|
|
|
|
sizeof(struct ib_flow_tcp_udp_filter));
|
|
|
|
memcpy(&ib_spec->tcp_udp.mask, &kern_spec->tcp_udp.mask,
|
|
|
|
sizeof(struct ib_flow_tcp_udp_filter));
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
int ib_uverbs_ex_create_flow(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
struct ib_udata *ucore,
|
|
|
|
struct ib_udata *uhw)
|
2013-08-14 18:58:30 +08:00
|
|
|
{
|
|
|
|
struct ib_uverbs_create_flow cmd;
|
|
|
|
struct ib_uverbs_create_flow_resp resp;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
struct ib_flow *flow_id;
|
2013-11-07 06:21:45 +08:00
|
|
|
struct ib_uverbs_flow_attr *kern_flow_attr;
|
2013-08-14 18:58:30 +08:00
|
|
|
struct ib_flow_attr *flow_attr;
|
|
|
|
struct ib_qp *qp;
|
|
|
|
int err = 0;
|
|
|
|
void *kern_spec;
|
|
|
|
void *ib_spec;
|
|
|
|
int i;
|
|
|
|
|
2013-12-12 06:01:52 +08:00
|
|
|
if (ucore->inlen < sizeof(cmd))
|
|
|
|
return -EINVAL;
|
|
|
|
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
if (ucore->outlen < sizeof(resp))
|
2013-08-14 18:58:30 +08:00
|
|
|
return -ENOSPC;
|
|
|
|
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
err = ib_copy_from_udata(&cmd, ucore, sizeof(cmd));
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
ucore->inbuf += sizeof(cmd);
|
|
|
|
ucore->inlen -= sizeof(cmd);
|
2013-08-14 18:58:30 +08:00
|
|
|
|
2013-09-01 23:39:52 +08:00
|
|
|
if (cmd.comp_mask)
|
|
|
|
return -EINVAL;
|
|
|
|
|
2013-08-14 18:58:30 +08:00
|
|
|
if ((cmd.flow_attr.type == IB_FLOW_ATTR_SNIFFER &&
|
|
|
|
!capable(CAP_NET_ADMIN)) || !capable(CAP_NET_RAW))
|
|
|
|
return -EPERM;
|
|
|
|
|
2013-11-07 06:21:44 +08:00
|
|
|
if (cmd.flow_attr.num_of_specs > IB_FLOW_SPEC_SUPPORT_LAYERS)
|
2013-09-01 23:39:52 +08:00
|
|
|
return -EINVAL;
|
|
|
|
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
if (cmd.flow_attr.size > ucore->inlen ||
|
2013-11-07 06:21:44 +08:00
|
|
|
cmd.flow_attr.size >
|
2013-11-07 06:21:46 +08:00
|
|
|
(cmd.flow_attr.num_of_specs * sizeof(struct ib_uverbs_flow_spec)))
|
2013-09-01 23:39:52 +08:00
|
|
|
return -EINVAL;
|
|
|
|
|
2013-12-12 06:01:49 +08:00
|
|
|
if (cmd.flow_attr.reserved[0] ||
|
|
|
|
cmd.flow_attr.reserved[1])
|
|
|
|
return -EINVAL;
|
|
|
|
|
2013-08-14 18:58:30 +08:00
|
|
|
if (cmd.flow_attr.num_of_specs) {
|
2013-11-07 06:21:44 +08:00
|
|
|
kern_flow_attr = kmalloc(sizeof(*kern_flow_attr) + cmd.flow_attr.size,
|
|
|
|
GFP_KERNEL);
|
2013-08-14 18:58:30 +08:00
|
|
|
if (!kern_flow_attr)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
memcpy(kern_flow_attr, &cmd.flow_attr, sizeof(*kern_flow_attr));
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
err = ib_copy_from_udata(kern_flow_attr + 1, ucore,
|
|
|
|
cmd.flow_attr.size);
|
|
|
|
if (err)
|
2013-08-14 18:58:30 +08:00
|
|
|
goto err_free_attr;
|
|
|
|
} else {
|
|
|
|
kern_flow_attr = &cmd.flow_attr;
|
|
|
|
}
|
|
|
|
|
|
|
|
uobj = kmalloc(sizeof(*uobj), GFP_KERNEL);
|
|
|
|
if (!uobj) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto err_free_attr;
|
|
|
|
}
|
|
|
|
init_uobj(uobj, 0, file->ucontext, &rule_lock_class);
|
|
|
|
down_write(&uobj->mutex);
|
|
|
|
|
|
|
|
qp = idr_read_qp(cmd.qp_handle, file->ucontext);
|
|
|
|
if (!qp) {
|
|
|
|
err = -EINVAL;
|
|
|
|
goto err_uobj;
|
|
|
|
}
|
|
|
|
|
2013-11-07 06:21:44 +08:00
|
|
|
flow_attr = kmalloc(sizeof(*flow_attr) + cmd.flow_attr.size, GFP_KERNEL);
|
2013-08-14 18:58:30 +08:00
|
|
|
if (!flow_attr) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto err_put;
|
|
|
|
}
|
|
|
|
|
|
|
|
flow_attr->type = kern_flow_attr->type;
|
|
|
|
flow_attr->priority = kern_flow_attr->priority;
|
|
|
|
flow_attr->num_of_specs = kern_flow_attr->num_of_specs;
|
|
|
|
flow_attr->port = kern_flow_attr->port;
|
|
|
|
flow_attr->flags = kern_flow_attr->flags;
|
|
|
|
flow_attr->size = sizeof(*flow_attr);
|
|
|
|
|
|
|
|
kern_spec = kern_flow_attr + 1;
|
|
|
|
ib_spec = flow_attr + 1;
|
2013-11-07 06:21:44 +08:00
|
|
|
for (i = 0; i < flow_attr->num_of_specs &&
|
2013-11-07 06:21:46 +08:00
|
|
|
cmd.flow_attr.size > offsetof(struct ib_uverbs_flow_spec, reserved) &&
|
2013-11-07 06:21:44 +08:00
|
|
|
cmd.flow_attr.size >=
|
2013-11-07 06:21:46 +08:00
|
|
|
((struct ib_uverbs_flow_spec *)kern_spec)->size; i++) {
|
2013-08-14 18:58:30 +08:00
|
|
|
err = kern_spec_to_ib_spec(kern_spec, ib_spec);
|
|
|
|
if (err)
|
|
|
|
goto err_free;
|
|
|
|
flow_attr->size +=
|
|
|
|
((union ib_flow_spec *) ib_spec)->size;
|
2013-11-07 06:21:46 +08:00
|
|
|
cmd.flow_attr.size -= ((struct ib_uverbs_flow_spec *)kern_spec)->size;
|
|
|
|
kern_spec += ((struct ib_uverbs_flow_spec *) kern_spec)->size;
|
2013-08-14 18:58:30 +08:00
|
|
|
ib_spec += ((union ib_flow_spec *) ib_spec)->size;
|
|
|
|
}
|
2013-11-07 06:21:44 +08:00
|
|
|
if (cmd.flow_attr.size || (i != flow_attr->num_of_specs)) {
|
|
|
|
pr_warn("create flow failed, flow %d: %d bytes left from uverb cmd\n",
|
|
|
|
i, cmd.flow_attr.size);
|
2013-12-12 06:01:50 +08:00
|
|
|
err = -EINVAL;
|
2013-08-14 18:58:30 +08:00
|
|
|
goto err_free;
|
|
|
|
}
|
|
|
|
flow_id = ib_create_flow(qp, flow_attr, IB_FLOW_DOMAIN_USER);
|
|
|
|
if (IS_ERR(flow_id)) {
|
|
|
|
err = PTR_ERR(flow_id);
|
|
|
|
goto err_free;
|
|
|
|
}
|
|
|
|
flow_id->qp = qp;
|
|
|
|
flow_id->uobject = uobj;
|
|
|
|
uobj->object = flow_id;
|
|
|
|
|
|
|
|
err = idr_add_uobj(&ib_uverbs_rule_idr, uobj);
|
|
|
|
if (err)
|
|
|
|
goto destroy_flow;
|
|
|
|
|
|
|
|
memset(&resp, 0, sizeof(resp));
|
|
|
|
resp.flow_handle = uobj->id;
|
|
|
|
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
err = ib_copy_to_udata(ucore,
|
|
|
|
&resp, sizeof(resp));
|
|
|
|
if (err)
|
2013-08-14 18:58:30 +08:00
|
|
|
goto err_copy;
|
|
|
|
|
|
|
|
put_qp_read(qp);
|
|
|
|
mutex_lock(&file->mutex);
|
|
|
|
list_add_tail(&uobj->list, &file->ucontext->rule_list);
|
|
|
|
mutex_unlock(&file->mutex);
|
|
|
|
|
|
|
|
uobj->live = 1;
|
|
|
|
|
|
|
|
up_write(&uobj->mutex);
|
|
|
|
kfree(flow_attr);
|
|
|
|
if (cmd.flow_attr.num_of_specs)
|
|
|
|
kfree(kern_flow_attr);
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
return 0;
|
2013-08-14 18:58:30 +08:00
|
|
|
err_copy:
|
|
|
|
idr_remove_uobj(&ib_uverbs_rule_idr, uobj);
|
|
|
|
destroy_flow:
|
|
|
|
ib_destroy_flow(flow_id);
|
|
|
|
err_free:
|
|
|
|
kfree(flow_attr);
|
|
|
|
err_put:
|
|
|
|
put_qp_read(qp);
|
|
|
|
err_uobj:
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
err_free_attr:
|
|
|
|
if (cmd.flow_attr.num_of_specs)
|
|
|
|
kfree(kern_flow_attr);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
int ib_uverbs_ex_destroy_flow(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
struct ib_udata *ucore,
|
|
|
|
struct ib_udata *uhw)
|
|
|
|
{
|
2013-08-14 18:58:30 +08:00
|
|
|
struct ib_uverbs_destroy_flow cmd;
|
|
|
|
struct ib_flow *flow_id;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
int ret;
|
|
|
|
|
2013-12-12 06:01:52 +08:00
|
|
|
if (ucore->inlen < sizeof(cmd))
|
|
|
|
return -EINVAL;
|
|
|
|
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
ret = ib_copy_from_udata(&cmd, ucore, sizeof(cmd));
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
2013-08-14 18:58:30 +08:00
|
|
|
|
2013-12-12 06:01:48 +08:00
|
|
|
if (cmd.comp_mask)
|
|
|
|
return -EINVAL;
|
|
|
|
|
2013-08-14 18:58:30 +08:00
|
|
|
uobj = idr_write_uobj(&ib_uverbs_rule_idr, cmd.flow_handle,
|
|
|
|
file->ucontext);
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
|
|
|
flow_id = uobj->object;
|
|
|
|
|
|
|
|
ret = ib_destroy_flow(flow_id);
|
|
|
|
if (!ret)
|
|
|
|
uobj->live = 0;
|
|
|
|
|
|
|
|
put_uobj_write(uobj);
|
|
|
|
|
|
|
|
idr_remove_uobj(&ib_uverbs_rule_idr, uobj);
|
|
|
|
|
|
|
|
mutex_lock(&file->mutex);
|
|
|
|
list_del(&uobj->list);
|
|
|
|
mutex_unlock(&file->mutex);
|
|
|
|
|
|
|
|
put_uobj(uobj);
|
|
|
|
|
IB/core: extended command: an improved infrastructure for uverbs commands
Commit 400dbc96583f ("IB/core: Infrastructure for extensible uverbs
commands") added an infrastructure for extensible uverbs commands
while later commit 436f2ad05a0b ("IB/core: Export ib_create/destroy_flow
through uverbs") exported ib_create_flow()/ib_destroy_flow() functions
using this new infrastructure.
According to the commit 400dbc96583f, the purpose of this
infrastructure is to support passing around provider (eg. hardware)
specific buffers when userspace issue commands to the kernel, so that
it would be possible to extend uverbs (eg. core) buffers independently
from the provider buffers.
But the new kernel command function prototypes were not modified to
take advantage of this extension. This issue was exposed by Roland
Dreier in a previous review[1].
So the following patch is an attempt to a revised extensible command
infrastructure.
This improved extensible command infrastructure distinguish between
core (eg. legacy)'s command/response buffers from provider
(eg. hardware)'s command/response buffers: each extended command
implementing function is given a struct ib_udata to hold core
(eg. uverbs) input and output buffers, and another struct ib_udata to
hold the hw (eg. provider) input and output buffers.
Having those buffers identified separately make it easier to increase
one buffer to support extension without having to add some code to
guess the exact size of each command/response parts: This should make
the extended functions more reliable.
Additionally, instead of relying on command identifier being greater
than IB_USER_VERBS_CMD_THRESHOLD, the proposed infrastructure rely on
unused bits in command field: on the 32 bits provided by command
field, only 6 bits are really needed to encode the identifier of
commands currently supported by the kernel. (Even using only 6 bits
leaves room for about 23 new commands).
So this patch makes use of some high order bits in command field to
store flags, leaving enough room for more command identifiers than one
will ever need (eg. 256).
The new flags are used to specify if the command should be processed
as an extended one or a legacy one. While designing the new command
format, care was taken to make usage of flags itself extensible.
Using high order bits of the commands field ensure that newer
libibverbs on older kernel will properly fail when trying to call
extended commands. On the other hand, older libibverbs on newer kernel
will never be able to issue calls to extended commands.
The extended command header includes the optional response pointer so
that output buffer length and output buffer pointer are located
together in the command, allowing proper parameters checking. This
should make implementing functions easier and safer.
Additionally the extended header ensure 64bits alignment, while making
all sizes multiple of 8 bytes, extending the maximum buffer size:
legacy extended
Maximum command buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
Maximum response buffer: 256KBytes 1024KBytes (512KBytes + 512KBytes)
For the purpose of doing proper buffer size accounting, the headers
size are no more taken in account in "in_words".
One of the odds of the current extensible infrastructure, reading
twice the "legacy" command header, is fixed by removing the "legacy"
command header from the extended command header: they are processed as
two different parts of the command: memory is read once and
information are not duplicated: it's making clear that's an extended
command scheme and not a different command scheme.
The proposed scheme will format input (command) and output (response)
buffers this way:
- command:
legacy header +
extended header +
command data (core + hw):
+----------------------------------------+
| flags | 00 00 | command |
| in_words | out_words |
+----------------------------------------+
| response |
| response |
| provider_in_words | provider_out_words |
| padding |
+----------------------------------------+
| |
. <uverbs input> .
. (in_words * 8) .
| |
+----------------------------------------+
| |
. <provider input> .
. (provider_in_words * 8) .
| |
+----------------------------------------+
- response, if present:
+----------------------------------------+
| |
. <uverbs output space> .
. (out_words * 8) .
| |
+----------------------------------------+
| |
. <provider output space> .
. (provider_out_words * 8) .
| |
+----------------------------------------+
The overall design is to ensure that the extensible infrastructure is
itself extensible while begin more reliable with more input and bound
checking.
Note:
The unused field in the extended header would be perfect candidate to
hold the command "comp_mask" (eg. bit field used to handle
compatibility). This was suggested by Roland Dreier in a previous
review[2]. But "comp_mask" field is likely to be present in the uverb
input and/or provider input, likewise for the response, as noted by
Matan Barak[3], so it doesn't make sense to put "comp_mask" in the
header.
[1]:
http://marc.info/?i=CAL1RGDWxmM17W2o_era24A-TTDeKyoL6u3NRu_=t_dhV_ZA9MA@mail.gmail.com
[2]:
http://marc.info/?i=CAL1RGDXJtrc849M6_XNZT5xO1+ybKtLWGq6yg6LhoSsKpsmkYA@mail.gmail.com
[3]:
http://marc.info/?i=525C1149.6000701@mellanox.com
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Link: http://marc.info/?i=cover.1383773832.git.ydroneaud@opteya.com
[ Convert "ret ? ret : 0" to the equivalent "ret". - Roland ]
Signed-off-by: Roland Dreier <roland@purestorage.com>
2013-11-07 06:21:49 +08:00
|
|
|
return ret;
|
2013-08-14 18:58:30 +08:00
|
|
|
}
|
|
|
|
|
2011-12-07 05:13:10 +08:00
|
|
|
static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2011-12-07 05:13:10 +08:00
|
|
|
struct ib_uverbs_create_xsrq *cmd,
|
|
|
|
struct ib_udata *udata)
|
2005-08-19 03:24:13 +08:00
|
|
|
{
|
|
|
|
struct ib_uverbs_create_srq_resp resp;
|
2011-05-26 08:08:38 +08:00
|
|
|
struct ib_usrq_object *obj;
|
2005-08-19 03:24:13 +08:00
|
|
|
struct ib_pd *pd;
|
|
|
|
struct ib_srq *srq;
|
2011-05-26 08:08:38 +08:00
|
|
|
struct ib_uobject *uninitialized_var(xrcd_uobj);
|
2005-08-19 03:24:13 +08:00
|
|
|
struct ib_srq_init_attr attr;
|
|
|
|
int ret;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
obj = kmalloc(sizeof *obj, GFP_KERNEL);
|
|
|
|
if (!obj)
|
2005-08-19 03:24:13 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
|
2012-05-01 01:27:26 +08:00
|
|
|
init_uobj(&obj->uevent.uobject, cmd->user_handle, file->ucontext, &srq_lock_class);
|
2011-05-26 08:08:38 +08:00
|
|
|
down_write(&obj->uevent.uobject.mutex);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
if (cmd->srq_type == IB_SRQT_XRC) {
|
|
|
|
attr.ext.xrc.xrcd = idr_read_xrcd(cmd->xrcd_handle, file->ucontext, &xrcd_uobj);
|
|
|
|
if (!attr.ext.xrc.xrcd) {
|
|
|
|
ret = -EINVAL;
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
goto err;
|
2011-05-26 08:08:38 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
obj->uxrcd = container_of(xrcd_uobj, struct ib_uxrcd_object, uobject);
|
|
|
|
atomic_inc(&obj->uxrcd->refcnt);
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
|
|
|
|
attr.ext.xrc.cq = idr_read_cq(cmd->cq_handle, file->ucontext, 0);
|
|
|
|
if (!attr.ext.xrc.cq) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put_xrcd;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
pd = idr_read_pd(cmd->pd_handle, file->ucontext);
|
|
|
|
if (!pd) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto err_put_cq;
|
2011-05-26 08:08:38 +08:00
|
|
|
}
|
|
|
|
|
2005-08-19 03:24:13 +08:00
|
|
|
attr.event_handler = ib_uverbs_srq_event_handler;
|
|
|
|
attr.srq_context = file;
|
2011-05-26 08:08:38 +08:00
|
|
|
attr.srq_type = cmd->srq_type;
|
|
|
|
attr.attr.max_wr = cmd->max_wr;
|
|
|
|
attr.attr.max_sge = cmd->max_sge;
|
|
|
|
attr.attr.srq_limit = cmd->srq_limit;
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
obj->uevent.events_reported = 0;
|
|
|
|
INIT_LIST_HEAD(&obj->uevent.event_list);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
srq = pd->device->create_srq(pd, &attr, udata);
|
2005-08-19 03:24:13 +08:00
|
|
|
if (IS_ERR(srq)) {
|
|
|
|
ret = PTR_ERR(srq);
|
2006-07-17 23:20:51 +08:00
|
|
|
goto err_put;
|
2005-08-19 03:24:13 +08:00
|
|
|
}
|
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
srq->device = pd->device;
|
|
|
|
srq->pd = pd;
|
|
|
|
srq->srq_type = cmd->srq_type;
|
|
|
|
srq->uobject = &obj->uevent.uobject;
|
2005-08-19 03:24:13 +08:00
|
|
|
srq->event_handler = attr.event_handler;
|
|
|
|
srq->srq_context = attr.srq_context;
|
2011-05-26 08:08:38 +08:00
|
|
|
|
|
|
|
if (cmd->srq_type == IB_SRQT_XRC) {
|
|
|
|
srq->ext.xrc.cq = attr.ext.xrc.cq;
|
|
|
|
srq->ext.xrc.xrcd = attr.ext.xrc.xrcd;
|
|
|
|
atomic_inc(&attr.ext.xrc.cq->usecnt);
|
|
|
|
atomic_inc(&attr.ext.xrc.xrcd->usecnt);
|
|
|
|
}
|
|
|
|
|
2005-08-19 03:24:13 +08:00
|
|
|
atomic_inc(&pd->usecnt);
|
|
|
|
atomic_set(&srq->usecnt, 0);
|
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
obj->uevent.uobject.object = srq;
|
|
|
|
ret = idr_add_uobj(&ib_uverbs_srq_idr, &obj->uevent.uobject);
|
2005-08-19 03:24:13 +08:00
|
|
|
if (ret)
|
|
|
|
goto err_destroy;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
memset(&resp, 0, sizeof resp);
|
2011-05-26 08:08:38 +08:00
|
|
|
resp.srq_handle = obj->uevent.uobject.id;
|
2006-02-24 04:36:18 +08:00
|
|
|
resp.max_wr = attr.attr.max_wr;
|
|
|
|
resp.max_sge = attr.attr.max_sge;
|
2011-05-26 08:08:38 +08:00
|
|
|
if (cmd->srq_type == IB_SRQT_XRC)
|
|
|
|
resp.srqn = srq->ext.xrc.srq_num;
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd->response,
|
2005-08-19 03:24:13 +08:00
|
|
|
&resp, sizeof resp)) {
|
|
|
|
ret = -EFAULT;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
goto err_copy;
|
2005-08-19 03:24:13 +08:00
|
|
|
}
|
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
if (cmd->srq_type == IB_SRQT_XRC) {
|
|
|
|
put_uobj_read(xrcd_uobj);
|
|
|
|
put_cq_read(attr.ext.xrc.cq);
|
|
|
|
}
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_pd_read(pd);
|
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
2011-05-26 08:08:38 +08:00
|
|
|
list_add_tail(&obj->uevent.uobject.list, &file->ucontext->srq_list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-09-28 06:07:25 +08:00
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
obj->uevent.uobject.live = 1;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
up_write(&obj->uevent.uobject.mutex);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
return 0;
|
2005-08-19 03:24:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err_copy:
|
2011-05-26 08:08:38 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_srq_idr, &obj->uevent.uobject);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
|
|
|
err_destroy:
|
|
|
|
ib_destroy_srq(srq);
|
|
|
|
|
2006-07-17 23:20:51 +08:00
|
|
|
err_put:
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
put_pd_read(pd);
|
2011-05-26 08:08:38 +08:00
|
|
|
|
|
|
|
err_put_cq:
|
|
|
|
if (cmd->srq_type == IB_SRQT_XRC)
|
|
|
|
put_cq_read(attr.ext.xrc.cq);
|
|
|
|
|
IB/uverbs: Lock SRQ / CQ / PD objects in a consistent order
Since XRC support was added, the uverbs code has locked SRQ, CQ and PD
objects needed during QP and SRQ creation in different orders
depending on the the code path. This leads to the (at least
theoretical) possibility of deadlock, and triggers the lockdep splat
below.
Fix this by making sure we always lock the SRQ first, then CQs and
finally the PD.
======================================================
[ INFO: possible circular locking dependency detected ]
3.4.0-rc5+ #34 Not tainted
-------------------------------------------------------
ibv_srq_pingpon/2484 is trying to acquire lock:
(SRQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
but task is already holding lock:
(CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (CQ-uobj){+++++.}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b16c3>] ib_uverbs_create_qp+0x180/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #1 (PD-uobj){++++++}:
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00af8ad>] __uverbs_create_xsrq+0x96/0x386 [ib_uverbs]
[<ffffffffa00b31b9>] ib_uverbs_detach_mcast+0x1cd/0x1e6 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
-> #0 (SRQ-uobj){+++++.}:
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
other info that might help us debug this:
Chain exists of:
SRQ-uobj --> PD-uobj --> CQ-uobj
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(CQ-uobj);
lock(PD-uobj);
lock(CQ-uobj);
lock(SRQ-uobj);
*** DEADLOCK ***
3 locks held by ibv_srq_pingpon/2484:
#0: (QP-uobj){+.+...}, at: [<ffffffffa00b162c>] ib_uverbs_create_qp+0xe9/0x684 [ib_uverbs]
#1: (PD-uobj){++++++}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
#2: (CQ-uobj){+++++.}, at: [<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
stack backtrace:
Pid: 2484, comm: ibv_srq_pingpon Not tainted 3.4.0-rc5+ #34
Call Trace:
[<ffffffff8137eff0>] print_circular_bug+0x1f8/0x209
[<ffffffff81070898>] __lock_acquire+0xa29/0xd06
[<ffffffffa00af37c>] ? __idr_get_uobj+0x20/0x5e [ib_uverbs]
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070fd0>] lock_acquire+0xbf/0xfe
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffff81070eee>] ? lock_release+0x166/0x189
[<ffffffff81384f28>] down_read+0x34/0x43
[<ffffffffa00af51b>] ? idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af51b>] idr_read_uobj+0x2f/0x4d [ib_uverbs]
[<ffffffffa00af542>] idr_read_obj+0x9/0x19 [ib_uverbs]
[<ffffffffa00b1728>] ib_uverbs_create_qp+0x1e5/0x684 [ib_uverbs]
[<ffffffff81070fec>] ? lock_acquire+0xdb/0xfe
[<ffffffff81070c09>] ? lock_release_non_nested+0x94/0x213
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffff810d470f>] ? might_fault+0x40/0x90
[<ffffffffa00ae3dd>] ib_uverbs_write+0xb7/0xc2 [ib_uverbs]
[<ffffffff810fe47f>] vfs_write+0xa7/0xee
[<ffffffff810ff736>] ? fget_light+0x3b/0x99
[<ffffffff810fe65f>] sys_write+0x45/0x69
[<ffffffff8138cdf9>] system_call_fastpath+0x16/0x1b
Reported-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>
2012-05-01 03:51:50 +08:00
|
|
|
err_put_xrcd:
|
|
|
|
if (cmd->srq_type == IB_SRQT_XRC) {
|
|
|
|
atomic_dec(&obj->uxrcd->refcnt);
|
|
|
|
put_uobj_read(xrcd_uobj);
|
|
|
|
}
|
2006-07-17 23:20:51 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
err:
|
2011-05-26 08:08:38 +08:00
|
|
|
put_uobj_write(&obj->uevent.uobject);
|
2005-08-19 03:24:13 +08:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2011-05-26 08:08:38 +08:00
|
|
|
ssize_t ib_uverbs_create_srq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2011-05-26 08:08:38 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_create_srq cmd;
|
|
|
|
struct ib_uverbs_create_xsrq xcmd;
|
|
|
|
struct ib_uverbs_create_srq_resp resp;
|
|
|
|
struct ib_udata udata;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
xcmd.response = cmd.response;
|
|
|
|
xcmd.user_handle = cmd.user_handle;
|
|
|
|
xcmd.srq_type = IB_SRQT_BASIC;
|
|
|
|
xcmd.pd_handle = cmd.pd_handle;
|
|
|
|
xcmd.max_wr = cmd.max_wr;
|
|
|
|
xcmd.max_sge = cmd.max_sge;
|
|
|
|
xcmd.srq_limit = cmd.srq_limit;
|
|
|
|
|
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd,
|
|
|
|
(unsigned long) cmd.response + sizeof resp,
|
|
|
|
in_len - sizeof cmd, out_len - sizeof resp);
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
ret = __uverbs_create_xsrq(file, ib_dev, &xcmd, &udata);
|
2011-05-26 08:08:38 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
ssize_t ib_uverbs_create_xsrq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2011-05-26 08:08:38 +08:00
|
|
|
const char __user *buf, int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_create_xsrq cmd;
|
|
|
|
struct ib_uverbs_create_srq_resp resp;
|
|
|
|
struct ib_udata udata;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd,
|
|
|
|
(unsigned long) cmd.response + sizeof resp,
|
|
|
|
in_len - sizeof cmd, out_len - sizeof resp);
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
ret = __uverbs_create_xsrq(file, ib_dev, &cmd, &udata);
|
2011-05-26 08:08:38 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
return in_len;
|
|
|
|
}
|
|
|
|
|
2005-08-19 03:24:13 +08:00
|
|
|
ssize_t ib_uverbs_modify_srq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-08-19 03:24:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_modify_srq cmd;
|
2006-08-12 05:58:09 +08:00
|
|
|
struct ib_udata udata;
|
2005-08-19 03:24:13 +08:00
|
|
|
struct ib_srq *srq;
|
|
|
|
struct ib_srq_attr attr;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
2006-08-12 05:58:09 +08:00
|
|
|
INIT_UDATA(&udata, buf + sizeof cmd, NULL, in_len - sizeof cmd,
|
|
|
|
out_len);
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
srq = idr_read_srq(cmd.srq_handle, file->ucontext);
|
|
|
|
if (!srq)
|
|
|
|
return -EINVAL;
|
2005-08-19 03:24:13 +08:00
|
|
|
|
|
|
|
attr.max_wr = cmd.max_wr;
|
|
|
|
attr.srq_limit = cmd.srq_limit;
|
|
|
|
|
2006-08-12 05:58:09 +08:00
|
|
|
ret = srq->device->modify_srq(srq, &attr, cmd.attr_mask, &udata);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_srq_read(srq);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
|
|
|
|
2006-02-14 08:31:57 +08:00
|
|
|
ssize_t ib_uverbs_query_srq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2006-02-14 08:31:57 +08:00
|
|
|
const char __user *buf,
|
|
|
|
int in_len, int out_len)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_query_srq cmd;
|
|
|
|
struct ib_uverbs_query_srq_resp resp;
|
|
|
|
struct ib_srq_attr attr;
|
|
|
|
struct ib_srq *srq;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (out_len < sizeof resp)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
srq = idr_read_srq(cmd.srq_handle, file->ucontext);
|
|
|
|
if (!srq)
|
|
|
|
return -EINVAL;
|
2006-02-14 08:31:57 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ret = ib_query_srq(srq, &attr);
|
2006-02-14 08:31:57 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_srq_read(srq);
|
2006-02-14 08:31:57 +08:00
|
|
|
|
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return ret;
|
2006-02-14 08:31:57 +08:00
|
|
|
|
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
|
|
|
|
resp.max_wr = attr.max_wr;
|
|
|
|
resp.max_sge = attr.max_sge;
|
|
|
|
resp.srq_limit = attr.srq_limit;
|
|
|
|
|
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return -EFAULT;
|
2006-02-14 08:31:57 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return in_len;
|
2006-02-14 08:31:57 +08:00
|
|
|
}
|
|
|
|
|
2005-08-19 03:24:13 +08:00
|
|
|
ssize_t ib_uverbs_destroy_srq(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2005-08-19 03:24:13 +08:00
|
|
|
const char __user *buf, int in_len,
|
|
|
|
int out_len)
|
|
|
|
{
|
2005-09-10 06:55:08 +08:00
|
|
|
struct ib_uverbs_destroy_srq cmd;
|
|
|
|
struct ib_uverbs_destroy_srq_resp resp;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_uobject *uobj;
|
2005-09-10 06:55:08 +08:00
|
|
|
struct ib_srq *srq;
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
struct ib_uevent_object *obj;
|
2005-09-10 06:55:08 +08:00
|
|
|
int ret = -EINVAL;
|
2013-08-01 23:49:54 +08:00
|
|
|
struct ib_usrq_object *us;
|
|
|
|
enum ib_srq_type srq_type;
|
2005-08-19 03:24:13 +08:00
|
|
|
|
|
|
|
if (copy_from_user(&cmd, buf, sizeof cmd))
|
|
|
|
return -EFAULT;
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
uobj = idr_write_uobj(&ib_uverbs_srq_idr, cmd.srq_handle, file->ucontext);
|
|
|
|
if (!uobj)
|
|
|
|
return -EINVAL;
|
|
|
|
srq = uobj->object;
|
|
|
|
obj = container_of(uobj, struct ib_uevent_object, uobject);
|
2013-08-01 23:49:54 +08:00
|
|
|
srq_type = srq->srq_type;
|
2005-09-10 06:55:08 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ret = ib_destroy_srq(srq);
|
|
|
|
if (!ret)
|
|
|
|
uobj->live = 0;
|
2005-08-19 03:24:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj_write(uobj);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
|
|
|
if (ret)
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
return ret;
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2013-08-01 23:49:54 +08:00
|
|
|
if (srq_type == IB_SRQT_XRC) {
|
|
|
|
us = container_of(obj, struct ib_usrq_object, uevent);
|
|
|
|
atomic_dec(&us->uxrcd->refcnt);
|
|
|
|
}
|
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
idr_remove_uobj(&ib_uverbs_srq_idr, uobj);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_lock(&file->mutex);
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
list_del(&uobj->list);
|
2006-01-14 06:51:39 +08:00
|
|
|
mutex_unlock(&file->mutex);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
ib_uverbs_release_uevent(file, obj);
|
2005-09-10 06:55:08 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
memset(&resp, 0, sizeof resp);
|
|
|
|
resp.events_reported = obj->events_reported;
|
2005-09-10 06:55:08 +08:00
|
|
|
|
IB/uverbs: Don't serialize with ib_uverbs_idr_mutex
Currently, all userspace verbs operations that call into the kernel
are serialized by ib_uverbs_idr_mutex. This can be a scalability
issue for some workloads, especially for devices driven by the ipath
driver, which needs to call into the kernel even for datapath
operations.
Fix this by adding reference counts to the userspace objects, and then
converting ib_uverbs_idr_mutex into a spinlock that only protects the
idrs long enough to take a reference on the object being looked up.
Because remove operations may fail, we have to do a slightly funky
two-step deletion, which is described in the comments at the top of
uverbs_cmd.c.
This also still leaves ib_uverbs_idr_lock as a single lock that is
possibly subject to contention. However, the lock hold time will only
be a single idr operation, so multiple threads should still be able to
make progress, even if ib_uverbs_idr_lock is being ping-ponged.
Surprisingly, these changes even shrink the object code:
add/remove: 23/5 grow/shrink: 4/21 up/down: 633/-693 (-60)
Signed-off-by: Roland Dreier <rolandd@cisco.com>
2006-06-18 11:44:49 +08:00
|
|
|
put_uobj(uobj);
|
2005-08-19 03:24:13 +08:00
|
|
|
|
2005-09-10 06:55:08 +08:00
|
|
|
if (copy_to_user((void __user *) (unsigned long) cmd.response,
|
|
|
|
&resp, sizeof resp))
|
|
|
|
ret = -EFAULT;
|
|
|
|
|
2005-08-19 03:24:13 +08:00
|
|
|
return ret ? ret : in_len;
|
|
|
|
}
|
2015-02-08 19:28:50 +08:00
|
|
|
|
|
|
|
int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
|
2015-08-13 23:32:04 +08:00
|
|
|
struct ib_device *ib_dev,
|
2015-02-08 19:28:50 +08:00
|
|
|
struct ib_udata *ucore,
|
|
|
|
struct ib_udata *uhw)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_ex_query_device_resp resp;
|
|
|
|
struct ib_uverbs_ex_query_device cmd;
|
|
|
|
struct ib_device_attr attr;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (ucore->inlen < sizeof(cmd))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
err = ib_copy_from_udata(&cmd, ucore, sizeof(cmd));
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
if (cmd.comp_mask)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (cmd.reserved)
|
|
|
|
return -EINVAL;
|
|
|
|
|
2015-02-08 19:28:51 +08:00
|
|
|
resp.response_length = offsetof(typeof(resp), odp_caps);
|
2015-02-08 19:28:50 +08:00
|
|
|
|
|
|
|
if (ucore->outlen < resp.response_length)
|
|
|
|
return -ENOSPC;
|
|
|
|
|
2015-06-11 21:35:24 +08:00
|
|
|
memset(&attr, 0, sizeof(attr));
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
err = ib_dev->query_device(ib_dev, &attr, uhw);
|
2015-02-08 19:28:50 +08:00
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
2015-08-13 23:32:04 +08:00
|
|
|
copy_query_dev_fields(file, ib_dev, &resp.base, &attr);
|
2015-02-08 19:28:50 +08:00
|
|
|
resp.comp_mask = 0;
|
|
|
|
|
2015-02-08 19:28:51 +08:00
|
|
|
if (ucore->outlen < resp.response_length + sizeof(resp.odp_caps))
|
|
|
|
goto end;
|
|
|
|
|
|
|
|
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
|
|
|
|
resp.odp_caps.general_caps = attr.odp_caps.general_caps;
|
|
|
|
resp.odp_caps.per_transport_caps.rc_odp_caps =
|
|
|
|
attr.odp_caps.per_transport_caps.rc_odp_caps;
|
|
|
|
resp.odp_caps.per_transport_caps.uc_odp_caps =
|
|
|
|
attr.odp_caps.per_transport_caps.uc_odp_caps;
|
|
|
|
resp.odp_caps.per_transport_caps.ud_odp_caps =
|
|
|
|
attr.odp_caps.per_transport_caps.ud_odp_caps;
|
|
|
|
resp.odp_caps.reserved = 0;
|
|
|
|
#else
|
|
|
|
memset(&resp.odp_caps, 0, sizeof(resp.odp_caps));
|
|
|
|
#endif
|
|
|
|
resp.response_length += sizeof(resp.odp_caps);
|
|
|
|
|
2015-06-11 21:35:24 +08:00
|
|
|
if (ucore->outlen < resp.response_length + sizeof(resp.timestamp_mask))
|
|
|
|
goto end;
|
|
|
|
|
|
|
|
resp.timestamp_mask = attr.timestamp_mask;
|
|
|
|
resp.response_length += sizeof(resp.timestamp_mask);
|
|
|
|
|
|
|
|
if (ucore->outlen < resp.response_length + sizeof(resp.hca_core_clock))
|
|
|
|
goto end;
|
|
|
|
|
|
|
|
resp.hca_core_clock = attr.hca_core_clock;
|
|
|
|
resp.response_length += sizeof(resp.hca_core_clock);
|
|
|
|
|
2015-02-08 19:28:51 +08:00
|
|
|
end:
|
2015-02-08 19:28:50 +08:00
|
|
|
err = ib_copy_to_udata(ucore, &resp, resp.response_length);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|