Commit Graph

143 Commits

Author SHA1 Message Date
Eddie Wai aa106202e4 [SCSI] bnx2fc: Bump version from 1.0.14 to 2.4.1
Signed-off-by: Eddie Wai <eddie.wai@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-10-25 09:58:03 +01:00
Eddie Wai 0680810c11 [SCSI] BNX2FC: hung task timeout warning observed when rmmod bnx2x with active FCoE targets
[v2] - removed the interface->enabled flag setting which prevented the
       fcoe ctlr link from being brought back up after a MTU change

A rtnl_lock deadlock was observed from the rmmod thread where it
tries to unregister the fcoe_ctlr device.  This unregistration
triggered a flush of the sysfs queue of the associated ctlr and led to
a call to the set_fcoe_ctlr_enabled routine.  This will eventually propagate
down to call the bnx2fc_disable routine and contented for the rtnl_lock
in the same context.

This patch creates a subset of the bnx2fc_enable/disable routine which
removes the unnecesary rtnl_lock and the bnx2fc_dev_lock acquisition from
the set_fcoe_ctlr_enabled path.

 kernel: INFO: task rmmod:7874 blocked for more than 120 seconds.
 kernel:      Tainted: G        W  ---------------    2.6.32-415.0.1.el6.x86_64 #1
 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 kernel: rmmod         D 000000000000000f     0  7874   6518 0x00000080
 kernel: ffff88022158f7d8 0000000000000086 0000000000000000 0000000000000000
 kernel: ffff88023fe72600 ffff88043c74d410 ffff88043c74d400 ffff88043c74d000
 kernel: ffff88021ecbe5f8 ffff88022158ffd8 000000000000fbc8 ffff88021ecbe5f8
 kernel: Call Trace:
 kernel: [<ffffffff81525985>] schedule_timeout+0x215/0x2e0
 kernel: [<ffffffff810680c0>] ? pick_next_task_fair+0xd0/0x130
 kernel: [<ffffffff81524858>] ? schedule+0x178/0x3b2
 kernel: [<ffffffff81525603>] wait_for_common+0x123/0x180
 kernel: [<ffffffff81066b40>] ? default_wake_function+0x0/0x20
 kernel: [<ffffffff811a486e>] ? ifind_fast+0x5e/0xb0
 kernel: [<ffffffff8152571d>] wait_for_completion+0x1d/0x20
 kernel: [<ffffffff81203868>] sysfs_addrm_finish+0x228/0x270
 kernel: [<ffffffff812014ab>] sysfs_hash_and_remove+0x5b/0x90
 kernel: [<ffffffff812056af>] sysfs_remove_group+0x5f/0x100
 kernel: [<ffffffff81367e8b>] device_remove_groups+0x3b/0x60
 kernel: [<ffffffff8136811d>] device_remove_attrs+0x3d/0x90
 kernel: [<ffffffff81368295>] device_del+0x125/0x1e0
 kernel: [<ffffffff81368372>] device_unregister+0x22/0x60
 kernel: [<ffffffffa038ead2>] fcoe_ctlr_device_delete+0xe2/0xf4 [libfcoe]
 kernel: [<ffffffffa03c43cb>] bnx2fc_interface_release+0x5b/0x90 [bnx2fc]
 kernel: [<ffffffffa03c4370>] ? bnx2fc_interface_release+0x0/0x90 [bnx2fc]
 kernel: [<ffffffff812835e7>] kref_put+0x37/0x70
 kernel: [<ffffffffa03c4192>] __bnx2fc_destroy+0x72/0xa0 [bnx2fc]
 kernel: [<ffffffffa03c5265>] bnx2fc_ulp_exit+0xf5/0x160 [bnx2fc]    <- got bnx2fc_dev_lock mutex_lock
 kernel: [<ffffffffa03b03c6>] cnic_ulp_exit+0xb6/0xc0 [cnic]
 kernel: [<ffffffffa03b5418>] cnic_netdev_event+0x368/0x370 [cnic]
 kernel: [<ffffffffa038c56c>] ? fcoe_del_netdev_mapping+0x8c/0xa0 [libfcoe]
 kernel: [<ffffffff8152a6e5>] notifier_call_chain+0x55/0x80
 kernel: [<ffffffff810a0a46>] raw_notifier_call_chain+0x16/0x20
 kernel: [<ffffffff81459beb>] call_netdevice_notifiers+0x1b/0x20
 kernel: [<ffffffff8145ab34>] rollback_registered_many+0x154/0x280
 kernel: [<ffffffff8145ad08>] rollback_registered+0x38/0x50
 kernel: [<ffffffff8145ad78>] unregister_netdevice_queue+0x58/0xa0
 kernel: [<ffffffff8145add0>] unregister_netdevice+0x10/0x20
 kernel: [<ffffffff8145adfe>] unregister_netdev+0x1e/0x30                  <- got rtnl_lock!!!!!!!!!
 kernel: [<ffffffffa0122278>] __bnx2x_remove+0x48/0x270 [bnx2x] <- got & rel rtnl_lock
 kernel: [<ffffffffa0122554>] bnx2x_remove_one+0x44/0x80 [bnx2x]
 kernel: [<ffffffff812a3af7>] pci_device_remove+0x37/0x70
 kernel: [<ffffffff8136b2ef>] __device_release_driver+0x6f/0xe0
 kernel: [<ffffffff8136b428>] driver_detach+0xc8/0xd0
 kernel: [<ffffffff8136a22e>] bus_remove_driver+0x8e/0x110
 kernel: [<ffffffff8136bc12>] driver_unregister+0x62/0xa0
 kernel: [<ffffffff812a3e04>] pci_unregister_driver+0x44/0xb0
 kernel: [<ffffffffa0191954>] bnx2x_cleanup+0x18/0x73 [bnx2x]
 kernel: [<ffffffff810b8be4>] sys_delete_module+0x194/0x260
 kernel: [<ffffffff810e1347>] ? audit_syscall_entry+0x1d7/0x200
 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

Signed-off-by: Eddie Wai <eddie.wai@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-10-25 09:58:02 +01:00
Eddie Wai 2183789609 [SCSI] bnx2fc: Fixed a SCSI CMD cmpl race condition between ABTS and CLEANUP
In the case when a SCSI_CMD times out, bnx2fc will initiate the sending of the
ABTS.  However, if the SCSI layer's SCSI command timer also times out, it'll
instantiate a task abort of the same xid.

The race condition this patch tries to fix is as follows:

SCSI_CMD timeout (20s)
thread 1                   thread 2
send ABTS
rx ABTS cmpl
                           task abort_eh
                           explicit LOGO since ABTS was engaged
                           CLEANUP cmpl
SCSI_CMD cmpl (ABTS cmpl)
instantiate RRQ
wait 10s
attempt to send RRQ (because of LOGO, it wouldn't continue)

Note that there is no call to scsi_done for this SCSI_CMD cmpletion
in this path.

The patch changes the path of execution to call scsi_done immediately
instead of instantiating the RRQ.

Signed-off-by: Eddie Wai <eddie.wai@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-10-25 09:58:02 +01:00
Michael Chan f78afb3563 bnx2x, cnic, bnx2i, bnx2fc: Fix bnx2i and bnx2fc regressions.
commit b9871bcfd2
    bnx2x: VF RSS support - PF side

changed the configuration of the doorbell HW and it broke iSCSI and FCoE.
We fix this by making compatible changes to the doorbell address in bnx2i
and bnx2fc.  For the userspace driver, we need to pass a modified CID
so that the existing userspace driver will calculate the correct doorbell
address and continue to work.

Signed-off-by: Ariel Elior <ariele@broadcom.com>
Signed-off-by: Eddie Wai <eddie.wai@broadcom.com>
Signed-off-by: Michael Chan <mchan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-09-18 12:24:32 -04:00
Bhanu Prakash Gollapudi 0a7d1d360a [SCSI] bnx2fc: Bumped version to 1.0.14
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-05-02 07:32:26 -07:00
Bhanu Prakash Gollapudi cf1221912f [SCSI] bnx2fc: Update copyright dates
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-05-02 07:32:25 -07:00
Bhanu Prakash Gollapudi 5d78f175d0 [SCSI] bnx2fc: Fix race condition between IO completion and abort
When IO is successfully completed while an abort is pending, eh_abort
incorrectly assumes that abort failed and performes recovery by issuing
cleanup. Howerver, cleanup timesout as the firmware has no clue about
this IO. Fix this by checking if the IO has already completed.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-05-02 07:32:25 -07:00
Bhanu Prakash Gollapudi c13d2b6d36 [SCSI] bnx2fc: Include chip number in the symbolic name
[jejb: move PCI_DEVICE_ID definitions to include/pci_ids.h]
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-05-02 07:32:24 -07:00
Bhanu Prakash Gollapudi 353c2ade1d [SCSI] bnx2fc: Enable cached tasks to improve performance
Set perf_config to 3 during firmware initialization to enable both
cached connections as well as cached tasks.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-05-02 07:32:24 -07:00
Bhanu Prakash Gollapudi 0eb43b4bb0 bnx2x, bnx2fc: Use per port max exchange resources
The firmware supports a maximum of 4K FCoE exchanges. In 4-port devices,
or when working in multi-function mode, this resource needs to be distributed
between the various possible FCoE functions.

This information needs to be calculated by bnx2x and propagated into bnx2fc
via cnic. bnx2fc can then use this value to calculate corresponding xid
resources instead of using global constants.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: Michael Chan <mchan@broadcom.com>
Signed-off-by: Yuval Mintz <yuvalmin@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-25 04:06:46 -04:00
Robert Love 0807619d3c libfc, fcoe, bnx2fc: Split fc_disc_init into fc_disc_{init, config}
Split discovery initialization in code that is setup once (fcoe_disc_init)
and code that can be re-configured (fcoe_disc_config).

Signed-off-by: Robert Love <robert.w.love@intel.com>
Tested-by: Jack Morgan <jack.morgan@intel.com>
Reviewed-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
2013-03-25 16:03:03 -07:00
Robert Love 8a9a713812 libfc, fcoe, bnx2fc: Always use fcoe_disc_init for discovery layer initialization
Currently libfcoe is doing some libfc discovery layer initialization outside of
libfc. This patch moves this code into libfc and sets up a split in discovery
(one time) initialization code and (re-configurable) settings that will come in
the next patch.

Signed-off-by: Robert Love <robert.w.love@intel.com>
Tested-by: Jack Morgan <jack.morgan@intel.com>
Reviewed-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
2013-03-25 16:01:10 -07:00
Robert Love 01bdcb626f bnx2fc: Make the fcoe_cltr the SCSI host parent
The fcoemon userspace daemon is searching for the a hostX
under the the /sys/bus/fcoe/devices/ctlrX/ entries. When
interfaces created using fcoe_sysfs and fcoe.ko this linkage
is setup correctly, but bnx2fc is not doing the same thing
and therefore fcoemon does not create the fcoe interface
for bnx2fc.

This patch sets up the correct linkage for bnx2fc such that
fcoemon will work correctly with fcoe_sysfs and bnx2fc.

Signed-off-by: Robert Love <robert.w.love@intel.com>
Acked-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
2013-03-25 12:19:15 -07:00
James Bottomley 3e34c1fc2b FCoE Updates for 3.9
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 
 iQIcBAABAgAGBQJRI992AAoJEEajxTw9cn4HCAcQAI0yWjsDYkTZUAN3vcU7Xh83
 e1Qt1dAWjY+sFspo54p4ndK7gkBQzBlzIPf1MdcTQOSBWSA2QYnvukpZuRm99OjH
 /60xky0TgGtvGUbNgCRG9zhrPQ+/rTvuenZrZa9ibWeIMM8RgyWBV76REDyetd3b
 K6q2Cz75MKDZwgwJpp7FPF7SJYQVgVbhbVJFIvN5rtS2xEUZI7Ltj/stW0QoJfkD
 R1XAlGCJETV+pzG0VY7aycxCGhN3HM7QYiA1EiKCfBkLzfFosf+oozWTgIkLm1PE
 UKO8b6RNR0d7BHXBkIgP0C18Wx9BrLWS5woNoiDANBK3FTJIgL/D32tiJJXhLFGP
 zzY4OXDT0OEUsn1oc8ldI5/LqALdvDJGzUBbiA1wTiXsJyHVAsBajJc3pP9btnom
 Z91xB4nRLQx/doJqaKOZiQJRCSeG7b/lQM06jOEHFAO7Ah9sRSqbelHd1+tmSV6J
 NfDhkZsi5rPZ9My23J5Nkcfy4Vi8hpyqNC0KZR7PTUYJwi1LrTKRSTkek53b5K3O
 Ee7ZtQ1cY3v90QuG2bNieWaLUKINX/nTKF5DIKH1l9dfxmcpHystPpjRZ06D5wUY
 vsH+9wNAq0v6+tAT8yvoc5qrZb2OEO/xkgCNIN8c5pNGiw0hjeXzhg5bZBGjNmGK
 ixtSPIR5QoaU1XFq62JH
 =r8PC
 -----END PGP SIGNATURE-----

[SCSI] Merge tag 'fcoe-02-19-13' into for-linus

FCoE Updates for 3.9

Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-03-01 09:10:08 +00:00
Julia Lawall b0d5e15cc4 [SCSI] bnx2fc: adjust duplicate test
Delete successive tests to the same location.  The code tested the result
of a previous allocation, that itself was already tested.  It is changed to
test the result of the most recent allocation.

A simplified version of the semantic match that finds this problem is as
follows: (http://coccinelle.lip6.fr/)

// <smpl>
@s exists@
local idexpression y;
expression x,e;
@@

*if ( \(x == NULL\|IS_ERR(x)\|y != 0\) )
 { ... when forall
   return ...; }
... when != \(y = e\|y += e\|y -= e\|y |= e\|y &= e\|y++\|y--\|&y\)
    when != \(XT_GETPAGE(...,y)\|WMI_CMD_BUF(...)\)
*if ( \(x == NULL\|IS_ERR(x)\|y != 0\) )
 { ... when forall
   return ...; }
// </smpl>

Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr>
Acked-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-30 14:04:54 +11:00
Bhanu Prakash Gollapudi 2b82d825a5 [SCSI] bnx2fc: Bumped version to 1.0.13
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-30 10:48:49 +11:00
Bhanu Prakash Gollapudi d450d778cd [SCSI] bnx2fc: Support max IO size to 512KB
Increase max_sectors from 512 to 1024 in order to support max IO size of 512KB.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-30 10:48:49 +11:00
Bhanu Prakash Gollapudi f246fe2a56 [SCSI] bnx2fc: Tx/Rx byte counts reset to 0 when exceeding 32 bit values
Since the FW counters are 32-bit, accumulate the stats in the driver.

[jejb: fix checkpatch warning]
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-30 10:48:48 +11:00
Bhanu Prakash Gollapudi e7f4fed5fa [SCSI] bnx2fc: Map the doorbell register between offload and enable requests
We used to map doorbell register after FW enable request is complete.
This causes a race condition when unsolicited event is received, and FW
sends a CQE for it. Since the doorbell is not mapped, driver does not
arm CQ, which means FW will not notify the driver for further CQ
completions.  To resolve this, map the doorbell between offload and
enable, so that driver is ready to receive the unsolicited packets and
arm the CQ as soon as FW enable is performed.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-30 10:48:48 +11:00
Bhanu Prakash Gollapudi 26bf62a3dd [SCSI] bnx2fc: Move offload/upload wait logic into a function
Moved the 'waiting for upload/offload completion' logic to respective
functions as it is called in multiple places. Also this is done in
preperation for the next patch.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-30 10:48:47 +11:00
Bhanu Prakash Gollapudi 50b7186f64 [SCSI] bnx2fc: support software fcoe target
Software FCoE target always advertises RETRY bit even when there are no
tape LUNs behind the target. This causes the driver to enable FW support
for sequence level error recovery and perform REC/SRR. This patch
arrests the behavior by not enabling SLER feature for this target.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-30 10:47:05 +11:00
Julia Lawall 33c7da0590 [SCSI] bnx2fc: Remove potential NULL dereference
If the NULL test is necessary, the initialization involving a dereference of
the tested value should be moved after the NULL test.

The sematic patch that fixes this problem is as follows:
(http://coccinelle.lip6.fr/)

// <smpl>
@@
type T;
expression E;
identifier i,fld;
statement S;
@@

- T i = E->fld;
+ T i;
  ... when != E
      when != i
  if (E == NULL) S
+ i = E->fld;
// </smpl>

Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-29 15:06:19 +11:00
Cyril Roelandt 0225fea06e [SCSI] bnx2fc: remove useless calls to memset().
These calls are followed by calls to memcpy() on the same memory area, so they
can be safely removed.

Signed-off-by: Cyril Roelandt <tipecaml@gmail.com>
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2013-01-29 15:02:22 +11:00
Greg Kroah-Hartman 6f03979051 Drivers: scsi: remove __dev* attributes.
CONFIG_HOTPLUG is going away as an option.  As a result, the __dev*
markings need to be removed.

This change removes the use of __devinit, __devexit_p, __devinitdata,
__devinitconst, and __devexit from these drivers.

Based on patches originally written by Bill Pemberton, but redone by me
in order to handle some of the coding style issues better, by hand.

Cc: Bill Pemberton <wfp5p@virginia.edu>
Cc: Adam Radford <linuxraid@lsi.com>
Cc: "James E.J. Bottomley" <JBottomley@parallels.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2013-01-03 15:57:01 -08:00
Yi Zou b8b3e697d1 bnx2fc: use fcoe_get_lesb/fcoe_ctlr_get_lesb() directly from libfcoe
Drop the bnx2fc_xxx versions as they are basically the same.

Signed-off-by: Yi Zou <yi.zou@intel.com>
Cc: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Tested-by: Marcus Dennis <marcusx.e.dennis@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
2012-12-14 10:38:56 -08:00
Yi Zou 0e0f9cd6a8 bnx2fc: use fcoe_link_speed_update() from the exported symbol in libfcoe
We have fcoe_link_speed_update() in libfcoe ready for use now, take out the
bnx2fc version which is almost the same.

Signed-off-by: Yi Zou <yi.zou@intel.com>
Cc: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Tested-by: Marcus Dennis <marcusx.e.dennis@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
2012-12-14 10:38:56 -08:00
Yi Zou c3d7909b86 bnx2fc: add support to get_netdev for bnx2f_interface
Adds support to fcoe_port's newly added get_netdev fucntion pointer for bnx2fc.

Signed-off-by: Yi Zou <yi.zou@intel.com>
Cc: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Tested-by: Marcus Dennis <marcusx.e.dennis@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
2012-12-14 10:38:55 -08:00
Robert Love 6e89ea3f10 bnx2fc: Use the fcoe_sysfs control interface
This patch adds support for the new fcoe_sysfs
control interface to bnx2fc.ko. It keeps the deprecated
interface in tact and therefore either the legacy
or the new control interfaces can be used. A mixed mode
is not supported. A user must either use the new
interfaces or the old ones, but not both.

The fcoe_ctlr's link state is now driven by both the
netdev link state as well as the fcoe_ctlr_device's
enabled attribute. The link must be up and the
fcoe_ctlr_device must be enabled before the FCoE
Controller starts discovery or login.

Signed-off-by: Robert Love <robert.w.love@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
2012-12-14 10:38:55 -08:00
Robert Love 6a891b071b libfcoe, fcoe, bnx2fc: Add new fcoe control interface
This patch does a few things.

1) Makes /sys/bus/fcoe/ctlr_{create,destroy} interfaces.
   These interfaces take an <ifname> and will either
   create an FCoE Controller or destroy an FCoE
   Controller depending on which file is written to.

   The new FCoE Controller will start in a DISABLED
   state and will not do discovery or login until it
   is ENABLED. This pause will allow us to configure
   the FCoE Controller before enabling it.

2) Makes the 'mode' attribute of a fcoe_ctlr_device
   writale. This allows the user to configure the mode
   in which the FCoE Controller will start in when it
   is ENABLED.

   Possible modes are 'Fabric', or 'VN2VN'.

   The default mode for a fcoe_ctlr{,_device} is 'Fabric'.
   Drivers must implement the set_fcoe_ctlr_mode routine
   to support this feature.

   libfcoe offers an exported routine to set a FCoE
   Controller's mode. The mode can only be changed
   when the FCoE Controller is DISABLED.

   This patch also removes the get_fcoe_ctlr_mode pointer
   in the fcoe_sysfs function template, the code in
   fcoe_ctlr.c to get the mode and the assignment of
   the fcoe_sysfs function pointer to the fcoe_ctlr.c
   implementation (in fcoe and bnx2fc). fcoe_sysfs can
   return that value for the mode without consulting the
   LLD.

3) Make a 'enabled' attribute of a fcoe_ctlr_device. On a
   read, fcoe_sysfs will return the attribute's value. On
   a write, fcoe_sysfs will call the LLD (if there is a
   callback) to notifiy that the enalbed state has changed.

This patch maintains the old FCoE control interfaces as
module parameters, but it adds comments pointing out that
the old interfaces are deprecated.

Signed-off-by: Robert Love <robert.w.love@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
2012-12-14 10:38:54 -08:00
Masanari Iida d755814808 treewide: fix typo of "suppport" in various comments
Signed-off-by: Masanari Iida <standby24x7@gmail.com>
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
2012-11-19 14:13:52 +01:00
Linus Torvalds 3151367f87 SCSI for-linus on 20121002
This is a large set of updates, mostly for drivers (qla2xxx [including support
 for new 83xx based card], qla4xxx, mpt2sas, bfa, zfcp, hpsa, be2iscsi, isci,
 lpfc, ipr, ibmvfc, ibmvscsi, megaraid_sas).  There's also a rework for tape
 adding virtually unlimited numbers of tape drives plus a set of dif fixes for
 sd and a fix for a live lock on hot remove of SCSI devices.
 
 This round includes a signed tag pull of isci-for-3.6
 
 Signed-off-by: James Bottomley <JBottomley@Parallels.com>
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.19 (GNU/Linux)
 
 iQEcBAABAgAGBQJQaqCFAAoJEDeqqVYsXL0MKJ4IALg/Obnk0/fNvBUNIrh5zRmj
 r9UlXFJnlEDT03qRGdn8okgWMChbgaD1ZrwDTQnjNsabVQoTXI6oO6/uL2c8crpY
 BFBwJvkNJS99nbcZv10CpJ3K7ykmRnKlkYon12iknhGwdtU+XJ14Z4PUcZkI9jmg
 sBQQ6uNVWyosaONNE+k6o+dw6OTttJkzRX8e9in3thstxNTcG+h9iB1zZ/ETkSEj
 tD4MyOgDiPf8kPV2awQThQGpni9Tu3SQr5dEn/iUUktUjiYsDNQuyaAk+QzyhUU7
 D35iIJnIHlXTSTMQkrG4qpJHBvqPkWlYJzaOmheQryQ3vzp2C5Ly/hS9il45uIQ=
 =49u9
 -----END PGP SIGNATURE-----

Merge tag 'scsi-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi

Pull first round of SCSI updates from James Bottomley:
 "This is a large set of updates, mostly for drivers (qla2xxx [including
  support for new 83xx based card], qla4xxx, mpt2sas, bfa, zfcp, hpsa,
  be2iscsi, isci, lpfc, ipr, ibmvfc, ibmvscsi, megaraid_sas).

  There's also a rework for tape adding virtually unlimited numbers of
  tape drives plus a set of dif fixes for sd and a fix for a live lock
  on hot remove of SCSI devices.

  This round includes a signed tag pull of isci-for-3.6

  Signed-off-by: James Bottomley <JBottomley@Parallels.com>"

Fix up trivial conflict in drivers/scsi/qla2xxx/qla_nx.c due to new PCI
helper function use in a function that was removed by this pull.

* tag 'scsi-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi: (198 commits)
  [SCSI] st: remove st_mutex
  [SCSI] sd: Ensure we correctly disable devices with unknown protection type
  [SCSI] hpsa: gen8plus Smart Array IDs
  [SCSI] qla4xxx: Update driver version to 5.03.00-k1
  [SCSI] qla4xxx: Disable generating pause frames for ISP83XX
  [SCSI] qla4xxx: Fix double clearing of risc_intr for ISP83XX
  [SCSI] qla4xxx: IDC implementation for Loopback
  [SCSI] qla4xxx: update copyrights in LICENSE.qla4xxx
  [SCSI] qla4xxx: Fix panic while rmmod
  [SCSI] qla4xxx: Fail probe_adapter if IRQ allocation fails
  [SCSI] qla4xxx: Prevent MSI/MSI-X falling back to INTx for ISP82XX
  [SCSI] qla4xxx: Update idc reg in case of PCI AER
  [SCSI] qla4xxx: Fix double IDC locking in qla4_8xxx_error_recovery
  [SCSI] qla4xxx: Clear interrupt while unloading driver for ISP83XX
  [SCSI] qla4xxx: Print correct IDC version
  [SCSI] qla4xxx: Added new mbox cmd to pass driver version to FW
  [SCSI] scsi_dh_alua: Enable STPG for unavailable ports
  [SCSI] scsi_remove_target: fix softlockup regression on hot remove
  [SCSI] ibmvscsi: Fix host config length field overflow
  [SCSI] ibmvscsi: Remove backend abstraction
  ...
2012-10-02 19:01:32 -07:00
Linus Torvalds 99dbb1632f Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
Pull the trivial tree from Jiri Kosina:
 "Tiny usual fixes all over the place"

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial: (34 commits)
  doc: fix old config name of kprobetrace
  fs/fs-writeback.c: cleanup riteback_sb_inodes kerneldoc
  btrfs: fix the commment for the action flags in delayed-ref.h
  btrfs: fix trivial typo for the comment of BTRFS_FREE_INO_OBJECTID
  vfs: fix kerneldoc for generic_fh_to_parent()
  treewide: fix comment/printk/variable typos
  ipr: fix small coding style issues
  doc: fix broken utf8 encoding
  nfs: comment fix
  platform/x86: fix asus_laptop.wled_type module parameter
  mfd: printk/comment fixes
  doc: getdelays.c: remember to close() socket on error in create_nl_socket()
  doc: aliasing-test: close fd on write error
  mmc: fix comment typos
  dma: fix comments
  spi: fix comment/printk typos in spi
  Coccinelle: fix typo in memdup_user.cocci
  tmiofb: missing NULL pointer checks
  tools: perf: Fix typo in tools/perf
  tools/testing: fix comment / output typos
  ...
2012-10-01 09:06:36 -07:00
Andi Kleen 16da05b115 [SCSI] Fix incorrect memset in bnx2fc_parse_fcp_rsp
gcc 4.8 warns because the memset only clears sizeof(char *) bytes, not
the whole buffer. Use the correct buffer size and clear the whole sense
buffer.

/backup/lsrc/git/linux-lto-2.6/drivers/scsi/bnx2fc/bnx2fc_io.c: In
function 'bnx2fc_parse_fcp_rsp':
/backup/lsrc/git/linux-lto-2.6/drivers/scsi/bnx2fc/bnx2fc_io.c:1810:41:
warning: argument to 'sizeof' in 'memset' call is the same expression as
the destination; did you mean to provide an explicit length?
[-Wsizeof-pointer-memaccess]
   memset(sc_cmd->sense_buffer, 0, sizeof(sc_cmd->sense_buffer));
                                         ^

Signed-off-by: Andi Kleen <ak@linux.intel.com>
Acked-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-09-24 12:11:00 +04:00
Masanari Iida 59e13d4833 scsi: fix various printk and comment typos
Correct spelling typo within drivers/scsi

Signed-off-by: Masanari Iida <standby24x7@gmail.com>
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
2012-09-01 08:24:35 -07:00
Linus Torvalds bdc0077af5 SCSI misc on 20120724
The most important feature of this patch set is the new async infrastructure
 that makes sure async_synchronize_full() synchronizes all domains and allows
 us to remove all the hacks (like having scsi_complete_async_scans() in the
 device base code) and means that the async infrastructure will "just work" in
 future. The rest is assorted driver updates (aacraid, bnx2fc, virto-scsi,
 megaraid, bfa, lpfc, qla2xxx, qla4xxx) plus a lot of infrastructure work in
 sas and FC.
 
 Signed-off-by: James Bottomley <JBottomley@Parallels.com>
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (GNU/Linux)
 
 iQEcBAABAgAGBQJQDjDCAAoJEDeqqVYsXL0M/sMH/jVgBfF1mjR+DQuTscKyD21w
 0BQLn5OmvDZDqo44iqQzNRObw7CxkBkUtHoozsknLijw+KggER653ZOAtUdIHfI/
 /uo7iJQ3J3D/Ezm99HYSpZiF2juZwsBRtFBoKkGqOpMlzFUx5o4hUbH5OcINxnHR
 VmvJU5K1kg8D77Q6zK+Atl14/Rfibc2IoufFmbYdplUAM/tV0BpBSSHJAJvqua76
 NGMl4KJcPZnXe/4LXcxZia5A2efdFFEzaQ2mM9rUVEAgHDAxc0Zg9IoDhGd08FX4
 G55NK+6+bKb9s7bgyva0T/iy817TRCzjteeYNFrb8nBRe7aQbAivaBHQFXIyvdQ=
 =y2sh
 -----END PGP SIGNATURE-----

Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi

Pull first round of SCSI updates from James Bottomley:
 "The most important feature of this patch set is the new async
  infrastructure that makes sure async_synchronize_full() synchronizes
  all domains and allows us to remove all the hacks (like having
  scsi_complete_async_scans() in the device base code) and means that
  the async infrastructure will "just work" in future.

  The rest is assorted driver updates (aacraid, bnx2fc, virto-scsi,
  megaraid, bfa, lpfc, qla2xxx, qla4xxx) plus a lot of infrastructure
  work in sas and FC.

  Signed-off-by: James Bottomley <JBottomley@Parallels.com>"

* tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi: (97 commits)
  [SCSI] Revert "[SCSI] fix async probe regression"
  [SCSI] cleanup usages of scsi_complete_async_scans
  [SCSI] queue async scan work to an async_schedule domain
  [SCSI] async: make async_synchronize_full() flush all work regardless of domain
  [SCSI] async: introduce 'async_domain' type
  [SCSI] bfa: Fix to set correct return error codes and misc cleanup.
  [SCSI] aacraid: Series 7 Async. (performance) mode support
  [SCSI] aha152x: Allow use on 64bit systems
  [SCSI] virtio-scsi: Add vdrv->scan for post VIRTIO_CONFIG_S_DRIVER_OK LUN scanning
  [SCSI] bfa: squelch lockdep complaint with a spin_lock_init
  [SCSI] qla2xxx: remove unnecessary reads of PCI_CAP_ID_EXP
  [SCSI] qla4xxx: remove unnecessary read of PCI_CAP_ID_EXP
  [SCSI] ufs: fix incorrect return value about SUCCESS and FAILED
  [SCSI] ufs: reverse the ufshcd_is_device_present logic
  [SCSI] ufs: use module_pci_driver
  [SCSI] usb-storage: update usb devices for write cache quirk in quirk list.
  [SCSI] usb-storage: add support for write cache quirk
  [SCSI] set to WCE if usb cache quirk is present.
  [SCSI] virtio-scsi: hotplug support for virtio-scsi
  [SCSI] virtio-scsi: split scatterlist per target
  ...
2012-07-24 18:11:22 -07:00
Bhanu Prakash Gollapudi eb47aa2c21 [SCSI] bnx2fc: Bumped version to 1.0.12
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-07-20 08:58:25 +01:00
Bhanu Prakash Gollapudi d71fb3bdee [SCSI] bnx2fc: use list_entry instead of explicit cast
Use list_for_each_entry_safe() instead of explicit cast to avoid relying on
struct layout

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-07-20 08:58:24 +01:00
Bhanu Prakash Gollapudi 5c17ae217a [SCSI] bnx2fc: Improve error recovery by handling parity errors
During parity errors, the ramrods are not issued to FW. bnx2fc waits for the
timeout value, and proceeds with cleaning up the IOs. Since we are already
out-of-sync with FW, cleanup commands timeout too, and do not get the
completion.  This operation takes 36 secs for each session to upload causing
huge delays. To fix this, bnx2fc now gets a PARITY_ERROR from cnic driver, and
upon failure, the driver does not issue any commands to the FW and finishes the
upload process sooner.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-07-20 08:58:23 +01:00
Bhanu Prakash Gollapudi 7adc5a3746 [SCSI] bnx2fc: Support interface creation on non-VLAN interface also.
bnx2fc had an assumption that the fcoe interface will always start on the vlan
dev. However, some switch implementations (Eg., HP virtual connect FlexFabric)
expects the fcoe interface to be started on physical interface. Do not error
out if the netdev is not a vlan dev.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-07-20 08:58:19 +01:00
Joe Perches 3db8cc1065 [SCSI] bnx2fc: Reduce object size by consolidating formats
Deduplication of formats and consolidating tests
makes the object much smaller.

Add bnx2fc_debug.c, add functions for a few logging
functions (BNX2FC_IO_DBG, BNX2FC_TGT_DBG, BNX2FC_HBA_DBG).
Use printf extension %pV.
Add and use pr_fmt and pr_<level>.
Move the debug #include below structure definitions.

$ size drivers/scsi/bnx2fc/built-in.o*
   text	   data	    bss	    dec	    hex	filename
 101563	   1165	  24976	 127704	  1f2d8	drivers/scsi/bnx2fc/built-in.o.new
 138473	   1109	  33400	 172982	  2a3b6	drivers/scsi/bnx2fc/built-in.o.old

Signed-off-by: Joe Perches <joe@perches.com>
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-07-20 08:58:17 +01:00
Eric Dumazet 69614270e1 [SCSI] bnx2fc: use kthread_create_on_node
Since bnx2fc_percpu_thread_create() creates percpu kthread, it makes
sense to use kthread_create_on_node() to get proper NUMA affinity for
kthread stack.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-07-20 08:58:14 +01:00
Vasu Dev 1bd49b4820 [SCSI] libfc, fcoe, bnx2fc: cleanup fcoe_dev_stats
The libfc is used by fcoe but fcoe agnostic,
and therefore should not have any fcoe references.

So renaming fcoe_dev_stats from libfc as its for fc_stats.
After that libfc is fcoe string free except some strings for
Open-FCoE.org.

Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Acked-by : Robert Love <robert.w.love@intel.com>
Tested-by: Ross Brattain <ross.b.brattain@intel.com>
Acked-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-07-20 08:31:47 +01:00
Barak Witkowski 2e499d3cc1 bnx2x, bnx2fc, bnx2i, cnic: Add statistics support and FCoE capabilities advertisement
1. When FCoE offload driver is registered, copy its capabilities to the chip
   scratchpad.
2. Copy FCoE/iSCSI MAC addresses in aligned manner to chip scratchpad.
3. Add FCoE/iSCSI statistics collection support

Signed-off-by: Barak Witkowski <barak@broadcom.com>
Signed-off-by: Eilon Greenstein <eilong@broadcom.com>
Signed-off-by: Eddie Wai <eddie.wai@broadcom.com>
Signed-off-by: Michael Chan <mchan@broadcom.com>
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-06-27 01:20:13 -07:00
Robert Love 8d55e507d2 [SCSI] fcoe, bnx2fc, libfcoe: SW FCoE and bnx2fc use FCoE Syfs
This patch has the SW FCoE driver and the bnx2fc
driver make use of the new fcoe_sysfs API added
earlier in this patch series.

After this patch a fcoe_ctlr_device is allocated with
private data in this order.

+------------------+   +------------------+
| fcoe_ctlr_device |   | fcoe_ctlr_device |
+------------------+   +------------------+
| fcoe_ctlr        |   | fcoe_ctlr        |
+------------------+   +------------------+
| fcoe_interface   |   | bnx2fc_interface |
+------------------+   +------------------+

libfcoe also takes part in this new model since it
discovers and manages fcoe_fcf instances. The memory
allocation is different for FCFs. I didn't want to
impact libfcoe's fcoe_fcf processing, so this patch
creates fcoe_fcf_device instances for each discovered
fcoe_fcf. The two are paired using a (void * priv)
member of the fcoe_ctlr_device. This allows libfcoe
to continue maintaining its list of fcoe_fcf instances
and simply attaches and detaches them from existing
or new fcoe_fcf_device instances.

Signed-off-by: Robert Love <robert.w.love@intel.com>
Tested-by: Ross Brattain <ross.b.brattain@intel.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-05-23 09:43:13 +01:00
Robert Love fd8f89027d [SCSI] bnx2fc: Allocate fcoe_ctlr with bnx2fc_interface, not as a member
Currently the fcoe_ctlr associated with an interface is allocated
    as a member of struct bnx2fc_interface. This causes problems when
    when later patches attempt to use the new fcoe_sysfs APIs which
    allow us to allocate the bnx2fc_interface as private data to a
    fcoe_ctlr_device instance. The problem is that libfcoe wants to
    be able use pointer math to find a fcoe_ctlr's fcoe_ctlr_device
    as well as finding a fcoe_ctlr_device's assocated fcoe_ctlr. To
    do this we need to allocate the fcoe_ctlr_device, with private
    data for the LLD. The private data will contain the fcoe_ctlr
    and its private data will be the bnx2fc_interface.

    +-------------------+
    | fcoe_ctlr_device  |
    +-------------------+
    | fcoe_ctlr         |
    +-------------------+
    | bnx2fc_interface  |
    +-------------------+

    This prep work will allow us to go from a fcoe_ctlr_device
    instance to its fcoe_ctlr as well as from a fcoe_ctlr to its
    fcoe_ctlr_device once the fcoe_sysfs API is in use (later
    patches in this series).

Signed-off-by: Robert Love <robert.w.love@intel.com>
Tested-by: Ross Brattain <ross.b.brattain@intel.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-05-23 09:38:11 +01:00
Bhanu Prakash Gollapudi e91506dbcf [SCSI] bnx2fc: Bumped version to 1.0.11
Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-05-22 10:03:00 +01:00
Bhanu Prakash Gollapudi 92886c9c97 [SCSI] bnx2fc: cleanup task management IO when it times out.
When the task management IO times out, or a flush operation is performed while
task management IO is pending, driver is not cleaning up the IO. This patch
cleans up the IO for the above cases.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-05-22 10:01:26 +01:00
Bhanu Prakash Gollapudi c1bb4f33de [SCSI] bnx2fc: Decrememnt io ref count when abort times out
When IO abort times out during eh_abort or a flush operation is performed while
abort is pending, the driver is not cleaning up the IO and thus not reducing
the IO reference count. With this change, as part of explicit logout, the IO is
cleaned up.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-05-22 10:00:06 +01:00
Bhanu Prakash Gollapudi de9c05fafc [SCSI] bnx2fc: Allow FLOGI to be retried when receiving bad responses.
Commit 907c07d451 added more cases to do FLOGI
retry on receiving bad response. Remove the code that drops the packet and
allow the stack to handle bad FLOGI response.

Signed-off-by: Bhanu Prakash Gollapudi <bprakash@broadcom.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
2012-05-22 09:50:36 +01:00
Linus Torvalds a75ee6ecd4 SCSI updates on 20120331
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (GNU/Linux)
 
 iQEcBAABAgAGBQJPdrIWAAoJEDeqqVYsXL0Mny4IAMTzXGOXCykpWhdIe2R8w0Ys
 eIoTJhBKoQWnLTV8cOODtwmtZcoQLeXkZmizZiAJvX6O1tOgueg+W4AFa9grxXGI
 O0d1bSb2ardzU7VZrZSY60Hd4bylMwn4Xv/0dRrQMwTJO0LEeGWsJPV2+2BuXwMB
 lGCNB67oUBXgMOI1jUZQRwx/mBzQ3e/gINjnpZTNKHia7YkX/yVTFISq7htgfDN7
 1wRGxymbHtVap3NbtUO96BUUndAiF5vom+4WNvaQUyPrCc6aoGWjv+J9DQXY/zgv
 AYjujAluK396D6YncGFAWBzYOg9WFbq54v0PRUanjcTTAu5ILs2BxqWdhmnvl14=
 =IH8T
 -----END PGP SIGNATURE-----

Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi-misc-2.6

Pull SCSI updates from James Bottomley:
 "This is primarily another round of driver updates (lpfc, bfa, fcoe,
  ipr) plus a new ufshcd driver.  There shouldn't be anything
  controversial in here (The final deletion of scsi proc_ops which
  caused some build breakage has been held over until the next merge
  window to give us more time to stabilise it).

  I'm afraid, with me moving continents at exactly the wrong time,
  anything submitted after the merge window opened has been held over to
  the next merge window."

* tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi-misc-2.6: (63 commits)
  [SCSI] ipr: Driver version 2.5.3
  [SCSI] ipr: Increase alignment boundary of command blocks
  [SCSI] ipr: Increase max concurrent oustanding commands
  [SCSI] ipr: Remove unnecessary memory barriers
  [SCSI] ipr: Remove unnecessary interrupt clearing on new adapters
  [SCSI] ipr: Fix target id allocation re-use problem
  [SCSI] atp870u, mpt2sas, qla4xxx use pci_dev->revision
  [SCSI] fcoe: Drop the rtnl_mutex before calling fcoe_ctlr_link_up
  [SCSI] bfa: Update the driver version to 3.0.23.0
  [SCSI] bfa: BSG and User interface fixes.
  [SCSI] bfa: Fix to avoid vport delete hang on request queue full scenario.
  [SCSI] bfa: Move service parameter programming logic into firmware.
  [SCSI] bfa: Revised Fabric Assigned Address(FAA) feature implementation.
  [SCSI] bfa: Flash controller IOC pll init fixes.
  [SCSI] bfa: Serialize the IOC hw semaphore unlock logic.
  [SCSI] bfa: Modify ISR to process pending completions
  [SCSI] bfa: Add fc host issue lip support
  [SCSI] mpt2sas: remove extraneous sas_log_info messages
  [SCSI] libfc: fcoe_transport_create fails in single-CPU environment
  [SCSI] fcoe: reduce contention for fcoe_rx_list lock [v2]
  ...
2012-03-31 13:31:23 -07:00