Add functions to fill in an FC header given a request header.
These reduces code lines in fc_lport and fc_rport and works
without an exchange/sequence assigned.
fc_fill_reply_hdr() fills a header for a final reply frame.
fc_fill_hdr() which is similar but allows specifying the
f_ctl parameter.
Add defines for F_CTL values FC_FCTL_REQ and FC_FCTL_RESP.
These can be used for most request and response sequences.
v2 of patch adds a line to copy the frame encapsulation
info from the received frame.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
To pave the way for eliminating exchanges from incoming requests,
add simple inline fc_frame_sid() and fc_frame_did() functions
which get the FC_IDs from the frame header. This can be almost
as efficient as getting them from the sequence/exchange.
Move ntohll, htonll, ntoh24 and hton24 to <scsi/fc_frame.h>
since we need them there and that's included by <scsi/libfc.h>
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The LOGO state hasn't been used in a while, except in a brief
transition to DELETE state while holding the rport mutex.
All port LOGO responses have been ignored as well as any timeout
if we don't get a response.
So this patch just removes LOGO state and simplifies the response handler.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
When an exchange is received with a FIP encapsulation, we need
to know that the response must be sent via FIP and what the original
ELS opcode was. This becomes important for VN2VN mode, where we may
receive FLOGI or LOGO from several peer VN_ports, and the LS_ACC or
LS_RJT must be sent FIP-encapsulated with the correct sub-type.
Add a field to the struct fc_frame, fr_encaps, to indicate the
encapsulation values. That term is chosen to be neutral and
LLD-agnostic in case non-FCoE/FIP LLDs might find it useful.
The frame fr_encaps is transferred from the ingress frame to the
exchange by fc_exch_recv_req(), and back to the outgoing frame
by fc_seq_send().
This is taking the last byte in the skb->cb array. If needed,
we could combine the info in sof, eof, flags, and encaps
together into one field, but it'd be better to do that if
and when its needed.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The FIP proposal for VN_port to VN_port point-to-multipoint
operation requires a FLOGI be sent to each remote port.
The FLOGI is sent with the assigned S_ID and D_IDs of the
local and remote ports. This and the response get
FIP-encapsulated for Ethernet.
Add FLOGI state to the remote port state machine.
This will be skipped if not in point-to-multipoint mode.
To reduce a little duplication between PLOGI and FLOGI
response handling, added fc_rport_login_complete(), which
handles the parameters for the rdata struct.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
For VN_port to VN_port mode, the transport sets the port_id and
there's no lport FLOGI. This is similar to FC loop mode.
Add a point_to_multipoint flag that indicates the local port is in
point-to-multipoint mode. This skips FLOGI and discovery.
It also skips resetting the port_id on resets other than link down.
Add function fc_lport_set_local_id() that sets the local port_id.
This is called by libfcoe on behalf of the low-level driver
to set the port_id when the link comes up.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
For VN_port to VN_port mode, FIP will do discovery and needs a
way to find its state from the local port or discovery structure.
It seems that any other LLD that implements its own discovery
would also need something like this.
Replace disc->lport with disc->priv, and use container_of to
find the lport. We could use disc->priv for that, but
container_of is smaller and faster.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Add pre-zeroed space after the allocation for fc_rport_priv
for use by the lower-level driver.
This is primarily for VN2VN FIP mode, but could be used in
other ways someday.
The space required is specified in lport->rport_priv_size.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
To allow LLD to do lookups on rports without grabbing a mutex,
make them RCU-safe. The caller of lport->tt.rport_lookup will
have the choice of holding disc_mutex or the rcu_read_lock().
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Remote ports were restarting indefinitely after getting
rejects in PRLI.
Fix by adding a counter of restarts and limiting that with
the port login retry limit as well.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch somewhat combines two fixes to remote port handing in libfc.
The first problem was that rport work could be queued on a deleted
and freed rport. This is handled by not resetting rdata->event
ton NONE if the rdata is about to be deleted.
However, that fix led to the second problem, described by
Bhanu Gollapudi, as follows:
> Here is the sequence of events. T1 is first LOGO receive thread, T2 is
> fc_rport_work() scheduled by T1 and T3 is second LOGO receive thread and
> T4 is fc_rport_work scheduled by T3.
>
> 1. (T1)Received 1st LOGO in state Ready
> 2. (T1)Delete port & enter to RESTART state.
> 3. (T1)schdule event_work, since event is RPORT_EV_NONE.
> 4. (T1)set event = RPORT_EV_LOGO
> 5. (T1)Enter RESTART state as disc_id is set.
> 6. (T2)remember to PLOGI, and set event = RPORT_EV_NONE
> 6. (T3)Received 2nd LOGO
> 7. (T3)Delete Port & enter to RESTART state.
> 8. (T3)schedule event_work, since event is RPORT_EV_NONE.
> 9. (T3)Enter RESTART state as disc_id is set.
> 9. (T3)set event = RPORT_EV_LOGO
> 10.(T2)work restart, enter PLOGI state and issues PLOGI
> 11.(T4)Since state is not RESTART anymore, restart is not set, and the
> event is not reset to RPORT_EV_NONE. (current event is RPORT_EV_LOGO).
> 12. Now, PLOGI succeeds and fc_rport_enter_ready() will not schedule
> event_work, and hence the rport will never be created, eventually losing
> the target after dev_loss_tmo.
So, the problem here is that we were tracking the desire for
the rport be restarted by state RESTART, which was otherwise
equivalent to DELETE. A contributing factor is that we dropped
the lock between steps 6 and 10 in thread T2, which allows the
state to change, and we didn't completely re-evaluate then.
This is hopefully corrected by the following minor redesign:
Simplify the rport restart logic by making the decision to
restart after deleting the transport rport. That decision
is based on a new STARTED flag that indicates fc_rport_login()
has been called and fc_rport_logoff() has not been called
since then. This replaces the need for the RESTART state.
Only restart if the rdata is still in DELETED state
and only if it still has the STARTED flag set.
Also now, since we clear the event code much later in the
work thread, allow for the possibility that the rport may
have become READY again via incoming PLOGI, and if so,
queue another event to handle that.
In the problem scenario, the second LOGO received will
cause the LOGO event to occur again.
Reported-by: Bhanu Gollapudi <bprakash@broadcom.com>
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch creates a port_id member in struct fc_lport.
This allows libfc to just deal with fc_lport instances
instead of calling into the fc_host to get the port_id.
This change helps in only using symbols necessary for
operation from the libfc structures. libfc still needs
to change the fc_host_port_id() if the port_id changes
so the presentation layer (scsi_transport_fc) can provide
the user with the correct value, but libfc shouldn't
rely on the presentation layer for operational values.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
hton24(p + 3, value) would fail to compile because
p + 3[0] is not a valid expression.
Went ahead and converted hton24 and ntoh24 to inline
functions, which is better because the parameters
are evalutated only once.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
When the kernel is configured for preemption, using smp_processor_id()
when preemption is enabled causes a warning backtrace and is wrong
since we could move off of that CPU as soon as we get the ID,
and we would be referencing the wrong CPU, and possibly an invalid one
if it could be hotswapped out.
Remove the fc_lport_get_stats() function and explicitly use per_cpu_ptr()
to get the statistics. Where preemption has been disabled by holding
a _bh lock continue to use smp_processor_id(), but otherwise use
get_cpu()/put_cpu().
In fcoe_recv_frame() also changed the cases where we return in the
middle to do a goto to the code which bumps ErrorFrames and does
a put_cpu(). Two of these cases didn't bump ErrorFrames before, but
doing so is harmless because they "can't happen", due to prior length
checks.
Also rearranged code in fcoe_recv_frame() to have only one call to
fc_exch_recv(). It's just as efficient and saves a call to put_cpu().
In fc_fcp.c, adjusted a FIXME comment for code which doesn't need fixing.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Add a member function pointer as get_lesb to libfc_function_template so LLD
can fill the LESB based on its own statistics. For fcoe, it fills the LESB
as a fcoe_fc_els_lesb struct according to FC-BB-5.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
FC-BB-5 Rev2.0, Clause 7.10 extends the FC-LS-3 LESB for FC-BB_E. We are
already tracking Link Failure Count so add the rest in this patch.
For VLinkFailureCount and MissDiscAdvCount, they are part of the per-cpu
fcoe_dev_stats. For SymbolErrorCount, ErroredBlockCount, and FCSErrorCount,
they are defined in IEEE 802.3-2008 and are per LLD. They are expected to
come from LLD.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Customers and certification tests have pointed out that we don't
show up on the switch management software as an initiator.
On some MDS switches 'show fcns database' command shows libfc
initiators as 'fcp' not 'fcp:init' like other initiators.
On others switches, I think the switch gets the features by doing a PRLI,
but it may be only certain models or under certain configurations.
Fix this by registering our FC4 features with the RFF_ID CT request
after local port login and after the RFT_ID.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This is to notify the LLD when an FC_ID is assigned to the local port.
The fnic driver needs to push the assigned FC_ID to firmware.
It currently does this by intercepting the FLOGI responses, and
in order to make that code more common with FIP and NPIV, it
makes more sense to wait until the local port has completely
handled the FLOGI or FDISC response. Also, when we fix
point-to-point FC_ID assignment, we'll need this callback as well.
Add a call to the libfc template, which is called whenever
the local port FC_ID is being assigned. It defaults to
fc_lport_set_fid(), supplied by libfc.
As additional benefit of this function, the LLD may determine
the MAC address that caused the change by looking at the received frame.
We also print the assigned port ID as long as it isn't 0.
Setting port ID to 0 happens often in reset while retrying FLOGI,
and would be uninteresting. This replaces the previous message
which didn't identify the host adapter instance.
patch v2 note: changed one word in a comment. "intercepted" -> "provided".
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch makes a variety of cleanup changes to all libfc files.
This patch adds kernel-doc headers to all functions lacking them
and attempts to better format existing headers. It also add kernel-doc
headers to structures.
This patch ensures that the current naming conventions for local ports,
remote ports and remote port private data is upheld in the following
manner.
struct instance (i.e. variable name)
--------------------------------------------------
fc_lport lport
fc_rport rport
fc_rport_libfc_priv rpriv
fc_rport_priv rdata
I also renamed dns_rp and ptp_rp to dns_rdata and ptp_rdata
respectively.
I used emacs 'indent-region' and 'tabify' on all libfc files
to correct spacing alignments.
I feel sorry for anyone attempting to review this patch.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This is the Open-FCoE implementation of the FC
passthrough support via bsg interface.
Passthrough support is added to both N_Ports and
VN_Ports.
Signed-off-by: Steve Ma <steve.ma@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Register the fc_host symbolic name as the symbolic port name
with the fabric name server.
Signed-off-by: Chris Leech <christopher.leech@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Register the fc_host symbolic name as the symbolic node name
with the fabric name server.
Signed-off-by: Chris Leech <christopher.leech@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
One could interpret FC-GS-5 to say that an explicit RNN_ID is required
before RSNN_NN is allowed to succeed, which is why RNN_ID was not obsoleted
along with RPN_ID acording to this document:
ftp://ftp.t11.org/t11/member/fc/gs-5/05-546v2.pdf
Signed-off-by: Chris Leech <christopher.leech@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
RPN_ID has been obsolete per FC-GS-5 for several years. The port name is
registered implicitly as part of FLOGI, and it is undesirable for ports to
change a registered port name using RPN_ID while logged into the fabric.
Signed-off-by: Chris Leech <christopher.leech@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The FIP code in libfcoe needed several changes to support NPIV
1) dst_src_addr needs to be managed per-n_port-ID for FPMA fabrics with NPIV
enabled. Managing the MAC address is now handled in fcoe, with some slight
changes to update_mac() and a new get_src_addr() function pointer.
2) The libfc elsct_send() hook is used to setup FCoE specific response
handlers for FIP encapsulated ELS exchanges. This lets the FCoE specific
handling know which VN_Port the exchange is for, and doesn't require
tracking OX_IDs. It might be possible to roll back to the full FIP frame
in these, but for now I've just stashed the contents of the MAC address
descriptor in the skb context block for later use. Also, because
fcoe_elsct_send() just passes control on to fc_elsct_send(), all transmits
still come through the normal frame_send() path.
3) The NPIV changes added a mutex hold in the keep alive sending, the lport
mutex is protecting the vport list. We can't take a mutex from a timer,
so move the FIP keep alive logic to the link work struct.
Signed-off-by: Chris Leech <christopher.leech@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
NPIV vports are managed in libfc by changing their virtual link state
when the parent N_Ports internal state changes. The vport link is only
online when the N_Port is in a ready state (logged into the fabric).
vport_state is updated as needed in this patch as well, currently the states
LINKDOWN, INITIALIZING, ACTIVE, DSIABLED, and NO_FABRIC_SUPP are used.
This also changes the fc_host port_state handling to differentiate between
LINKDOWN and OFFLINE.
Signed-off-by: Chris Leech <christopher.leech@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adds a function to create a new VN_Port instances, which share the EM
list with the N_Port, VN_Port lookup by fabric ID when responding to a new
request (otherwise the exchange lookup from the N_Ports EM list is trusted to
return an exchange with a cached lport value for the correct VN_Port),
a pointer to a fc_vport structure for VN_Ports, and flags to indicate if an
N_Port supports NPIV and if the switch/fabric allows it.
Signed-off-by: Chris Leech <christopher.leech@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
I'd like to keep basic initialization together with allocation, which means
this can't just be a tail-call to scsi_host_alloc.
This is needed to create a generic libfc host allocation routine for NPIV
VN_Ports, which will share the exchange ID space (through sharing exchange
manager structures) with the parent lport. In order to clone the exchange
manager list when the lport is allocated, the list head must be initialized
earlier.
Also, update fnic to use the libfc_host_alloc so that later changes do not break
it. (contribution by Joe Eykholt)
Signed-off-by: Chris Leech <christopher.leech@intel.com>
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
include/scsi/libfc.h is currently loaded with common code
shared between libfc's sub-modules as well as shared between
libfc and fcoe. Previous patches attempted to move out
non-common code. This patch creates two files for common
libfc routines that will not be shared with fcoe, fnic or
any other LLDs.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This function is never used, let's remove it.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch moves all non-common routines and function prototypes
out of libfc.h and into the appropriate .c files. It makes these
routines 'static' when necessary and removes any unnecessary EXPORT_SYMBOL
statements.
A result of moving the fc_exch_seq_send, fc_seq_els_rsp_send, fc_exch_alloc
and fc_seq_start_next prototypes out of libfc.h is that they were no longer
being imported into fc_exch.c when libfc.h was included. This caused errors
where routines in fc_exch.c were looking for undefined symbols. To fix this
this patch reorganizes fc_seq_alloc, fc_seq_start_next and
fc_seq_start_next_locked. This move also made it so that
fc_seq_start_next_locked did not need to be prototyped at the top of
fc_exch.c.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch modifies scsi_host_template->change_queue_depth so that
it takes an argument indicating why it is being called. This will be
used so that if a LLD needs to do some extra processing when
handling queue fulls or later ramp ups, it can do so.
This is a simple port of the drivers setting a change_queue_depth
callback. In the patch I just have these LLDs adjust the queue depth
if the user was requesting it.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
[Vasu.Dev: v2
Also converted pmcraid_change_queue_depth and then verified
all modules compile using "make allmodconfig" for any new build
warnings on X86_64.
Updated original description after combing two original
patches from Mike to make this patch git bisectable.]
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
[jejb: fixed up 53c700]
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Timer crashes were caused by freeing a struct fc_rport_priv
with a timer pending, causing the timer facility list to be
corrupted. This was during FC uplink flap tests with a lot
of targets.
After discovery, we were doing an PLOGI on an rdata that was
in DELETE state but not yet removed from the lookup list.
This moved the rdata from DELETE state to PLOGI state.
If the PLOGI exchange allocation failed and needed to be
retried, the timer scheduling could race with the free
being done by fc_rport_work().
When fc_rport_login() is called on a rport in DELETE state,
move it to a new state RESTART. In fc_rport_work, when
handling a LOGO, STOPPED or FAILED event, look for restart
state. In the RESTART case, don't take the rdata off the
list and after the transport remote port is deleted and
exchanges are reset, re-login to the remote port.
Note that the new RESTART state also corrects a problem we
had when re-discovering a port that had moved to DELETE state.
In that case, a new rdata was created, but the old rdata
would do an exchange manager reset affecting the FC_ID
for both the new rdata and old rdata. With the new state,
the new port isn't logged into until after any old exchanges
are reset.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Reported-by: Alex Lyakas <alexl@mellanox.co.il>
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
When rport_login is called on an rport that is already thought
to be logged in, use ADISC. If that fails, redo PLOGI.
This is less disruptive after fabric changes that don't affect
the state of the target.
Implement the sending of ADISC via fc_els_fill.
Add ADISC state to the rport state machine. This is entered from READY
and returns to READY after successful completion. If it fails, the rport
is either logged off and deleted or re-does PLOGI.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Improve lport and rport debug messages to indicate whether
the response is LS_ACC, LS_RJT, closed, or timeout.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This moves the remote port lookup for incoming ELS requests into
fc_rport.c, in preparation for handing PLOGI and LOGO from
unknown rports.
This changes the arg to rport_recv_req from an rdata to an lport.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Currently these values are initialized by the callers. This was exposed
by a later patch that adds PLOGI request support. The patch failed to
initialize the new remote port's roles and it caused problems. This patch
has the rport_create routine initialize the identifiers and then the
callers can override them with real values.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
When receiving an RSCN, do not log off all rports. This is
extremely disruptive. If, after the GPN_FT response, some
rports haven't been listed, delete them.
Add field disc_id to structs fc_rport_priv and fc_disc.
disc_id is an arbitrary serial number used to identify the
rports found by the latest discovery. This eliminates the need
to go through the rport list when restarting discovery.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Delete unused disc->delay element.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
There was no need to have the discovery status stored in struct fc_disc.
Change fc_disc_done() to take the discovery status as an argument
and just pass it on to the discovery callback.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Don't create a "dummy" remote port to go with fc_rport_priv.
Make the rport truly optional by allocating fc_rport_priv separately
and not requiring a dummy rport to be there if we haven't yet done
fc_remote_port_add().
The fc_rport_libfc_priv remains as a structure attached to the
rport for I/O purposes.
Be sure to hold references on rdata when the lock is dropped in
fc_rport_work().
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Remote ports will become READY more than once after
ADISC is implemented in a later patch.
The event callback that has been called "CREATED" will mean "READY".
Rename it now in preparation for those changes.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Allow a struct fc_rport_priv to have no fc_rport associated with it.
This sets up to remove the need for "rogue" rports.
Add a few fields to fc_rport_priv that are needed before the fc_rport
is created. These are the ids, maxframe_size, classes, and rport pointer.
Remove the macro PRIV_TO_RPORT(). Just use rdata->rport where appropriate.
To take the place of the get_device()/put_device ops that were used to
hold both the rport and rdata, add a reference count to rdata structures
using kref. When kref_get decrements the refcount to zero, a new template
function releasing the rdata should be called. This will take care of
freeing the rdata and releasing the hold on the rport (for now). After
subsequent patches make the rport truly optional, this release function
will simply free the rdata.
Remove the simple inline function fc_rport_set_name(), which becomes
semanticly ambiguous otherwise. The caller will set the port_name and
node_name in the rdata->Ids, which will later be copied to the rport
when it its created.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
tt.elsct_send is used by both FCP and by the rport state machine.
After further patches, these two modules will use different
structures for the remote port.
So, change elsct_send to use the FC_ID instead of the fc_rport_priv
as its argument. It currently only uses the FC_ID anyway.
For CT requests the destination FC_ID is still implicitly 0xfffffc.
After further patches the did arg on CT requests will be used to
specify the FC_ID being inquired about for GPN_ID or other queries.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The rport and discovery modules deal with remote ports
before fc_remote_port_add() can be done, because the
full set of rport identifiers is not known at early stages.
In preparation for splitting the fc_rport/fc_rport_priv allocation,
make fc_rport_priv the primary interface for the remote port and
discovery engines.
The FCP / SCSI layers still deal with fc_rport and
fc_rport_libfc_priv, however.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
These macros introduce extra undesirable semicolons that keep
them from being used in expressions, and they don't protect
against being passed an expression.
Add parens and remove the semicolons.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The interface for lport->tt.rport_create() takes a fc_disc_port arg,
which is unnatural for most calls. The only reason for this was
to avoid passing in the local port as an argument, but otherwise
added to complexity.
Simplify by just using lport and fc_rport_identifiers.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
While the I/O and LLD interfaces use fc_rport_libfc_priv, the
disc and rport interfaces will use fc_rport_priv, which will
be separately allocated.
Change the disc and rport usage of fc_rport_libfc_priv to fc_rport_priv.
Use #define temporarily to make both names equivalent until a
subsequent patch splits them.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
1. Updates fcoe_rcv() to queue incoming frames to the fcoe per
cpu thread on which this frame's exch was originated and simply
use current cpu for request exch not originated by initiator.
It is redundant to add this code under CONFIG_SMP, so removes
CONFIG_SMP uses around this code.
2. Updates fc_exch_em_alloc, fc_exch_delete, fc_exch_find to use
per cpu exch pools, here fc_exch_delete is rename of older
fc_exch_mgr_delete_ep since ep/exch are now deleted in pools
of EM and so brief new name is sufficient and better name.
Updates these functions to map exch id to their index into exch
pool using fc_cpu_mask, fc_cpu_order and EM min_xid.
This mapping is as per detailed explanation about this in
last patch and basically this is just as lower fc_cpu_mask
bits of exch id as cpu number and upper bit sum of EM min_xid
and exch index in pool.
Uses pool next_index to keep track of exch allocation from
pool along with pool_max_index as upper bound of exches array
in pool.
3. Adds exch pool ptr to fc_exch to free exch to its pool in
fc_exch_delete.
4. Updates fc_exch_mgr_reset to reset all exch pools of an EM,
this required adding fc_exch_pool_reset func to reset exches
in pool and then have fc_exch_mgr_reset call fc_exch_pool_reset
for each pool within each EM for a lport.
5. Removes no longer needed exches array, em_lock, next_xid, and
total_exches from struct fc_exch_mgr, these are not needed after
use of per cpu exch pool, also removes not used max_read,
last_read from struct fc_exch_mgr.
6. Updates locking notes for exch pool lock with fc_exch lock and
uses pool lock in exch allocation, lookup and reset.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adds per cpu exch pool for these reasons:-
1. Currently an EM instance is shared across all cpus to manage
all exches for all cpus. This required em_lock across all
cpus for an exch alloc, free, lookup and reset each frame
and that made em_lock expensive, so instead having per cpu
exch pool with their own per cpu pool lock will likely reduce
locking contention in fast path for an exch alloc, free and
lookup.
2. Per cpu exch pool will likely improve cache hit ratio since
all frames of an exch will be processed on the same cpu on
which exch originated.
This patch is only prep work to help in keeping complexity of next
patch low, so this patch only sets up per cpu exch pool and related
helper funcs to be used by next patch. The next patch fully makes
use of per cpu exch pool in all code paths ie. tx, rx and reset.
Divides per EM exch id range equally across all cpus to setup per
cpu exch pool. This division is such that lower bits of exch id
carries cpu number info on which exch originated, later a simple
bitwise AND operation on exch id of incoming frame with fc_cpu_mask
retrieves cpu number info to direct all frames to same cpu on which
exch originated. This required a global fc_cpu_mask and fc_cpu_order
initialized to max possible cpus number nr_cpu_ids rounded up to 2's
power, this will be used in mapping exch id and exch ptr array
index in pool during exch allocation, find or reset code paths.
Adds a check in fc_exch_mgr_alloc() to ensure specified min_xid
lower bits are zero since these bits are used to carry cpu info.
Adds and initializes struct fc_exch_pool with all required fields
to manage exches in pool.
Allocates per cpu struct fc_exch_pool with memory for exches array
for range of exches per pool. The exches array memory is followed
by struct fc_exch_pool.
Adds fc_exch_ptr_get/set() helper functions to get/set exch ptr in
pool exches array at specified array index.
Increases default FCOE_MAX_XID to 0x0FFF from 0x07EF, so that more
exches are available per cpu after above described exch id range
division across all cpus to each pool.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Modifies current code to use EM anchor list in EM allocation, EM free,
EM reset, exch allocation and exch lookup code paths.
1. Modifies fc_exch_mgr_alloc to accept EM match function and then
have allocated EM added to the lport using fc_exch_mgr_add API
while also updating EM kref for newly added EM.
2. Updates fc_exch_mgr_free API to accept only lport pointer instead
EM and then have this API free all EMs of the lport from EM anchor
list.
3. Removes single lport pointer link from the EM, which was used in
associating lport pointer in newly allocated exchange. Instead have
lport pointer passed along new exchange allocation call path and
then store passed lport pointer in newly allocated exchange, this
will allow a single EM instance to be used across more than one
lport and used in EM reset to reset only lport specific exchanges.
4. Modifies fc_exch_mgr_reset to reset all EMs from the EM anchor list
of the lport, adds additional exch lport pointer (ep->lp) check for
shared EM case to reset exchange specific to a lport requested reset.
5. Updates exch allocation API fc_exch_alloc to use EM anchor list and
its anchor match func pointer. The fc_exch_alloc will walk the list
of EMs until it finds a match, a match will be either null match
func pointer or call to match function returning true value.
6. Updates fc_exch_recv to accept incoming frame on local port using
only lport pointer and frame pointer without specifying EM instance
of incoming frame. Instead modified fc_exch_recv to locate EM for the
incoming frame by matching xid of incoming frame against a EM xid range.
This change was required to use EM list in libfc Rx path and after this
change the lport fc_exch_mgr pointer emp is not needed anymore, so
removed emp pointer.
7. Updates fnic for removed lport emp pointer and above modified libfc APIs
fc_exch_recv, fc_exch_mgr_alloc and fc_exch_mgr_free.
8. Removes exch_get and exch_put from libfc_function_template as these
are no longer needed with EM anchor list and its match function use.
Also removes its default function fc_exch_get.
A defect this patch introduced regarding the libfc initialization order in
the fnic driver was fixed by Joe Eykholt <jeykholt@cisco.com>.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Currently there is a 1:1 relationship between the lport
and exchange manager. This macro takes an EM as an argument
and determines the lport from it. However, later patches
will use an EM list per lport, so we will no longer have
this 1:1 relationship- this macro must change.
The FC_EM_DBG macro is rarely used. There are four callers,
two can use FC_LPORT_DBG instead and two can be removed
since they're not necessary. This patch makes those changes
and removes the macro.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adds EM list using a anchor struct fc_exch_mgr_anchor, anchor is used
to allow same EM instance sharing across more than one lport on a eth
device, this implementation is per discussed design posted at
http://www.open-fcoe.org/pipermail/devel/2009-June/002566.html.
The shared EM is required for multiple lports on eth device when
using multiple VLANs or NPIV.
Adds fc_exch_mgr_add API to add a EM to the lport and fc_exch_mgr_del
API to delete previously added EM.
Also adds function fc_exch_mgr_destroy() to destroy allocated EM.
The kref is added to the EM to keep track of EM usage count, the EM is
destroyed when no longer in use upon kref reaching to zero.
The caller can specify match function to fc_exch_mgr_add, this
will be used in determining exchange allocation from its EM or not.
Moved calling of fcoe_em_config below fcoe_libfc_config calling,
so that list head lp->ema_list is initialized before configuring
EM.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
State RPORT_ST_NONE was intented to be an invalid state (0), never used.
This was a misguided attempt to be sure it was always initialized.
Having an extra state meaning nothing requires switch statements to
have a case covering that state.
State NONE has been used instead to mean the remote port is being deleted.
Changing the name to RPORT_ST_DELETE.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The state NONE was meant to be invalid, but has been used as
the initial state. Rename it to be DISABLED, as more descriptive.
Further patches will make it the like the RESET state, except
it won't transition to FLOGI until fc_lport_fabric_login() is called.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
libfc debug messages currently show 'lport: <fc-id>:'
wher <fc-id> is the hex assigned port-id. When the lport
is logged off, that will be zero, so its hard to distinguish
which instance is involved. The FC-ID can change
if the port is re-patched or changes VSANs.
Two lports may even have the same FC-ID if connected to isolated SANs.
Change the debug messages to print the SCSI host number "hostN:",
which will not change for the life of the lport.
Still show the FC_ID on lport messages.
Also, add a macro to FC_RPORT_ID_DBG for rport debugging where there's
no rdata structure involved. It takes the lport and port_id as parameters.
Use this in fc_rport_recv_plogi_req() and fc_rport_recv_logo_req().
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This is unlikely to cause any problems, but the libfc debug macros
introduce extra undesirable semicolons.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch adds the /sys/module/libfc/parameters/debug_logging
file to sysfs as a module parameter. It accepts an integer
bitmask for logging. Currently it supports:
bit
LSB 0 = general libfc debugging
1 = lport debugging
2 = disc debugging
3 = rport debugging
4 = fcp debugging
5 = EM debugging
6 = exch/seq debugging
7 = scsi logging (mostly error handling)
the other bits are not used at this time.
The patch converts all of the libfc source files to use
these new macros and removes the old FC_DBG macro.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This allows fnic to configure number of retries for lport and rport
separately.
Signed-off-by: Abhijeet Joglekar <abjoglek@cisco.com>
Acked-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Rogue ports are currently not tracked on any list. The only reference
to them is through any outstanding exchanges pending on the rogue ports.
If the module is removed while a retry is set on a rogue port
(say a Plogi retry for instance), this retry is not cancelled because there
is no reference to the rogue port in the discovery rports list. Thus the
local port can clean itself up, delete the exchange pool, and then the
rogue port timeout can fire and try to start up another exchange.
This patch tracks the rogue ports in a new list disc->rogue_rports. Creating
a new list instead of using the disc->rports list keeps remote port code
change to a minimum.
1) Whenever a rogue port is created, it is immediately added to the
disc->rogue_rports list.
2) When the rogues port goes to ready, it is removed from the rogue list
and the real remote port is added to the disc->rports list
3) The removal of the rogue from the disc->rogue_rports list is done in
the context of the fc_rport_work() workQ thread in discovery callback.
4) Real rports are removed from the disc->rports list like before. Lookup
is done only in the real rports list. This avoids making large changes
to the remote port code.
5) In fc_disc_stop_rports, the rogues list is traversed in addition to the
real list to stop the rogue ports and issue logoffs on them. This way, rogue
ports get cleaned up when the local port goes away.
6) rogue remote ports are not removed from the list right away, but
removed late in fc_rport_work() context, multiple threads can find the same
remote port in the list and call rport_logoff(). Rport_logoff() only
continues with the logoff if port is not in NONE state, thus preventing
multiple logoffs and multiple list deletions.
7) Since the rport is removed from the disc list at a later stage
(in the disc callback), incoming frames can find the rport even if
rport_logoff() has been called on the rport. When rport_logoff() is called,
the rport state is set to NONE, and we are trying to cancel all exchanges
and retries on that port. While in this state, if an incoming
Plogi/Prli/Logo or other frames match the rport, we should not reply
because the rport is in the NONE state. Just drop the frame, since the
rport will be deleted soon in the disc callback (fc_rport_work)
8) In fc_disc_single(), remove rport lookup and call to fc_disc_del_target.
fc_disc_single() is called from recv_rscn_req() where rport lookup
and rport_logoff is already done.
Signed-off-by: Abhijeet Joglekar <abjoglek@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The foce_softc mem was reserved by libfc_host_alloc as well as
by fcoe_host_alloc.
Removes one liner fcoe_host_alloc completely, instead directly calls
libfc_host_alloc to alloc scsi_host with libfc for just one fcoe_softc
as fcoe private data.
Moves libfc_host_alloc to libfc.h since it is a libfc API, placed
lport_priv API adjacent to libfc_host_alloc since this is related
to scsi_host priv data.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Remove the hotplug creation of dev_stats, we allocate for all possible CPUs
now when we allocate the lport.
v2: Durring the 2.6.30 merge window, before these patches were comitted,
'percpu_ptr' was renamed 'per_cpu_ptr'. This latest update updates this
patch for the name change.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When LLD supports direct data placement (ddp) for large receive of an scsi
i/o coming into fc_fcp, we call into libfc_function_template's ddp_setup()
to prepare for a ddp of large receive for this read I/O. When I/O is complete,
we call the corresponding ddp_done() to get the length of data ddped as well
as to let LLD do clean up.
fc_fcp_ddp_setup()/fc_fcp_ddp_done() are added to setup and complete a ddped
read I/O described by the given fc_fcp_pkt. They would call into corresponding
ddp_setup/ddp_done implemented by the fcoe layer. Eventually, fcoe layer calls
into LLD's ddp_setup/ddp_done provided through net_device
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This checks if net_devices supports FCoE offload ops in netdev_ops and it
if it does, then sets up the corresponding flags in the associated fc_lport.
For large send offload, the maximum length supported in one large send is now
described by the added lso_max in fc_lport, which is setup initially from
netdev->gso_max_size.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
1) Added '()' for function names in kerneldoc comments
2) Changed comment bookends from '**/' to '*/'. The comment on the the
mailing list was that '**/' "is consistently unconventional. Not
wrong, just odd." The Documentation/kernel-doc-nano-HOWTO.txt
states that kerneldoc comment blocks should end with '**/' but most
(if not all) instance I found under drivers/scsi/ were only using
the '*/' so I converted to that style.
3) Removed incorrect linebreaks in kerneldoc comments where found
4) Removed a few unnecessary blank comment lines in kerneldoc comment
blocks
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Made the comments more like the comments for struct scsi_host_template.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The fcoe_xmit could call fc_pause in case the pending skb queue len is larger
than FCOE_MAX_QUEUE_DEPTH, the fc_pause was trying to grab lport->lp_muex to
change lport->link_status and that had these issues :-
1. The fcoe_xmit was getting called with bh disabled, thus causing
"BUG: scheduling while atomic" when grabbing lport->lp_muex with bh disabled.
2. fc_linkup and fc_linkdown function calls lport_enter function with
lport->lp_mutex held and these enter function in turn calls fcoe_xmit to send
lport related FC frame, e.g. fc_linkup => fc_lport_enter_flogi to send flogi
req. In this case grabbing the same lport->lp_mutex again in fc_puase from
fcoe_xmit would cause deadlock.
The lport->lp_mutex was used for setting FC_PAUSE in fcoe_xmit path but
FC_PAUSE bit was not used anywhere beside just setting and clear this
bit in lport->link_status, instead used a separate field qfull in fc_lport
to eliminate need for lport->lp_mutex to track pending queue full condition
and in turn avoid above described two locking issues.
Also added check for lp->qfull in fc_fcp_lport_queue_ready to trigger
SCSI_MLQUEUE_HOST_BUSY when lp->qfull is set to prevent more scsi-ml cmds
while lp->qfull is set.
This patch eliminated FC_LINK_UP and FC_PAUSE and instead used dedicated
fields in fc_lport for this, this simplified all related conditional
code.
Also removed fc_pause and fc_unpause functions and instead used newly added
lport->qfull directly in fcoe.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
fc_exch_mgr structure is private to fc_exch.c. To export exch_mgr_reset to
transport, transport needs access to the exch manager. Change
exch_mgr_reset to use lport param which is the shared structure between
libFC and transport.
Alternatively, fc_exch_mgr definition can be moved to libfc.h so that lport
can be accessed from mp*.
Signed-off-by: Abhijeet Joglekar <abjoglek@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
libFC is composed of 4 blocks supported by an exchange manager
and a framing library. The upper 4 layers are fc_lport, fc_disc,
fc_rport and fc_fcp. A LLD that uses libfc could choose to
either use libfc's block, or using the transport template
defined in libfc.h, override one or more blocks with its own
implementation.
The EM (Exchange Manager) manages exhcanges/sequences for all
commands- ELS, CT and FCP.
The framing library frames ELS and CT commands.
The fc_lport block manages the library's representation of the
host's FC enabled ports.
The fc_disc block manages discovery of targets as well as
handling changes that occur in the FC fabric (via. RSCN events).
The fc_rport block manages the library's representation of other
entities in the FC fabric. Currently the library uses this block
for targets, its peer when in point-to-point mode and the
directory server, but can be extended for other entities if
needed.
The fc_fcp block interacts with the scsi-ml and handles all
I/O.
Signed-off-by: Robert Love <robert.w.love@intel.com>
[jejb: added include of delay.h to fix ppc64 compile prob spotted by sfr]
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Joe Eykholt