2014-07-25 00:04:10 +08:00
|
|
|
/*
|
|
|
|
* Copyright © 2014 Intel Corporation
|
|
|
|
*
|
|
|
|
* Permission is hereby granted, free of charge, to any person obtaining a
|
|
|
|
* copy of this software and associated documentation files (the "Software"),
|
|
|
|
* to deal in the Software without restriction, including without limitation
|
|
|
|
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
|
|
|
|
* and/or sell copies of the Software, and to permit persons to whom the
|
|
|
|
* Software is furnished to do so, subject to the following conditions:
|
|
|
|
*
|
|
|
|
* The above copyright notice and this permission notice (including the next
|
|
|
|
* paragraph) shall be included in all copies or substantial portions of the
|
|
|
|
* Software.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
|
|
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
|
|
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
|
|
|
|
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
|
|
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
|
|
|
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
|
|
|
|
* IN THE SOFTWARE.
|
|
|
|
*
|
|
|
|
* Authors:
|
|
|
|
* Ben Widawsky <ben@bwidawsk.net>
|
|
|
|
* Michel Thierry <michel.thierry@intel.com>
|
|
|
|
* Thomas Daniel <thomas.daniel@intel.com>
|
|
|
|
* Oscar Mateo <oscar.mateo@intel.com>
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
2014-07-25 00:04:48 +08:00
|
|
|
/**
|
|
|
|
* DOC: Logical Rings, Logical Ring Contexts and Execlists
|
|
|
|
*
|
|
|
|
* Motivation:
|
2014-07-25 00:04:10 +08:00
|
|
|
* GEN8 brings an expansion of the HW contexts: "Logical Ring Contexts".
|
|
|
|
* These expanded contexts enable a number of new abilities, especially
|
|
|
|
* "Execlists" (also implemented in this file).
|
|
|
|
*
|
2014-07-25 00:04:48 +08:00
|
|
|
* One of the main differences with the legacy HW contexts is that logical
|
|
|
|
* ring contexts incorporate many more things to the context's state, like
|
|
|
|
* PDPs or ringbuffer control registers:
|
|
|
|
*
|
|
|
|
* The reason why PDPs are included in the context is straightforward: as
|
|
|
|
* PPGTTs (per-process GTTs) are actually per-context, having the PDPs
|
|
|
|
* contained there mean you don't need to do a ppgtt->switch_mm yourself,
|
|
|
|
* instead, the GPU will do it for you on the context switch.
|
|
|
|
*
|
|
|
|
* But, what about the ringbuffer control registers (head, tail, etc..)?
|
|
|
|
* shouldn't we just need a set of those per engine command streamer? This is
|
|
|
|
* where the name "Logical Rings" starts to make sense: by virtualizing the
|
|
|
|
* rings, the engine cs shifts to a new "ring buffer" with every context
|
|
|
|
* switch. When you want to submit a workload to the GPU you: A) choose your
|
|
|
|
* context, B) find its appropriate virtualized ring, C) write commands to it
|
|
|
|
* and then, finally, D) tell the GPU to switch to that context.
|
|
|
|
*
|
|
|
|
* Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switch
|
|
|
|
* to a contexts is via a context execution list, ergo "Execlists".
|
|
|
|
*
|
|
|
|
* LRC implementation:
|
|
|
|
* Regarding the creation of contexts, we have:
|
|
|
|
*
|
|
|
|
* - One global default context.
|
|
|
|
* - One local default context for each opened fd.
|
|
|
|
* - One local extra context for each context create ioctl call.
|
|
|
|
*
|
|
|
|
* Now that ringbuffers belong per-context (and not per-engine, like before)
|
|
|
|
* and that contexts are uniquely tied to a given engine (and not reusable,
|
|
|
|
* like before) we need:
|
|
|
|
*
|
|
|
|
* - One ringbuffer per-engine inside each context.
|
|
|
|
* - One backing object per-engine inside each context.
|
|
|
|
*
|
|
|
|
* The global default context starts its life with these new objects fully
|
|
|
|
* allocated and populated. The local default context for each opened fd is
|
|
|
|
* more complex, because we don't know at creation time which engine is going
|
|
|
|
* to use them. To handle this, we have implemented a deferred creation of LR
|
|
|
|
* contexts:
|
|
|
|
*
|
|
|
|
* The local context starts its life as a hollow or blank holder, that only
|
|
|
|
* gets populated for a given engine once we receive an execbuffer. If later
|
|
|
|
* on we receive another execbuffer ioctl for the same context but a different
|
|
|
|
* engine, we allocate/populate a new ringbuffer and context backing object and
|
|
|
|
* so on.
|
|
|
|
*
|
|
|
|
* Finally, regarding local contexts created using the ioctl call: as they are
|
|
|
|
* only allowed with the render ring, we can allocate & populate them right
|
|
|
|
* away (no need to defer anything, at least for now).
|
|
|
|
*
|
|
|
|
* Execlists implementation:
|
2014-07-25 00:04:10 +08:00
|
|
|
* Execlists are the new method by which, on gen8+ hardware, workloads are
|
|
|
|
* submitted for execution (as opposed to the legacy, ringbuffer-based, method).
|
2014-07-25 00:04:48 +08:00
|
|
|
* This method works as follows:
|
|
|
|
*
|
|
|
|
* When a request is committed, its commands (the BB start and any leading or
|
|
|
|
* trailing commands, like the seqno breadcrumbs) are placed in the ringbuffer
|
|
|
|
* for the appropriate context. The tail pointer in the hardware context is not
|
|
|
|
* updated at this time, but instead, kept by the driver in the ringbuffer
|
|
|
|
* structure. A structure representing this request is added to a request queue
|
|
|
|
* for the appropriate engine: this structure contains a copy of the context's
|
|
|
|
* tail after the request was written to the ring buffer and a pointer to the
|
|
|
|
* context itself.
|
|
|
|
*
|
|
|
|
* If the engine's request queue was empty before the request was added, the
|
|
|
|
* queue is processed immediately. Otherwise the queue will be processed during
|
|
|
|
* a context switch interrupt. In any case, elements on the queue will get sent
|
|
|
|
* (in pairs) to the GPU's ExecLists Submit Port (ELSP, for short) with a
|
|
|
|
* globally unique 20-bits submission ID.
|
|
|
|
*
|
|
|
|
* When execution of a request completes, the GPU updates the context status
|
|
|
|
* buffer with a context complete event and generates a context switch interrupt.
|
|
|
|
* During the interrupt handling, the driver examines the events in the buffer:
|
|
|
|
* for each context complete event, if the announced ID matches that on the head
|
|
|
|
* of the request queue, then that request is retired and removed from the queue.
|
|
|
|
*
|
|
|
|
* After processing, if any requests were retired and the queue is not empty
|
|
|
|
* then a new execution list can be submitted. The two requests at the front of
|
|
|
|
* the queue are next to be submitted but since a context may not occur twice in
|
|
|
|
* an execution list, if subsequent requests have the same ID as the first then
|
|
|
|
* the two requests must be combined. This is done simply by discarding requests
|
|
|
|
* at the head of the queue until either only one requests is left (in which case
|
|
|
|
* we use a NULL second context) or the first two requests have unique IDs.
|
|
|
|
*
|
|
|
|
* By always executing the first two requests in the queue the driver ensures
|
|
|
|
* that the GPU is kept as busy as possible. In the case where a single context
|
|
|
|
* completes but a second context is still executing, the request for this second
|
|
|
|
* context will be at the head of the queue when we remove the first one. This
|
|
|
|
* request will then be resubmitted along with a new request for a different context,
|
|
|
|
* which will cause the hardware to continue executing the second request and queue
|
|
|
|
* the new request (the GPU detects the condition of a context getting preempted
|
|
|
|
* with the same context and optimizes the context switch flow by not doing
|
|
|
|
* preemption, but just sampling the new tail pointer).
|
|
|
|
*
|
2014-07-25 00:04:10 +08:00
|
|
|
*/
|
drm/i915: Move execlists irq handler to a bottom half
Doing a lot of work in the interrupt handler introduces huge
latencies to the system as a whole.
Most dramatic effect can be seen by running an all engine
stress test like igt/gem_exec_nop/all where, when the kernel
config is lean enough, the whole system can be brought into
multi-second periods of complete non-interactivty. That can
look for example like this:
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:3:143]
Modules linked in: [redacted for brevity]
CPU: 0 PID: 143 Comm: kworker/u8:3 Tainted: G U L 4.5.0-160321+ #183
Hardware name: Intel Corporation Broadwell Client platform/WhiteTip Mountain 1
Workqueue: i915 gen6_pm_rps_work [i915]
task: ffff8800aae88000 ti: ffff8800aae90000 task.ti: ffff8800aae90000
RIP: 0010:[<ffffffff8104a3c2>] [<ffffffff8104a3c2>] __do_softirq+0x72/0x1d0
RSP: 0000:ffff88014f403f38 EFLAGS: 00000206
RAX: ffff8800aae94000 RBX: 0000000000000000 RCX: 00000000000006e0
RDX: 0000000000000020 RSI: 0000000004208060 RDI: 0000000000215d80
RBP: ffff88014f403f80 R08: 0000000b1b42c180 R09: 0000000000000022
R10: 0000000000000004 R11: 00000000ffffffff R12: 000000000000a030
R13: 0000000000000082 R14: ffff8800aa4d0080 R15: 0000000000000082
FS: 0000000000000000(0000) GS:ffff88014f400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa53b90c000 CR3: 0000000001a0a000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
042080601b33869f ffff8800aae94000 00000000fffc2678 ffff88010000000a
0000000000000000 000000000000a030 0000000000005302 ffff8800aa4d0080
0000000000000206 ffff88014f403f90 ffffffff8104a716 ffff88014f403fa8
Call Trace:
<IRQ>
[<ffffffff8104a716>] irq_exit+0x86/0x90
[<ffffffff81031e7d>] smp_apic_timer_interrupt+0x3d/0x50
[<ffffffff814f3eac>] apic_timer_interrupt+0x7c/0x90
<EOI>
[<ffffffffa01c5b40>] ? gen8_write64+0x1a0/0x1a0 [i915]
[<ffffffff814f2b39>] ? _raw_spin_unlock_irqrestore+0x9/0x20
[<ffffffffa01c5c44>] gen8_write32+0x104/0x1a0 [i915]
[<ffffffff8132c6a2>] ? n_tty_receive_buf_common+0x372/0xae0
[<ffffffffa017cc9e>] gen6_set_rps_thresholds+0x1be/0x330 [i915]
[<ffffffffa017eaf0>] gen6_set_rps+0x70/0x200 [i915]
[<ffffffffa0185375>] intel_set_rps+0x25/0x30 [i915]
[<ffffffffa01768fd>] gen6_pm_rps_work+0x10d/0x2e0 [i915]
[<ffffffff81063852>] ? finish_task_switch+0x72/0x1c0
[<ffffffff8105ab29>] process_one_work+0x139/0x350
[<ffffffff8105b186>] worker_thread+0x126/0x490
[<ffffffff8105b060>] ? rescuer_thread+0x320/0x320
[<ffffffff8105fa64>] kthread+0xc4/0xe0
[<ffffffff8105f9a0>] ? kthread_create_on_node+0x170/0x170
[<ffffffff814f351f>] ret_from_fork+0x3f/0x70
[<ffffffff8105f9a0>] ? kthread_create_on_node+0x170/0x170
I could not explain, or find a code path, which would explain
a +20 second lockup, but from some instrumentation it was
apparent the interrupts off proportion of time was between
10-25% under heavy load which is quite bad.
When a interrupt "cliff" is reached, which was >~320k irq/s on
my machine, the whole system goes into a terrible state of the
above described multi-second lockups.
By moving the GT interrupt handling to a tasklet in a most
simple way, the problem above disappears completely.
Testing the effect on sytem-wide latencies using
igt/gem_syslatency shows the following before this patch:
gem_syslatency: cycles=1532739, latency mean=416531.829us max=2499237us
gem_syslatency: cycles=1839434, latency mean=1458099.157us max=4998944us
gem_syslatency: cycles=1432570, latency mean=2688.451us max=1201185us
gem_syslatency: cycles=1533543, latency mean=416520.499us max=2498886us
This shows that the unrelated process is experiencing huge
delays in its wake-up latency. After the patch the results
look like this:
gem_syslatency: cycles=808907, latency mean=53.133us max=1640us
gem_syslatency: cycles=862154, latency mean=62.778us max=2117us
gem_syslatency: cycles=856039, latency mean=58.079us max=2123us
gem_syslatency: cycles=841683, latency mean=56.914us max=1667us
Showing a huge improvement in the unrelated process wake-up
latency. It also shows an approximate halving in the number
of total empty batches submitted during the test. This may
not be worrying since the test puts the driver under
a very unrealistic load with ncpu threads doing empty batch
submission to all GPU engines each.
Another benefit compared to the hard-irq handling is that now
work on all engines can be dispatched in parallel since we can
have up to number of CPUs active tasklets. (While previously
a single hard-irq would serially dispatch on one engine after
another.)
More interesting scenario with regards to throughput is
"gem_latency -n 100" which shows 25% better throughput and
CPU usage, and 14% better dispatch latencies.
I did not find any gains or regressions with Synmark2 or
GLbench under light testing. More benchmarking is certainly
required.
v2:
* execlists_lock should be taken as spin_lock_bh when
queuing work from userspace now. (Chris Wilson)
* uncore.lock must be taken with spin_lock_irq when
submitting requests since that now runs from either
softirq or process context.
v3:
* Expanded commit message with more testing data;
* converted missed locking sites to _bh;
* added execlist_lock comment. (Chris Wilson)
v4:
* Mention dispatch parallelism in commit. (Chris Wilson)
* Do not hold uncore.lock over MMIO reads since the block
is already serialised per-engine via the tasklet itself.
(Chris Wilson)
* intel_lrc_irq_handler should be static. (Chris Wilson)
* Cancel/sync the tasklet on GPU reset. (Chris Wilson)
* Document and WARN that tasklet cannot be active/pending
on engine cleanup. (Chris Wilson/Imre Deak)
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Imre Deak <imre.deak@intel.com>
Testcase: igt/gem_exec_nop/all
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=94350
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/1459768316-6670-1-git-send-email-tvrtko.ursulin@linux.intel.com
2016-04-04 19:11:56 +08:00
|
|
|
#include <linux/interrupt.h>
|
2014-07-25 00:04:10 +08:00
|
|
|
|
|
|
|
#include <drm/drmP.h>
|
|
|
|
#include <drm/i915_drm.h>
|
|
|
|
#include "i915_drv.h"
|
2017-11-10 22:26:34 +08:00
|
|
|
#include "i915_gem_render_state.h"
|
2018-06-29 04:12:07 +08:00
|
|
|
#include "i915_vgpu.h"
|
2018-01-24 08:43:49 +08:00
|
|
|
#include "intel_lrc_reg.h"
|
2015-07-11 01:13:11 +08:00
|
|
|
#include "intel_mocs.h"
|
2018-04-11 00:12:46 +08:00
|
|
|
#include "intel_workarounds.h"
|
2014-07-25 00:04:11 +08:00
|
|
|
|
drm/i915/bdw: Handle context switch events
Handle all context status events in the context status buffer on every
context switch interrupt. We only remove work from the execlist queue
after a context status buffer reports that it has completed and we only
attempt to schedule new contexts on interrupt when a previously submitted
context completes (unless no contexts are queued, which means the GPU is
free).
We canot call intel_runtime_pm_get() in an interrupt (or with a spinlock
grabbed, FWIW), because it might sleep, which is not a nice thing to do.
Instead, do the runtime_pm get/put together with the create/destroy request,
and handle the forcewake get/put directly.
Signed-off-by: Thomas Daniel <thomas.daniel@intel.com>
v2: Unreferencing the context when we are freeing the request might free
the backing bo, which requires the struct_mutex to be grabbed, so defer
unreferencing and freeing to a bottom half.
v3:
- Ack the interrupt inmediately, before trying to handle it (fix for
missing interrupts by Bob Beckett <robert.beckett@intel.com>).
- Update the Context Status Buffer Read Pointer, just in case (spotted
by Damien Lespiau).
v4: New namespace and multiple rebase changes.
v5: Squash with "drm/i915/bdw: Do not call intel_runtime_pm_get() in an
interrupt", as suggested by Daniel.
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
[danvet: Checkpatch ...]
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:39 +08:00
|
|
|
#define RING_EXECLIST_QFULL (1 << 0x2)
|
|
|
|
#define RING_EXECLIST1_VALID (1 << 0x3)
|
|
|
|
#define RING_EXECLIST0_VALID (1 << 0x4)
|
|
|
|
#define RING_EXECLIST_ACTIVE_STATUS (3 << 0xE)
|
|
|
|
#define RING_EXECLIST1_ACTIVE (1 << 0x11)
|
|
|
|
#define RING_EXECLIST0_ACTIVE (1 << 0x12)
|
|
|
|
|
|
|
|
#define GEN8_CTX_STATUS_IDLE_ACTIVE (1 << 0)
|
|
|
|
#define GEN8_CTX_STATUS_PREEMPTED (1 << 1)
|
|
|
|
#define GEN8_CTX_STATUS_ELEMENT_SWITCH (1 << 2)
|
|
|
|
#define GEN8_CTX_STATUS_ACTIVE_IDLE (1 << 3)
|
|
|
|
#define GEN8_CTX_STATUS_COMPLETE (1 << 4)
|
|
|
|
#define GEN8_CTX_STATUS_LITE_RESTORE (1 << 15)
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
|
2016-09-09 21:11:46 +08:00
|
|
|
#define GEN8_CTX_STATUS_COMPLETED_MASK \
|
2017-11-20 20:34:56 +08:00
|
|
|
(GEN8_CTX_STATUS_COMPLETE | GEN8_CTX_STATUS_PREEMPTED)
|
2016-09-09 21:11:46 +08:00
|
|
|
|
2016-04-29 16:07:06 +08:00
|
|
|
/* Typical size of the average request (2 pipecontrols and a MI_BB) */
|
|
|
|
#define EXECLISTS_REQUEST_SIZE 64 /* bytes */
|
2016-10-05 04:11:26 +08:00
|
|
|
#define WA_TAIL_DWORDS 2
|
2017-09-29 03:38:59 +08:00
|
|
|
#define WA_TAIL_BYTES (sizeof(u32) * WA_TAIL_DWORDS)
|
2016-10-05 04:11:26 +08:00
|
|
|
|
2016-05-24 21:53:34 +08:00
|
|
|
static int execlists_context_deferred_alloc(struct i915_gem_context *ctx,
|
2018-05-18 05:26:32 +08:00
|
|
|
struct intel_engine_cs *engine,
|
|
|
|
struct intel_context *ce);
|
2016-10-05 04:11:26 +08:00
|
|
|
static void execlists_init_reg_state(u32 *reg_state,
|
|
|
|
struct i915_gem_context *ctx,
|
|
|
|
struct intel_engine_cs *engine,
|
|
|
|
struct intel_ring *ring);
|
2014-11-13 18:28:56 +08:00
|
|
|
|
2018-02-22 22:22:29 +08:00
|
|
|
static inline struct i915_priolist *to_priolist(struct rb_node *rb)
|
|
|
|
{
|
|
|
|
return rb_entry(rb, struct i915_priolist, node);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int rq_prio(const struct i915_request *rq)
|
|
|
|
{
|
2018-04-19 02:40:52 +08:00
|
|
|
return rq->sched.attr.priority;
|
2018-02-22 22:22:29 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool need_preempt(const struct intel_engine_cs *engine,
|
|
|
|
const struct i915_request *last,
|
|
|
|
int prio)
|
|
|
|
{
|
2018-04-04 02:35:37 +08:00
|
|
|
return (intel_engine_has_preemption(engine) &&
|
2018-05-01 20:21:31 +08:00
|
|
|
__execlists_need_preempt(prio, rq_prio(last)) &&
|
|
|
|
!i915_request_completed(last));
|
2018-02-22 22:22:29 +08:00
|
|
|
}
|
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
/*
|
2016-01-15 23:10:27 +08:00
|
|
|
* The context descriptor encodes various attributes of a context,
|
|
|
|
* including its GTT address and some flags. Because it's fairly
|
|
|
|
* expensive to calculate, we'll just do it once and cache the result,
|
|
|
|
* which remains valid until the context is unpinned.
|
|
|
|
*
|
2016-07-16 03:48:06 +08:00
|
|
|
* This is what a descriptor looks like, from LSB to MSB::
|
|
|
|
*
|
2017-01-27 21:03:09 +08:00
|
|
|
* bits 0-11: flags, GEN8_CTX_* (cached in ctx->desc_template)
|
2016-07-16 03:48:06 +08:00
|
|
|
* bits 12-31: LRCA, GTT address of (the HWSP of) this context
|
2018-06-02 19:29:45 +08:00
|
|
|
* bits 32-52: ctx ID, a globally unique tag (highest bit used by GuC)
|
2016-07-16 03:48:06 +08:00
|
|
|
* bits 53-54: mbz, reserved for use by hardware
|
|
|
|
* bits 55-63: group ID, currently unused and set to 0
|
2018-03-03 00:14:58 +08:00
|
|
|
*
|
|
|
|
* Starting from Gen11, the upper dword of the descriptor has a new format:
|
|
|
|
*
|
|
|
|
* bits 32-36: reserved
|
|
|
|
* bits 37-47: SW context ID
|
|
|
|
* bits 48:53: engine instance
|
|
|
|
* bit 54: mbz, reserved for use by hardware
|
|
|
|
* bits 55-60: SW counter
|
|
|
|
* bits 61-63: engine class
|
|
|
|
*
|
|
|
|
* engine info, SW context ID and SW counter need to form a unique number
|
|
|
|
* (Context ID) per lrc.
|
2014-07-25 00:04:48 +08:00
|
|
|
*/
|
2016-01-15 23:10:27 +08:00
|
|
|
static void
|
2016-05-24 21:53:34 +08:00
|
|
|
intel_lr_context_descriptor_update(struct i915_gem_context *ctx,
|
2018-05-18 05:26:32 +08:00
|
|
|
struct intel_engine_cs *engine,
|
|
|
|
struct intel_context *ce)
|
2014-07-25 00:04:36 +08:00
|
|
|
{
|
2016-04-28 16:56:52 +08:00
|
|
|
u64 desc;
|
2014-07-25 00:04:36 +08:00
|
|
|
|
2018-03-03 00:14:58 +08:00
|
|
|
BUILD_BUG_ON(MAX_CONTEXT_HW_ID > (BIT(GEN8_CTX_ID_WIDTH)));
|
|
|
|
BUILD_BUG_ON(GEN11_MAX_CONTEXT_HW_ID > (BIT(GEN11_SW_CTX_ID_WIDTH)));
|
2014-07-25 00:04:36 +08:00
|
|
|
|
2017-01-27 21:03:09 +08:00
|
|
|
desc = ctx->desc_template; /* bits 0-11 */
|
2018-03-03 00:14:58 +08:00
|
|
|
GEM_BUG_ON(desc & GENMASK_ULL(63, 12));
|
|
|
|
|
2017-09-13 16:56:00 +08:00
|
|
|
desc |= i915_ggtt_offset(ce->state) + LRC_HEADER_PAGES * PAGE_SIZE;
|
2016-05-24 21:53:37 +08:00
|
|
|
/* bits 12-31 */
|
2018-03-03 00:14:58 +08:00
|
|
|
GEM_BUG_ON(desc & GENMASK_ULL(63, 32));
|
|
|
|
|
2018-06-02 19:29:46 +08:00
|
|
|
/*
|
|
|
|
* The following 32bits are copied into the OA reports (dword 2).
|
|
|
|
* Consider updating oa_get_render_ctx_id in i915_perf.c when changing
|
|
|
|
* anything below.
|
|
|
|
*/
|
2018-03-03 00:14:58 +08:00
|
|
|
if (INTEL_GEN(ctx->i915) >= 11) {
|
|
|
|
GEM_BUG_ON(ctx->hw_id >= BIT(GEN11_SW_CTX_ID_WIDTH));
|
|
|
|
desc |= (u64)ctx->hw_id << GEN11_SW_CTX_ID_SHIFT;
|
|
|
|
/* bits 37-47 */
|
|
|
|
|
|
|
|
desc |= (u64)engine->instance << GEN11_ENGINE_INSTANCE_SHIFT;
|
|
|
|
/* bits 48-53 */
|
|
|
|
|
|
|
|
/* TODO: decide what to do with SW counter (bits 55-60) */
|
|
|
|
|
|
|
|
desc |= (u64)engine->class << GEN11_ENGINE_CLASS_SHIFT;
|
|
|
|
/* bits 61-63 */
|
|
|
|
} else {
|
|
|
|
GEM_BUG_ON(ctx->hw_id >= BIT(GEN8_CTX_ID_WIDTH));
|
|
|
|
desc |= (u64)ctx->hw_id << GEN8_CTX_ID_SHIFT; /* bits 32-52 */
|
|
|
|
}
|
2015-09-04 19:59:15 +08:00
|
|
|
|
2016-05-24 21:53:37 +08:00
|
|
|
ce->lrc_desc = desc;
|
2015-09-04 19:59:15 +08:00
|
|
|
}
|
|
|
|
|
2017-09-17 04:44:13 +08:00
|
|
|
static struct i915_priolist *
|
2018-05-08 08:30:45 +08:00
|
|
|
lookup_priolist(struct intel_engine_cs *engine, int prio)
|
2017-09-17 04:44:12 +08:00
|
|
|
{
|
2017-09-22 20:43:03 +08:00
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
2017-09-17 04:44:12 +08:00
|
|
|
struct i915_priolist *p;
|
|
|
|
struct rb_node **parent, *rb;
|
|
|
|
bool first = true;
|
|
|
|
|
2017-09-22 20:43:03 +08:00
|
|
|
if (unlikely(execlists->no_priolist))
|
2017-09-17 04:44:12 +08:00
|
|
|
prio = I915_PRIORITY_NORMAL;
|
|
|
|
|
|
|
|
find_priolist:
|
|
|
|
/* most positive priority is scheduled first, equal priorities fifo */
|
|
|
|
rb = NULL;
|
2018-06-29 15:53:20 +08:00
|
|
|
parent = &execlists->queue.rb_root.rb_node;
|
2017-09-17 04:44:12 +08:00
|
|
|
while (*parent) {
|
|
|
|
rb = *parent;
|
2018-02-22 22:22:29 +08:00
|
|
|
p = to_priolist(rb);
|
2017-09-17 04:44:12 +08:00
|
|
|
if (prio > p->priority) {
|
|
|
|
parent = &rb->rb_left;
|
|
|
|
} else if (prio < p->priority) {
|
|
|
|
parent = &rb->rb_right;
|
|
|
|
first = false;
|
|
|
|
} else {
|
2017-09-17 04:44:13 +08:00
|
|
|
return p;
|
2017-09-17 04:44:12 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (prio == I915_PRIORITY_NORMAL) {
|
2017-09-22 20:43:03 +08:00
|
|
|
p = &execlists->default_priolist;
|
2017-09-17 04:44:12 +08:00
|
|
|
} else {
|
|
|
|
p = kmem_cache_alloc(engine->i915->priorities, GFP_ATOMIC);
|
|
|
|
/* Convert an allocation failure to a priority bump */
|
|
|
|
if (unlikely(!p)) {
|
|
|
|
prio = I915_PRIORITY_NORMAL; /* recurses just once */
|
|
|
|
|
|
|
|
/* To maintain ordering with all rendering, after an
|
|
|
|
* allocation failure we have to disable all scheduling.
|
|
|
|
* Requests will then be executed in fifo, and schedule
|
|
|
|
* will ensure that dependencies are emitted in fifo.
|
|
|
|
* There will be still some reordering with existing
|
|
|
|
* requests, so if userspace lied about their
|
|
|
|
* dependencies that reordering may be visible.
|
|
|
|
*/
|
2017-09-22 20:43:03 +08:00
|
|
|
execlists->no_priolist = true;
|
2017-09-17 04:44:12 +08:00
|
|
|
goto find_priolist;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
p->priority = prio;
|
2017-09-17 04:44:13 +08:00
|
|
|
INIT_LIST_HEAD(&p->requests);
|
2017-09-17 04:44:12 +08:00
|
|
|
rb_link_node(&p->node, rb, parent);
|
2018-06-29 15:53:20 +08:00
|
|
|
rb_insert_color_cached(&p->node, &execlists->queue, first);
|
2017-09-17 04:44:12 +08:00
|
|
|
|
2018-02-22 22:22:29 +08:00
|
|
|
return p;
|
2017-09-17 04:44:12 +08:00
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static void unwind_wa_tail(struct i915_request *rq)
|
2017-09-29 03:38:59 +08:00
|
|
|
{
|
|
|
|
rq->tail = intel_ring_wrap(rq->ring, rq->wa_tail - WA_TAIL_BYTES);
|
|
|
|
assert_ring_tail_valid(rq->ring, rq->tail);
|
|
|
|
}
|
|
|
|
|
2017-10-26 04:00:18 +08:00
|
|
|
static void __unwind_incomplete_requests(struct intel_engine_cs *engine)
|
2017-09-29 03:38:59 +08:00
|
|
|
{
|
2018-02-21 17:56:36 +08:00
|
|
|
struct i915_request *rq, *rn;
|
2017-09-29 03:39:01 +08:00
|
|
|
struct i915_priolist *uninitialized_var(p);
|
|
|
|
int last_prio = I915_PRIORITY_INVALID;
|
2017-09-29 03:38:59 +08:00
|
|
|
|
2018-05-03 00:38:39 +08:00
|
|
|
lockdep_assert_held(&engine->timeline.lock);
|
2017-09-29 03:38:59 +08:00
|
|
|
|
|
|
|
list_for_each_entry_safe_reverse(rq, rn,
|
2018-05-03 00:38:39 +08:00
|
|
|
&engine->timeline.requests,
|
2017-09-29 03:38:59 +08:00
|
|
|
link) {
|
2018-02-21 17:56:36 +08:00
|
|
|
if (i915_request_completed(rq))
|
2017-09-29 03:38:59 +08:00
|
|
|
return;
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
__i915_request_unsubmit(rq);
|
2017-09-29 03:38:59 +08:00
|
|
|
unwind_wa_tail(rq);
|
|
|
|
|
2018-02-22 22:22:29 +08:00
|
|
|
GEM_BUG_ON(rq_prio(rq) == I915_PRIORITY_INVALID);
|
|
|
|
if (rq_prio(rq) != last_prio) {
|
|
|
|
last_prio = rq_prio(rq);
|
2018-05-08 08:30:45 +08:00
|
|
|
p = lookup_priolist(engine, last_prio);
|
2017-09-29 03:39:01 +08:00
|
|
|
}
|
|
|
|
|
2018-05-08 08:30:46 +08:00
|
|
|
GEM_BUG_ON(p->priority != rq_prio(rq));
|
2018-04-19 02:40:51 +08:00
|
|
|
list_add(&rq->sched.link, &p->requests);
|
2017-09-29 03:38:59 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
drm/i915/guc: Preemption! With GuC
Pretty similar to what we have on execlists.
We're reusing most of the GEM code, however, due to GuC quirks we need a
couple of extra bits.
Preemption is implemented as GuC action, and actions can be pretty slow.
Because of that, we're using a mutex to serialize them. Since we're
requesting preemption from the tasklet, the task of creating a workitem
and wrapping it in GuC action is delegated to a worker.
To distinguish that preemption has finished, we're using additional
piece of HWSP, and since we're not getting context switch interrupts,
we're also adding a user interrupt.
The fact that our special preempt context has completed unfortunately
doesn't mean that we're ready to submit new work. We also need to wait
for GuC to finish its own processing.
v2: Don't compile out the wait for GuC, handle workqueue flush on reset,
no need for ordered workqueue, put on a reviewer hat when looking at my own
patches (Chris)
Move struct work around in intel_guc, move user interruput outside of
conditional (Michał)
Keep ring around rather than chase though intel_context
v3: Extract WA for flushing ggtt writes to a helper (Chris)
Keep work_struct in intel_guc rather than engine (Michał)
Use ordered workqueue for inject_preempt worker to avoid GuC quirks.
v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele)
Drop stray newlines, use container_of for intel_guc in worker,
check for presence of workqueue when flushing it, rather than
enable_guc_submission modparam, reorder preempt postprocessing (Chris)
v5: Make wq NULL after destroying it
v6: Swap struct guc_preempt_work members (Michał)
Signed-off-by: Michał Winiarski <michal.winiarski@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Jeff McGee <jeff.mcgee@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Oscar Mateo <oscar.mateo@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 21:35:58 +08:00
|
|
|
void
|
2017-10-26 04:00:18 +08:00
|
|
|
execlists_unwind_incomplete_requests(struct intel_engine_execlists *execlists)
|
|
|
|
{
|
|
|
|
struct intel_engine_cs *engine =
|
|
|
|
container_of(execlists, typeof(*engine), execlists);
|
2018-05-09 05:03:17 +08:00
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&engine->timeline.lock, flags);
|
2017-10-26 04:00:18 +08:00
|
|
|
|
|
|
|
__unwind_incomplete_requests(engine);
|
2018-05-09 05:03:17 +08:00
|
|
|
|
|
|
|
spin_unlock_irqrestore(&engine->timeline.lock, flags);
|
2017-10-26 04:00:18 +08:00
|
|
|
}
|
|
|
|
|
2016-09-09 21:11:45 +08:00
|
|
|
static inline void
|
2018-02-21 17:56:36 +08:00
|
|
|
execlists_context_status_change(struct i915_request *rq, unsigned long status)
|
2014-07-25 00:04:36 +08:00
|
|
|
{
|
2016-09-09 21:11:45 +08:00
|
|
|
/*
|
|
|
|
* Only used when GVT-g is enabled now. When GVT-g is disabled,
|
|
|
|
* The compiler should eliminate this function as dead-code.
|
|
|
|
*/
|
|
|
|
if (!IS_ENABLED(CONFIG_DRM_I915_GVT))
|
|
|
|
return;
|
2015-01-16 17:34:35 +08:00
|
|
|
|
2017-03-13 10:47:11 +08:00
|
|
|
atomic_notifier_call_chain(&rq->engine->context_status_notifier,
|
|
|
|
status, rq);
|
2014-07-25 00:04:36 +08:00
|
|
|
}
|
|
|
|
|
2018-03-31 21:06:26 +08:00
|
|
|
inline void
|
|
|
|
execlists_user_begin(struct intel_engine_execlists *execlists,
|
|
|
|
const struct execlist_port *port)
|
|
|
|
{
|
|
|
|
execlists_set_active_once(execlists, EXECLISTS_ACTIVE_USER);
|
|
|
|
}
|
|
|
|
|
|
|
|
inline void
|
|
|
|
execlists_user_end(struct intel_engine_execlists *execlists)
|
|
|
|
{
|
|
|
|
execlists_clear_active(execlists, EXECLISTS_ACTIVE_USER);
|
|
|
|
}
|
|
|
|
|
2017-11-22 02:18:47 +08:00
|
|
|
static inline void
|
2018-02-21 17:56:36 +08:00
|
|
|
execlists_context_schedule_in(struct i915_request *rq)
|
2017-11-22 02:18:47 +08:00
|
|
|
{
|
|
|
|
execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_IN);
|
2017-11-22 02:18:48 +08:00
|
|
|
intel_engine_context_in(rq->engine);
|
2017-11-22 02:18:47 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
2018-05-03 07:02:02 +08:00
|
|
|
execlists_context_schedule_out(struct i915_request *rq, unsigned long status)
|
2017-11-22 02:18:47 +08:00
|
|
|
{
|
2017-11-22 02:18:48 +08:00
|
|
|
intel_engine_context_out(rq->engine);
|
2018-05-03 07:02:02 +08:00
|
|
|
execlists_context_status_change(rq, status);
|
|
|
|
trace_i915_request_out(rq);
|
2017-11-22 02:18:47 +08:00
|
|
|
}
|
|
|
|
|
2016-02-27 00:58:32 +08:00
|
|
|
static void
|
|
|
|
execlists_update_context_pdps(struct i915_hw_ppgtt *ppgtt, u32 *reg_state)
|
|
|
|
{
|
|
|
|
ASSIGN_CTX_PDP(ppgtt, reg_state, 3);
|
|
|
|
ASSIGN_CTX_PDP(ppgtt, reg_state, 2);
|
|
|
|
ASSIGN_CTX_PDP(ppgtt, reg_state, 1);
|
|
|
|
ASSIGN_CTX_PDP(ppgtt, reg_state, 0);
|
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static u64 execlists_update_context(struct i915_request *rq)
|
2014-07-25 00:04:37 +08:00
|
|
|
{
|
2018-05-18 05:26:32 +08:00
|
|
|
struct intel_context *ce = rq->hw_context;
|
2017-02-06 18:37:16 +08:00
|
|
|
struct i915_hw_ppgtt *ppgtt =
|
2018-05-18 05:26:30 +08:00
|
|
|
rq->gem_context->ppgtt ?: rq->i915->mm.aliasing_ppgtt;
|
2016-09-09 21:11:46 +08:00
|
|
|
u32 *reg_state = ce->lrc_reg_state;
|
2014-07-25 00:04:37 +08:00
|
|
|
|
2017-04-25 21:00:49 +08:00
|
|
|
reg_state[CTX_RING_TAIL+1] = intel_ring_set_tail(rq->ring, rq->tail);
|
2014-07-25 00:04:37 +08:00
|
|
|
|
2016-02-27 00:58:32 +08:00
|
|
|
/* True 32b PPGTT with dynamic page allocation: update PDP
|
|
|
|
* registers and point the unallocated PDPs to scratch page.
|
|
|
|
* PML4 is allocated during ppgtt init, so this is not needed
|
|
|
|
* in 48-bit mode.
|
|
|
|
*/
|
2018-06-05 23:37:58 +08:00
|
|
|
if (ppgtt && !i915_vm_is_48bit(&ppgtt->vm))
|
2016-02-27 00:58:32 +08:00
|
|
|
execlists_update_context_pdps(ppgtt, reg_state);
|
2016-09-09 21:11:46 +08:00
|
|
|
|
|
|
|
return ce->lrc_desc;
|
2014-07-25 00:04:37 +08:00
|
|
|
}
|
|
|
|
|
2018-03-03 00:14:59 +08:00
|
|
|
static inline void write_desc(struct intel_engine_execlists *execlists, u64 desc, u32 port)
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
{
|
2018-03-03 00:14:59 +08:00
|
|
|
if (execlists->ctrl_reg) {
|
|
|
|
writel(lower_32_bits(desc), execlists->submit_reg + port * 2);
|
|
|
|
writel(upper_32_bits(desc), execlists->submit_reg + port * 2 + 1);
|
|
|
|
} else {
|
|
|
|
writel(upper_32_bits(desc), execlists->submit_reg);
|
|
|
|
writel(lower_32_bits(desc), execlists->submit_reg);
|
|
|
|
}
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
}
|
|
|
|
|
2016-09-09 21:11:46 +08:00
|
|
|
static void execlists_submit_ports(struct intel_engine_cs *engine)
|
2016-09-09 21:11:45 +08:00
|
|
|
{
|
2018-03-03 00:14:59 +08:00
|
|
|
struct intel_engine_execlists *execlists = &engine->execlists;
|
|
|
|
struct execlist_port *port = execlists->port;
|
2017-05-17 20:10:00 +08:00
|
|
|
unsigned int n;
|
2016-09-09 21:11:45 +08:00
|
|
|
|
2018-07-19 15:50:29 +08:00
|
|
|
/*
|
|
|
|
* We can skip acquiring intel_runtime_pm_get() here as it was taken
|
|
|
|
* on our behalf by the request (see i915_gem_mark_busy()) and it will
|
|
|
|
* not be relinquished until the device is idle (see
|
|
|
|
* i915_gem_idle_work_handler()). As a precaution, we make sure
|
|
|
|
* that all ELSP are drained i.e. we have processed the CSB,
|
|
|
|
* before allowing ourselves to idle and calling intel_runtime_pm_put().
|
|
|
|
*/
|
|
|
|
GEM_BUG_ON(!engine->i915->gt.awake);
|
|
|
|
|
2018-03-03 00:14:59 +08:00
|
|
|
/*
|
|
|
|
* ELSQ note: the submit queue is not cleared after being submitted
|
|
|
|
* to the HW so we need to make sure we always clean it up. This is
|
|
|
|
* currently ensured by the fact that we always write the same number
|
|
|
|
* of elsq entries, keep this in mind before changing the loop below.
|
|
|
|
*/
|
|
|
|
for (n = execlists_num_ports(execlists); n--; ) {
|
2018-02-21 17:56:36 +08:00
|
|
|
struct i915_request *rq;
|
2017-05-17 20:10:00 +08:00
|
|
|
unsigned int count;
|
|
|
|
u64 desc;
|
|
|
|
|
|
|
|
rq = port_unpack(&port[n], &count);
|
|
|
|
if (rq) {
|
|
|
|
GEM_BUG_ON(count > !n);
|
|
|
|
if (!count++)
|
2017-11-22 02:18:47 +08:00
|
|
|
execlists_context_schedule_in(rq);
|
2017-05-17 20:10:00 +08:00
|
|
|
port_set(&port[n], port_pack(rq, count));
|
|
|
|
desc = execlists_update_context(rq);
|
|
|
|
GEM_DEBUG_EXEC(port[n].context_id = upper_32_bits(desc));
|
2017-11-09 22:30:19 +08:00
|
|
|
|
2018-04-06 20:35:14 +08:00
|
|
|
GEM_TRACE("%s in[%d]: ctx=%d.%d, global=%d (fence %llx:%d) (current %d), prio=%d\n",
|
2017-11-09 22:30:19 +08:00
|
|
|
engine->name, n,
|
2017-12-20 06:09:16 +08:00
|
|
|
port[n].context_id, count,
|
2018-02-22 22:22:29 +08:00
|
|
|
rq->global_seqno,
|
2018-04-06 20:35:14 +08:00
|
|
|
rq->fence.context, rq->fence.seqno,
|
2018-03-28 05:01:57 +08:00
|
|
|
intel_engine_get_seqno(engine),
|
2018-02-22 22:22:29 +08:00
|
|
|
rq_prio(rq));
|
2017-05-17 20:10:00 +08:00
|
|
|
} else {
|
|
|
|
GEM_BUG_ON(!n);
|
|
|
|
desc = 0;
|
|
|
|
}
|
2016-09-09 21:11:45 +08:00
|
|
|
|
2018-03-03 00:14:59 +08:00
|
|
|
write_desc(execlists, desc, n);
|
2017-05-17 20:10:00 +08:00
|
|
|
}
|
2018-03-03 00:14:59 +08:00
|
|
|
|
|
|
|
/* we need to manually load the submit queue */
|
|
|
|
if (execlists->ctrl_reg)
|
|
|
|
writel(EL_CTRL_LOAD, execlists->ctrl_reg);
|
|
|
|
|
|
|
|
execlists_clear_active(execlists, EXECLISTS_ACTIVE_HWACK);
|
2016-09-09 21:11:45 +08:00
|
|
|
}
|
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
static bool ctx_single_port_submission(const struct intel_context *ce)
|
2014-07-25 00:04:36 +08:00
|
|
|
{
|
2016-09-09 21:11:46 +08:00
|
|
|
return (IS_ENABLED(CONFIG_DRM_I915_GVT) &&
|
2018-05-18 05:26:32 +08:00
|
|
|
i915_gem_context_force_single_submission(ce->gem_context));
|
2016-09-09 21:11:46 +08:00
|
|
|
}
|
2014-07-25 00:04:36 +08:00
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
static bool can_merge_ctx(const struct intel_context *prev,
|
|
|
|
const struct intel_context *next)
|
2016-09-09 21:11:46 +08:00
|
|
|
{
|
|
|
|
if (prev != next)
|
|
|
|
return false;
|
2016-03-17 20:59:46 +08:00
|
|
|
|
2016-09-09 21:11:46 +08:00
|
|
|
if (ctx_single_port_submission(prev))
|
|
|
|
return false;
|
2016-03-17 20:59:46 +08:00
|
|
|
|
2016-09-09 21:11:46 +08:00
|
|
|
return true;
|
2014-07-25 00:04:36 +08:00
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static void port_assign(struct execlist_port *port, struct i915_request *rq)
|
2017-05-17 20:10:00 +08:00
|
|
|
{
|
|
|
|
GEM_BUG_ON(rq == port_request(port));
|
|
|
|
|
|
|
|
if (port_isset(port))
|
2018-02-21 17:56:36 +08:00
|
|
|
i915_request_put(port_request(port));
|
2017-05-17 20:10:00 +08:00
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
port_set(port, port_pack(i915_request_get(rq), port_count(port)));
|
2017-05-17 20:10:00 +08:00
|
|
|
}
|
|
|
|
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
static void inject_preempt_context(struct intel_engine_cs *engine)
|
|
|
|
{
|
2018-03-03 00:14:59 +08:00
|
|
|
struct intel_engine_execlists *execlists = &engine->execlists;
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
struct intel_context *ce =
|
2018-04-30 21:15:01 +08:00
|
|
|
to_intel_context(engine->i915->preempt_context, engine);
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
unsigned int n;
|
|
|
|
|
2018-03-03 00:14:59 +08:00
|
|
|
GEM_BUG_ON(execlists->preempt_complete_status !=
|
2018-02-08 05:05:44 +08:00
|
|
|
upper_32_bits(ce->lrc_desc));
|
2018-01-25 19:24:42 +08:00
|
|
|
GEM_BUG_ON((ce->lrc_reg_state[CTX_CONTEXT_CONTROL + 1] &
|
|
|
|
_MASKED_BIT_ENABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT |
|
|
|
|
CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT)) !=
|
|
|
|
_MASKED_BIT_ENABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT |
|
|
|
|
CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT));
|
|
|
|
|
2018-02-22 22:22:29 +08:00
|
|
|
/*
|
|
|
|
* Switch to our empty preempt context so
|
|
|
|
* the state of the GPU is known (idle).
|
|
|
|
*/
|
2017-12-20 17:06:26 +08:00
|
|
|
GEM_TRACE("%s\n", engine->name);
|
2018-03-03 00:14:59 +08:00
|
|
|
for (n = execlists_num_ports(execlists); --n; )
|
|
|
|
write_desc(execlists, 0, n);
|
|
|
|
|
|
|
|
write_desc(execlists, ce->lrc_desc, n);
|
|
|
|
|
|
|
|
/* we need to manually load the submit queue */
|
|
|
|
if (execlists->ctrl_reg)
|
|
|
|
writel(EL_CTRL_LOAD, execlists->ctrl_reg);
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
|
2018-05-17 02:33:50 +08:00
|
|
|
execlists_clear_active(execlists, EXECLISTS_ACTIVE_HWACK);
|
|
|
|
execlists_set_active(execlists, EXECLISTS_ACTIVE_PREEMPT);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void complete_preempt_context(struct intel_engine_execlists *execlists)
|
|
|
|
{
|
|
|
|
GEM_BUG_ON(!execlists_is_active(execlists, EXECLISTS_ACTIVE_PREEMPT));
|
|
|
|
|
2018-07-16 21:21:54 +08:00
|
|
|
if (inject_preempt_hang(execlists))
|
|
|
|
return;
|
|
|
|
|
2018-05-17 02:33:50 +08:00
|
|
|
execlists_cancel_port_requests(execlists);
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
__unwind_incomplete_requests(container_of(execlists,
|
|
|
|
struct intel_engine_cs,
|
|
|
|
execlists));
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
}
|
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
static void execlists_dequeue(struct intel_engine_cs *engine)
|
2014-07-25 00:04:38 +08:00
|
|
|
{
|
2017-09-22 20:43:06 +08:00
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
|
|
|
struct execlist_port *port = execlists->port;
|
2017-09-22 20:43:07 +08:00
|
|
|
const struct execlist_port * const last_port =
|
|
|
|
&execlists->port[execlists->port_mask];
|
2018-02-21 17:56:36 +08:00
|
|
|
struct i915_request *last = port_request(port);
|
2016-11-15 04:41:03 +08:00
|
|
|
struct rb_node *rb;
|
2016-09-09 21:11:46 +08:00
|
|
|
bool submit = false;
|
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
/*
|
|
|
|
* Hardware submission is through 2 ports. Conceptually each port
|
2016-09-09 21:11:46 +08:00
|
|
|
* has a (RING_START, RING_HEAD, RING_TAIL) tuple. RING_START is
|
|
|
|
* static for a context, and unique to each, so we only execute
|
|
|
|
* requests belonging to a single context from each ring. RING_HEAD
|
|
|
|
* is maintained by the CS in the context image, it marks the place
|
|
|
|
* where it got up to last time, and through RING_TAIL we tell the CS
|
|
|
|
* where we want to execute up to this time.
|
|
|
|
*
|
|
|
|
* In this list the requests are in order of execution. Consecutive
|
|
|
|
* requests from the same context are adjacent in the ringbuffer. We
|
|
|
|
* can combine these requests into a single RING_TAIL update:
|
|
|
|
*
|
|
|
|
* RING_HEAD...req1...req2
|
|
|
|
* ^- RING_TAIL
|
|
|
|
* since to execute req2 the CS must first execute req1.
|
|
|
|
*
|
|
|
|
* Our goal then is to point each port to the end of a consecutive
|
|
|
|
* sequence of requests as being the most optimal (fewest wake ups
|
|
|
|
* and context switches) submission.
|
2015-05-11 23:03:27 +08:00
|
|
|
*/
|
2014-07-25 00:04:38 +08:00
|
|
|
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
if (last) {
|
|
|
|
/*
|
|
|
|
* Don't resubmit or switch until all outstanding
|
|
|
|
* preemptions (lite-restore) are seen. Then we
|
|
|
|
* know the next preemption status we see corresponds
|
|
|
|
* to this ELSP update.
|
|
|
|
*/
|
2018-03-24 20:58:29 +08:00
|
|
|
GEM_BUG_ON(!execlists_is_active(execlists,
|
|
|
|
EXECLISTS_ACTIVE_USER));
|
2017-11-20 20:34:58 +08:00
|
|
|
GEM_BUG_ON(!port_count(&port[0]));
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
|
2017-11-20 20:34:58 +08:00
|
|
|
/*
|
|
|
|
* If we write to ELSP a second time before the HW has had
|
|
|
|
* a chance to respond to the previous write, we can confuse
|
|
|
|
* the HW and hit "undefined behaviour". After writing to ELSP,
|
|
|
|
* we must then wait until we see a context-switch event from
|
|
|
|
* the HW to indicate that it has had a chance to respond.
|
|
|
|
*/
|
|
|
|
if (!execlists_is_active(execlists, EXECLISTS_ACTIVE_HWACK))
|
2018-06-29 04:12:04 +08:00
|
|
|
return;
|
2017-11-20 20:34:58 +08:00
|
|
|
|
2018-02-22 22:22:29 +08:00
|
|
|
if (need_preempt(engine, last, execlists->queue_priority)) {
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
inject_preempt_context(engine);
|
2018-06-29 04:12:04 +08:00
|
|
|
return;
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
}
|
2018-02-22 22:22:29 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* In theory, we could coalesce more requests onto
|
|
|
|
* the second port (the first port is active, with
|
|
|
|
* no preemptions pending). However, that means we
|
|
|
|
* then have to deal with the possible lite-restore
|
|
|
|
* of the second port (as we submit the ELSP, there
|
|
|
|
* may be a context-switch) but also we may complete
|
|
|
|
* the resubmission before the context-switch. Ergo,
|
|
|
|
* coalescing onto the second port will cause a
|
|
|
|
* preemption event, but we cannot predict whether
|
|
|
|
* that will affect port[0] or port[1].
|
|
|
|
*
|
|
|
|
* If the second port is already active, we can wait
|
|
|
|
* until the next context-switch before contemplating
|
|
|
|
* new requests. The GPU will be busy and we should be
|
|
|
|
* able to resubmit the new ELSP before it idles,
|
|
|
|
* avoiding pipeline bubbles (momentary pauses where
|
|
|
|
* the driver is unable to keep up the supply of new
|
|
|
|
* work). However, we have to double check that the
|
|
|
|
* priorities of the ports haven't been switch.
|
|
|
|
*/
|
|
|
|
if (port_count(&port[1]))
|
2018-06-29 04:12:04 +08:00
|
|
|
return;
|
2018-02-22 22:22:29 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* WaIdleLiteRestore:bdw,skl
|
|
|
|
* Apply the wa NOOPs to prevent
|
|
|
|
* ring:HEAD == rq:TAIL as we resubmit the
|
|
|
|
* request. See gen8_emit_breadcrumb() for
|
|
|
|
* where we prepare the padding after the
|
|
|
|
* end of the request.
|
|
|
|
*/
|
|
|
|
last->tail = last->wa_tail;
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
}
|
|
|
|
|
2018-06-29 15:53:20 +08:00
|
|
|
while ((rb = rb_first_cached(&execlists->queue))) {
|
2018-02-22 22:22:29 +08:00
|
|
|
struct i915_priolist *p = to_priolist(rb);
|
2018-02-21 17:56:36 +08:00
|
|
|
struct i915_request *rq, *rn;
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
|
2018-04-19 02:40:51 +08:00
|
|
|
list_for_each_entry_safe(rq, rn, &p->requests, sched.link) {
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
/*
|
|
|
|
* Can we combine this request with the current port?
|
|
|
|
* It has to be the same context/ringbuffer and not
|
|
|
|
* have any exceptions (e.g. GVT saying never to
|
|
|
|
* combine contexts).
|
|
|
|
*
|
|
|
|
* If we can combine the requests, we can execute both
|
|
|
|
* by updating the RING_TAIL to point to the end of the
|
|
|
|
* second request, and so we never need to tell the
|
|
|
|
* hardware about the first.
|
2016-09-09 21:11:46 +08:00
|
|
|
*/
|
2018-05-18 05:26:32 +08:00
|
|
|
if (last &&
|
|
|
|
!can_merge_ctx(rq->hw_context, last->hw_context)) {
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
/*
|
|
|
|
* If we are on the second port and cannot
|
|
|
|
* combine this request with the last, then we
|
|
|
|
* are done.
|
|
|
|
*/
|
2017-09-22 20:43:07 +08:00
|
|
|
if (port == last_port) {
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
__list_del_many(&p->requests,
|
2018-04-19 02:40:51 +08:00
|
|
|
&rq->sched.link);
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If GVT overrides us we only ever submit
|
|
|
|
* port[0], leaving port[1] empty. Note that we
|
|
|
|
* also have to be careful that we don't queue
|
|
|
|
* the same context (even though a different
|
|
|
|
* request) to the second port.
|
|
|
|
*/
|
2018-05-18 05:26:32 +08:00
|
|
|
if (ctx_single_port_submission(last->hw_context) ||
|
|
|
|
ctx_single_port_submission(rq->hw_context)) {
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
__list_del_many(&p->requests,
|
2018-04-19 02:40:51 +08:00
|
|
|
&rq->sched.link);
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
GEM_BUG_ON(last->hw_context == rq->hw_context);
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
|
|
|
|
if (submit)
|
|
|
|
port_assign(port, last);
|
|
|
|
port++;
|
2017-09-22 20:43:06 +08:00
|
|
|
|
|
|
|
GEM_BUG_ON(port_isset(port));
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
}
|
2016-09-09 21:11:46 +08:00
|
|
|
|
2018-04-19 02:40:51 +08:00
|
|
|
INIT_LIST_HEAD(&rq->sched.link);
|
2018-02-21 17:56:36 +08:00
|
|
|
__i915_request_submit(rq);
|
|
|
|
trace_i915_request_in(rq, port_index(port, execlists));
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
last = rq;
|
|
|
|
submit = true;
|
2016-09-09 21:11:46 +08:00
|
|
|
}
|
2016-11-15 04:40:59 +08:00
|
|
|
|
2018-06-29 15:53:20 +08:00
|
|
|
rb_erase_cached(&p->node, &execlists->queue);
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
INIT_LIST_HEAD(&p->requests);
|
|
|
|
if (p->priority != I915_PRIORITY_NORMAL)
|
2017-05-17 20:10:04 +08:00
|
|
|
kmem_cache_free(engine->i915->priorities, p);
|
2018-02-22 22:22:29 +08:00
|
|
|
}
|
2018-04-11 18:39:29 +08:00
|
|
|
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
done:
|
2018-04-11 18:39:29 +08:00
|
|
|
/*
|
|
|
|
* Here be a bit of magic! Or sleight-of-hand, whichever you prefer.
|
|
|
|
*
|
|
|
|
* We choose queue_priority such that if we add a request of greater
|
|
|
|
* priority than this, we kick the submission tasklet to decide on
|
|
|
|
* the right order of submitting the requests to hardware. We must
|
|
|
|
* also be prepared to reorder requests as they are in-flight on the
|
|
|
|
* HW. We derive the queue_priority then as the first "hole" in
|
|
|
|
* the HW submission ports and if there are no available slots,
|
|
|
|
* the priority of the lowest executing request, i.e. last.
|
|
|
|
*
|
|
|
|
* When we do receive a higher priority request ready to run from the
|
|
|
|
* user, see queue_request(), the queue_priority is bumped to that
|
|
|
|
* request triggering preemption on the next dequeue (or subsequent
|
|
|
|
* interrupt for secondary ports).
|
|
|
|
*/
|
|
|
|
execlists->queue_priority =
|
|
|
|
port != execlists->port ? rq_prio(last) : INT_MIN;
|
|
|
|
|
2018-06-29 04:12:04 +08:00
|
|
|
if (submit) {
|
2017-05-17 20:10:00 +08:00
|
|
|
port_assign(port, last);
|
2018-06-29 04:12:04 +08:00
|
|
|
execlists_submit_ports(engine);
|
|
|
|
}
|
2018-02-16 00:25:53 +08:00
|
|
|
|
|
|
|
/* We must always keep the beast fed if we have work piled up */
|
2018-06-29 15:53:20 +08:00
|
|
|
GEM_BUG_ON(rb_first_cached(&execlists->queue) &&
|
|
|
|
!port_isset(execlists->port));
|
2018-02-16 00:25:53 +08:00
|
|
|
|
2018-05-09 05:03:17 +08:00
|
|
|
/* Re-evaluate the executing context setup after each preemptive kick */
|
|
|
|
if (last)
|
2018-03-31 21:06:26 +08:00
|
|
|
execlists_user_begin(execlists, execlists->port);
|
2018-05-09 05:03:17 +08:00
|
|
|
|
2018-06-29 04:12:04 +08:00
|
|
|
/* If the engine is now idle, so should be the flag; and vice versa. */
|
|
|
|
GEM_BUG_ON(execlists_is_active(&engine->execlists,
|
|
|
|
EXECLISTS_ACTIVE_USER) ==
|
|
|
|
!port_isset(engine->execlists.port));
|
2018-05-09 05:03:17 +08:00
|
|
|
}
|
|
|
|
|
drm/i915/guc: Preemption! With GuC
Pretty similar to what we have on execlists.
We're reusing most of the GEM code, however, due to GuC quirks we need a
couple of extra bits.
Preemption is implemented as GuC action, and actions can be pretty slow.
Because of that, we're using a mutex to serialize them. Since we're
requesting preemption from the tasklet, the task of creating a workitem
and wrapping it in GuC action is delegated to a worker.
To distinguish that preemption has finished, we're using additional
piece of HWSP, and since we're not getting context switch interrupts,
we're also adding a user interrupt.
The fact that our special preempt context has completed unfortunately
doesn't mean that we're ready to submit new work. We also need to wait
for GuC to finish its own processing.
v2: Don't compile out the wait for GuC, handle workqueue flush on reset,
no need for ordered workqueue, put on a reviewer hat when looking at my own
patches (Chris)
Move struct work around in intel_guc, move user interruput outside of
conditional (Michał)
Keep ring around rather than chase though intel_context
v3: Extract WA for flushing ggtt writes to a helper (Chris)
Keep work_struct in intel_guc rather than engine (Michał)
Use ordered workqueue for inject_preempt worker to avoid GuC quirks.
v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele)
Drop stray newlines, use container_of for intel_guc in worker,
check for presence of workqueue when flushing it, rather than
enable_guc_submission modparam, reorder preempt postprocessing (Chris)
v5: Make wq NULL after destroying it
v6: Swap struct guc_preempt_work members (Michał)
Signed-off-by: Michał Winiarski <michal.winiarski@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Jeff McGee <jeff.mcgee@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Oscar Mateo <oscar.mateo@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 21:35:58 +08:00
|
|
|
void
|
2017-10-26 04:00:18 +08:00
|
|
|
execlists_cancel_port_requests(struct intel_engine_execlists * const execlists)
|
2017-09-22 20:43:05 +08:00
|
|
|
{
|
2017-09-25 20:49:27 +08:00
|
|
|
struct execlist_port *port = execlists->port;
|
2017-10-10 19:48:57 +08:00
|
|
|
unsigned int num_ports = execlists_num_ports(execlists);
|
2017-09-22 20:43:05 +08:00
|
|
|
|
2017-09-25 20:49:27 +08:00
|
|
|
while (num_ports-- && port_isset(port)) {
|
2018-02-21 17:56:36 +08:00
|
|
|
struct i915_request *rq = port_request(port);
|
2017-09-26 18:17:19 +08:00
|
|
|
|
2018-04-06 20:35:14 +08:00
|
|
|
GEM_TRACE("%s:port%u global=%d (fence %llx:%d), (current %d)\n",
|
|
|
|
rq->engine->name,
|
|
|
|
(unsigned int)(port - execlists->port),
|
|
|
|
rq->global_seqno,
|
|
|
|
rq->fence.context, rq->fence.seqno,
|
|
|
|
intel_engine_get_seqno(rq->engine));
|
|
|
|
|
2017-10-24 05:32:36 +08:00
|
|
|
GEM_BUG_ON(!execlists->active);
|
2018-05-03 07:02:02 +08:00
|
|
|
execlists_context_schedule_out(rq,
|
|
|
|
i915_request_completed(rq) ?
|
|
|
|
INTEL_CONTEXT_SCHEDULE_OUT :
|
|
|
|
INTEL_CONTEXT_SCHEDULE_PREEMPTED);
|
2018-03-06 10:15:57 +08:00
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
i915_request_put(rq);
|
2017-09-26 18:17:19 +08:00
|
|
|
|
2017-09-25 20:49:27 +08:00
|
|
|
memset(port, 0, sizeof(*port));
|
|
|
|
port++;
|
|
|
|
}
|
2018-03-24 20:58:29 +08:00
|
|
|
|
2018-07-16 20:54:24 +08:00
|
|
|
execlists_clear_all_active(execlists);
|
2017-09-22 20:43:05 +08:00
|
|
|
}
|
|
|
|
|
2018-06-29 04:12:08 +08:00
|
|
|
static void reset_csb_pointers(struct intel_engine_execlists *execlists)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* After a reset, the HW starts writing into CSB entry [0]. We
|
|
|
|
* therefore have to set our HEAD pointer back one entry so that
|
|
|
|
* the *first* entry we check is entry 0. To complicate this further,
|
|
|
|
* as we don't wait for the first interrupt after reset, we have to
|
|
|
|
* fake the HW write to point back to the last entry so that our
|
|
|
|
* inline comparison of our cached head position against the last HW
|
|
|
|
* write works even before the first interrupt.
|
|
|
|
*/
|
|
|
|
execlists->csb_head = execlists->csb_write_reset;
|
|
|
|
WRITE_ONCE(*execlists->csb_write, execlists->csb_write_reset);
|
|
|
|
}
|
|
|
|
|
2018-07-16 16:03:30 +08:00
|
|
|
static void nop_submission_tasklet(unsigned long data)
|
|
|
|
{
|
|
|
|
/* The driver is wedged; don't process any more events. */
|
|
|
|
}
|
|
|
|
|
2017-09-16 01:31:00 +08:00
|
|
|
static void execlists_cancel_requests(struct intel_engine_cs *engine)
|
|
|
|
{
|
2017-09-22 20:43:03 +08:00
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
2018-02-21 17:56:36 +08:00
|
|
|
struct i915_request *rq, *rn;
|
2017-09-16 01:31:00 +08:00
|
|
|
struct rb_node *rb;
|
|
|
|
unsigned long flags;
|
|
|
|
|
2018-04-06 20:35:14 +08:00
|
|
|
GEM_TRACE("%s current %d\n",
|
|
|
|
engine->name, intel_engine_get_seqno(engine));
|
drm/i915: Suspend submission tasklets around wedging
After staring hard at sequences like
[ 28.199013] systemd-1 2..s. 26062228us : execlists_submission_tasklet: rcs0 cs-irq head=0 [0?], tail=1 [1?]
[ 28.199095] systemd-1 2..s. 26062229us : execlists_submission_tasklet: rcs0 csb[1]: status=0x00000018:0x00000000, active=0x1
[ 28.199177] systemd-1 2..s. 26062230us : execlists_submission_tasklet: rcs0 out[0]: ctx=0.1, seqno=3, prio=-1024
[ 28.199258] systemd-1 2..s. 26062231us : execlists_submission_tasklet: rcs0 completed ctx=0
[ 28.199340] gem_eio-829 1..s1 26066853us : execlists_submission_tasklet: rcs0 in[0]: ctx=1.1, seqno=1, prio=0
[ 28.199421] <idle>-0 2..s. 26066863us : execlists_submission_tasklet: rcs0 cs-irq head=1 [1?], tail=2 [2?]
[ 28.199503] <idle>-0 2..s. 26066865us : execlists_submission_tasklet: rcs0 csb[2]: status=0x00000001:0x00000000, active=0x1
[ 28.199585] gem_eio-829 1..s1 26067077us : execlists_submission_tasklet: rcs0 in[1]: ctx=3.1, seqno=2, prio=0
[ 28.199667] gem_eio-829 1..s1 26067078us : execlists_submission_tasklet: rcs0 in[0]: ctx=1.2, seqno=1, prio=0
[ 28.199749] <idle>-0 2..s. 26067084us : execlists_submission_tasklet: rcs0 cs-irq head=2 [2?], tail=3 [3?]
[ 28.199830] <idle>-0 2..s. 26067085us : execlists_submission_tasklet: rcs0 csb[3]: status=0x00008002:0x00000001, active=0x1
[ 28.199912] <idle>-0 2..s. 26067086us : execlists_submission_tasklet: rcs0 out[0]: ctx=1.2, seqno=1, prio=0
[ 28.199994] gem_eio-829 2..s. 28246084us : execlists_submission_tasklet: rcs0 cs-irq head=3 [3?], tail=4 [4?]
[ 28.200096] gem_eio-829 2..s. 28246088us : execlists_submission_tasklet: rcs0 csb[4]: status=0x00000014:0x00000001, active=0x5
[ 28.200178] gem_eio-829 2..s. 28246089us : execlists_submission_tasklet: rcs0 out[0]: ctx=0.0, seqno=0, prio=0
[ 28.200260] gem_eio-829 2..s. 28246127us : execlists_submission_tasklet: execlists_submission_tasklet:886 GEM_BUG_ON(buf[2 * head + 1] != port->context_id)
the conclusion is that the only place where the ports are reset to zero,
is from engine->cancel_requests called during i915_gem_set_wedged().
The race is horrible as it results from calling set-wedged on active HW
(the GPU reset failed) and as such we need to be careful as the HW state
changes beneath us. Fortunately, it's the same scary conditions as
affect normal reset, so we can reuse the same machinery to disable state
tracking as we clobber it.
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=104945
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Michel Thierry <michel.thierry@intel.com>
Fixes: af7a8ffad9c5 ("drm/i915: Use rcu instead of stop_machine in set_wedged")
Reviewed-by: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180302113324.23189-2-chris@chris-wilson.co.uk
2018-03-02 19:33:24 +08:00
|
|
|
|
2018-03-02 22:32:45 +08:00
|
|
|
/*
|
|
|
|
* Before we call engine->cancel_requests(), we should have exclusive
|
|
|
|
* access to the submission state. This is arranged for us by the
|
|
|
|
* caller disabling the interrupt generation, the tasklet and other
|
|
|
|
* threads that may then access the same state, giving us a free hand
|
|
|
|
* to reset state. However, we still need to let lockdep be aware that
|
|
|
|
* we know this state may be accessed in hardirq context, so we
|
|
|
|
* disable the irq around this manipulation and we want to keep
|
|
|
|
* the spinlock focused on its duties and not accidentally conflate
|
|
|
|
* coverage to the submission's irq state. (Similarly, although we
|
|
|
|
* shouldn't need to disable irq around the manipulation of the
|
|
|
|
* submission's irq state, we also wish to remind ourselves that
|
|
|
|
* it is irq state.)
|
|
|
|
*/
|
2018-06-29 04:12:05 +08:00
|
|
|
spin_lock_irqsave(&engine->timeline.lock, flags);
|
2017-09-16 01:31:00 +08:00
|
|
|
|
|
|
|
/* Cancel the requests on the HW and clear the ELSP tracker. */
|
2017-10-26 04:00:18 +08:00
|
|
|
execlists_cancel_port_requests(execlists);
|
2018-07-16 20:54:24 +08:00
|
|
|
execlists_user_end(execlists);
|
2017-09-16 01:31:00 +08:00
|
|
|
|
|
|
|
/* Mark all executing requests as skipped. */
|
2018-05-03 00:38:39 +08:00
|
|
|
list_for_each_entry(rq, &engine->timeline.requests, link) {
|
2017-09-16 01:31:00 +08:00
|
|
|
GEM_BUG_ON(!rq->global_seqno);
|
2018-02-21 17:56:36 +08:00
|
|
|
if (!i915_request_completed(rq))
|
2017-09-16 01:31:00 +08:00
|
|
|
dma_fence_set_error(&rq->fence, -EIO);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Flush the queued requests to the timeline list (for retiring). */
|
2018-06-29 15:53:20 +08:00
|
|
|
while ((rb = rb_first_cached(&execlists->queue))) {
|
2018-02-22 22:22:29 +08:00
|
|
|
struct i915_priolist *p = to_priolist(rb);
|
2017-09-16 01:31:00 +08:00
|
|
|
|
2018-04-19 02:40:51 +08:00
|
|
|
list_for_each_entry_safe(rq, rn, &p->requests, sched.link) {
|
|
|
|
INIT_LIST_HEAD(&rq->sched.link);
|
2017-09-16 01:31:00 +08:00
|
|
|
|
|
|
|
dma_fence_set_error(&rq->fence, -EIO);
|
2018-02-21 17:56:36 +08:00
|
|
|
__i915_request_submit(rq);
|
2017-09-16 01:31:00 +08:00
|
|
|
}
|
|
|
|
|
2018-06-29 15:53:20 +08:00
|
|
|
rb_erase_cached(&p->node, &execlists->queue);
|
2017-09-16 01:31:00 +08:00
|
|
|
INIT_LIST_HEAD(&p->requests);
|
|
|
|
if (p->priority != I915_PRIORITY_NORMAL)
|
|
|
|
kmem_cache_free(engine->i915->priorities, p);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Remaining _unready_ requests will be nop'ed when submitted */
|
|
|
|
|
2018-02-22 22:22:29 +08:00
|
|
|
execlists->queue_priority = INT_MIN;
|
2018-06-29 15:53:20 +08:00
|
|
|
execlists->queue = RB_ROOT_CACHED;
|
2017-09-25 20:49:27 +08:00
|
|
|
GEM_BUG_ON(port_isset(execlists->port));
|
2017-09-16 01:31:00 +08:00
|
|
|
|
2018-07-16 16:03:30 +08:00
|
|
|
GEM_BUG_ON(__tasklet_is_enabled(&execlists->tasklet));
|
|
|
|
execlists->tasklet.func = nop_submission_tasklet;
|
|
|
|
|
2018-06-29 04:12:05 +08:00
|
|
|
spin_unlock_irqrestore(&engine->timeline.lock, flags);
|
2017-09-16 01:31:00 +08:00
|
|
|
}
|
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
static inline bool
|
|
|
|
reset_in_progress(const struct intel_engine_execlists *execlists)
|
|
|
|
{
|
|
|
|
return unlikely(!__tasklet_is_enabled(&execlists->tasklet));
|
|
|
|
}
|
|
|
|
|
2018-05-17 02:33:53 +08:00
|
|
|
static void process_csb(struct intel_engine_cs *engine)
|
drm/i915/bdw: Handle context switch events
Handle all context status events in the context status buffer on every
context switch interrupt. We only remove work from the execlist queue
after a context status buffer reports that it has completed and we only
attempt to schedule new contexts on interrupt when a previously submitted
context completes (unless no contexts are queued, which means the GPU is
free).
We canot call intel_runtime_pm_get() in an interrupt (or with a spinlock
grabbed, FWIW), because it might sleep, which is not a nice thing to do.
Instead, do the runtime_pm get/put together with the create/destroy request,
and handle the forcewake get/put directly.
Signed-off-by: Thomas Daniel <thomas.daniel@intel.com>
v2: Unreferencing the context when we are freeing the request might free
the backing bo, which requires the struct_mutex to be grabbed, so defer
unreferencing and freeing to a bottom half.
v3:
- Ack the interrupt inmediately, before trying to handle it (fix for
missing interrupts by Bob Beckett <robert.beckett@intel.com>).
- Update the Context Status Buffer Read Pointer, just in case (spotted
by Damien Lespiau).
v4: New namespace and multiple rebase changes.
v5: Squash with "drm/i915/bdw: Do not call intel_runtime_pm_get() in an
interrupt", as suggested by Daniel.
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
[danvet: Checkpatch ...]
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:39 +08:00
|
|
|
{
|
2017-09-22 20:43:03 +08:00
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
2018-03-31 21:06:26 +08:00
|
|
|
struct execlist_port *port = execlists->port;
|
2018-06-29 04:12:07 +08:00
|
|
|
const u32 * const buf = execlists->csb_status;
|
|
|
|
u8 head, tail;
|
2016-02-27 00:58:32 +08:00
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
/*
|
|
|
|
* Note that csb_write, csb_status may be either in HWSP or mmio.
|
|
|
|
* When reading from the csb_write mmio register, we have to be
|
|
|
|
* careful to only use the GEN8_CSB_WRITE_PTR portion, which is
|
|
|
|
* the low 4bits. As it happens we know the next 4bits are always
|
|
|
|
* zero and so we can simply masked off the low u8 of the register
|
|
|
|
* and treat it identically to reading from the HWSP (without having
|
|
|
|
* to use explicit shifting and masking, and probably bifurcating
|
|
|
|
* the code to handle the legacy mmio read).
|
|
|
|
*/
|
|
|
|
head = execlists->csb_head;
|
|
|
|
tail = READ_ONCE(*execlists->csb_write);
|
|
|
|
GEM_TRACE("%s cs-irq head=%d, tail=%d\n", engine->name, head, tail);
|
|
|
|
if (unlikely(head == tail))
|
|
|
|
return;
|
2018-06-15 17:31:37 +08:00
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
/*
|
|
|
|
* Hopefully paired with a wmb() in HW!
|
|
|
|
*
|
|
|
|
* We must complete the read of the write pointer before any reads
|
|
|
|
* from the CSB, so that we do not see stale values. Without an rmb
|
|
|
|
* (lfence) the HW may speculatively perform the CSB[] reads *before*
|
|
|
|
* we perform the READ_ONCE(*csb_write).
|
|
|
|
*/
|
|
|
|
rmb();
|
2017-09-13 16:56:05 +08:00
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
do {
|
2018-06-29 04:12:06 +08:00
|
|
|
struct i915_request *rq;
|
|
|
|
unsigned int status;
|
|
|
|
unsigned int count;
|
|
|
|
|
|
|
|
if (++head == GEN8_CSB_ENTRIES)
|
|
|
|
head = 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We are flying near dragons again.
|
|
|
|
*
|
|
|
|
* We hold a reference to the request in execlist_port[]
|
|
|
|
* but no more than that. We are operating in softirq
|
|
|
|
* context and so cannot hold any mutex or sleep. That
|
|
|
|
* prevents us stopping the requests we are processing
|
|
|
|
* in port[] from being retired simultaneously (the
|
|
|
|
* breadcrumb will be complete before we see the
|
|
|
|
* context-switch). As we only hold the reference to the
|
|
|
|
* request, any pointer chasing underneath the request
|
|
|
|
* is subject to a potential use-after-free. Thus we
|
|
|
|
* store all of the bookkeeping within port[] as
|
|
|
|
* required, and avoid using unguarded pointers beneath
|
|
|
|
* request itself. The same applies to the atomic
|
|
|
|
* status notifier.
|
|
|
|
*/
|
|
|
|
|
|
|
|
GEM_TRACE("%s csb[%d]: status=0x%08x:0x%08x, active=0x%x\n",
|
|
|
|
engine->name, head,
|
2018-06-29 04:12:07 +08:00
|
|
|
buf[2 * head + 0], buf[2 * head + 1],
|
2018-06-29 04:12:06 +08:00
|
|
|
execlists->active);
|
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
status = buf[2 * head];
|
2018-06-29 04:12:06 +08:00
|
|
|
if (status & (GEN8_CTX_STATUS_IDLE_ACTIVE |
|
|
|
|
GEN8_CTX_STATUS_PREEMPTED))
|
|
|
|
execlists_set_active(execlists,
|
|
|
|
EXECLISTS_ACTIVE_HWACK);
|
|
|
|
if (status & GEN8_CTX_STATUS_ACTIVE_IDLE)
|
|
|
|
execlists_clear_active(execlists,
|
|
|
|
EXECLISTS_ACTIVE_HWACK);
|
|
|
|
|
|
|
|
if (!(status & GEN8_CTX_STATUS_COMPLETED_MASK))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/* We should never get a COMPLETED | IDLE_ACTIVE! */
|
|
|
|
GEM_BUG_ON(status & GEN8_CTX_STATUS_IDLE_ACTIVE);
|
|
|
|
|
|
|
|
if (status & GEN8_CTX_STATUS_COMPLETE &&
|
|
|
|
buf[2*head + 1] == execlists->preempt_complete_status) {
|
|
|
|
GEM_TRACE("%s preempt-idle\n", engine->name);
|
|
|
|
complete_preempt_context(execlists);
|
|
|
|
continue;
|
2017-09-13 16:56:05 +08:00
|
|
|
}
|
2017-09-22 20:43:03 +08:00
|
|
|
|
2018-06-29 04:12:06 +08:00
|
|
|
if (status & GEN8_CTX_STATUS_PREEMPTED &&
|
|
|
|
execlists_is_active(execlists,
|
|
|
|
EXECLISTS_ACTIVE_PREEMPT))
|
|
|
|
continue;
|
2017-03-26 04:10:53 +08:00
|
|
|
|
2018-06-29 04:12:06 +08:00
|
|
|
GEM_BUG_ON(!execlists_is_active(execlists,
|
|
|
|
EXECLISTS_ACTIVE_USER));
|
2016-09-09 21:11:46 +08:00
|
|
|
|
2018-06-29 04:12:06 +08:00
|
|
|
rq = port_unpack(port, &count);
|
|
|
|
GEM_TRACE("%s out[0]: ctx=%d.%d, global=%d (fence %llx:%d) (current %d), prio=%d\n",
|
|
|
|
engine->name,
|
|
|
|
port->context_id, count,
|
|
|
|
rq ? rq->global_seqno : 0,
|
|
|
|
rq ? rq->fence.context : 0,
|
|
|
|
rq ? rq->fence.seqno : 0,
|
|
|
|
intel_engine_get_seqno(engine),
|
|
|
|
rq ? rq_prio(rq) : 0);
|
|
|
|
|
|
|
|
/* Check the context/desc id for this event matches */
|
|
|
|
GEM_DEBUG_BUG_ON(buf[2 * head + 1] != port->context_id);
|
|
|
|
|
|
|
|
GEM_BUG_ON(count == 0);
|
|
|
|
if (--count == 0) {
|
2018-05-17 02:33:53 +08:00
|
|
|
/*
|
2018-06-29 04:12:06 +08:00
|
|
|
* On the final event corresponding to the
|
|
|
|
* submission of this context, we expect either
|
|
|
|
* an element-switch event or a completion
|
|
|
|
* event (and on completion, the active-idle
|
|
|
|
* marker). No more preemptions, lite-restore
|
|
|
|
* or otherwise.
|
2017-02-07 01:05:02 +08:00
|
|
|
*/
|
2018-06-29 04:12:06 +08:00
|
|
|
GEM_BUG_ON(status & GEN8_CTX_STATUS_PREEMPTED);
|
|
|
|
GEM_BUG_ON(port_isset(&port[1]) &&
|
|
|
|
!(status & GEN8_CTX_STATUS_ELEMENT_SWITCH));
|
|
|
|
GEM_BUG_ON(!port_isset(&port[1]) &&
|
|
|
|
!(status & GEN8_CTX_STATUS_ACTIVE_IDLE));
|
2017-02-07 01:05:02 +08:00
|
|
|
|
2018-06-29 04:12:06 +08:00
|
|
|
/*
|
|
|
|
* We rely on the hardware being strongly
|
|
|
|
* ordered, that the breadcrumb write is
|
|
|
|
* coherent (visible from the CPU) before the
|
|
|
|
* user interrupt and CSB is processed.
|
|
|
|
*/
|
|
|
|
GEM_BUG_ON(!i915_request_completed(rq));
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
|
2018-06-29 04:12:06 +08:00
|
|
|
execlists_context_schedule_out(rq,
|
|
|
|
INTEL_CONTEXT_SCHEDULE_OUT);
|
|
|
|
i915_request_put(rq);
|
2018-02-21 23:23:01 +08:00
|
|
|
|
2018-06-29 04:12:06 +08:00
|
|
|
GEM_TRACE("%s completed ctx=%d\n",
|
|
|
|
engine->name, port->context_id);
|
2018-02-21 23:23:01 +08:00
|
|
|
|
2018-06-29 04:12:06 +08:00
|
|
|
port = execlists_port_complete(execlists, port);
|
|
|
|
if (port_isset(port))
|
|
|
|
execlists_user_begin(execlists, port);
|
|
|
|
else
|
|
|
|
execlists_user_end(execlists);
|
|
|
|
} else {
|
|
|
|
port_set(port, port_pack(rq, count));
|
2017-03-26 04:10:53 +08:00
|
|
|
}
|
2018-06-29 04:12:07 +08:00
|
|
|
} while (head != tail);
|
drm/i915/bdw: Handle context switch events
Handle all context status events in the context status buffer on every
context switch interrupt. We only remove work from the execlist queue
after a context status buffer reports that it has completed and we only
attempt to schedule new contexts on interrupt when a previously submitted
context completes (unless no contexts are queued, which means the GPU is
free).
We canot call intel_runtime_pm_get() in an interrupt (or with a spinlock
grabbed, FWIW), because it might sleep, which is not a nice thing to do.
Instead, do the runtime_pm get/put together with the create/destroy request,
and handle the forcewake get/put directly.
Signed-off-by: Thomas Daniel <thomas.daniel@intel.com>
v2: Unreferencing the context when we are freeing the request might free
the backing bo, which requires the struct_mutex to be grabbed, so defer
unreferencing and freeing to a bottom half.
v3:
- Ack the interrupt inmediately, before trying to handle it (fix for
missing interrupts by Bob Beckett <robert.beckett@intel.com>).
- Update the Context Status Buffer Read Pointer, just in case (spotted
by Damien Lespiau).
v4: New namespace and multiple rebase changes.
v5: Squash with "drm/i915/bdw: Do not call intel_runtime_pm_get() in an
interrupt", as suggested by Daniel.
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
[danvet: Checkpatch ...]
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:39 +08:00
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
execlists->csb_head = head;
|
2018-05-17 02:33:53 +08:00
|
|
|
}
|
2016-02-27 00:58:32 +08:00
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
static void __execlists_submission_tasklet(struct intel_engine_cs *const engine)
|
2018-05-17 02:33:53 +08:00
|
|
|
{
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
lockdep_assert_held(&engine->timeline.lock);
|
2018-05-17 02:33:53 +08:00
|
|
|
|
2018-06-29 04:12:10 +08:00
|
|
|
process_csb(engine);
|
2018-05-17 02:33:53 +08:00
|
|
|
if (!execlists_is_active(&engine->execlists, EXECLISTS_ACTIVE_PREEMPT))
|
|
|
|
execlists_dequeue(engine);
|
drm/i915/bdw: Handle context switch events
Handle all context status events in the context status buffer on every
context switch interrupt. We only remove work from the execlist queue
after a context status buffer reports that it has completed and we only
attempt to schedule new contexts on interrupt when a previously submitted
context completes (unless no contexts are queued, which means the GPU is
free).
We canot call intel_runtime_pm_get() in an interrupt (or with a spinlock
grabbed, FWIW), because it might sleep, which is not a nice thing to do.
Instead, do the runtime_pm get/put together with the create/destroy request,
and handle the forcewake get/put directly.
Signed-off-by: Thomas Daniel <thomas.daniel@intel.com>
v2: Unreferencing the context when we are freeing the request might free
the backing bo, which requires the struct_mutex to be grabbed, so defer
unreferencing and freeing to a bottom half.
v3:
- Ack the interrupt inmediately, before trying to handle it (fix for
missing interrupts by Bob Beckett <robert.beckett@intel.com>).
- Update the Context Status Buffer Read Pointer, just in case (spotted
by Damien Lespiau).
v4: New namespace and multiple rebase changes.
v5: Squash with "drm/i915/bdw: Do not call intel_runtime_pm_get() in an
interrupt", as suggested by Daniel.
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
[danvet: Checkpatch ...]
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:39 +08:00
|
|
|
}
|
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
/*
|
|
|
|
* Check the unread Context Status Buffers and manage the submission of new
|
|
|
|
* contexts to the ELSP accordingly.
|
|
|
|
*/
|
|
|
|
static void execlists_submission_tasklet(unsigned long data)
|
|
|
|
{
|
|
|
|
struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
GEM_TRACE("%s awake?=%d, active=%x\n",
|
|
|
|
engine->name,
|
|
|
|
engine->i915->gt.awake,
|
|
|
|
engine->execlists.active);
|
|
|
|
|
|
|
|
spin_lock_irqsave(&engine->timeline.lock, flags);
|
2018-07-19 15:50:29 +08:00
|
|
|
__execlists_submission_tasklet(engine);
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
spin_unlock_irqrestore(&engine->timeline.lock, flags);
|
|
|
|
}
|
|
|
|
|
2018-02-22 22:22:29 +08:00
|
|
|
static void queue_request(struct intel_engine_cs *engine,
|
2018-04-19 02:40:51 +08:00
|
|
|
struct i915_sched_node *node,
|
2018-02-22 22:22:29 +08:00
|
|
|
int prio)
|
2017-09-17 04:44:13 +08:00
|
|
|
{
|
2018-04-19 02:40:51 +08:00
|
|
|
list_add_tail(&node->link,
|
2018-05-08 08:30:45 +08:00
|
|
|
&lookup_priolist(engine, prio)->requests);
|
2018-02-22 22:22:29 +08:00
|
|
|
}
|
2017-09-17 04:44:13 +08:00
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
static void __update_queue(struct intel_engine_cs *engine, int prio)
|
2018-03-26 19:50:34 +08:00
|
|
|
{
|
|
|
|
engine->execlists.queue_priority = prio;
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void __submit_queue_imm(struct intel_engine_cs *engine)
|
|
|
|
{
|
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
|
|
|
|
|
|
|
if (reset_in_progress(execlists))
|
|
|
|
return; /* defer until we restart the engine following reset */
|
|
|
|
|
|
|
|
if (execlists->tasklet.func == execlists_submission_tasklet)
|
|
|
|
__execlists_submission_tasklet(engine);
|
|
|
|
else
|
|
|
|
tasklet_hi_schedule(&execlists->tasklet);
|
2018-03-26 19:50:34 +08:00
|
|
|
}
|
|
|
|
|
2018-02-22 22:22:29 +08:00
|
|
|
static void submit_queue(struct intel_engine_cs *engine, int prio)
|
|
|
|
{
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
if (prio > engine->execlists.queue_priority) {
|
|
|
|
__update_queue(engine, prio);
|
|
|
|
__submit_queue_imm(engine);
|
|
|
|
}
|
2017-09-17 04:44:13 +08:00
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static void execlists_submit_request(struct i915_request *request)
|
2014-07-25 00:04:38 +08:00
|
|
|
{
|
2016-03-16 19:00:38 +08:00
|
|
|
struct intel_engine_cs *engine = request->engine;
|
2016-09-09 21:11:54 +08:00
|
|
|
unsigned long flags;
|
2014-07-25 00:04:38 +08:00
|
|
|
|
2016-11-15 04:41:00 +08:00
|
|
|
/* Will be called from irq-context when using foreign fences. */
|
2018-05-03 00:38:39 +08:00
|
|
|
spin_lock_irqsave(&engine->timeline.lock, flags);
|
2014-07-25 00:04:38 +08:00
|
|
|
|
2018-04-19 02:40:51 +08:00
|
|
|
queue_request(engine, &request->sched, rq_prio(request));
|
2014-07-25 00:04:38 +08:00
|
|
|
|
2018-06-29 15:53:20 +08:00
|
|
|
GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
|
2018-04-19 02:40:51 +08:00
|
|
|
GEM_BUG_ON(list_empty(&request->sched.link));
|
drm/i915: Split execlist priority queue into rbtree + linked list
All the requests at the same priority are executed in FIFO order. They
do not need to be stored in the rbtree themselves, as they are a simple
list within a level. If we move the requests at one priority into a list,
we can then reduce the rbtree to the set of priorities. This should keep
the height of the rbtree small, as the number of active priorities can not
exceed the number of active requests and should be typically only a few.
Currently, we have ~2k possible different priority levels, that may
increase to allow even more fine grained selection. Allocating those in
advance seems a waste (and may be impossible), so we opt for allocating
upon first use, and freeing after its requests are depleted. To avoid
the possibility of an allocation failure causing us to lose a request,
we preallocate the default priority (0) and bump any request to that
priority if we fail to allocate it the appropriate plist. Having a
request (that is ready to run, so not leading to corruption) execute
out-of-order is better than leaking the request (and its dependency
tree) entirely.
There should be a benefit to reducing execlists_dequeue() to principally
using a simple list (and reducing the frequency of both rbtree iteration
and balancing on erase) but for typical workloads, request coalescing
should be small enough that we don't notice any change. The main gain is
from improving PI calls to schedule, and the explicit list within a
level should make request unwinding simpler (we just need to insert at
the head of the list rather than the tail and not have to make the
rbtree search more complicated).
v2: Avoid use-after-free when deleting a depleted priolist
v3: Michał found the solution to handling the allocation failure
gracefully. If we disable all priority scheduling following the
allocation failure, those requests will be executed in fifo and we will
ensure that this request and its dependencies are in strict fifo (even
when it doesn't realise it is only a single list). Normal scheduling is
restored once we know the device is idle, until the next failure!
Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 20:10:03 +08:00
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
submit_queue(engine, rq_prio(request));
|
|
|
|
|
2018-05-03 00:38:39 +08:00
|
|
|
spin_unlock_irqrestore(&engine->timeline.lock, flags);
|
2014-07-25 00:04:38 +08:00
|
|
|
}
|
|
|
|
|
2018-04-19 02:40:51 +08:00
|
|
|
static struct i915_request *sched_to_request(struct i915_sched_node *node)
|
2017-10-04 04:34:50 +08:00
|
|
|
{
|
2018-04-19 02:40:51 +08:00
|
|
|
return container_of(node, struct i915_request, sched);
|
2017-10-04 04:34:50 +08:00
|
|
|
}
|
|
|
|
|
2016-11-15 04:41:03 +08:00
|
|
|
static struct intel_engine_cs *
|
2018-04-19 02:40:51 +08:00
|
|
|
sched_lock_engine(struct i915_sched_node *node, struct intel_engine_cs *locked)
|
2016-11-15 04:41:03 +08:00
|
|
|
{
|
2018-04-19 02:40:51 +08:00
|
|
|
struct intel_engine_cs *engine = sched_to_request(node)->engine;
|
2017-03-28 04:21:43 +08:00
|
|
|
|
|
|
|
GEM_BUG_ON(!locked);
|
2016-11-15 04:41:03 +08:00
|
|
|
|
|
|
|
if (engine != locked) {
|
2018-05-03 00:38:39 +08:00
|
|
|
spin_unlock(&locked->timeline.lock);
|
|
|
|
spin_lock(&engine->timeline.lock);
|
2016-11-15 04:41:03 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return engine;
|
|
|
|
}
|
|
|
|
|
2018-04-19 02:40:52 +08:00
|
|
|
static void execlists_schedule(struct i915_request *request,
|
|
|
|
const struct i915_sched_attr *attr)
|
2016-11-15 04:41:03 +08:00
|
|
|
{
|
2018-05-08 08:30:46 +08:00
|
|
|
struct i915_priolist *uninitialized_var(pl);
|
|
|
|
struct intel_engine_cs *engine, *last;
|
2016-11-15 04:41:03 +08:00
|
|
|
struct i915_dependency *dep, *p;
|
|
|
|
struct i915_dependency stack;
|
2018-04-19 02:40:52 +08:00
|
|
|
const int prio = attr->priority;
|
2016-11-15 04:41:03 +08:00
|
|
|
LIST_HEAD(dfs);
|
|
|
|
|
2017-09-29 03:39:00 +08:00
|
|
|
GEM_BUG_ON(prio == I915_PRIORITY_INVALID);
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
if (i915_request_completed(request))
|
2018-01-06 18:56:18 +08:00
|
|
|
return;
|
|
|
|
|
2018-04-19 02:40:52 +08:00
|
|
|
if (prio <= READ_ONCE(request->sched.attr.priority))
|
2016-11-15 04:41:03 +08:00
|
|
|
return;
|
|
|
|
|
2016-11-28 22:36:49 +08:00
|
|
|
/* Need BKL in order to use the temporary link inside i915_dependency */
|
|
|
|
lockdep_assert_held(&request->i915->drm.struct_mutex);
|
2016-11-15 04:41:03 +08:00
|
|
|
|
2018-04-19 02:40:51 +08:00
|
|
|
stack.signaler = &request->sched;
|
2016-11-15 04:41:03 +08:00
|
|
|
list_add(&stack.dfs_link, &dfs);
|
|
|
|
|
2018-01-02 23:12:26 +08:00
|
|
|
/*
|
|
|
|
* Recursively bump all dependent priorities to match the new request.
|
2016-11-15 04:41:03 +08:00
|
|
|
*
|
|
|
|
* A naive approach would be to use recursion:
|
2018-04-19 02:40:51 +08:00
|
|
|
* static void update_priorities(struct i915_sched_node *node, prio) {
|
|
|
|
* list_for_each_entry(dep, &node->signalers_list, signal_link)
|
2016-11-15 04:41:03 +08:00
|
|
|
* update_priorities(dep->signal, prio)
|
2018-04-19 02:40:51 +08:00
|
|
|
* queue_request(node);
|
2016-11-15 04:41:03 +08:00
|
|
|
* }
|
|
|
|
* but that may have unlimited recursion depth and so runs a very
|
|
|
|
* real risk of overunning the kernel stack. Instead, we build
|
|
|
|
* a flat list of all dependencies starting with the current request.
|
|
|
|
* As we walk the list of dependencies, we add all of its dependencies
|
|
|
|
* to the end of the list (this may include an already visited
|
|
|
|
* request) and continue to walk onwards onto the new dependencies. The
|
|
|
|
* end result is a topological list of requests in reverse order, the
|
|
|
|
* last element in the list is the request we must execute first.
|
|
|
|
*/
|
2018-01-02 23:12:27 +08:00
|
|
|
list_for_each_entry(dep, &dfs, dfs_link) {
|
2018-04-19 02:40:51 +08:00
|
|
|
struct i915_sched_node *node = dep->signaler;
|
2016-11-15 04:41:03 +08:00
|
|
|
|
2018-01-02 23:12:26 +08:00
|
|
|
/*
|
|
|
|
* Within an engine, there can be no cycle, but we may
|
2017-03-28 04:21:43 +08:00
|
|
|
* refer to the same dependency chain multiple times
|
|
|
|
* (redundant dependencies are not eliminated) and across
|
|
|
|
* engines.
|
|
|
|
*/
|
2018-04-19 02:40:51 +08:00
|
|
|
list_for_each_entry(p, &node->signalers_list, signal_link) {
|
2018-01-02 23:12:26 +08:00
|
|
|
GEM_BUG_ON(p == dep); /* no cycles! */
|
|
|
|
|
2018-04-19 02:40:51 +08:00
|
|
|
if (i915_sched_node_signaled(p->signaler))
|
2017-10-04 04:34:50 +08:00
|
|
|
continue;
|
|
|
|
|
2018-04-19 02:40:52 +08:00
|
|
|
GEM_BUG_ON(p->signaler->attr.priority < node->attr.priority);
|
|
|
|
if (prio > READ_ONCE(p->signaler->attr.priority))
|
2016-11-15 04:41:03 +08:00
|
|
|
list_move_tail(&p->dfs_link, &dfs);
|
2017-03-28 04:21:43 +08:00
|
|
|
}
|
2016-11-15 04:41:03 +08:00
|
|
|
}
|
|
|
|
|
2018-01-02 23:12:26 +08:00
|
|
|
/*
|
|
|
|
* If we didn't need to bump any existing priorities, and we haven't
|
2017-05-17 20:10:05 +08:00
|
|
|
* yet submitted this request (i.e. there is no potential race with
|
|
|
|
* execlists_submit_request()), we can set our own priority and skip
|
|
|
|
* acquiring the engine locks.
|
|
|
|
*/
|
2018-04-19 02:40:52 +08:00
|
|
|
if (request->sched.attr.priority == I915_PRIORITY_INVALID) {
|
2018-04-19 02:40:51 +08:00
|
|
|
GEM_BUG_ON(!list_empty(&request->sched.link));
|
2018-04-19 02:40:52 +08:00
|
|
|
request->sched.attr = *attr;
|
2017-05-17 20:10:05 +08:00
|
|
|
if (stack.dfs_link.next == stack.dfs_link.prev)
|
|
|
|
return;
|
|
|
|
__list_del_entry(&stack.dfs_link);
|
|
|
|
}
|
|
|
|
|
2018-05-08 08:30:46 +08:00
|
|
|
last = NULL;
|
2017-03-28 04:21:43 +08:00
|
|
|
engine = request->engine;
|
2018-05-03 00:38:39 +08:00
|
|
|
spin_lock_irq(&engine->timeline.lock);
|
2017-03-28 04:21:43 +08:00
|
|
|
|
2016-11-15 04:41:03 +08:00
|
|
|
/* Fifo and depth-first replacement ensure our deps execute before us */
|
|
|
|
list_for_each_entry_safe_reverse(dep, p, &dfs, dfs_link) {
|
2018-04-19 02:40:51 +08:00
|
|
|
struct i915_sched_node *node = dep->signaler;
|
2016-11-15 04:41:03 +08:00
|
|
|
|
|
|
|
INIT_LIST_HEAD(&dep->dfs_link);
|
|
|
|
|
2018-04-19 02:40:51 +08:00
|
|
|
engine = sched_lock_engine(node, engine);
|
2016-11-15 04:41:03 +08:00
|
|
|
|
2018-04-19 02:40:52 +08:00
|
|
|
if (prio <= node->attr.priority)
|
2016-11-15 04:41:03 +08:00
|
|
|
continue;
|
|
|
|
|
2018-04-19 02:40:52 +08:00
|
|
|
node->attr.priority = prio;
|
2018-04-19 02:40:51 +08:00
|
|
|
if (!list_empty(&node->link)) {
|
2018-05-08 08:30:46 +08:00
|
|
|
if (last != engine) {
|
|
|
|
pl = lookup_priolist(engine, prio);
|
|
|
|
last = engine;
|
|
|
|
}
|
|
|
|
GEM_BUG_ON(pl->priority != prio);
|
|
|
|
list_move_tail(&node->link, &pl->requests);
|
2017-03-28 04:21:43 +08:00
|
|
|
}
|
2018-03-26 19:50:34 +08:00
|
|
|
|
|
|
|
if (prio > engine->execlists.queue_priority &&
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
i915_sw_fence_done(&sched_to_request(node)->submit)) {
|
|
|
|
/* defer submission until after all of our updates */
|
|
|
|
__update_queue(engine, prio);
|
|
|
|
tasklet_hi_schedule(&engine->execlists.tasklet);
|
|
|
|
}
|
2016-11-15 04:41:03 +08:00
|
|
|
}
|
|
|
|
|
2018-05-03 00:38:39 +08:00
|
|
|
spin_unlock_irq(&engine->timeline.lock);
|
2016-11-15 04:41:03 +08:00
|
|
|
}
|
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
static void execlists_context_destroy(struct intel_context *ce)
|
|
|
|
{
|
|
|
|
GEM_BUG_ON(ce->pin_count);
|
|
|
|
|
2018-06-25 18:06:03 +08:00
|
|
|
if (!ce->state)
|
|
|
|
return;
|
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
intel_ring_free(ce->ring);
|
2018-06-25 18:06:04 +08:00
|
|
|
|
|
|
|
GEM_BUG_ON(i915_gem_object_is_active(ce->state->obj));
|
|
|
|
i915_gem_object_put(ce->state->obj);
|
2018-05-18 05:26:32 +08:00
|
|
|
}
|
|
|
|
|
2018-05-18 05:26:33 +08:00
|
|
|
static void execlists_context_unpin(struct intel_context *ce)
|
2018-05-18 05:26:32 +08:00
|
|
|
{
|
|
|
|
intel_ring_unpin(ce->ring);
|
|
|
|
|
|
|
|
ce->state->obj->pin_global--;
|
|
|
|
i915_gem_object_unpin_map(ce->state->obj);
|
|
|
|
i915_vma_unpin(ce->state);
|
|
|
|
|
|
|
|
i915_gem_context_put(ce->gem_context);
|
|
|
|
}
|
|
|
|
|
2017-11-10 22:26:32 +08:00
|
|
|
static int __context_pin(struct i915_gem_context *ctx, struct i915_vma *vma)
|
|
|
|
{
|
|
|
|
unsigned int flags;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Clear this page out of any CPU caches for coherent swap-in/out.
|
|
|
|
* We only want to do this on the first bind so that we do not stall
|
|
|
|
* on an active context (which by nature is already on the GPU).
|
|
|
|
*/
|
|
|
|
if (!(vma->flags & I915_VMA_GLOBAL_BIND)) {
|
|
|
|
err = i915_gem_object_set_to_gtt_domain(vma->obj, true);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
flags = PIN_GLOBAL | PIN_HIGH;
|
|
|
|
if (ctx->ggtt_offset_bias)
|
|
|
|
flags |= PIN_OFFSET_BIAS | ctx->ggtt_offset_bias;
|
|
|
|
|
|
|
|
return i915_vma_pin(vma, 0, GEN8_LR_CONTEXT_ALIGN, flags);
|
|
|
|
}
|
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
static struct intel_context *
|
|
|
|
__execlists_context_pin(struct intel_engine_cs *engine,
|
|
|
|
struct i915_gem_context *ctx,
|
|
|
|
struct intel_context *ce)
|
drm/i915/bdw: Pin the context backing objects to GGTT on-demand
Up until now, we have pinned every logical ring context backing object
during creation, and left it pinned until destruction. This made my life
easier, but it's a harmful thing to do, because we cause fragmentation
of the GGTT (and, eventually, we would run out of space).
This patch makes the pinning on-demand: the backing objects of the two
contexts that are written to the ELSP are pinned right before submission
and unpinned once the hardware is done with them. The only context that
is still pinned regardless is the global default one, so that the HWS can
still be accessed in the same way (ring->status_page).
v2: In the early version of this patch, we were pinning the context as
we put it into the ELSP: on the one hand, this is very efficient because
only a maximum two contexts are pinned at any given time, but on the other
hand, we cannot really pin in interrupt time :(
v3: Use a mutex rather than atomic_t to protect pin count to avoid races.
Do not unpin default context in free_request.
v4: Break out pin and unpin into functions. Fix style problems reported
by checkpatch
v5: Remove unpin_lock as all pinning and unpinning is done with the struct
mutex already locked. Add WARN_ONs to make sure this is the case in future.
Issue: VIZ-4277
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Signed-off-by: Thomas Daniel <thomas.daniel@intel.com>
Reviewed-by: Akash Goel <akash.goels@gmail.com>
Reviewed-by: Deepak S<deepak.s@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-11-13 18:28:10 +08:00
|
|
|
{
|
2016-04-12 22:40:42 +08:00
|
|
|
void *vaddr;
|
2016-01-15 23:10:27 +08:00
|
|
|
int ret;
|
drm/i915/bdw: Pin the context backing objects to GGTT on-demand
Up until now, we have pinned every logical ring context backing object
during creation, and left it pinned until destruction. This made my life
easier, but it's a harmful thing to do, because we cause fragmentation
of the GGTT (and, eventually, we would run out of space).
This patch makes the pinning on-demand: the backing objects of the two
contexts that are written to the ELSP are pinned right before submission
and unpinned once the hardware is done with them. The only context that
is still pinned regardless is the global default one, so that the HWS can
still be accessed in the same way (ring->status_page).
v2: In the early version of this patch, we were pinning the context as
we put it into the ELSP: on the one hand, this is very efficient because
only a maximum two contexts are pinned at any given time, but on the other
hand, we cannot really pin in interrupt time :(
v3: Use a mutex rather than atomic_t to protect pin count to avoid races.
Do not unpin default context in free_request.
v4: Break out pin and unpin into functions. Fix style problems reported
by checkpatch
v5: Remove unpin_lock as all pinning and unpinning is done with the struct
mutex already locked. Add WARN_ONs to make sure this is the case in future.
Issue: VIZ-4277
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Signed-off-by: Thomas Daniel <thomas.daniel@intel.com>
Reviewed-by: Akash Goel <akash.goels@gmail.com>
Reviewed-by: Deepak S<deepak.s@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-11-13 18:28:10 +08:00
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
ret = execlists_context_deferred_alloc(ctx, engine, ce);
|
2018-01-26 20:18:46 +08:00
|
|
|
if (ret)
|
|
|
|
goto err;
|
2017-01-05 23:30:20 +08:00
|
|
|
GEM_BUG_ON(!ce->state);
|
drm/i915: Unify active context tracking between legacy/execlists/guc
The requests conversion introduced a nasty bug where we could generate a
new request in the middle of constructing a request if we needed to idle
the system in order to evict space for a context. The request to idle
would be executed (and waited upon) before the current one, creating a
minor havoc in the seqno accounting, as we will consider the current
request to already be completed (prior to deferred seqno assignment) but
ring->last_retired_head would have been updated and still could allow
us to overwrite the current request before execution.
We also employed two different mechanisms to track the active context
until it was switched out. The legacy method allowed for waiting upon an
active context (it could forcibly evict any vma, including context's),
but the execlists method took a step backwards by pinning the vma for
the entire active lifespan of the context (the only way to evict was to
idle the entire GPU, not individual contexts). However, to circumvent
the tricky issue of locking (i.e. we cannot take struct_mutex at the
time of i915_gem_request_submit(), where we would want to move the
previous context onto the active tracker and unpin it), we take the
execlists approach and keep the contexts pinned until retirement.
The benefit of the execlists approach, more important for execlists than
legacy, was the reduction in work in pinning the context for each
request - as the context was kept pinned until idle, it could short
circuit the pinning for all active contexts.
We introduce new engine vfuncs to pin and unpin the context
respectively. The context is pinned at the start of the request, and
only unpinned when the following request is retired (this ensures that
the context is idle and coherent in main memory before we unpin it). We
move the engine->last_context tracking into the retirement itself
(rather than during request submission) in order to allow the submission
to be reordered or unwound without undue difficultly.
And finally an ulterior motive for unifying context handling was to
prepare for mock requests.
v2: Rename to last_retired_context, split out legacy_context tracking
for MI_SET_CONTEXT.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161218153724.8439-3-chris@chris-wilson.co.uk
2016-12-18 23:37:20 +08:00
|
|
|
|
2017-11-10 22:26:32 +08:00
|
|
|
ret = __context_pin(ctx, ce->state);
|
2015-09-11 19:53:46 +08:00
|
|
|
if (ret)
|
2016-04-28 16:56:53 +08:00
|
|
|
goto err;
|
2014-11-13 18:28:56 +08:00
|
|
|
|
2016-08-15 17:48:54 +08:00
|
|
|
vaddr = i915_gem_object_pin_map(ce->state->obj, I915_MAP_WB);
|
2016-04-12 22:40:42 +08:00
|
|
|
if (IS_ERR(vaddr)) {
|
|
|
|
ret = PTR_ERR(vaddr);
|
2016-08-15 17:48:54 +08:00
|
|
|
goto unpin_vma;
|
2016-01-16 01:12:45 +08:00
|
|
|
}
|
|
|
|
|
2017-04-03 19:34:25 +08:00
|
|
|
ret = intel_ring_pin(ce->ring, ctx->i915, ctx->ggtt_offset_bias);
|
2015-09-11 19:53:46 +08:00
|
|
|
if (ret)
|
2016-04-12 22:40:42 +08:00
|
|
|
goto unpin_map;
|
drm/i915: Integrate GuC-based command submission
GuC-based submission is mostly the same as execlist mode, up to
intel_logical_ring_advance_and_submit(), where the context being
dispatched would be added to the execlist queue; at this point
we submit the context to the GuC backend instead.
There are, however, a few other changes also required, notably:
1. Contexts must be pinned at GGTT addresses accessible by the GuC
i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the
PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls.
2. The GuC's TLB must be invalidated after a context is pinned at
a new GGTT address.
3. GuC firmware uses the one page before Ring Context as shared data.
Therefore, whenever driver wants to get base address of LRC, we
will offset one page for it. LRC_PPHWSP_PN is defined as the page
number of LRCA.
4. In the work queue used to pass requests to the GuC, the GuC
firmware requires the ring-tail-offset to be represented as an
11-bit value, expressed in QWords. Therefore, the ringbuffer
size must be reduced to the representable range (4 pages).
v2:
Defer adding #defines until needed [Chris Wilson]
Rationalise type declarations [Chris Wilson]
v4:
Squashed kerneldoc patch into here [Daniel Vetter]
v5:
Update request->tail in code common to both GuC and execlist modes.
Add a private version of lr_context_update(), as sharing the
execlist version leads to race conditions when the CPU and
the GuC both update TAIL in the context image.
Conversion of error-captured HWS page to string must account
for offset from start of object to actual HWS (LRC_PPHWSP_PN).
Issue: VIZ-4884
Signed-off-by: Alex Dai <yu.dai@intel.com>
Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 22:43:43 +08:00
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
intel_lr_context_descriptor_update(ctx, engine, ce);
|
2016-05-24 21:53:37 +08:00
|
|
|
|
2016-10-05 04:11:26 +08:00
|
|
|
ce->lrc_reg_state = vaddr + LRC_STATE_PN * PAGE_SIZE;
|
|
|
|
ce->lrc_reg_state[CTX_RING_BUFFER_START+1] =
|
2016-08-15 17:49:07 +08:00
|
|
|
i915_ggtt_offset(ce->ring->vma);
|
2018-06-11 19:08:45 +08:00
|
|
|
GEM_BUG_ON(!intel_ring_offset_valid(ce->ring, ce->ring->head));
|
2018-03-28 05:01:36 +08:00
|
|
|
ce->lrc_reg_state[CTX_RING_HEAD+1] = ce->ring->head;
|
2016-10-05 04:11:26 +08:00
|
|
|
|
2017-10-14 04:26:16 +08:00
|
|
|
ce->state->obj->pin_global++;
|
2016-07-20 20:31:50 +08:00
|
|
|
i915_gem_context_get(ctx);
|
2018-05-18 05:26:32 +08:00
|
|
|
return ce;
|
2014-11-13 18:28:56 +08:00
|
|
|
|
2016-04-12 22:40:42 +08:00
|
|
|
unpin_map:
|
2016-08-15 17:48:54 +08:00
|
|
|
i915_gem_object_unpin_map(ce->state->obj);
|
|
|
|
unpin_vma:
|
|
|
|
__i915_vma_unpin(ce->state);
|
2016-04-28 16:56:53 +08:00
|
|
|
err:
|
2016-05-24 21:53:37 +08:00
|
|
|
ce->pin_count = 0;
|
2017-05-04 17:33:08 +08:00
|
|
|
return ERR_PTR(ret);
|
2015-09-11 19:53:46 +08:00
|
|
|
}
|
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
static const struct intel_context_ops execlists_context_ops = {
|
|
|
|
.unpin = execlists_context_unpin,
|
|
|
|
.destroy = execlists_context_destroy,
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct intel_context *
|
|
|
|
execlists_context_pin(struct intel_engine_cs *engine,
|
|
|
|
struct i915_gem_context *ctx)
|
2015-09-11 19:53:46 +08:00
|
|
|
{
|
2018-04-30 21:15:01 +08:00
|
|
|
struct intel_context *ce = to_intel_context(ctx, engine);
|
2015-09-11 19:53:46 +08:00
|
|
|
|
2016-07-05 17:40:23 +08:00
|
|
|
lockdep_assert_held(&ctx->i915->drm.struct_mutex);
|
2016-01-28 18:29:55 +08:00
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
if (likely(ce->pin_count++))
|
|
|
|
return ce;
|
|
|
|
GEM_BUG_ON(!ce->pin_count); /* no overflow please! */
|
drm/i915/bdw: Pin the context backing objects to GGTT on-demand
Up until now, we have pinned every logical ring context backing object
during creation, and left it pinned until destruction. This made my life
easier, but it's a harmful thing to do, because we cause fragmentation
of the GGTT (and, eventually, we would run out of space).
This patch makes the pinning on-demand: the backing objects of the two
contexts that are written to the ELSP are pinned right before submission
and unpinned once the hardware is done with them. The only context that
is still pinned regardless is the global default one, so that the HWS can
still be accessed in the same way (ring->status_page).
v2: In the early version of this patch, we were pinning the context as
we put it into the ELSP: on the one hand, this is very efficient because
only a maximum two contexts are pinned at any given time, but on the other
hand, we cannot really pin in interrupt time :(
v3: Use a mutex rather than atomic_t to protect pin count to avoid races.
Do not unpin default context in free_request.
v4: Break out pin and unpin into functions. Fix style problems reported
by checkpatch
v5: Remove unpin_lock as all pinning and unpinning is done with the struct
mutex already locked. Add WARN_ONs to make sure this is the case in future.
Issue: VIZ-4277
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Signed-off-by: Thomas Daniel <thomas.daniel@intel.com>
Reviewed-by: Akash Goel <akash.goels@gmail.com>
Reviewed-by: Deepak S<deepak.s@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-11-13 18:28:10 +08:00
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
ce->ops = &execlists_context_ops;
|
2016-01-28 18:29:55 +08:00
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
return __execlists_context_pin(engine, ctx, ce);
|
drm/i915/bdw: Pin the context backing objects to GGTT on-demand
Up until now, we have pinned every logical ring context backing object
during creation, and left it pinned until destruction. This made my life
easier, but it's a harmful thing to do, because we cause fragmentation
of the GGTT (and, eventually, we would run out of space).
This patch makes the pinning on-demand: the backing objects of the two
contexts that are written to the ELSP are pinned right before submission
and unpinned once the hardware is done with them. The only context that
is still pinned regardless is the global default one, so that the HWS can
still be accessed in the same way (ring->status_page).
v2: In the early version of this patch, we were pinning the context as
we put it into the ELSP: on the one hand, this is very efficient because
only a maximum two contexts are pinned at any given time, but on the other
hand, we cannot really pin in interrupt time :(
v3: Use a mutex rather than atomic_t to protect pin count to avoid races.
Do not unpin default context in free_request.
v4: Break out pin and unpin into functions. Fix style problems reported
by checkpatch
v5: Remove unpin_lock as all pinning and unpinning is done with the struct
mutex already locked. Add WARN_ONs to make sure this is the case in future.
Issue: VIZ-4277
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Signed-off-by: Thomas Daniel <thomas.daniel@intel.com>
Reviewed-by: Akash Goel <akash.goels@gmail.com>
Reviewed-by: Deepak S<deepak.s@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-11-13 18:28:10 +08:00
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static int execlists_request_alloc(struct i915_request *request)
|
2016-12-18 23:37:19 +08:00
|
|
|
{
|
2017-11-15 23:12:04 +08:00
|
|
|
int ret;
|
2016-12-18 23:37:19 +08:00
|
|
|
|
2018-05-18 05:26:32 +08:00
|
|
|
GEM_BUG_ON(!request->hw_context->pin_count);
|
drm/i915: Unify active context tracking between legacy/execlists/guc
The requests conversion introduced a nasty bug where we could generate a
new request in the middle of constructing a request if we needed to idle
the system in order to evict space for a context. The request to idle
would be executed (and waited upon) before the current one, creating a
minor havoc in the seqno accounting, as we will consider the current
request to already be completed (prior to deferred seqno assignment) but
ring->last_retired_head would have been updated and still could allow
us to overwrite the current request before execution.
We also employed two different mechanisms to track the active context
until it was switched out. The legacy method allowed for waiting upon an
active context (it could forcibly evict any vma, including context's),
but the execlists method took a step backwards by pinning the vma for
the entire active lifespan of the context (the only way to evict was to
idle the entire GPU, not individual contexts). However, to circumvent
the tricky issue of locking (i.e. we cannot take struct_mutex at the
time of i915_gem_request_submit(), where we would want to move the
previous context onto the active tracker and unpin it), we take the
execlists approach and keep the contexts pinned until retirement.
The benefit of the execlists approach, more important for execlists than
legacy, was the reduction in work in pinning the context for each
request - as the context was kept pinned until idle, it could short
circuit the pinning for all active contexts.
We introduce new engine vfuncs to pin and unpin the context
respectively. The context is pinned at the start of the request, and
only unpinned when the following request is retired (this ensures that
the context is idle and coherent in main memory before we unpin it). We
move the engine->last_context tracking into the retirement itself
(rather than during request submission) in order to allow the submission
to be reordered or unwound without undue difficultly.
And finally an ulterior motive for unifying context handling was to
prepare for mock requests.
v2: Rename to last_retired_context, split out legacy_context tracking
for MI_SET_CONTEXT.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161218153724.8439-3-chris@chris-wilson.co.uk
2016-12-18 23:37:20 +08:00
|
|
|
|
2016-12-18 23:37:19 +08:00
|
|
|
/* Flush enough space to reduce the likelihood of waiting after
|
|
|
|
* we start building the request - in which case we will just
|
|
|
|
* have to repeat work.
|
|
|
|
*/
|
|
|
|
request->reserved_space += EXECLISTS_REQUEST_SIZE;
|
|
|
|
|
2017-11-15 23:12:04 +08:00
|
|
|
ret = intel_ring_wait_for_space(request->ring, request->reserved_space);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
2016-12-18 23:37:19 +08:00
|
|
|
|
|
|
|
/* Note that after this point, we have committed to using
|
|
|
|
* this request as it is being used to both track the
|
|
|
|
* state of engine initialisation and liveness of the
|
|
|
|
* golden renderstate above. Think twice before you try
|
|
|
|
* to cancel/unwind this request now.
|
|
|
|
*/
|
|
|
|
|
|
|
|
request->reserved_space -= EXECLISTS_REQUEST_SIZE;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-07-03 21:27:31 +08:00
|
|
|
/*
|
|
|
|
* In this WA we need to set GEN8_L3SQCREG4[21:21] and reset it after
|
|
|
|
* PIPE_CONTROL instruction. This is required for the flush to happen correctly
|
|
|
|
* but there is a slight complication as this is applied in WA batch where the
|
|
|
|
* values are only initialized once so we cannot take register value at the
|
|
|
|
* beginning and reuse it further; hence we save its value to memory, upload a
|
|
|
|
* constant value with bit21 set and then we restore it back with the saved value.
|
|
|
|
* To simplify the WA, a constant value is formed by using the default value
|
|
|
|
* of this register. This shouldn't be a problem because we are only modifying
|
|
|
|
* it for a short period and this batch in non-premptible. We can ofcourse
|
|
|
|
* use additional instructions that read the actual value of the register
|
|
|
|
* at that time and set our bit of interest but it makes the WA complicated.
|
|
|
|
*
|
|
|
|
* This WA is also required for Gen9 so extracting as a function avoids
|
|
|
|
* code duplication.
|
|
|
|
*/
|
2017-02-17 15:58:59 +08:00
|
|
|
static u32 *
|
|
|
|
gen8_emit_flush_coherentl3_wa(struct intel_engine_cs *engine, u32 *batch)
|
2015-06-20 02:07:01 +08:00
|
|
|
{
|
2017-02-17 15:58:59 +08:00
|
|
|
*batch++ = MI_STORE_REGISTER_MEM_GEN8 | MI_SRM_LRM_GLOBAL_GTT;
|
|
|
|
*batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
|
|
|
|
*batch++ = i915_ggtt_offset(engine->scratch) + 256;
|
|
|
|
*batch++ = 0;
|
|
|
|
|
|
|
|
*batch++ = MI_LOAD_REGISTER_IMM(1);
|
|
|
|
*batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
|
|
|
|
*batch++ = 0x40400000 | GEN8_LQSC_FLUSH_COHERENT_LINES;
|
|
|
|
|
2017-02-16 20:23:25 +08:00
|
|
|
batch = gen8_emit_pipe_control(batch,
|
|
|
|
PIPE_CONTROL_CS_STALL |
|
|
|
|
PIPE_CONTROL_DC_FLUSH_ENABLE,
|
|
|
|
0);
|
2017-02-17 15:58:59 +08:00
|
|
|
|
|
|
|
*batch++ = MI_LOAD_REGISTER_MEM_GEN8 | MI_SRM_LRM_GLOBAL_GTT;
|
|
|
|
*batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
|
|
|
|
*batch++ = i915_ggtt_offset(engine->scratch) + 256;
|
|
|
|
*batch++ = 0;
|
|
|
|
|
|
|
|
return batch;
|
2015-06-20 02:07:01 +08:00
|
|
|
}
|
|
|
|
|
2016-07-16 03:48:06 +08:00
|
|
|
/*
|
|
|
|
* Typically we only have one indirect_ctx and per_ctx batch buffer which are
|
|
|
|
* initialized at the beginning and shared across all contexts but this field
|
|
|
|
* helps us to have multiple batches at different offsets and select them based
|
|
|
|
* on a criteria. At the moment this batch always start at the beginning of the page
|
|
|
|
* and at this point we don't have multiple wa_ctx batch buffers.
|
2015-06-23 22:50:43 +08:00
|
|
|
*
|
2016-07-16 03:48:06 +08:00
|
|
|
* The number of WA applied are not known at the beginning; we use this field
|
|
|
|
* to return the no of DWORDS written.
|
2015-06-20 02:07:01 +08:00
|
|
|
*
|
2016-07-16 03:48:06 +08:00
|
|
|
* It is to be noted that this batch does not contain MI_BATCH_BUFFER_END
|
|
|
|
* so it adds NOOPs as padding to make it cacheline aligned.
|
|
|
|
* MI_BATCH_BUFFER_END will be added to perctx batch and both of them together
|
|
|
|
* makes a complete batch buffer.
|
2015-06-20 02:07:01 +08:00
|
|
|
*/
|
2017-02-17 15:58:59 +08:00
|
|
|
static u32 *gen8_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
|
2015-06-20 02:07:01 +08:00
|
|
|
{
|
2015-06-20 01:37:12 +08:00
|
|
|
/* WaDisableCtxRestoreArbitration:bdw,chv */
|
2017-02-17 15:58:59 +08:00
|
|
|
*batch++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2015-06-20 01:37:13 +08:00
|
|
|
/* WaFlushCoherentL3CacheLinesAtContextSwitch:bdw */
|
2017-02-17 15:58:59 +08:00
|
|
|
if (IS_BROADWELL(engine->i915))
|
|
|
|
batch = gen8_emit_flush_coherentl3_wa(engine, batch);
|
2015-06-20 01:37:13 +08:00
|
|
|
|
2015-06-23 22:46:57 +08:00
|
|
|
/* WaClearSlmSpaceAtContextSwitch:bdw,chv */
|
|
|
|
/* Actual scratch location is at 128 bytes offset */
|
2017-02-16 20:23:25 +08:00
|
|
|
batch = gen8_emit_pipe_control(batch,
|
|
|
|
PIPE_CONTROL_FLUSH_L3 |
|
|
|
|
PIPE_CONTROL_GLOBAL_GTT_IVB |
|
|
|
|
PIPE_CONTROL_CS_STALL |
|
|
|
|
PIPE_CONTROL_QW_WRITE,
|
|
|
|
i915_ggtt_offset(engine->scratch) +
|
|
|
|
2 * CACHELINE_BYTES);
|
2015-06-23 22:46:57 +08:00
|
|
|
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
*batch++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
|
|
|
|
|
2015-06-20 02:07:01 +08:00
|
|
|
/* Pad to end of cacheline */
|
2017-02-17 15:58:59 +08:00
|
|
|
while ((unsigned long)batch % CACHELINE_BYTES)
|
|
|
|
*batch++ = MI_NOOP;
|
2015-06-20 02:07:01 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* MI_BATCH_BUFFER_END is not required in Indirect ctx BB because
|
|
|
|
* execution depends on the length specified in terms of cache lines
|
|
|
|
* in the register CTX_RCS_INDIRECT_CTX
|
|
|
|
*/
|
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
return batch;
|
2015-06-20 02:07:01 +08:00
|
|
|
}
|
|
|
|
|
2018-06-18 17:41:50 +08:00
|
|
|
struct lri {
|
|
|
|
i915_reg_t reg;
|
|
|
|
u32 value;
|
|
|
|
};
|
|
|
|
|
|
|
|
static u32 *emit_lri(u32 *batch, const struct lri *lri, unsigned int count)
|
2015-07-14 22:01:27 +08:00
|
|
|
{
|
2018-06-18 17:41:50 +08:00
|
|
|
GEM_BUG_ON(!count || count > 63);
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
|
2018-06-18 17:41:50 +08:00
|
|
|
*batch++ = MI_LOAD_REGISTER_IMM(count);
|
|
|
|
do {
|
|
|
|
*batch++ = i915_mmio_reg_offset(lri->reg);
|
|
|
|
*batch++ = lri->value;
|
|
|
|
} while (lri++, --count);
|
|
|
|
*batch++ = MI_NOOP;
|
2015-07-14 22:01:29 +08:00
|
|
|
|
2018-06-18 17:41:50 +08:00
|
|
|
return batch;
|
|
|
|
}
|
2018-06-16 03:06:05 +08:00
|
|
|
|
2018-06-18 17:41:50 +08:00
|
|
|
static u32 *gen9_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
|
|
|
|
{
|
|
|
|
static const struct lri lri[] = {
|
|
|
|
/* WaDisableGatherAtSetShaderCommonSlice:skl,bxt,kbl,glk */
|
|
|
|
{
|
|
|
|
COMMON_SLICE_CHICKEN2,
|
|
|
|
__MASKED_FIELD(GEN9_DISABLE_GATHER_AT_SET_SHADER_COMMON_SLICE,
|
|
|
|
0),
|
|
|
|
},
|
|
|
|
|
|
|
|
/* BSpec: 11391 */
|
|
|
|
{
|
|
|
|
FF_SLICE_CHICKEN,
|
|
|
|
__MASKED_FIELD(FF_SLICE_CHICKEN_CL_PROVOKING_VERTEX_FIX,
|
|
|
|
FF_SLICE_CHICKEN_CL_PROVOKING_VERTEX_FIX),
|
|
|
|
},
|
|
|
|
|
|
|
|
/* BSpec: 11299 */
|
|
|
|
{
|
|
|
|
_3D_CHICKEN3,
|
|
|
|
__MASKED_FIELD(_3D_CHICKEN_SF_PROVOKING_VERTEX_FIX,
|
|
|
|
_3D_CHICKEN_SF_PROVOKING_VERTEX_FIX),
|
|
|
|
}
|
|
|
|
};
|
2018-06-16 03:06:05 +08:00
|
|
|
|
2018-06-18 17:41:50 +08:00
|
|
|
*batch++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
|
2018-06-16 03:06:05 +08:00
|
|
|
|
2018-06-18 17:41:50 +08:00
|
|
|
/* WaFlushCoherentL3CacheLinesAtContextSwitch:skl,bxt,glk */
|
|
|
|
batch = gen8_emit_flush_coherentl3_wa(engine, batch);
|
2018-06-16 03:06:05 +08:00
|
|
|
|
2018-06-18 17:41:50 +08:00
|
|
|
batch = emit_lri(batch, lri, ARRAY_SIZE(lri));
|
2016-07-20 19:26:13 +08:00
|
|
|
|
2016-06-07 22:19:15 +08:00
|
|
|
/* WaClearSlmSpaceAtContextSwitch:kbl */
|
|
|
|
/* Actual scratch location is at 128 bytes offset */
|
2017-02-17 15:58:59 +08:00
|
|
|
if (IS_KBL_REVID(engine->i915, 0, KBL_REVID_A0)) {
|
2017-02-16 20:23:25 +08:00
|
|
|
batch = gen8_emit_pipe_control(batch,
|
|
|
|
PIPE_CONTROL_FLUSH_L3 |
|
|
|
|
PIPE_CONTROL_GLOBAL_GTT_IVB |
|
|
|
|
PIPE_CONTROL_CS_STALL |
|
|
|
|
PIPE_CONTROL_QW_WRITE,
|
|
|
|
i915_ggtt_offset(engine->scratch)
|
|
|
|
+ 2 * CACHELINE_BYTES);
|
2016-06-07 22:19:15 +08:00
|
|
|
}
|
2016-07-05 17:01:30 +08:00
|
|
|
|
2017-01-26 17:16:58 +08:00
|
|
|
/* WaMediaPoolStateCmdInWABB:bxt,glk */
|
2016-07-05 17:01:30 +08:00
|
|
|
if (HAS_POOLED_EU(engine->i915)) {
|
|
|
|
/*
|
|
|
|
* EU pool configuration is setup along with golden context
|
|
|
|
* during context initialization. This value depends on
|
|
|
|
* device type (2x6 or 3x6) and needs to be updated based
|
|
|
|
* on which subslice is disabled especially for 2x6
|
|
|
|
* devices, however it is safe to load default
|
|
|
|
* configuration of 3x6 device instead of masking off
|
|
|
|
* corresponding bits because HW ignores bits of a disabled
|
|
|
|
* subslice and drops down to appropriate config. Please
|
|
|
|
* see render_state_setup() in i915_gem_render_state.c for
|
|
|
|
* possible configurations, to avoid duplication they are
|
|
|
|
* not shown here again.
|
|
|
|
*/
|
2017-02-17 15:58:59 +08:00
|
|
|
*batch++ = GEN9_MEDIA_POOL_STATE;
|
|
|
|
*batch++ = GEN9_MEDIA_POOL_ENABLE;
|
|
|
|
*batch++ = 0x00777000;
|
|
|
|
*batch++ = 0;
|
|
|
|
*batch++ = 0;
|
|
|
|
*batch++ = 0;
|
2016-07-05 17:01:30 +08:00
|
|
|
}
|
|
|
|
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
*batch++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
|
|
|
|
|
2015-07-14 22:01:27 +08:00
|
|
|
/* Pad to end of cacheline */
|
2017-02-17 15:58:59 +08:00
|
|
|
while ((unsigned long)batch % CACHELINE_BYTES)
|
|
|
|
*batch++ = MI_NOOP;
|
2015-07-14 22:01:27 +08:00
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
return batch;
|
2015-07-14 22:01:27 +08:00
|
|
|
}
|
|
|
|
|
2018-02-06 07:33:30 +08:00
|
|
|
static u32 *
|
|
|
|
gen10_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* WaPipeControlBefore3DStateSamplePattern: cnl
|
|
|
|
*
|
|
|
|
* Ensure the engine is idle prior to programming a
|
|
|
|
* 3DSTATE_SAMPLE_PATTERN during a context restore.
|
|
|
|
*/
|
|
|
|
batch = gen8_emit_pipe_control(batch,
|
|
|
|
PIPE_CONTROL_CS_STALL,
|
|
|
|
0);
|
|
|
|
/*
|
|
|
|
* WaPipeControlBefore3DStateSamplePattern says we need 4 dwords for
|
|
|
|
* the PIPE_CONTROL followed by 12 dwords of 0x0, so 16 dwords in
|
|
|
|
* total. However, a PIPE_CONTROL is 6 dwords long, not 4, which is
|
|
|
|
* confusing. Since gen8_emit_pipe_control() already advances the
|
|
|
|
* batch by 6 dwords, we advance the other 10 here, completing a
|
|
|
|
* cacheline. It's not clear if the workaround requires this padding
|
|
|
|
* before other commands, or if it's just the regular padding we would
|
|
|
|
* already have for the workaround bb, so leave it here for now.
|
|
|
|
*/
|
|
|
|
for (i = 0; i < 10; i++)
|
|
|
|
*batch++ = MI_NOOP;
|
|
|
|
|
|
|
|
/* Pad to end of cacheline */
|
|
|
|
while ((unsigned long)batch % CACHELINE_BYTES)
|
|
|
|
*batch++ = MI_NOOP;
|
|
|
|
|
|
|
|
return batch;
|
|
|
|
}
|
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
#define CTX_WA_BB_OBJ_SIZE (PAGE_SIZE)
|
|
|
|
|
|
|
|
static int lrc_setup_wa_ctx(struct intel_engine_cs *engine)
|
2015-06-20 02:07:01 +08:00
|
|
|
{
|
2016-08-15 17:49:04 +08:00
|
|
|
struct drm_i915_gem_object *obj;
|
|
|
|
struct i915_vma *vma;
|
|
|
|
int err;
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
obj = i915_gem_object_create(engine->i915, CTX_WA_BB_OBJ_SIZE);
|
2016-08-15 17:49:04 +08:00
|
|
|
if (IS_ERR(obj))
|
|
|
|
return PTR_ERR(obj);
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2018-06-05 23:37:58 +08:00
|
|
|
vma = i915_vma_instance(obj, &engine->i915->ggtt.vm, NULL);
|
2016-08-15 17:49:04 +08:00
|
|
|
if (IS_ERR(vma)) {
|
|
|
|
err = PTR_ERR(vma);
|
|
|
|
goto err;
|
2015-06-20 02:07:01 +08:00
|
|
|
}
|
|
|
|
|
2016-08-15 17:49:04 +08:00
|
|
|
err = i915_vma_pin(vma, 0, PAGE_SIZE, PIN_GLOBAL | PIN_HIGH);
|
|
|
|
if (err)
|
|
|
|
goto err;
|
|
|
|
|
|
|
|
engine->wa_ctx.vma = vma;
|
2015-06-20 02:07:01 +08:00
|
|
|
return 0;
|
2016-08-15 17:49:04 +08:00
|
|
|
|
|
|
|
err:
|
|
|
|
i915_gem_object_put(obj);
|
|
|
|
return err;
|
2015-06-20 02:07:01 +08:00
|
|
|
}
|
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
static void lrc_destroy_wa_ctx(struct intel_engine_cs *engine)
|
2015-06-20 02:07:01 +08:00
|
|
|
{
|
2018-07-21 20:50:37 +08:00
|
|
|
i915_vma_unpin_and_release(&engine->wa_ctx.vma, 0);
|
2015-06-20 02:07:01 +08:00
|
|
|
}
|
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
typedef u32 *(*wa_bb_func_t)(struct intel_engine_cs *engine, u32 *batch);
|
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
static int intel_init_workaround_bb(struct intel_engine_cs *engine)
|
2015-06-20 02:07:01 +08:00
|
|
|
{
|
2016-08-15 17:49:04 +08:00
|
|
|
struct i915_ctx_workarounds *wa_ctx = &engine->wa_ctx;
|
2017-02-17 15:58:59 +08:00
|
|
|
struct i915_wa_ctx_bb *wa_bb[2] = { &wa_ctx->indirect_ctx,
|
|
|
|
&wa_ctx->per_ctx };
|
|
|
|
wa_bb_func_t wa_bb_fn[2];
|
2015-06-20 02:07:01 +08:00
|
|
|
struct page *page;
|
2017-02-17 15:58:59 +08:00
|
|
|
void *batch, *batch_ptr;
|
|
|
|
unsigned int i;
|
2016-08-15 17:49:04 +08:00
|
|
|
int ret;
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2018-01-19 18:00:04 +08:00
|
|
|
if (GEM_WARN_ON(engine->id != RCS))
|
2017-02-17 15:58:59 +08:00
|
|
|
return -EINVAL;
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
switch (INTEL_GEN(engine->i915)) {
|
2018-05-09 05:29:23 +08:00
|
|
|
case 11:
|
|
|
|
return 0;
|
2017-08-16 07:16:48 +08:00
|
|
|
case 10:
|
2018-02-06 07:33:30 +08:00
|
|
|
wa_bb_fn[0] = gen10_init_indirectctx_bb;
|
|
|
|
wa_bb_fn[1] = NULL;
|
|
|
|
break;
|
2017-02-17 15:58:59 +08:00
|
|
|
case 9:
|
|
|
|
wa_bb_fn[0] = gen9_init_indirectctx_bb;
|
2017-09-21 21:54:44 +08:00
|
|
|
wa_bb_fn[1] = NULL;
|
2017-02-17 15:58:59 +08:00
|
|
|
break;
|
|
|
|
case 8:
|
|
|
|
wa_bb_fn[0] = gen8_init_indirectctx_bb;
|
2017-10-04 04:34:49 +08:00
|
|
|
wa_bb_fn[1] = NULL;
|
2017-02-17 15:58:59 +08:00
|
|
|
break;
|
|
|
|
default:
|
|
|
|
MISSING_CASE(INTEL_GEN(engine->i915));
|
2015-06-23 22:50:44 +08:00
|
|
|
return 0;
|
2015-07-14 22:01:27 +08:00
|
|
|
}
|
2015-06-23 22:50:44 +08:00
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
ret = lrc_setup_wa_ctx(engine);
|
2015-06-20 02:07:01 +08:00
|
|
|
if (ret) {
|
|
|
|
DRM_DEBUG_DRIVER("Failed to setup context WA page: %d\n", ret);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2016-08-15 17:49:04 +08:00
|
|
|
page = i915_gem_object_get_dirty_page(wa_ctx->vma->obj, 0);
|
2017-02-17 15:58:59 +08:00
|
|
|
batch = batch_ptr = kmap_atomic(page);
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
/*
|
|
|
|
* Emit the two workaround batch buffers, recording the offset from the
|
|
|
|
* start of the workaround batch buffer object for each and their
|
|
|
|
* respective sizes.
|
|
|
|
*/
|
|
|
|
for (i = 0; i < ARRAY_SIZE(wa_bb_fn); i++) {
|
|
|
|
wa_bb[i]->offset = batch_ptr - batch;
|
2018-01-26 20:18:46 +08:00
|
|
|
if (GEM_WARN_ON(!IS_ALIGNED(wa_bb[i]->offset,
|
|
|
|
CACHELINE_BYTES))) {
|
2017-02-17 15:58:59 +08:00
|
|
|
ret = -EINVAL;
|
|
|
|
break;
|
|
|
|
}
|
2017-09-21 21:54:43 +08:00
|
|
|
if (wa_bb_fn[i])
|
|
|
|
batch_ptr = wa_bb_fn[i](engine, batch_ptr);
|
2017-02-17 15:58:59 +08:00
|
|
|
wa_bb[i]->size = batch_ptr - (batch + wa_bb[i]->offset);
|
2015-06-20 02:07:01 +08:00
|
|
|
}
|
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
BUG_ON(batch_ptr - batch > CTX_WA_BB_OBJ_SIZE);
|
|
|
|
|
2015-06-20 02:07:01 +08:00
|
|
|
kunmap_atomic(batch);
|
|
|
|
if (ret)
|
2017-02-17 15:58:59 +08:00
|
|
|
lrc_destroy_wa_ctx(engine);
|
2015-06-20 02:07:01 +08:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2018-01-02 23:12:34 +08:00
|
|
|
static void enable_execlists(struct intel_engine_cs *engine)
|
2014-07-25 00:04:24 +08:00
|
|
|
{
|
2016-05-06 22:40:21 +08:00
|
|
|
struct drm_i915_private *dev_priv = engine->i915;
|
2018-01-02 23:12:34 +08:00
|
|
|
|
|
|
|
I915_WRITE(RING_HWSTAM(engine->mmio_base), 0xffffffff);
|
2018-01-30 21:49:17 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Make sure we're not enabling the new 12-deep CSB
|
|
|
|
* FIFO as that requires a slightly updated handling
|
|
|
|
* in the ctx switch irq. Since we're currently only
|
|
|
|
* using only 2 elements of the enhanced execlists the
|
|
|
|
* deeper FIFO it's not needed and it's not worth adding
|
|
|
|
* more statements to the irq handler to support it.
|
|
|
|
*/
|
|
|
|
if (INTEL_GEN(dev_priv) >= 11)
|
|
|
|
I915_WRITE(RING_MODE_GEN7(engine),
|
|
|
|
_MASKED_BIT_DISABLE(GEN11_GFX_DISABLE_LEGACY_MODE));
|
|
|
|
else
|
|
|
|
I915_WRITE(RING_MODE_GEN7(engine),
|
|
|
|
_MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE));
|
|
|
|
|
drm/i915: Flush the ring stop bit after clearing RING_HEAD in reset
Inside the live_hangcheck (reset) selftests, we occasionally see
failures like
<7>[ 239.094840] i915_gem_set_wedged rcs0
<7>[ 239.094843] i915_gem_set_wedged current seqno 19a98, last 19a9a, hangcheck 0 [5158 ms]
<7>[ 239.094846] i915_gem_set_wedged Reset count: 6239 (global 1)
<7>[ 239.094848] i915_gem_set_wedged Requests:
<7>[ 239.095052] i915_gem_set_wedged first 19a99 [e8c:5f] prio=1024 @ 5159ms: (null)
<7>[ 239.095056] i915_gem_set_wedged last 19a9a [e81:1a] prio=139 @ 5159ms: igt/rcs0[5977]/1
<7>[ 239.095059] i915_gem_set_wedged active 19a99 [e8c:5f] prio=1024 @ 5159ms: (null)
<7>[ 239.095062] i915_gem_set_wedged [head 0220, postfix 0280, tail 02a8, batch 0xffffffff_ffffffff]
<7>[ 239.100050] i915_gem_set_wedged ring->start: 0x00283000
<7>[ 239.100053] i915_gem_set_wedged ring->head: 0x000001f8
<7>[ 239.100055] i915_gem_set_wedged ring->tail: 0x000002a8
<7>[ 239.100057] i915_gem_set_wedged ring->emit: 0x000002a8
<7>[ 239.100059] i915_gem_set_wedged ring->space: 0x00000f10
<7>[ 239.100085] i915_gem_set_wedged RING_START: 0x00283000
<7>[ 239.100088] i915_gem_set_wedged RING_HEAD: 0x00000260
<7>[ 239.100091] i915_gem_set_wedged RING_TAIL: 0x000002a8
<7>[ 239.100094] i915_gem_set_wedged RING_CTL: 0x00000001
<7>[ 239.100097] i915_gem_set_wedged RING_MODE: 0x00000300 [idle]
<7>[ 239.100100] i915_gem_set_wedged RING_IMR: fffffefe
<7>[ 239.100104] i915_gem_set_wedged ACTHD: 0x00000000_0000609c
<7>[ 239.100108] i915_gem_set_wedged BBADDR: 0x00000000_0000609d
<7>[ 239.100111] i915_gem_set_wedged DMA_FADDR: 0x00000000_00283260
<7>[ 239.100114] i915_gem_set_wedged IPEIR: 0x00000000
<7>[ 239.100117] i915_gem_set_wedged IPEHR: 0x02800000
<7>[ 239.100120] i915_gem_set_wedged Execlist status: 0x00044052 00000002
<7>[ 239.100124] i915_gem_set_wedged Execlist CSB read 5 [5 cached], write 5 [5 from hws], interrupt posted? no, tasklet queued? no (enabled)
<7>[ 239.100128] i915_gem_set_wedged ELSP[0] count=1, ring->start=00283000, rq: 19a99 [e8c:5f] prio=1024 @ 5164ms: (null)
<7>[ 239.100132] i915_gem_set_wedged ELSP[1] count=1, ring->start=00257000, rq: 19a9a [e81:1a] prio=139 @ 5164ms: igt/rcs0[5977]/1
<7>[ 239.100135] i915_gem_set_wedged HW active? 0x5
<7>[ 239.100250] i915_gem_set_wedged E 19a99 [e8c:5f] prio=1024 @ 5164ms: (null)
<7>[ 239.100338] i915_gem_set_wedged E 19a9a [e81:1a] prio=139 @ 5164ms: igt/rcs0[5977]/1
<7>[ 239.100340] i915_gem_set_wedged Queue priority: 139
<7>[ 239.100343] i915_gem_set_wedged Q 0 [e98:19] prio=132 @ 5164ms: igt/rcs0[5977]/8
<7>[ 239.100346] i915_gem_set_wedged Q 0 [e84:19] prio=121 @ 5165ms: igt/rcs0[5977]/2
<7>[ 239.100349] i915_gem_set_wedged Q 0 [e87:19] prio=82 @ 5165ms: igt/rcs0[5977]/3
<7>[ 239.100352] i915_gem_set_wedged Q 0 [e84:1a] prio=44 @ 5164ms: igt/rcs0[5977]/2
<7>[ 239.100356] i915_gem_set_wedged Q 0 [e8b:19] prio=20 @ 5165ms: igt/rcs0[5977]/4
<7>[ 239.100362] i915_gem_set_wedged drv_selftest [5894] waiting for 19a99
where the GPU saw an arbitration point and idles; AND HAS NOT BEEN RESET!
The RING_MODE indicates that is idle and has the STOP_RING bit set, so
try clearing it.
v2: Only clear the bit on restarting the ring, as we want to be sure the
STOP_RING bit is kept if reset fails on wedging.
v3: Spot when the ring state doesn't make sense when re-initialising the
engine and dump it to the logs so that we don't have to wait for an
error later and try to guess what happened earlier.
v4: Prepare to print all the unexpected state, not just the first.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180518100933.2239-1-chris@chris-wilson.co.uk
2018-05-18 18:09:33 +08:00
|
|
|
I915_WRITE(RING_MI_MODE(engine->mmio_base),
|
|
|
|
_MASKED_BIT_DISABLE(STOP_RING));
|
|
|
|
|
2018-01-02 23:12:34 +08:00
|
|
|
I915_WRITE(RING_HWS_PGA(engine->mmio_base),
|
|
|
|
engine->status_page.ggtt_offset);
|
|
|
|
POSTING_READ(RING_HWS_PGA(engine->mmio_base));
|
|
|
|
}
|
|
|
|
|
drm/i915: Flush the ring stop bit after clearing RING_HEAD in reset
Inside the live_hangcheck (reset) selftests, we occasionally see
failures like
<7>[ 239.094840] i915_gem_set_wedged rcs0
<7>[ 239.094843] i915_gem_set_wedged current seqno 19a98, last 19a9a, hangcheck 0 [5158 ms]
<7>[ 239.094846] i915_gem_set_wedged Reset count: 6239 (global 1)
<7>[ 239.094848] i915_gem_set_wedged Requests:
<7>[ 239.095052] i915_gem_set_wedged first 19a99 [e8c:5f] prio=1024 @ 5159ms: (null)
<7>[ 239.095056] i915_gem_set_wedged last 19a9a [e81:1a] prio=139 @ 5159ms: igt/rcs0[5977]/1
<7>[ 239.095059] i915_gem_set_wedged active 19a99 [e8c:5f] prio=1024 @ 5159ms: (null)
<7>[ 239.095062] i915_gem_set_wedged [head 0220, postfix 0280, tail 02a8, batch 0xffffffff_ffffffff]
<7>[ 239.100050] i915_gem_set_wedged ring->start: 0x00283000
<7>[ 239.100053] i915_gem_set_wedged ring->head: 0x000001f8
<7>[ 239.100055] i915_gem_set_wedged ring->tail: 0x000002a8
<7>[ 239.100057] i915_gem_set_wedged ring->emit: 0x000002a8
<7>[ 239.100059] i915_gem_set_wedged ring->space: 0x00000f10
<7>[ 239.100085] i915_gem_set_wedged RING_START: 0x00283000
<7>[ 239.100088] i915_gem_set_wedged RING_HEAD: 0x00000260
<7>[ 239.100091] i915_gem_set_wedged RING_TAIL: 0x000002a8
<7>[ 239.100094] i915_gem_set_wedged RING_CTL: 0x00000001
<7>[ 239.100097] i915_gem_set_wedged RING_MODE: 0x00000300 [idle]
<7>[ 239.100100] i915_gem_set_wedged RING_IMR: fffffefe
<7>[ 239.100104] i915_gem_set_wedged ACTHD: 0x00000000_0000609c
<7>[ 239.100108] i915_gem_set_wedged BBADDR: 0x00000000_0000609d
<7>[ 239.100111] i915_gem_set_wedged DMA_FADDR: 0x00000000_00283260
<7>[ 239.100114] i915_gem_set_wedged IPEIR: 0x00000000
<7>[ 239.100117] i915_gem_set_wedged IPEHR: 0x02800000
<7>[ 239.100120] i915_gem_set_wedged Execlist status: 0x00044052 00000002
<7>[ 239.100124] i915_gem_set_wedged Execlist CSB read 5 [5 cached], write 5 [5 from hws], interrupt posted? no, tasklet queued? no (enabled)
<7>[ 239.100128] i915_gem_set_wedged ELSP[0] count=1, ring->start=00283000, rq: 19a99 [e8c:5f] prio=1024 @ 5164ms: (null)
<7>[ 239.100132] i915_gem_set_wedged ELSP[1] count=1, ring->start=00257000, rq: 19a9a [e81:1a] prio=139 @ 5164ms: igt/rcs0[5977]/1
<7>[ 239.100135] i915_gem_set_wedged HW active? 0x5
<7>[ 239.100250] i915_gem_set_wedged E 19a99 [e8c:5f] prio=1024 @ 5164ms: (null)
<7>[ 239.100338] i915_gem_set_wedged E 19a9a [e81:1a] prio=139 @ 5164ms: igt/rcs0[5977]/1
<7>[ 239.100340] i915_gem_set_wedged Queue priority: 139
<7>[ 239.100343] i915_gem_set_wedged Q 0 [e98:19] prio=132 @ 5164ms: igt/rcs0[5977]/8
<7>[ 239.100346] i915_gem_set_wedged Q 0 [e84:19] prio=121 @ 5165ms: igt/rcs0[5977]/2
<7>[ 239.100349] i915_gem_set_wedged Q 0 [e87:19] prio=82 @ 5165ms: igt/rcs0[5977]/3
<7>[ 239.100352] i915_gem_set_wedged Q 0 [e84:1a] prio=44 @ 5164ms: igt/rcs0[5977]/2
<7>[ 239.100356] i915_gem_set_wedged Q 0 [e8b:19] prio=20 @ 5165ms: igt/rcs0[5977]/4
<7>[ 239.100362] i915_gem_set_wedged drv_selftest [5894] waiting for 19a99
where the GPU saw an arbitration point and idles; AND HAS NOT BEEN RESET!
The RING_MODE indicates that is idle and has the STOP_RING bit set, so
try clearing it.
v2: Only clear the bit on restarting the ring, as we want to be sure the
STOP_RING bit is kept if reset fails on wedging.
v3: Spot when the ring state doesn't make sense when re-initialising the
engine and dump it to the logs so that we don't have to wait for an
error later and try to guess what happened earlier.
v4: Prepare to print all the unexpected state, not just the first.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180518100933.2239-1-chris@chris-wilson.co.uk
2018-05-18 18:09:33 +08:00
|
|
|
static bool unexpected_starting_state(struct intel_engine_cs *engine)
|
|
|
|
{
|
|
|
|
struct drm_i915_private *dev_priv = engine->i915;
|
|
|
|
bool unexpected = false;
|
|
|
|
|
|
|
|
if (I915_READ(RING_MI_MODE(engine->mmio_base)) & STOP_RING) {
|
|
|
|
DRM_DEBUG_DRIVER("STOP_RING still set in RING_MI_MODE\n");
|
|
|
|
unexpected = true;
|
|
|
|
}
|
|
|
|
|
|
|
|
return unexpected;
|
|
|
|
}
|
|
|
|
|
2018-01-02 23:12:34 +08:00
|
|
|
static int gen8_init_common_ring(struct intel_engine_cs *engine)
|
|
|
|
{
|
2016-09-09 21:11:53 +08:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
ret = intel_mocs_init_engine(engine);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
2014-07-25 00:04:24 +08:00
|
|
|
|
2016-10-07 14:53:26 +08:00
|
|
|
intel_engine_reset_breadcrumbs(engine);
|
2016-09-09 21:11:53 +08:00
|
|
|
|
drm/i915: Flush the ring stop bit after clearing RING_HEAD in reset
Inside the live_hangcheck (reset) selftests, we occasionally see
failures like
<7>[ 239.094840] i915_gem_set_wedged rcs0
<7>[ 239.094843] i915_gem_set_wedged current seqno 19a98, last 19a9a, hangcheck 0 [5158 ms]
<7>[ 239.094846] i915_gem_set_wedged Reset count: 6239 (global 1)
<7>[ 239.094848] i915_gem_set_wedged Requests:
<7>[ 239.095052] i915_gem_set_wedged first 19a99 [e8c:5f] prio=1024 @ 5159ms: (null)
<7>[ 239.095056] i915_gem_set_wedged last 19a9a [e81:1a] prio=139 @ 5159ms: igt/rcs0[5977]/1
<7>[ 239.095059] i915_gem_set_wedged active 19a99 [e8c:5f] prio=1024 @ 5159ms: (null)
<7>[ 239.095062] i915_gem_set_wedged [head 0220, postfix 0280, tail 02a8, batch 0xffffffff_ffffffff]
<7>[ 239.100050] i915_gem_set_wedged ring->start: 0x00283000
<7>[ 239.100053] i915_gem_set_wedged ring->head: 0x000001f8
<7>[ 239.100055] i915_gem_set_wedged ring->tail: 0x000002a8
<7>[ 239.100057] i915_gem_set_wedged ring->emit: 0x000002a8
<7>[ 239.100059] i915_gem_set_wedged ring->space: 0x00000f10
<7>[ 239.100085] i915_gem_set_wedged RING_START: 0x00283000
<7>[ 239.100088] i915_gem_set_wedged RING_HEAD: 0x00000260
<7>[ 239.100091] i915_gem_set_wedged RING_TAIL: 0x000002a8
<7>[ 239.100094] i915_gem_set_wedged RING_CTL: 0x00000001
<7>[ 239.100097] i915_gem_set_wedged RING_MODE: 0x00000300 [idle]
<7>[ 239.100100] i915_gem_set_wedged RING_IMR: fffffefe
<7>[ 239.100104] i915_gem_set_wedged ACTHD: 0x00000000_0000609c
<7>[ 239.100108] i915_gem_set_wedged BBADDR: 0x00000000_0000609d
<7>[ 239.100111] i915_gem_set_wedged DMA_FADDR: 0x00000000_00283260
<7>[ 239.100114] i915_gem_set_wedged IPEIR: 0x00000000
<7>[ 239.100117] i915_gem_set_wedged IPEHR: 0x02800000
<7>[ 239.100120] i915_gem_set_wedged Execlist status: 0x00044052 00000002
<7>[ 239.100124] i915_gem_set_wedged Execlist CSB read 5 [5 cached], write 5 [5 from hws], interrupt posted? no, tasklet queued? no (enabled)
<7>[ 239.100128] i915_gem_set_wedged ELSP[0] count=1, ring->start=00283000, rq: 19a99 [e8c:5f] prio=1024 @ 5164ms: (null)
<7>[ 239.100132] i915_gem_set_wedged ELSP[1] count=1, ring->start=00257000, rq: 19a9a [e81:1a] prio=139 @ 5164ms: igt/rcs0[5977]/1
<7>[ 239.100135] i915_gem_set_wedged HW active? 0x5
<7>[ 239.100250] i915_gem_set_wedged E 19a99 [e8c:5f] prio=1024 @ 5164ms: (null)
<7>[ 239.100338] i915_gem_set_wedged E 19a9a [e81:1a] prio=139 @ 5164ms: igt/rcs0[5977]/1
<7>[ 239.100340] i915_gem_set_wedged Queue priority: 139
<7>[ 239.100343] i915_gem_set_wedged Q 0 [e98:19] prio=132 @ 5164ms: igt/rcs0[5977]/8
<7>[ 239.100346] i915_gem_set_wedged Q 0 [e84:19] prio=121 @ 5165ms: igt/rcs0[5977]/2
<7>[ 239.100349] i915_gem_set_wedged Q 0 [e87:19] prio=82 @ 5165ms: igt/rcs0[5977]/3
<7>[ 239.100352] i915_gem_set_wedged Q 0 [e84:1a] prio=44 @ 5164ms: igt/rcs0[5977]/2
<7>[ 239.100356] i915_gem_set_wedged Q 0 [e8b:19] prio=20 @ 5165ms: igt/rcs0[5977]/4
<7>[ 239.100362] i915_gem_set_wedged drv_selftest [5894] waiting for 19a99
where the GPU saw an arbitration point and idles; AND HAS NOT BEEN RESET!
The RING_MODE indicates that is idle and has the STOP_RING bit set, so
try clearing it.
v2: Only clear the bit on restarting the ring, as we want to be sure the
STOP_RING bit is kept if reset fails on wedging.
v3: Spot when the ring state doesn't make sense when re-initialising the
engine and dump it to the logs so that we don't have to wait for an
error later and try to guess what happened earlier.
v4: Prepare to print all the unexpected state, not just the first.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180518100933.2239-1-chris@chris-wilson.co.uk
2018-05-18 18:09:33 +08:00
|
|
|
if (GEM_SHOW_DEBUG() && unexpected_starting_state(engine)) {
|
|
|
|
struct drm_printer p = drm_debug_printer(__func__);
|
|
|
|
|
|
|
|
intel_engine_dump(engine, &p, NULL);
|
|
|
|
}
|
|
|
|
|
2018-01-02 23:12:34 +08:00
|
|
|
enable_execlists(engine);
|
2014-07-25 00:04:24 +08:00
|
|
|
|
2016-09-09 21:11:53 +08:00
|
|
|
return 0;
|
2014-07-25 00:04:24 +08:00
|
|
|
}
|
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
static int gen8_init_render_ring(struct intel_engine_cs *engine)
|
2014-07-25 00:04:24 +08:00
|
|
|
{
|
2016-05-06 22:40:21 +08:00
|
|
|
struct drm_i915_private *dev_priv = engine->i915;
|
2014-07-25 00:04:24 +08:00
|
|
|
int ret;
|
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
ret = gen8_init_common_ring(engine);
|
2014-07-25 00:04:24 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
2018-04-14 20:27:54 +08:00
|
|
|
intel_whitelist_workarounds_apply(engine);
|
2018-04-11 00:12:47 +08:00
|
|
|
|
2014-07-25 00:04:24 +08:00
|
|
|
/* We need to disable the AsyncFlip performance optimisations in order
|
|
|
|
* to use MI_WAIT_FOR_EVENT within the CS. It should already be
|
|
|
|
* programmed to '1' on all products.
|
|
|
|
*
|
|
|
|
* WaDisableAsyncFlipPerfMode:snb,ivb,hsw,vlv,bdw,chv
|
|
|
|
*/
|
|
|
|
I915_WRITE(MI_MODE, _MASKED_BIT_ENABLE(ASYNC_FLIP_PERF_DISABLE));
|
|
|
|
|
|
|
|
I915_WRITE(INSTPM, _MASKED_BIT_ENABLE(INSTPM_FORCE_ORDERING));
|
|
|
|
|
2018-04-11 00:12:47 +08:00
|
|
|
return 0;
|
2014-07-25 00:04:24 +08:00
|
|
|
}
|
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
static int gen9_init_render_ring(struct intel_engine_cs *engine)
|
2015-02-10 03:33:08 +08:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
ret = gen8_init_common_ring(engine);
|
2015-02-10 03:33:08 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
2018-04-14 20:27:54 +08:00
|
|
|
intel_whitelist_workarounds_apply(engine);
|
2018-04-11 00:12:47 +08:00
|
|
|
|
|
|
|
return 0;
|
2015-02-10 03:33:08 +08:00
|
|
|
}
|
|
|
|
|
2018-05-17 02:33:51 +08:00
|
|
|
static struct i915_request *
|
|
|
|
execlists_reset_prepare(struct intel_engine_cs *engine)
|
|
|
|
{
|
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
2018-05-17 02:33:54 +08:00
|
|
|
struct i915_request *request, *active;
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
unsigned long flags;
|
2018-05-17 02:33:51 +08:00
|
|
|
|
|
|
|
GEM_TRACE("%s\n", engine->name);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Prevent request submission to the hardware until we have
|
|
|
|
* completed the reset in i915_gem_reset_finish(). If a request
|
|
|
|
* is completed by one engine, it may then queue a request
|
|
|
|
* to a second via its execlists->tasklet *just* as we are
|
|
|
|
* calling engine->init_hw() and also writing the ELSP.
|
|
|
|
* Turning off the execlists->tasklet until the reset is over
|
|
|
|
* prevents the race.
|
|
|
|
*/
|
|
|
|
__tasklet_disable_sync_once(&execlists->tasklet);
|
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
spin_lock_irqsave(&engine->timeline.lock, flags);
|
|
|
|
|
2018-05-17 02:33:54 +08:00
|
|
|
/*
|
|
|
|
* We want to flush the pending context switches, having disabled
|
|
|
|
* the tasklet above, we can assume exclusive access to the execlists.
|
|
|
|
* For this allows us to catch up with an inflight preemption event,
|
|
|
|
* and avoid blaming an innocent request if the stall was due to the
|
|
|
|
* preemption itself.
|
|
|
|
*/
|
2018-06-29 04:12:10 +08:00
|
|
|
process_csb(engine);
|
2018-05-17 02:33:54 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The last active request can then be no later than the last request
|
|
|
|
* now in ELSP[0]. So search backwards from there, so that if the GPU
|
|
|
|
* has advanced beyond the last CSB update, it will be pardoned.
|
|
|
|
*/
|
|
|
|
active = NULL;
|
|
|
|
request = port_request(execlists->port);
|
|
|
|
if (request) {
|
2018-05-17 02:33:55 +08:00
|
|
|
/*
|
|
|
|
* Prevent the breadcrumb from advancing before we decide
|
|
|
|
* which request is currently active.
|
|
|
|
*/
|
|
|
|
intel_engine_stop_cs(engine);
|
|
|
|
|
2018-05-17 02:33:54 +08:00
|
|
|
list_for_each_entry_from_reverse(request,
|
|
|
|
&engine->timeline.requests,
|
|
|
|
link) {
|
|
|
|
if (__i915_request_completed(request,
|
|
|
|
request->global_seqno))
|
|
|
|
break;
|
|
|
|
|
|
|
|
active = request;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
drm/i915/execlists: Direct submission of new requests (avoid tasklet/ksoftirqd)
Back in commit 27af5eea54d1 ("drm/i915: Move execlists irq handler to a
bottom half"), we came to the conclusion that running our CSB processing
and ELSP submission from inside the irq handler was a bad idea. A really
bad idea as we could impose nearly 1s latency on other users of the
system, on average! Deferring our work to a tasklet allowed us to do the
processing with irqs enabled, reducing the impact to an average of about
50us.
We have since eradicated the use of forcewaked mmio from inside the CSB
processing and ELSP submission, bringing the impact down to around 5us
(on Kabylake); an order of magnitude better than our measurements 2
years ago on Broadwell and only about 2x worse on average than the
gem_syslatency on an unladen system.
In this iteration of the tasklet-vs-direct submission debate, we seek a
compromise where by we submit new requests immediately to the HW but
defer processing the CS interrupt onto a tasklet. We gain the advantage
of low-latency and ksoftirqd avoidance when waking up the HW, while
avoiding the system-wide starvation of our CS irq-storms.
Comparing the impact on the maximum latency observed (that is the time
stolen from an RT process) over a 120s interval, repeated several times
(using gem_syslatency, similar to RT's cyclictest) while the system is
fully laden with i915 nops, we see that direct submission an actually
improve the worse case.
Maximum latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 2)
x Always using tasklets (a couple of >1000us outliers removed)
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| + |
| + |
| + |
| + + |
| + + + |
| + + + + x x x |
| +++ + + + x x x x x x |
| +++ + ++ + + *x x x x x x |
| +++ + ++ + * *x x * x x x |
| + +++ + ++ * * +*xxx * x x xx |
| * +++ + ++++* *x+**xx+ * x x xxxx x |
| **x++++*++**+*x*x****x+ * +x xx xxxx x x |
|x* ******+***************++*+***xxxxxx* xx*x xxx + x+|
| |__________MA___________| |
| |______M__A________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 118 91 186 124 125.28814 16.279137
+ 120 92 187 109 112.00833 13.458617
Difference at 95.0% confidence
-13.2798 +/- 3.79219
-10.5994% +/- 3.02677%
(Student's t, pooled s = 14.9237)
However the mean latency is adversely affected:
Mean latency in microseconds of a third party RT thread
(gem_syslatency -t 120 -f 1)
x Always using tasklets
+ Only using tasklets from CS irq, direct submission of requests
+------------------------------------------------------------------------+
| xxxxxx + ++ |
| xxxxxx + ++ |
| xxxxxx + +++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ ++ |
| xxxxxxx +++++ +++ |
| xxxxxxx + ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxx ++ ++++++++++ |
| xxxxxxxxxx +++++++++++++++ |
| xxxxxxxxxxx x +++++++++++++++ |
|x xxxxxxxxxxxxx x + + ++++++++++++++++++ +|
| |__A__| |
| |____A___| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 3.506 3.727 3.631 3.6321417 0.02773109
+ 120 3.834 4.149 4.039 4.0375167 0.041221676
Difference at 95.0% confidence
0.405375 +/- 0.00888913
11.1608% +/- 0.244735%
(Student's t, pooled s = 0.03513)
However, since the mean latency corresponds to the amount of irqsoff
processing we have to do for a CS interrupt, we only need to speed that
up to benefit not just system latency but our own throughput.
v2: Remember to defer submissions when under reset.
v4: Only use direct submission for new requests
v5: Be aware that with mixing direct tasklet evaluation and deferred
tasklets, we may end up idling before running the deferred tasklet.
v6: Remove the redudant likely() from tasklet_is_enabled(), restrict the
annotation to reset_in_progress().
v7: Take the full timeline.lock when enabling perf_pmu stats as the
tasklet is no longer a valid guard. A consequence is that the stats are
now only valid for engines also using the timeline.lock to process
state.
Testcase: igt/gem_exec_latency/*rthog*
References: 27af5eea54d1 ("drm/i915: Move execlists irq handler to a bottom half")
Suggested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180628201211.13837-9-chris@chris-wilson.co.uk
2018-06-29 04:12:11 +08:00
|
|
|
spin_unlock_irqrestore(&engine->timeline.lock, flags);
|
|
|
|
|
2018-05-17 02:33:54 +08:00
|
|
|
return active;
|
2018-05-17 02:33:51 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void execlists_reset(struct intel_engine_cs *engine,
|
|
|
|
struct i915_request *request)
|
2016-09-09 21:11:53 +08:00
|
|
|
{
|
2017-09-22 20:43:03 +08:00
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
2017-09-17 04:44:14 +08:00
|
|
|
unsigned long flags;
|
2018-04-28 19:15:32 +08:00
|
|
|
u32 *regs;
|
2017-07-21 20:32:22 +08:00
|
|
|
|
2018-04-06 20:35:14 +08:00
|
|
|
GEM_TRACE("%s request global=%x, current=%d\n",
|
|
|
|
engine->name, request ? request->global_seqno : 0,
|
|
|
|
intel_engine_get_seqno(engine));
|
2018-01-02 23:12:32 +08:00
|
|
|
|
2018-06-29 04:12:05 +08:00
|
|
|
spin_lock_irqsave(&engine->timeline.lock, flags);
|
2017-09-17 04:44:14 +08:00
|
|
|
|
2017-07-21 20:32:22 +08:00
|
|
|
/*
|
|
|
|
* Catch up with any missed context-switch interrupts.
|
|
|
|
*
|
|
|
|
* Ideally we would just read the remaining CSB entries now that we
|
|
|
|
* know the gpu is idle. However, the CSB registers are sometimes^W
|
|
|
|
* often trashed across a GPU reset! Instead we have to rely on
|
|
|
|
* guessing the missed context-switch events by looking at what
|
|
|
|
* requests were completed.
|
|
|
|
*/
|
2017-10-26 04:00:18 +08:00
|
|
|
execlists_cancel_port_requests(execlists);
|
2017-07-21 20:32:22 +08:00
|
|
|
|
2017-09-17 04:44:14 +08:00
|
|
|
/* Push back any incomplete requests for replay after the reset. */
|
2017-10-26 04:00:18 +08:00
|
|
|
__unwind_incomplete_requests(engine);
|
2017-07-21 20:32:22 +08:00
|
|
|
|
2018-05-31 16:22:45 +08:00
|
|
|
/* Following the reset, we need to reload the CSB read/write pointers */
|
2018-06-29 04:12:08 +08:00
|
|
|
reset_csb_pointers(&engine->execlists);
|
2018-05-31 16:22:45 +08:00
|
|
|
|
2018-06-29 04:12:05 +08:00
|
|
|
spin_unlock_irqrestore(&engine->timeline.lock, flags);
|
2018-03-02 21:12:46 +08:00
|
|
|
|
2018-03-02 22:32:45 +08:00
|
|
|
/*
|
|
|
|
* If the request was innocent, we leave the request in the ELSP
|
2017-02-07 23:24:37 +08:00
|
|
|
* and will try to replay it on restarting. The context image may
|
|
|
|
* have been corrupted by the reset, in which case we may have
|
|
|
|
* to service a new GPU hang, but more likely we can continue on
|
|
|
|
* without impact.
|
|
|
|
*
|
|
|
|
* If the request was guilty, we presume the context is corrupt
|
|
|
|
* and have to at least restore the RING register in the context
|
|
|
|
* image back to the expected values to skip over the guilty request.
|
|
|
|
*/
|
2017-09-17 04:44:14 +08:00
|
|
|
if (!request || request->fence.error != -EIO)
|
2017-02-07 23:24:37 +08:00
|
|
|
return;
|
2016-09-09 21:11:53 +08:00
|
|
|
|
2018-03-02 22:32:45 +08:00
|
|
|
/*
|
|
|
|
* We want a simple context + ring to execute the breadcrumb update.
|
2016-10-05 04:11:26 +08:00
|
|
|
* We cannot rely on the context being intact across the GPU hang,
|
|
|
|
* so clear it and rebuild just what we need for the breadcrumb.
|
|
|
|
* All pending requests for this context will be zapped, and any
|
|
|
|
* future request will be after userspace has had the opportunity
|
|
|
|
* to recreate its own state.
|
|
|
|
*/
|
2018-05-18 05:26:32 +08:00
|
|
|
regs = request->hw_context->lrc_reg_state;
|
2018-05-18 17:02:11 +08:00
|
|
|
if (engine->pinned_default_state) {
|
|
|
|
memcpy(regs, /* skip restoring the vanilla PPHWSP */
|
|
|
|
engine->pinned_default_state + LRC_STATE_PN * PAGE_SIZE,
|
|
|
|
engine->context_size - PAGE_SIZE);
|
2018-04-28 19:15:32 +08:00
|
|
|
}
|
2018-05-18 05:26:30 +08:00
|
|
|
execlists_init_reg_state(regs,
|
|
|
|
request->gem_context, engine, request->ring);
|
2016-10-05 04:11:26 +08:00
|
|
|
|
2016-09-09 21:11:53 +08:00
|
|
|
/* Move the RING_HEAD onto the breadcrumb, past the hanging batch */
|
2018-04-28 19:15:32 +08:00
|
|
|
regs[CTX_RING_BUFFER_START + 1] = i915_ggtt_offset(request->ring->vma);
|
2016-10-05 04:11:26 +08:00
|
|
|
|
2018-06-11 19:08:45 +08:00
|
|
|
request->ring->head = intel_ring_wrap(request->ring, request->postfix);
|
|
|
|
regs[CTX_RING_HEAD + 1] = request->ring->head;
|
|
|
|
|
2016-09-09 21:11:53 +08:00
|
|
|
intel_ring_update_space(request->ring);
|
|
|
|
|
2016-10-05 04:11:26 +08:00
|
|
|
/* Reset WaIdleLiteRestore:bdw,skl as well */
|
2017-09-29 03:38:59 +08:00
|
|
|
unwind_wa_tail(request);
|
2016-09-09 21:11:53 +08:00
|
|
|
}
|
|
|
|
|
2018-05-17 02:33:51 +08:00
|
|
|
static void execlists_reset_finish(struct intel_engine_cs *engine)
|
|
|
|
{
|
2018-06-04 15:34:40 +08:00
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
|
|
|
|
|
|
|
/* After a GPU reset, we may have requests to replay */
|
2018-06-29 15:53:20 +08:00
|
|
|
if (!RB_EMPTY_ROOT(&execlists->queue.rb_root))
|
2018-06-04 15:34:40 +08:00
|
|
|
tasklet_schedule(&execlists->tasklet);
|
|
|
|
|
2018-05-22 18:19:37 +08:00
|
|
|
/*
|
|
|
|
* Flush the tasklet while we still have the forcewake to be sure
|
|
|
|
* that it is not allowed to sleep before we restart and reload a
|
|
|
|
* context.
|
|
|
|
*
|
|
|
|
* As before (with execlists_reset_prepare) we rely on the caller
|
|
|
|
* serialising multiple attempts to reset so that we know that we
|
|
|
|
* are the only one manipulating tasklet state.
|
|
|
|
*/
|
2018-06-04 15:34:40 +08:00
|
|
|
__tasklet_enable_sync_once(&execlists->tasklet);
|
2018-05-17 02:33:51 +08:00
|
|
|
|
|
|
|
GEM_TRACE("%s\n", engine->name);
|
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static int intel_logical_ring_emit_pdps(struct i915_request *rq)
|
2015-06-26 20:46:14 +08:00
|
|
|
{
|
2018-05-18 05:26:30 +08:00
|
|
|
struct i915_hw_ppgtt *ppgtt = rq->gem_context->ppgtt;
|
2018-02-21 17:56:36 +08:00
|
|
|
struct intel_engine_cs *engine = rq->engine;
|
2017-02-28 23:28:10 +08:00
|
|
|
const int num_lri_cmds = GEN8_3LVL_PDPES * 2;
|
2017-02-14 19:32:42 +08:00
|
|
|
u32 *cs;
|
|
|
|
int i;
|
2015-06-26 20:46:14 +08:00
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
cs = intel_ring_begin(rq, num_lri_cmds * 2 + 2);
|
2017-02-14 19:32:42 +08:00
|
|
|
if (IS_ERR(cs))
|
|
|
|
return PTR_ERR(cs);
|
2015-06-26 20:46:14 +08:00
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
*cs++ = MI_LOAD_REGISTER_IMM(num_lri_cmds);
|
2017-02-28 23:28:10 +08:00
|
|
|
for (i = GEN8_3LVL_PDPES - 1; i >= 0; i--) {
|
2015-06-26 20:46:14 +08:00
|
|
|
const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i);
|
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
*cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(engine, i));
|
|
|
|
*cs++ = upper_32_bits(pd_daddr);
|
|
|
|
*cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(engine, i));
|
|
|
|
*cs++ = lower_32_bits(pd_daddr);
|
2015-06-26 20:46:14 +08:00
|
|
|
}
|
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
*cs++ = MI_NOOP;
|
2018-02-21 17:56:36 +08:00
|
|
|
intel_ring_advance(rq, cs);
|
2015-06-26 20:46:14 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static int gen8_emit_bb_start(struct i915_request *rq,
|
2016-08-03 05:50:27 +08:00
|
|
|
u64 offset, u32 len,
|
2017-02-28 23:28:08 +08:00
|
|
|
const unsigned int flags)
|
2014-07-25 00:04:32 +08:00
|
|
|
{
|
2017-02-14 19:32:42 +08:00
|
|
|
u32 *cs;
|
2014-07-25 00:04:32 +08:00
|
|
|
int ret;
|
|
|
|
|
2015-06-26 20:46:14 +08:00
|
|
|
/* Don't rely in hw updating PDPs, specially in lite-restore.
|
|
|
|
* Ideally, we should set Force PD Restore in ctx descriptor,
|
|
|
|
* but we can't. Force Restore would be a second option, but
|
|
|
|
* it is unsafe in case of lite-restore (because the ctx is
|
2015-07-30 18:06:23 +08:00
|
|
|
* not idle). PML4 is allocated during ppgtt init so this is
|
|
|
|
* not needed in 48-bit.*/
|
2018-05-18 05:26:30 +08:00
|
|
|
if (rq->gem_context->ppgtt &&
|
|
|
|
(intel_engine_flag(rq->engine) & rq->gem_context->ppgtt->pd_dirty_rings) &&
|
2018-06-05 23:37:58 +08:00
|
|
|
!i915_vm_is_48bit(&rq->gem_context->ppgtt->vm) &&
|
2018-02-21 17:56:36 +08:00
|
|
|
!intel_vgpu_active(rq->i915)) {
|
|
|
|
ret = intel_logical_ring_emit_pdps(rq);
|
2017-02-28 23:28:08 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
2015-06-26 20:46:14 +08:00
|
|
|
|
2018-05-18 05:26:30 +08:00
|
|
|
rq->gem_context->ppgtt->pd_dirty_rings &= ~intel_engine_flag(rq->engine);
|
2015-06-26 20:46:14 +08:00
|
|
|
}
|
|
|
|
|
2018-05-04 03:54:16 +08:00
|
|
|
cs = intel_ring_begin(rq, 6);
|
2017-02-14 19:32:42 +08:00
|
|
|
if (IS_ERR(cs))
|
|
|
|
return PTR_ERR(cs);
|
2014-07-25 00:04:32 +08:00
|
|
|
|
2017-10-06 03:10:05 +08:00
|
|
|
/*
|
|
|
|
* WaDisableCtxRestoreArbitration:bdw,chv
|
|
|
|
*
|
|
|
|
* We don't need to perform MI_ARB_ENABLE as often as we do (in
|
|
|
|
* particular all the gen that do not need the w/a at all!), if we
|
|
|
|
* took care to make sure that on every switch into this context
|
|
|
|
* (both ordinary and for preemption) that arbitrartion was enabled
|
|
|
|
* we would be fine. However, there doesn't seem to be a downside to
|
|
|
|
* being paranoid and making sure it is set before each batch and
|
|
|
|
* every context-switch.
|
|
|
|
*
|
|
|
|
* Note that if we fail to enable arbitration before the request
|
|
|
|
* is complete, then we do not see the context-switch interrupt and
|
|
|
|
* the engine hangs (with RING_HEAD == RING_TAIL).
|
|
|
|
*
|
|
|
|
* That satisfies both the GPGPU w/a and our heavy-handed paranoia.
|
|
|
|
*/
|
2017-10-04 04:34:49 +08:00
|
|
|
*cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
|
|
|
|
|
2014-07-25 00:04:32 +08:00
|
|
|
/* FIXME(BDW): Address space and security selectors. */
|
2017-02-28 23:28:08 +08:00
|
|
|
*cs++ = MI_BATCH_BUFFER_START_GEN8 |
|
|
|
|
(flags & I915_DISPATCH_SECURE ? 0 : BIT(8)) |
|
|
|
|
(flags & I915_DISPATCH_RS ? MI_BATCH_RESOURCE_STREAMER : 0);
|
2017-02-14 19:32:42 +08:00
|
|
|
*cs++ = lower_32_bits(offset);
|
|
|
|
*cs++ = upper_32_bits(offset);
|
2018-05-04 03:54:16 +08:00
|
|
|
|
|
|
|
*cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
|
|
|
|
*cs++ = MI_NOOP;
|
2018-02-21 17:56:36 +08:00
|
|
|
intel_ring_advance(rq, cs);
|
2014-07-25 00:04:32 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-07-02 00:23:27 +08:00
|
|
|
static void gen8_logical_ring_enable_irq(struct intel_engine_cs *engine)
|
2014-07-25 00:04:31 +08:00
|
|
|
{
|
2016-05-06 22:40:21 +08:00
|
|
|
struct drm_i915_private *dev_priv = engine->i915;
|
2016-07-02 00:23:27 +08:00
|
|
|
I915_WRITE_IMR(engine,
|
|
|
|
~(engine->irq_enable_mask | engine->irq_keep_mask));
|
|
|
|
POSTING_READ_FW(RING_IMR(engine->mmio_base));
|
2014-07-25 00:04:31 +08:00
|
|
|
}
|
|
|
|
|
2016-07-02 00:23:27 +08:00
|
|
|
static void gen8_logical_ring_disable_irq(struct intel_engine_cs *engine)
|
2014-07-25 00:04:31 +08:00
|
|
|
{
|
2016-05-06 22:40:21 +08:00
|
|
|
struct drm_i915_private *dev_priv = engine->i915;
|
2016-07-02 00:23:27 +08:00
|
|
|
I915_WRITE_IMR(engine, ~engine->irq_keep_mask);
|
2014-07-25 00:04:31 +08:00
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static int gen8_emit_flush(struct i915_request *request, u32 mode)
|
2014-07-25 00:04:28 +08:00
|
|
|
{
|
2017-02-14 19:32:42 +08:00
|
|
|
u32 cmd, *cs;
|
2014-07-25 00:04:28 +08:00
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
cs = intel_ring_begin(request, 4);
|
|
|
|
if (IS_ERR(cs))
|
|
|
|
return PTR_ERR(cs);
|
2014-07-25 00:04:28 +08:00
|
|
|
|
|
|
|
cmd = MI_FLUSH_DW + 1;
|
|
|
|
|
2015-01-22 21:42:00 +08:00
|
|
|
/* We always require a command barrier so that subsequent
|
|
|
|
* commands, such as breadcrumb interrupts, are strictly ordered
|
|
|
|
* wrt the contents of the write cache being flushed to memory
|
|
|
|
* (and thus being coherent from the CPU).
|
|
|
|
*/
|
|
|
|
cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
|
|
|
|
|
2016-08-03 05:50:25 +08:00
|
|
|
if (mode & EMIT_INVALIDATE) {
|
2015-01-22 21:42:00 +08:00
|
|
|
cmd |= MI_INVALIDATE_TLB;
|
2016-08-03 05:50:19 +08:00
|
|
|
if (request->engine->id == VCS)
|
2015-01-22 21:42:00 +08:00
|
|
|
cmd |= MI_INVALIDATE_BSD;
|
2014-07-25 00:04:28 +08:00
|
|
|
}
|
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
*cs++ = cmd;
|
|
|
|
*cs++ = I915_GEM_HWS_SCRATCH_ADDR | MI_FLUSH_DW_USE_GTT;
|
|
|
|
*cs++ = 0; /* upper addr */
|
|
|
|
*cs++ = 0; /* value */
|
|
|
|
intel_ring_advance(request, cs);
|
2014-07-25 00:04:28 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static int gen8_emit_flush_render(struct i915_request *request,
|
2016-08-03 05:50:25 +08:00
|
|
|
u32 mode)
|
2014-07-25 00:04:28 +08:00
|
|
|
{
|
2016-08-03 05:50:18 +08:00
|
|
|
struct intel_engine_cs *engine = request->engine;
|
2016-08-15 17:49:07 +08:00
|
|
|
u32 scratch_addr =
|
|
|
|
i915_ggtt_offset(engine->scratch) + 2 * CACHELINE_BYTES;
|
2016-06-07 22:19:10 +08:00
|
|
|
bool vf_flush_wa = false, dc_flush_wa = false;
|
2017-02-14 19:32:42 +08:00
|
|
|
u32 *cs, flags = 0;
|
2016-06-07 22:19:10 +08:00
|
|
|
int len;
|
2014-07-25 00:04:28 +08:00
|
|
|
|
|
|
|
flags |= PIPE_CONTROL_CS_STALL;
|
|
|
|
|
2016-08-03 05:50:25 +08:00
|
|
|
if (mode & EMIT_FLUSH) {
|
2014-07-25 00:04:28 +08:00
|
|
|
flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
|
|
|
|
flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
|
2016-01-14 10:59:39 +08:00
|
|
|
flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
|
2015-08-21 23:08:41 +08:00
|
|
|
flags |= PIPE_CONTROL_FLUSH_ENABLE;
|
2014-07-25 00:04:28 +08:00
|
|
|
}
|
|
|
|
|
2016-08-03 05:50:25 +08:00
|
|
|
if (mode & EMIT_INVALIDATE) {
|
2014-07-25 00:04:28 +08:00
|
|
|
flags |= PIPE_CONTROL_TLB_INVALIDATE;
|
|
|
|
flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
|
|
|
|
flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
|
|
|
|
flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
|
|
|
|
flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
|
|
|
|
flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
|
|
|
|
flags |= PIPE_CONTROL_QW_WRITE;
|
|
|
|
flags |= PIPE_CONTROL_GLOBAL_GTT_IVB;
|
|
|
|
|
2015-12-18 01:49:57 +08:00
|
|
|
/*
|
|
|
|
* On GEN9: before VF_CACHE_INVALIDATE we need to emit a NULL
|
|
|
|
* pipe control.
|
|
|
|
*/
|
2016-05-06 22:40:21 +08:00
|
|
|
if (IS_GEN9(request->i915))
|
2015-12-18 01:49:57 +08:00
|
|
|
vf_flush_wa = true;
|
2016-06-07 22:19:10 +08:00
|
|
|
|
|
|
|
/* WaForGAMHang:kbl */
|
|
|
|
if (IS_KBL_REVID(request->i915, 0, KBL_REVID_B0))
|
|
|
|
dc_flush_wa = true;
|
2015-12-18 01:49:57 +08:00
|
|
|
}
|
2015-01-26 05:27:11 +08:00
|
|
|
|
2016-06-07 22:19:10 +08:00
|
|
|
len = 6;
|
|
|
|
|
|
|
|
if (vf_flush_wa)
|
|
|
|
len += 6;
|
|
|
|
|
|
|
|
if (dc_flush_wa)
|
|
|
|
len += 12;
|
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
cs = intel_ring_begin(request, len);
|
|
|
|
if (IS_ERR(cs))
|
|
|
|
return PTR_ERR(cs);
|
2014-07-25 00:04:28 +08:00
|
|
|
|
2017-02-16 20:23:25 +08:00
|
|
|
if (vf_flush_wa)
|
|
|
|
cs = gen8_emit_pipe_control(cs, 0, 0);
|
2015-01-26 05:27:11 +08:00
|
|
|
|
2017-02-16 20:23:25 +08:00
|
|
|
if (dc_flush_wa)
|
|
|
|
cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_DC_FLUSH_ENABLE,
|
|
|
|
0);
|
2016-06-07 22:19:10 +08:00
|
|
|
|
2017-02-16 20:23:25 +08:00
|
|
|
cs = gen8_emit_pipe_control(cs, flags, scratch_addr);
|
2016-06-07 22:19:10 +08:00
|
|
|
|
2017-02-16 20:23:25 +08:00
|
|
|
if (dc_flush_wa)
|
|
|
|
cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_CS_STALL, 0);
|
2016-06-07 22:19:10 +08:00
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
intel_ring_advance(request, cs);
|
2014-07-25 00:04:28 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-01-20 21:43:35 +08:00
|
|
|
/*
|
|
|
|
* Reserve space for 2 NOOPs at the end of each request to be
|
|
|
|
* used as a workaround for not being allowed to do lite
|
|
|
|
* restore with HEAD==TAIL (WaIdleLiteRestore).
|
|
|
|
*/
|
2018-02-21 17:56:36 +08:00
|
|
|
static void gen8_emit_wa_tail(struct i915_request *request, u32 *cs)
|
2014-07-25 00:04:27 +08:00
|
|
|
{
|
drm/i915/execlists: Preemption!
When we write to ELSP, it triggers a context preemption at the earliest
arbitration point (3DPRIMITIVE, some PIPECONTROLs, a few other
operations and the explicit MI_ARB_CHECK). If this is to the same
context, it triggers a LITE_RESTORE where the RING_TAIL is merely
updated (used currently to chain requests from the same context
together, avoiding bubbles). However, if it is to a different context, a
full context-switch is performed and it will start to execute the new
context saving the image of the old for later execution.
Previously we avoided preemption by only submitting a new context when
the old was idle. But now we wish embrace it, and if the new request has
a higher priority than the currently executing request, we write to the
ELSP regardless, thus triggering preemption, but we tell the GPU to
switch to our special preemption context (not the target). In the
context-switch interrupt handler, we know that the previous contexts
have finished execution and so can unwind all the incomplete requests
and compute the new highest priority request to execute.
It would be feasible to avoid the switch-to-idle intermediate by
programming the ELSP with the target context. The difficulty is in
tracking which request that should be whilst maintaining the dependency
change, the error comes in with coalesced requests. As we only track the
most recent request and its priority, we may run into the issue of being
tricked in preempting a high priority request that was followed by a
low priority request from the same context (e.g. for PI); worse still
that earlier request may be our own dependency and the order then broken
by preemption. By injecting the switch-to-idle and then recomputing the
priority queue, we avoid the issue with tracking in-flight coalesced
requests. Having tried the preempt-to-busy approach, and failed to find
a way around the coalesced priority issue, Michal's original proposal to
inject an idle context (based on handling GuC preemption) succeeds.
The current heuristic for deciding when to preempt are only if the new
request is of higher priority, and has the privileged priority of
greater than 0. Note that the scheduler remains unfair!
v2: Disable for gen8 (bdw/bsw) as we need additional w/a for GPGPU.
Since, the feature is now conditional and not always available when we
have a scheduler, make it known via the HAS_SCHEDULER GETPARAM (now a
capability mask).
v3: Stylistic tweaks.
v4: Appease Joonas with a snippet of kerneldoc, only to fuel to fire of
the preempt vs preempting debate.
Suggested-by: Michal Winiarski <michal.winiarski@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Ben Widawsky <benjamin.widawsky@intel.com>
Cc: Zhenyu Wang <zhenyuw@linux.intel.com>
Cc: Zhi Wang <zhi.a.wang@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-8-chris@chris-wilson.co.uk
2017-10-04 04:34:52 +08:00
|
|
|
/* Ensure there's always at least one preemption point per-request. */
|
|
|
|
*cs++ = MI_ARB_CHECK;
|
2017-02-14 19:32:42 +08:00
|
|
|
*cs++ = MI_NOOP;
|
|
|
|
request->wa_tail = intel_ring_offset(request, cs);
|
2016-10-28 20:58:52 +08:00
|
|
|
}
|
2014-07-25 00:04:27 +08:00
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static void gen8_emit_breadcrumb(struct i915_request *request, u32 *cs)
|
2016-10-28 20:58:52 +08:00
|
|
|
{
|
2016-01-20 21:43:35 +08:00
|
|
|
/* w/a: bit 5 needs to be zero for MI_FLUSH_DW address. */
|
|
|
|
BUILD_BUG_ON(I915_GEM_HWS_INDEX_ADDR & (1 << 5));
|
2014-07-25 00:04:27 +08:00
|
|
|
|
2017-10-26 04:00:15 +08:00
|
|
|
cs = gen8_emit_ggtt_write(cs, request->global_seqno,
|
|
|
|
intel_hws_seqno_address(request->engine));
|
2017-02-14 19:32:42 +08:00
|
|
|
*cs++ = MI_USER_INTERRUPT;
|
2018-05-04 03:54:16 +08:00
|
|
|
*cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
|
2017-02-14 19:32:42 +08:00
|
|
|
request->tail = intel_ring_offset(request, cs);
|
2017-03-27 21:14:12 +08:00
|
|
|
assert_ring_tail_valid(request->ring, request->tail);
|
2016-10-28 20:58:52 +08:00
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
gen8_emit_wa_tail(request, cs);
|
2016-01-20 21:43:35 +08:00
|
|
|
}
|
2016-10-28 20:58:51 +08:00
|
|
|
static const int gen8_emit_breadcrumb_sz = 6 + WA_TAIL_DWORDS;
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static void gen8_emit_breadcrumb_rcs(struct i915_request *request, u32 *cs)
|
2016-01-20 21:43:35 +08:00
|
|
|
{
|
2016-04-12 21:51:55 +08:00
|
|
|
/* We're using qword write, seqno should be aligned to 8 bytes. */
|
|
|
|
BUILD_BUG_ON(I915_GEM_HWS_INDEX & 1);
|
|
|
|
|
2017-10-26 04:00:15 +08:00
|
|
|
cs = gen8_emit_ggtt_write_rcs(cs, request->global_seqno,
|
|
|
|
intel_hws_seqno_address(request->engine));
|
2017-02-14 19:32:42 +08:00
|
|
|
*cs++ = MI_USER_INTERRUPT;
|
2018-05-04 03:54:16 +08:00
|
|
|
*cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
|
2017-02-14 19:32:42 +08:00
|
|
|
request->tail = intel_ring_offset(request, cs);
|
2017-03-27 21:14:12 +08:00
|
|
|
assert_ring_tail_valid(request->ring, request->tail);
|
2016-10-28 20:58:52 +08:00
|
|
|
|
2017-02-14 19:32:42 +08:00
|
|
|
gen8_emit_wa_tail(request, cs);
|
2014-07-25 00:04:27 +08:00
|
|
|
}
|
2017-10-26 04:00:15 +08:00
|
|
|
static const int gen8_emit_breadcrumb_rcs_sz = 8 + WA_TAIL_DWORDS;
|
2016-10-28 20:58:51 +08:00
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
static int gen8_init_rcs_context(struct i915_request *rq)
|
2014-12-02 20:50:48 +08:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2018-04-11 00:12:47 +08:00
|
|
|
ret = intel_ctx_workarounds_emit(rq);
|
2014-12-02 20:50:48 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
ret = intel_rcs_context_init_mocs(rq);
|
2015-07-11 01:13:11 +08:00
|
|
|
/*
|
|
|
|
* Failing to program the MOCS is non-fatal.The system will not
|
|
|
|
* run at peak performance. So generate an error and carry on.
|
|
|
|
*/
|
|
|
|
if (ret)
|
|
|
|
DRM_ERROR("MOCS failed to program: expect performance issues.\n");
|
|
|
|
|
2018-02-21 17:56:36 +08:00
|
|
|
return i915_gem_render_state_emit(rq);
|
2014-12-02 20:50:48 +08:00
|
|
|
}
|
|
|
|
|
2014-07-25 00:04:48 +08:00
|
|
|
/**
|
|
|
|
* intel_logical_ring_cleanup() - deallocate the Engine Command Streamer
|
2016-06-03 21:02:17 +08:00
|
|
|
* @engine: Engine Command Streamer.
|
2014-07-25 00:04:48 +08:00
|
|
|
*/
|
2016-03-16 19:00:37 +08:00
|
|
|
void intel_logical_ring_cleanup(struct intel_engine_cs *engine)
|
2014-07-25 00:04:22 +08:00
|
|
|
{
|
2014-10-31 20:00:26 +08:00
|
|
|
struct drm_i915_private *dev_priv;
|
2014-07-25 00:04:30 +08:00
|
|
|
|
drm/i915: Move execlists irq handler to a bottom half
Doing a lot of work in the interrupt handler introduces huge
latencies to the system as a whole.
Most dramatic effect can be seen by running an all engine
stress test like igt/gem_exec_nop/all where, when the kernel
config is lean enough, the whole system can be brought into
multi-second periods of complete non-interactivty. That can
look for example like this:
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:3:143]
Modules linked in: [redacted for brevity]
CPU: 0 PID: 143 Comm: kworker/u8:3 Tainted: G U L 4.5.0-160321+ #183
Hardware name: Intel Corporation Broadwell Client platform/WhiteTip Mountain 1
Workqueue: i915 gen6_pm_rps_work [i915]
task: ffff8800aae88000 ti: ffff8800aae90000 task.ti: ffff8800aae90000
RIP: 0010:[<ffffffff8104a3c2>] [<ffffffff8104a3c2>] __do_softirq+0x72/0x1d0
RSP: 0000:ffff88014f403f38 EFLAGS: 00000206
RAX: ffff8800aae94000 RBX: 0000000000000000 RCX: 00000000000006e0
RDX: 0000000000000020 RSI: 0000000004208060 RDI: 0000000000215d80
RBP: ffff88014f403f80 R08: 0000000b1b42c180 R09: 0000000000000022
R10: 0000000000000004 R11: 00000000ffffffff R12: 000000000000a030
R13: 0000000000000082 R14: ffff8800aa4d0080 R15: 0000000000000082
FS: 0000000000000000(0000) GS:ffff88014f400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa53b90c000 CR3: 0000000001a0a000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
042080601b33869f ffff8800aae94000 00000000fffc2678 ffff88010000000a
0000000000000000 000000000000a030 0000000000005302 ffff8800aa4d0080
0000000000000206 ffff88014f403f90 ffffffff8104a716 ffff88014f403fa8
Call Trace:
<IRQ>
[<ffffffff8104a716>] irq_exit+0x86/0x90
[<ffffffff81031e7d>] smp_apic_timer_interrupt+0x3d/0x50
[<ffffffff814f3eac>] apic_timer_interrupt+0x7c/0x90
<EOI>
[<ffffffffa01c5b40>] ? gen8_write64+0x1a0/0x1a0 [i915]
[<ffffffff814f2b39>] ? _raw_spin_unlock_irqrestore+0x9/0x20
[<ffffffffa01c5c44>] gen8_write32+0x104/0x1a0 [i915]
[<ffffffff8132c6a2>] ? n_tty_receive_buf_common+0x372/0xae0
[<ffffffffa017cc9e>] gen6_set_rps_thresholds+0x1be/0x330 [i915]
[<ffffffffa017eaf0>] gen6_set_rps+0x70/0x200 [i915]
[<ffffffffa0185375>] intel_set_rps+0x25/0x30 [i915]
[<ffffffffa01768fd>] gen6_pm_rps_work+0x10d/0x2e0 [i915]
[<ffffffff81063852>] ? finish_task_switch+0x72/0x1c0
[<ffffffff8105ab29>] process_one_work+0x139/0x350
[<ffffffff8105b186>] worker_thread+0x126/0x490
[<ffffffff8105b060>] ? rescuer_thread+0x320/0x320
[<ffffffff8105fa64>] kthread+0xc4/0xe0
[<ffffffff8105f9a0>] ? kthread_create_on_node+0x170/0x170
[<ffffffff814f351f>] ret_from_fork+0x3f/0x70
[<ffffffff8105f9a0>] ? kthread_create_on_node+0x170/0x170
I could not explain, or find a code path, which would explain
a +20 second lockup, but from some instrumentation it was
apparent the interrupts off proportion of time was between
10-25% under heavy load which is quite bad.
When a interrupt "cliff" is reached, which was >~320k irq/s on
my machine, the whole system goes into a terrible state of the
above described multi-second lockups.
By moving the GT interrupt handling to a tasklet in a most
simple way, the problem above disappears completely.
Testing the effect on sytem-wide latencies using
igt/gem_syslatency shows the following before this patch:
gem_syslatency: cycles=1532739, latency mean=416531.829us max=2499237us
gem_syslatency: cycles=1839434, latency mean=1458099.157us max=4998944us
gem_syslatency: cycles=1432570, latency mean=2688.451us max=1201185us
gem_syslatency: cycles=1533543, latency mean=416520.499us max=2498886us
This shows that the unrelated process is experiencing huge
delays in its wake-up latency. After the patch the results
look like this:
gem_syslatency: cycles=808907, latency mean=53.133us max=1640us
gem_syslatency: cycles=862154, latency mean=62.778us max=2117us
gem_syslatency: cycles=856039, latency mean=58.079us max=2123us
gem_syslatency: cycles=841683, latency mean=56.914us max=1667us
Showing a huge improvement in the unrelated process wake-up
latency. It also shows an approximate halving in the number
of total empty batches submitted during the test. This may
not be worrying since the test puts the driver under
a very unrealistic load with ncpu threads doing empty batch
submission to all GPU engines each.
Another benefit compared to the hard-irq handling is that now
work on all engines can be dispatched in parallel since we can
have up to number of CPUs active tasklets. (While previously
a single hard-irq would serially dispatch on one engine after
another.)
More interesting scenario with regards to throughput is
"gem_latency -n 100" which shows 25% better throughput and
CPU usage, and 14% better dispatch latencies.
I did not find any gains or regressions with Synmark2 or
GLbench under light testing. More benchmarking is certainly
required.
v2:
* execlists_lock should be taken as spin_lock_bh when
queuing work from userspace now. (Chris Wilson)
* uncore.lock must be taken with spin_lock_irq when
submitting requests since that now runs from either
softirq or process context.
v3:
* Expanded commit message with more testing data;
* converted missed locking sites to _bh;
* added execlist_lock comment. (Chris Wilson)
v4:
* Mention dispatch parallelism in commit. (Chris Wilson)
* Do not hold uncore.lock over MMIO reads since the block
is already serialised per-engine via the tasklet itself.
(Chris Wilson)
* intel_lrc_irq_handler should be static. (Chris Wilson)
* Cancel/sync the tasklet on GPU reset. (Chris Wilson)
* Document and WARN that tasklet cannot be active/pending
on engine cleanup. (Chris Wilson/Imre Deak)
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Imre Deak <imre.deak@intel.com>
Testcase: igt/gem_exec_nop/all
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=94350
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/1459768316-6670-1-git-send-email-tvrtko.ursulin@linux.intel.com
2016-04-04 19:11:56 +08:00
|
|
|
/*
|
|
|
|
* Tasklet cannot be active at this point due intel_mark_active/idle
|
|
|
|
* so this is just for documentation.
|
|
|
|
*/
|
2017-11-16 21:32:37 +08:00
|
|
|
if (WARN_ON(test_bit(TASKLET_STATE_SCHED,
|
|
|
|
&engine->execlists.tasklet.state)))
|
|
|
|
tasklet_kill(&engine->execlists.tasklet);
|
drm/i915: Move execlists irq handler to a bottom half
Doing a lot of work in the interrupt handler introduces huge
latencies to the system as a whole.
Most dramatic effect can be seen by running an all engine
stress test like igt/gem_exec_nop/all where, when the kernel
config is lean enough, the whole system can be brought into
multi-second periods of complete non-interactivty. That can
look for example like this:
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:3:143]
Modules linked in: [redacted for brevity]
CPU: 0 PID: 143 Comm: kworker/u8:3 Tainted: G U L 4.5.0-160321+ #183
Hardware name: Intel Corporation Broadwell Client platform/WhiteTip Mountain 1
Workqueue: i915 gen6_pm_rps_work [i915]
task: ffff8800aae88000 ti: ffff8800aae90000 task.ti: ffff8800aae90000
RIP: 0010:[<ffffffff8104a3c2>] [<ffffffff8104a3c2>] __do_softirq+0x72/0x1d0
RSP: 0000:ffff88014f403f38 EFLAGS: 00000206
RAX: ffff8800aae94000 RBX: 0000000000000000 RCX: 00000000000006e0
RDX: 0000000000000020 RSI: 0000000004208060 RDI: 0000000000215d80
RBP: ffff88014f403f80 R08: 0000000b1b42c180 R09: 0000000000000022
R10: 0000000000000004 R11: 00000000ffffffff R12: 000000000000a030
R13: 0000000000000082 R14: ffff8800aa4d0080 R15: 0000000000000082
FS: 0000000000000000(0000) GS:ffff88014f400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa53b90c000 CR3: 0000000001a0a000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
042080601b33869f ffff8800aae94000 00000000fffc2678 ffff88010000000a
0000000000000000 000000000000a030 0000000000005302 ffff8800aa4d0080
0000000000000206 ffff88014f403f90 ffffffff8104a716 ffff88014f403fa8
Call Trace:
<IRQ>
[<ffffffff8104a716>] irq_exit+0x86/0x90
[<ffffffff81031e7d>] smp_apic_timer_interrupt+0x3d/0x50
[<ffffffff814f3eac>] apic_timer_interrupt+0x7c/0x90
<EOI>
[<ffffffffa01c5b40>] ? gen8_write64+0x1a0/0x1a0 [i915]
[<ffffffff814f2b39>] ? _raw_spin_unlock_irqrestore+0x9/0x20
[<ffffffffa01c5c44>] gen8_write32+0x104/0x1a0 [i915]
[<ffffffff8132c6a2>] ? n_tty_receive_buf_common+0x372/0xae0
[<ffffffffa017cc9e>] gen6_set_rps_thresholds+0x1be/0x330 [i915]
[<ffffffffa017eaf0>] gen6_set_rps+0x70/0x200 [i915]
[<ffffffffa0185375>] intel_set_rps+0x25/0x30 [i915]
[<ffffffffa01768fd>] gen6_pm_rps_work+0x10d/0x2e0 [i915]
[<ffffffff81063852>] ? finish_task_switch+0x72/0x1c0
[<ffffffff8105ab29>] process_one_work+0x139/0x350
[<ffffffff8105b186>] worker_thread+0x126/0x490
[<ffffffff8105b060>] ? rescuer_thread+0x320/0x320
[<ffffffff8105fa64>] kthread+0xc4/0xe0
[<ffffffff8105f9a0>] ? kthread_create_on_node+0x170/0x170
[<ffffffff814f351f>] ret_from_fork+0x3f/0x70
[<ffffffff8105f9a0>] ? kthread_create_on_node+0x170/0x170
I could not explain, or find a code path, which would explain
a +20 second lockup, but from some instrumentation it was
apparent the interrupts off proportion of time was between
10-25% under heavy load which is quite bad.
When a interrupt "cliff" is reached, which was >~320k irq/s on
my machine, the whole system goes into a terrible state of the
above described multi-second lockups.
By moving the GT interrupt handling to a tasklet in a most
simple way, the problem above disappears completely.
Testing the effect on sytem-wide latencies using
igt/gem_syslatency shows the following before this patch:
gem_syslatency: cycles=1532739, latency mean=416531.829us max=2499237us
gem_syslatency: cycles=1839434, latency mean=1458099.157us max=4998944us
gem_syslatency: cycles=1432570, latency mean=2688.451us max=1201185us
gem_syslatency: cycles=1533543, latency mean=416520.499us max=2498886us
This shows that the unrelated process is experiencing huge
delays in its wake-up latency. After the patch the results
look like this:
gem_syslatency: cycles=808907, latency mean=53.133us max=1640us
gem_syslatency: cycles=862154, latency mean=62.778us max=2117us
gem_syslatency: cycles=856039, latency mean=58.079us max=2123us
gem_syslatency: cycles=841683, latency mean=56.914us max=1667us
Showing a huge improvement in the unrelated process wake-up
latency. It also shows an approximate halving in the number
of total empty batches submitted during the test. This may
not be worrying since the test puts the driver under
a very unrealistic load with ncpu threads doing empty batch
submission to all GPU engines each.
Another benefit compared to the hard-irq handling is that now
work on all engines can be dispatched in parallel since we can
have up to number of CPUs active tasklets. (While previously
a single hard-irq would serially dispatch on one engine after
another.)
More interesting scenario with regards to throughput is
"gem_latency -n 100" which shows 25% better throughput and
CPU usage, and 14% better dispatch latencies.
I did not find any gains or regressions with Synmark2 or
GLbench under light testing. More benchmarking is certainly
required.
v2:
* execlists_lock should be taken as spin_lock_bh when
queuing work from userspace now. (Chris Wilson)
* uncore.lock must be taken with spin_lock_irq when
submitting requests since that now runs from either
softirq or process context.
v3:
* Expanded commit message with more testing data;
* converted missed locking sites to _bh;
* added execlist_lock comment. (Chris Wilson)
v4:
* Mention dispatch parallelism in commit. (Chris Wilson)
* Do not hold uncore.lock over MMIO reads since the block
is already serialised per-engine via the tasklet itself.
(Chris Wilson)
* intel_lrc_irq_handler should be static. (Chris Wilson)
* Cancel/sync the tasklet on GPU reset. (Chris Wilson)
* Document and WARN that tasklet cannot be active/pending
on engine cleanup. (Chris Wilson/Imre Deak)
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Imre Deak <imre.deak@intel.com>
Testcase: igt/gem_exec_nop/all
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=94350
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: http://patchwork.freedesktop.org/patch/msgid/1459768316-6670-1-git-send-email-tvrtko.ursulin@linux.intel.com
2016-04-04 19:11:56 +08:00
|
|
|
|
2016-05-06 22:40:21 +08:00
|
|
|
dev_priv = engine->i915;
|
2014-10-31 20:00:26 +08:00
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
if (engine->buffer) {
|
|
|
|
WARN_ON((I915_READ_MODE(engine) & MODE_IDLE) == 0);
|
2015-12-08 23:02:36 +08:00
|
|
|
}
|
2014-07-25 00:04:23 +08:00
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
if (engine->cleanup)
|
|
|
|
engine->cleanup(engine);
|
2014-07-25 00:04:23 +08:00
|
|
|
|
drm/i915: Unify active context tracking between legacy/execlists/guc
The requests conversion introduced a nasty bug where we could generate a
new request in the middle of constructing a request if we needed to idle
the system in order to evict space for a context. The request to idle
would be executed (and waited upon) before the current one, creating a
minor havoc in the seqno accounting, as we will consider the current
request to already be completed (prior to deferred seqno assignment) but
ring->last_retired_head would have been updated and still could allow
us to overwrite the current request before execution.
We also employed two different mechanisms to track the active context
until it was switched out. The legacy method allowed for waiting upon an
active context (it could forcibly evict any vma, including context's),
but the execlists method took a step backwards by pinning the vma for
the entire active lifespan of the context (the only way to evict was to
idle the entire GPU, not individual contexts). However, to circumvent
the tricky issue of locking (i.e. we cannot take struct_mutex at the
time of i915_gem_request_submit(), where we would want to move the
previous context onto the active tracker and unpin it), we take the
execlists approach and keep the contexts pinned until retirement.
The benefit of the execlists approach, more important for execlists than
legacy, was the reduction in work in pinning the context for each
request - as the context was kept pinned until idle, it could short
circuit the pinning for all active contexts.
We introduce new engine vfuncs to pin and unpin the context
respectively. The context is pinned at the start of the request, and
only unpinned when the following request is retired (this ensures that
the context is idle and coherent in main memory before we unpin it). We
move the engine->last_context tracking into the retirement itself
(rather than during request submission) in order to allow the submission
to be reordered or unwound without undue difficultly.
And finally an ulterior motive for unifying context handling was to
prepare for mock requests.
v2: Rename to last_retired_context, split out legacy_context tracking
for MI_SET_CONTEXT.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161218153724.8439-3-chris@chris-wilson.co.uk
2016-12-18 23:37:20 +08:00
|
|
|
intel_engine_cleanup_common(engine);
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2017-02-17 15:58:59 +08:00
|
|
|
lrc_destroy_wa_ctx(engine);
|
2018-01-02 23:12:34 +08:00
|
|
|
|
2016-05-06 22:40:21 +08:00
|
|
|
engine->i915 = NULL;
|
drm/i915: Allocate intel_engine_cs structure only for the enabled engines
With the possibility of addition of many more number of rings in future,
the drm_i915_private structure could bloat as an array, of type
intel_engine_cs, is embedded inside it.
struct intel_engine_cs engine[I915_NUM_ENGINES];
Though this is still fine as generally there is only a single instance of
drm_i915_private structure used, but not all of the possible rings would be
enabled or active on most of the platforms. Some memory can be saved by
allocating intel_engine_cs structure only for the enabled/active engines.
Currently the engine/ring ID is kept static and dev_priv->engine[] is simply
indexed using the enums defined in intel_engine_id.
To save memory and continue using the static engine/ring IDs, 'engine' is
defined as an array of pointers.
struct intel_engine_cs *engine[I915_NUM_ENGINES];
dev_priv->engine[engine_ID] will be NULL for disabled engine instances.
There is a text size reduction of 928 bytes, from 1028200 to 1027272, for
i915.o file (but for i915.ko file text size remain same as 1193131 bytes).
v2:
- Remove the engine iterator field added in drm_i915_private structure,
instead pass a local iterator variable to the for_each_engine**
macros. (Chris)
- Do away with intel_engine_initialized() and instead directly use the
NULL pointer check on engine pointer. (Chris)
v3:
- Remove for_each_engine_id() macro, as the updated macro for_each_engine()
can be used in place of it. (Chris)
- Protect the access to Render engine Fault register with a NULL check, as
engine specific init is done later in Driver load sequence.
v4:
- Use !!dev_priv->engine[VCS] style for the engine check in getparam. (Chris)
- Kill the superfluous init_engine_lists().
v5:
- Cleanup the intel_engines_init() & intel_engines_setup(), with respect to
allocation of intel_engine_cs structure. (Chris)
v6:
- Rebase.
v7:
- Optimize the for_each_engine_masked() macro. (Chris)
- Change the type of 'iter' local variable to enum intel_engine_id. (Chris)
- Rebase.
v8: Rebase.
v9: Rebase.
v10:
- For index calculation use engine ID instead of pointer based arithmetic in
intel_engine_sync_index() as engine pointers are not contiguous now (Chris)
- For appropriateness, rename local enum variable 'iter' to 'id'. (Joonas)
- Use for_each_engine macro for cleanup in intel_engines_init() and remove
check for NULL engine pointer in cleanup() routines. (Joonas)
v11: Rebase.
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Akash Goel <akash.goel@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/1476378888-7372-1-git-send-email-akash.goel@intel.com
2016-10-14 01:14:48 +08:00
|
|
|
dev_priv->engine[engine->id] = NULL;
|
|
|
|
kfree(engine);
|
2014-07-25 00:04:22 +08:00
|
|
|
}
|
|
|
|
|
2018-07-18 04:29:32 +08:00
|
|
|
void intel_execlists_set_default_submission(struct intel_engine_cs *engine)
|
2016-08-03 05:50:31 +08:00
|
|
|
{
|
2017-03-17 01:13:03 +08:00
|
|
|
engine->submit_request = execlists_submit_request;
|
2017-09-16 01:31:00 +08:00
|
|
|
engine->cancel_requests = execlists_cancel_requests;
|
2017-03-17 01:13:03 +08:00
|
|
|
engine->schedule = execlists_schedule;
|
2017-11-16 21:32:37 +08:00
|
|
|
engine->execlists.tasklet.func = execlists_submission_tasklet;
|
2017-10-25 22:39:41 +08:00
|
|
|
|
2018-05-17 02:33:52 +08:00
|
|
|
engine->reset.prepare = execlists_reset_prepare;
|
|
|
|
|
2017-10-25 22:39:41 +08:00
|
|
|
engine->park = NULL;
|
|
|
|
engine->unpark = NULL;
|
2017-11-29 18:28:05 +08:00
|
|
|
|
|
|
|
engine->flags |= I915_ENGINE_SUPPORTS_STATS;
|
2018-04-04 02:35:37 +08:00
|
|
|
if (engine->i915->preempt_context)
|
|
|
|
engine->flags |= I915_ENGINE_HAS_PREEMPTION;
|
2018-02-08 05:05:43 +08:00
|
|
|
|
|
|
|
engine->i915->caps.scheduler =
|
|
|
|
I915_SCHEDULER_CAP_ENABLED |
|
|
|
|
I915_SCHEDULER_CAP_PRIORITY;
|
2018-04-04 02:35:37 +08:00
|
|
|
if (intel_engine_has_preemption(engine))
|
2018-02-08 05:05:43 +08:00
|
|
|
engine->i915->caps.scheduler |= I915_SCHEDULER_CAP_PREEMPTION;
|
2016-08-03 05:50:31 +08:00
|
|
|
}
|
|
|
|
|
2016-01-13 01:32:34 +08:00
|
|
|
static void
|
2016-05-06 22:40:20 +08:00
|
|
|
logical_ring_default_vfuncs(struct intel_engine_cs *engine)
|
2016-01-13 01:32:34 +08:00
|
|
|
{
|
|
|
|
/* Default vfuncs which can be overriden by each engine. */
|
2016-03-16 19:00:37 +08:00
|
|
|
engine->init_hw = gen8_init_common_ring;
|
2018-05-17 02:33:51 +08:00
|
|
|
|
|
|
|
engine->reset.prepare = execlists_reset_prepare;
|
|
|
|
engine->reset.reset = execlists_reset;
|
|
|
|
engine->reset.finish = execlists_reset_finish;
|
drm/i915: Unify active context tracking between legacy/execlists/guc
The requests conversion introduced a nasty bug where we could generate a
new request in the middle of constructing a request if we needed to idle
the system in order to evict space for a context. The request to idle
would be executed (and waited upon) before the current one, creating a
minor havoc in the seqno accounting, as we will consider the current
request to already be completed (prior to deferred seqno assignment) but
ring->last_retired_head would have been updated and still could allow
us to overwrite the current request before execution.
We also employed two different mechanisms to track the active context
until it was switched out. The legacy method allowed for waiting upon an
active context (it could forcibly evict any vma, including context's),
but the execlists method took a step backwards by pinning the vma for
the entire active lifespan of the context (the only way to evict was to
idle the entire GPU, not individual contexts). However, to circumvent
the tricky issue of locking (i.e. we cannot take struct_mutex at the
time of i915_gem_request_submit(), where we would want to move the
previous context onto the active tracker and unpin it), we take the
execlists approach and keep the contexts pinned until retirement.
The benefit of the execlists approach, more important for execlists than
legacy, was the reduction in work in pinning the context for each
request - as the context was kept pinned until idle, it could short
circuit the pinning for all active contexts.
We introduce new engine vfuncs to pin and unpin the context
respectively. The context is pinned at the start of the request, and
only unpinned when the following request is retired (this ensures that
the context is idle and coherent in main memory before we unpin it). We
move the engine->last_context tracking into the retirement itself
(rather than during request submission) in order to allow the submission
to be reordered or unwound without undue difficultly.
And finally an ulterior motive for unifying context handling was to
prepare for mock requests.
v2: Rename to last_retired_context, split out legacy_context tracking
for MI_SET_CONTEXT.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161218153724.8439-3-chris@chris-wilson.co.uk
2016-12-18 23:37:20 +08:00
|
|
|
|
|
|
|
engine->context_pin = execlists_context_pin;
|
2016-12-18 23:37:24 +08:00
|
|
|
engine->request_alloc = execlists_request_alloc;
|
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
engine->emit_flush = gen8_emit_flush;
|
2016-10-28 20:58:50 +08:00
|
|
|
engine->emit_breadcrumb = gen8_emit_breadcrumb;
|
2016-10-28 20:58:51 +08:00
|
|
|
engine->emit_breadcrumb_sz = gen8_emit_breadcrumb_sz;
|
2017-03-17 01:13:03 +08:00
|
|
|
|
2018-07-18 04:29:32 +08:00
|
|
|
engine->set_default_submission = intel_execlists_set_default_submission;
|
2016-08-03 05:50:31 +08:00
|
|
|
|
2018-03-03 00:14:56 +08:00
|
|
|
if (INTEL_GEN(engine->i915) < 11) {
|
|
|
|
engine->irq_enable = gen8_logical_ring_enable_irq;
|
|
|
|
engine->irq_disable = gen8_logical_ring_disable_irq;
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* TODO: On Gen11 interrupt masks need to be clear
|
|
|
|
* to allow C6 entry. Keep interrupts enabled at
|
|
|
|
* and take the hit of generating extra interrupts
|
|
|
|
* until a more refined solution exists.
|
|
|
|
*/
|
|
|
|
}
|
2016-03-16 19:00:37 +08:00
|
|
|
engine->emit_bb_start = gen8_emit_bb_start;
|
2016-01-13 01:32:34 +08:00
|
|
|
}
|
|
|
|
|
2016-01-13 01:32:35 +08:00
|
|
|
static inline void
|
2016-07-13 23:03:35 +08:00
|
|
|
logical_ring_default_irqs(struct intel_engine_cs *engine)
|
2016-01-13 01:32:35 +08:00
|
|
|
{
|
2018-03-15 02:26:53 +08:00
|
|
|
unsigned int shift = 0;
|
|
|
|
|
|
|
|
if (INTEL_GEN(engine->i915) < 11) {
|
|
|
|
const u8 irq_shifts[] = {
|
|
|
|
[RCS] = GEN8_RCS_IRQ_SHIFT,
|
|
|
|
[BCS] = GEN8_BCS_IRQ_SHIFT,
|
|
|
|
[VCS] = GEN8_VCS1_IRQ_SHIFT,
|
|
|
|
[VCS2] = GEN8_VCS2_IRQ_SHIFT,
|
|
|
|
[VECS] = GEN8_VECS_IRQ_SHIFT,
|
|
|
|
};
|
|
|
|
|
|
|
|
shift = irq_shifts[engine->id];
|
|
|
|
}
|
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
engine->irq_enable_mask = GT_RENDER_USER_INTERRUPT << shift;
|
|
|
|
engine->irq_keep_mask = GT_CONTEXT_SWITCH_INTERRUPT << shift;
|
2016-01-13 01:32:35 +08:00
|
|
|
}
|
|
|
|
|
2016-07-13 23:03:36 +08:00
|
|
|
static void
|
|
|
|
logical_ring_setup(struct intel_engine_cs *engine)
|
|
|
|
{
|
2016-07-13 23:03:41 +08:00
|
|
|
intel_engine_setup_common(engine);
|
|
|
|
|
2016-07-13 23:03:36 +08:00
|
|
|
/* Intentionally left blank. */
|
|
|
|
engine->buffer = NULL;
|
|
|
|
|
2017-11-16 21:32:37 +08:00
|
|
|
tasklet_init(&engine->execlists.tasklet,
|
|
|
|
execlists_submission_tasklet, (unsigned long)engine);
|
2016-07-13 23:03:36 +08:00
|
|
|
|
|
|
|
logical_ring_default_vfuncs(engine);
|
|
|
|
logical_ring_default_irqs(engine);
|
|
|
|
}
|
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
static bool csb_force_mmio(struct drm_i915_private *i915)
|
|
|
|
{
|
|
|
|
/* Older GVT emulation depends upon intercepting CSB mmio */
|
|
|
|
return intel_vgpu_active(i915) && !intel_vgpu_has_hwsp_emulation(i915);
|
|
|
|
}
|
|
|
|
|
2017-09-13 16:56:02 +08:00
|
|
|
static int logical_ring_init(struct intel_engine_cs *engine)
|
2016-06-23 21:52:41 +08:00
|
|
|
{
|
2018-06-29 04:12:07 +08:00
|
|
|
struct drm_i915_private *i915 = engine->i915;
|
|
|
|
struct intel_engine_execlists * const execlists = &engine->execlists;
|
2016-06-23 21:52:41 +08:00
|
|
|
int ret;
|
|
|
|
|
2016-07-13 23:03:41 +08:00
|
|
|
ret = intel_engine_init_common(engine);
|
2016-06-23 21:52:41 +08:00
|
|
|
if (ret)
|
|
|
|
goto error;
|
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
if (HAS_LOGICAL_RING_ELSQ(i915)) {
|
|
|
|
execlists->submit_reg = i915->regs +
|
2018-03-03 00:14:59 +08:00
|
|
|
i915_mmio_reg_offset(RING_EXECLIST_SQ_CONTENTS(engine));
|
2018-06-29 04:12:07 +08:00
|
|
|
execlists->ctrl_reg = i915->regs +
|
2018-03-03 00:14:59 +08:00
|
|
|
i915_mmio_reg_offset(RING_EXECLIST_CONTROL(engine));
|
|
|
|
} else {
|
2018-06-29 04:12:07 +08:00
|
|
|
execlists->submit_reg = i915->regs +
|
2018-03-03 00:14:59 +08:00
|
|
|
i915_mmio_reg_offset(RING_ELSP(engine));
|
|
|
|
}
|
2018-01-02 23:12:33 +08:00
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
execlists->preempt_complete_status = ~0u;
|
|
|
|
if (i915->preempt_context) {
|
2018-04-30 21:15:01 +08:00
|
|
|
struct intel_context *ce =
|
2018-06-29 04:12:07 +08:00
|
|
|
to_intel_context(i915->preempt_context, engine);
|
2018-04-30 21:15:01 +08:00
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
execlists->preempt_complete_status =
|
2018-04-30 21:15:01 +08:00
|
|
|
upper_32_bits(ce->lrc_desc);
|
|
|
|
}
|
2018-02-08 05:05:44 +08:00
|
|
|
|
2018-06-29 04:12:07 +08:00
|
|
|
execlists->csb_read =
|
|
|
|
i915->regs + i915_mmio_reg_offset(RING_CONTEXT_STATUS_PTR(engine));
|
|
|
|
if (csb_force_mmio(i915)) {
|
|
|
|
execlists->csb_status = (u32 __force *)
|
|
|
|
(i915->regs + i915_mmio_reg_offset(RING_CONTEXT_STATUS_BUF_LO(engine, 0)));
|
|
|
|
|
|
|
|
execlists->csb_write = (u32 __force *)execlists->csb_read;
|
2018-06-29 04:12:08 +08:00
|
|
|
execlists->csb_write_reset =
|
|
|
|
_MASKED_FIELD(GEN8_CSB_WRITE_PTR_MASK,
|
|
|
|
GEN8_CSB_ENTRIES - 1);
|
2018-06-29 04:12:07 +08:00
|
|
|
} else {
|
|
|
|
execlists->csb_status =
|
|
|
|
&engine->status_page.page_addr[I915_HWS_CSB_BUF0_INDEX];
|
|
|
|
|
|
|
|
execlists->csb_write =
|
|
|
|
&engine->status_page.page_addr[intel_hws_csb_write_index(i915)];
|
2018-06-29 04:12:08 +08:00
|
|
|
execlists->csb_write_reset = GEN8_CSB_ENTRIES - 1;
|
2018-06-29 04:12:07 +08:00
|
|
|
}
|
2018-06-29 04:12:08 +08:00
|
|
|
reset_csb_pointers(execlists);
|
2018-05-31 16:22:45 +08:00
|
|
|
|
2016-06-23 21:52:41 +08:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
error:
|
|
|
|
intel_logical_ring_cleanup(engine);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2016-07-13 23:03:40 +08:00
|
|
|
int logical_render_ring_init(struct intel_engine_cs *engine)
|
2016-06-23 21:52:41 +08:00
|
|
|
{
|
|
|
|
struct drm_i915_private *dev_priv = engine->i915;
|
|
|
|
int ret;
|
|
|
|
|
2016-07-13 23:03:36 +08:00
|
|
|
logical_ring_setup(engine);
|
|
|
|
|
2016-06-23 21:52:41 +08:00
|
|
|
if (HAS_L3_DPF(dev_priv))
|
|
|
|
engine->irq_keep_mask |= GT_RENDER_L3_PARITY_ERROR_INTERRUPT;
|
|
|
|
|
|
|
|
/* Override some for render ring. */
|
|
|
|
if (INTEL_GEN(dev_priv) >= 9)
|
|
|
|
engine->init_hw = gen9_init_render_ring;
|
|
|
|
else
|
|
|
|
engine->init_hw = gen8_init_render_ring;
|
|
|
|
engine->init_context = gen8_init_rcs_context;
|
|
|
|
engine->emit_flush = gen8_emit_flush_render;
|
2017-10-26 04:00:15 +08:00
|
|
|
engine->emit_breadcrumb = gen8_emit_breadcrumb_rcs;
|
|
|
|
engine->emit_breadcrumb_sz = gen8_emit_breadcrumb_rcs_sz;
|
2016-06-23 21:52:41 +08:00
|
|
|
|
2017-01-10 22:47:34 +08:00
|
|
|
ret = intel_engine_create_scratch(engine, PAGE_SIZE);
|
2016-06-23 21:52:41 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
ret = intel_init_workaround_bb(engine);
|
|
|
|
if (ret) {
|
|
|
|
/*
|
|
|
|
* We continue even if we fail to initialize WA batch
|
|
|
|
* because we only expect rare glitches but nothing
|
|
|
|
* critical to prevent us from using GPU
|
|
|
|
*/
|
|
|
|
DRM_ERROR("WA batch buffer initialization failed: %d\n",
|
|
|
|
ret);
|
|
|
|
}
|
|
|
|
|
2016-12-16 21:18:42 +08:00
|
|
|
return logical_ring_init(engine);
|
2016-06-23 21:52:41 +08:00
|
|
|
}
|
|
|
|
|
2016-07-13 23:03:40 +08:00
|
|
|
int logical_xcs_ring_init(struct intel_engine_cs *engine)
|
2016-07-13 23:03:36 +08:00
|
|
|
{
|
|
|
|
logical_ring_setup(engine);
|
|
|
|
|
|
|
|
return logical_ring_init(engine);
|
2014-07-25 00:04:22 +08:00
|
|
|
}
|
|
|
|
|
2015-02-14 00:27:56 +08:00
|
|
|
static u32
|
2016-05-06 22:40:21 +08:00
|
|
|
make_rpcs(struct drm_i915_private *dev_priv)
|
2015-02-14 00:27:56 +08:00
|
|
|
{
|
|
|
|
u32 rpcs = 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* No explicit RPCS request is needed to ensure full
|
|
|
|
* slice/subslice/EU enablement prior to Gen9.
|
|
|
|
*/
|
2016-05-06 22:40:21 +08:00
|
|
|
if (INTEL_GEN(dev_priv) < 9)
|
2015-02-14 00:27:56 +08:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Starting in Gen9, render power gating can leave
|
|
|
|
* slice/subslice/EU in a partially enabled state. We
|
|
|
|
* must make an explicit request through RPCS for full
|
|
|
|
* enablement.
|
|
|
|
*/
|
2016-09-01 00:13:02 +08:00
|
|
|
if (INTEL_INFO(dev_priv)->sseu.has_slice_pg) {
|
2015-02-14 00:27:56 +08:00
|
|
|
rpcs |= GEN8_RPCS_S_CNT_ENABLE;
|
2016-09-01 00:13:04 +08:00
|
|
|
rpcs |= hweight8(INTEL_INFO(dev_priv)->sseu.slice_mask) <<
|
2015-02-14 00:27:56 +08:00
|
|
|
GEN8_RPCS_S_CNT_SHIFT;
|
|
|
|
rpcs |= GEN8_RPCS_ENABLE;
|
|
|
|
}
|
|
|
|
|
2016-09-01 00:13:02 +08:00
|
|
|
if (INTEL_INFO(dev_priv)->sseu.has_subslice_pg) {
|
2015-02-14 00:27:56 +08:00
|
|
|
rpcs |= GEN8_RPCS_SS_CNT_ENABLE;
|
2018-03-06 20:28:52 +08:00
|
|
|
rpcs |= hweight8(INTEL_INFO(dev_priv)->sseu.subslice_mask[0]) <<
|
2015-02-14 00:27:56 +08:00
|
|
|
GEN8_RPCS_SS_CNT_SHIFT;
|
|
|
|
rpcs |= GEN8_RPCS_ENABLE;
|
|
|
|
}
|
|
|
|
|
2016-09-01 00:13:02 +08:00
|
|
|
if (INTEL_INFO(dev_priv)->sseu.has_eu_pg) {
|
|
|
|
rpcs |= INTEL_INFO(dev_priv)->sseu.eu_per_subslice <<
|
2015-02-14 00:27:56 +08:00
|
|
|
GEN8_RPCS_EU_MIN_SHIFT;
|
2016-09-01 00:13:02 +08:00
|
|
|
rpcs |= INTEL_INFO(dev_priv)->sseu.eu_per_subslice <<
|
2015-02-14 00:27:56 +08:00
|
|
|
GEN8_RPCS_EU_MAX_SHIFT;
|
|
|
|
rpcs |= GEN8_RPCS_ENABLE;
|
|
|
|
}
|
|
|
|
|
|
|
|
return rpcs;
|
|
|
|
}
|
|
|
|
|
2016-03-16 19:00:37 +08:00
|
|
|
static u32 intel_lr_indirect_ctx_offset(struct intel_engine_cs *engine)
|
2016-02-23 18:31:49 +08:00
|
|
|
{
|
|
|
|
u32 indirect_ctx_offset;
|
|
|
|
|
2016-05-06 22:40:21 +08:00
|
|
|
switch (INTEL_GEN(engine->i915)) {
|
2016-02-23 18:31:49 +08:00
|
|
|
default:
|
2016-05-06 22:40:21 +08:00
|
|
|
MISSING_CASE(INTEL_GEN(engine->i915));
|
2016-02-23 18:31:49 +08:00
|
|
|
/* fall through */
|
2018-03-03 00:15:00 +08:00
|
|
|
case 11:
|
|
|
|
indirect_ctx_offset =
|
|
|
|
GEN11_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
|
|
|
|
break;
|
2017-06-07 04:30:38 +08:00
|
|
|
case 10:
|
|
|
|
indirect_ctx_offset =
|
|
|
|
GEN10_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
|
|
|
|
break;
|
2016-02-23 18:31:49 +08:00
|
|
|
case 9:
|
|
|
|
indirect_ctx_offset =
|
|
|
|
GEN9_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
|
|
|
|
break;
|
|
|
|
case 8:
|
|
|
|
indirect_ctx_offset =
|
|
|
|
GEN8_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
return indirect_ctx_offset;
|
|
|
|
}
|
|
|
|
|
2017-02-21 17:58:39 +08:00
|
|
|
static void execlists_init_reg_state(u32 *regs,
|
2016-10-05 04:11:26 +08:00
|
|
|
struct i915_gem_context *ctx,
|
|
|
|
struct intel_engine_cs *engine,
|
|
|
|
struct intel_ring *ring)
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
{
|
2016-10-05 04:11:26 +08:00
|
|
|
struct drm_i915_private *dev_priv = engine->i915;
|
|
|
|
struct i915_hw_ppgtt *ppgtt = ctx->ppgtt ?: dev_priv->mm.aliasing_ppgtt;
|
2017-02-21 17:58:39 +08:00
|
|
|
u32 base = engine->mmio_base;
|
2018-05-18 05:26:32 +08:00
|
|
|
bool rcs = engine->class == RENDER_CLASS;
|
2017-02-21 17:58:39 +08:00
|
|
|
|
|
|
|
/* A context is actually a big batch buffer with several
|
|
|
|
* MI_LOAD_REGISTER_IMM commands followed by (reg, value) pairs. The
|
|
|
|
* values we are setting here are only for the first context restore:
|
|
|
|
* on a subsequent save, the GPU will recreate this batchbuffer with new
|
|
|
|
* values (including all the missing MI_LOAD_REGISTER_IMM commands that
|
|
|
|
* we are not initializing here).
|
|
|
|
*/
|
|
|
|
regs[CTX_LRI_HEADER_0] = MI_LOAD_REGISTER_IMM(rcs ? 14 : 11) |
|
|
|
|
MI_LRI_FORCE_POSTED;
|
|
|
|
|
|
|
|
CTX_REG(regs, CTX_CONTEXT_CONTROL, RING_CONTEXT_CONTROL(engine),
|
2018-01-25 19:24:42 +08:00
|
|
|
_MASKED_BIT_DISABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT |
|
|
|
|
CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT) |
|
2017-02-21 17:58:39 +08:00
|
|
|
_MASKED_BIT_ENABLE(CTX_CTRL_INHIBIT_SYN_CTX_SWITCH |
|
|
|
|
(HAS_RESOURCE_STREAMER(dev_priv) ?
|
|
|
|
CTX_CTRL_RS_CTX_ENABLE : 0)));
|
|
|
|
CTX_REG(regs, CTX_RING_HEAD, RING_HEAD(base), 0);
|
|
|
|
CTX_REG(regs, CTX_RING_TAIL, RING_TAIL(base), 0);
|
|
|
|
CTX_REG(regs, CTX_RING_BUFFER_START, RING_START(base), 0);
|
|
|
|
CTX_REG(regs, CTX_RING_BUFFER_CONTROL, RING_CTL(base),
|
|
|
|
RING_CTL_SIZE(ring->size) | RING_VALID);
|
|
|
|
CTX_REG(regs, CTX_BB_HEAD_U, RING_BBADDR_UDW(base), 0);
|
|
|
|
CTX_REG(regs, CTX_BB_HEAD_L, RING_BBADDR(base), 0);
|
|
|
|
CTX_REG(regs, CTX_BB_STATE, RING_BBSTATE(base), RING_BB_PPGTT);
|
|
|
|
CTX_REG(regs, CTX_SECOND_BB_HEAD_U, RING_SBBADDR_UDW(base), 0);
|
|
|
|
CTX_REG(regs, CTX_SECOND_BB_HEAD_L, RING_SBBADDR(base), 0);
|
|
|
|
CTX_REG(regs, CTX_SECOND_BB_STATE, RING_SBBSTATE(base), 0);
|
|
|
|
if (rcs) {
|
2017-09-21 21:54:43 +08:00
|
|
|
struct i915_ctx_workarounds *wa_ctx = &engine->wa_ctx;
|
|
|
|
|
2017-02-21 17:58:39 +08:00
|
|
|
CTX_REG(regs, CTX_RCS_INDIRECT_CTX, RING_INDIRECT_CTX(base), 0);
|
|
|
|
CTX_REG(regs, CTX_RCS_INDIRECT_CTX_OFFSET,
|
|
|
|
RING_INDIRECT_CTX_OFFSET(base), 0);
|
2017-09-21 21:54:43 +08:00
|
|
|
if (wa_ctx->indirect_ctx.size) {
|
2016-08-15 17:49:07 +08:00
|
|
|
u32 ggtt_offset = i915_ggtt_offset(wa_ctx->vma);
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2017-02-21 17:58:39 +08:00
|
|
|
regs[CTX_RCS_INDIRECT_CTX + 1] =
|
2017-02-17 15:58:59 +08:00
|
|
|
(ggtt_offset + wa_ctx->indirect_ctx.offset) |
|
|
|
|
(wa_ctx->indirect_ctx.size / CACHELINE_BYTES);
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2017-02-21 17:58:39 +08:00
|
|
|
regs[CTX_RCS_INDIRECT_CTX_OFFSET + 1] =
|
2016-03-16 19:00:37 +08:00
|
|
|
intel_lr_indirect_ctx_offset(engine) << 6;
|
2017-09-21 21:54:43 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
CTX_REG(regs, CTX_BB_PER_CTX_PTR, RING_BB_PER_CTX_PTR(base), 0);
|
|
|
|
if (wa_ctx->per_ctx.size) {
|
|
|
|
u32 ggtt_offset = i915_ggtt_offset(wa_ctx->vma);
|
2015-06-20 02:07:01 +08:00
|
|
|
|
2017-02-21 17:58:39 +08:00
|
|
|
regs[CTX_BB_PER_CTX_PTR + 1] =
|
2017-02-17 15:58:59 +08:00
|
|
|
(ggtt_offset + wa_ctx->per_ctx.offset) | 0x01;
|
2015-06-20 02:07:01 +08:00
|
|
|
}
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
}
|
2017-02-21 17:58:39 +08:00
|
|
|
|
|
|
|
regs[CTX_LRI_HEADER_1] = MI_LOAD_REGISTER_IMM(9) | MI_LRI_FORCE_POSTED;
|
|
|
|
|
|
|
|
CTX_REG(regs, CTX_CTX_TIMESTAMP, RING_CTX_TIMESTAMP(base), 0);
|
2015-11-05 05:20:11 +08:00
|
|
|
/* PDP values well be assigned later if needed */
|
2017-02-21 17:58:39 +08:00
|
|
|
CTX_REG(regs, CTX_PDP3_UDW, GEN8_RING_PDP_UDW(engine, 3), 0);
|
|
|
|
CTX_REG(regs, CTX_PDP3_LDW, GEN8_RING_PDP_LDW(engine, 3), 0);
|
|
|
|
CTX_REG(regs, CTX_PDP2_UDW, GEN8_RING_PDP_UDW(engine, 2), 0);
|
|
|
|
CTX_REG(regs, CTX_PDP2_LDW, GEN8_RING_PDP_LDW(engine, 2), 0);
|
|
|
|
CTX_REG(regs, CTX_PDP1_UDW, GEN8_RING_PDP_UDW(engine, 1), 0);
|
|
|
|
CTX_REG(regs, CTX_PDP1_LDW, GEN8_RING_PDP_LDW(engine, 1), 0);
|
|
|
|
CTX_REG(regs, CTX_PDP0_UDW, GEN8_RING_PDP_UDW(engine, 0), 0);
|
|
|
|
CTX_REG(regs, CTX_PDP0_LDW, GEN8_RING_PDP_LDW(engine, 0), 0);
|
drm/i915/gen8: Dynamic page table allocations
This finishes off the dynamic page tables allocations, in the legacy 3
level style that already exists. Most everything has already been setup
to this point, the patch finishes off the enabling by setting the
appropriate function pointers.
In LRC mode, contexts need to know the PDPs when they are populated. With
dynamic page table allocations, these PDPs may not exist yet. Check if
PDPs have been allocated and use the scratch page if they do not exist yet.
Before submission, update the PDPs in the logic ring context as PDPs
have been allocated.
v2: Update aliasing/true ppgtt allocate/teardown/clear functions for
gen 6 & 7.
v3: Rebase.
v4: Remove BUG() from ppgtt_unbind_vma, but keep checking that either
teardown_va_range or clear_range functions exist (Daniel).
v5: Similar to gen6, in init, gen8_ppgtt_clear_range call is only needed
for aliasing ppgtt. Zombie tracking was originally added for teardown
function and is no longer required.
v6: Update err_out case in gen8_alloc_va_range (missed from lastest
rebase).
v7: Rebase after s/page_tables/page_table/.
v8: Updated scratch_pt check after scratch flag was removed in previous
patch.
v9: Note that lrc mode needs to be updated to support init state without
any PDP.
v10: Unmap correct page_table in gen8_alloc_va_range's error case, clean-up
gen8_aliasing_ppgtt_init (remove duplicated map), and initialize PTs
during page table allocation.
v11: Squashed LRC enabling commit, otherwise LRC mode would be left broken
until it was updated to handle the init case without any PDP.
v12: Do not overallocate new_pts bitmap, make alloc_gen8_temp_bitmaps
static and don't abuse of inline functions. (Mika)
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
Signed-off-by: Michel Thierry <michel.thierry@intel.com> (v2+)
Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-04-08 19:13:34 +08:00
|
|
|
|
2018-06-05 23:37:58 +08:00
|
|
|
if (ppgtt && i915_vm_is_48bit(&ppgtt->vm)) {
|
2015-07-30 18:06:23 +08:00
|
|
|
/* 64b PPGTT (48bit canonical)
|
|
|
|
* PDP0_DESCRIPTOR contains the base address to PML4 and
|
|
|
|
* other PDP Descriptors are ignored.
|
|
|
|
*/
|
2017-02-21 17:58:39 +08:00
|
|
|
ASSIGN_CTX_PML4(ppgtt, regs);
|
2015-07-30 18:06:23 +08:00
|
|
|
}
|
|
|
|
|
2017-02-21 17:58:39 +08:00
|
|
|
if (rcs) {
|
|
|
|
regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
|
|
|
|
CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
|
|
|
|
make_rpcs(dev_priv));
|
2017-06-13 19:23:03 +08:00
|
|
|
|
|
|
|
i915_oa_init_reg_state(engine, ctx, regs);
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
}
|
2016-10-05 04:11:26 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static int
|
|
|
|
populate_lr_context(struct i915_gem_context *ctx,
|
|
|
|
struct drm_i915_gem_object *ctx_obj,
|
|
|
|
struct intel_engine_cs *engine,
|
|
|
|
struct intel_ring *ring)
|
|
|
|
{
|
|
|
|
void *vaddr;
|
2017-11-10 22:26:33 +08:00
|
|
|
u32 *regs;
|
2016-10-05 04:11:26 +08:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
ret = i915_gem_object_set_to_cpu_domain(ctx_obj, true);
|
|
|
|
if (ret) {
|
|
|
|
DRM_DEBUG_DRIVER("Could not set to CPU domain\n");
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
vaddr = i915_gem_object_pin_map(ctx_obj, I915_MAP_WB);
|
|
|
|
if (IS_ERR(vaddr)) {
|
|
|
|
ret = PTR_ERR(vaddr);
|
|
|
|
DRM_DEBUG_DRIVER("Could not map object pages! (%d)\n", ret);
|
|
|
|
return ret;
|
|
|
|
}
|
2016-10-28 20:58:35 +08:00
|
|
|
ctx_obj->mm.dirty = true;
|
2016-10-05 04:11:26 +08:00
|
|
|
|
2017-11-10 22:26:33 +08:00
|
|
|
if (engine->default_state) {
|
|
|
|
/*
|
|
|
|
* We only want to copy over the template context state;
|
|
|
|
* skipping over the headers reserved for GuC communication,
|
|
|
|
* leaving those as zero.
|
|
|
|
*/
|
|
|
|
const unsigned long start = LRC_HEADER_PAGES * PAGE_SIZE;
|
|
|
|
void *defaults;
|
|
|
|
|
|
|
|
defaults = i915_gem_object_pin_map(engine->default_state,
|
|
|
|
I915_MAP_WB);
|
2018-03-01 19:46:39 +08:00
|
|
|
if (IS_ERR(defaults)) {
|
|
|
|
ret = PTR_ERR(defaults);
|
|
|
|
goto err_unpin_ctx;
|
|
|
|
}
|
2017-11-10 22:26:33 +08:00
|
|
|
|
|
|
|
memcpy(vaddr + start, defaults + start, engine->context_size);
|
|
|
|
i915_gem_object_unpin_map(engine->default_state);
|
|
|
|
}
|
|
|
|
|
2016-10-05 04:11:26 +08:00
|
|
|
/* The second page of the context object contains some fields which must
|
|
|
|
* be set up prior to the first execution. */
|
2017-11-10 22:26:33 +08:00
|
|
|
regs = vaddr + LRC_STATE_PN * PAGE_SIZE;
|
|
|
|
execlists_init_reg_state(regs, ctx, engine, ring);
|
|
|
|
if (!engine->default_state)
|
|
|
|
regs[CTX_CONTEXT_CONTROL + 1] |=
|
|
|
|
_MASKED_BIT_ENABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT);
|
2018-03-03 00:14:59 +08:00
|
|
|
if (ctx == ctx->i915->preempt_context && INTEL_GEN(engine->i915) < 11)
|
2018-01-24 05:04:12 +08:00
|
|
|
regs[CTX_CONTEXT_CONTROL + 1] |=
|
|
|
|
_MASKED_BIT_ENABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT |
|
|
|
|
CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT);
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
|
2018-03-01 19:46:39 +08:00
|
|
|
err_unpin_ctx:
|
2016-04-12 22:40:42 +08:00
|
|
|
i915_gem_object_unpin_map(ctx_obj);
|
2018-03-01 19:46:39 +08:00
|
|
|
return ret;
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
}
|
|
|
|
|
2016-05-24 21:53:34 +08:00
|
|
|
static int execlists_context_deferred_alloc(struct i915_gem_context *ctx,
|
2018-05-18 05:26:32 +08:00
|
|
|
struct intel_engine_cs *engine,
|
|
|
|
struct intel_context *ce)
|
2014-07-25 00:04:12 +08:00
|
|
|
{
|
drm/i915/bdw: A bit more advanced LR context alloc/free
Now that we have the ability to allocate our own context backing objects
and we have multiplexed one of them per engine inside the context structs,
we can finally allocate and free them correctly.
Regarding the context size, reading the register to calculate the sizes
can work, I think, however the docs are very clear about the actual
context sizes on GEN8, so just hardcode that and use it.
v2: Rebased on top of the Full PPGTT series. It is important to notice
that at this point we have one global default context per engine, all
of them using the aliasing PPGTT (as opposed to the single global
default context we have with legacy HW contexts).
v3:
- Go back to one single global default context, this time with multiple
backing objects inside.
- Use different context sizes for non-render engines, as suggested by
Damien (still hardcoded, since the information about the context size
registers in the BSpec is, well, *lacking*).
- Render ctx size is 20 (or 19) pages, but not 21 (caught by Damien).
- Move default context backing object creation to intel_init_ring (so
that we don't waste memory in rings that might not get initialized).
v4:
- Reuse the HW legacy context init/fini.
- Create a separate free function.
- Rename the functions with an intel_ preffix.
v5: Several rebases to account for the changes in the previous patches.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:14 +08:00
|
|
|
struct drm_i915_gem_object *ctx_obj;
|
2016-08-15 17:48:54 +08:00
|
|
|
struct i915_vma *vma;
|
drm/i915/bdw: A bit more advanced LR context alloc/free
Now that we have the ability to allocate our own context backing objects
and we have multiplexed one of them per engine inside the context structs,
we can finally allocate and free them correctly.
Regarding the context size, reading the register to calculate the sizes
can work, I think, however the docs are very clear about the actual
context sizes on GEN8, so just hardcode that and use it.
v2: Rebased on top of the Full PPGTT series. It is important to notice
that at this point we have one global default context per engine, all
of them using the aliasing PPGTT (as opposed to the single global
default context we have with legacy HW contexts).
v3:
- Go back to one single global default context, this time with multiple
backing objects inside.
- Use different context sizes for non-render engines, as suggested by
Damien (still hardcoded, since the information about the context size
registers in the BSpec is, well, *lacking*).
- Render ctx size is 20 (or 19) pages, but not 21 (caught by Damien).
- Move default context backing object creation to intel_init_ring (so
that we don't waste memory in rings that might not get initialized).
v4:
- Reuse the HW legacy context init/fini.
- Create a separate free function.
- Rename the functions with an intel_ preffix.
v5: Several rebases to account for the changes in the previous patches.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:14 +08:00
|
|
|
uint32_t context_size;
|
2016-08-03 05:50:21 +08:00
|
|
|
struct intel_ring *ring;
|
2018-05-03 00:38:39 +08:00
|
|
|
struct i915_timeline *timeline;
|
drm/i915/bdw: A bit more advanced LR context alloc/free
Now that we have the ability to allocate our own context backing objects
and we have multiplexed one of them per engine inside the context structs,
we can finally allocate and free them correctly.
Regarding the context size, reading the register to calculate the sizes
can work, I think, however the docs are very clear about the actual
context sizes on GEN8, so just hardcode that and use it.
v2: Rebased on top of the Full PPGTT series. It is important to notice
that at this point we have one global default context per engine, all
of them using the aliasing PPGTT (as opposed to the single global
default context we have with legacy HW contexts).
v3:
- Go back to one single global default context, this time with multiple
backing objects inside.
- Use different context sizes for non-render engines, as suggested by
Damien (still hardcoded, since the information about the context size
registers in the BSpec is, well, *lacking*).
- Render ctx size is 20 (or 19) pages, but not 21 (caught by Damien).
- Move default context backing object creation to intel_init_ring (so
that we don't waste memory in rings that might not get initialized).
v4:
- Reuse the HW legacy context init/fini.
- Create a separate free function.
- Rename the functions with an intel_ preffix.
v5: Several rebases to account for the changes in the previous patches.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:14 +08:00
|
|
|
int ret;
|
|
|
|
|
2018-01-26 20:18:46 +08:00
|
|
|
if (ce->state)
|
|
|
|
return 0;
|
2014-07-25 00:04:12 +08:00
|
|
|
|
2017-04-28 15:53:36 +08:00
|
|
|
context_size = round_up(engine->context_size, I915_GTT_PAGE_SIZE);
|
drm/i915/bdw: A bit more advanced LR context alloc/free
Now that we have the ability to allocate our own context backing objects
and we have multiplexed one of them per engine inside the context structs,
we can finally allocate and free them correctly.
Regarding the context size, reading the register to calculate the sizes
can work, I think, however the docs are very clear about the actual
context sizes on GEN8, so just hardcode that and use it.
v2: Rebased on top of the Full PPGTT series. It is important to notice
that at this point we have one global default context per engine, all
of them using the aliasing PPGTT (as opposed to the single global
default context we have with legacy HW contexts).
v3:
- Go back to one single global default context, this time with multiple
backing objects inside.
- Use different context sizes for non-render engines, as suggested by
Damien (still hardcoded, since the information about the context size
registers in the BSpec is, well, *lacking*).
- Render ctx size is 20 (or 19) pages, but not 21 (caught by Damien).
- Move default context backing object creation to intel_init_ring (so
that we don't waste memory in rings that might not get initialized).
v4:
- Reuse the HW legacy context init/fini.
- Create a separate free function.
- Rename the functions with an intel_ preffix.
v5: Several rebases to account for the changes in the previous patches.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:14 +08:00
|
|
|
|
2017-09-13 16:56:00 +08:00
|
|
|
/*
|
|
|
|
* Before the actual start of the context image, we insert a few pages
|
|
|
|
* for our own use and for sharing with the GuC.
|
|
|
|
*/
|
|
|
|
context_size += LRC_HEADER_PAGES * PAGE_SIZE;
|
drm/i915: Integrate GuC-based command submission
GuC-based submission is mostly the same as execlist mode, up to
intel_logical_ring_advance_and_submit(), where the context being
dispatched would be added to the execlist queue; at this point
we submit the context to the GuC backend instead.
There are, however, a few other changes also required, notably:
1. Contexts must be pinned at GGTT addresses accessible by the GuC
i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the
PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls.
2. The GuC's TLB must be invalidated after a context is pinned at
a new GGTT address.
3. GuC firmware uses the one page before Ring Context as shared data.
Therefore, whenever driver wants to get base address of LRC, we
will offset one page for it. LRC_PPHWSP_PN is defined as the page
number of LRCA.
4. In the work queue used to pass requests to the GuC, the GuC
firmware requires the ring-tail-offset to be represented as an
11-bit value, expressed in QWords. Therefore, the ringbuffer
size must be reduced to the representable range (4 pages).
v2:
Defer adding #defines until needed [Chris Wilson]
Rationalise type declarations [Chris Wilson]
v4:
Squashed kerneldoc patch into here [Daniel Vetter]
v5:
Update request->tail in code common to both GuC and execlist modes.
Add a private version of lr_context_update(), as sharing the
execlist version leads to race conditions when the CPU and
the GuC both update TAIL in the context image.
Conversion of error-captured HWS page to string must account
for offset from start of object to actual HWS (LRC_PPHWSP_PN).
Issue: VIZ-4884
Signed-off-by: Alex Dai <yu.dai@intel.com>
Signed-off-by: Dave Gordon <david.s.gordon@intel.com>
Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 22:43:43 +08:00
|
|
|
|
2016-12-01 22:16:37 +08:00
|
|
|
ctx_obj = i915_gem_object_create(ctx->i915, context_size);
|
2018-06-11 23:33:32 +08:00
|
|
|
if (IS_ERR(ctx_obj))
|
|
|
|
return PTR_ERR(ctx_obj);
|
drm/i915/bdw: A bit more advanced LR context alloc/free
Now that we have the ability to allocate our own context backing objects
and we have multiplexed one of them per engine inside the context structs,
we can finally allocate and free them correctly.
Regarding the context size, reading the register to calculate the sizes
can work, I think, however the docs are very clear about the actual
context sizes on GEN8, so just hardcode that and use it.
v2: Rebased on top of the Full PPGTT series. It is important to notice
that at this point we have one global default context per engine, all
of them using the aliasing PPGTT (as opposed to the single global
default context we have with legacy HW contexts).
v3:
- Go back to one single global default context, this time with multiple
backing objects inside.
- Use different context sizes for non-render engines, as suggested by
Damien (still hardcoded, since the information about the context size
registers in the BSpec is, well, *lacking*).
- Render ctx size is 20 (or 19) pages, but not 21 (caught by Damien).
- Move default context backing object creation to intel_init_ring (so
that we don't waste memory in rings that might not get initialized).
v4:
- Reuse the HW legacy context init/fini.
- Create a separate free function.
- Rename the functions with an intel_ preffix.
v5: Several rebases to account for the changes in the previous patches.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:14 +08:00
|
|
|
|
2018-06-05 23:37:58 +08:00
|
|
|
vma = i915_vma_instance(ctx_obj, &ctx->i915->ggtt.vm, NULL);
|
2016-08-15 17:48:54 +08:00
|
|
|
if (IS_ERR(vma)) {
|
|
|
|
ret = PTR_ERR(vma);
|
|
|
|
goto error_deref_obj;
|
|
|
|
}
|
|
|
|
|
2018-05-03 00:38:39 +08:00
|
|
|
timeline = i915_timeline_create(ctx->i915, ctx->name);
|
|
|
|
if (IS_ERR(timeline)) {
|
|
|
|
ret = PTR_ERR(timeline);
|
|
|
|
goto error_deref_obj;
|
|
|
|
}
|
|
|
|
|
|
|
|
ring = intel_engine_create_ring(engine, timeline, ctx->ring_size);
|
|
|
|
i915_timeline_put(timeline);
|
2016-08-03 05:50:20 +08:00
|
|
|
if (IS_ERR(ring)) {
|
|
|
|
ret = PTR_ERR(ring);
|
2015-09-11 19:53:46 +08:00
|
|
|
goto error_deref_obj;
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
}
|
|
|
|
|
2016-08-03 05:50:20 +08:00
|
|
|
ret = populate_lr_context(ctx, ctx_obj, engine, ring);
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
if (ret) {
|
|
|
|
DRM_DEBUG_DRIVER("Failed to populate LRC: %d\n", ret);
|
2016-08-03 05:50:20 +08:00
|
|
|
goto error_ring_free;
|
2014-07-25 00:04:15 +08:00
|
|
|
}
|
|
|
|
|
2016-08-03 05:50:20 +08:00
|
|
|
ce->ring = ring;
|
2016-08-15 17:48:54 +08:00
|
|
|
ce->state = vma;
|
2014-07-25 00:04:12 +08:00
|
|
|
|
|
|
|
return 0;
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
|
2016-08-03 05:50:20 +08:00
|
|
|
error_ring_free:
|
2016-08-03 05:50:21 +08:00
|
|
|
intel_ring_free(ring);
|
2015-09-11 19:53:46 +08:00
|
|
|
error_deref_obj:
|
2016-07-20 20:31:53 +08:00
|
|
|
i915_gem_object_put(ctx_obj);
|
drm/i915/bdw: Populate LR contexts (somewhat)
For the most part, logical ring context objects are similar to hardware
contexts in that the backing object is meant to be opaque. There are
some exceptions where we need to poke certain offsets of the object for
initialization, updating the tail pointer or updating the PDPs.
For our basic execlist implementation we'll only need our PPGTT PDs,
and ringbuffer addresses in order to set up the context. With previous
patches, we have both, so start prepping the context to be load.
Before running a context for the first time you must populate some
fields in the context object. These fields begin 1 PAGE + LRCA, ie. the
first page (in 0 based counting) of the context image. These same
fields will be read and written to as contexts are saved and restored
once the system is up and running.
Many of these fields are completely reused from previous global
registers: ringbuffer head/tail/control, context control matches some
previous MI_SET_CONTEXT flags, and page directories. There are other
fields which we don't touch which we may want in the future.
v2: CTX_LRI_HEADER_0 is MI_LOAD_REGISTER_IMM(14) for render and (11)
for other engines.
v3: Several rebases and general changes to the code.
v4: Squash with "Extract LR context object populating"
Also, Damien's review comments:
- Set the Force Posted bit on the LRI header, as the BSpec suggest we do.
- Prevent warning when compiling a 32-bits kernel without HIGHMEM64.
- Add a clarifying comment to the context population code.
v5: Damien's review comments:
- The third MI_LOAD_REGISTER_IMM in the context does not set Force Posted.
- Remove dead code.
v6: Add a note about the (presumed) differences between BDW and CHV state
contexts. Also, Brad's review comments:
- Use the _MASKED_BIT_ENABLE, upper_32_bits and lower_32_bits macros.
- Be less magical about how we set the ring size in the context.
Signed-off-by: Ben Widawsky <ben@bwidawsk.net> (v1)
Signed-off-by: Rafael Barbalho <rafael.barbalho@intel.com> (v2)
Signed-off-by: Oscar Mateo <oscar.mateo@intel.com>
Reviewed-by: Damien Lespiau <damien.lespiau@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2014-07-25 00:04:17 +08:00
|
|
|
return ret;
|
2014-07-25 00:04:12 +08:00
|
|
|
}
|
2015-02-17 00:12:53 +08:00
|
|
|
|
2016-09-09 21:11:53 +08:00
|
|
|
void intel_lr_context_resume(struct drm_i915_private *dev_priv)
|
2015-02-17 00:12:53 +08:00
|
|
|
{
|
2016-03-16 19:00:36 +08:00
|
|
|
struct intel_engine_cs *engine;
|
2016-09-21 21:51:08 +08:00
|
|
|
struct i915_gem_context *ctx;
|
drm/i915: Allocate intel_engine_cs structure only for the enabled engines
With the possibility of addition of many more number of rings in future,
the drm_i915_private structure could bloat as an array, of type
intel_engine_cs, is embedded inside it.
struct intel_engine_cs engine[I915_NUM_ENGINES];
Though this is still fine as generally there is only a single instance of
drm_i915_private structure used, but not all of the possible rings would be
enabled or active on most of the platforms. Some memory can be saved by
allocating intel_engine_cs structure only for the enabled/active engines.
Currently the engine/ring ID is kept static and dev_priv->engine[] is simply
indexed using the enums defined in intel_engine_id.
To save memory and continue using the static engine/ring IDs, 'engine' is
defined as an array of pointers.
struct intel_engine_cs *engine[I915_NUM_ENGINES];
dev_priv->engine[engine_ID] will be NULL for disabled engine instances.
There is a text size reduction of 928 bytes, from 1028200 to 1027272, for
i915.o file (but for i915.ko file text size remain same as 1193131 bytes).
v2:
- Remove the engine iterator field added in drm_i915_private structure,
instead pass a local iterator variable to the for_each_engine**
macros. (Chris)
- Do away with intel_engine_initialized() and instead directly use the
NULL pointer check on engine pointer. (Chris)
v3:
- Remove for_each_engine_id() macro, as the updated macro for_each_engine()
can be used in place of it. (Chris)
- Protect the access to Render engine Fault register with a NULL check, as
engine specific init is done later in Driver load sequence.
v4:
- Use !!dev_priv->engine[VCS] style for the engine check in getparam. (Chris)
- Kill the superfluous init_engine_lists().
v5:
- Cleanup the intel_engines_init() & intel_engines_setup(), with respect to
allocation of intel_engine_cs structure. (Chris)
v6:
- Rebase.
v7:
- Optimize the for_each_engine_masked() macro. (Chris)
- Change the type of 'iter' local variable to enum intel_engine_id. (Chris)
- Rebase.
v8: Rebase.
v9: Rebase.
v10:
- For index calculation use engine ID instead of pointer based arithmetic in
intel_engine_sync_index() as engine pointers are not contiguous now (Chris)
- For appropriateness, rename local enum variable 'iter' to 'id'. (Joonas)
- Use for_each_engine macro for cleanup in intel_engines_init() and remove
check for NULL engine pointer in cleanup() routines. (Joonas)
v11: Rebase.
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Akash Goel <akash.goel@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/1476378888-7372-1-git-send-email-akash.goel@intel.com
2016-10-14 01:14:48 +08:00
|
|
|
enum intel_engine_id id;
|
2016-09-21 21:51:08 +08:00
|
|
|
|
|
|
|
/* Because we emit WA_TAIL_DWORDS there may be a disparity
|
|
|
|
* between our bookkeeping in ce->ring->head and ce->ring->tail and
|
|
|
|
* that stored in context. As we only write new commands from
|
|
|
|
* ce->ring->tail onwards, everything before that is junk. If the GPU
|
|
|
|
* starts reading from its RING_HEAD from the context, it may try to
|
|
|
|
* execute that junk and die.
|
|
|
|
*
|
|
|
|
* So to avoid that we reset the context images upon resume. For
|
|
|
|
* simplicity, we just zero everything out.
|
|
|
|
*/
|
2017-06-20 19:05:45 +08:00
|
|
|
list_for_each_entry(ctx, &dev_priv->contexts.list, link) {
|
drm/i915: Allocate intel_engine_cs structure only for the enabled engines
With the possibility of addition of many more number of rings in future,
the drm_i915_private structure could bloat as an array, of type
intel_engine_cs, is embedded inside it.
struct intel_engine_cs engine[I915_NUM_ENGINES];
Though this is still fine as generally there is only a single instance of
drm_i915_private structure used, but not all of the possible rings would be
enabled or active on most of the platforms. Some memory can be saved by
allocating intel_engine_cs structure only for the enabled/active engines.
Currently the engine/ring ID is kept static and dev_priv->engine[] is simply
indexed using the enums defined in intel_engine_id.
To save memory and continue using the static engine/ring IDs, 'engine' is
defined as an array of pointers.
struct intel_engine_cs *engine[I915_NUM_ENGINES];
dev_priv->engine[engine_ID] will be NULL for disabled engine instances.
There is a text size reduction of 928 bytes, from 1028200 to 1027272, for
i915.o file (but for i915.ko file text size remain same as 1193131 bytes).
v2:
- Remove the engine iterator field added in drm_i915_private structure,
instead pass a local iterator variable to the for_each_engine**
macros. (Chris)
- Do away with intel_engine_initialized() and instead directly use the
NULL pointer check on engine pointer. (Chris)
v3:
- Remove for_each_engine_id() macro, as the updated macro for_each_engine()
can be used in place of it. (Chris)
- Protect the access to Render engine Fault register with a NULL check, as
engine specific init is done later in Driver load sequence.
v4:
- Use !!dev_priv->engine[VCS] style for the engine check in getparam. (Chris)
- Kill the superfluous init_engine_lists().
v5:
- Cleanup the intel_engines_init() & intel_engines_setup(), with respect to
allocation of intel_engine_cs structure. (Chris)
v6:
- Rebase.
v7:
- Optimize the for_each_engine_masked() macro. (Chris)
- Change the type of 'iter' local variable to enum intel_engine_id. (Chris)
- Rebase.
v8: Rebase.
v9: Rebase.
v10:
- For index calculation use engine ID instead of pointer based arithmetic in
intel_engine_sync_index() as engine pointers are not contiguous now (Chris)
- For appropriateness, rename local enum variable 'iter' to 'id'. (Joonas)
- Use for_each_engine macro for cleanup in intel_engines_init() and remove
check for NULL engine pointer in cleanup() routines. (Joonas)
v11: Rebase.
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Akash Goel <akash.goel@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/1476378888-7372-1-git-send-email-akash.goel@intel.com
2016-10-14 01:14:48 +08:00
|
|
|
for_each_engine(engine, dev_priv, id) {
|
2018-04-30 21:15:01 +08:00
|
|
|
struct intel_context *ce =
|
|
|
|
to_intel_context(ctx, engine);
|
2016-09-21 21:51:08 +08:00
|
|
|
u32 *reg;
|
2015-02-17 00:12:53 +08:00
|
|
|
|
2016-09-21 21:51:08 +08:00
|
|
|
if (!ce->state)
|
|
|
|
continue;
|
2016-04-12 22:40:42 +08:00
|
|
|
|
2016-09-21 21:51:08 +08:00
|
|
|
reg = i915_gem_object_pin_map(ce->state->obj,
|
|
|
|
I915_MAP_WB);
|
|
|
|
if (WARN_ON(IS_ERR(reg)))
|
|
|
|
continue;
|
2015-02-17 00:12:53 +08:00
|
|
|
|
2016-09-21 21:51:08 +08:00
|
|
|
reg += LRC_STATE_PN * PAGE_SIZE / sizeof(*reg);
|
|
|
|
reg[CTX_RING_HEAD+1] = 0;
|
|
|
|
reg[CTX_RING_TAIL+1] = 0;
|
2015-02-17 00:12:53 +08:00
|
|
|
|
2016-10-28 20:58:35 +08:00
|
|
|
ce->state->obj->mm.dirty = true;
|
2016-09-21 21:51:08 +08:00
|
|
|
i915_gem_object_unpin_map(ce->state->obj);
|
2015-02-17 00:12:53 +08:00
|
|
|
|
2017-04-25 21:00:49 +08:00
|
|
|
intel_ring_reset(ce->ring, 0);
|
2016-09-21 21:51:08 +08:00
|
|
|
}
|
2015-02-17 00:12:53 +08:00
|
|
|
}
|
|
|
|
}
|
2018-04-04 17:33:29 +08:00
|
|
|
|
|
|
|
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
|
|
|
|
#include "selftests/intel_lrc.c"
|
|
|
|
#endif
|