UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity
OpenCloudOS-Kernel/drivers/gpu/drm/radeon/radeon_kfd.c

697 lines
18 KiB
C
Raw Normal View History

drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
/*
* Copyright 2014 Advanced Micro Devices, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*/
#include <linux/module.h>
#include <linux/fdtable.h>
#include <linux/uaccess.h>
#include <drm/drmP.h>
#include "radeon.h"
#include "cikd.h"
#include "cik_reg.h"
#include "radeon_kfd.h"
drm/radeon: Add implementation of get_fw_version This patch implements a new interface that was added to the kfd-->kgd interface. The new interface function retrieves the firmware version that is currently in use by a specific engine. The firmware was uploaded to the engine by the radeon driver. v2: Returns the fw version of the specific engine, as passed into the function by a new parameter Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-11-09 18:45:11 +08:00
#include "radeon_ucode.h"
#include <linux/firmware.h>
drm/radeon: Use new cik_structs.h file Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-03 05:33:20 +08:00
#include "cik_structs.h"
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
#define CIK_PIPE_PER_MEC (4)
struct kgd_mem {
drm/radeon: Impl. new gtt allocate/free functions This patch adds the implementation of the gtt interface functions. The allocate function will allocate a single bo, pin and map it to kernel memory. It will return the gpu address and cpu ptr as arguments. v2: The bulk of the allocations in the GART is for MQDs. MQDs represent active user-mode queues, which are on the current runlist. It is important to remember that active queues doesn't necessarily mean scheduled/running queues, especially if there is over-subscription of queues or more than a single HSA process. Because the scheduling of the user-mode queues is done by the CP firmware, amdkfd doesn't have any indication if the queue is scheduled or not. If the CP will try to schedule a queue, and its MQD is not present, this will probably stuck the CP permanently, as it will load garbage from the GART (the address of the MQD is given to the CP inside the runlist packet). In addition, there are a couple of small allocations which also should always be pinned - runlist packets (2 packets) and HPDs. runlist packets can be quite large, depending on number of processes and queues. This new allocate function represents the short/mid-term solution of limiting the total memory consumption to around 4MB by default. The long-term solution is to create a mechanism through which radeon/ttm can ask amdkfd to clear GART/VRAM memory due to memory pressure. Then, amdkfd will preempt the running queues and wait until the memory pressure is over. After that, amdkfd will reschedule the queues. Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-10-27 02:52:55 +08:00
struct radeon_bo *bo;
drm/radeon: Remove old radeon_sa usage from kfd-->kgd interface Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-10-26 15:43:43 +08:00
uint64_t gpu_addr;
drm/radeon: Impl. new gtt allocate/free functions This patch adds the implementation of the gtt interface functions. The allocate function will allocate a single bo, pin and map it to kernel memory. It will return the gpu address and cpu ptr as arguments. v2: The bulk of the allocations in the GART is for MQDs. MQDs represent active user-mode queues, which are on the current runlist. It is important to remember that active queues doesn't necessarily mean scheduled/running queues, especially if there is over-subscription of queues or more than a single HSA process. Because the scheduling of the user-mode queues is done by the CP firmware, amdkfd doesn't have any indication if the queue is scheduled or not. If the CP will try to schedule a queue, and its MQD is not present, this will probably stuck the CP permanently, as it will load garbage from the GART (the address of the MQD is given to the CP inside the runlist packet). In addition, there are a couple of small allocations which also should always be pinned - runlist packets (2 packets) and HPDs. runlist packets can be quite large, depending on number of processes and queues. This new allocate function represents the short/mid-term solution of limiting the total memory consumption to around 4MB by default. The long-term solution is to create a mechanism through which radeon/ttm can ask amdkfd to clear GART/VRAM memory due to memory pressure. Then, amdkfd will preempt the running queues and wait until the memory pressure is over. After that, amdkfd will reschedule the queues. Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-10-27 02:52:55 +08:00
void *cpu_ptr;
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
};
drm/radeon: Impl. new gtt allocate/free functions This patch adds the implementation of the gtt interface functions. The allocate function will allocate a single bo, pin and map it to kernel memory. It will return the gpu address and cpu ptr as arguments. v2: The bulk of the allocations in the GART is for MQDs. MQDs represent active user-mode queues, which are on the current runlist. It is important to remember that active queues doesn't necessarily mean scheduled/running queues, especially if there is over-subscription of queues or more than a single HSA process. Because the scheduling of the user-mode queues is done by the CP firmware, amdkfd doesn't have any indication if the queue is scheduled or not. If the CP will try to schedule a queue, and its MQD is not present, this will probably stuck the CP permanently, as it will load garbage from the GART (the address of the MQD is given to the CP inside the runlist packet). In addition, there are a couple of small allocations which also should always be pinned - runlist packets (2 packets) and HPDs. runlist packets can be quite large, depending on number of processes and queues. This new allocate function represents the short/mid-term solution of limiting the total memory consumption to around 4MB by default. The long-term solution is to create a mechanism through which radeon/ttm can ask amdkfd to clear GART/VRAM memory due to memory pressure. Then, amdkfd will preempt the running queues and wait until the memory pressure is over. After that, amdkfd will reschedule the queues. Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-10-27 02:52:55 +08:00
static int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
void **mem_obj, uint64_t *gpu_addr,
void **cpu_ptr);
static void free_gtt_mem(struct kgd_dev *kgd, void *mem_obj);
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
static uint64_t get_vmem_size(struct kgd_dev *kgd);
static uint64_t get_gpu_clock_counter(struct kgd_dev *kgd);
static uint32_t get_max_engine_clock_in_mhz(struct kgd_dev *kgd);
drm/radeon: Add implementation of get_fw_version This patch implements a new interface that was added to the kfd-->kgd interface. The new interface function retrieves the firmware version that is currently in use by a specific engine. The firmware was uploaded to the engine by the radeon driver. v2: Returns the fw version of the specific engine, as passed into the function by a new parameter Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-11-09 18:45:11 +08:00
static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type);
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
/*
* Register access functions
*/
static void kgd_program_sh_mem_settings(struct kgd_dev *kgd, uint32_t vmid,
uint32_t sh_mem_config, uint32_t sh_mem_ape1_base,
uint32_t sh_mem_ape1_limit, uint32_t sh_mem_bases);
static int kgd_set_pasid_vmid_mapping(struct kgd_dev *kgd, unsigned int pasid,
unsigned int vmid);
static int kgd_init_pipeline(struct kgd_dev *kgd, uint32_t pipe_id,
uint32_t hpd_size, uint64_t hpd_gpu_addr);
static int kgd_hqd_load(struct kgd_dev *kgd, void *mqd, uint32_t pipe_id,
uint32_t queue_id, uint32_t __user *wptr);
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
static int kgd_hqd_sdma_load(struct kgd_dev *kgd, void *mqd);
drm/amd: Fixing typos in kfd<->kgd interface Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-09 18:00:09 +08:00
static bool kgd_hqd_is_occupied(struct kgd_dev *kgd, uint64_t queue_address,
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
uint32_t pipe_id, uint32_t queue_id);
static int kgd_hqd_destroy(struct kgd_dev *kgd, uint32_t reset_type,
unsigned int timeout, uint32_t pipe_id,
uint32_t queue_id);
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
static bool kgd_hqd_sdma_is_occupied(struct kgd_dev *kgd, void *mqd);
static int kgd_hqd_sdma_destroy(struct kgd_dev *kgd, void *mqd,
unsigned int timeout);
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
static const struct kfd2kgd_calls kfd2kgd = {
drm/radeon: Impl. new gtt allocate/free functions This patch adds the implementation of the gtt interface functions. The allocate function will allocate a single bo, pin and map it to kernel memory. It will return the gpu address and cpu ptr as arguments. v2: The bulk of the allocations in the GART is for MQDs. MQDs represent active user-mode queues, which are on the current runlist. It is important to remember that active queues doesn't necessarily mean scheduled/running queues, especially if there is over-subscription of queues or more than a single HSA process. Because the scheduling of the user-mode queues is done by the CP firmware, amdkfd doesn't have any indication if the queue is scheduled or not. If the CP will try to schedule a queue, and its MQD is not present, this will probably stuck the CP permanently, as it will load garbage from the GART (the address of the MQD is given to the CP inside the runlist packet). In addition, there are a couple of small allocations which also should always be pinned - runlist packets (2 packets) and HPDs. runlist packets can be quite large, depending on number of processes and queues. This new allocate function represents the short/mid-term solution of limiting the total memory consumption to around 4MB by default. The long-term solution is to create a mechanism through which radeon/ttm can ask amdkfd to clear GART/VRAM memory due to memory pressure. Then, amdkfd will preempt the running queues and wait until the memory pressure is over. After that, amdkfd will reschedule the queues. Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-10-27 02:52:55 +08:00
.init_gtt_mem_allocation = alloc_gtt_mem,
.free_gtt_mem = free_gtt_mem,
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
.get_vmem_size = get_vmem_size,
.get_gpu_clock_counter = get_gpu_clock_counter,
.get_max_engine_clock_in_mhz = get_max_engine_clock_in_mhz,
.program_sh_mem_settings = kgd_program_sh_mem_settings,
.set_pasid_vmid_mapping = kgd_set_pasid_vmid_mapping,
.init_pipeline = kgd_init_pipeline,
.hqd_load = kgd_hqd_load,
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
.hqd_sdma_load = kgd_hqd_sdma_load,
drm/amd: Fixing typos in kfd<->kgd interface Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-09 18:00:09 +08:00
.hqd_is_occupied = kgd_hqd_is_occupied,
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
.hqd_sdma_is_occupied = kgd_hqd_sdma_is_occupied,
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
.hqd_destroy = kgd_hqd_destroy,
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
.hqd_sdma_destroy = kgd_hqd_sdma_destroy,
drm/radeon: Add implementation of get_fw_version This patch implements a new interface that was added to the kfd-->kgd interface. The new interface function retrieves the firmware version that is currently in use by a specific engine. The firmware was uploaded to the engine by the radeon driver. v2: Returns the fw version of the specific engine, as passed into the function by a new parameter Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-11-09 18:45:11 +08:00
.get_fw_version = get_fw_version
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
};
static const struct kgd2kfd_calls *kgd2kfd;
bool radeon_kfd_init(void)
{
drm/radeon: Init amdkfd only if it was compiled This patch changes the radeon_kfd_init(), which is used to initialize the interface between radeon and amdkfd, so the interface will be initialized only if amdkfd was build, either as module or inside the kernel image. In the modules case, the symbol_request() will be used (same as old code). In the in-image compilation case, a direct call to kgd2kfd_init() will be done. For other cases, radeon_kfd_init() will just return false. This patch is necessary because in case of the following specific configuration: kernel 32-bit, no modules support, random kernel base and no hibernation, the symbol_request() doesn't work as expected - it doesn't return NULL if the symbol doesn't exists - which makes the kernel panic. Reviewed-by: Christian König <christian.koenig@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-22 17:19:23 +08:00
#if defined(CONFIG_HSA_AMD_MODULE)
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
bool (*kgd2kfd_init_p)(unsigned, const struct kfd2kgd_calls*,
const struct kgd2kfd_calls**);
kgd2kfd_init_p = symbol_request(kgd2kfd_init);
if (kgd2kfd_init_p == NULL)
return false;
if (!kgd2kfd_init_p(KFD_INTERFACE_VERSION, &kfd2kgd, &kgd2kfd)) {
symbol_put(kgd2kfd_init);
kgd2kfd = NULL;
return false;
}
return true;
drm/radeon: Init amdkfd only if it was compiled This patch changes the radeon_kfd_init(), which is used to initialize the interface between radeon and amdkfd, so the interface will be initialized only if amdkfd was build, either as module or inside the kernel image. In the modules case, the symbol_request() will be used (same as old code). In the in-image compilation case, a direct call to kgd2kfd_init() will be done. For other cases, radeon_kfd_init() will just return false. This patch is necessary because in case of the following specific configuration: kernel 32-bit, no modules support, random kernel base and no hibernation, the symbol_request() doesn't work as expected - it doesn't return NULL if the symbol doesn't exists - which makes the kernel panic. Reviewed-by: Christian König <christian.koenig@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-22 17:19:23 +08:00
#elif defined(CONFIG_HSA_AMD)
if (!kgd2kfd_init(KFD_INTERFACE_VERSION, &kfd2kgd, &kgd2kfd)) {
kgd2kfd = NULL;
return false;
}
return true;
#else
return false;
#endif
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
}
void radeon_kfd_fini(void)
{
if (kgd2kfd) {
kgd2kfd->exit();
symbol_put(kgd2kfd_init);
}
}
void radeon_kfd_device_probe(struct radeon_device *rdev)
{
if (kgd2kfd)
rdev->kfd = kgd2kfd->probe((struct kgd_dev *)rdev, rdev->pdev);
}
void radeon_kfd_device_init(struct radeon_device *rdev)
{
if (rdev->kfd) {
struct kgd2kfd_shared_resources gpu_resources = {
.compute_vmid_bitmap = 0xFF00,
.first_compute_pipe = 1,
.compute_pipe_count = 8 - 1,
};
radeon_doorbell_get_kfd_info(rdev,
&gpu_resources.doorbell_physical_address,
&gpu_resources.doorbell_aperture_size,
&gpu_resources.doorbell_start_offset);
kgd2kfd->device_init(rdev->kfd, &gpu_resources);
}
}
void radeon_kfd_device_fini(struct radeon_device *rdev)
{
if (rdev->kfd) {
kgd2kfd->device_exit(rdev->kfd);
rdev->kfd = NULL;
}
}
void radeon_kfd_interrupt(struct radeon_device *rdev, const void *ih_ring_entry)
{
if (rdev->kfd)
kgd2kfd->interrupt(rdev->kfd, ih_ring_entry);
}
void radeon_kfd_suspend(struct radeon_device *rdev)
{
if (rdev->kfd)
kgd2kfd->suspend(rdev->kfd);
}
int radeon_kfd_resume(struct radeon_device *rdev)
{
int r = 0;
if (rdev->kfd)
r = kgd2kfd->resume(rdev->kfd);
return r;
}
drm/radeon: Impl. new gtt allocate/free functions This patch adds the implementation of the gtt interface functions. The allocate function will allocate a single bo, pin and map it to kernel memory. It will return the gpu address and cpu ptr as arguments. v2: The bulk of the allocations in the GART is for MQDs. MQDs represent active user-mode queues, which are on the current runlist. It is important to remember that active queues doesn't necessarily mean scheduled/running queues, especially if there is over-subscription of queues or more than a single HSA process. Because the scheduling of the user-mode queues is done by the CP firmware, amdkfd doesn't have any indication if the queue is scheduled or not. If the CP will try to schedule a queue, and its MQD is not present, this will probably stuck the CP permanently, as it will load garbage from the GART (the address of the MQD is given to the CP inside the runlist packet). In addition, there are a couple of small allocations which also should always be pinned - runlist packets (2 packets) and HPDs. runlist packets can be quite large, depending on number of processes and queues. This new allocate function represents the short/mid-term solution of limiting the total memory consumption to around 4MB by default. The long-term solution is to create a mechanism through which radeon/ttm can ask amdkfd to clear GART/VRAM memory due to memory pressure. Then, amdkfd will preempt the running queues and wait until the memory pressure is over. After that, amdkfd will reschedule the queues. Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-10-27 02:52:55 +08:00
static int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
void **mem_obj, uint64_t *gpu_addr,
void **cpu_ptr)
{
struct radeon_device *rdev = (struct radeon_device *)kgd;
struct kgd_mem **mem = (struct kgd_mem **) mem_obj;
int r;
BUG_ON(kgd == NULL);
BUG_ON(gpu_addr == NULL);
BUG_ON(cpu_ptr == NULL);
*mem = kmalloc(sizeof(struct kgd_mem), GFP_KERNEL);
if ((*mem) == NULL)
return -ENOMEM;
r = radeon_bo_create(rdev, size, PAGE_SIZE, true, RADEON_GEM_DOMAIN_GTT,
RADEON_GEM_GTT_WC, NULL, NULL, &(*mem)->bo);
if (r) {
dev_err(rdev->dev,
"failed to allocate BO for amdkfd (%d)\n", r);
return r;
}
/* map the buffer */
r = radeon_bo_reserve((*mem)->bo, true);
if (r) {
dev_err(rdev->dev, "(%d) failed to reserve bo for amdkfd\n", r);
goto allocate_mem_reserve_bo_failed;
}
r = radeon_bo_pin((*mem)->bo, RADEON_GEM_DOMAIN_GTT,
&(*mem)->gpu_addr);
if (r) {
dev_err(rdev->dev, "(%d) failed to pin bo for amdkfd\n", r);
goto allocate_mem_pin_bo_failed;
}
*gpu_addr = (*mem)->gpu_addr;
r = radeon_bo_kmap((*mem)->bo, &(*mem)->cpu_ptr);
if (r) {
dev_err(rdev->dev,
"(%d) failed to map bo to kernel for amdkfd\n", r);
goto allocate_mem_kmap_bo_failed;
}
*cpu_ptr = (*mem)->cpu_ptr;
radeon_bo_unreserve((*mem)->bo);
return 0;
allocate_mem_kmap_bo_failed:
radeon_bo_unpin((*mem)->bo);
allocate_mem_pin_bo_failed:
radeon_bo_unreserve((*mem)->bo);
allocate_mem_reserve_bo_failed:
radeon_bo_unref(&(*mem)->bo);
return r;
}
static void free_gtt_mem(struct kgd_dev *kgd, void *mem_obj)
{
struct kgd_mem *mem = (struct kgd_mem *) mem_obj;
BUG_ON(mem == NULL);
radeon_bo_reserve(mem->bo, true);
radeon_bo_kunmap(mem->bo);
radeon_bo_unpin(mem->bo);
radeon_bo_unreserve(mem->bo);
radeon_bo_unref(&(mem->bo));
kfree(mem);
}
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
static uint64_t get_vmem_size(struct kgd_dev *kgd)
{
struct radeon_device *rdev = (struct radeon_device *)kgd;
BUG_ON(kgd == NULL);
return rdev->mc.real_vram_size;
}
static uint64_t get_gpu_clock_counter(struct kgd_dev *kgd)
{
struct radeon_device *rdev = (struct radeon_device *)kgd;
return rdev->asic->get_gpu_clock_counter(rdev);
}
static uint32_t get_max_engine_clock_in_mhz(struct kgd_dev *kgd)
{
struct radeon_device *rdev = (struct radeon_device *)kgd;
/* The sclk is in quantas of 10kHz */
return rdev->pm.dpm.dyn_state.max_clock_voltage_on_ac.sclk / 100;
}
static inline struct radeon_device *get_radeon_device(struct kgd_dev *kgd)
{
return (struct radeon_device *)kgd;
}
static void write_register(struct kgd_dev *kgd, uint32_t offset, uint32_t value)
{
struct radeon_device *rdev = get_radeon_device(kgd);
writel(value, (void __iomem *)(rdev->rmmio + offset));
}
static uint32_t read_register(struct kgd_dev *kgd, uint32_t offset)
{
struct radeon_device *rdev = get_radeon_device(kgd);
return readl((void __iomem *)(rdev->rmmio + offset));
}
static void lock_srbm(struct kgd_dev *kgd, uint32_t mec, uint32_t pipe,
uint32_t queue, uint32_t vmid)
{
struct radeon_device *rdev = get_radeon_device(kgd);
uint32_t value = PIPEID(pipe) | MEID(mec) | VMID(vmid) | QUEUEID(queue);
mutex_lock(&rdev->srbm_mutex);
write_register(kgd, SRBM_GFX_CNTL, value);
}
static void unlock_srbm(struct kgd_dev *kgd)
{
struct radeon_device *rdev = get_radeon_device(kgd);
write_register(kgd, SRBM_GFX_CNTL, 0);
mutex_unlock(&rdev->srbm_mutex);
}
static void acquire_queue(struct kgd_dev *kgd, uint32_t pipe_id,
uint32_t queue_id)
{
uint32_t mec = (++pipe_id / CIK_PIPE_PER_MEC) + 1;
uint32_t pipe = (pipe_id % CIK_PIPE_PER_MEC);
lock_srbm(kgd, mec, pipe, queue_id, 0);
}
static void release_queue(struct kgd_dev *kgd)
{
unlock_srbm(kgd);
}
static void kgd_program_sh_mem_settings(struct kgd_dev *kgd, uint32_t vmid,
uint32_t sh_mem_config,
uint32_t sh_mem_ape1_base,
uint32_t sh_mem_ape1_limit,
uint32_t sh_mem_bases)
{
lock_srbm(kgd, 0, 0, 0, vmid);
write_register(kgd, SH_MEM_CONFIG, sh_mem_config);
write_register(kgd, SH_MEM_APE1_BASE, sh_mem_ape1_base);
write_register(kgd, SH_MEM_APE1_LIMIT, sh_mem_ape1_limit);
write_register(kgd, SH_MEM_BASES, sh_mem_bases);
unlock_srbm(kgd);
}
static int kgd_set_pasid_vmid_mapping(struct kgd_dev *kgd, unsigned int pasid,
unsigned int vmid)
{
/*
* We have to assume that there is no outstanding mapping.
* The ATC_VMID_PASID_MAPPING_UPDATE_STATUS bit could be 0
* because a mapping is in progress or because a mapping finished and
* the SW cleared it.
* So the protocol is to always wait & clear.
*/
uint32_t pasid_mapping = (pasid == 0) ? 0 :
(uint32_t)pasid | ATC_VMID_PASID_MAPPING_VALID;
write_register(kgd, ATC_VMID0_PASID_MAPPING + vmid*sizeof(uint32_t),
pasid_mapping);
while (!(read_register(kgd, ATC_VMID_PASID_MAPPING_UPDATE_STATUS) &
(1U << vmid)))
cpu_relax();
write_register(kgd, ATC_VMID_PASID_MAPPING_UPDATE_STATUS, 1U << vmid);
drm/radeon: Assign VMID to PASID for IH in non-HWS mode This patch fixes a bug in kgd_set_pasid_vmid_mapping(), where the function only updated the ATC registers (IOMMU) with the new VMID <--> PASID mapping, but didn't update the IH (Interrupt) registers. The bug only occurs when using non-HWS mode. In HWS mode, the CP automatically does the VMID <--> PASID mapping. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Acked-by: Alex Deucher <alexander.deucher@amd.com>
2014-12-17 20:09:10 +08:00
/* Mapping vmid to pasid also for IH block */
write_register(kgd, IH_VMID_0_LUT + vmid * sizeof(uint32_t),
pasid_mapping);
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
return 0;
}
static int kgd_init_pipeline(struct kgd_dev *kgd, uint32_t pipe_id,
uint32_t hpd_size, uint64_t hpd_gpu_addr)
{
uint32_t mec = (++pipe_id / CIK_PIPE_PER_MEC) + 1;
uint32_t pipe = (pipe_id % CIK_PIPE_PER_MEC);
lock_srbm(kgd, mec, pipe, 0, 0);
write_register(kgd, CP_HPD_EOP_BASE_ADDR,
lower_32_bits(hpd_gpu_addr >> 8));
write_register(kgd, CP_HPD_EOP_BASE_ADDR_HI,
upper_32_bits(hpd_gpu_addr >> 8));
write_register(kgd, CP_HPD_EOP_VMID, 0);
write_register(kgd, CP_HPD_EOP_CONTROL, hpd_size);
unlock_srbm(kgd);
return 0;
}
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
static inline uint32_t get_sdma_base_addr(struct cik_sdma_rlc_registers *m)
{
uint32_t retval;
retval = m->sdma_engine_id * SDMA1_REGISTER_OFFSET +
m->sdma_queue_id * KFD_CIK_SDMA_QUEUE_OFFSET;
pr_debug("kfd: sdma base address: 0x%x\n", retval);
return retval;
}
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
static inline struct cik_mqd *get_mqd(void *mqd)
{
return (struct cik_mqd *)mqd;
}
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
static inline struct cik_sdma_rlc_registers *get_sdma_mqd(void *mqd)
{
return (struct cik_sdma_rlc_registers *)mqd;
}
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
static int kgd_hqd_load(struct kgd_dev *kgd, void *mqd, uint32_t pipe_id,
uint32_t queue_id, uint32_t __user *wptr)
{
uint32_t wptr_shadow, is_wptr_shadow_valid;
struct cik_mqd *m;
m = get_mqd(mqd);
is_wptr_shadow_valid = !get_user(wptr_shadow, wptr);
acquire_queue(kgd, pipe_id, queue_id);
write_register(kgd, CP_MQD_BASE_ADDR, m->cp_mqd_base_addr_lo);
write_register(kgd, CP_MQD_BASE_ADDR_HI, m->cp_mqd_base_addr_hi);
write_register(kgd, CP_MQD_CONTROL, m->cp_mqd_control);
write_register(kgd, CP_HQD_PQ_BASE, m->cp_hqd_pq_base_lo);
write_register(kgd, CP_HQD_PQ_BASE_HI, m->cp_hqd_pq_base_hi);
write_register(kgd, CP_HQD_PQ_CONTROL, m->cp_hqd_pq_control);
write_register(kgd, CP_HQD_IB_CONTROL, m->cp_hqd_ib_control);
write_register(kgd, CP_HQD_IB_BASE_ADDR, m->cp_hqd_ib_base_addr_lo);
write_register(kgd, CP_HQD_IB_BASE_ADDR_HI, m->cp_hqd_ib_base_addr_hi);
write_register(kgd, CP_HQD_IB_RPTR, m->cp_hqd_ib_rptr);
write_register(kgd, CP_HQD_PERSISTENT_STATE,
m->cp_hqd_persistent_state);
write_register(kgd, CP_HQD_SEMA_CMD, m->cp_hqd_sema_cmd);
write_register(kgd, CP_HQD_MSG_TYPE, m->cp_hqd_msg_type);
write_register(kgd, CP_HQD_ATOMIC0_PREOP_LO,
m->cp_hqd_atomic0_preop_lo);
write_register(kgd, CP_HQD_ATOMIC0_PREOP_HI,
m->cp_hqd_atomic0_preop_hi);
write_register(kgd, CP_HQD_ATOMIC1_PREOP_LO,
m->cp_hqd_atomic1_preop_lo);
write_register(kgd, CP_HQD_ATOMIC1_PREOP_HI,
m->cp_hqd_atomic1_preop_hi);
write_register(kgd, CP_HQD_PQ_RPTR_REPORT_ADDR,
m->cp_hqd_pq_rptr_report_addr_lo);
write_register(kgd, CP_HQD_PQ_RPTR_REPORT_ADDR_HI,
m->cp_hqd_pq_rptr_report_addr_hi);
write_register(kgd, CP_HQD_PQ_RPTR, m->cp_hqd_pq_rptr);
write_register(kgd, CP_HQD_PQ_WPTR_POLL_ADDR,
m->cp_hqd_pq_wptr_poll_addr_lo);
write_register(kgd, CP_HQD_PQ_WPTR_POLL_ADDR_HI,
m->cp_hqd_pq_wptr_poll_addr_hi);
write_register(kgd, CP_HQD_PQ_DOORBELL_CONTROL,
m->cp_hqd_pq_doorbell_control);
write_register(kgd, CP_HQD_VMID, m->cp_hqd_vmid);
write_register(kgd, CP_HQD_QUANTUM, m->cp_hqd_quantum);
write_register(kgd, CP_HQD_PIPE_PRIORITY, m->cp_hqd_pipe_priority);
write_register(kgd, CP_HQD_QUEUE_PRIORITY, m->cp_hqd_queue_priority);
write_register(kgd, CP_HQD_IQ_RPTR, m->cp_hqd_iq_rptr);
if (is_wptr_shadow_valid)
write_register(kgd, CP_HQD_PQ_WPTR, wptr_shadow);
write_register(kgd, CP_HQD_ACTIVE, m->cp_hqd_active);
release_queue(kgd);
return 0;
}
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
static int kgd_hqd_sdma_load(struct kgd_dev *kgd, void *mqd)
{
struct cik_sdma_rlc_registers *m;
uint32_t sdma_base_addr;
m = get_sdma_mqd(mqd);
sdma_base_addr = get_sdma_base_addr(m);
write_register(kgd,
sdma_base_addr + SDMA0_RLC0_VIRTUAL_ADDR,
m->sdma_rlc_virtual_addr);
write_register(kgd,
sdma_base_addr + SDMA0_RLC0_RB_BASE,
m->sdma_rlc_rb_base);
write_register(kgd,
sdma_base_addr + SDMA0_RLC0_RB_BASE_HI,
m->sdma_rlc_rb_base_hi);
write_register(kgd,
sdma_base_addr + SDMA0_RLC0_RB_RPTR_ADDR_LO,
m->sdma_rlc_rb_rptr_addr_lo);
write_register(kgd,
sdma_base_addr + SDMA0_RLC0_RB_RPTR_ADDR_HI,
m->sdma_rlc_rb_rptr_addr_hi);
write_register(kgd,
sdma_base_addr + SDMA0_RLC0_DOORBELL,
m->sdma_rlc_doorbell);
write_register(kgd,
sdma_base_addr + SDMA0_RLC0_RB_CNTL,
m->sdma_rlc_rb_cntl);
return 0;
}
drm/amd: Fixing typos in kfd<->kgd interface Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-09 18:00:09 +08:00
static bool kgd_hqd_is_occupied(struct kgd_dev *kgd, uint64_t queue_address,
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
uint32_t pipe_id, uint32_t queue_id)
{
uint32_t act;
bool retval = false;
uint32_t low, high;
acquire_queue(kgd, pipe_id, queue_id);
act = read_register(kgd, CP_HQD_ACTIVE);
if (act) {
low = lower_32_bits(queue_address >> 8);
high = upper_32_bits(queue_address >> 8);
if (low == read_register(kgd, CP_HQD_PQ_BASE) &&
high == read_register(kgd, CP_HQD_PQ_BASE_HI))
retval = true;
}
release_queue(kgd);
return retval;
}
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
static bool kgd_hqd_sdma_is_occupied(struct kgd_dev *kgd, void *mqd)
{
struct cik_sdma_rlc_registers *m;
uint32_t sdma_base_addr;
uint32_t sdma_rlc_rb_cntl;
m = get_sdma_mqd(mqd);
sdma_base_addr = get_sdma_base_addr(m);
sdma_rlc_rb_cntl = read_register(kgd,
sdma_base_addr + SDMA0_RLC0_RB_CNTL);
if (sdma_rlc_rb_cntl & SDMA_RB_ENABLE)
return true;
return false;
}
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
static int kgd_hqd_destroy(struct kgd_dev *kgd, uint32_t reset_type,
unsigned int timeout, uint32_t pipe_id,
uint32_t queue_id)
{
uint32_t temp;
acquire_queue(kgd, pipe_id, queue_id);
write_register(kgd, CP_HQD_PQ_DOORBELL_CONTROL, 0);
write_register(kgd, CP_HQD_DEQUEUE_REQUEST, reset_type);
while (true) {
temp = read_register(kgd, CP_HQD_ACTIVE);
if (temp & 0x1)
break;
if (timeout == 0) {
pr_err("kfd: cp queue preemption time out (%dms)\n",
temp);
drm/radeon: do not leave queue acquired if timeout happens in kgd_hqd_destroy() If timeout happens, kgd_hqd_destroy() just returns -ETIME leaving queue acquired. It may cause a deadlock, so the patch proposes to release queue before return. Found by Linux Driver Verification project (linuxtesting.org). Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2015-01-04 07:31:20 +08:00
release_queue(kgd);
drm/radeon: Add radeon <--> amdkfd interface This patch adds the interface between the radeon driver and the amdkfd driver. The interface implementation is contained in radeon_kfd.c and radeon_kfd.h. The interface itself is represented by a pointer to struct kfd_dev. The pointer is located inside radeon_device structure. All the register accesses that amdkfd need are done using this interface. This allows us to avoid direct register accesses in amdkfd proper, while also avoiding locking between amdkfd and radeon. The single exception is the doorbells that are used in both of the drivers. However, because they are located in separate pci bar pages, the danger of sharing registers between the drivers is minimal. Having said that, we are planning to move the doorbells as well to radeon. v3: Add interface for sa manager init and fini. The init function will allocate a buffer on system memory and pin it to the GART address space via the radeon sa manager. All mappings of buffers to GART address space are done via the radeon sa manager. The interface of allocate memory will use the radeon sa manager to sub allocate from the single buffer that was allocated during the init function. Change lower_32/upper_32 calls to use linux macros Add documentation for the interface v4: Change ptr field type in kgd_mem from uint32_t* to void* to match to type that is returned by radeon_sa_bo_cpu_addr v5: Change format of mqd structure to work with latest KV firmware Add support for AQL queues creation to enable working with open-source HSA runtime. Move generic kfd-->kgd interface and other generic kgd definitions to a generic header file that will be used by AMD's radeon and amdgpu drivers Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-15 18:53:32 +08:00
return -ETIME;
}
msleep(20);
timeout -= 20;
}
release_queue(kgd);
return 0;
}
drm/radeon: Add implementation of get_fw_version This patch implements a new interface that was added to the kfd-->kgd interface. The new interface function retrieves the firmware version that is currently in use by a specific engine. The firmware was uploaded to the engine by the radeon driver. v2: Returns the fw version of the specific engine, as passed into the function by a new parameter Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-11-09 18:45:11 +08:00
drm/radeon: Implement SDMA interface functions This patch implements the new SDMA interface functions. It also adds defines and structures related to SDMA registers. v2: Removed init_sdma_engines() from interface. Initialization is done in radeon. v3: - Removed unused defines. - Added SDMA_ prefix to defines that didn't have them. Signed-off-by: Ben Goz <ben.goz@amd.com> Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 04:12:30 +08:00
static int kgd_hqd_sdma_destroy(struct kgd_dev *kgd, void *mqd,
unsigned int timeout)
{
struct cik_sdma_rlc_registers *m;
uint32_t sdma_base_addr;
uint32_t temp;
m = get_sdma_mqd(mqd);
sdma_base_addr = get_sdma_base_addr(m);
temp = read_register(kgd, sdma_base_addr + SDMA0_RLC0_RB_CNTL);
temp = temp & ~SDMA_RB_ENABLE;
write_register(kgd, sdma_base_addr + SDMA0_RLC0_RB_CNTL, temp);
while (true) {
temp = read_register(kgd, sdma_base_addr +
SDMA0_RLC0_CONTEXT_STATUS);
if (temp & SDMA_RLC_IDLE)
break;
if (timeout == 0)
return -ETIME;
msleep(20);
timeout -= 20;
}
write_register(kgd, sdma_base_addr + SDMA0_RLC0_DOORBELL, 0);
write_register(kgd, sdma_base_addr + SDMA0_RLC0_RB_RPTR, 0);
write_register(kgd, sdma_base_addr + SDMA0_RLC0_RB_WPTR, 0);
write_register(kgd, sdma_base_addr + SDMA0_RLC0_RB_BASE, 0);
return 0;
}
drm/radeon: Add implementation of get_fw_version This patch implements a new interface that was added to the kfd-->kgd interface. The new interface function retrieves the firmware version that is currently in use by a specific engine. The firmware was uploaded to the engine by the radeon driver. v2: Returns the fw version of the specific engine, as passed into the function by a new parameter Signed-off-by: Oded Gabbay <oded.gabbay@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-11-09 18:45:11 +08:00
static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type)
{
struct radeon_device *rdev = (struct radeon_device *) kgd;
const union radeon_firmware_header *hdr;
BUG_ON(kgd == NULL || rdev->mec_fw == NULL);
switch (type) {
case KGD_ENGINE_PFP:
hdr = (const union radeon_firmware_header *) rdev->pfp_fw->data;
break;
case KGD_ENGINE_ME:
hdr = (const union radeon_firmware_header *) rdev->me_fw->data;
break;
case KGD_ENGINE_CE:
hdr = (const union radeon_firmware_header *) rdev->ce_fw->data;
break;
case KGD_ENGINE_MEC1:
hdr = (const union radeon_firmware_header *) rdev->mec_fw->data;
break;
case KGD_ENGINE_MEC2:
hdr = (const union radeon_firmware_header *)
rdev->mec2_fw->data;
break;
case KGD_ENGINE_RLC:
hdr = (const union radeon_firmware_header *) rdev->rlc_fw->data;
break;
case KGD_ENGINE_SDMA:
hdr = (const union radeon_firmware_header *)
rdev->sdma_fw->data;
break;
default:
return 0;
}
if (hdr == NULL)
return 0;
/* Only 12 bit in use*/
return hdr->common.ucode_version;
}