Commit Graph

46 Commits

Author SHA1 Message Date
Xuan Zhuo a10fba0377 virtio: find_vqs() add arg sizes
find_vqs() adds a new parameter sizes to specify the size of each vq
vring.

NULL as sizes means that all queues in find_vqs() use the maximum size.
A value in the array is 0, which means that the corresponding queue uses
the maximum size.

In the split scenario, the meaning of size is the largest size, because
it may be limited by memory, the virtio core will try a smaller size.
And the size is power of 2.

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Acked-by: Hans de Goede <hdegoede@redhat.com>
Reviewed-by: Mathieu Poirier <mathieu.poirier@linaro.org>
Acked-by: Jason Wang <jasowang@redhat.com>
Message-Id: <20220801063902.129329-34-xuanzhuo@linux.alibaba.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2022-08-11 04:06:40 -04:00
Xuan Zhuo da80296183 virtio: record the maximum queue num supported by the device.
virtio-net can display the maximum (supported by hardware) ring size in
ethtool -g eth0.

When the subsequent patch implements vring reset, it can judge whether
the ring size passed by the driver is legal based on this.

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Acked-by: Jason Wang <jasowang@redhat.com>
Message-Id: <20220801063902.129329-2-xuanzhuo@linux.alibaba.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2022-08-11 04:06:38 -04:00
Xuan Zhuo c2a052a4a9 remoteproc: rename len of rpoc_vring to num
Rename the member len in the structure rpoc_vring to num. And remove 'in
bytes' from the comment of it. This is misleading. Because this actually
refers to the size of the virtio vring to be created. The unit is not
bytes.

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Message-Id: <20220624025621.128843-2-xuanzhuo@linux.alibaba.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2022-08-11 04:06:37 -04:00
Arnaud Pouliquen 9955548919 remoteproc: Remove vdev_to_rvdev and vdev_to_rproc from remoteproc API
These both functions are only used by the remoteproc_virtio.
There is no reason to expose them in the API.
Move the functions in remoteproc_virtio.c

Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
Link: https://lore.kernel.org/r/20211001101234.4247-4-arnaud.pouliquen@foss.st.com
2021-10-15 09:49:55 -05:00
Suman Anna f286743400 remoteproc: Fix various kernel-doc warnings
Fix all the kernel-doc warnings in various remoteproc core files.
Some of them just needed a formatting cleanup change, while others
needed the Return statement to be added, or documenting the missed
structure elements.

Reviewed-by: Mathieu Poirier <mathieu.poirier@linaro.org>
Signed-off-by: Suman Anna <s-anna@ti.com>
Link: https://lore.kernel.org/r/20210519180304.23563-3-s-anna@ti.com
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2021-05-27 22:07:43 -05:00
Christoph Hellwig 0a0f0d8be7 dma-mapping: split <linux/dma-mapping.h>
Split out all the bits that are purely for dma_map_ops implementations
and related code into a new <linux/dma-map-ops.h> header so that they
don't get pulled into all the drivers.  That also means the architecture
specific <asm/dma-mapping.h> is not pulled in by <linux/dma-mapping.h>
any more, which leads to a missing includes that were pulled in by the
x86 or arm versions in a few not overly portable drivers.

Signed-off-by: Christoph Hellwig <hch@lst.de>
2020-10-06 07:07:03 +02:00
Tero Kristo db9178a4f8 remoteproc: Fall back to using parent memory pool if no dedicated available
In some cases, like with OMAP remoteproc, we are not creating dedicated
memory pool for the virtio device. Instead, we use the same memory pool
for all shared memories. The current virtio memory pool handling forces
a split between these two, as a separate device is created for it,
causing memory to be allocated from bad location if the dedicated pool
is not available. Fix this by falling back to using the parent device
memory pool if dedicated is not available.

Cc: stable@vger.kernel.org
Reviewed-by: Mathieu Poirier <mathieu.poirier@linaro.org>
Acked-by: Arnaud Pouliquen <arnaud.pouliquen@st.com>
Fixes: 086d08725d ("remoteproc: create vdev subdevice with specific dma memory pool")
Signed-off-by: Tero Kristo <t-kristo@ti.com>
Signed-off-by: Suman Anna <s-anna@ti.com>
Link: https://lore.kernel.org/r/20200420160600.10467-2-s-anna@ti.com
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2020-05-12 16:08:00 -07:00
Christophe JAILLET 2fb75ceaf7 remoteproc: Add missing '\n' in log messages
Message logged by 'dev_xxx()' or 'pr_xxx()' should end with a '\n'.

Fixes: 791c13b709 ("remoteproc: Fix NULL pointer dereference in rproc_virtio_notify")
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
Link: https://lore.kernel.org/r/20200411160750.32573-1-christophe.jaillet@wanadoo.fr
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2020-04-22 22:20:40 -07:00
Nikita Shubin 791c13b709 remoteproc: Fix NULL pointer dereference in rproc_virtio_notify
Undefined rproc_ops .kick method in remoteproc driver will result in
"Unable to handle kernel NULL pointer dereference" in rproc_virtio_notify,
after firmware loading if:

 1) .kick method wasn't defined in driver
 2) resource_table exists in firmware and has "Virtio device entry" defined

Let's refuse to register an rproc-induced virtio device if no kick method was
defined for rproc.

[   13.180049][  T415] 8<--- cut here ---
[   13.190558][  T415] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[   13.212544][  T415] pgd = (ptrval)
[   13.217052][  T415] [00000000] *pgd=00000000
[   13.224692][  T415] Internal error: Oops: 80000005 [#1] PREEMPT SMP ARM
[   13.231318][  T415] Modules linked in: rpmsg_char imx_rproc virtio_rpmsg_bus rpmsg_core [last unloaded: imx_rproc]
[   13.241687][  T415] CPU: 0 PID: 415 Comm: unload-load.sh Not tainted 5.5.2-00002-g707df13bbbdd #6
[   13.250561][  T415] Hardware name: Freescale i.MX7 Dual (Device Tree)
[   13.257009][  T415] PC is at 0x0
[   13.260249][  T415] LR is at rproc_virtio_notify+0x2c/0x54
[   13.265738][  T415] pc : [<00000000>]    lr : [<8050f6b0>]    psr: 60010113
[   13.272702][  T415] sp : b8d47c48  ip : 00000001  fp : bc04de00
[   13.278625][  T415] r10: bc04c000  r9 : 00000cc0  r8 : b8d46000
[   13.284548][  T415] r7 : 00000000  r6 : b898f200  r5 : 00000000  r4 : b8a29800
[   13.291773][  T415] r3 : 00000000  r2 : 990a3ad4  r1 : 00000000  r0 : b8a29800
[   13.299000][  T415] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   13.306833][  T415] Control: 10c5387d  Table: b8b4806a  DAC: 00000051
[   13.313278][  T415] Process unload-load.sh (pid: 415, stack limit = 0x(ptrval))
[   13.320591][  T415] Stack: (0xb8d47c48 to 0xb8d48000)
[   13.325651][  T415] 7c40:                   b895b680 00000001 b898f200 803c6430 b895bc80 7f00ae18
[   13.334531][  T415] 7c60: 00000035 00000000 00000000 b9393200 80b3ed80 00004000 b9393268 bbf5a9a2
[   13.343410][  T415] 7c80: 00000e00 00000200 00000000 7f00aff0 7f00a014 b895b680 b895b800 990a3ad4
[   13.352290][  T415] 7ca0: 00000001 b898f210 b898f200 00000000 00000000 7f00e000 00000001 00000000
[   13.361170][  T415] 7cc0: 00000000 803c62e0 80b2169c 802a0924 b898f210 00000000 00000000 b898f210
[   13.370049][  T415] 7ce0: 80b9ba44 00000000 80b9ba48 00000000 7f00e000 00000008 80b2169c 80400114
[   13.378929][  T415] 7d00: 80b2169c 8061fd64 b898f210 7f00e000 80400744 b8d46000 80b21634 80b21634
[   13.387809][  T415] 7d20: 80b2169c 80400614 80b21634 80400718 7f00e000 00000000 b8d47d7c 80400744
[   13.396689][  T415] 7d40: b8d46000 80b21634 80b21634 803fe338 b898f254 b80fe76c b8d32e38 990a3ad4
[   13.405569][  T415] 7d60: fffffff3 b898f210 b8d46000 00000001 b898f254 803ffe7c 80857a90 b898f210
[   13.414449][  T415] 7d80: 00000001 990a3ad4 b8d46000 b898f210 b898f210 80b17aec b8a29c20 803ff0a4
[   13.423328][  T415] 7da0: b898f210 00000000 b8d46000 803fb8e0 b898f200 00000000 80b17aec b898f210
[   13.432209][  T415] 7dc0: b8a29c20 990a3ad4 b895b900 b898f200 8050fb7c 80b17aec b898f210 b8a29c20
[   13.441088][  T415] 7de0: b8a29800 b895b900 b8a29a04 803c5ec0 b8a29c00 b898f200 b8a29a20 00000007
[   13.449968][  T415] 7e00: b8a29c20 8050fd78 b8a29800 00000000 b8a29a20 b8a29c04 b8a29820 b8a299d0
[   13.458848][  T415] 7e20: b895b900 8050e5a4 b8a29800 b8a299d8 b8d46000 b8a299e0 b8a29820 b8a299d0
[   13.467728][  T415] 7e40: b895b900 8050e008 000041ed 00000000 b8b8c440 b8a299d8 b8a299e0 b8a299d8
[   13.476608][  T415] 7e60: b8b8c440 990a3ad4 00000000 b8a29820 b8b8c400 00000006 b8a29800 b895b880
[   13.485487][  T415] 7e80: b8d47f78 00000000 00000000 8050f4b4 00000006 b895b890 b8b8c400 008fbea0
[   13.494367][  T415] 7ea0: b895b880 8029f530 00000000 00000000 b8d46000 00000006 b8d46000 008fbea0
[   13.503246][  T415] 7ec0: 8029f434 00000000 b8d46000 00000000 00000000 8021e2e4 0000000a 8061fd0c
[   13.512125][  T415] 7ee0: 0000000a b8af0c00 0000000a b8af0c40 00000001 b8af0c40 00000000 8061f910
[   13.521005][  T415] 7f00: 0000000a 80240af4 00000002 b8d46000 00000000 8061fd0c 00000002 80232d7c
[   13.529884][  T415] 7f20: 00000000 b8d46000 00000000 990a3ad4 00000000 00000006 b8a62d80 008fbea0
[   13.538764][  T415] 7f40: b8d47f78 00000000 b8d46000 00000000 00000000 802210c0 b88f2900 00000000
[   13.547644][  T415] 7f60: b8a62d80 b8a62d80 b8d46000 00000006 008fbea0 80221320 00000000 00000000
[   13.556524][  T415] 7f80: b8af0c00 990a3ad4 0000006c 008fbea0 76f1cda0 00000004 80101204 00000004
[   13.565403][  T415] 7fa0: 00000000 80101000 0000006c 008fbea0 00000001 008fbea0 00000006 00000000
[   13.574283][  T415] 7fc0: 0000006c 008fbea0 76f1cda0 00000004 00000006 00000006 00000000 00000000
[   13.583162][  T415] 7fe0: 00000004 7ebaf7d0 76eb4c0b 76e3f206 600d0030 00000001 00000000 00000000
[   13.592056][  T415] [<8050f6b0>] (rproc_virtio_notify) from [<803c6430>] (virtqueue_notify+0x1c/0x34)
[   13.601298][  T415] [<803c6430>] (virtqueue_notify) from [<7f00ae18>] (rpmsg_probe+0x280/0x380 [virtio_rpmsg_bus])
[   13.611663][  T415] [<7f00ae18>] (rpmsg_probe [virtio_rpmsg_bus]) from [<803c62e0>] (virtio_dev_probe+0x1f8/0x2c4)
[   13.622022][  T415] [<803c62e0>] (virtio_dev_probe) from [<80400114>] (really_probe+0x200/0x450)
[   13.630817][  T415] [<80400114>] (really_probe) from [<80400614>] (driver_probe_device+0x16c/0x1ac)
[   13.639873][  T415] [<80400614>] (driver_probe_device) from [<803fe338>] (bus_for_each_drv+0x84/0xc8)
[   13.649102][  T415] [<803fe338>] (bus_for_each_drv) from [<803ffe7c>] (__device_attach+0xd4/0x164)
[   13.658069][  T415] [<803ffe7c>] (__device_attach) from [<803ff0a4>] (bus_probe_device+0x84/0x8c)
[   13.666950][  T415] [<803ff0a4>] (bus_probe_device) from [<803fb8e0>] (device_add+0x444/0x768)
[   13.675572][  T415] [<803fb8e0>] (device_add) from [<803c5ec0>] (register_virtio_device+0xa4/0xfc)
[   13.684541][  T415] [<803c5ec0>] (register_virtio_device) from [<8050fd78>] (rproc_add_virtio_dev+0xcc/0x1b8)
[   13.694466][  T415] [<8050fd78>] (rproc_add_virtio_dev) from [<8050e5a4>] (rproc_start+0x148/0x200)
[   13.703521][  T415] [<8050e5a4>] (rproc_start) from [<8050e008>] (rproc_boot+0x384/0x5c0)
[   13.711708][  T415] [<8050e008>] (rproc_boot) from [<8050f4b4>] (state_store+0x3c/0xc8)
[   13.719723][  T415] [<8050f4b4>] (state_store) from [<8029f530>] (kernfs_fop_write+0xfc/0x214)
[   13.728348][  T415] [<8029f530>] (kernfs_fop_write) from [<8021e2e4>] (__vfs_write+0x30/0x1cc)
[   13.736971][  T415] [<8021e2e4>] (__vfs_write) from [<802210c0>] (vfs_write+0xac/0x17c)
[   13.744985][  T415] [<802210c0>] (vfs_write) from [<80221320>] (ksys_write+0x64/0xe4)
[   13.752825][  T415] [<80221320>] (ksys_write) from [<80101000>] (ret_fast_syscall+0x0/0x54)
[   13.761178][  T415] Exception stack(0xb8d47fa8 to 0xb8d47ff0)
[   13.766932][  T415] 7fa0:                   0000006c 008fbea0 00000001 008fbea0 00000006 00000000
[   13.775811][  T415] 7fc0: 0000006c 008fbea0 76f1cda0 00000004 00000006 00000006 00000000 00000000
[   13.784687][  T415] 7fe0: 00000004 7ebaf7d0 76eb4c0b 76e3f206
[   13.790442][  T415] Code: bad PC value
[   13.839214][  T415] ---[ end trace 1fe21ecfc9f28852 ]---

Reviewed-by: Mathieu Poirier <mathieu.poirier@linaro.org>
Signed-off-by: Nikita Shubin <NShubin@topcon.com>
Fixes: 7a18694162 ("remoteproc: remove the single rpmsg vdev limitation")
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/r/20200306072452.24743-1-NShubin@topcon.com
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2020-03-25 22:29:44 -07:00
Arnaud Pouliquen 2e7d4c2c4b remoteproc: fix kernel-doc warnings
Fix the following warnings when documentation is built:
drivers/remoteproc/remoteproc_virtio.c:330: warning: Function parameter
or member 'id' not described in 'rproc_add_virtio_dev'
drivers/remoteproc/remoteproc_core.c:243: warning: Function parameter
or member 'name' not described in 'rproc_find_carveout_by_name'
drivers/remoteproc/remoteproc_core.c:473: warning: Function parameter
or member 'offset' not described in 'rproc_handle_vdev'
drivers/remoteproc/remoteproc_core.c:604: warning: Function parameter
or member 'offset' not described in 'rproc_handle_trace'
drivers/remoteproc/remoteproc_core.c:678: warning: Function parameter
or member 'offset' not described in 'rproc_handle_devmem'
drivers/remoteproc/remoteproc_core.c:873: warning: Function parameter
or member 'offset' not described in 'rproc_handle_carveout'
drivers/remoteproc/remoteproc_core.c:1029: warning: cannot understand function
prototype: 'rproc_handle_resource_t rproc_loading_handlers[RSC_LAST] = '
drivers/remoteproc/remoteproc_core.c:1693: warning: Function parameter
or member 'work' not described in 'rproc_crash_handler_work'

Acked-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@st.com>
Link: https://lore.kernel.org/r/20200212161956.10358-1-arnaud.pouliquen@st.com
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2020-03-25 22:29:38 -07:00
Thomas Gleixner 9c92ab6191 treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 282
Based on 1 normalized pattern(s):

  this software is licensed under the terms of the gnu general public
  license version 2 as published by the free software foundation and
  may be copied distributed and modified under those terms this
  program is distributed in the hope that it will be useful but
  without any warranty without even the implied warranty of
  merchantability or fitness for a particular purpose see the gnu
  general public license for more details

extracted by the scancode license scanner the SPDX license identifier

  GPL-2.0-only

has been chosen to replace the boilerplate/reference in 285 file(s).

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Alexios Zavras <alexios.zavras@intel.com>
Reviewed-by: Allison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190529141900.642774971@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-06-05 17:36:37 +02:00
Linus Torvalds 2f194646fe remoteproc updates for v5.1
This contains the last patches in Loic's remoteproc resource table
 handling changes, a number of updates to documentation, support for
 invoking the crash handler (for testing purposes), a fix for the
 handling of virtio devices during recovery, performance state votes in
 Qualcomm modem driver, support for specifying board specific firmware
 path for Qualcomm modem driver and improved support for graceful
 shutdown of Qualcomm remoteprocs.
 -----BEGIN PGP SIGNATURE-----
 
 iQJPBAABCAA5FiEEBd4DzF816k8JZtUlCx85Pw2ZrcUFAlyJS1kbHGJqb3JuLmFu
 ZGVyc3NvbkBsaW5hcm8ub3JnAAoJEAsfOT8Nma3FgtYP/RGxDkPOAXZmJSlurz+Z
 syWop/9s1nLqkSVtXDZ6pthuPtHrQEyAIOfdCbot6X0ZCuAzTnhtdJUzMFlXMBrd
 MDwdwxZEobr1bngz8VENrt1Lny0oy6Zm8ajyr/+xdb3Xw74mcmJ7UDjyIh1w8YPm
 QA6acCYCzRBIwskhlBSqVT2Mf8imAXr1bBV2nh5ngS9+5r1UkGO4/dnXs8mZQRWU
 ZaIfWT9kN2cIxHQH5c5bHCaYP/dmoz5FGyBVYq4OZSuFq+ziG3dQ+VA3YJFv7yUI
 /3P9r4VEJYpR8Tg08lG6SvEtvZ0e0zdtNX0FzYYF8KnpSjB3Vf8k5u/RNnZRn33v
 k5QQivC+YGpG7Rwy1pgyNtZQ881KsmN6JVQxT1rm6ddriSl4TnQXO32lswD2YmM4
 Gt/+Sgu18KEa1ASbOUQk/cRS8Zr3kg2+xDrgBh+Sp0265t+6V9/cGb+PJKkJn9RD
 HRE3fXDdQRwEmp/rhx03kSdlhbfg5A9mTLxb/T5kBR/nC1qLYZV592mvBZS08imk
 7T2zquqP+anwKDOLE++PgJbx6nlzFnPqDTjzohHWns5MfYSGEHagQ21IrDJuzkhG
 nr2r1Gpwkl1dDarvAgYUMCcWGCWvSrUpLfbKYR9QaKWo1JTTDbocBIFF8j7a+eVf
 EiKlGv7SX1LUpXk4K+WabbCr
 =4xC9
 -----END PGP SIGNATURE-----

Merge tag 'rproc-v5.1' of git://github.com/andersson/remoteproc

Pull remoteproc updates from Bjorn Andersson:
 "This contains the last patches in Loic's remoteproc resource table
  handling changes, a number of updates to documentation, support for
  invoking the crash handler (for testing purposes), a fix for the
  handling of virtio devices during recovery, performance state votes in
  Qualcomm modem driver, support for specifying board specific firmware
  path for Qualcomm modem driver and improved support for graceful
  shutdown of Qualcomm remoteprocs"

* tag 'rproc-v5.1' of git://github.com/andersson/remoteproc: (33 commits)
  remoteproc: fix for "dma-mapping: remove the DMA_MEMORY_EXCLUSIVE flag"
  remoteproc: fix rproc_check_carveout_da() returned error and comments
  remoteproc: fix trace buffer va initialization
  remoteproc: fix rproc_alloc_carveout() for rproc with iommu domain
  remoteproc: add warning on resource table cast
  remoteproc: fix rproc_alloc_carveout() bad variable cast
  remoteproc: fix rproc_da_to_va in case of unallocated carveout
  remoteproc: correct rproc_mem_entry_init() comments
  remoteproc: fix recovery procedure
  rpmsg: virtio: change header file sort style
  rpmsg: virtio: allocate buffer from parent
  remoteproc: st: add reserved memory support
  remoteproc: create vdev subdevice with specific dma memory pool
  remoteproc: q6v5_adsp: Remove voting for lpass_aon clock
  dt-binding: remoteproc: Remove lpass_aon clock from adsp pil clock list
  remoteproc: q6v5-mss: Active powerdomain for SDM845
  remoteproc: q6v5-mss: Vote for rpmh power domains
  remoteproc: qcom: Add support for parsing fw dt bindings
  remoteproc: qcom_q6v5: don't auto boot remote processor
  remoteproc: qcom: Wait for shutdown-ack/ind on sysmon shutdown
  ...
2019-03-14 09:00:06 -07:00
Stephen Rothwell d664ce75ae remoteproc: fix for "dma-mapping: remove the DMA_MEMORY_EXCLUSIVE flag"
The commit 82c5de0ab8 ("dma-mapping: remove the DMA_MEMORY_EXCLUSIVE
flag") removed the "flags" parameter for dma_declare_coherent_memory().
Remove the parameter from the call in rproc_add_virtio_dev().

Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
[bjorn: Extended commit message]
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2019-03-12 17:00:42 -07:00
Loic Pallardy d4c036fec3 remoteproc: fix recovery procedure
Commit 7e83cab824a87e83cab824a8 ("remoteproc: Modify recovery path
to use rproc_{start,stop}()") replaces rproc_{shutdown,boot}() with
rproc_{stop,start}(), which skips destroy the virtio device at stop
but re-initializes it again at start.

Issue is that struct virtio_dev is not correctly reinitialized like done
at initial allocation thanks to kzalloc() and kobject is considered as
already initialized by kernel. That is due to the fact struct virtio_dev
is allocated and released at vdev resource handling level managed and
virtio device is registered and unregistered at rproc subdevices level.

Moreover kernel documentation mentions that device struct must be
zero initialized before calling device_initialize().

This patch disentangles struct virtio_dev from struct rproc_vdev as
the two struct don't have the same life-cycle.

struct virtio_dev is now allocated on rproc_start() and released
on rproc_stop().

This patch applies on top of patch
remoteproc: create vdev subdevice with specific dma memory pool [1]

[1]: https://patchwork.kernel.org/patch/10755781/

Fixes: 7e83cab824 ("remoteproc: Modify recovery path to use rproc_{start,stop}()")

Reported-by: Xiang Xiao <xiaoxiang781216@gmail.com>
Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2019-02-20 21:34:17 -08:00
Loic Pallardy 086d08725d remoteproc: create vdev subdevice with specific dma memory pool
This patch creates a dedicated vdev subdevice for each vdev declared
in firmware resource table and associates carveout named "vdev%dbuffer"
(with %d vdev index in resource table) if any as dma coherent memory pool.

Then vdev subdevice is used as parent for virtio device.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2019-02-20 21:15:35 -08:00
Wei Wang a229989d97 virtio: don't allocate vqs when names[i] = NULL
Some vqs may not need to be allocated when their related feature bits
are disabled. So callers may pass in such vqs with "names = NULL".
Then we skip such vq allocations.

Signed-off-by: Wei Wang <wei.w.wang@intel.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Cc: stable@vger.kernel.org
Fixes: 86a559787e ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")
2019-01-14 20:15:19 -05:00
Tiwei Bie 3a814fdf27 virtio_ring: disable packed ring on unsupported transports
Currently, ccw, vop and remoteproc need some legacy virtio
APIs to create or access virtio rings, which are not supported
by packed ring. So disable packed ring on these transports
for now.

Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-11-26 22:17:40 -08:00
Loic Pallardy c6aed238b7 remoteproc: modify vring allocation to rely on centralized carveout allocator
Current version of rproc_alloc_vring function supports only dynamic vring
allocation.

This patch allows to allocate vrings based on memory region declatation.
Vrings are now manage like memory carveouts, to communize memory management
code in rproc_alloc_registered_carveouts().

Allocated buffer is retrieved in rp_find_vq() thanks to
rproc_find_carveout_by_name() functions for.

This patch sets vrings names to vdev"x"vring"y" with x vdev index in
resource table and y vring index in vdev. This will be updated when
name will be associated to vdev in firmware resource table.

Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2018-10-14 23:37:40 -07:00
Loic Pallardy 276ec99342 remoteproc: replace "%p" with "%pK"
The format specifier "%p" can leak kernel addresses.
Use "%pK" instead.

This patch proposes changes for remoteproc core only.

Acked-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Loic Pallardy <loic.pallardy@st.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2018-07-30 23:24:15 -07:00
weiping zhang 900a163ede virtio_remoteproc: correct put_device virtio_device.dev
rproc_virtio_dev_release will be called iff virtio_device.dev's
reference count drops to 0. Here we just put vdev.dev, and then
rproc->dev's cleanup will be done in rproc_virtio_dev_release.

Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: weiping zhang <zhangweiping@didichuxing.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2018-01-02 09:32:30 -08:00
Michael S. Tsirkin f94682dde5 virtio: add context flag to find vqs
Allows maintaining extra context per vq.  For ease of use, passing in
NULL is legal and disables the feature for all vqs.

Includes fixes by Christian for s390, acked by Cornelia.

Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Acked-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2017-05-02 23:41:43 +03:00
Christoph Hellwig fb5e31d970 virtio: allow drivers to request IRQ affinity when creating VQs
Add a struct irq_affinity pointer to the find_vqs methods, which if set
is used to tell the PCI layer to create the MSI-X vectors for our I/O
virtqueues with the proper affinity from the start.  Compared to after
the fact affinity hints this gives us an instantly working setup and
allows to allocate the irq descritors node-local and avoid interconnect
traffic.  Last but not least this will allow blk-mq queues are created
based on the interrupt affinity for storage drivers.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2017-02-27 20:54:04 +02:00
Bjorn Andersson a863af5d41 remoteproc: virtio: Anchor vring life cycle in vdev
Instead of having the vrings being allocated and freed as they are
requested by the virtio device tie their life cycle to the vdev
resource. This allows us to decouple the vdev resource management from
the virtio device management.

Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2016-11-14 21:52:16 -08:00
Bjorn Andersson aab8d80223 remoteproc: Assign kref to rproc_vdev
No functional change

Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2016-11-14 21:52:15 -08:00
Bjorn Andersson ddf711872c remoteproc: Introduce auto-boot flag
Introduce an "auto-boot" flag on rprocs to make it possible to flag
remote processors without vdevs to automatically boot once the firmware
is found.

Preserve previous behavior of the wkup_m3 processor being explicitly
booted by a consumer.

Cc: Lee Jones <lee.jones@linaro.org>
Cc: Loic Pallardy <loic.pallardy@st.com>
Cc: Suman Anna <s-anna@ti.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2016-08-17 23:17:42 -07:00
Anna, Suman 730f84ce6d remoteproc: align code with open parenthesis
This patch fixes the existing alignment checkpatch check
warnings of the type "Alignment should match open parenthesis"
in the remoteproc core source files.

Signed-off-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2016-08-12 21:27:49 -07:00
Anna, Suman f145928d49 remoteproc: fix bare unsigned type usage
While there is nothing wrong with defining an unsigned integer
variable or argument using the bare unsigned type, it is better
to use the checkpatch preferred 'unsigned int' type.

Signed-off-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2016-08-12 21:27:48 -07:00
Lee Jones 3d87fa1d47 remoteproc: core: Task sync during rproc_fw_boot()
By default, rproc_fw_boot() needs to wait for rproc to be configured,
but a race may occur when using rpmsg/virtio.  In this case, it can
be called locally in a safe manor.

This patch represents two usecases:

 - External call (via exported rproc_boot()), which waits
 - Internal call can use 'nowait' version of rproc_boot()

Signed-off-by: Fabrice Gasnier <fabrice.gasnier@st.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2016-05-06 11:46:36 -07:00
Stefan Hajnoczi f7ad26ff95 virtio: make find_vqs() checkpatch.pl-friendly
checkpatch.pl wants arrays of strings declared as follows:

  static const char * const names[] = { "vq-1", "vq-2", "vq-3" };

Currently the find_vqs() function takes a const char *names[] argument
so passing checkpatch.pl's const char * const names[] results in a
compiler error due to losing the second const.

This patch adjusts the find_vqs() prototype and updates all virtio
transports.  This makes it possible for virtio_balloon.c, virtio_input.c,
virtgpu_kms.c, and virtio_rpmsg_bus.c to use the checkpatch.pl-friendly
type.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: Bjorn Andersson <bjorn.andersson@sonymobile.com>
2016-01-12 20:47:06 +02:00
Michael S. Tsirkin 5c609a5ef0 virtio: allow finalize_features to fail
This will make it easy for transports to validate features and return
failure.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2014-12-09 16:32:32 +02:00
Michael S. Tsirkin 93d389f820 virtio: assert 32 bit features in transports
At this point, no transports set any of the high 32 feature bits.
Since transports generally can't (yet) cope with such bits, add BUG_ON
checks to make sure they are not set by mistake.

Based on rproc patch by Rusty.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
2014-12-09 12:05:24 +02:00
Michael S. Tsirkin d025477368 virtio: add support for 64 bit features.
Change u32 to u64, and use BIT_ULL and 1ULL everywhere.

Note: transports are unchanged, and only set low 32 bit.
This guarantees that no transport sets e.g. VERSION_1
by mistake without proper support.

Based on patch by Rusty.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
2014-12-09 12:05:24 +02:00
Michael S. Tsirkin e16e12be34 virtio: use u32, not bitmap for features
It seemed like a good idea to use bitmap for features
in struct virtio_device, but it's actually a pain,
and seems to become even more painful when we get more
than 32 feature bits.  Just change it to a u32 for now.

Based on patch by Rusty.

Suggested-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
2014-12-09 12:05:23 +02:00
Heinz Graalfs 46f9c2b925 virtio_ring: change host notification API
Currently a host kick error is silently ignored and not reflected in
the virtqueue of a particular virtio device.

Changing the notify API for guest->host notification seems to be one
prerequisite in order to be able to handle such errors in the context
where the kick is triggered.

This patch changes the notify API. The notify function must return a
bool return value. It returns false if the host notification failed.

Signed-off-by: Heinz Graalfs <graalfs@linux.vnet.ibm.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2013-10-29 11:28:11 +10:30
Sjur Brændeland 92b38f8514 remoteproc: support virtio config space.
Support virtio configuration space and device status. The virtio
device can now access the resource table in shared memory.

Signed-off-by: Sjur Brændeland <sjur.brandeland@stericsson.com>
Acked-by: Ido Yariv <ido@wizery.com>
[rebase and style changes]
Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
2013-04-07 14:06:17 +03:00
Stephen Hemminger 9350393239 virtio: make config_ops const
It is just a table of function pointers, make it const for cleanliness and security
reasons.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2013-02-11 15:32:17 +10:30
Ohad Ben-Cohen dab55bbafd remoteproc: fix error path of ->find_vqs
Eliminate an erroneous invocation of rproc_shutdown inside
the error path of rproc_virtio_find_vqs.

Reported-by: Ido Yariv <ido@wizery.com>
Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
2012-11-29 10:05:09 +02:00
Michael S. Tsirkin 6457f126c8 virtio: support reserved vqs
virtio network device multiqueue support reserves
vq 3 for future use (useful both for future extensions and to make it
pretty - this way receive vqs have even and transmit - odd numbers).
Make it possible to skip initialization for
specific vq numbers by specifying NULL for name.
Document this usage as well as (existing) NULL callback.

Drivers using this not coded up yet, so I simply tested
with virtio-pci and verified that this patch does
not break existing drivers.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-09-28 15:05:15 +09:30
Jason Wang 17bb6d4088 virtio-ring: move queue_index to vring_virtqueue
Instead of storing the queue index in transport-specific virtio structs,
this patch moves them to vring_virtqueue and introduces an helper to get
the value.  This lets drivers simplify their management and tracing of
virtqueues.

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2012-09-28 15:05:15 +09:30
Ohad Ben-Cohen 7183a2a799 remoteproc: remove the now-redundant kref
Now that every rproc instance contains a device, we don't need a
kref anymore to maintain the refcount of the rproc instances:
that's what device are good with!

This patch removes the now-redundant kref, and switches to
{get, put}_device instead of kref_{get, put}.

We also don't need the kref's release function anymore, and instead,
we just utilize the class's release handler (which is now responsible
for all memory de-allocations).

Cc: Stephen Boyd <sboyd@codeaurora.org>
Cc: Fernando Guzman Lugo <fernando.lugo@ti.com>
Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
2012-07-06 00:53:25 +03:00
Ohad Ben-Cohen b5ab5e24e9 remoteproc: maintain a generic child device for each rproc
For each registered rproc, maintain a generic remoteproc device whose
parent is the low level platform-specific device (commonly a pdev, but
it may certainly be any other type of device too).

With this in hand, the resulting device hierarchy might then look like:

omap-rproc.0
 |
 - remoteproc0  <---- new !
    |
    - virtio0
    |
    - virtio1
       |
       - rpmsg0
       |
       - rpmsg1
       |
       - rpmsg2

Where:
- omap-rproc.0 is the low level device that's bound to the
  driver which invokes rproc_register()
- remoteproc0 is the result of this patch, and will be added by the
  remoteproc framework when rproc_register() is invoked
- virtio0 and virtio1 are vdevs that are registered by remoteproc
  when it realizes that they are supported by the firmware
  of the physical remote processor represented by omap-rproc.0
- rpmsg0, rpmsg1 and rpmsg2 are rpmsg devices that represent rpmsg
  channels, and are registerd by the rpmsg bus when it gets notified
  about their existence

Technically, this patch:
- changes 'struct rproc' to contain this generic remoteproc.x device
- creates a new "remoteproc" type, to which this new generic remoteproc.x
  device belong to.
- adds a super simple enumeration method for the indices of the
  remoteproc.x devices
- updates all dev_* messaging to use the generic remoteproc.x device
  instead of the low level platform-specific device
- updates all dma_* allocations to use the parent of remoteproc.x (where
  the platform-specific memory pools, most commonly CMA, are to be found)

Adding this generic device has several merits:
- we can now add remoteproc runtime PM support simply by hooking onto the
  new "remoteproc" type
- all remoteproc log messages will now carry a common name prefix
  instead of having a platform-specific one
- having a device as part of the rproc struct makes it possible to simplify
  refcounting (see subsequent patch)

Thanks to Stephen Boyd <sboyd@codeaurora.org> for suggesting and
discussing these ideas in one of the remoteproc review threads and
to Fernando Guzman Lugo <fernando.lugo@ti.com> for trying them out
with the (upcoming) runtime PM support for remoteproc.

Cc: Fernando Guzman Lugo <fernando.lugo@ti.com>
Reviewed-by: Stephen Boyd <sboyd@codeaurora.org>
Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
2012-07-06 00:53:03 +03:00
Ohad Ben-Cohen 6db20ea8d8 remoteproc: allocate vrings on demand, free when not needed
Dynamically allocate the vrings' DMA when the remote processor
is about to be powered on (i.e. when ->find_vqs() is invoked),
and release them as soon as it is powered off (i.e. when ->del_vqs()
is invoked).

The obvious and immediate benefit is better memory utilization, since
memory for the vrings is now only allocated when the relevant remote
processor is used.

Additionally, this approach also makes recovery of a (crashing)
remote processor easier: one just needs to remove the relevant
vdevs, and the entire vrings cleanup takes place automagically.

Tested-by: Fernando Guzman Lugo <fernando.lugo@ti.com>
Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
2012-07-04 13:27:04 +03:00
Ohad Ben-Cohen 63140e0ed2 remoteproc: remove the hardcoded vring alignment
Remove the hardcoded vring alignment of 4096 bytes,
and instead utilize tha vring alignment as specified in
the resource table.

This is needed for remote processors that have rigid
memory requirement, and which have found the alignment of
4096 bytes to be excessively big.

Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Brian Swetland <swetland@google.com>
Cc: Iliyan Malchev <malchev@google.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Grant Likely <grant.likely@secretlab.ca>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Mark Grosen <mgrosen@ti.com>
Cc: John Williams <john.williams@petalogix.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Loic PALLARDY <loic.pallardy@stericsson.com>
Cc: Ludovic BARRE <ludovic.barre@stericsson.com>
Cc: Omar Ramirez Luna <omar.luna@linaro.org>
Cc: Guzman Lugo Fernando <fernando.lugo@ti.com>
Cc: Anna Suman <s-anna@ti.com>
Cc: Clark Rob <rob@ti.com>
Cc: Stephen Boyd <sboyd@codeaurora.org>
Cc: Saravana Kannan <skannan@codeaurora.org>
Cc: David Brown <davidb@codeaurora.org>
Cc: Kieran Bingham <kieranbingham@gmail.com>
Cc: Tony Lindgren <tony@atomide.com>
2012-03-06 19:14:37 +02:00
Ohad Ben-Cohen 7a18694162 remoteproc: remove the single rpmsg vdev limitation
Now that the resource table supports publishing a virtio device
in a single resource entry, firmware images can start supporting
more than a single vdev.

This patch removes the single vdev limitation of the remoteproc
framework so multi-vdev firmwares can be leveraged: VDEV resource
entries are parsed when the rproc is registered, and as a result
their vrings are set up and the virtio devices are registered
(and they go away when the rproc goes away).

Moreover, we no longer only support VIRTIO_ID_RPMSG vdevs; any
virtio device type goes now. As a result, there's no more any
rpmsg-specific APIs or code in remoteproc: it all becomes generic
virtio handling.

Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Brian Swetland <swetland@google.com>
Cc: Iliyan Malchev <malchev@google.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Grant Likely <grant.likely@secretlab.ca>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Mark Grosen <mgrosen@ti.com>
Cc: John Williams <john.williams@petalogix.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Loic PALLARDY <loic.pallardy@stericsson.com>
Cc: Ludovic BARRE <ludovic.barre@stericsson.com>
Cc: Omar Ramirez Luna <omar.luna@linaro.org>
Cc: Guzman Lugo Fernando <fernando.lugo@ti.com>
Cc: Anna Suman <s-anna@ti.com>
Cc: Clark Rob <rob@ti.com>
Cc: Stephen Boyd <sboyd@codeaurora.org>
Cc: Saravana Kannan <skannan@codeaurora.org>
Cc: David Brown <davidb@codeaurora.org>
Cc: Kieran Bingham <kieranbingham@gmail.com>
Cc: Tony Lindgren <tony@atomide.com>
2012-03-06 19:14:12 +02:00
Ohad Ben-Cohen 41a6ee09ee remoteproc: safer boot/shutdown order
Boot the remote processor only after setting up the virtqueues,
and shut it down before deleting them.

Remote processors should obey virtio status changes, and
therefore not manipulate/trigger the virtqueues while the virtio
driver isn't ready, but it's just safer not to rely on that
(plus a vq access might already be inflight while a vdev
status is changed).

We also don't have yet status change notifications, but that's
a temporary limitation.

Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Brian Swetland <swetland@google.com>
Cc: Iliyan Malchev <malchev@google.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Grant Likely <grant.likely@secretlab.ca>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Mark Grosen <mgrosen@ti.com>
Cc: John Williams <john.williams@petalogix.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Loic PALLARDY <loic.pallardy@stericsson.com>
Cc: Ludovic BARRE <ludovic.barre@stericsson.com>
Cc: Omar Ramirez Luna <omar.luna@linaro.org>
Cc: Guzman Lugo Fernando <fernando.lugo@ti.com>
Cc: Anna Suman <s-anna@ti.com>
Cc: Clark Rob <rob@ti.com>
Cc: Stephen Boyd <sboyd@codeaurora.org>
Cc: Saravana Kannan <skannan@codeaurora.org>
Cc: David Brown <davidb@codeaurora.org>
Cc: Kieran Bingham <kieranbingham@gmail.com>
Cc: Tony Lindgren <tony@atomide.com>
2012-03-06 19:13:53 +02:00
Ohad Ben-Cohen 04a9016e82 remoteproc: remoteproc_rpmsg -> remoteproc_virtio
At this point remoteproc can only register a single VIRTIO_ID_RPMSG virtio
device.

This limitation is going away soon: remoteproc is getting support for
registering any number of virtio devices and of any type (as
published by the firmware of the remote processor).

Rename remoteproc_rpmsg.c to remoteproc_virtio.c in preparation of
this generalization work.

Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Brian Swetland <swetland@google.com>
Cc: Iliyan Malchev <malchev@google.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Grant Likely <grant.likely@secretlab.ca>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Mark Grosen <mgrosen@ti.com>
Cc: John Williams <john.williams@petalogix.com>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Loic PALLARDY <loic.pallardy@stericsson.com>
Cc: Ludovic BARRE <ludovic.barre@stericsson.com>
Cc: Omar Ramirez Luna <omar.luna@linaro.org>
Cc: Guzman Lugo Fernando <fernando.lugo@ti.com>
Cc: Anna Suman <s-anna@ti.com>
Cc: Clark Rob <rob@ti.com>
Cc: Stephen Boyd <sboyd@codeaurora.org>
Cc: Saravana Kannan <skannan@codeaurora.org>
Cc: David Brown <davidb@codeaurora.org>
Cc: Kieran Bingham <kieranbingham@gmail.com>
Cc: Tony Lindgren <tony@atomide.com>
2012-03-06 19:13:45 +02:00