The current sub-module unload exit path is a mess, it tries to abuse the idr. Just keep a list of devices per driver struct and free them in-order on rmmod. Signed-off-by: Dave Airlie <airlied@redhat.com>