ZTS: Provide for nested cleanup routines

Shared test library functions lack a simple way to ensure proper
cleanup in the event of a failure.  The `log_onexit` cleanup pattern
cannot be used in library functions because it uses one global
variable to store the cleanup command.

An example of where this is a serious issue is when a tunable that
artifically stalls kernel progress gets activated and then some check
fails.  Unless the caller knows about the tunable and sets it back,
the system will be left in a bad state.

To solve this problem, turn the global cleanup variable into a stack.
Provide push and pop functions to add additional cleanup steps and
remove them after it is safe again.

The first use of this new functionality is in attempt_during_removal,
which sets REMOVAL_SUSPEND_PROGRESS.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10080
This commit is contained in:
Ryan Moeller 2020-03-03 13:28:09 -05:00 committed by GitHub
parent 9bb907bc3f
commit 0a0f9a7dc6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 25 additions and 5 deletions

View File

@ -281,7 +281,23 @@ function log_pos
function log_onexit
{
_CLEANUP="$@"
_CLEANUP=("$*")
}
# Push an exit handler on the cleanup stack
#
# $@ - function(s) to perform on exit
function log_onexit_push
{
_CLEANUP+=("$*")
}
# Pop an exit handler off the cleanup stack
function log_onexit_pop
{
_CLEANUP=("${_CLEANUP[@]:0:${#_CLEANUP[@]}-1}")
}
#
@ -425,12 +441,14 @@ function _endlog
_execute_testfail_callbacks
fi
if [[ -n $_CLEANUP ]] ; then
typeset cleanup=$_CLEANUP
log_onexit ""
typeset stack=("${_CLEANUP[@]}")
log_onexit ""
typeset i=${#stack[@]}
while (( i-- )); do
typeset cleanup="${stack[i]}"
log_note "Performing local cleanup via log_onexit ($cleanup)"
$cleanup
fi
done
exit $exitcode
}

View File

@ -60,6 +60,7 @@ function attempt_during_removal # pool disk callback [args]
typeset callback=$3
shift 3
log_onexit_push set_tunable32 REMOVAL_SUSPEND_PROGRESS 0
set_tunable32 REMOVAL_SUSPEND_PROGRESS 1
log_must zpool remove $pool $disk
@ -80,6 +81,7 @@ function attempt_during_removal # pool disk callback [args]
log_must is_pool_removing $pool
set_tunable32 REMOVAL_SUSPEND_PROGRESS 0
log_onexit_pop
log_must wait_for_removal $pool
log_mustnot vdevs_in_pool $pool $disk