From 1392bd370c9274cdfdd59474d83ad822b868f5c2 Mon Sep 17 00:00:00 2001 From: The-Themis-Benchmarks <70174675+the-themis-benchmarks@users.noreply.github.com> Date: Wed, 2 Jun 2021 20:13:36 -0700 Subject: [PATCH 1/3] Update README.md --- README.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 8ef20d7..9541fc1 100644 --- a/README.md +++ b/README.md @@ -732,10 +732,25 @@ You can follow the similar instructions in **Step 2** to check the bug finding r ### Notes -(1) you can substitute `--monkey` with `--ape`, `--combo`, `--humandroid`, `--qtesting` or `--timemachine` to try the corresponding tool. You also need to change the output directory `-o ../monkey-results/` to a distinct directory, e.g., `-o ../ape-results`. You can follow the similar steps described above to inspect whether the target bug was found or not and the related info. +(1) You can substitute `--monkey` with `--ape`, `--combo`, `--qtesting` or `--timemachine` to directly run the corresponding tool. You may need to change the output directory `-o ../monkey-results/` to a distinct directory, e.g., `-o ../ape-results`. You can follow the similar steps described above to inspect whether the target bug was found or not and the related info. -(2) If the app under test requires user login, you should specify the login script. Themis will call the login script before testing. For example, if we run `Monkey` on `../commons/commons-2.11.0-#3244.apk` which requires user login, the command line should be: +(2) Specifically, for `humanoid`, before running, you need to setup the specific running environment of Humanoid. Open a termina, and run: +``` +cd /home/themis/the-themis-benchmark/tools/Humanoid-tool +source venv/bin/activate # Humanoid depends on tensorflow 1.12, which requires specific Python version +cd Humanoid +python3 agent.py -c config.json # start the server of Humanoid +``` + +Open a new terminal, run `Humanoid` (which internally runs `droidbot`) on the emulator `Android7.1_Humanoid` (with specific screen size): +``` +cd /home/themis/the-themis-benchmark/scripts/ +python3 themis.py --no-headless --avd Android7.1_Humanoid --apk ../ActivityDiary/ActivityDiary-1.1.8-debug-#118.apk --time 10m -o ../humanoid-results --humandroid +``` + + +(3) If the app under test requires user login, you should specify the login script. Themis will call the login script before testing. For example, if we run `Monkey` on `../commons/commons-2.11.0-#3244.apk` which requires user login, the command line should be: ``` python3 themis.py --avd Android7.1 --apk ../commons/commons-2.11.0-#3244.apk -n 1 --repeat 5 --time 6h -o ../monkey-results/ --login ../commons/login-2.11.0-#3244.py --monkey @@ -792,7 +807,7 @@ pip3 install --upgrade --pre uiautomator2 6. If you run Themis on remote servers, please omit the option `--no-headless` which turns off the emulator GUI. -7. Install all the necessary dependecies required by the respective testing tools. +7. Install all the necessary dependecies required by the respective testing tools (see the repository of each tools). ## Extend Themis for Future Research From 361fbcede042d952f0de5adb7c2eb1f674d56552 Mon Sep 17 00:00:00 2001 From: The-Themis-Benchmarks <70174675+the-themis-benchmarks@users.noreply.github.com> Date: Wed, 2 Jun 2021 23:01:44 -0700 Subject: [PATCH 2/3] Update README.md --- README.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9541fc1..9fa6cc9 100644 --- a/README.md +++ b/README.md @@ -732,10 +732,10 @@ You can follow the similar instructions in **Step 2** to check the bug finding r ### Notes -(1) You can substitute `--monkey` with `--ape`, `--combo`, `--qtesting` or `--timemachine` to directly run the corresponding tool. You may need to change the output directory `-o ../monkey-results/` to a distinct directory, e.g., `-o ../ape-results`. You can follow the similar steps described above to inspect whether the target bug was found or not and the related info. +(1) You can substitute `--monkey` with `--ape` or `--combo` to directly run the corresponding tool. You may need to change the output directory `-o ../monkey-results/` to a distinct directory, e.g., `-o ../ape-results`. You can follow the similar steps described above to inspect whether the target bug was found or not and the related info. -(2) Specifically, for `humanoid`, before running, you need to setup the specific running environment of Humanoid. Open a termina, and run: +(2) Specifically, for `humanoid`, before running, you need to setup the specific running environment of Humanoid. Open a terminal, and run: ``` cd /home/themis/the-themis-benchmark/tools/Humanoid-tool source venv/bin/activate # Humanoid depends on tensorflow 1.12, which requires specific Python version @@ -749,8 +749,22 @@ cd /home/themis/the-themis-benchmark/scripts/ python3 themis.py --no-headless --avd Android7.1_Humanoid --apk ../ActivityDiary/ActivityDiary-1.1.8-debug-#118.apk --time 10m -o ../humanoid-results --humandroid ``` +(3) Specifically, for `Q-testing`, before running, you need to setup the specific running environment of Humanoid. Open a terminal, and run: +``` +cd /home/themis/the-themis-benchmark/tools/Q-testing +source venv/bin/activate # Q-testing depends on Python 2.7 +``` +And then, run the command line: +``` +cd /home/themis/the-themis-benchmark/scripts/ +python3 themis.py --no-headless --avd Android7.1 --apk ../ActivityDiary/ActivityDiary-1.1.8-debug-#118.apk --time 10m -o ../qtesting-results/ --qtest +ing +``` -(3) If the app under test requires user login, you should specify the login script. Themis will call the login script before testing. For example, if we run `Monkey` on `../commons/commons-2.11.0-#3244.apk` which requires user login, the command line should be: +(4) Specifically, for `TimeMachine`, it depends on VirtualBox and docker by iteself. Thus, we cannot build `TimeMachine` within this VM because. `TimeMachine` can be built on native machines. + + +(5) If the app under test requires user login, you should specify the login script. Themis will call the login script before testing. For example, if we run `Monkey` on `../commons/commons-2.11.0-#3244.apk` which requires user login, the command line should be: ``` python3 themis.py --avd Android7.1 --apk ../commons/commons-2.11.0-#3244.apk -n 1 --repeat 5 --time 6h -o ../monkey-results/ --login ../commons/login-2.11.0-#3244.py --monkey From d6c979b6b60f817b4cdf8b971947b79624f8fe76 Mon Sep 17 00:00:00 2001 From: The-Themis-Benchmarks <70174675+the-themis-benchmarks@users.noreply.github.com> Date: Thu, 3 Jun 2021 00:20:40 -0700 Subject: [PATCH 3/3] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9fa6cc9..15a3ef7 100644 --- a/README.md +++ b/README.md @@ -22,9 +22,9 @@ For each bug, we provide: - The stack trace of the bug -- Metadata for supporting evaluation (e.g., bug explanation, login script, configuration files used by the Themis's infrastructure) +- Metadata for supporting evaluation (e.g., app login scripts and configuration files used by Themis for code coverage computation) -- the app source code w.r.t each bug +- The app source code w.r.t each bug ### List of crash bugs