2017-05-10 00:13:54 +08:00
# React Benchmarking
2015-10-29 04:09:43 +08:00
2017-05-10 00:13:54 +08:00
## Commands
benchmarking: measure and analyze scripts
This uses wall-clock time (for now) so it's noisier than alternatives
(cachegrind, CPU perf-counters), but it's still valuable. In a future diff we
can make it use those.
`measure.py` outputs something that `analyze.py` can understand, but you can use `analyze.py` without `measure.py` too. The file format is simple:
```
$ cat measurements.txt
factory_ms_jsc_jit 13.580322265625
factory_ms_jsc_jit 13.659912109375
factory_ms_jsc_jit 13.67919921875
factory_ms_jsc_nojit 12.827880859375
factory_ms_jsc_nojit 13.105224609375
factory_ms_jsc_nojit 13.195068359375
factory_ms_node 40.4891400039196
factory_ms_node 40.6669420003891
factory_ms_node 43.52413299679756
ssr_pe_cold_ms_jsc_jit 43.06005859375
...
```
(The lines do not need to be sorted.)
Comparing 0.14.0 vs master:
```
$ ./measure.py react-0.14.0.min.js >014.txt
Measuring SSR for PE benchmark (30 trials)
..............................
Measuring SSR for PE with warm JIT (3 slow trials)
...
$ ./measure.py react.min.js >master.txt
Measuring SSR for PE benchmark (30 trials)
..............................
Measuring SSR for PE with warm JIT (3 slow trials)
...
$ ./analyze.py 014.txt master.txt
Comparing 014.txt (control) vs master.txt (test)
Significant differences marked by ***
% change from control to test, with 99% CIs:
* factory_ms_jsc_jit
% change: -0.56% [ -2.51%, +1.39%]
means: 14.037 (control), 13.9593 (test)
* factory_ms_jsc_nojit
% change: +1.23% [ -1.18%, +3.64%]
means: 13.2586 (control), 13.4223 (test)
* factory_ms_node
% change: +3.53% [ +0.29%, +6.77%] ***
means: 42.0529 (control), 43.54 (test)
* ssr_pe_cold_ms_jsc_jit
% change: -6.84% [ -9.04%, -4.65%] ***
means: 44.2444 (control), 41.2187 (test)
* ssr_pe_cold_ms_jsc_nojit
% change: -11.81% [-14.66%, -8.96%] ***
means: 52.9449 (control), 46.6953 (test)
* ssr_pe_cold_ms_node
% change: -2.70% [ -4.52%, -0.88%] ***
means: 96.8909 (control), 94.2741 (test)
* ssr_pe_warm_ms_jsc_jit
% change: -17.60% [-22.04%, -13.16%] ***
means: 13.763 (control), 11.3439 (test)
* ssr_pe_warm_ms_jsc_nojit
% change: -20.65% [-22.62%, -18.68%] ***
means: 30.8829 (control), 24.5074 (test)
* ssr_pe_warm_ms_node
% change: -8.76% [-13.48%, -4.03%] ***
means: 30.0193 (control), 27.3964 (test)
$
```
2015-11-19 08:16:34 +08:00
2017-05-10 00:13:54 +08:00
In most cases, the only two commands you might want to use are:
2016-09-13 02:58:00 +08:00
2017-12-02 08:00:29 +08:00
- `yarn start`
2018-11-13 00:25:32 +08:00
- `yarn --cwd=../../ build react/index,react-dom/index --type=UMD_PROD && yarn start --skip-build`
2016-09-13 02:58:00 +08:00
2017-12-02 08:00:29 +08:00
The first command will run benchmarks with all the default settings. A local and remote build will occur on React and ReactDOM UMD bundles, both local and remote repos will be run against all benchmarks.
benchmarking: measure and analyze scripts
This uses wall-clock time (for now) so it's noisier than alternatives
(cachegrind, CPU perf-counters), but it's still valuable. In a future diff we
can make it use those.
`measure.py` outputs something that `analyze.py` can understand, but you can use `analyze.py` without `measure.py` too. The file format is simple:
```
$ cat measurements.txt
factory_ms_jsc_jit 13.580322265625
factory_ms_jsc_jit 13.659912109375
factory_ms_jsc_jit 13.67919921875
factory_ms_jsc_nojit 12.827880859375
factory_ms_jsc_nojit 13.105224609375
factory_ms_jsc_nojit 13.195068359375
factory_ms_node 40.4891400039196
factory_ms_node 40.6669420003891
factory_ms_node 43.52413299679756
ssr_pe_cold_ms_jsc_jit 43.06005859375
...
```
(The lines do not need to be sorted.)
Comparing 0.14.0 vs master:
```
$ ./measure.py react-0.14.0.min.js >014.txt
Measuring SSR for PE benchmark (30 trials)
..............................
Measuring SSR for PE with warm JIT (3 slow trials)
...
$ ./measure.py react.min.js >master.txt
Measuring SSR for PE benchmark (30 trials)
..............................
Measuring SSR for PE with warm JIT (3 slow trials)
...
$ ./analyze.py 014.txt master.txt
Comparing 014.txt (control) vs master.txt (test)
Significant differences marked by ***
% change from control to test, with 99% CIs:
* factory_ms_jsc_jit
% change: -0.56% [ -2.51%, +1.39%]
means: 14.037 (control), 13.9593 (test)
* factory_ms_jsc_nojit
% change: +1.23% [ -1.18%, +3.64%]
means: 13.2586 (control), 13.4223 (test)
* factory_ms_node
% change: +3.53% [ +0.29%, +6.77%] ***
means: 42.0529 (control), 43.54 (test)
* ssr_pe_cold_ms_jsc_jit
% change: -6.84% [ -9.04%, -4.65%] ***
means: 44.2444 (control), 41.2187 (test)
* ssr_pe_cold_ms_jsc_nojit
% change: -11.81% [-14.66%, -8.96%] ***
means: 52.9449 (control), 46.6953 (test)
* ssr_pe_cold_ms_node
% change: -2.70% [ -4.52%, -0.88%] ***
means: 96.8909 (control), 94.2741 (test)
* ssr_pe_warm_ms_jsc_jit
% change: -17.60% [-22.04%, -13.16%] ***
means: 13.763 (control), 11.3439 (test)
* ssr_pe_warm_ms_jsc_nojit
% change: -20.65% [-22.62%, -18.68%] ***
means: 30.8829 (control), 24.5074 (test)
* ssr_pe_warm_ms_node
% change: -8.76% [-13.48%, -4.03%] ***
means: 30.0193 (control), 27.3964 (test)
$
```
2015-11-19 08:16:34 +08:00
2017-05-19 07:13:59 +08:00
The second command will run all benchmarks but skip the build process. This is useful for when doing local performance tweaking and the remote repo has already had its bundles built. Both local and remote repos will be run against all benchmarks with this command too.
2016-09-13 02:58:00 +08:00
2017-05-10 00:13:54 +08:00
The other commands are as follows:
2016-09-13 02:58:00 +08:00
2017-05-10 00:13:54 +08:00
```bash
# will compare local repo vs remote merge base repo
2017-12-02 08:00:29 +08:00
yarn start
2016-09-13 02:58:00 +08:00
2017-05-10 00:13:54 +08:00
# will compare local repo vs remote merge base repo
# this can significantly improve bench times due to no build
2017-12-02 08:00:29 +08:00
yarn start --skip-build
2015-10-29 04:09:43 +08:00
2017-05-10 00:13:54 +08:00
# will only build and run local repo against benchmarks (no remote values will be shown)
2017-12-02 08:00:29 +08:00
yarn start --local
2015-10-29 04:09:43 +08:00
2017-05-10 00:13:54 +08:00
# will only build and run remote merge base repo against benchmarks (no local values will be shown)
2017-12-02 08:00:29 +08:00
yarn start --remote
2015-10-29 04:09:43 +08:00
2021-06-30 02:26:24 +08:00
# will only build and run remote main repo against benchmarks
yarn start --remote=main
2015-10-29 04:09:43 +08:00
2017-12-02 08:00:29 +08:00
# same as "yarn start"
yarn start --remote --local
2015-10-29 04:09:43 +08:00
2017-05-10 00:13:54 +08:00
# runs benchmarks with Chrome in headless mode
2017-12-02 08:00:29 +08:00
yarn start --headless
2015-10-29 04:09:43 +08:00
2017-05-10 00:13:54 +08:00
# runs only specific string matching benchmarks
2017-12-02 08:00:29 +08:00
yarn start --benchmark=hacker
2018-05-18 16:18:35 +08:00
```
