criterion performance measurements
overview
want to understand this report?
Data.Vector/1000x1000/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.15767056159381496 | 0.16180864945833287 | 0.1690642239299096 |
| Standard deviation | 2.7531007734191383e-3 | 7.702419624026191e-3 | 1.129528113903033e-2 |
Outlying measurements have moderate (0.1227534878937303%) effect on estimated standard deviation.
Data.Vector/1000x1000/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.16156839362136105 | 0.1638656798555396 | 0.1663139982472713 |
| Standard deviation | 2.5130458266518253e-3 | 3.467866337772096e-3 | 4.547316762080607e-3 |
Outlying measurements have moderate (0.12244897959183675%) effect on estimated standard deviation.
Data.Vector/1000x1000/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.10030205030753467 | 0.11087983277979356 | 0.12315495203781195 |
| Standard deviation | 1.1928286604341842e-2 | 1.6831271539471705e-2 | 2.1675590972482322e-2 |
Outlying measurements have moderate (0.4788094943884869%) effect on estimated standard deviation.
Data.Vector/1000x1000/FoldZipWith6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.706445364116167e-2 | 8.108502028095689e-2 | 9.005874317441472e-2 |
| Standard deviation | 2.430563300658737e-3 | 8.271232079334057e-3 | 1.3117823782121125e-2 |
Outlying measurements have moderate (0.3147947794719566%) effect on estimated standard deviation.
Data.Vector/1000x1000/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.186346773830088 | 0.1940727830219573 | 0.19914335913360254 |
| Standard deviation | 5.1964088217552815e-3 | 7.85281986419e-3 | 1.07476784410042e-2 |
Outlying measurements have moderate (0.13888888888888873%) effect on estimated standard deviation.
Data.Vector/1000x1000/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.19694115276916452 | 0.2060023851483833 | 0.2185146189067512 |
| Standard deviation | 1.012942102759376e-2 | 1.291881749793593e-2 | 1.496623137238347e-2 |
Outlying measurements have moderate (0.14827312983634433%) effect on estimated standard deviation.
Data.Vector/1000x1000/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.18287176576559214 | 0.19263854065544733 | 0.19675780710443025 |
| Standard deviation | 2.2863166352852585e-3 | 7.802271455603468e-3 | 1.1281075772410781e-2 |
Outlying measurements have moderate (0.13888888888888876%) effect on estimated standard deviation.
Data.Vector/1000x1000/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.13746461865667242 | 0.1412199109306905 | 0.14649046804396465 |
| Standard deviation | 6.880191263538742e-4 | 5.568598908239879e-3 | 6.265726632967736e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
Data.Vector/1000x1000/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1463870210311913 | 0.15139345107197708 | 0.15656535617519257 |
| Standard deviation | 4.630460994799094e-3 | 6.615946450336113e-3 | 9.550623240539008e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
Data.Vector/1000x1000/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.14530771008710416 | 0.15553222214173493 | 0.18317909281278758 |
| Standard deviation | 2.1786175178418535e-3 | 2.0483688141420153e-2 | 2.6879828929610664e-2 |
Outlying measurements have moderate (0.3166512337371697%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0269943701094013e-3 | 2.1356789694762885e-3 | 2.2664842297599524e-3 |
| Standard deviation | 2.652777347418475e-4 | 3.4746405018974946e-4 | 4.846354320136945e-4 |
Outlying measurements have severe (0.8410274526617928%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.027383018645429e-3 | 2.113000856260666e-3 | 2.1911177291638936e-3 |
| Standard deviation | 2.1391088428710458e-4 | 2.5820290497443075e-4 | 3.200444294224807e-4 |
Outlying measurements have severe (0.7617698103720179%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.293573999460397e-2 | 3.4784027447199405e-2 | 3.6564438471769614e-2 |
| Standard deviation | 2.7875634294345164e-3 | 3.7267240085457963e-3 | 5.541597453942374e-3 |
Outlying measurements have moderate (0.3854804212233718%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/FoldZipWith6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4452483726004755e-2 | 3.593350505579914e-2 | 3.7832679275348044e-2 |
| Standard deviation | 2.1886343766765346e-3 | 3.341173919245547e-3 | 4.698528449929579e-3 |
Outlying measurements have moderate (0.3199462102837654%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.376299949806481e-4 | 8.638206366448692e-4 | 8.885771452641523e-4 |
| Standard deviation | 6.443019683109898e-5 | 7.836592996733357e-5 | 1.0556415957674264e-4 |
Outlying measurements have severe (0.6867104390650919%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.766784242870923e-4 | 9.188847022800436e-4 | 9.730960388254242e-4 |
| Standard deviation | 1.1345090854640432e-4 | 1.483741792075195e-4 | 2.129718748646095e-4 |
Outlying measurements have severe (0.8772700339406796%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.08440289701073e-4 | 8.324235636937159e-4 | 8.5681799547617e-4 |
| Standard deviation | 6.884520763788765e-5 | 7.907530682641436e-5 | 9.472451815057847e-5 |
Outlying measurements have severe (0.7057266246170508%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.635440401847958e-4 | 8.868025379641735e-4 | 9.115015202084175e-4 |
| Standard deviation | 6.14549128271965e-5 | 7.381074927477033e-5 | 8.782167179674697e-5 |
Outlying measurements have severe (0.6432287226767744%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1016461348605823e-3 | 1.1297900151252705e-3 | 1.1631064966154613e-3 |
| Standard deviation | 8.04265902216326e-5 | 9.750217324993922e-5 | 1.1371553681559323e-4 |
Outlying measurements have severe (0.6474136149418432%) effect on estimated standard deviation.
Data.Vector.Storable/1000x1000/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0606417744749095e-3 | 1.1019839296836993e-3 | 1.1302302777675324e-3 |
| Standard deviation | 8.257883358050679e-5 | 9.90326620359163e-5 | 1.1847367811669611e-4 |
Outlying measurements have severe (0.6632841901159898%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4789124616738886e-3 | 1.5321660680938693e-3 | 1.5773980944815465e-3 |
| Standard deviation | 1.2656098392259448e-4 | 1.5873478284207272e-4 | 1.980680315598799e-4 |
Outlying measurements have severe (0.715707613207583%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.495819268159258e-3 | 1.5570785117861468e-3 | 1.6221044840779727e-3 |
| Standard deviation | 1.3799194406742353e-4 | 1.9226685418486617e-4 | 3.024047989168437e-4 |
Outlying measurements have severe (0.7855878780895957%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1982325462112057e-2 | 2.3096158456238965e-2 | 2.3771130849376824e-2 |
| Standard deviation | 1.3429120800850334e-3 | 1.9446727001388253e-3 | 3.0286519421217355e-3 |
Outlying measurements have moderate (0.3526760851104454%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/FoldZipWith6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8311208015135186e-2 | 2.9552930481459877e-2 | 3.1150090500586917e-2 |
| Standard deviation | 2.0957129744687032e-3 | 2.67165734375497e-3 | 3.7734256945588865e-3 |
Outlying measurements have moderate (0.3389319113557659%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.069602470568974e-4 | 7.277889449560005e-4 | 7.477297420245351e-4 |
| Standard deviation | 5.427276894598896e-5 | 6.42930481186221e-5 | 7.623248863229298e-5 |
Outlying measurements have severe (0.6852434017315993%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.106591147765525e-4 | 7.31171727699737e-4 | 7.517877164644216e-4 |
| Standard deviation | 5.46573498513108e-5 | 6.548405578428707e-5 | 7.624430802723945e-5 |
Outlying measurements have severe (0.6856056856164325%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.984093713372514e-4 | 7.209829430669261e-4 | 7.413723270838058e-4 |
| Standard deviation | 5.637984530108937e-5 | 6.68689475581048e-5 | 7.829245161798508e-5 |
Outlying measurements have severe (0.6997029603738842%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.514476046338513e-4 | 7.775143316082495e-4 | 7.992760017289154e-4 |
| Standard deviation | 5.833511879989423e-5 | 6.98494476510283e-5 | 8.237405984572456e-5 |
Outlying measurements have severe (0.6948420351094279%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0866228136280605e-3 | 1.123861366418248e-3 | 1.1499730901832692e-3 |
| Standard deviation | 7.952175762067983e-5 | 9.789778371496513e-5 | 1.1951344343195251e-4 |
Outlying measurements have severe (0.6476742035328631%) effect on estimated standard deviation.
Data.Vector.Unboxed/1000x1000/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.807007721322053e-4 | 1.0090541594482605e-3 | 1.032929061448373e-3 |
| Standard deviation | 7.316753299403236e-5 | 8.983760553558225e-5 | 1.094507541833262e-4 |
Outlying measurements have severe (0.6726988427416527%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.