criterion performance measurements
overview
want to understand this report?
Data.Vector/1x1000000/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.018028066332339e-4 | 9.446991994752079e-4 | 9.793003316981425e-4 |
| Standard deviation | 8.305889314684694e-5 | 1.0289136727246484e-4 | 1.2939095165951295e-4 |
Outlying measurements have severe (0.7288967931090492%) effect on estimated standard deviation.
Data.Vector/1x1000000/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.898049393908494e-4 | 9.272363713501813e-4 | 9.663665354192545e-4 |
| Standard deviation | 8.506109980321425e-5 | 1.0108274677947279e-4 | 1.264900336152883e-4 |
Outlying measurements have severe (0.7333691484855385%) effect on estimated standard deviation.
Data.Vector/1x1000000/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.844120392127565e-4 | 9.34951020188891e-4 | 9.753933730289599e-4 |
| Standard deviation | 9.642365014899166e-5 | 1.1482610625453555e-4 | 1.3647420615503535e-4 |
Outlying measurements have severe (0.7689882728176689%) effect on estimated standard deviation.
Data.Vector/1x1000000/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.836082928010482e-3 | 1.9149507453298778e-3 | 1.9872520816562518e-3 |
| Standard deviation | 1.7610634221517692e-4 | 2.1220698309284473e-4 | 2.529133623749362e-4 |
Outlying measurements have severe (0.6950333829270513%) effect on estimated standard deviation.
Data.Vector/1x1000000/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8741868984964228e-3 | 1.964506016322936e-3 | 2.073124198531972e-3 |
| Standard deviation | 2.2175232129118802e-4 | 2.854617855827477e-4 | 3.8245715262315213e-4 |
Outlying measurements have severe (0.7982185283789495%) effect on estimated standard deviation.
Data.Vector/1x1000000/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.913983139740194e-4 | 9.245934224066044e-4 | 9.687399522471502e-4 |
| Standard deviation | 8.708085804975599e-5 | 1.0569421489193658e-4 | 1.25963148940582e-4 |
Outlying measurements have severe (0.750922559949055%) effect on estimated standard deviation.
Data.Vector/1x1000000/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7902241095381025e-3 | 1.8685803120865175e-3 | 1.9365434406545317e-3 |
| Standard deviation | 1.597192560584448e-4 | 2.0489165789070444e-4 | 2.4332678021414845e-4 |
Outlying measurements have severe (0.6947492876893747%) effect on estimated standard deviation.
Data.Vector/1x1000000/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8656383509592918e-3 | 1.990939683736417e-3 | 2.213074871577918e-3 |
| Standard deviation | 2.6877057305565576e-4 | 4.8074568238586264e-4 | 7.885192010268329e-4 |
Outlying measurements have severe (0.9296920099827211%) effect on estimated standard deviation.
Data.Vector/1x1000000/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9035317918032367e-3 | 2.0382353680648436e-3 | 2.2964032102986295e-3 |
| Standard deviation | 3.156271846887129e-4 | 5.358520459193359e-4 | 8.023524231370154e-4 |
Outlying measurements have severe (0.9319546973154298%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.970675474620219e-9 | 7.480067311799094e-9 | 8.441207043539003e-9 |
| Standard deviation | 1.2306038128256166e-9 | 2.0870297342024165e-9 | 3.594969478408915e-9 |
Outlying measurements have severe (0.98958694250982%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.395980133767946e-9 | 8.842067337234421e-9 | 9.230906907661354e-9 |
| Standard deviation | 1.042412616231445e-9 | 1.2351713022911799e-9 | 1.5822827943404184e-9 |
Outlying measurements have severe (0.958096248897892%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.016909830210579e-9 | 7.551795687682028e-9 | 8.046743728585566e-9 |
| Standard deviation | 1.0911634000525115e-9 | 1.457043297622503e-9 | 2.006730146165969e-9 |
Outlying measurements have severe (0.9790737367962216%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.477499726690709e-4 | 4.8039877819079955e-4 | 5.393082822024026e-4 |
| Standard deviation | 4.9843989715592606e-5 | 1.2240953821441653e-4 | 2.2079957697891724e-4 |
Outlying measurements have severe (0.9602079259110194%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.542755659605859e-4 | 7.907778547800667e-4 | 8.357898677896374e-4 |
| Standard deviation | 9.704599761929306e-5 | 1.1940085423731775e-4 | 1.572039834367742e-4 |
Outlying measurements have severe (0.8658786794882969%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.354040260489685e-9 | 6.683789996803642e-9 | 6.903326252265887e-9 |
| Standard deviation | 6.626588724212627e-10 | 7.742247222413676e-10 | 9.378129979009113e-10 |
Outlying measurements have severe (0.9414302600869959%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.9024681186372394e-4 | 4.1199828330548395e-4 | 4.268703760094032e-4 |
| Standard deviation | 4.130203288701002e-5 | 5.1140308983504643e-5 | 6.091363799380248e-5 |
Outlying measurements have severe (0.8346829703115991%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.9561015587554415e-4 | 4.1517748805050385e-4 | 4.320927798839237e-4 |
| Standard deviation | 4.5340096422042506e-5 | 5.489048064136495e-5 | 6.749439139760176e-5 |
Outlying measurements have severe (0.8475407839135731%) effect on estimated standard deviation.
Data.Vector.Storable/1x1000000/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.7874747198647505e-4 | 3.957192428179931e-4 | 4.151849359754334e-4 |
| Standard deviation | 4.449204357966468e-5 | 5.4609515278605746e-5 | 6.917369506123971e-5 |
Outlying measurements have severe (0.8616708602465094%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.909388800756507e-9 | 7.1679986495031155e-9 | 7.418319465576648e-9 |
| Standard deviation | 6.58889886587137e-10 | 7.85411490319172e-10 | 9.628306277061957e-10 |
Outlying measurements have severe (0.9343800416565183%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.030285713642875e-9 | 7.324560088285401e-9 | 7.612528453121602e-9 |
| Standard deviation | 6.379649650813515e-10 | 7.852651935669301e-10 | 9.613514312201517e-10 |
Outlying measurements have severe (0.9307597501525352%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.524587200982693e-9 | 6.846336680237059e-9 | 7.243135337574177e-9 |
| Standard deviation | 7.171580119918775e-10 | 9.641463993108005e-10 | 1.3905649377817912e-9 |
Outlying measurements have severe (0.9586764065359523%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.9394548230521056e-4 | 4.124556087261565e-4 | 4.313093887459311e-4 |
| Standard deviation | 3.785031626131389e-5 | 4.669651536687429e-5 | 5.841852282739579e-5 |
Outlying measurements have severe (0.8094191029636043%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.50462824691118e-4 | 7.968851400625608e-4 | 8.581357423125306e-4 |
| Standard deviation | 1.0458782933711384e-4 | 1.454349079965471e-4 | 1.9273267966768976e-4 |
Outlying measurements have severe (0.9086837544638235%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.90658159728585e-9 | 9.269646278270276e-9 | 9.716048239328004e-9 |
| Standard deviation | 9.268365834643642e-10 | 1.1287867497355437e-9 | 1.4556507348161751e-9 |
Outlying measurements have severe (0.9440746917510131%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.076301842650724e-4 | 4.2822530350193676e-4 | 4.460145153416746e-4 |
| Standard deviation | 5.217431696148545e-5 | 6.39511469606485e-5 | 8.12027133081776e-5 |
Outlying measurements have severe (0.8848566568078535%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.971344823187184e-4 | 4.16910709360035e-4 | 4.3801562428243156e-4 |
| Standard deviation | 4.683609586213249e-5 | 5.485654008952764e-5 | 6.519488724621087e-5 |
Outlying measurements have severe (0.8474650098252664%) effect on estimated standard deviation.
Data.Vector.Unboxed/1x1000000/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.878855952906457e-4 | 4.115444777618405e-4 | 4.2762883780641966e-4 |
| Standard deviation | 4.732671572864916e-5 | 5.566011380413195e-5 | 6.915206248707682e-5 |
Outlying measurements have severe (0.8613674060387755%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.