criterion performance measurements
overview
want to understand this report?
Data.Vector/200x200/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3446414699760257e-3 | 2.409400115344888e-3 | 2.5918729660995453e-3 |
| Standard deviation | 1.1936326635680298e-4 | 3.0913978691054535e-4 | 6.26443683848233e-4 |
Outlying measurements have severe (0.7851512030898539%) effect on estimated standard deviation.
Data.Vector/200x200/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3647829970157475e-3 | 2.4183500388147757e-3 | 2.550258821176352e-3 |
| Standard deviation | 8.088541938374702e-5 | 2.5231561004773084e-4 | 5.034379540097153e-4 |
Outlying measurements have severe (0.6905406690656486%) effect on estimated standard deviation.
Data.Vector/200x200/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5054735877460357e-3 | 2.6034944165175275e-3 | 2.8231553018929164e-3 |
| Standard deviation | 1.4888145116049873e-4 | 5.375694407717335e-4 | 9.532459784325443e-4 |
Outlying measurements have severe (0.8999886016478541%) effect on estimated standard deviation.
Data.Vector/200x200/FoldZipWith6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.9271624122752535e-3 | 3.1586844504604313e-3 | 3.5977782564827576e-3 |
| Standard deviation | 7.353914835844297e-4 | 1.0156320359214074e-3 | 1.5890730958703018e-3 |
Outlying measurements have severe (0.9554416436897468%) effect on estimated standard deviation.
Data.Vector/200x200/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.531742367834068e-3 | 2.7012427495641334e-3 | 2.9300412096351955e-3 |
| Standard deviation | 5.176151036382253e-4 | 7.019093233595463e-4 | 9.233696963082528e-4 |
Outlying measurements have severe (0.9395102726234958%) effect on estimated standard deviation.
Data.Vector/200x200/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0065909694119157e-3 | 2.065639270830839e-3 | 2.1797127938004137e-3 |
| Standard deviation | 1.507515510192944e-4 | 2.645877012635046e-4 | 4.342372807010278e-4 |
Outlying measurements have severe (0.7752156962174853%) effect on estimated standard deviation.
Data.Vector/200x200/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0070811402252563e-3 | 2.0592237010251166e-3 | 2.146867214183811e-3 |
| Standard deviation | 1.2494005887868834e-4 | 2.2513208420183172e-4 | 3.8073283787948286e-4 |
Outlying measurements have severe (0.7243904722483185%) effect on estimated standard deviation.
Data.Vector/200x200/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9515313106142843e-3 | 2.00266133820991e-3 | 2.153507089362899e-3 |
| Standard deviation | 4.611822301208313e-5 | 3.139541775557754e-4 | 5.996548693221562e-4 |
Outlying measurements have severe (0.8514064110318633%) effect on estimated standard deviation.
Data.Vector/200x200/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3693859694557014e-3 | 2.5331498018659458e-3 | 2.8846843086980205e-3 |
| Standard deviation | 4.0415263664424683e-4 | 7.665020739529675e-4 | 1.2925290652590446e-3 |
Outlying measurements have severe (0.9592826987679752%) effect on estimated standard deviation.
Data.Vector/200x200/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1322953824150907e-3 | 2.1862752675708133e-3 | 2.301583853854964e-3 |
| Standard deviation | 1.3980505554055122e-4 | 2.6750122570674026e-4 | 4.923562679550663e-4 |
Outlying measurements have severe (0.7568683246901234%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.860722208615437e-5 | 8.16432374760698e-5 | 9.116942990726164e-5 |
| Standard deviation | 7.000096744618853e-6 | 1.7966550386690746e-5 | 3.584910833074439e-5 |
Outlying measurements have severe (0.9644490087127735%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.52880185692604e-5 | 8.388238360726249e-5 | 1.1654353602607005e-4 |
| Standard deviation | 4.159770703952201e-6 | 5.300214841769865e-5 | 1.1227635450623614e-4 |
Outlying measurements have severe (0.9914877303827881%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0679148350684615e-3 | 1.084481288038463e-3 | 1.1191641804821541e-3 |
| Standard deviation | 5.0229509680027814e-5 | 7.445364455327325e-5 | 1.1872439125595364e-4 |
Outlying measurements have severe (0.5498686003705977%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/FoldZipWith6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0285793175433009e-3 | 1.0446891780839366e-3 | 1.0793558001373708e-3 |
| Standard deviation | 4.53246946105559e-5 | 7.942591550791666e-5 | 1.447256648213417e-4 |
Outlying measurements have severe (0.5997139361576823%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.722419072615307e-5 | 3.727866850331107e-5 | 3.743874091192027e-5 |
| Standard deviation | 1.278967441986344e-7 | 2.9372807617797007e-7 | 5.690117566158855e-7 |
Outlying measurements have no (7.2459907293940945e-3%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.702242992870149e-5 | 3.715561142721487e-5 | 3.738667108559493e-5 |
| Standard deviation | 3.099836108091367e-7 | 5.650243014333817e-7 | 9.237269780018003e-7 |
Outlying measurements have moderate (0.10357676538090843%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.739264485582071e-5 | 3.889470648911344e-5 | 4.4749718893577854e-5 |
| Standard deviation | 1.3083892010190847e-7 | 9.4486629660681e-6 | 1.819238346268373e-5 |
Outlying measurements have severe (0.9687542685998195%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.217592458144459e-5 | 4.2272067925602125e-5 | 4.254009026929657e-5 |
| Standard deviation | 1.6001058644234424e-7 | 4.7365433710440027e-7 | 8.713577764170262e-7 |
Outlying measurements have slight (6.297455942697043e-2%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.24170388954505e-5 | 5.247006069008071e-5 | 5.2531378385271814e-5 |
| Standard deviation | 1.4222970297783662e-7 | 2.0056274186298221e-7 | 3.1663574916070687e-7 |
Outlying measurements have no (7.6331360946743e-3%) effect on estimated standard deviation.
Data.Vector.Storable/200x200/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.321783063080756e-5 | 5.3295784757122983e-5 | 5.337113656656198e-5 |
| Standard deviation | 1.9046334050530748e-7 | 2.738871541315033e-7 | 3.9406776972511313e-7 |
Outlying measurements have no (7.69184544198087e-3%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/FoldZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.989000502909045e-5 | 7.032744704470489e-5 | 7.156329003996987e-5 |
| Standard deviation | 9.142062581123713e-7 | 2.1417278971342637e-6 | 4.168599262637132e-6 |
Outlying measurements have moderate (0.29369183011833316%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/RecurseZip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.163131886767375e-5 | 7.204818057480207e-5 | 7.260729705727421e-5 |
| Standard deviation | 1.2504496248253198e-6 | 1.659129207813356e-6 | 2.3560524996359743e-6 |
Outlying measurements have moderate (0.19335579004914297%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/RecurseZipWithN
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0039350655837848e-3 | 1.0281918732207918e-3 | 1.0797522623978628e-3 |
| Standard deviation | 7.009302639195955e-5 | 1.1544840611985654e-4 | 1.97353430534569e-4 |
Outlying measurements have severe (0.774784195767109%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/FoldZipWith6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.003432837364261e-3 | 1.0187330559805914e-3 | 1.0416818006187786e-3 |
| Standard deviation | 4.581427542703358e-5 | 6.201400915407868e-5 | 8.995855532558973e-5 |
Outlying measurements have moderate (0.4983536486723545%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/UncheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.290213916698037e-5 | 3.412685123232057e-5 | 3.897544504390881e-5 |
| Standard deviation | 1.2997914756037443e-7 | 7.895326619113415e-6 | 1.678723955440216e-5 |
Outlying measurements have severe (0.9692419717557254%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/UncheckedStFromBackModify
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.288710963950813e-5 | 3.3020755816588765e-5 | 3.342658877650348e-5 |
| Standard deviation | 2.727950782246585e-7 | 6.971433004539136e-7 | 1.3450488786129697e-6 |
Outlying measurements have moderate (0.18472907269998962%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/UncheckedStFromBackBailEmpty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.262824688899175e-5 | 3.373537854157007e-5 | 3.7036922468804186e-5 |
| Standard deviation | 1.9317810631444669e-7 | 7.0390859042259805e-6 | 1.3568951432444536e-5 |
Outlying measurements have severe (0.9616292597817715%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/UncheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.752967058805091e-5 | 3.756838684083611e-5 | 3.760741428470781e-5 |
| Standard deviation | 1.0310269980882292e-7 | 1.3050058926362972e-7 | 1.7430209972730207e-7 |
Outlying measurements have no (7.245990729394214e-3%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/CheckedStFromBack
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.158598085586455e-5 | 5.184085233671138e-5 | 5.271399406274116e-5 |
| Standard deviation | 2.1099824632515304e-7 | 1.4260061384463406e-6 | 2.991645183162446e-6 |
Outlying measurements have moderate (0.26474169815701715%) effect on estimated standard deviation.
Data.Vector.Unboxed/200x200/CheckedStFromFront
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.900973861759643e-5 | 4.910431392354418e-5 | 4.952333362274094e-5 |
| Standard deviation | 1.7138176397388278e-7 | 4.923512394929238e-7 | 1.1295915566463018e-6 |
Outlying measurements have no (7.575316123769143e-3%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.