We should produce a benchmark that clearly shows the performance improvements
from 2.3 to 2.4.

This involves recruiting volunteers - preferably on all three major operating
systems. They should produce numbers for invocations of darcs 2.3.1 and darcs 2.4.

I've posted a call for volunteers on

  http://lists.osuosl.org/pipermail/darcs-users/2009-December/022599.html

If we don't get any hands raised by the end of the week, we'll have to do it
ourselves.

Now that this is done, we need to determine what is the right way to get the
benchmarks.  Presumably, it's just a matter of refining the darcs-benchmark
program (cabal install darcs-benchmark, folks)

http://wiki.darcs.net/Development/Benchmarks

Petr: do you believe that in its current released state, darcs-benchmark is
adequate for producing benchmarks that are useful for a release announcement? 
If not, do you have a clear picture on how we'd need to extend it?

Thanks!

I've been hacking away at darcs-benchmark recently and soliciting help
from volunteers.

I think we're almost there.  Infrastructure keeps creeping forward a
little bit.  Still need to account for optimize --pristine.

When they're done, the benchmarks will appear at
http://wiki.darcs.net/Benchmarks

We now have benchmarks for MacOS X, Linux and Windows.
They're still a bit hard to read, but I think we've reached a state
where we can call this done:
http://wiki.darcs.net/Benchmarks

Further refinements ahead, particularly, issue1631!

Op donderdag 25 februari 2010 12:57 schreef je:
> We now have benchmarks for MacOS X, Linux and Windows.
> They're still a bit hard to read

I asked on IRC in what form people would like to see the benchmarking data for 
the 2.4 release. Igloo thought we should make graphs, and I agree. But what 
would be the best approach? I'm not very skilled at data visualization.

I can imagine we'd want some sort of box plot. Is it easy to make that by hand 
in Excel or OO Calc? Or should we add a gnuplot output format to darcs-
benchmark to automate it?

Reinier

The right answer in these kinds of situations is (almost) always to
automate.  

We could conceivably extend darcs-benchmark report with a gnuplot option
(hopefully re-using as much pre-existing code as possible).  The bad
news is that this would mean impinging on our volunteers for another
round of benchmarking.  The good news is that the upcoming version of
darcs-benchmark now saves results, so if we do ask them for more work,
it should be the last time we'll need their computers to be crunching
away.  I'm hoping that the plan of making it really easy for benchmarks
to accumulate works out.  Moar data!

[NB. I may not be able to work on the visualisation aspect of
darcs-benchmark until Sunday if at all, and it's looking like my
endurance is starting to thin out.  So no promises!  Better send in
those patches :-)]

Hi,

Op donderdag 25 februari 2010 22:48 schreef je:
> We could conceivably extend darcs-benchmark report with a gnuplot option
> (hopefully re-using as much pre-existing code as possible).  The bad
> news is that this would mean impinging on our volunteers for another
> round of benchmarking. 

I don't think I'm willing to wait for a new benchmarking round anymore. But a 
gnuplot output option would still be jolly useful for 2.5 of course :)

Reinier

On Fri, Feb 26, 2010 at 18:31:35 +0000, Reinier Lamers wrote:
> I don't think I'm willing to wait for a new benchmarking round anymore. But a 
> gnuplot output option would still be jolly useful for 2.5 of course :)

OK. I've done some manual reformatting to make the tables easier to read
(this will no longer be necessary once people start collecting data with
the new darcs-benchmark).

Shall we mark this as resolved for the purposes of this release?

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9

On Thu, Feb 25, 2010 at 19:41:22 -0500, Max Battcher wrote:
> Another approach might be to use Google Visualization API at the
> Wiki. Low tech immediate solution (which I may try tonight/tomorrow,
> it should be reasonably quick) would be simply to cobble together
> the necessary URL(s) and stick it on the wiki.

That would be great, if you wouldn't mind giving it a shot!

It may make sense to go with a manual solution for expediency, and also
to get an idea of how the data ought to be visualised so that we know
how to automate things.

I hope my recent reformatting work didn't disrupt any of this.  Perhaps
it made it easier, if you haven't already started.

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9

Eric Kow wrote:
> Eric Kow <kowey@darcs.net> added the comment:
> 
> On Thu, Feb 25, 2010 at 19:41:22 -0500, Max Battcher wrote:
>> Another approach might be to use Google Visualization API at the
>> Wiki. Low tech immediate solution (which I may try tonight/tomorrow,
>> it should be reasonably quick) would be simply to cobble together
>> the necessary URL(s) and stick it on the wiki.
> 
> That would be great, if you wouldn't mind giving it a shot!
 >
> It may make sense to go with a manual solution for expediency, and also
> to get an idea of how the data ought to be visualised so that we know
> how to automate things.

Tested it on my own system's page:

http://wiki.darcs.net/Benchmarks/Quasar

I ended up with a trio of Python scripts to semi-automate the task. They 
are ugly, quick hacks, but they seem to work as a start. One scrapes the 
wiki tables into CSV files, and the other two create graph URLs from CSV 
files.

This is my stopping point for the night, but additional pages should be 
relatively quick... A few obvious TODO items:

* Add units labels to y-axis
* Better unit handling all around (currently hardcoded and "dumb")
* Consider a way to standardize version color/order across graph tool runs
* Graphs for standard deviation? A box plot-like error marker on the 
time graphs?
* Remove duplications between graph scripts
* Comment, document, further automate, etc...

> I hope my recent reformatting work didn't disrupt any of this.  Perhaps
> it made it easier, if you haven't already started.

Didn't start until after your reformatting.

Let me know what y'all think of my quick hack for the visualization 
department,

--
--Max Battcher--
http://worldmaker.net

On Sat, Feb 27, 2010 at 01:40:12 -0500, Max Battcher wrote:
> Tested it on my own system's page:
> 
> http://wiki.darcs.net/Benchmarks/Quasar
...
> Let me know what y'all think of my quick hack for the visualization
> department,

Those look great!  The first thing I'd say is that the graphical
columns should match the order of the text tables.

See http://irclog.perlgeek.de/darcs/2010-02-27#i_2044699
for feedback, some of it seems to be mentioned below.

> I ended up with a trio of Python scripts to semi-automate the task.
> They are ugly, quick hacks, but they seem to work as a start. One
> scrapes the wiki tables into CSV files, and the other two create
> graph URLs from CSV files.
> 
> This is my stopping point for the night, but additional pages should
> be relatively quick... A few obvious TODO items:
> 
> * Add units labels to y-axis
> * Better unit handling all around (currently hardcoded and "dumb")
> * Consider a way to standardize version color/order across graph tool runs
> * Graphs for standard deviation? A box plot-like error marker on the
> time graphs?
> * Remove duplications between graph scripts
> * Comment, document, further automate, etc...

And now that you've shown the way (using Google Chart is great because
this means people don't have to worry about uploading images to the
wiki, just paste and bang!), I'll point out that there is a Haskell
package for GoogleChart that we should plug in darcs-benchmark:
http://hackage.haskell.org/package/GoogleChart

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9

Take two of my graphs have been posted to the wiki (including now for 
Apricot and Vs2). I'm attaching the two scripts as they stand tonight. 
They still are in the "quick hack" department, but looking better than 
they did last night. csvscrape.py is a very simple BeautifulSoup-based 
scraper to take all of a (wiki) page's tables and dump them into a set 
of CSV files. graph.py takes the CSV files and converts them to a series 
of reST image directives. Throw in the unix xclip command to pipe 
graph.py's output to the clipboard and the combined scrape, graph, copy, 
and paste labor isn't bad at all.

Eric Kow wrote:
> And now that you've shown the way (using Google Chart is great because
> this means people don't have to worry about uploading images to the
> wiki, just paste and bang!), I'll point out that there is a Haskell
> package for GoogleChart that we should plug in darcs-benchmark:
> http://hackage.haskell.org/package/GoogleChart

I think I might look into this. It would be a good opportunity to try my 
hand at bettering my Haskell skills, and definitely the way forward from 
my quick hack solution.

--
--Max Battcher--
http://worldmaker.net

Attachments

• csvscrape.py
• graph.py

OK, I've tweaked the code a little bit to hopefully make it easier for
you to look into integrating the GoogleCharts stuff.

Meanwhile, let's mark this resolved and do any future tracking in the
darcs-benchmark tracker (as managed by bugs-everywhere).  

If that's too cumbersome, just yell at me and we can think of something
else :-)