On Mon, Mar 03, 2008 at 03:49:02PM -0000, Eric Kow wrote:
> We are developing a Haskell benchmarking library called Maybench.  The project
> also includes a darcs-specific wrapper (darcs-benchmark), which we are primarily
> focusing on.
> 
> See http://code.google.com/p/maybench
> 
> Or darcs get http://code.haskell.org/maybench

I should perhaps mention that darcs-benchmark isn't currently working for
me... I'm not sure why.  It hangs on the call to darcs record.  :(
-- 
David Roundy
Department of Physics
Oregon State University

I guess now that we have something which sort of works, I should point out that
work has started on this.  

We are developing a Haskell benchmarking library called Maybench.  The project
also includes a darcs-specific wrapper (darcs-benchmark), which we are primarily
focusing on.

See http://code.google.com/p/maybench

Or darcs get http://code.haskell.org/maybench

This is what the output of maybench is on the yi repository; this is basically
what ertai mentioned he was working on, with a little bit of fiddling by me and
David.

------------------8<----------------------------------
Running... /usr/local/bin/darcs get --quiet /foo/yi/ main
Running... /usr/local/bin/darcs changes --quiet --repodir main
unrecord last 1
Running... /usr/local/bin/darcs unrecord --quiet --last 1 --all --repodir main
0.110191s
record
Running... /usr/local/bin/darcs record --quiet --all -m "test patch" --repodir main
0.062309s
obliterate last 1
Running... /usr/local/bin/darcs obliterate --quiet --last 1 --all --repodir main
1.185752s
pull 1
Running... /usr/local/bin/darcs pull --quiet --all --repodir main
1.117513s
obliterate last 50
Running... /usr/local/bin/darcs obliterate --quiet --last 50 --all --repodir main
1.53781s
pull 50
Running... /usr/local/bin/darcs pull --quiet --all --repodir main
1.582683s
obliterate last 500
Running... /usr/local/bin/darcs obliterate --quiet --last 500 --all --repodir main
14.193587s
pull 500
Running... /usr/local/bin/darcs pull --quiet --all --repodir main
9.192307s
--------------------8<-------------------

As you can see, it marks benches.  There is some extra work to do, like making
the output nice and pretty, or generating comparisons with other versions of
darcs.  Perhaps even doing automated performance regression testing (raise an
alarm if darcs ever becomes 3X slower, or something like that).

Help definitely wanted, even if it's "just" to write benchmarks.

All are invited to join in!  If you've started working on something like this,
you should especially consider submitting some code.  Unfortunately, the mailing
list settings are a bit strict,; your message will be rejected if you do not
first subscribe.  We're working on loosening that up a bit.  Also, we're
experimenting with a liberal right-to-push model, so just ask if you want to be
able to push patches in directly.

I should mention that I've found a couple of patterns useful in tracking down
performance bugs:

darcs obliterate --last 100 -a && darcs pull -a

darcs unrecord --last 1 -a && darcs record -a -m foo

It would be helpful for a benchmarking code to focus on inverse pairs like this,
because they're very easy to work with when tracking down code.

And a good start for a benchmarking framework would simply be something that
sets up a few repos and then runs the above on each of them.

David

I started on this about three years ago. You are welcome to re-use my work from
then:

http://lists.osuosl.org/pipermail/darcs-devel/2005-March/001524.html

http://mark.stosberg.com/darcs_hive/darcs_speed_test/

    Mark

Excerpts from bugs's message of Wed Feb 06 17:24:02 UTC 2008:
> 
> New submission from Eric Kow <eric.kow@gmail.com>:
> 
> It would be really useful to have a quick and dirty benchmarking script that we
> can either distribute with or alongside darcs
> 
> The script would just encode some current darcs benchmarking practices: namely,
> getting a repository, obliterating a 1000 patches and pulling them back.  Having
> it do the best of N trials is probably a good idea too.  Automatically
> summarising the results (% improvement) would be nice too.
> 
> Perhaps a good way to do it is to have it be parameterisable with two
> directories (with darcs built in both) and a repository.  To avoid people having
> to choose a repository, we could also distribute the GHC one as the prototypical
> large-ish repository and hopefully set our sights higher one day.
> 
> Also, I think it's more important for us to have a basic version of this script
> now, than a fancy one later :-)

I'm  currently working on such a script. I'm playing with it in my spare time,
and it's not yet usable.

On Feb 6, 2008 9:24 AM, Eric Kow <bugs@darcs.net> wrote:
>
> New submission from Eric Kow <eric.kow@gmail.com>:
>
> It would be really useful to have a quick and dirty benchmarking script that we
> can either distribute with or alongside darcs
>
> The script would just encode some current darcs benchmarking practices: namely,
> getting a repository, obliterating a 1000 patches and pulling them back.  Having
> it do the best of N trials is probably a good idea too.  Automatically
> summarising the results (% improvement) would be nice too.
>
> Perhaps a good way to do it is to have it be parameterisable with two
> directories (with darcs built in both) and a repository.  To avoid people having
> to choose a repository, we could also distribute the GHC one as the prototypical
> large-ish repository and hopefully set our sights higher one day.
>
> Also, I think it's more important for us to have a basic version of this script
> now, than a fancy one later :-)

I started making a fancy one as a class project (it's in scala) and
works okay, it just needs more usage patterns/benchmarks implemented.
It's actually quite simple and I spend most of the time on the project
just learning scala, so a Haskell re-implementation could probably be
made easily.  BTW, I see this as an argument for having a libdarcs.
It would be easier to create one-off versions of darcs for
benchmarking using a libdarcs than it is to create wrappers around
darcs.

Here is the link if you want to tear it apart or get ideas:
http://www.codersbase.com/index.php/DarcsSim

Probably the most useful thing you'll get out of it (if you don't use
it) is the way I use ghc's run-time to extract metrics.

Jason

On Feb 6, 2008, at 10:24 AM, Eric Kow wrote:

> It would be really useful to have a quick and dirty benchmarking  
> script that we
> can either distribute with or alongside darcs

That's an excellent idea.

Regards,

Zooko

It would be really useful to have a quick and dirty benchmarking script that we
can either distribute with or alongside darcs

The script would just encode some current darcs benchmarking practices: namely,
getting a repository, obliterating a 1000 patches and pulling them back.  Having
it do the best of N trials is probably a good idea too.  Automatically
summarising the results (% improvement) would be nice too.

Perhaps a good way to do it is to have it be parameterisable with two
directories (with darcs built in both) and a repository.  To avoid people having
to choose a repository, we could also distribute the GHC one as the prototypical
large-ish repository and hopefully set our sights higher one day.

Also, I think it's more important for us to have a basic version of this script
now, than a fancy one later :-)