The darcs repo

    http://urchin.earth.li/darcs/ian/memory/

contains a library that, when LD_PRELOADed, tracks memory usage over the
execution of a program. It isn't perfect, but it probably works well
enough.

As I mentioned a while ago on one of the lists, it would be really
useful to have a darcs memory-suite (and probably also time-suite) in
addition to the test-suite. The hardest part is actually finding the
peak memory usage of a program's execution, but the above library should
solve that.

The ideal situation would be to have a project-independent
memory/time-suite with an elegant (E?)DSL and an easy way to get a list
of commands to reproduce failures by hand, but some individual tests
hacked together would be better than nothing.

Thanks
Ian

On Tue, Jan 17, 2006 at 03:46:27PM +0000, Ian Lynagh wrote:
> ...
> contains a library that, when LD_PRELOADed, tracks memory usage over the
> execution of a program. It isn't perfect, but it probably works well
> enough.

I like this idea, and LD_PRELOAD does seem like a relatively fool-proof
approach.  Is LD_PRELOAD purely linux-specific?

> As I mentioned a while ago on one of the lists, it would be really useful
> to have a darcs memory-suite (and probably also time-suite) in addition
> to the test-suite. The hardest part is actually finding the peak memory
> usage of a program's execution, but the above library should solve that.

Indeed, it would seem that with this library we could moderately easily
write a shell script (or haskell program) that could be run with something
like

./test-memory 25m ./darcs foobar

which would run "./darcs foobar" and print the peak memory use, and would
also print "FAILED" if the memory use is greater than 25 megabytes.  Such a
script would then make it very easy to add tests to the memory suite.

When this is written and added to the Makefile, the remaining tough part
would seem to be generating largish repositories upon which to test
darcs--which would be a very useful feature to have, but could take some
thought.  Ideally, one would like to have a repo-generating program that
could generate repos with various lengths histories, size patches,
directory depths, numbers of files per directory, length files, etc.
-- 
David Roundy
http://www.darcs.net

On Wed, Jan 18, 2006 at 11:35:33AM +0000, David Roundy wrote:
> 
> David Roundy <droundy@darcs.net> added the comment:
> 
> On Tue, Jan 17, 2006 at 03:46:27PM +0000, Ian Lynagh wrote:
> > ...
> > contains a library that, when LD_PRELOADed, tracks memory usage over the
> > execution of a program. It isn't perfect, but it probably works well
> > enough.
> 
> I like this idea, and LD_PRELOAD does seem like a relatively fool-proof
> approach.  Is LD_PRELOAD purely linux-specific?

No, but it's not universal either. I'd expect most space bugs to exist
either everywhere or nowhere, though.

The best second OS to cover in order to catch as many bugs as possible
is probably Windows, but I have no idea how to do this sort of thing
there.

> > As I mentioned a while ago on one of the lists, it would be really useful
> > to have a darcs memory-suite (and probably also time-suite) in addition
> > to the test-suite. The hardest part is actually finding the peak memory
> > usage of a program's execution, but the above library should solve that.
> 
> Indeed, it would seem that with this library we could moderately easily
> write a shell script (or haskell program) that could be run with something
> like
> 
> ./test-memory 25m ./darcs foobar

Ah, I hadn't thought about passing a maximum in, good point. We can
probably get it to the library via an environment variable.

> which would run "./darcs foobar" and print the peak memory use,

I don't think it should print to stdout/stderr; currently it writes to a
file called data, although this could also be set via env vars. We might
want one file for the step by step mem usage and another for general
stats like peak memory usage.

> When this is written and added to the Makefile, the remaining tough part
> would seem to be generating largish repositories upon which to test
> darcs--which would be a very useful feature to have, but could take some
> thought.  Ideally, one would like to have a repo-generating program that
> could generate repos with various lengths histories, size patches,
> directory depths, numbers of files per directory, length files, etc.

This should be simple enough with a few bits of perl, like

perl -e 'print "Foo\n"x10000' > foo

(or the generating script could be a perl program itself.

Thanks
Ian

On Wed, Jan 18, 2006 at 01:44:41PM +0000, Ian Lynagh wrote:
> > Indeed, it would seem that with this library we could moderately easily
> > write a shell script (or haskell program) that could be run with something
> > like
> > 
> > ./test-memory 25m ./darcs foobar
> 
> Ah, I hadn't thought about passing a maximum in, good point. We can
> probably get it to the library via an environment variable.
>
> > which would run "./darcs foobar" and print the peak memory use,
> 
> I don't think it should print to stdout/stderr; currently it writes to a
> file called data, although this could also be set via env vars. We might
> want one file for the step by step mem usage and another for general
> stats like peak memory usage.

I was thinking of ./test-memory as a driver script, which would parse the
output file and then present it in a pretty manner to the caller (on
stdout).  Definitely the library should be dumping to a file, so we can
simultaneously check the values of stdout if we so desire.

> > When this is written and added to the Makefile, the remaining tough
> > part would seem to be generating largish repositories upon which to
> > test darcs--which would be a very useful feature to have, but could
> > take some thought.  Ideally, one would like to have a repo-generating
> > program that could generate repos with various lengths histories, size
> > patches, directory depths, numbers of files per directory, length
> > files, etc.
> 
> This should be simple enough with a few bits of perl, like
> 
> perl -e 'print "Foo\n"x10000' > foo
> 
> (or the generating script could be a perl program itself.

That'll handle a few files, but if we want to generate repositories with
entire directory structures and long histories--so that we could, for
example, check the memory usage of darcs changes -v, it would be nice to
stick have a nice framework (probably written in perl) for generating our
repositories.  The point of the framework being to allow us to change any
dimension we like of the repository to see how memory use or time scale
with that.  e.g. the length of files, number of files, size of changes
(i.e. just a few lines in a large files, or many separate hunks in each
file, or one huge hunk in each file), number of changes, etc.  There are so
many dimensions here, many of which critically will affect some time or
space inefficiency, that I'd like a unified way of specifying them.

If we were thorough enough, we'd then be able to create nice plots like
"time versus number of patches since the last tag upon which the inventory
is split, with everything else held fixed".
-- 
David Roundy
http://www.darcs.net

Ian, if you have a spare moment, do you think you could turn this into reality,
i.e. as a part of the darcs darcs repository slash makefile?

We have four distinct bugs about memory consumption (see in particular issue622 ).

There is also the maybench project and its darcs benchmark... but here I do not
particularly care about generated tests.  Handwritten, reproducible ones would
be good.

I don't expect to have a spare minute for this in the near future, sorry.

If I understand Jason correctly, we need a volunteer to 'bake this into darcs
and use a reporting technique like ghc uses.Print "<<mmap: bytes allocated
:mmap>>" to stderr when --track-mmap is passed to darcs.'

Does Petr's darcs-benchmark do what we want?

  http://hackage.haskell.org/package/darcs-benchmark

It prints memory use