I don't know to what extent this is possible, but it would be nicer (for
commands like darcs whatsnew and darcs record) if darcs wouldn't spend so much
time reading, copying from or synchronising pristine.

I guess we can file this under micro-performance.  It takes a few seconds for it
to do this, which is not very long, but long enough to induce a sort of
impatience/frustration.  It would be nice if recording was instantaneous.

Any ideas?

Ah-hah!  The key here is that I'm on an NFS share.  My repository has these
characteristics (unfortunately, I can't share it yet):
523 files, 152 patches (yay for darcs show files | wc -l and darcs changes --count).

Doing a darcs mv of a directory takes 8 seconds on NFS and barely a second on
the local hard disk.

I think Ian had some ideas about simulating slow disks?

Would something like this be useful for debugging this? 

slow-fs
=======
I wish there was some kind of tool that would let me simulate a slow file system
(e.g. an NFS share).   The idea is that we would do something like (*):

  mount.slow-fs /pretend/slow/path /actual/location 0.1

And that whenever I tried to read or write from /pretend/slow/path, slow-fs
would insert an artificial 0.1 ms (*) delay before actually reading or writing
from /actual/location.

(*)  I don't know what actual mount.foo syntax should look like
(**) or whatever units are appropriate

My nfs-mounted repo is 1196 files and 1975 patches. The latter took 20 seconds
to report. whatsnew regularly takes 30+ seconds, which means I don't use darcs
as much as I would like to.

Another data point we might want to be collecting is "du -Hs _darcs/pristine
_darcs/patches". Mine is 18M pristine tree; 35M of patches. I'm still on
darcs-1.0 format, and I'm running darcs 2.1.2.

I don't think your slow-fs idea reflects actual network filesystems very well 
(or at all). Benchmarking on real NFS would be both much more useful and much 
easier. YMMV.

On Thu, Jan 15, 2009 at 20:12:41 -0000, Petr Ročkai wrote:
> I don't think your slow-fs idea reflects actual network filesystems very well 
> (or at all). Benchmarking on real NFS would be both much more useful and much 
> easier. YMMV.

Is there any way to simulate the kind of slow downs you get from using a
network filesystem?

The point would be to let would-be performance hackers benchmark darcs
(a) on some kind of fair and reproducible environment and (b) without
needing access to real NFS shares... That is, we want to make it so that
people that can hack on performance do so independently of those of us
who suffer from its performance.  Surely there must some way to achieve
this?

The "slowfs" idea could be implemented on Linux using FUSE,
using something like rofs as an example.  rofs is only 500 lines
of C, and what it does is make an existing filesystem appear else-
where in the directory tree, but read-only (i.e. like the new mount
-o bind,ro).  You could do a similar thing, except instead of making
the system read-only, just introduce a 100ms sleep before each
operation.

This should even work without root privileges, because sshfs works
without root privileges.

> The "slowfs" idea could be implemented on Linux using FUSE,
> using something like rofs as an example.

Is this something that would be easy for you (Trent) to just
crank out?  It may be fun experimenting with it maybe releasing
it as a general open source project (who knows who else may want
to do such benchmarking).  It may not be particularly realistic,
but it may be good enough for benchmarking...

On Fri, Jan 16, 2009 at 10:03:09AM -0000, Eric Kow wrote:
>> The "slowfs" idea could be implemented on Linux using FUSE, using
>> something like rofs as an example.
>
> Is this something that would be easy for you (Trent) to just crank
> out?

As I have no experience with libfuse, I would need to spend an hour or
two looking at its documentation and rofs' source before I could give
a sensible time estimate.  Right now my estimates (best, expected,
worst) would be (¼, 1, 5) man-days, where one day is eight hours.

> It may not be particularly realistic, but it may be good enough for
> benchmarking...

Precisely.

On Fri, Jan 16, 2009 at 10:13:12 -0000, Trent Buck wrote:
> As I have no experience with libfuse, I would need to spend an hour or
> two looking at its documentation and rofs' source before I could give
> a sensible time estimate.  Right now my estimates (best, expected,
> worst) would be (¼, 1, 5) man-days, where one day is eight hours.

Nice estimating!

I'd appreciate it if you could go ahead and refine it, i.e. look at the
libfuse documentation (and rofs source) so that at the very least we
have a clearer idea how easy it would be for another volunteer (or you)
to do this.

Status update.

On Wed, Jan 21, 2009 at 11:34:25PM +0000, Eric Kow wrote:
> On Fri, Jan 16, 2009 at 10:13:12 -0000, Trent Buck wrote:
> > As I have no experience with libfuse, I would need to spend an hour or
> > two looking at its documentation and rofs' source before I could give
> > a sensible time estimate.  Right now my estimates (best, expected,
> > worst) would be (¼, 1, 5) man-days, where one day is eight hours.
> 
> Nice estimating!
> 
> I'd appreciate it if you could go ahead and refine it, i.e. look at the
> libfuse documentation (and rofs source) so that at the very least we
> have a clearer idea how easy it would be for another volunteer (or you)
> to do this.

I had a look at this tonight, and it looks like basically you have a
single slowfs.c file that has the following layout:

  - a global variable real_path :: [Char].
  - main (), which calls fuse_main ().
  - a function translate_path path = real_path </> path
  - a bunch of functions slow_foo(args), that call the underlying
    syscall foo(args) after using translate_path on the appropriate
    argument.

  - a global structure of the form { fuse_foo = slow_foo; ... } that
    is passed to fuse_main.

All of which sounds pretty straightforward, but a bit tedious and
detailed, in particular since it seems that sometimes the underlying
syscall of slow_foo isn't called foo, but rather lfoo or foo64 or even
bar.

I also found that there is a HSFuse wrapper, which may or may not
simplify matters.  Isaac Jones has used it for [1], at least.  Is
Isaac interested in Darcs development?

I'm going to bed now; I don't know if I'll feel like working on this
tomorrow.

[1] http://www.haskell.org/halfs/

Hi Isaac,

You were mentioned in Trent's estimate below.  We were trying to see how
much work it would take to develop a 'slowfs' wrapper to help us improve
our darcs benchmarking efforts by simulating (however poorly) a network
drive.

By any chance, would you have any comments, particularly on...

> I also found that there is a HSFuse wrapper, which may or may not
> simplify matters.  Isaac Jones has used it for [1], at least.  Is
> Isaac interested in Darcs development?

> [1] http://www.haskell.org/halfs/

Thanks! -Eric

On Sun, Jan 25, 2009 at 14:18:17 -0000, Trent Buck wrote:
> On Wed, Jan 21, 2009 at 11:34:25PM +0000, Eric Kow wrote:
> > On Fri, Jan 16, 2009 at 10:13:12 -0000, Trent Buck wrote:
> > > As I have no experience with libfuse, I would need to spend an hour or
> > > two looking at its documentation and rofs' source before I could give
> > > a sensible time estimate.  Right now my estimates (best, expected,
> > > worst) would be (¼, 1, 5) man-days, where one day is eight hours.
> > 
> > Nice estimating!
> > 
> > I'd appreciate it if you could go ahead and refine it, i.e. look at the
> > libfuse documentation (and rofs source) so that at the very least we
> > have a clearer idea how easy it would be for another volunteer (or you)
> > to do this.
> 
> I had a look at this tonight, and it looks like basically you have a
> single slowfs.c file that has the following layout:
> 
>   - a global variable real_path :: [Char].
>   - main (), which calls fuse_main ().
>   - a function translate_path path = real_path </> path
>   - a bunch of functions slow_foo(args), that call the underlying
>     syscall foo(args) after using translate_path on the appropriate
>     argument.
> 
>   - a global structure of the form { fuse_foo = slow_foo; ... } that
>     is passed to fuse_main.
> 
> All of which sounds pretty straightforward, but a bit tedious and
> detailed, in particular since it seems that sometimes the underlying
> syscall of slow_foo isn't called foo, but rather lfoo or foo64 or even
> bar.
> 
> I also found that there is a HSFuse wrapper, which may or may not
> simplify matters.  Isaac Jones has used it for [1], at least.  Is
> Isaac interested in Darcs development?
> 
> I'm going to bed now; I don't know if I'll feel like working on this
> tomorrow.
> 
> [1] http://www.haskell.org/halfs/

Ok, no comments from Isaac.  I'm going to mark this need-volunteer, with the
next action being to write a slowfs.c as described by Trent in
http://bugs.darcs.net/msg7178