Issue 1283 run tests in separate subdirs

Title run tests in separate subdirs
Priority feature Status resolved
Milestone Resolved in 2.8.0
Superseder Nosy List ckeen, darcs-devel, dmitry.kurochkin, kowey, mornfall, thorkilnaur, twb
Assigned To mornfall
Topics Devel

Created on 2008-12-26.12:52:52 by twb, last changed 2010-09-03.08:41:03 by kowey.

msg6893 (view) Author: twb Date: 2008-12-26.12:52:48
Currently, all the tests in tests/ run directly in tests-foo/ and many
have a prelude along the lines of

    rm -rf test                 # cleanup after a previous run
    mkdir test
    cd test

This is stupid for two reasons: firstly, it is duplication of code.
Secondly, screw-ups are not obvious because a previous test script
might be inadvertently "fixing" the broken script.  We found two such
bugs merely by reversing the order in which test scripts ran!

Instead, the script harness should create a completely clean scratch
directory for each test, and cd into it before running the test.  It
should also automatically remove this scratch directory at the end of
the test -- UNLESS the test failed, in which case the scratch
directory might be left around as a diagnostic tool.

I propose that "cabal test" continue to use tests.<repo format>/
directories as it does now, but within that directory create a <script
name>/ directory for each test.

PS: when a test fails, perhaps its stderr and stdout could also be
placed in this test-specific directory, rather than printing it to
standard output (which tends to be overwhelming when you initially
just want a list of failing tests).
msg6978 (view) Author: thorkilnaur Date: 2009-01-05.14:14:49
Thanks Trent. I whole-heartedly agree with these suggestions. You seem to have 
assigned yourself to this task (?), so perhaps you would take the trouble, 
whenever suitable, to change the status to "in-progress".

Thanks again and best regards
msg6994 (view) Author: twb Date: 2009-01-06.00:26:14
This isn't as easy as I first expected, and I've made no progress on it.  Unassigning.
msg6996 (view) Author: kowey Date: 2009-01-06.09:07:48
Sounds like Christian might have some thoughts about this.
I'll assign it in case he's interested in looking into the issue.

P.S. Resent because everytime I try to use the email interface, I get the
syntax wrong and get rebuffed by roundup.  Sticking to ru and to the web
interface instead.
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
msg12307 (view) Author: kowey Date: 2010-08-25.17:51:08
I think Christian may be too busy for this, and our need for this has 
been crystal clear.

Perhaps we should work on this during the 2010-10 sprint

BTS training: what's the difference between feature/wishlist? Mostly a 
notion of importance.
msg12314 (view) Author: mornfall Date: 2010-08-26.12:12:33
I'll look into this as soon as I'm done with darcs-benchmark refactor.
msg12434 (view) Author: kowey Date: 2010-09-03.08:41:02
Implemented by Petr.  Hooray!
Date User Action Args
2008-12-26 12:52:52twbcreate
2009-01-05 14:14:59thorkilnaursetpriority: wishlist
status: unread -> needs-reproduction
topic: + X_DeprecatedTopic_Testing
messages: + msg6978
nosy: kowey, simon, twb, thorkilnaur, dmitry.kurochkin
2009-01-06 00:26:16twbsetassignedto: twb -> (no value)
messages: + msg6994
nosy: kowey, simon, twb, thorkilnaur, dmitry.kurochkin
2009-01-06 09:06:18koweysetassignedto: ckeen
nosy: + ckeen
2009-01-06 09:07:51koweysetnosy: kowey, simon, twb, thorkilnaur, dmitry.kurochkin, ckeen
messages: + msg6996
2009-08-10 16:18:25koweysettopic: + Devel, - X_DeprecatedTopic_Testing
nosy: kowey, simon, twb, thorkilnaur, dmitry.kurochkin, ckeen
2009-08-25 17:36:55adminsetnosy: + darcs-devel, - simon
2009-08-27 14:21:55adminsetnosy: kowey, darcs-devel, twb, thorkilnaur, dmitry.kurochkin, ckeen
2009-09-02 20:18:29koweysetstatus: needs-reproduction -> needs-implementation
nosy: kowey, darcs-devel, twb, thorkilnaur, dmitry.kurochkin, ckeen
2010-08-25 17:49:11koweylinkissue1930 superseder
2010-08-25 17:51:08koweysetpriority: wishlist -> feature
assignedto: ckeen -> (no value)
messages: + msg12307
2010-08-26 12:12:34mornfallsetassignedto: mornfall
messages: + msg12314
nosy: + mornfall
2010-09-03 08:41:03koweysetstatus: needs-implementation -> resolved
messages: + msg12434
resolvedin: 2.8.0