This set of patches make the testsuite always run failing tests. It also
renames a couple of tests that no longer fail.

I'd like someone else to look over this, before I push it to the main repo,
since I seem to recall it being a slightly controversial idea?

5 patches for repository http://darcs.net:

Sat Jul 21 14:31:20 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * Always run the failing tests, with their result being negated
  Unless told not to, failing tests will now always run. Since they are expected
  to fail, a test run is considered a success if the test fails, and vice-versa.

Sat Jul 21 14:35:39 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * Failing issue2017 test now passes

Sat Jul 21 14:52:09 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * Issue 68 test no longer fails (also rework for a more efficient test)

Sat Jul 21 14:55:58 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * failing issue1401 test should be skipped on hashed repos, not fail

Sat Jul 21 15:42:01 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * Skip failing test 1829 on hashed repos, since its a problem with Conflictors

Attachments

• always-run-the-failing-tests_-with-their-result-being-negated.dpatch
• patch-preview.txt
• unnamed

I have a couple of concerns:
 - How do we mark tests that fail intermittently or in only on certain 
platforms?
 - How do we signal what's going on very clearly so that a future darcs 
developer who fixes a failing test in passing doesn't think they've 
actually broken something?

Other than that I think this concept is fine.

>  - How do we mark tests that fail intermittently or in only on certain 
> platforms?

Not sure, but do we handle this at all at the moment? I'm not sure we do.

>  - How do we signal what's going on very clearly so that a future darcs 
> developer who fixes a failing test in passing doesn't think they've 
> actually broken something?

Yeah, this is a concern that I had. I looked at test-framework but I
couldn't
see an easy way to modify the "[Success]" or "[Failed]" messages (such
that we
could make it obvious that a failed on a failing test was a good thing.

>>  - How do we mark tests that fail intermittently or in only on 
certain 
>> platforms?
> 
> Not sure, but do we handle this at all at the moment? I'm not sure we 
do.

We can mark them as failing.

The problem after this change will be that eiither marking them as 
passing or as failing will lead to unexpected failures in some 
scenarios.

> We can mark them as failing.
>
> The problem after this change will be that eiither marking them as 
> passing or as failing will lead to unexpected failures in some 
> scenarios.

Good point. Maybe this patchset is a bad idea overal then. I suppose
when changing code you are more worried about breaking existing
things/regressing to an old bug. 

My only worry (as was evidenced by the last few patches here) with the
status quo was that no-longer-failing tests would get left in failing-*.

What about the tests that no longer fail? Do they belong to the sort
that sometimes fail or can we screen the renamings?

I think we should introduce a new maybe-failing- class of tests. These
should be run by default, so when they fail it is apparent that we don't
have a new issue.

The main change ("Always run the failing tests...") no longer applies to
screened w/o lots of conflicts. Since the idea seems to be controversial
and nothing happened here for a long time I have set the status to
reject and instead re-sent the (uncontroversial) test renaming patches
as a separate bundle.