darcs

Patch 887 Always run the failing tests, with their... (and 4 more)

Title Always run the failing tests, with their... (and 4 more)
Superseder Nosy List owst
Related Issues
Status rejected Assigned To owst
Milestone

Created on 2012-07-21.15:01:53 by owst, last changed 2015-06-17.19:28:13 by bfrk.

Files
File name Status Uploaded Type Edit Remove
always-run-the-failing-tests_-with-their-result-being-negated.dpatch owst, 2012-07-21.15:01:52 application/x-darcs-patch
patch-preview.txt owst, 2012-07-21.15:01:52 text/x-darcs-patch
unnamed owst, 2012-07-21.15:01:52
See mailing list archives for discussion on individual patches.
Messages
msg15904 (view) Author: owst Date: 2012-07-21.15:01:52
This set of patches make the testsuite always run failing tests. It also
renames a couple of tests that no longer fail.

I'd like someone else to look over this, before I push it to the main repo,
since I seem to recall it being a slightly controversial idea?

5 patches for repository http://darcs.net:

Sat Jul 21 14:31:20 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * Always run the failing tests, with their result being negated
  Unless told not to, failing tests will now always run. Since they are expected
  to fail, a test run is considered a success if the test fails, and vice-versa.

Sat Jul 21 14:35:39 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * Failing issue2017 test now passes

Sat Jul 21 14:52:09 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * Issue 68 test no longer fails (also rework for a more efficient test)

Sat Jul 21 14:55:58 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * failing issue1401 test should be skipped on hashed repos, not fail

Sat Jul 21 15:42:01 BST 2012  Owen Stephens <darcs@owenstephens.co.uk>
  * Skip failing test 1829 on hashed repos, since its a problem with Conflictors
Attachments
msg15913 (view) Author: ganesh Date: 2012-07-21.18:45:07
I have a couple of concerns:
 - How do we mark tests that fail intermittently or in only on certain 
platforms?
 - How do we signal what's going on very clearly so that a future darcs 
developer who fixes a failing test in passing doesn't think they've 
actually broken something?

Other than that I think this concept is fine.
msg15914 (view) Author: owst Date: 2012-07-21.19:36:14
>  - How do we mark tests that fail intermittently or in only on certain 
> platforms?

Not sure, but do we handle this at all at the moment? I'm not sure we do.

>  - How do we signal what's going on very clearly so that a future darcs 
> developer who fixes a failing test in passing doesn't think they've 
> actually broken something?

Yeah, this is a concern that I had. I looked at test-framework but I
couldn't
see an easy way to modify the "[Success]" or "[Failed]" messages (such
that we
could make it obvious that a failed on a failing test was a good thing.
msg15915 (view) Author: ganesh Date: 2012-07-21.19:46:31
>>  - How do we mark tests that fail intermittently or in only on 
certain 
>> platforms?
> 
> Not sure, but do we handle this at all at the moment? I'm not sure we 
do.

We can mark them as failing.

The problem after this change will be that eiither marking them as 
passing or as failing will lead to unexpected failures in some 
scenarios.
msg15916 (view) Author: owst Date: 2012-07-21.19:57:34
> We can mark them as failing.
>
> The problem after this change will be that eiither marking them as 
> passing or as failing will lead to unexpected failures in some 
> scenarios.

Good point. Maybe this patchset is a bad idea overal then. I suppose
when changing code you are more worried about breaking existing
things/regressing to an old bug. 

My only worry (as was evidenced by the last few patches here) with the
status quo was that no-longer-failing tests would get left in failing-*.
msg18020 (view) Author: bfrk Date: 2015-02-05.15:50:58
What about the tests that no longer fail? Do they belong to the sort
that sometimes fail or can we screen the renamings?

I think we should introduce a new maybe-failing- class of tests. These
should be run by default, so when they fail it is apparent that we don't
have a new issue.
msg18538 (view) Author: bfrk Date: 2015-06-17.19:28:12
The main change ("Always run the failing tests...") no longer applies to
screened w/o lots of conflicts. Since the idea seems to be controversial
and nothing happened here for a long time I have set the status to
reject and instead re-sent the (uncontroversial) test renaming patches
as a separate bundle.
History
Date User Action Args
2012-07-21 15:01:53owstcreate
2012-07-21 18:45:08ganeshsetmessages: + msg15913
2012-07-21 19:36:14owstsetmessages: + msg15914
2012-07-21 19:46:31ganeshsetmessages: + msg15915
2012-07-21 19:57:34owstsetmessages: + msg15916
2012-08-26 00:31:45ganeshsetstatus: needs-screening -> in-discussion
2012-09-21 10:38:36owstsetassignedto: owst
2015-02-05 15:50:58bfrksetmessages: + msg18020
2015-06-17 19:28:13bfrksetstatus: in-discussion -> rejected
messages: + msg18538