Issue 2720 Mysterious test failures on MacOS with ghc >= 9.6

Title Mysterious test failures on MacOS with ghc >= 9.6
Priority bug Status unknown
Milestone Resolved in
Superseder Nosy List bfrk
Assigned To

Created on 2024-05-18.20:02:57 by bfrk, last changed 2024-05-20.12:04:40 by bfrk.

msg23826 (view) Author: bfrk Date: 2024-05-18.20:02:56
With ghc >= 9.6 we get test failures on MacOS for tests/issue2333.sh and 
tests/issue1465_ortryrunning.sh. For issue2333 it says

| PATH='' $thedarcs push ../S -a --debug # Try to push patches between repos.
| + PATH=
| + /Users/runner/work/darcs-ci/darcs-ci/dist-newstyle/build/x86_64-osx/ghc-
9.6.4/darcs-2.19.1/build/darcs/darcs push ../S -a --debug
| darcs: security: createProcess: posix_spawnp: does not exist (No such file or 

I added the --debug flag and also added a debug message to be issued right before 
we run the 'darcs apply'. I am probably overlooking something here...

Full log:

msg23831 (view) Author: bfrk Date: 2024-05-18.21:13:04
The problem in tests/issue1465_ortryrunning.sh can be avoided by a minor 
change in the test script, which anyway doesn't quite test (at this 
particular invocation) what it claims to. Will send a patch.
msg23847 (view) Author: ganesh Date: 2024-05-19.13:58:38
I will look into the issue2333 failure.
msg23859 (view) Author: ganesh Date: 2024-05-19.20:02:57
Following up on the issue2333 failure in issue2722.
msg23866 (view) Author: ganesh Date: 2024-05-19.21:10:55
Never mind, it looks like the two have the same root cause.

In both cases, the tests that fail are emptying out $PATH. In
the case of issue2333, the error points to the 'security' binary,
which lives in /usr/bin.

Putting /usr/bin into the explicit PATH in both tests fixes them.
But that is quite hacky, particularly as other things like darcs itself
might be in /usr/bin.

I'll dig more into what the invocation of /usr/bin/security is for.
msg23867 (view) Author: ganesh Date: 2024-05-19.21:17:07
It seems likely that it's used to find certificates, and that it's this
line or something similar:


But I don't know why it would have only started failing with more recent GHCs.
I'll explore that a bit more.
msg23872 (view) Author: ganesh Date: 2024-05-19.22:26:02
It doesn't look like it's specific to recent GHCs, I get it with 9.0 and 
9.4 (but not 8.2) without any special package choices.

It feels like a bit of an edge case to be running with such a
cut down PATH - we're only really doing it to manage the test

Given we're not actually making any remote calls in those tests,
maybe we should see if we can make the certificate read more lazy,
so that it is only triggered when actually needed.
msg23873 (view) Author: ganesh Date: 2024-05-19.22:57:04
I haven't tracked down all the details but I don't know if we can easily make the certificate read lazily.

I guess that this unsafePerformIO here will happen unconditionally:


and I think that it ultimately calls `getSystemCertificateStore`.

Plus we'd have to rework our use of the things in Network.HTTP.Simple to pass a manager explicitly.
msg23874 (view) Author: bfrk Date: 2024-05-19.22:59:33
Hm, so they run a program named "security". And use whatever is found 
first in the user's PATH. That doesn't strike me as being secure in the 
first place. If they called it with the full path it would at least 
guarantee that it's the intended program; and incidentally solve the 
issue as well.
msg23879 (view) Author: ganesh Date: 2024-05-20.08:19:57
It's just to get a list of trusted certificates for https. If someone
has managed to manipulate the system to change the result of that, they
have bigger problems.

I don't know if it's reasonable to assume it'll always be in /usr/bin.
Most of the recommendations I found online when searching for more 
information (that talked about other languages etc) didn't include the
full path either.
msg23881 (view) Author: bfrk Date: 2024-05-20.08:45:52
Regardless of whether one regards running the "security" program from the PATH as a 
potential security problem or not: preventing *any* program that uses x509-system 
directly or indirectly (including any program that uses http-client) from working 
just because /usr/bin is not in the PATH is a severe and unnecessary restriction. 
The more so, considering the effort it has cost us to find out what goes wrong 
here. I think raising an issue with x509-system is therefore more than justified. 
At least they should allow to override the default, e.g. with a special environment 
variable. And in case the program fails to execute, they should issue a message 
that suggests how to fix it e.g. "Either make sure that the 'security' program is 
in the PATH or set X509_SYSTEM_OSX_SECURITY=/usr/bin/security." or something like 
msg23882 (view) Author: bfrk Date: 2024-05-20.08:52:14

The responses list a number of places where certificates are stored. So 
another way to fix the issue in x509-system is to use those paths as a 
fallback if running the 'security' command fails.
msg23886 (view) Author: ganesh Date: 2024-05-20.10:52:15
The reason for not being able to use `http-client` at all is a mix of
the x509 package shelling out to a separate
program, and whatever has a top-level `unsafePerformIO` that invokes
that logic, so it's not totally clear-cut where the best fix is, IMO.
I agree with your suggestions as possible mitigations.
msg23887 (view) Author: bfrk Date: 2024-05-20.12:04:40
Date User Action Args
2024-05-18 20:02:57bfrkcreate
2024-05-18 21:13:04bfrksetmessages: + msg23831
2024-05-19 13:58:39ganeshsetmessages: + msg23847
2024-05-19 20:02:57ganeshsetmessages: + msg23859
2024-05-19 21:10:56ganeshsetmessages: + msg23866
2024-05-19 21:17:07ganeshsetmessages: + msg23867
2024-05-19 22:26:02ganeshsetmessages: + msg23872
2024-05-19 22:57:04ganeshsetmessages: + msg23873
2024-05-19 22:59:33bfrksetmessages: + msg23874
2024-05-20 08:19:58ganeshsetmessages: + msg23879
2024-05-20 08:45:52bfrksetmessages: + msg23881
2024-05-20 08:52:14bfrksetmessages: + msg23882
2024-05-20 10:52:15ganeshsetmessages: + msg23886
2024-05-20 12:04:40bfrksetmessages: + msg23887