When issue64 has been solved, darcs stores patch metadata as Unicode strings.
However, the code that displays the metadata was not written with non-ASCII
characters in mind. I believe I saw that it replaces non-ASCII characters with
numerical escapes, but I have not been able to find code for that in the Printer
module.

There is also the conceptual question of how to make Printer.renderPS treat
non-ASCII characters.

Is this worthy of a Darcs 2.5.1?
Lele just experienced an issue similar to issue1990

If we get a fix, yes.

I looked at it for a bit and ultimately what happens is that hPutStr
writes only the lowest 8 bits of a character with a code point > 255. To
solve this, we must either explicitly encode the text with the locale
encoding before hPutStr'ing it, or we require GHC >= 6.12 and don't
switch file handles to binary mode.

The latter solution seems better because it is what the IO library is
designed for. But I don't know which parts of darcs require the file
handles to be in binary mode, and why.

Note that for the Windows line ending behavior that the "binary mode"
flag in C is for, you don't have to put the handle in binary mode in
Haskell. 'hSetNewlineMode noNewlineTranslation' will do that for you and
still write characters outside of latin1 correctly.

Reinier Lamers <bugs@darcs.net> writes:
> I looked at it for a bit and ultimately what happens is that hPutStr
> writes only the lowest 8 bits of a character with a code point > 255. To
> solve this, we must either explicitly encode the text with the locale
> encoding before hPutStr'ing it, or we require GHC >= 6.12 and don't
> switch file handles to binary mode.

We can't do the latter, since it breaks windows. Non-binary handles only
work with ghc <= 6.10 on that platform, encountering a "bad" character
with 6.12 or later will crash the program. Happens to darcs a lot for
various reasons. You really need to use binary handles for everything.

Yours,
  Petr.

We made an impromptu attempt at working out what was going on in
http://irclog.perlgeek.de/darcs/2010-12-16#i_3094602

It probably duplicates a lot of stuff that Reinier already knows, but 
never hurts to repeat the what-the-heck-is-darcs-doing exercise across 
different members of the team

These two emails may be relevant:

 - http://lists.osuosl.org/pipermail/darcs-users/2010-
December/025932.html
 - http://lists.osuosl.org/pipermail/darcs-users/2010-May/024023.html

Bumping to 2.5.2 - if a fix appears soon it might still make 2.5.1.

I just want to note that, due to this bug (in particular, the kind of
output described in issue 1990), I am still using darcs 2.4.4.

This bug's status is "needs-reproduction". Does this mean that the
maintainers have not been able to replicate it?

If that's the case, here's a quick way I can reproduce the bug:

> darcs init foo
> cd foo
> touch foo
> darcs add foo
> darcs record
As the patch name, use "берегам" (or any other unicode-containing string)
> darcs log

This displays

  * <U+0431><U+0435><U+0440><U+0435><U+0433><U+0430><U+043C>

if DARCS_DONT_ESCAPE_8BIT=0 and

  * 15@530<
if DARCS_DONT_ESCAPE_8BIT=1

We shouldn't have left this hanging so long, it's pretty fundamental to 
anyone in a non-ASCII locale. I'm taking a look at it now.

I've spent some time looking at this and sent draft patch1239

I don't really like it as it's a layer of sticking plaster on top of an already messy 
situation. It also doesn't make the situation any better on Windows or in non-UTF8 
locales, though I don't think it makes it any worse.

The patch basically adds a new parameter to all the printing code to decide whether to 
locale-encode Strings as they are output or not. Bytestrings are left alone as they 
typically seem to be sourced from patch metadata which is already UTF8-encoded.

I don't think we can do this translation unconditionally because the same code is used 
to print out patch files on disk etc. It seems to me that the right long-term solution 
would be to use phantom types to tag both Strings and Bytestrings with their encoding 
(e.g. locale, UTF8, 7bit-only). This would make it clear where in the code we should 
convert things. We should also cleanly split out the "print for user consumption" and 
"print for persistence" use cases; they're currently tangled together all the way up 
to classes like ShowPatch.

Users will also need to set the environment variable DARCS_DONT_ESCAPE_8BIT=1 to get 
UTF8 output. It feels like we should make that the default, but I don't fully 
understand the implications and why it isn't already the default.

Patch1239 was a first attempt at bringing this closer to a conclusion.
It makes it apparent (and easily changeable) when and where a possible
encoding step gets inserted.

However, the question remains, how do we know in general when to pass
Encode and when to pass Standard? Deciding this requires non-local
knowledge about what kind of data we are processing which is error prone.

My gut feeling is that the *producer* should put that information into
the Doc, rather than the consumer guessing it. If I am right with that,
the distinction can and should be made apparent in the type of the data,
for instance by adding a type parameter

data Doc (enc :: RenderMode) = ...

Here, RenderMode is promoted to a kind. This needs the DataKinds
extension which is available since ghc-7.6.x.

I have moved the milestone to 3.0.0 because I do not think we have a
definite solution yet.

It works with `export DARCS_DONT_ESCAPE_8BIT=1`.

Perhaps we could display *always* UTF-8 as UTF-8 except otherwise is
noted (perhaps putting some file in _darcs directory for explicit use of
encodings other than UTF-8).