I think it would be good for darcs to include better support for Unicode.

UTF-8 seems to work okay, mostly because it was designed in a clever way 
so as to be largely compatible with ASCII, or at least any system that 
supports an arbitrary high-byte range, for example, end of line markers 
are the same. (And the manual also has an environment setting you can 
use to get correct output.)

It would be nice to also have some minimal support for UTF-16, at the 
very least a way to override the automatic treatment as binary for files 
containing \0 (basic ASCII characters include \0, for example 'A' is 00 
41). I think there is already an item on the list to allow this 
automatic treatment to be overridden, which goes part of the way.

The other main issue would probably be detecting end of line for diffs. 
In some cases this may actually work by coincidence, with the last 0A 
(or 0D) being recognised, despite the proceeding 00 (depending on byte 
order). This isn't a complete solution, however, so proper end of line 
detection is needed.

Apart from detecting end of line, the rest of the system should be 
adequate -- once lines are correctly identified, a diff is any byte, 
whether or not a Unicode sequence, still makes them different. More 
advanced stuff, like identifying the start/end character of the 
difference within the line, or white space ignore, would require more work.

- Sly

Attachments

• stephen_gryphon.vcf

Tuomo has made some comments on the darcs-devel thread:

http://www.abridgegame.org/pipermail/darcs-devel/2006-September/004775.html

No comment myself (yet)

We won't add support for utf-16 to darcs' hunk patches.  Text patches to a
utf-16 file would need to be a new patch type.  I wouldn't object to someone
writing such a patch type, but I doubt it'll happen, so I'm marking this wish as
wont-fix.

David