jeluf wrote:

Same in French wiki at
http://fr.wikipedia.org/wiki/Image:Lieutenant-colonel_des_armes_a%CC%80_cheval.png
Note the "a%CC%80"

I have dug up some mor info on this:

The crucial point is that *some* canonisation (normal form) should be used as
internal representation. For compatibility reasons, this should probably be a
composed form, as the decomposed forms are rendered badly on some systems. Here
is the official document about unicode normal forms:

http://www.unicode.org/reports/tr15/

HTH

jeluf wrote:

This has to be done at least for:

• User names
• File names
• Page titles

and should be also done for wikitext, at least in the searchindex.

Bug seems to be specific to Safari. Firefox 0.9.1 and IE 5.2.3 both normalize the name to the precomposed form.

Uploaded from Safari:
http://meta.wikimedia.org/wiki/Image:Wiki_test_e%CC%81.png

Uploaded from Firefox and IE:
http://meta.wikimedia.org/wiki/Image:Wiki_test_%C3%A9.png

Nonetheless we certainly should be normalizing input... Check if there's an iconv or mb_* function for doing this efficiently.

I spent a few minutes googling and came up with nothing useful pre-existing in PHP. Guess I'll have to write another hack. :P
All the necessary data should be in the Unicode data tables... It may be possible to write a DSO extension that makes use of
existing library functions (libidn seems to have UTF-8-based normalization functions for instance) but we'll need a 'native'
PHP version anyway for general distribution.

jeluf wrote:

libidn provides a stringprep_utf8_nfkc_normalize() function. The glyphs created
by this normalization differ from the input. e.g. ² becomes 2.
When using this for user names, would we want to preserve the original string
for displaying but use an internal representation for comparing?

There is a PHP-libidn binding at http://php-idn.bayour.com/ but it looks they do
not yet provide access to stringprep_utf8_nfkc_normalize().

jeluf wrote:

The ucdata library might be interesting, it provides both composition and
decomposition, upper case, etc.

http://crl.nmsu.edu/~mleisher/ucdata.html

The download page at that site is broken, there is rev 2.5 available at
ftp://crl.nmsu.edu/CLR/multiling/unicode/ucdata-2.5.tar.gz

A further note: in addition to being decomposed, Safari actually is sending the
filename with HTML character references: "Wiki test é.png"

Adding an accept-charset attribute to the <form> unfortunately doesn't seem to
change anything. Also, in current 1.4 cvs the # is now stripped to - before we
get to the point where we normalize the title and would be interpreting the
character, so things get even weirder.

Now fixed in 1.4 CVS.

Might consider backporting the isolated filename normalization part to 1.3 on account of the safari problem without
risking the general case; leaving this bug open for the moment.

1.4 nearing release; not backporting.