Page MenuHomePhabricator

Links mixing latin1 literal characters and entities produce bad UTF-8
Closed, ResolvedPublic

Description

Author: timwi

Description:
BUG MIGRATED FROM SOURCEFORGE
http://sourceforge.net/tracker/index.php?func=detail&aid=978668&group_id=34373&atid=411192
Originally submitted by Ilguiz Latypov (ilgiz) 2004-06-24 04:29

This bug description may require changing the browser's
Character Encoding to UTF-8 in order to see Unicode
characters.

The Unicode link at

http://en.wikipedia.org/wiki/Tatar_language

was entered as

[[:tt:]

It is displayed correctly as:

tt:

and the resulting link is correctly encoded in UTF-8:

http://tt.wikipedia.org/wiki/%C5%9E%FCr%E4le

However, on clicking the link, I am redirected to a
different URL

http://tt.wikipedia.org/wiki/%C3%85%C5%BE%C3%BCr%C3%A4le

This URL rewriting happens on the server, as the
following log shows:

ilgiz@ei:~$ telnet tt.wikipedia.org 80
Trying 207.142.131.248...
Connected to tt.wikipedia.org.
Escape character is '^]'.
GET /wiki/%C5%9E%FCr%E4le HTTP/1.1
Host: tt.wikipedia.org

HTTP/1.0 301 Moved Permanently
Date: Thu, 24 Jun 2004 02:23:29 GMT
Server: Apache/1.3.29 (Unix) PHP/4.3.4
X-Powered-By: PHP/4.3.4
Vary: Accept-Encoding,Cookie
Expires: -1
Cache-Control: private, s-maxage=0, max-age=0,
must-revalidate
Last-Modified: Thu, 24 Jun 2004 02:23:29 GMT
Location:
http://tt.wikipedia.org/wiki/%C3%85%C5%BE%C3%BCr%C3%A4le
Content-Type: text/html
X-Cache: MISS from wikipedia.org
Connection: close

It seems that the URL rewriting attempts to encode the
link names from a supposed character set into another
one. Because the original link was UTF-8 encoded, the
additional translation corrupted the link.

  • Additional comments ------------------------

Date: 2004-06-24 07:40
Sender: SF user vibber

The following link is corrupt:
http://tt.wikipedia.org/wiki/%C5%9E%FCr%E4le

The is encoded according to UTF-8, but the literal haracters are encoded according to 8-bit ISO 8859-1. Thus, when
received the wiki sees that the incoming URL is *not* valid UTF-8
and
tries to upconvert it from ISO 8859-1. This correctly converts
the \x15ut of course the is further corrupted.

To workaround the inconsistency, use numerical references or
hard-
coded % codes in the link.


Date: 2004-06-24 14:32
Sender: SF user ilgiz

Thanks for getting into the root cause. I've changed the
rest of the link into numeric HTML entity characters, and it
worked.

I now realize the server should have treated all the
characters I entered in the edit field as Unicode only.
This would be possible if the ISO-8859-1 character coding of
the :en: pages was replaced with UTF-8.

In trying to do the best, the server recognized that the
first Unicode character U+015E in the POST data couldn't be
kept in the page's ISO-8859-1 code page and correctly
presented it as an HTML entity &350;. However, the rest of
the Unicode characters in the input field (U+00FC, ü
and U+00E4, ä) could be represented by the ISO-8859-1
characters, and the server returned their ISO-8859-1
encoding in the saved page text, in agreement with the
page's character coding.

In fact, it could be the Firefox browser's problem triggered
by the presence of characters outside the page's character
coding in the form of numeric HTML entities. The browser
should have correctly encoded the text of link into UTF-8,
as you mentioned, regardless of the page's character
encoding. When I move the mouse over the link, the browser
incorrectly shows the first character as 2 ISO-8859-1
characters (probably derived from the UTF presentation).
Besides, the browser keeps the ISO-8859-1 encoding even when
I click the link.


Date: 2004-06-24 18:40
Sender: SF user vibber

Firefox has nothing to do with it; the link is inherently wrong
and would
work under no circumstances.


Version: unspecified
Severity: major

Details

Reference
bz65

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 21 2014, 6:41 PM
bzimport added a project: MediaWiki-Parser.
bzimport set Reference to bz65.

timwi wrote:

*** Bug 67 has been marked as a duplicate of this bug. ***

timwi wrote:

*** Bug 66 has been marked as a duplicate of this bug. ***

ilatypov wrote:

Thank you again for pointing out that the original link I used for testing

http://tt.wikipedia.org/wiki/%C5%9E%FCr%E4le

was internally inconsistent. I agree that replacing all non-ASCII characters in
the editable links is the way to avoid confusion.
And you are right, there wasn't a problem with firefox. The wikipedia server
doesn't encode href attributes correctly.

The above inconsistent link was produced by the wikipedia server from the valid
input

Tatar myths, including the story of [[:tt:Şüräle]]

in the page by the address

http://en.wikipedia.org/wiki/Tatar_language

Here is what the page source would look like when typing/pasting the non-ISO
link in the edit form literally:

Tatar myths,  including the story of <a

href="http://tt.wikipedia.org/wiki/%C5%9E%FCr%E4le" class='extiw'

title="tt:Å&#382;üräle">tt:&#350;üräle</a></p>

If I encode the link manually as you suggest,

Tatar myths, including the story of [[:tt:&#350;&#252;r&#228;le]]

then the href attribute produced by the wikipedia server becomes correct:

Tatar myths,  including the story of <a

href="http://tt.wikipedia.org/wiki/%C5%9E%C3%BCr%C3%A4le" class='extiw'

title="tt:">tt:&#350;üräle</a>

In both cases the text node of the <a> tag (the word in front of </a>) is
generated correctly.

I still suspect that having to enter non-ASCII wiki names using the "HTML
entity" &#NUMBER; or the "UTF-8/URL" %HEXBYTE encoding might not be convenient.
People who create new links may not know about the existing problem and enter
the links literally.

A note on HTML 4.0 suggests that all non-ASCII href characters should be
represented the "UTF-8/URL" way in the HTML page regardless of the character set
of the page:

http://www.w3.org/TR/PR-html40-971107/appendix/notes.html#h-B.1

Interestingly, the tt.wikipedia.org server seems to convert any non-ASCII
character in href according to the w3 recommendation. This could be related to
the fact that the tt server produces the UTF-8 pages while the en server
produces ISO-8859-1 pages. So the quick fix might be as easy as changing the
default encoding of the en server to UTF-8. If that solution is impossible due
to some other concerns, the href encoding of the en server needs to be fixed.

jeluf wrote:

Changing en to UTF-8 is not as trivial as it sounds.
It would require converting the entire database.
This will probably happen in the future, but not in the next days.

And I agree, it's probably the only sane way to fix this bug.
Always creating interwiki links as UTF-8 would break links to
the few remaining latin-1 wikis. For the time being, please use
the workaround.

Creating interwiki links as UTF-8 should work fine with the Latin-1 wikis; they'll detect the encoding in the
URL and convert it. However this means knowing what you're dealing with before trying to interpret the
character references.

  • Bug 555 has been marked as a duplicate of this bug. ***
  • Bug 909 has been marked as a duplicate of this bug. ***

Changed summary to explain problem

  • Bug 1554 has been marked as a duplicate of this bug. ***

ilatypov wrote:

Changed the summary from

Links mixing latin1 literal characters and entities produce bad UTF-8

to
Typing in a non-latin1 interwiki link produces a wrong hex UTF-8 URL in "a href"

  • Bug 1591 has been marked as a duplicate of this bug. ***
  • Bug 1597 has been marked as a duplicate of this bug. ***

This is getting reported a lot and it's starting to piss me off. ;)

Raising priority.

gangleri wrote:

Thanks Brion!

http://en.wikipedia.org/wiki/User:Gangleri/tests/bugzilla:1691 gives more
examples and variations for the two characters ú and š.

Regards Reinhardt

I've managed to whip up some code to normalize 'mixed' interwiki links to UTF-8,
which fixes nearly all of ganglieri's test cases.

I'll make a couple more tweaks in the morning and check it in.

Checked in and put live. Seems to be working.

A couple notes re: Ganglieri's test cases...

  • The w:ro links are broken due to bug 563.
  • NEVER use &#154; or &#x9A; for s-caron. Numeric character references always

refer to Unicode code points, and U+009A is a reserved control character, *not* s-
caron. It might appear to work sometimes due to a fluke and crappy workarounds
for compatibility with a Windows bug, but should definitely not be relied upon. Use
the real Unicode number, &#353;. The same goes for the other characters in the
Windows CP1252 extended range (see http://en.wikipedia.org/wiki/ISO_8859
-1#Windows-1252 )

  • For the moment the only named character references that will work in links are the

ISO 8859-1 ones (s-caron does not appear in ISO 8859-1). Stick with the numbers
for now.

Someone removed all the CCs on this bug and put in an apparently unrelated page as the sample URL without explanation. Removing.

gangleri wrote:

Hi Brion!

Sorry for reopening this bug. You managed to solve 80%.

See: http://jadesukka.homelinux.org:8180/betawiki/Bugzilla_0065
and http://test.leuksman.com/index.php/Bugzilla_0065

THERE links as [[wikipedia:bg:Бела Барток]] will FAIL.
THERE links as [[wikipedia:sr:User:Горан Анђелковић]] are OK ( User is used )
but [[wikipedia:sr:Корисник:Горан Анђелковић]] will FAIL.


Please remember that at en.wikipedia links as [[w:he:א]] will fail because this
is RTL.

For your help: At the pages mentioned above:

THERE links as [[wikipedia:he:User:גראנ]] are OK ( User is used ) but
THERE links as wikipedia:he:משתמש:גראנ]] will FAIL.


Please see also:
http://jadesukka.homelinux.org:8180/betawiki/Bugzilla_unknown_02
http://test.leuksman.com/index.php/Bugzilla_unknown_02
[[en:User:Gangleri/tests/sister projects]]

where two problems are described:

  • <nowiki>[[wikibooks:]]</nowiki> generates
    • [[wikibooks:]]
    • this looks like: '''<span

class="plainlinks">http://en.wikibooks.org/wiki/</span>"
title="wikibooks:">wikibooks:'''

    • [[wikibooks:Main Page]] is '''OK'''
  • <nowiki>[[rfc:]]</nowiki> generates
    • [[rfc:]]
    • this looks like: '''<span

class="plainlinks">http://www.rfc-editor.org/rfc/rfc.txt</span>" title="rfc:">rfc'''


At the described pages the following link will fail:

[[wikipedia:da:]]
[[wikipedia:de:]]
[[wikipedia:eo:]]
[[wikipedia:es:]]
[[wikipedia:ar:]]
[[wikipedia:he:]]

It seems to be the same problem as experienced at
[[en:User:Gangleri/tests/sister projects]] where

[[w:]], [[b:]], [[n:]] and [[q:]] are OK but
[[w:en:]], [[b:en:]], [[n:en:]] and [[q:en:]] will FAIL.

Thanks for your patience an d fixing these bugs.

Best regards Reinhardt [[user:gangleri]]

Right-to-leftness probably has nothing at all to do with anything, since that's a
property of text *DISPLAY* and is far, far, far, far outside anything that can
affect this.

Most likely a character validity check is tripping on conflicting bytes again.

gangleri wrote:

[[wikipedia:bg:Бела Барток]] still fails

changed Severity to "major" - Regards Reinhardt

Comments after reopening appear unrelated to this bug, which by definition can only occur on
Latin-1 wikis. Re-closing; please file a separate bug report.

gangleri wrote:

parts of comment 19 where reported as individual bugs
bug 2342 - w:he:(aleph) is a valid interwiki link at en: but not at other wiki's
bug 2372 - [[interwiki_foo:]] will generate wrong code and render incorrect