Detect Asian Characters 2

[sen5241b]

The header may not be reliable to use for this purpose. Is there a way to detect Asian characters like is these websites:

http://www.centamap.com/gc/home.aspx?ref=alias&ck=gbase&alias=centamap2.asp

http://www.smugmug.com/gallery/6426839_nndvr#407123091_bd2xo

 

[feherke]

Hi

The first document is delivered with this HTTP header :

[tt]Content-Type: text/html; charset=big5[/tt]

The Big-5 character set makes clear that is Asian text.

The second document is delivered with this HTTP header :

[tt]Content-Type: text/html; charset=utf-8[/tt]

The CJK ideographs contained by UTF-8 characters set have character codes in the range 4E00-9FFF. You will have to check for the existence of such character in the document.

Or I misunderstand your problem.

Feherke.

http://rootshell.be/~feherke/

 

[sen5241b]

You do understand my problem. Language headers are not always reliable. So I would check if the page is UTF8 and search for characters in the range 4E00-9FFF? Also, what does 'CJK' stand for?

 

[feherke]

Hi

At least according to List of Unicode characters article on Wikipedia, that is the subset of characters you are interested in.

Also, what does 'CJK' stand for?

http://abbreviations.com/cjk[/URL]]Chinese Japanese Korean

Feherke.

http://rootshell.be/~feherke/

 

[tsuji]

[1] To guess the character encoding of a page is no small business. The heritage of mozilla having been open-sourced has a well-deserved renowned piece of code for it chardet.dll by Frank Y F Tang. It is ported to different implementations (java, python,...?).

[2] You can check out the jchardet, java plotting, source-forge project, free download.

http://jchardet.sourceforge.net/

[3] With jchardet, the "guess" for you will be done by running a command line like this with properly located in the current directory.
[tt] java -classpath chardet.jar org.mozilla.intl.chardet.HtmlCharsetDetector [ignore]

http://www.centamap.com/gc/home.aspx?ref=alias&ck=gbase&alias=centamap2.asp

[/ignore][/tt]

 

[sen5241b]

The Vietnamese alphabets are listed in several noncontiguous Unicode ranges:

Basic Latin {U+0000..U+007F}
Latin-1 Supplement {U+0080..U+00FF}
Latin Extended-A, -B {U+0100..U+024F}
Latin Extended Additional {U+1E00..U+1EFF}
Combining Diacritical Marks {U+0300..U+036F}

Is there some simple way, short of regexs, to search for these unicode chars.?

 

[tsuji]

The regex would not be the appropriate tool for the search. The search must be conducted at the binary level, not at php string level which has already supposed some decoding scheme.

 

[sen5241b]

Any advice on how to do a binary search of an HTML page?

 

[sen5241b]

Ok after a ridiculous amount of time searching the web I found this:

http://www.developertutorials.com/pear-manual/package.text.text-languagedetect.html

I'm going to start experimenting with it. Its alpha. I'll let you all know if it works.

 

[sen5241b]

Doesn't seem to work so well.

http://languagedetect.org/