Smart questions
Smart answers
Smart people
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Member Login

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips now!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

Join Tek-Tips
*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

LINK TO THIS FORUM!

Add Stickiness To Your Site By Linking To This Professionally Managed Technical Forum.
Just copy and paste the
code below into your site.

Partner With Us!

"Best Of Breed" Forums Add Stickiness To Your Site
Partner Button
(Download This Button Today!)

Feedback

"...love the site and am constantly recommending it to (selected !) clients here in ireland..."

Geography

Where in the world do Tek-Tips members come from?

perl: remove non-english sentence

diera (Programmer)
18 May 12 3:57
Hi,

I would like to filter all the sentence in my dataset which is not in English language. I already take a look to remove for non-ASCII character but sometime the character for some language are the same with English.

Do you know any Perl library or dictionary to filter non-english word.
any help is much appreciated.


-DATA-
11.10.2009 12.09 @ariel di sini kabar sihat saje
12.10.2009 17:47 start learning at mid night again. Did I stop learning? sure not. I'll learn until die.
12.10.2009 17:48 RT ggapj e tinha PiruliCópteros!! quem não teve?? #twittesuainfancia
12.10.2009 21:29 @anaelisa04 coooomédia meemo em? =D
13.10.2009 14:47 Reply w/Answer for Chance to Win a FREE Phoenix Bat: Who holds the MLB record for most home runs hit in opening day games?
13.10.2009 14:57 @alexforrestitv some treatment are good apparently in the region.Hope you will come back there to show something more cheerful? :)
13.10.2009 15:18 QUER #AUMENTAR SEUS #SEGUIDORES RÁPIDO? Siga o @MeusFollowers. Leia as instruções e em poucos minutos você terá centenas de seguidores!!

duncdude (Programmer)
19 May 12 9:37

CODE --> perl

#!/usr/bin/perl -w use strict; use warnings; my %english = (); open ( DICTIONARY, '<', '/usr/share/dict/words' ); while ( <DICTIONARY> ) { $_ = lc $_; print; chomp; $english{ $_ } = 1; } close DICTIONARY; while ( <DATA> ) { $_ = lc $_; my $finds = 0; my @split_up = split( /\s+/ ); foreach ( @split_up ) { $finds++ if exists $english{ $_ }; } print "<<< $finds find(s) >>>\t$_\n"; print "\t... more than half of the words were found in the dictionary\n\n" if $finds >= (( scalar @split_up ) / 2 ); } __DATA__ 11.10.2009 12.09 @ariel di sini kabar sihat saje 12.10.2009 17:47 start learning at mid night again. Did I stop learning? sure not. I'll learn until die. 12.10.2009 17:48 RT ggapj e tinha PiruliCópteros!! quem não teve?? #twittesuainfancia 12.10.2009 21:29 @anaelisa04 coooomédia meemo em? =D 13.10.2009 14:47 Reply w/Answer for Chance to Win a FREE Phoenix Bat: Who holds the MLB record for most home runs hit in opening day games? 13.10.2009 14:57 @alexforrestitv some treatment are good apparently in the region.Hope you will come back there to show something more cheerful? :) 13.10.2009 15:18 QUER #AUMENTAR SEUS #SEGUIDORES RÁPIDO? Siga o @MeusFollowers. Leia as instruções e em poucos minutos você terá centenas de seguidores!!

Kind Regards
Duncan

feherke (Programmer)
20 May 12 8:33
Hi

Nice idea, Duncan. But could you specify the package which provides that file ? Thank you.

( The systems I checked show quite different situations : Frugalware - no /usr/share/dict/; Debian - /usr/share/dict/words is broken symlink; Ubuntu - /usr/share/dict/ exists but empty. )

Feherke.
http://feherke.github.com/

diera (Programmer)
21 May 12 5:04
Hi,

thank you Duncan. As feherke's question, can you specify the package?

i have run the code, it seem the file not exist.

'Can't open dictionary: No such file or directory at C:....'

thank you very much.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Back To Forum

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close