Character Encoding Issue

[plunkett]

I'm establishing a TCP connection to a remote chat server. Once connected, we exchange data back and forth in a typical fashion.

The server sends data UTF8 encoded. I read from a socket, and then convert the bytes into chars, then read them from a BufferedReader. That is then converted to a string and printed to a JTextArea.

The problem is this: When converting the bytes, if I do nothing (accept the default charset when converting), then nothing but the lower ASCII characters appear. If I convert the bytes using UTF-8 decoder, then many more characters show up correctly, but many others still look like boxes in my JTextArea.




So if I have no utf8 decoder and I read bytes and append them to a JTextArea, no special characters show up correctly (they look like unconverted UTF8 characters)...

in = new BufferedReader(new InputStreamReader(socket.getInputStream()));


If I have defined a utf8 decoder, when I append to JTextArea, many more characters (the extended ones) show up correctly now but others (the even more extended ones) now show up as boxes in the JTextArea.


Charset charset = Charset.forName("UTF-8");
CharsetDecoder myUTF8Decoder = charset.newDecoder();

in = new BufferedReader(new InputStreamReader(socket.getInputStream(), myUTF8Decoder));



So, am I decoding these wrong? Or is the JTextArea not setup to show these extended characters? Doesn't Java use unicode by default? Is it a locale problem?


Any help would be greatly appreciated. I've battled with this for days!

 

[prosper]

I have copied and modified the code from the following website

http://darksleep.com/player/SocketExample/

import java.net.Socket;
import java.net.ServerSocket;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.IOException;
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;

public class VerySimpleClient extends JFrame implements ActionListener{
    private String serverHostname = null;
    private int serverPort = 0;
    private byte[] data = null;
    private Socket sock = null;
    private InputStream sockInput = null;
    private OutputStream sockOutput = null;
    private JTextField jt = null;
    private JButton jb = null;
    public VerySimpleClient(String serverHostname, int serverPort){
        super();
        getContentPane().setLayout(new BorderLayout());
        jt = new JTextField();
        getContentPane().add(jt,BorderLayout.CENTER);
        jb = new JButton("Send Message");
        getContentPane().add(jb,BorderLayout.SOUTH);
        jb.addActionListener(this);
        this.serverHostname =  serverHostname;
        this.serverPort = serverPort;
        this.data = data;
    }
    
    public void sendSomeMessages(int iterations) {
        System.err.println("Opening connection to "+serverHostname+" port "+serverPort);
        try {
            sock = new Socket(serverHostname, serverPort);
            sockInput = sock.getInputStream();
            sockOutput = sock.getOutputStream();
        }
        catch (IOException e){
            e.printStackTrace(System.err);
            return;
        }

        System.err.println("About to start reading/writing to/from socket.");
        
        if (jt.getText()!=null && jt.getText().length()>0){
             data = jt.getText().getBytes();
        }
        else{
             data = "nothing type".getBytes();
        }

        byte[] buf = new byte[data.length];
        int bytes_read = 0;
        for(int loopi = 1; loopi <= 2; loopi++) {
            try {
                sockOutput.write(data, 0, data.length);        
                bytes_read = sockInput.read(buf, 0, buf.length);
            }
            catch (IOException e){
                e.printStackTrace(System.err);
            }
            if(bytes_read < data.length) {
                System.err.println("run: Sent "+data.length+" bytes, server should have sent them back, read "+bytes_read+" bytes, not the same number of bytes.");
            }
            else {
                System.err.println("Sent "+bytes_read+" bytes to server and received them back again, msg = "+(new String(data)));
            }

            // Sleep for a bit so the action doesn't happen to fast - this is purely for reasons of demonstration, and not required technically.
            try { Thread.sleep(50);} catch (Exception e) {}; 
        }
        System.err.println("Done reading/writing to/from socket, closing socket.");

        try {
            sock.close();
        }
        catch (IOException e){
            System.err.println("Exception closing socket.");
            e.printStackTrace(System.err);
        }
        System.err.println("Exiting.");
    }
    public void actionPerformed(ActionEvent e) {
           sendSomeMessages(100);
    }
    public static void main(String argv[]) {
        String hostname = "localhost";
        int port = 54321;
        
        VerySimpleClient client = new VerySimpleClient(hostname, port);
        client.setSize(400,400);
        client.setVisible(true);
        //client.sendSomeMessages(100);
    }
}
// Tue Nov  2 18:33:43 EST 2004
//
// Written by Sean R. Owens, released to the public
// domain.  Share and enjoy.  [URL unfurl="true"]http://darksleep.com/player[/URL]

import java.net.Socket;
import java.net.ServerSocket;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.IOException;

// An example of a very simple socket server.  Start by looking at the
// static main() method at the bottom of this file.
public class VerySimpleServer {
    private int serverPort = 0;
    private ServerSocket serverSock = null;

    public VerySimpleServer(int serverPort) {
        this.serverPort = serverPort;

         try {
            serverSock = new ServerSocket(serverPort);
         }
         catch (IOException e){
            e.printStackTrace(System.err);
         }
    }
    
    // All this method does is wait for some bytes from the
    // connection, read them, then write them back again, until the
    // socket is closed from the other side.
    public void handleConnection(InputStream sockInput, OutputStream sockOutput) {
        while(true) {
            byte[] buf=new byte[1024];
            int bytes_read = 0;
            try {
                // This call to read() will wait forever, until the
                // program on the other side either sends some data,
                // or closes the socket.
                bytes_read = sockInput.read(buf, 0, buf.length);

                // If the socket is closed, sockInput.read() will return -1.
                if(bytes_read < 0) {
                    System.err.println("Tried to read from socket, read() returned < 0,  Closing socket.");
                    return;
                }
                System.err.println("Received "+bytes_read
                                   +" bytes, sending them back to client, data="
                                   +(new String(buf, 0, bytes_read)));
                sockOutput.write(buf, 0, bytes_read);
                // This call to flush() is optional - we're saying go
                // ahead and send the data now instead of buffering
                // it.
                sockOutput.flush();
            }
            catch (Exception e){
                System.err.println("Exception reading from/writing to socket, e="+e);
                e.printStackTrace(System.err);
                return;
            }
        }

    }

    public void waitForConnections() {
        Socket sock = null;
        InputStream sockInput = null;
        OutputStream sockOutput = null;
        while (true) {
            try {
                // This method call, accept(), blocks and waits
                // (forever if necessary) until some other program
                // opens a socket connection to our server.  When some
                // other program opens a connection to our server,
                // accept() creates a new socket to represent that
                // connection and returns.
                sock = serverSock.accept();
                System.err.println("Have accepted new socket.");

                // From this point on, no new socket connections can
                // be made to our server until we call accept() again.

                sockInput = sock.getInputStream();
                sockOutput = sock.getOutputStream();
            }
            catch (IOException e){
                e.printStackTrace(System.err);
            }

            // Do something with the socket - read bytes from the
            // socket and write them back to the socket until the
            // other side closes the connection.
            handleConnection(sockInput, sockOutput);

            // Now we close the socket.
            try {
                System.err.println("Closing socket.");
                sock.close();
            }
            catch (Exception e){
                System.err.println("Exception while closing socket.");
                e.printStackTrace(System.err);
            }

            System.err.println("Finished with socket, waiting for next connection.");
        }
    }

    public static void main(String argv[]) {
        int port = 54321;
        VerySimpleServer server = new VerySimpleServer(port);
        server.waitForConnections();
    }
}

The original code of the author can handle Chinese character. I have modified his code and let user input words in a JTextField
in a JFrame. The GUI I created is ugly and it is used for showing that the original code can accept Chinese words and Japanese KanJi.
If the example do not work with your input, please tell us the problem.