problem fetching long documents through URL class

[developerinlondon]

I am using the URL class to fetch a long document from a remote machine, but it seems my program automatically disconnects after reading for about 1 minute 10 seconds. My code is something like the following :

         try {
            // Send data
            URL url = new URL(this.url);            
            URLConnection conn = url.openConnection();
            conn.setDoOutput(true);
            logger.debug("timeout is "+conn.getReadTimeout());  // this returns 0, so no timeouts according to API

            OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
            wr.write(this.xml_out);
            wr.flush();
              
            // Get the response
            BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
            //conn.
            String line;
            while ((line = rd.readLine()) != null) {
                this.xml_in += line + "\n";
            }
            wr.close();
            rd.close();
            
            return true;
        } catch (Exception e) {
            // todo : report error
            logger.error(e.getMessage());
            return false;
        }

any fix?

 

[sedj]

Instead of reading the document line by line, have you tried reading in chunks (maybe 4 k a time or something) using a DataInputStream ?

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

good idea.

i tried the following, but seems I am missing something obvious here as I am not getting any data:

         try {
            // Send data
            URL url = new URL(this.url);            
            URLConnection conn = url.openConnection();
            conn.setDoOutput(true);
            logger.debug("timeout is "+conn.getReadTimeout());

            OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
            wr.write(this.xml_out);
            wr.flush();
              
            // Get the response
            DataInputStream rd = new DataInputStream(conn.getInputStream());
       
            byte[] b = null;
            while (rd.read(b, 0, 4000) > 0) {
                this.xml_in += b.toString();
            }
            wr.close();
            rd.close();
            
            return true;
        } catch (Exception e) {
            // todo : report error
            logger.error(e.getMessage());
            return false;
        }
    }

 

[sedj]

A few errors :

- You must initialize the byte array before use.
- You should create a new String from the read byte array rather than doing toString()
- You should use a StringBuffer as it is far more optimized for concatenating strings than the '+' operator.

         try {
            // Send data
            URL url = new URL(this.url);            
            URLConnection conn = url.openConnection();
            conn.setDoOutput(true);
            logger.debug("timeout is "+conn.getReadTimeout());

            OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
            wr.write(this.xml_out);
            wr.flush();
              
            // Get the response
            DataInputStream rd = new DataInputStream(conn.getInputStream());
       
            byte[] b = new byte[4096];
            StringBuffer sb = new StringBuffer(4096);
            while (rd.read(b, 0, 4096) > 0) {
                sb.append(new String(b));
            }
            this.xml_in = sb.toString();
            wr.close();
            rd.close();
            
            return true;
        } catch (Exception e) {
            // todo : report error
            logger.error(e.getMessage());
            return false;
        }
    }

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

thanks for the corrections.
but i still seem to be getting some strange behaviours.
1) i cant see anything coming in on xml_in variable. when i put a log inside the while loop i can see all the data coming in.
2) the data that comes in the logs are sometimes broken, or badly placed. eg :
<FlightCode>TFSEZB</FlightCode>
<Route>LGWTFS</Route>
<DepartureDate>05.04.05</DepartureDate>
<DepartureTime>06:00</DepartureTime>
<ArrivalDate>05.04.05</ArrArrivalDate>
<ArrivalTime>
[2005-03-21 08:30:14,078] DEBUG 157688 [123_tel_avail] ntools.Jurl - string is :ivalDate>
<ArrivalTime>1ghtNumber>BA6996</FlightNumber>
<Available>A</Available>
</CharterFlight>
<CharterFlight>
it seems the <ArrivalDate> tag is repeating itself.

The frequency of this malformatted data was reduced after i changed the buffer size to 1024.
I was not sure if this was because you set the sb size also to 4096, so i tried using the xml_in.concat method, but this was no use either.

any ideas how I can get the data to show and to come in proper format?

 

[sedj]

Try this instead :

byte[] buffer = new byte[4096];
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int i = -1;

while (true) {
	i = rd.read(buffer, 0, buffer.length);
	if (i == -1) break;
	baos.write(buffer, 0, i);
}

xml_in = new String(baos.toByteArray());
baos.close();

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

ok this works better now!

however it seems there is also a bottleneck when i am sending an update query to the database, it seems to only take a certain amount of data from the xml_in variable no matter how big it is.
heres a snippet of the query:

            PreparedStatement stmt = conn.prepareStatement("update session_data set xml_in = '?' where sessionid = '?' and source = '?'");
            stmt.setEscapeProcessing(true);
           
            stmt.setString(1, xml_in);
            stmt.setString(2,sessionID);
            stmt.setString(3, source);
            stmt.execute();

I tried using the setBlob method but I am not sure how I can convert a String object to a Blob object and whether that will be of any use.

 

[developerinlondon]

actually it doesnt seem to do anything with the parameterized query. It was updating with a concatenated xml_in value when I was using :

            PreparedStatement stmt = conn.prepareStatement("update session_data set xml_in = '"+xml_in.replace('\'', '"')+"' where sessionid = '"+this.sessionID+"' and source = '" + this.source+"'");

 

[developerinlondon]

ok got it working now. we need longtext field in mysql to store these values. But still its not working on the parameterized query. any ideas?

 

[sedj]

Any errors ?
You should use a clob field for the xml_in field BTW.

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

nope no errors. thats what i dont like about the apache dbcp, i still havent figured out a way to log what goes on behind the scene.

so you mean to use Clob instead of String type? MySQL doesnt have any CLOB type as far as I know? whats the benefit?

 

[sedj]

Perhaps if you think DBCP is so bad, you should look into using a different pool ?!

I can assure you MySQL does have clobs (binary varchar in mysql).

http://www.mysqlfreaks.com/mysql/manual_Reference.html#BLOB

--------------------------------------------------
Free Database Connection Pooling Software

http://www.primrose.org.uk

 

[developerinlondon]

ok i got an error. it says

"Parameter index out of range (1 > number of parameters, which is 0)"
it seems not to be able to see the "?"s.

 

[developerinlondon]

ok got the fix also. we dont need the single quotes (') beside the question mark (?).

 

[timw]

You don't quote the ? in a preparedStatement.
Try

PreparedStatement stmt = conn.prepareStatement("update session_data set xml_in =? where sessionid=? and source=?");

Tim