problem with HTTPUrlConnection using SOAP

[developerinlondon]

I am using HTTPURLConnection object to fetch documents, however when getting SOAP documents they reply a HTTP Header 500 when there is an error message inside the document, this is correct according to the SOAP specification. However I am unable to get the contents of the document, instead Java throws an exception saying there was error 500, and we cant seem to get the actual content of the reply.

Below is the code thats being tried -

         /**************************************************************************************
            * now open a url connection, it automatically turns into httpsurlconnection if needed
            **************************************************************************************/
             HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            conn = null;
            //if (this.url.startsWith("https")) {
            if ((this.url.startsWith("https")) && (this.url.lastIndexOf(new String("bcp"))<1)) {
                conn = (HttpsURLConnection) url.openConnection();
                logger.info("Using custom security manager for secure connection.");
            } else {
                conn = (HttpURLConnection) url.openConnection();
            }


            /*********************************
             * add any custom headers passed *
             *********************************/
            add_headers(header_out, conn);

            /********************************
            * configure timeouts + method   *
            *********************************/
            conn.setDoOutput(true);
            conn.setConnectTimeout(connection_timeout); // maximum 30 seconds connection timeout.
            conn.setReadTimeout(read_timeout); // maximum 30 seconds read timeout.
            conn.setRequestMethod(this.request_method); // set the request method

            /********************************
            * send output (if passed)       *
            *********************************/
            // send output if there is any.
            OutputStreamWriter wr = null;
            if (xml_out != null) {
                wr = new OutputStreamWriter(conn.getOutputStream());
               // wr.write (this.header_out);
                wr.write (this.xml_out);
                wr.flush();
            }
            conn.connect();
            /***********************************
            * read the reply and put in xml_in *
            ************************************/
            logger.info("Reading Reply from "+this.url+".");
            InputStream in = conn.getInputStream();
// **** WE GET THE EXCEPTION THROWN AFTER THE ABOVE LINE! *** //
            // Get the response
            DataInputStream rd = new DataInputStream(in);
            byte[] buffer = new byte[4096];
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            int i = -1;
            while (true) {
                i = rd.read(buffer, 0, buffer.length);
                if (i == -1) break;
                baos.write(buffer, 0, i);
            }

            byte[] byteData = baos.toByteArray();

            xml_in = new String(byteData);

            baos.close();

            //logger.debug(xml_in);

            if (wr != null) {
                wr.close();
            }
            rd.close();

any thoughts on how we can get around this? We dont need a fullblown SOAP Parser, only the ability to read the content instead of just throwing an error 500.