Apache error log - DNS lookup failure for: hostname

[yarafatin]

I am in the midst of performance testing a web application and I get these errors in Apache logs

[Sat Apr 18 03:01:01 2015] [error] [client xx.xx.xxx.253] proxy: DNS lookup failure for: jboss-host-name returned by /some/url, referer:

http://f5hostname/context/URI

It occurs only at high load levels which indicates that it may not be the configuration that is causing it. I have seen posts where this error occurs due to incorrect proxy configuration, but my case doesn't fall under that I guess. At the very end of the test I get another error - "All workers are in error state". I think the first DNS error causes all workers to shutdown.

Can someone point where I could start looking? Please let me know if you need more details than whats mentioned below.

More details:
Apache version: 2.2
mod_cluster: 1.2.4.Final

I have an F5 in front of two Apache servers. There are ten JBoss servers in the balancer.

Apache configuration snippet: (I am not using proxypass, just modcluster with a proxylist on Jboss)

<IfModule mod_proxy.c>
    ProxyRequests Off
    ProxyVia On
</IfModule>

<IfModule mpm_worker_module>
    ServerLimit, StartServers and worker related attributes configured here
</IfModule>

ProxyPreserveHost On
MemManagerFile /var/cache/httpd

<VirtualHost *:1280>
    ManagerBalancerName some2
    ........

Jboss config:

<subsystem xmlns="urn:jboss:domain:modcluster:1.1">
    <mod-cluster-config advertise-socket="modcluster" proxy-list="host:1280,host-2:1280" balancer="some2" advertise="false" excluded-contexts="invoker,jbossws,juddi,console" connector="http">
        <dynamic-load-provider>
            <load-metric type="busyness"/>
        </dynamic-load-provider>
    </mod-cluster-config>
</subsystem>

 

[ChrisHirst]

if "jboss-host-name" is the hostname for your installation, a reverse DNS lookup is ALWAYS going to fail because it is not a Fully Qualified Domain Name (FQDN).

Other than that set HostnameLookups Off in Apache's configuration to stop reverse DNS lookups.

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

 

[yarafatin]

jboss-host-name is not the actual hostname, it is the hostname with FQDN. I had to shield the client's internal server names on external posts.

Jboss is started with -b jboss-host-name, so mod_cluster publishes the hostname to httpd and establishes its availability. Reverse lookup is not applicable in this situation and I dont understand how that could cause the DNS lookup errors at high load volume. The system works fine at low load levels. Did I misread your message?

 

[yarafatin]

In fact, HostnameLookups is already set to Off.

 

[ChrisHirst]

Okay my knowledge of JBoss is limited to ... well, .... nothing at all, but this;

I dont understand how that could cause the DNS lookup errors at high load volume. The system works fine at low load levels.

may be indicative of some latency or loading 'issue' with whatever a F5 may happen to be.

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

 

[yarafatin]

Thank you for the input Chris.

It has to do with either Apache, network, tcp or the Jboss servers. The issue occurs when apache is communicating to/from Jboss.
I looked up Apache code that throws this error and found
if (err != APR_SUCCESS) {
return ap_proxyerror(r, HTTP_BAD_GATEWAY,
apr_pstrcat(p, "DNS lookup failure for: ",
conn->hostname, NULL));
}

Variable err is value from the function call - apr_sockaddr_info_get found in sockaddr.c which has a bunch of logic and if anything goes wrong there, it could be misinterpreted as DNS lookup failure.