IP/DNS Grab - More Efficent Method??

[chitownclone]

I am using the following methods to collect IP and server data for a large number (500k+) urls, which is then inserted into a MySQL database.

I need:
- IP
- DNS Server Name
- Server Info (Type and OS)

Is there a more efficient method to collect this data? Is it redundant to use 3 modules..ie can this data be collected with 1 or 2?

Any suggestions would be hugely appreciated.


-- Thanks

#     ######## IP PING ##########
$p = Net::Ping->new("syn");
$p->{port_num} = getservbyname("http", "tcp");
$p->ping($url);
($host,$rtt,$ip) = $p->ack; 
$p->close(); 
($ip1, $ip2, $ip3,$ip4) = split /\./, $ip, 4;


#     ######## DNS Name Grab ##########
$res = Net::DNS::Resolver->new;
$query = $res->query("$ip", "PTR");
if ($query) {
  foreach $rr (grep { $_->type eq 'PTR' } $query>answer) {
     $ptrname = $rr->ptrdname;
  }	
}


#     ######## Server Data Grab ##########
$robot = new LWP::UserAgent;
$robot->timeout (4);
$request  = new HTTP::Request GET => $url; 
$response = $robot->request( $request );
@data = $response->content . "\n";

if ($response->is_success) {
   $server=$response->server;
}

 

[icrf]

By "efficient" are you looking for shorter code or faster code? Bets are good if there's a module that does it all, it calls a lot of those other ones in the background for you.

________________________________________
Andrew

I work for a gift card company!

 

[mlibeson]

Be careful your IP Ping portion will not be accurate as there are now IPs with 5 sets of numbers, not just 4 anymore.


Michael Libeson

 

[icrf]

huh? There's the 128 bit hexadecimal IPv6 addresses, but I've not heard of anything breaking into five groups of numbers...these are computers, they love powers of 2.

________________________________________
Andrew

I work for a gift card company!

 

[mlibeson]

I have processed hundreds of web site logs and the 5 group of number IP addresses are out there.


Michael Libeson

 

[chitownclone]

'By "efficient" are you looking for shorter code or faster code?'

FASTER! The processing time current takes days. Most of the lag comes from contacting the DN servers, so I know its gonna be long regardless. The only way I could figure that I could decrease the 'crawl' time is to contact the server once/twice instead of three times.

As far as the IP Addresses.....is anyone currently using those with 5 sets? I have a copy of the ARIN IP Block list (which has 4 sets) so I am curious as to who is registering these new ones.

-- Thank you

 

[icrf]

I'm still going to have to guess it's a logging fluke. IPv4 and IPv6 are the only internet protocol addresses I've ever heard of being used, and neither is represented by 5-tuples.

I'll try and remember to poke around tonight on combining some of that. Off the top of my head, I'd be you can skip the first ping and get an IP address from LWP in the last step. I'm not sure you can get much more than that. You need to contact the DNS servers for its PTR record and the server itself for the host software, so two RTT's, one to each server, is a minumum.

________________________________________
Andrew

I work for a gift card company!

 

[azzazzello]

I would love to find out at least 1 of those 5-group IPs...and also let me know how I can ping it, or get to it...because if I put a 5-group "IP" in most applications' IP field, they will spit an error back at me