Telnet checking for open port

[siki85]

Hi guys,

I´m here to know more about using telnet for checking open ports...

There is a database that use port 1526 to receive connections from users....when the app works fine, if I try to do telnet x.x.x.x 1526 the cmd windows show the following

C:\Documents and Settings\USER> telnet x.x.x.x 1526
-
C:\Documents and Settings\USER>

As you can see, I can get inside the server through the port, and then it lets me to do enter to show me again the documents and settings path.

But, this application has been causing some problems, mainly connections where users get slow connections...when this happens if I try to do the same check

C:\Documents and Settings\USER> telnet x.x.x.x 1526
-
-
-
-
-
-
C:\Documents and Settings\USER>

So as you can see, it seems that the telnet check hangs for a moment...I´m asking about this problem because I couldn´t figure out what´s going on with this problem. I already check the network, check the server for viruses....without much luck.

Thanks for your help and time...any advice would be great mates....

 

[DrB0b]

Does this happen both in office and out of office? Are your clients using a VPN to tunnel in if from the outside? Is there a third party firewall active, new AV recently or updates to an old AV since the issue came about? Windows Firewall active and or any Windows Updates since the issue? What is the OS of the server and the general OS of the clients? Work better on some than others?

Learning - A never ending quest for knowledge usually attained by being thrown in a situation and told to fix it NOW.

 

[siki85]

Hi my friend,

- The OS is Linux Suse 11, the database is informix 11. The problem affects all users (same LAN and remote branches). I have different VLANs for servers and clients, and there are routes to direct traffic between them. There are not ACL configured.

- The server has the port open through the linux firewall. We have an applicance as firewall too, but it is located between the LAN and WAN and it manages packets routed to internet, not for internal usage.

- Users run windowx xp. All of them have the windows firewall disabled.

The problem appears when the informix reach a specific concurrent connections. As far we can see, it seems that the database hangs till 102 concurrent connections, after that it works slowly, there are moments when it is fast again, and so it. We also check the server processes using top command, the application use oninit as the process but it doesn´t consume much cpu resources.

I installed wireshark in the server, sniff some packets but I haven´t seen anything weird. Everything looks normal, users trying to connect through port 1526, there are no broadcast storm, just normal packages from DNS server.

We have another server as backup running in a different branch (both server were configured to work using replication data) and I don´t have the problem there. But, it is in a different branch.

I suspect it could be the LAN, but as I wrote above, I can´t figure out what is causing it...

Thanks for your time my friend....

 

[siki85]

The telnet connection I did it from vlan 192.168.1.x (clients) and 10.102.2.x (servers). There is a L3 switch which has to manage the traffic, it routes traffic between 2 subnets.

 

[DrB0b]

Does the server house any shared drives/folders or anything else to test the general network traffic to and from to see if it could be as simple as a failing LAN card/onboard LAN port? Does a ping drop any responses to either the internet or a local IP? Tracerouter show for an internal IP correctly and without issue from the server?

Learning - A never ending quest for knowledge usually attained by being thrown in a situation and told to fix it NOW.

 

[DrB0b]

I know you know I messed this up, but for other readers who may be knowledgeable, the command is "traceroute" not tracerouter. Really wish we could edit our responses.....

Learning - A never ending quest for knowledge usually attained by being thrown in a situation and told to fix it NOW.

 

[siki85]

Well, the server is PER710 from DELL. It has 5 disks, each of 1 TB.

When the problem appears, I have an extended ping and I don´t see lost responses, I receive answers of 1ms without drops.

A traceroute show me the following (I´m doing it from my PC):
C:\Documents and Settings\USER>tracert 10.102.2.30 (server)

Traza a la dirección 10.102.2.30
sobre un máximo de 30 saltos:

1 2 ms 12 ms 17 ms 192.168.1.1 (L3 switch)
2 2 ms <1 ms <1 ms 10.102.2.30

Attached you could see a quick description of our network.

 

[siki85]

Sorry, I have problems trying to upload the image...

 

[siki85]

Well...I guess I´ll have to draw it here...

INTERNET -- CARRIER L2 SWITCH--- FIREWALL ---- L3 SWITCH ------ L2 SWITCH ---- SERVERS (10.102.2.30)
| | |
|--- CARRIER ROUTER -------| | (TRUNK)
(BRANCHES) |
| L2 SWITCH (PCs 192.168.1.x)
|
|
BRANCH A , BRANCH B, BRANCH C

Sorry, I don´t know how to upload an image....

 

[siki85]

Now I figure it out....

Here is the image...

 

[DrB0b]

How long has this problem been an issue? If just recently, has anything changed that you can think of? New Computer/OS/server implementations, new network hardware, firmware/route upgrades, etc?

Learning - A never ending quest for knowledge usually attained by being thrown in a situation and told to fix it NOW.

 

[siki85]

We haven´t done new changes in the network, this has been working fine until last Saturday 9 when the problem began. Suddenly, users started to notify about the problem. We haven´t installed new switches or routers, nothing.

Also, we revised the linux system logs, but they don´t show anything weird. The errors we have are from informix, which are:

- listener-thread: err = -408: oserr = 0: errstr = : Invalid message type received from the sqlexec process.

- listener-thread: err = -25582: oserr = 0: errstr = : Network connection is broken.

- ASF Echo-Thread Server: asfcode = -25580: oserr = 32: errstr = : System error occurred in network function.

All of them have different solutions pointing to the network.

Thanks in advanced...

 

[DrB0b]

Have you restarted the network services and/or informix since the issue? Checked the logs after a successful restart of the services and program?

Learning - A never ending quest for knowledge usually attained by being thrown in a situation and told to fix it NOW.

 

[siki85]

Yes.

We have restarted the server like three times since the problem arrived.

We look into the logs and it shows that the system and services were succesfully restarted.

I also look for interface errors, I haven´t seen collisions...

Is it possible that the informix was corrupted or something ?
If you try to use the system at night, it works perfectly. You can telnet the port and it responds very fast. At the beginning, we thought it was the quantity of concurrent connections, but we started almost 200 connections at night and the system works...the problem began when users connect in the morning...

 

[VinceWhirlwind]

Could the concurrent connections be a problem with the firewall, rather than the server?

 

[DrB0b]

If it is time related, it sounds like an internal networking device error to me. If you can have 200 connections at night work fine but 200 in the day time makes it slow, that would infer to me that one of your network devices is packed full of other, non-informix connections, are either taking precedence or there is so much other network traffic it is being bogged down. Have you attempted any switch/router restarts or jumped into any logs to see usage statistics day vs night?

Learning - A never ending quest for knowledge usually attained by being thrown in a situation and told to fix it NOW.

 

[siki85]

At the beginning I thought it was the NIC too. We used another NIC port in the server (it has 4 ports), but the problem doesn´t disappeared.

I also checked the network history using gnome monitor, but traffic doesn´t reach 100 Mbps, it remains constant (100 kbps, 200 kbps...).

 

[DrB0b]

I dont mean traffic on just the server. If a switch is getting bombarded throughout the day by employees "working"(see surfing the net and downloading), the bottleneck could be one of your switches or possibly a soon to be failing one.

Learning - A never ending quest for knowledge usually attained by being thrown in a situation and told to fix it NOW.

 

[siki85]

We run wireshark in the server, I don´t see anything as broadcast or something. Most packets come from users connecting to the server through port 1526.

The only thing I see in wireshark are packets from the server with checksum errors. For this case, we disabled the checksum offload in the NIC, but it doesn´t work.