Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Incremental socket file descriptor
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Thu Jul 04, 2013 2:42 pm    Post subject: [SOLVED] Incremental socket file descriptor Reply with quote

Hello there,
I've developed a server that accept connections from clients. Connections are very fast and they can get disconnected many times (for network issues, 3g bad connections and so on).
My question is this:
After a long periond (about 10 days), if I list socket connection using lsof -i :server_port, I see the number of socket the system uses is always incremental:
Code:

Server  1778   395u 
Server  1778   398u 
Server  1778   400u 
Server  1778   406u 
Server  1778   408u 
Server  1778   409u 
Server  1778   410u 
Server  1778   411u 
Server  1778   412u 
Server  1778   413u 
Server  1778   416u 
Server  1778   417u 


These connections are always incremental (previous to 395 they never been used by system). On this machine there are other server (web and mysql) but number of connections they established surely are not 394 (more or less)
This causes that after 2 months my server gets "too many files opened" (because it reaches the maximum permitted files).
I'am using SO_REUSEADDR and tcp_tw_reuse but these don't impact on behaviour.
How can I tell system to rescue fd in the gap?

Thanks
Gianni
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/


Last edited by DevOne on Wed Jul 10, 2013 8:23 am; edited 1 time in total
Back to top
View user's profile Send private message
Pes88
Apprentice
Apprentice


Joined: 06 May 2009
Posts: 243

PostPosted: Thu Jul 04, 2013 4:17 pm    Post subject: Reply with quote

Hi,
To help us to understand the problem you should also post some relevant part of your code.

However, I think you are forgetting to release file descriptors when the connection is terminated. For example :
Code:

while(1) {
   sock = accept(list_sock ....);
   read/write(sock, ...);
  close (sock);
}

_________________
Pygoscelis papua, chiamato Gentoo dagli abitanti delle isole Falkland/Malvinas, noto per essere il pinguino più veloce!!! XD
Back to top
View user's profile Send private message
v_andal
Guru
Guru


Joined: 26 Aug 2008
Posts: 541
Location: Germany

PostPosted: Fri Jul 05, 2013 7:07 am    Post subject: Reply with quote

Run "netstat -a -n" and see if you have sockets in state CLOSE_WAIT, those will be your missing sockets. As Pes88 has pointed out, the highest possible reason is not closed file descriptor.
Back to top
View user's profile Send private message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Fri Jul 05, 2013 2:14 pm    Post subject: Reply with quote

Hello thank you for responding...there are no pending connection because I've log that always show current active connection managed by my server.
What you see in lsof command is what it prints on the log (I mean the number of socket opened).
netstat reveals me that the only connections are in ESTABLISHED mode. CLOSE_WAIT are shown as well, and then cleaned up by my server.

Due to I cannot track closed connection by clients, my server puts all accepted connections in a queue that steadly it is checked to find closed connections by clients, and then it calls close() on it
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/
Back to top
View user's profile Send private message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Sun Jul 07, 2013 9:57 am    Post subject: Reply with quote

Just doing a more detail 'lsof -p pid' of my server, I have discovered a bad thing:
Code:

Server  1778 ---  455u  sock                0,6      0t0 303826 can't identify protocol
Server  1778 ---  456u  sock                0,6      0t0 305411 can't identify protocol
Server  1778 ---  457u  sock                0,6      0t0 308059 can't identify protocol
Server  1778 ---  458u  sock                0,6      0t0 307429 can't identify protocol
Server  1778 ---  459u  sock                0,6      0t0 305627 can't identify protocol
Server  1778 ---  460u  sock                0,6      0t0 305634 can't identify protocol
Server  1778 ---  461u  sock                0,6      0t0 307270 can't identify protocol
Server  1778 ---  462u  sock                0,6      0t0 308569 can't identify protocol
Server  1778 ---  463u  sock                0,6      0t0 307833 can't identify protocol
Server  1778 ---  464u  sock                0,6      0t0 307343 can't identify protocol
Server  1778 ---  465u  sock                0,6      0t0 308047 can't identify protocol
Server  1778 ---  466u  sock                0,6      0t0 308060 can't identify protocol
Server  1778 ---  467u  sock                0,6      0t0 307806 can't identify protocol
Server  1778 ---  468u  sock                0,6      0t0 308061 can't identify protocol
Server  1778 ---  469u  sock                0,6      0t0 308070 can't identify protocol
Server  1778 ---  470u  sock                0,6      0t0 308071 can't identify protocol
Server  1778 ---  471u  sock                0,6      0t0 308625 can't identify protocol


that's very strange...
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/
Back to top
View user's profile Send private message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Sun Jul 07, 2013 4:51 pm    Post subject: Reply with quote

Moreover...
the valid connections have type "inet", the wrong ones have type "sock"...
what does it mean???
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sun Jul 07, 2013 5:23 pm    Post subject: Reply with quote

DevOne wrote:
what does it mean?

I'd be willing to bet you aren't checking the error return from a function.
candide wrote:
A testcase is a minimal compilable example exhibiting your symptoms. "Minimal" means just the bare essentials required to illustrate your question. "Compilable" means that there is enough code to compile it on our machines, and therefore use our debuggers. Please paste a testcase at http://ideone.com/ or http://codepad.org/ to help us assist you.
Back to top
View user's profile Send private message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Sun Jul 07, 2013 6:02 pm    Post subject: Reply with quote

steveL wrote:
DevOne wrote:
what does it mean?



I write all errno value in my log for each new connection and dispatch...it's always 0
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21489

PostPosted: Sun Jul 07, 2013 6:29 pm    Post subject: Reply with quote

DevOne: you have now had two people ask you for a sample of the faulty code so that it can be analyzed. Consider my post the third request. If you want us to help you, we need more to work with than some vague claims that you think your code is correct with regard to the blind suggestions we have offered. Some people may have the patience to continue offering blind suggestions, but you will get more and better help if you give us enough that we can review the problem independently.
Back to top
View user's profile Send private message
v_andal
Guru
Guru


Joined: 26 Aug 2008
Posts: 541
Location: Germany

PostPosted: Mon Jul 08, 2013 6:52 am    Post subject: Reply with quote

DevOne wrote:
Moreover...
the valid connections have type "inet", the wrong ones have type "sock"...
what does it mean???


It just means, that these sockets use non-standard protocol. For example, I get the same type of output for sockets talking M3UA over SCTP protocol. Does your application use protocols other than IP, TCP, UDP?

Of course, the best would be to see the code. But really, the only reason for too many open sockets is not closing them. If you are not using TCP protocol, then you won't have status CLOSE_WAIT in netstat, but the sockets shall still be open. The kernel can't reuse socket if the application didn't close it.
Back to top
View user's profile Send private message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Mon Jul 08, 2013 1:57 pm    Post subject: Reply with quote

v_andal wrote:

It just means, that these sockets use non-standard protocol. For example, I get the same type of output for sockets talking M3UA over SCTP protocol. Does your application use protocols other than IP, TCP, UDP?

Of course, the best would be to see the code. But really, the only reason for too many open sockets is not closing them. If you are not using TCP protocol, then you won't have status CLOSE_WAIT in netstat, but the sockets shall still be open. The kernel can't reuse socket if the application didn't close it.


Hi, I'm just using TCP and UDP...I cannot post code because it's not just a code inside my program...it's a complex framework I have built from scratch using a mix of tcp and udp.
The only problem I can see, I'm using a garbage procedure timer (since it's not just open connection -> elaborate -> close).
I'am using getsockopt() to retrieve tcp_info structure for each socket I collect in my queue:
I just use to check ESTABLISHED, CLOSE_TIME_WAIT to force closing after a long period or to close pending connection...maybe a bug in tcp_info flag can cause this problem, since this type of error comes up sometimes and not for every connection (this place if the only one in which I close all tcp socket)
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/
Back to top
View user's profile Send private message
v_andal
Guru
Guru


Joined: 26 Aug 2008
Posts: 541
Location: Germany

PostPosted: Tue Jul 09, 2013 6:54 am    Post subject: Reply with quote

I've done couple of tests. The unconnected UDP and TCP sockets are also reported as "can't identify protocol" by lsof. Also, if you are using UDP sockets, then you can't get tcp_info for them. The getsockopt in this case returns error indication.

Your approach with timed garbage collecting for sockets is very strange. For TCP sockets one should just react to EOF indicator. As for UDP sockets, I don't get why would anyone create and delete them constantly. I mean, I can imagine creation of UDP sock, sending some data and then closing it after that, but keeping such sockets around seems unusual. If I expect to receive data over UDP, then I create one socket listening on specified port and keep using it. Well, the point is, timed garbage collector is not needed for UDP sockets neither.

Anyway, since you can't provide code, you are on your own in searching for the bug. But I assure you, the bug is in your program, not in tcp_info support nor any other system code.
Back to top
View user's profile Send private message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Tue Jul 09, 2013 8:03 am    Post subject: Reply with quote

v_andal wrote:
I've done couple of tests. The unconnected UDP and TCP sockets are also reported as "can't identify protocol" by lsof. Also, if you are using UDP sockets, then you can't get tcp_info for them. The getsockopt in this case returns error indication.

Your approach with timed garbage collecting for sockets is very strange. For TCP sockets one should just react to EOF indicator. As for UDP sockets, I don't get why would anyone create and delete them constantly. I mean, I can imagine creation of UDP sock, sending some data and then closing it after that, but keeping such sockets around seems unusual. If I expect to receive data over UDP, then I create one socket listening on specified port and keep using it. Well, the point is, timed garbage collector is not needed for UDP sockets neither.

Anyway, since you can't provide code, you are on your own in searching for the bug. But I assure you, the bug is in your program, not in tcp_info support nor any other system code.


Hi,
thanks to take time to make tests...I'm just using garbage procedure on tcp connections and right now every condition (TCP_CLOSE_WAIT, TCP_CLOSING and so on...) ends up with a close(sock_fd) and no other case or error is generated using else conditions (so right now I'm sure no sockets are left opened); I'm still receiving "can't identify protocol" even all case are handled with close(fd).
It's strange for you because you are thinking of just an http server or similar, where you start a request, handle that and then send result closing socket...my framework can do that, but it permits to create persistent connections for chatting or sending every time informations (from server to client and back) and can handle multiple connections when client goes down with internet and similar....when client sends close request just only the system is notified, there is no callback triggered towards server app to handle that.
Anyway I think problem is not in my garbage...and seems to happen just sometimes
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Tue Jul 09, 2013 12:01 pm    Post subject: Reply with quote

DevOne wrote:
<A load of crap>

You are a complete and utter time-waster.

Good luck with your rubbish code.

Makes note never to run anything from "DevOne", nor indeed to bother helping him with anything.
Back to top
View user's profile Send private message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Tue Jul 09, 2013 12:21 pm    Post subject: Reply with quote

steveL wrote:
DevOne wrote:
<A load of crap>

You are a complete and utter time-waster.

Good luck with your rubbish code.

Makes note never to run anything from "DevOne", nor indeed to bother helping him with anything.


i didn't catch your thoughts...what is your problem?!?
what you call craps it's working 24h on 24h for months without any human help, without eats extra memory from start and managing hundred and hundred of connections...
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/
Back to top
View user's profile Send private message
DevOne
Guru
Guru


Joined: 13 Jul 2005
Posts: 419

PostPosted: Wed Jul 10, 2013 8:23 am    Post subject: Reply with quote

Ok I have resolved my problem...I had to compile all my code (core libraries, dynamic commands and plugin) on target machine and now it works.
I think problem was in commands in which they got mysql connection objects thru plugin loaded by core library and this sometimes caused to not properly close connections.

Thank you so much for all people have supported me so lovely.

I'm a lucky developer since I made "rubbish" code! :lol:
_________________
Real-time cooperative softair: http://www.softairrealfight.net
Qt generic-compass contribution: https://codereview.qt-project.org/#/c/92017
Open-source Virtual Keyboard QT widget: https://www.linux-apps.com/p/1132203/
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum