Squid file descriptors problem on specific hits

Taking care of your Linux box.
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Squid file descriptors problem on specific hits

Post by osama1 »

Hello,

I am facing a problem in squid. I get these packets and like these many more in squid cache. filedescriptors limit is 4096. I think these packets just establish connection and fill the filedescriptors limit. After that squid does not respond and net goes down. If I block these IPs (like 94.76.213.217) mentioned in squid log then after a while net starts and everything starts working. I thought it was a virus and spreading in network and disables squid from working. any suggestion what might be the permanent solution for this. I have installed dansguardian and squid.

dansguardian is not enabled for all clients since everyone does not like filtering.

1255143629.157 1031 192.168.3.78 TCP_MISS/503 1526 GET http://94.76.213.217/images/? - DIRECT/94.76.213.217 text/html
1255143746.976 1323 192.168.3.78 TCP_MISS/302 977 GET http://www.google.com/ - DIRECT/216.239.59.99 text/html
1255143748.095 909 192.168.3.78 TCP_MISS/200 1568 GET http://www.google.com.pk/ - DIRECT/216.239.59.147 text/html
1255143749.316 867 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143749.363 878 192.168.3.78 TCP_MISS/302 622 GET http://119.42.231.250/search? - DIRECT/119.42.231.250 text/html
1255143749.417 876 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143749.753 433 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143750.819 1249 192.168.3.78 TCP_MISS/200 709 GET http://static.alimama.com/static/tbk/index.html? - DIRECT/218.60.35.189 text/html
1255143756.668 407 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143764.090 404 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143764.371 686 192.168.3.78 TCP_MISS/000 24 GET http://149.20.56.32/search? - DIRECT/149.20.56.32 -
1255143764.524 688 192.168.3.78 TCP_MISS/000 24 GET http://149.20.56.32/search? - DIRECT/149.20.56.32 -
1255143764.547 866 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143764.752 1073 192.168.3.78 TCP_MISS/200 343 GET http://205.188.161.4/search? - DIRECT/205.188.161.4 text/plain
1255143771.652 404 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143771.666 411 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143772.112 690 192.168.3.78 TCP_MISS/000 24 GET http://149.20.56.32/search? - DIRECT/149.20.56.32 -
1255143779.734 879 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143779.745 880 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143780.953 2085 192.168.3.78 TCP_MISS/502 1518 GET http://199.2.137.252/search? - DIRECT/199.2.137.252 text/html
1255143786.724 404 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143787.178 867 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143787.182 869 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143787.190 881 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143787.198 878 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143794.645 877 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255150390.001 179996 127.0.0.1 TCP_MISS/504 1526 GET http://74.208.164.166/search? - DIRECT/74.208.164.166 text/html

1255150382.014 180001 127.0.0.1 TCP_MISS/504 1526 GET http://74.208.164.166/search? - DIRECT/74.208.164.166 text/html
1255150375.015 179971 127.0.0.1 TCP_MISS/504 1520 GET http://97.74.200.45/search? - DIRECT/97.74.200.45 text/html
lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Post by lambda »

you'll need to find some pattern in common with these requests to block them effectively. perhaps they send some other header that you can track, or something like that.

an alternative is to limit the number of concurrent connections per ip to something like 20. that might slow down the problem, but not stop it.
Watch out for the Manners Taliban!
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Post by osama1 »

can you point a manual to apply this limit either through squid or iptables or dansguardian.
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Post by osama1 »

I just applied a few rules and I think it works. lemme test it for a couple of days
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Post by osama1 »

I got this. Any suggestion what are these packets and how to stop them.

I want to block these based on flags rather then IP

Oct 12 19:59:17 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39462 DF PROTO=TCP SPT=4783 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:25 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39704 DF PROTO=TCP SPT=4786 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:25 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39708 DF PROTO=TCP SPT=4787 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:25 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39710 DF PROTO=TCP SPT=4788 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:32 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39980 DF PROTO=TCP SPT=4793 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:40 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=40219 DF PROTO=TCP SPT=4796 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:40 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=40220 DF PROTO=TCP SPT=4797 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:48 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=40470 DF PROTO=TCP SPT=4800 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:48 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=40471 DF PROTO=TCP SPT=4801 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Post by lambda »

that sounds like a bad idea in general. are you positive no other traffic will be matched by your rules?

which flags?
Watch out for the Manners Taliban!
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Post by osama1 »

> I applied these rules and its working.

$IPTABLES -t nat -A PREROUTING -p tcp --dport 80 ! --syn -m state --state NEW -j DROP

or

$IPTABLES -t nat -A PREROUTING -p tcp ! --syn -m state --state NEW -j DROP

2nd rule is better

> These rules set a limit to 25 new connection per minute per ip ; just to avoid flooding

$IPTABLES -t nat -A PREROUTING -p tcp --dport 80 -i $INTERNAL_INTERFACE -m state --state NEW -m recent --set


$IPTABLES -t nat -A PREROUTING -p tcp --dport 80 -i $INTERNAL_INTERFACE -m state --state NEW -m recent --update --seconds 60 --hitcount 25 -j DROP

> to check open connections per ip

netstat -atnp -A inet | grep ":3128" | awk -F " " '{print $5} '| awk -F ":" '{print $1}'| sort | uniq -c | sort -nr

> to check squid occupied file descrptors
squidclient -p 3128 mgr:info | grep 'file descri'
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Post by osama1 »

Now i am facing another virus like thing. Any suggestion and permanent solution to stop these packets?

>> netstat -atnp -A inet | grep ":18085" |grep TIME_WAIT | awk -F " " '{print $5} '| awk -F ":" '{print $1}'| sort | uniq -c | sort -nr
3383 192.168.3.71
19 192.168.3.41
7 192.168.3.120
6 192.168.3.83

>>in squid log

Fri Oct 16 23:46:31 2009 1 192.168.3.71 TCP_NEGATIVE_HIT/403 219 GET http://www.autoregalia.co.uk/ - NONE/- -
Fri Oct 16 23:46:31 2009 1 192.168.3.71 TCP_NEGATIVE_HIT/404 817 GET http://regionbukovel.info/ru/ - NONE/- text/html
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/403 219 GET http://www.autoregalia.co.uk/ - NONE/- -
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/404 817 GET http://regionbukovel.info/ru/ - NONE/- text/html
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/404 817 GET http://regionbukovel.info/ru/ - NONE/- text/html
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/404 817 GET http://regionbukovel.info/ru/ - NONE/- text/html
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/404 817 GET http://regionbukovel.info/ru/ - NONE/- text/html
Fri Oct 16 23:46:31 2009 1 192.168.3.71 TCP_NEGATIVE_HIT/404 817 GET http://regionbukovel.info/ru/ - NONE/- text/html
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/404 817 GET http://regionbukovel.info/ru/ - NONE/- text/html
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/404 817 GET http://regionbukovel.info/ru/ - NONE/- text/html
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/403 219 GET http://www.autoregalia.co.uk/ - NONE/- -
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/403 219 GET http://www.autoregalia.co.uk/ - NONE/- -
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/403 219 GET http://www.autoregalia.co.uk/ - NONE/- -
Fri Oct 16 23:46:31 2009 1 192.168.3.71 TCP_NEGATIVE_HIT/403 219 GET http://www.autoregalia.co.uk/ - NONE/- -
Fri Oct 16 23:46:31 2009 0 192.168.3.71 TCP_NEGATIVE_HIT/403 219 GET http://www.autoregalia.co.uk/ - NONE/- -
lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Post by lambda »

every time you see this, you have to block the user, and get them to clean their system.
Watch out for the Manners Taliban!
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

rules to tackle syn flood

Post by osama1 »

Yes lambda you r right. Better is to clean the client system but in a LAN as a service provider it’s very difficult to restrict clients or clean clients computer on regular basis. There should be some kind of fault tolerance in Linux box to tackle problem.

I have applied previously mentioned rules with some modification. I have managed to mitigate the affect of syn flood by using these rules. Many other sites also suggest these rules with minor modifications. In my case I am now satisfied with these rules


$IPTABLES -t nat -A PREROUTING -p tcp ! --syn -m state --state NEW -j DROP

$IPTABLES -t nat -N syn-flood
$IPTABLES -t nat -A PREROUTING -p tcp --dport 80 -i $INT_INTERFACE --syn -j syn-flood
$IPTABLES -t nat -A syn-flood -m recent --set
$IPTABLES -t nat -A syn-flood -m recent --update --seconds 5 --hitcount 16 -j DROP
lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Post by lambda »

There should be some kind of fault tolerance in Linux box to tackle problem.
you're not getting it.

let me give you an example. you have a cell phone. suppose i (and several other people) call your cell phone fifty times a day each, just to ask you "hey, what's the time?"

what can you do about it? you can turn your phone off, but that means you can't get any calls from anyone (or make calls). you can simply press the cancel button when you see a call from me or the others, but that still wastes your time and energy to deal with my call. you can ignore the calls, but while the phone's ringing, other people can't call you. you can change your number, but if you give your number out, i or the others might find it and start calling you.

are you going to say "there should be some kind of fault tolerence in nokia to tackle the problem?"

here's another example: do you have an email address? do you get spam? do you blame linux (or windows -- whatever the server runs) for not blocking all the spam while still letting non-spam messages through?

please don't blame linux (or windows, or squid, or iptables). it's not the operating system's responsibility to deal with infected clients; it's yours. technology can't solve all your problems.

if you think you can't block them because you're a service provider, think again. companies like twitter have no problem with blocking infected users, or at least informing them.
Watch out for the Manners Taliban!
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Post by osama1 »

Well I think linux is capable of controlling anything and everything. There is no blaming. We just dont know the solution but solution is there. Just like I tried to search and resolved my problem. I think many here switched to dedicated hardware solutions due to these problems. Dedicated hardware solution is also an option but after all thats also linux.
Linux is capable of evading anykind of Virus.
So Linux is perfect... Cheerrsssss.
lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Post by lambda »

so, maybe you should upgrade your lan clients to linux.
Watch out for the Manners Taliban!
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Post by osama1 »

I wish i could. but again we cannot control the client :(
We only control server
LinuxFreaK
Site Admin
Posts: 5132
Joined: Fri May 02, 2003 10:24 am
Location: Karachi
Contact:

Re:

Post by LinuxFreaK »

Dear osama1,
Hello,

You can take a look into maxconn under squid configuration.

FYI, http://www.visolve.com/squid/squid24s1/ ... ntrols.php

Best Regards.
Farrukh Ahmed
Post Reply