Squid file descriptors problem on specific hits

Taking care of your Linux box.
osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Squid file descriptors problem on specific hits

Postby osama1 » Wed Dec 09, 2009 9:36 pm

Hello,

I am facing a problem in squid. I get these packets and like these many more in squid cache. filedescriptors limit is 4096. I think these packets just establish connection and fill the filedescriptors limit. After that squid does not respond and net goes down. If I block these IPs (like 94.76.213.217) mentioned in squid log then after a while net starts and everything starts working. I thought it was a virus and spreading in network and disables squid from working. any suggestion what might be the permanent solution for this. I have installed dansguardian and squid.

dansguardian is not enabled for all clients since everyone does not like filtering.

1255143629.157 1031 192.168.3.78 TCP_MISS/503 1526 GET http://94.76.213.217/images/? - DIRECT/94.76.213.217 text/html
1255143746.976 1323 192.168.3.78 TCP_MISS/302 977 GET http://www.google.com/ - DIRECT/216.239.59.99 text/html
1255143748.095 909 192.168.3.78 TCP_MISS/200 1568 GET http://www.google.com.pk/ - DIRECT/216.239.59.147 text/html
1255143749.316 867 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143749.363 878 192.168.3.78 TCP_MISS/302 622 GET http://119.42.231.250/search? - DIRECT/119.42.231.250 text/html
1255143749.417 876 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143749.753 433 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143750.819 1249 192.168.3.78 TCP_MISS/200 709 GET http://static.alimama.com/static/tbk/index.html? - DIRECT/218.60.35.189 text/html
1255143756.668 407 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143764.090 404 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143764.371 686 192.168.3.78 TCP_MISS/000 24 GET http://149.20.56.32/search? - DIRECT/149.20.56.32 -
1255143764.524 688 192.168.3.78 TCP_MISS/000 24 GET http://149.20.56.32/search? - DIRECT/149.20.56.32 -
1255143764.547 866 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143764.752 1073 192.168.3.78 TCP_MISS/200 343 GET http://205.188.161.4/search? - DIRECT/205.188.161.4 text/plain
1255143771.652 404 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143771.666 411 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143772.112 690 192.168.3.78 TCP_MISS/000 24 GET http://149.20.56.32/search? - DIRECT/149.20.56.32 -
1255143779.734 879 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143779.745 880 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143780.953 2085 192.168.3.78 TCP_MISS/502 1518 GET http://199.2.137.252/search? - DIRECT/199.2.137.252 text/html
1255143786.724 404 192.168.3.78 TCP_MISS/404 629 GET http://83.68.16.6/search? - DIRECT/83.68.16.6 text/html
1255143787.178 867 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143787.182 869 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143787.190 881 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143787.198 878 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255143794.645 877 192.168.3.78 TCP_MISS/404 372 GET http://221.7.91.31/search? - DIRECT/221.7.91.31 text/html
1255150390.001 179996 127.0.0.1 TCP_MISS/504 1526 GET http://74.208.164.166/search? - DIRECT/74.208.164.166 text/html

1255150382.014 180001 127.0.0.1 TCP_MISS/504 1526 GET http://74.208.164.166/search? - DIRECT/74.208.164.166 text/html
1255150375.015 179971 127.0.0.1 TCP_MISS/504 1520 GET http://97.74.200.45/search? - DIRECT/97.74.200.45 text/html

lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Postby lambda » Thu Dec 10, 2009 8:33 pm

you'll need to find some pattern in common with these requests to block them effectively. perhaps they send some other header that you can track, or something like that.

an alternative is to limit the number of concurrent connections per ip to something like 20. that might slow down the problem, but not stop it.
Watch out for the !
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?

osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Postby osama1 » Thu Dec 10, 2009 9:27 pm

can you point a manual to apply this limit either through squid or iptables or dansguardian.

osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Postby osama1 » Thu Dec 10, 2009 11:16 pm

I just applied a few rules and I think it works. lemme test it for a couple of days

osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Postby osama1 » Fri Dec 11, 2009 8:28 pm

I got this. Any suggestion what are these packets and how to stop them.

I want to block these based on flags rather then IP

Oct 12 19:59:17 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39462 DF PROTO=TCP SPT=4783 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:25 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39704 DF PROTO=TCP SPT=4786 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:25 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39708 DF PROTO=TCP SPT=4787 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:25 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39710 DF PROTO=TCP SPT=4788 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:32 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=39980 DF PROTO=TCP SPT=4793 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:40 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=40219 DF PROTO=TCP SPT=4796 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:40 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=40220 DF PROTO=TCP SPT=4797 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:48 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=40470 DF PROTO=TCP SPT=4800 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Oct 12 19:59:48 office kernel: IN=eth0 OUT= MAC=00:a0:24:ac:d1:ee:00:06:5b:c5:d7:a1:08:00 SRC=192.168.3.94 DST=221.7.91.31 LEN=48 TOS=0x00 PREC=0x00 TTL=128 ID=40471 DF PROTO=TCP SPT=4801 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0

lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Postby lambda » Fri Dec 11, 2009 10:31 pm

that sounds like a bad idea in general. are you positive no other traffic will be matched by your rules?

which flags?
Watch out for the !
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?

osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Postby osama1 » Sat Dec 12, 2009 12:41 pm

> I applied these rules and its working.

$IPTABLES -t nat -A PREROUTING -p tcp --dport 80 ! --syn -m state --state NEW -j DROP

or

$IPTABLES -t nat -A PREROUTING -p tcp ! --syn -m state --state NEW -j DROP

2nd rule is better

> These rules set a limit to 25 new connection per minute per ip ; just to avoid flooding

$IPTABLES -t nat -A PREROUTING -p tcp --dport 80 -i $INTERNAL_INTERFACE -m state --state NEW -m recent --set


$IPTABLES -t nat -A PREROUTING -p tcp --dport 80 -i $INTERNAL_INTERFACE -m state --state NEW -m recent --update --seconds 60 --hitcount 25 -j DROP

> to check open connections per ip

netstat -atnp -A inet | grep ":3128" | awk -F " " '{print $5} '| awk -F ":" '{print $1}'| sort | uniq -c | sort -nr

> to check squid occupied file descrptors
squidclient -p 3128 mgr:info | grep 'file descri'

osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Postby osama1 » Tue Dec 15, 2009 11:59 pm


lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Postby lambda » Wed Dec 16, 2009 2:59 pm

Watch out for the !
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?

osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

rules to tackle syn flood

Postby osama1 » Thu Dec 17, 2009 11:20 am


lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Postby lambda » Thu Dec 17, 2009 3:15 pm

Watch out for the !
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?

osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Postby osama1 » Thu Dec 17, 2009 3:30 pm


lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Postby lambda » Thu Dec 17, 2009 3:39 pm

Watch out for the !
Isn't it amazing how so many people can type "linuxpakistan.net" into their browsers but not "google.com"?

osama1
Lance Naik
Posts: 33
Joined: Fri Jul 17, 2009 10:02 am

Postby osama1 » Thu Dec 17, 2009 5:36 pm


LinuxFreaK
Site Admin
Posts: 5132
Joined: Fri May 02, 2003 10:24 am
Location: Karachi
Contact:

Re:

Postby LinuxFreaK » Tue Jan 26, 2010 11:44 am

Dear osama1,
Hello,

You can take a look into maxconn under squid configuration.

FYI, http://www.visolve.com/squid/squid24s1/ ... ntrols.php

Best Regards.
Farrukh Ahmed


Return to “Administration”

Who is online

Users browsing this forum: No registered users and 1 guest