On this page
Ultimate Security Proxy With Tor
Nowadays, within the growing web 2.0 environment you may want to have some anonymity, and use other IP addresses than your own IP. Or, for some special purposes - a few IPs or more, frequently changed. So no one will be able to track you. A solution exists, and it is called Tor Project, or simply tor. There are a lot of articles and howtos giving you the idea of how it works, I'm not going to describe here onion routing and its principles, I'll rather tell you how practically pull out the maximum out of it.
So, let's start.
Major disadvantages of tor, with all its benefits like security (however, tor manufacturers advice not to rely on its strong anonymity, because there are some cases, when it is possible to track young h4x0r...) are:
1. Long enough session (60 seconds for one connection per one exit node, that means, that you will have one external IP for a minute)
2. Slow perfomance (request takes up to 20 seconds to complete, what makes surfing sites with lot of elements a disaster)
3. All the requests come through one node, and possibly route how your requests arrive can be calculated.
In order to fight these three we are going to use:
1. 8 tor processes, each using separate spool directory
2. 8 privoxy processes, each configured to talk to separate tor.
3. First squid, with malware domains blacklist, will have 8 round robin cache peers configured. (squid-in)
4. Havp, with squid-in as parent. (Anti-virus proxy, using clamav :) )
5. Second squid, that will use havp as parent (squid-out). Users will connect to this one.
Schema:
Okay, so you got it, how all it looks like. Now let's do all the required configuration.
For squid-in about 2Gb of disc cache will be enough.
For squid-out it is better to have about 12-15 Gb of disc cache to store clean elements.
So you'll need at least 20 Gb HDD.
Now about RAM.
Each tor process consumes ~ 6 Mb of Ram. 6x8 = 48 ~ 64 Mb of RAM for Tor processes.
Privoxy eats about 2 Mb each. 2x8 = 16 ~ 32 Mb of RAM.
So, 128 Mb of RAM will be enough for Tor and Privoxy processes.
Squid-in will have memory footprint approx 128 Mb. We assume it eats 256 Mb.
Same for Squid-out. We got 768 Mb of RAM total required. If it is gonna be a real system - we assume 1 Gb hardware RAM
installed.
CPU.
As you know, anti-virus scanning is very CPU demanding.
I've used a 2.0 GHz P4.
I've tested this system under concurrent load under 10 users, and it did now show it was exhausted.
Finally, we have:
CPU: 2.0 GHz
RAM: 1.0 Gb
HDD: 20 Gb
Used software:
Privoxy
privoxy --version
Privoxy version 3.0.8 (http://www.privoxy.org/)
Tor
tor --version
Oct 11 19:49:27.394 [notice] Tor v0.1.2.19. This is experimental software. Do not rely on it for strong anonymity.
Tor version 0.1.2.19.
Squid
squid -v
Squid Cache: Version 2.7.STABLE3
configure options: '--prefix=/usr' '--host=x86_64-pc-linux-gnu' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib'
'--sysconfdir=/etc/squid' '--libexecdir=/usr/libexec/squid' '--localstatedir=/var'
'--datadir=/usr/share/squid' '--enable-auth=basic,digest,ntlm' '--enable-removal-policies=lru,heap'
'--enable-digest-auth-helpers=password'
'--enable-basic-auth-helpers=SASL,PAM,SMB,multi-domain-NTLM,getpwnam,NCSA,MSNT'
'--enable-external-acl-helpers=wbinfo_group,ip_user,session,unix_group'
'--enable-ntlm-auth-helpers=SMB,fakeauth' '--enable-ident-lookups' '--enable-useragent-log'
'--enable-cache-digests' '--enable-delay-pools' '--enable-referer-log' '--enable-arp-acl' '--with-pthreads'
'--with-large-files' '--enable-htcp' '--enable-carp' '--enable-follow-x-forwarded-for' '--with-maxfd=8192'
'--enable-snmp' '--enable-ssl' '--enable-storeio=ufs,diskd,coss,aufs,null' '--enable-async-io'
'--enable-linux-netfilter' '--libdir=/usr/lib64' '--build=x86_64-pc-linux-gnu'
'build_alias=x86_64-pc-linux-gnu' 'host_alias=x86_64-pc-linux-gnu' 'CC=x86_64-pc-linux-gnu-gcc'
'CFLAGS=-march=k8 -O2 -pipe -fomit-frame-pointer' 'LDFLAGS=-Wl,-O1'
Tor Configuration
/etc/tor/torrc
SocksPort 9050 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor
/etc/tor/torrc2
SocksPort 9150 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor2 ControlPort 9151
/etc/tor/torrc3
SocksPort 9250 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor3 ControlPort 9251
/etc/tor/torrc4
SocksPort 9350 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor4 ControlPort 9351
/etc/tor/torrc5
SocksPort 9450 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor5 ControlPort 9451
/etc/tor/torrc6
SocksPort 9550 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor6 ControlPort 9551
/etc/tor/torrc7
SocksPort 9650 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor7 ControlPort 9651
/etc/tor/torrc8
SocksPort 9750 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor8 ControlPort 9751
Configuration files for privoxy
/etc/privoxy/config
trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8118 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9050 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0
/etc/privoxy/config2
trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy2 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8129 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9150 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0
/etc/privoxy/config3
trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy3 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8230 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9250 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0
/etc/privoxy/config4
trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy4 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8321 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9350 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0
/etc/privoxy/config5
trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy5 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8421 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9450 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0
/etc/privoxy/config6
trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy6 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8522 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9550 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0
/etc/privoxy/config7
trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy7 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8623 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9650 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0
/etc/privoxy/config8
trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy8 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8724 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9750 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0
Startup scripts
Privoxy: /etc/rc.d/privoxy (Arch linux)
#!/bin/bash
# source application-specific settings
[ -f /etc/conf.d/privoxy ] && . /etc/conf.d/privoxy
. /etc/rc.conf
. /etc/rc.d/functions
#PID=`pidof -o %PPID /usr/sbin/privoxy`
start() {
stat_busy "Starting Privoxy, squid, havp"
chown privoxy:privoxy /var/log/privoxy
# create missing logfiles
for i in logfile jarfile; do
touch /var/log/privoxy/$i
chgrp $PRIVOXY_GROUP /var/log/privoxy/$i
chmod 660 /var/log/privoxy/$i
done
chown privoxy:privoxy /var/log/privoxy2
# create missing logfiles
for i in logfile jarfile; do
touch /var/log/privoxy2/$i
chgrp $PRIVOXY_GROUP /var/log/privoxy2/$i
chmod 660 /var/log/privoxy2/$i
done
chown privoxy:privoxy /var/log/privoxy3
# create missing logfiles
for i in logfile jarfile; do
touch /var/log/privoxy3/$i
chgrp $PRIVOXY_GROUP /var/log/privoxy3/$i
chmod 660 /var/log/privoxy3/$i
done
chown privoxy:privoxy /var/log/privoxy4
# create missing logfiles
for i in logfile jarfile; do
touch /var/log/privoxy4/$i
chgrp $PRIVOXY_GROUP /var/log/privoxy4/$i
chmod 660 /var/log/privoxy4/$i
done
chown privoxy:privoxy /var/log/privoxy5
# create missing logfiles
for i in logfile jarfile; do
touch /var/log/privoxy5/$i
chgrp $PRIVOXY_GROUP /var/log/privoxy5/$i
chmod 660 /var/log/privoxy5/$i
done
chown privoxy:privoxy /var/log/privoxy6
# create missing logfiles
for i in logfile jarfile; do
touch /var/log/privoxy6/$i
chgrp $PRIVOXY_GROUP /var/log/privoxy6/$i
chmod 660 /var/log/privoxy6/$i
done
chown privoxy:privoxy /var/log/privoxy7
# create missing logfiles
for i in logfile jarfile; do
touch /var/log/privoxy7/$i
chgrp $PRIVOXY_GROUP /var/log/privoxy7/$i
chmod 660 /var/log/privoxy7/$i
done
chown privoxy:privoxy /var/log/privoxy8
# create missing logfiles
for i in logfile jarfile; do
touch /var/log/privoxy8/$i
chgrp $PRIVOXY_GROUP /var/log/privoxy8/$i
chmod 660 /var/log/privoxy8/$i
done
#[ -z "$PID" ] && /usr/sbin/privoxy $PRIVOXY_ARGS 2>/dev/null
/usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy.pid /etc/privoxy/config
/usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy2.pid /etc/privoxy/config2
/usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy3.pid /etc/privoxy/config3
/usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy4.pid /etc/privoxy/config4
/usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy5.pid /etc/privoxy/config5
/usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy6.pid /etc/privoxy/config6
/usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy7.pid /etc/privoxy/config7
/usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy8.pid /etc/privoxy/config8
havp -c /etc/havp/havp.config
squid -f /etc/squid/squid-in.conf -DYC
squid -f /etc/squid/squid-out.conf -DYC
if [ $? -gt 0 ]; then
stat_fail
else
add_daemon privoxy
stat_done
fi
}
stop() {
stat_busy "Stopping Privoxy, Squid, Havp"
#[ ! -z "$PID" ] && kill $PID &> /dev/null
killall privoxy
killall squid
killall havp
if [ $? -gt 0 ]; then
stat_fail
else
rm_daemon privoxy
stat_done
fi
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
$0 stop
sleep 1
$0 start
;;
*)
echo "usage: $0 {start|stop|restart}"
esac
exit 0
Tor startup script: /etc/rc.d/tor (Arch linux)
#!/bin/bash
. /etc/rc.conf
. /etc/rc.d/functions
#PID=`pidof -o %PPID /usr/bin/tor`
tor='/usr/bin/tor'
case "$1" in
start)
stat_busy "Starting Tor Daemon"
#[ -z "$PID" ] && /usr/bin/tor &>/dev/null
${tor} -f /etc/tor/torrc
${tor} -f /etc/tor/torrc2
${tor} -f /etc/tor/torrc3
${tor} -f /etc/tor/torrc4
${tor} -f /etc/tor/torrc5
${tor} -f /etc/tor/torrc6
${tor} -f /etc/tor/torrc7
${tor} -f /etc/tor/torrc8
if [ $? -gt 0 ]; then
stat_fail
else
add_daemon tor
stat_done
fi
;;
stop)
stat_busy "Stopping Tor Daemon"
#[ ! -z "$PID" ] && kill $PID &> /dev/null
killall tor
if [ $? -gt 0 ]; then
stat_fail
else
rm_daemon tor
stat_done
fi
;;
restart)
$0 stop
sleep 3
$0 start
;;
*)
echo "usage: $0 {start|stop|restart}"
esac
exit 0
# vim: ft=sh ts=2 sw=2
Squid-in config
/etc/squid/squid-in.conf
acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports acl malware_domains url_regex '/etc/squid/Malware-domains.txt' http_access deny malware_domains http_access allow localhost http_access deny all icp_access deny all http_port 3400 icp_port 0 hierarchy_stoplist cgi-bin ? refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 cache_peer localhost parent 8118 0 round-robin no-query cache_peer localhost2 parent 8129 0 round-robin no-query cache_peer localhost3 parent 8230 0 round-robin no-query cache_peer localhost4 parent 8321 0 round-robin no-query cache_peer localhost5 parent 8421 0 round-robin no-query cache_peer localhost6 parent 8522 0 round-robin no-query cache_peer localhost7 parent 8623 0 round-robin no-query cache_peer localhost8 parent 8724 0 round-robin no-query never_direct allow all always_direct deny all acl apache rep_header Server ^Apache broken_vary_encoding allow apache forwarded_for off coredump_dir /var/cache/squid-in cache_dir ufs /var/cache/squid-in 100 16 256 pid_filename /var/run/squid-in.pid access_log /var/log/squid/access.squid-in.log cache_store_log /var/log/squid/store.squid-in.log cache_log /var/log/squid/cache.squid-in.log
Squid-out config
/etc/squid/squid-out.conf
acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access allow localhost http_access deny all icp_access deny all http_port 3128 icp_port 0 hierarchy_stoplist cgi-bin ? refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 cache_peer localhost parent 3410 0 round-robin no-query never_direct allow all always_direct deny all acl apache rep_header Server ^Apache broken_vary_encoding allow apache forwarded_for off coredump_dir /var/cache/squid-out cache_dir ufs /var/cache/squid-out 100 16 256 pid_filename /var/run/squid-out.pid access_log /var/log/squid/access.squid-out.log cache_store_log /var/log/squid/store.squid-out.log cache_log /var/log/squid/cache.squid-out.log
Havp antivirus proxy config
/etc/havp/havp.config
USER havp GROUP havp DAEMON true PIDFILE /tmp/havp.pid SERVERNUMBER 4 MAXSERVERS 256 SCANTEMPFILE /tmp/havp-XXXXXX TEMPDIR /tmp DBRELOAD 60 PARENTPROXY localhost PARENTPORT 3400 FORWARDED_IP false X_FORWARDED_FOR false PORT 3410 BIND_ADDRESS 127.0.0.1 ENABLECLAMLIB true ENABLECLAMD false ENABLEFPROT false ENABLEAVG false ENABLEAVESERVER false ENABLESOPHIE false ENABLETROPHIE false ENABLENOD32 false ENABLEAVAST false ENABLEARCAVIR false ENABLEDRWEB false
Domain List updater script (can be launched from cron)
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Author: Bohdan Turkynewych, AKA Gh0st, 2006-2008, tb0hdan[at]gmail[dot]com
# Distributed under the terms of GPLv2
# Parts of code from neutron jabber bot project, http://code.google.com/p/neutron/
from urllib2 import Request as urllib2_Request, urlopen as urllib2_urlopen
from sys import exit as sys_exit
from re import search as re_search
from time import ctime as time_ctime, time as time_time
from re import compile as re_compile
def t_conv(timestamp):
reply = ''
seconds = timestamp % 60
minutes = (timestamp / 60) % 60
hours = (timestamp / 3600) % 60
days = timestamp / 216000
if days: reply += str(days) + ' day(s) '
if hours: reply += str(hours) + ' hour(s) '
if minutes: reply += str(minutes) + ' minute(s) '
reply += str(seconds) + ' second(s)'
return reply
def uptime():
global BOOTUP_TIMESTAMP
if BOOTUP_TIMESTAMP:
idletime = int(time_time() - BOOTUP_TIMESTAMP)
reply = t_conv(idletime)
else:
reply = 'Unknown'
return reply
strip_tags = re_compile(r'<[^]+>')
def decode(text):
data = text.replace('<br>','\n').replace(' ', ' ').replace('<', '<').replace('>', '>').replace('"', '"').replace('<br>','\n').replace('
<li>','\r\n')
return strip_tags.sub('', data)
def get_data(message):
reply=[]
try:
for line in message.split('\n'):
if re_search('wbr',line):
#
od = re_search('',line)
d0main = line[od.end():]
d0main = d0main[:re_search('',d0main).start()].strip()
#
if d0main == '-':
od = re_search('-',line)
bookurl = line[od.end():]
bookurl = bookurl[:re_search('',bookurl).start()].strip()
bookurl = decode(bookurl)
else:
bookurl = decode(d0main)
d0main = ''
stripper = '[/|:]'
if re_search(stripper,bookurl):
#d0main = bookurl
od = re_search(stripper,bookurl)
d0main = bookurl[od.end():]
d0main = bookurl[:re_search(stripper,bookurl).start()].strip()
else:
d0main = bookurl
if not d0main in reply:
reply.append(d0main)
return reply
except:
raise
def update_list(filename):
req2 = urllib2_Request('http://www.malwaredomainlist.com/mdl.php?sort=Domain&ascordesc=DESC&search=&colsearch=All&quantity=All')
req2.add_header = ('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.9) Gecko/20071030 SeaMonkey/1.1.6')
print '[+] Update process started at ' + time_ctime()
try:
r = urllib2_urlopen(req2)
target = r.read()
urls = get_data(target)
except:
raise
print '[-] Error occured while fetching / parsing data.'
sys_exit(1)
print '[+] Got ' + str(len(urls)) + ' domains/ips.'
try:
file = open(filename,'w')
for line in urls:
file.write(line + '\n')
file.close()
except:
print '[-] Error occured while flushing data to file'
sys_exit(5)
elapsed = uptime()
print '[+] Update process finished at ' + time_ctime() + '. Took ' + elapsed
BOOTUP_TIMESTAMP = time_time()
update_list('/etc/squid/Malware-domains.txt')
Have fun!
P.S. Future optimisation requires 8 havp instances as well due to slowup.
P.S.S You may exclude havp and squid-out proxy from your system, and use only squid-in. That will be much, much faster.
