Independent Suspension

2007-07-01

ipython in zopectl debug

A long time ago (6mos? a year) I got tired of manually typing support for history and tab completion into the zope debugger. So I just hooked the debug command up to read my .pythonrc file by importing user in the do_debug method of zopectl.py.

def do_debug(self, arg):
   cmdline = self.get_startup_cmd(self.options.python + ' -i',
   "import Zope2; app=Zope2.app(); import user;")

Today I tried getting ipython working as described in the plone docs. I also tried vanrees notes

and a couple others. I could not get it going though. So I decided to return to the hack above and extend it to load ipython.

def do_debug(self, arg):
    cmdline = self.get_startup_cmd(self.options.python + ' -i',
    "import Zope2; app=Zope2.app(); import user;     
     ns={'__name__':'msrd','app':app}; import IPython;   
     IPython.Shell.IPShell(user_ns=ns).mainloop(sys_exit=1);")

Poof, now debug comes up with ipython. Yay... What does that mean you may ask?

In [2]: ??app.msrd
Type: ImplicitAcquirerWrapper
Base Class:
String Form:
Namespace: Interactive
Length: 1
Docstring [source file open failed]:
Make PloneSite subclass CMFSite and add some methods.
This will be useful for adding more things later on.

I know this is probably the wrong way to do this, but nyah nyah. It is working now. I do welcome feedback...

As a note: I also have import user in my args. This picks up stuff from your file defined in PYTHONSTARTUP. But then if you already use PYTHONSTARTUP you probably already know that.

2007-01-19

Plone site broken? Fix ir through the zodb

Today someone changed the ip on a dev box running an instance that had the ip as part of the url in CacheFu. So of course it won't load a page anymore. What to do? reinstall? nah. Let 's think for a second. Plone is a nice interface to Zope/CMF. Zope/CMF is a lovely interface to ZODB. ZODB is a persistence engine for Python objects. CacheFu's settings are persisted as atributes of a cache object in the ZODB. Let's just fix it directly....

/usr/local/{instance}/bin/zopectl debug
Starting debugger (the name "app" is bound to the top-level Zope object)
...{ 8< snip } ...
>>> import readline, rlcompleter, transaction
>>> readline.parse_and_bind("tab: complete")
# now we have tab completion
# bind the plone root to s
>>> s = app.itcd
# bind our (s)ites cache settings to cc
>>> cc = s.portal_cache_settings
>>> cc.getDomains()
('http://1.2.3.4:80',) ## <<== look there is our old value!!
# yup it needs to be this new value
>>> cc.setDomains("http://4.3.2.1:8080")
# and persisted. Otherwise it will be aborted at exit
>>> transaction.commit()

That was easy, no?

2007-01-05

Quick and dirty regexen search for spamd & postfix logs


#!/usr/bin/env python
#-------------------------------------------------------------
# Name:        finder.py
# Purpose:     This is a script to search through  Posfix 
#              and spamd logs for the last x days and return 
#              hits. Mostly it is a quick and dirty way to 
#              find entris regarding emails blocked by 
#              spamd/postfix   
# Author:      Reed L. O'Brien reed at reedobrien com
#                                                                                                           
# Created:     2007-01-05
# Modified:    2007-01-05
# Copyright:   (c) Reed L. O'Brien 2007
# License:     DWYWWI (improvements welcome)                         
#--------------------------------------------------------------                      


#Do the imports
import os, re, bz2, sys, time

# make sure there is a regex
try:
    # compile the regex
  regx = re.compile(sys.argv[1], re.IGNORECASE)
except IndexError:
  print """
  usage:
         finder  [days back to search]

         ex: finder foobar 2
         will find all occurences of 'foobar' in the last 2
         days of spamd and maillog files.\n\tThe number of days
         is optional and defaults to 1 if not given."""
  sys.exit(0)

# empty list to store hits in
found = []

# move to the log directory
os.chdir('/var/log')
# Start a counter for the number counted
s = 0

#get and set days
try:
 days = int(sys.argv[2])
except:
 days = 1
# Get a list of qualifying files NOTE: you may need stat(x).[st_ctime|st_mtime] depending on your OS
L = [f for f in os.listdir('.')
   if os.stat(f).st_birthtime > time.time() - (days * 86400)
   and (f.startswith('spamd') or f.startswith('maillog'))]
# get a count of how many files to search
n = len(L)

# start a loop on the list
for f in L:
    # If it is a bz2 open it as a bz2 object
if (f.startswith('spamd') or f.startswith('maillog')) and f.endswith('2'):
 # tell em what is happening
   sys.stdout.write("\rsearching: %2s  remain %s                   " % (f,n))
   sys.stdout.flush()
        # set a line count
   c = 1
        # get a handle on the file
        handle = bz2.BZ2File(f)
        # iterate through the lines
        for line in handle:
        # if the regex is found
       if regx.search(line):
           # append the filename, line count and line content to the found list
           found.append("%-10s : %s\n%s" % (c, f, line))
                # increment the line count
           c += 1
       else:
           # or just increment the count if no regex match 
           c += 1
                # decrement the number of files remaining
           n -= 1  
                # increment the number of files searched
           s += 1

    ## DO the same as above as a regular file object if not a bz2 file SEE NOTES FOR last loop
if (f.startswith('spamd') or f.startswith('maillog')) and not f.endswith('2'):
   sys.stdout.write("\rsearching: %2s  remain %s                     " % (f,n))
   sys.stdout.flush()
   c = 1
   handle = open(f)
   for line in handle:
       if regx.search(line):
           found.append("%-10s : %s\n%s" % (c, f, line))
           c += 1
       else:
           c += 1
           n -= 1
           s += 1
                ##make some space to overwrite the sys.stdout text 
print '\n\n\n\n\n'
print 'Searched:', s # Print how many files were searched

#print the results from the found list.
for x in found:
print x

2006-12-28

Mostly functional email regex

for word based delims in text:
em = re.compile(r"\b['A-Z0-9._%-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b",
   re.IGNORECASE)


for start and endline delims:
em = re.compile(r"^['A-Z0-9._%-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$",
    re.IGNORECASE)

2006-12-14

less spam with spamd and pf

untested but should be pretty close

##Assumes Freebsd built with postfix and pf:
##requires spamd

#Add line to /etc/fstab
# Device Mountpoint FStype Options Dump Pass
fdescfs /dev/fd fdecfs rw 0 0

# mount it
sudo mount -a

## build spamd
cd /usr/ports/mail/spamd && sudo make install

#answer yes
This system has no entry for spamd in /etc/services
Would you like to add it automatically? (y/n) [y]? y
This system has no entry for spamd-cfg in /etc/services
Would you like to add it automatically? (y/n) [y]? y

# setup spamd.conf
all:\
:spamhaus:spamhausDROP:whitelist:spews1:spews2:china:korea:becks:blacklist:

# Mirrored from http://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2
spamhaus:\
:black:\
:msg="SPAM. Your address %A is in the Spamhaus Block List\n\
See http://www.spamhaus.org/sbl and\
http://www.abuse.net/sbl.phtml?IP=%A for more details":\
:method=http:\
:file=www.openbsd.org/spamd/SBL.cidr.gz:

spamhausDROP:\
:black:\
:msg="SPAM. Your address %A is in the Spamhaus DROP List\n\
See http://www.spamhaus.org/sbl and\
http://www.abuse.net/sbl.phtml?IP=%A for more details":\
:method=http:\
:file=www.spamhaus.org/DROP/drop.lasso:

becks:\ # experimental, probably blocks some good ips
:black:\
:msg="SPAM. Your address %A has sent spam within the last 24 hours":\
:method=http:\
:file=www.openbsd.org/spamd/traplist.gz

# Mirrored from http://www.spews.org/spews_list_level1.txt
spews1:\
:black:\
:msg="SPAM. Your address %A is in the spews level 1 database\n\
See http://www.spews.org/ask.cgi?x=%A for more details":\
:method=http:\
:file=www.openbsd.org/spamd/spews_list_level1.txt.gz:

# Mirrored from http://www.spews.org/spews_list_level2.txt
spews2:\
:black:\
:msg="SPAM. Your address %A is in the spews level 2 database\n\
See http://www.spews.org/ask.cgi?x=%A for more details":\
:method=http:\
:file=www.openbsd.org/spamd/spews_list_level2.txt.gz:

# Mirrored from http://www.okean.com/chinacidr.txt
china:\
:black:\
:msg="SPAM. Your address %A appears to be from China\n\
See http://www.okean.com/asianspamblocks.html for more details":\
:method=http:\
:file=www.openbsd.org/spamd/chinacidr.txt.gz:

# Mirrored from http://www.okean.com/koreacidr.txt
korea:\
:black:\
:msg="SPAM. Your address %A appears to be from Korea\n\
See http://www.okean.com/asianspamblocks.html for more details":\
:method=http:\
:file=www.openbsd.org/spamd/koreacidr.txt.gz:

# Whitelists are done like this, and must be added to "all" after each
# blacklist from which you want the addresses in the whitelist removed.
#
whitelist:\
:white:\
:file=/var/mail/whitelist.txt:

blacklist:\
:black:\
:msg=/var/mail/blackmsg.txt:\
:method=file:\
:file=/var/mail/blacklist.txt:

# touch the spamd and chown it
sudo touch /var/db/spamd && sudo chown nobody:wheel /var/db/spamd

#create the inital log file
sudo touch /var/log/spamd

# make spamd log in it's own log in /etc/syslog.conf
!spamd
*.* /var/log/spamd

#restart it
sudo /etc/rc.d/syslogd restart

# make it rotate IN /etc/newsyslog.conf
/var/log/spamd 640 1000 * @T00 JC

#edit pf.conf
# macros
nic = "xl0"

tcp_services = "{ 22, 25, 8080}"
icmp_types = "echoreq"
udp_services = "{ 123, 53 }"
web_service = "{ 80 }"
mail_host = "127.0.0.1"

priv_nets = "{ 127.0.0.0/8, 172.16.0.0/12, 10.0.0.0/8 }"
my_nets = "{ 192.168.68.0/24}"

# options
set block-policy return
set loginterface $nic
set skip on lo0

# scrub
scrub in all

#########
## Spamd
#########

# grey host list
table persist

#white host list
table persist
table persist file "/var/mail/whitelist.txt"

# forward white listed ips
rdr pass on $nic inet proto tcp from to $nic port smtp -> 127.0.0.1 port 8025

rdr pass on $nic proto tcp from to $nic port smtp -> $mail_host port smtp

rdr pass on $nic proto tcp from to $nic port smtp -> $mail_host port smtp

# send all suspects to the spamd daemon

rdr pass on $nic inet proto tcp from ! to $nic port smtp -> 127.0.0.1 port 8025

rdr pass on $nic inet proto tcp from any to $nic port smtp -> $mail_host port smtp

# filter rules
block all

block drop in quick on $nic from $priv_nets to any
block drop out quick on $nic from any to $priv_nets

pass in on $nic inet proto tcp from any to $nic port smtp flags S/SA keep state

pass in on $nic inet proto tcp from $my_nets to $nic port $tcp_services flags S/SA keep state

pass in on $nic inet proto tcp from any to $nic port $web_service flags S/SA keep state

pass in on $nic inet proto udp from any to $nic port $udp_services

pass in on $nic inet proto udp from any to $nic port 123

pass in inet proto icmp all icmp-type $icmp_types keep state

pass out on $nic proto tcp all modulate state flags S/SA
pass out on $nic proto { udp, icmp } all keep state

#make /var/mail/whitelist.txt
#FDS
192.251.225.192/26
#apple
17.0.0.0/8
#aol.com
152.163.225.0/24
205.188.139.0/24
205.188.144.0/24
205.188.156.0/23
205.188.159.0/24
64.12.136.0/23
64.12.138.0/24
152.163.225.0/24
205.188.139.0/24
205.188.144.0/24
205.188.156.0/23
205.188.159.0/24
64.12.136.0/23
64.12.138.0/24
#amazon.com
207.171.160.0/19
87.238.80.0/21
72.21.196.0/24
72.21.208.0/24
207.171.160.32/28
207.171.180.176/28
207.171.164.32/28
207.171.190.0/28
87.238.80.24/29
87.238.84.24/29
72.21.196.0/24
72.21.208.0/24
#_spf.google.com
216.239.56.0/23
64.233.160.0/19
66.249.80.0/20
72.14.192.0/18
#spf-a.hotmail.com
209.240.192.0/19
65.52.0.0/14
131.107.0.0/16
157.54.0.0/15
157.56.0.0/14
157.60.0.0/16
167.220.0.0/16
204.79.135.0/24
204.79.188.0/24
204.79.252.0/24
207.46.0.0/16
199.2.137.0/24
#spf-b.hotmail.com
199.103.90.0/23
204.182.144.0/24
204.255.244.0/23
206.138.168.0/21
64.4.0.0/18
65.54.128.0/17
207.68.128.0/18
207.68.192.0/20
207.82.250.0/23
207.82.252.0/23
209.1.112.0/23
#spf-c.hotmail.com
209.185.128.0/23
209.185.130.0/23
209.185.240.0/22
216.32.180.0/22
216.32.240.0/22
216.33.148.0/22
216.33.151.0/24
216.33.236.0/22
216.33.240.0/22
216.200.206.0/24
204.95.96.0/20
#spf-d.hotmail.com
65.59.232.0/23
65.59.234.0/24
209.1.15.0/24
64.41.193.0/24
216.34.51.0/24
#_spf-a.microsoft.com
213.199.128.139
213.199.128.145
207.46.50.72
207.46.50.82
#_spf-b.microsoft.com
131.107.65.22
131.107.65.131
131.107.1.101
131.107.1.102
217.77.141.52
217.77.141.59
#_spf-c.microsoft.com
131.107.1.18
131.107.1.19
131.107.1.20
131.107.70.12
131.107.70.16
#s._spf.ebay.com.
66.135.209.192/27
66.135.197.0/27
64.4.240.64/27
64.4.244.64/27
#m._spf.ebay.com
66.135.215.224/27
216.33.244.96/27
216.33.244.84
#p._spf.ebay.com
67.72.99.26
206.165.246.83
206.165.246.84
206.165.246.85
206.165.246.86
64.127.115.252
194.64.234.129/27
#c._spf.ebay.com
12.155.144.75
62.22.61.131
63.104.149.126
64.68.79.253
64.94.204.222
66.135.215.134
67.72.12.29

#make /var/mail/blacklist.txt
1.2.3.4 My-black

#make /var/mail/blackmsg.txt
SPAM. Your address %A is in my blacklist

# in postfix main.cf
strict_rfc821_envelopes = yes
smtpd_helo_required = yes
smtpd_delay_reject = yes

smtpd_recipient_restrictions =
warn_if_reject reject_unknown_client,
reject_invalid_hostname,
reject_non_fqdn_hostname,
reject_non_fqdn_sender,
reject_non_fqdn_recipient,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
permit_mynetworks,
reject_unauth_destination,
reject_rhsbl_sender dsn.rfc-ignorant.org
reject_rhsbl_sender bogusmx.rfc-ignorant.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client sbl-xbl.spamhaus.org,
reject_rbl_client dnsbl.sorbs.net,
reject_rbl_client list.dsbl.org
reject_rbl_client relays.ordb.org,
permit

smtpd_data_restrictions =
reject_unauth_pipelining,
permit

relay_domains = $mydestination /usr/local/etc/postfix/relay_domains.txt
smtp_recipient_restrictions = permit_mynetworks reject_unauth_destination
relay_recipient_maps = hash:/usr/local/etc/postfix/relay_recipients
transport_maps = hash:/usr/local/etc/postfix/transport

#/usr/local/etc/postfix/relay_domains.txt
example.com
another.net
three.org

#/usr/local/etc/postfix/relay_recipients
@example.com x
@another.net x
justme@three.org x

#/usr/local/etc/postfix/transport
example.com relay:[mail.example.com]
another.net relay:[mail.example.com]
three.org relay:[mail.three.org]

#do
sudo postmap /usr/local/etc/postfix/relay_recipients
sudo postmap /usr/local/etc/postfix/transports

#put spamd into rc.conf
pfspamd_enable="YES"
pfspamd_flags="-v -4 -g -G25:4:864 -s10"

2006-09-02

First Post

Great more data smog.