Independent Suspension

Using a TCP proxy to bypass firewall

2008-05-26T03:18:00.000-04:00

In the middle of a month long holiday in Africa, I finally am at a place where I can connect my notebook to the internet. I have been able to get to my mail via webmail, but I don't have all the filters I like set up on the webmail system, so it is still way to noisy to get past all the mailing lists...

IMAP is a blessing. Except when the provider firewall blocks port 993. In this particular case it doesn't block 10993 though, so I decided to set up a proxy which would get me my mail. It was really quite simple.

So this is an example of how to use balance on freebsd 7 to set up a proxy for a TCP service. In my case i am proxying imaps via an unprivileged port on my utility server to the imaps port on the actual mail server.

I connected to my freebsd box over ublocked ssh and...


reed@host $ cd /usr/ports/net/balance
reed@host $ sudo make install

I did a quick connection test like so:


reed@host $ sudo balance -fp 10993 mail.serverbox.com:993

and a quick telnet confirmed connection to the destination so I then added some config to rc.conf:


### balance                                                                                                                             
balance_enable="YES"
balance_hosts="host1"
balance_host1_address="my.proxy.box.com"
balance_host1_ports="10993"
balance_host1_targets="mail.server.box.com:imaps"

And I am now receiving and filtering email with Mail...

Cake

create an installer dmg of a program

2008-04-14T11:53:00.000-04:00

Open the PackageMaker program.
Click "Assist me..."
Add the files you want to install and give them a destination on the destination drive.
Select files appropriate for a Welcome page, README, license, and conclusion.
Add a bg image if so desired.

Save and move to the directory in a terminal then

$hdiutil create Subversion-1.4.3 -srcFolder ~/projects/Subversion-1.4.3.mpkg

viola!

Aquamacs and Python

2008-04-14T11:21:00.000-04:00

Aquamacs with Python completion and ipython

Install Pymacs (python setup.py install)
Make sure you have pymac-services and rebox in your PATH. They install where the python binaries are.
Install python-mode.el in your site-lisp (/Library/Application\ Support/Emacs/site-lisp (create if necessary))

from the python mode folder copy pycomplete.py to your python site-packages (or otherwise add it to your python path.) and the .el files to your site-lisp directory

Install ipython
edit your .emacs

(your path may differ, as well as your mileage)

(setq auto-mode-alist (cons '("\\.py$" . python-mode) auto-mode-alist))
(setq interpreter-mode-alist (cons '("python" . python-mode) interpreter-mode-alist))
(autoload 'python-mode "python-mode" "Python editing mode." t)
(autoload 'pymacs-load "pymacs" nil t)
(autoload 'pymacs-eval "pymacs" nil t)
(autoload 'pymacs-apply "pymacs")
(autoload 'pymacs-call "pymacs")
(require 'pycomplete)
(setq ipython-command "/Library/Frameworks/Python.framework/Versions/Current/bin/ipython")
(require 'ipython)

Mapping specific requests to a single ZEO client instance

2008-04-08T21:18:00.000-04:00

As a follow up to Apache Round Robin for ZeoClients:

we needed to send session specific things to a single instance.

An example of why this is necessary is for collective.captcha. If the connections are being round robin-ed you get audio and image files from different instances. That is BAD. It doesn't work.

First make sure varnish doesn't cache the captcha images and audio or your captcha protected form page.
In varnish.vcl add:


# Do NOT cache captcha image, audio, request
  if (   req.url ~ "/@@captcha"
      || req.url ~ "/contact-info"
     ) {
        pipe;
       }

Next we need to map a server to do our captcha stuff.
In zopeservers.map add the line:


CAPTCHA localhost:6970

I will leave it to you to think of how to creatively balance requests to the other instances. Perhaps remove teh one that does some captcha and other special work from ALL, or maybe add the other instances more than once... OK I didn't leave much, but there are probably smarter ways to do it.

Finally we need to add a rule for mapping captcha requests to the correct instance.
To the appropriate apache.conf (http.conf) add:


RewriteRule ^/(@@captcha/(audio|image)|contact-info)   \
   http://${zopeservers:CAPTCHA}/VirtualHostBase/http/%{SERVER_NAME}:80/msrd/VirtualHostRoot/$1 [L,P]

Don't forget to restart apache and varnish!!

Thanks to Chris Shenton for suggesting the solution.

Error building psycopg custom egg

2008-02-26T18:04:00.000-05:00

I was running a buildout and got this error:


/psycopg/config.h:119: error: static declaration of 'round' follows non-static declaration

I know little to nothing about C so I asked Chris Shenton. He said, "Sounds like it is already defined. Try commenting it out." It worked.

I assume this is new in FreeBSD 7 which I am running. So I think this should work:


--- psycopg/config.h    2008-02-26 21:47:43 +0000
+++ psycopg/config.h    2008-02-26 21:56:06 +0000
@@ -113,7 +113,7 @@
 #define inline
 #endif


-#if defined(__FreeBSD__) || (defined(_WIN32) && !defined(__GNUC__)) || defined(__sun__)
+#if defined(__FREEBSD__ < 7) || (defined(_WIN32) && !defined(__GNUC__)) || defined(__sun__)
 /* what's this, we have no round function either? */
 static double round(double num)
 {

I went to file a bug at initd.org, but their system was down. So I put it here. Thanks for the clue Chris.

Apache Round Robin for ZeoClients

2008-01-15T18:21:00.000-05:00

There is a common approach to running Plone/Zope behind some service that distributes requests across numerous ZeoClients. Here's how I have been doing it using Apache (2.2.x).



                                  ________Zeoclient(6968)
                                 | _______ZeoClient(6969)
client-->varnish(80)-->apache(81)|<_______ZeoClient(6970)
                                 |________ZeoClient(6971)

Everybody knows how to setup varnish, right? So I am only going to show the Apache bits here.

First you need a rewrite map.


#zopeservers.map
ALL     localhost:6968|localhost:6969|localhost:6970|localhost:6917

Of course there can be more or less hosts in the map and they don't have to be all localhost either.

Next you need to enter the right stuff in the apache configuration. As here in a VirtualHost.


<VirtualHost *:81>
...
RewriteMap  zopeservers rnd:/usr/local/apache2/conf/zopeservers.map
RewriteRule ^/(.*) \
    http://${zopeservers:ALL}/VirtualHostBase/http/%{SERVER_NAME}:80/ploneroot/VirtualHostRoot/$1 [L,P]
...
</VirtualHost>

How does Emiril say... BAM!

Not so hard at all. A couple of notes:

1. I don't think apache knows how to detect if a client is down or broken. So it will serve up all the errors it gets.

2. Varnish doesn't cache https, so it needs to be handled wihtout varnish...

Seeded (Salted) SHA Passwords

2008-01-12T00:19:00.000-05:00

This week I needed to work with passwords from an OpenLDAP database. I needed to create users and encode their passwords as SSHA. After much googling and reading of authentication code examples, this is what I came up with.


import hashlib
import os
from base64 import urlsafe_b64encode as encode
from base64 import urlsafe_b64decode as decode

def makeSecret(password):
 salt = os.urandom(4)
 h = hashlib.sha1(password)
 h.update(salt)
 return "{SSHA}" + encode(h.digest() + salt)

def checkPassword(challenge_password, password):
 challenge_bytes = decode(challenge_password[6:])
 digest = challenge_bytes[:20]
 salt = challenge_bytes[20:]
 hr = hashlib.sha1(password)
 hr.update(salt)
 return digest == hr.digest()

Wow, python rules. So first I want to make sure I can validate a password against one generated by slappasswd


reedobrien$ slappasswd -s topsecret
{SSHA}Ccpjsip2UZL2CR2VsWTH7aF0vWKHQ7jn

And then see if I can validate it


>>> checkPassword('{SSHA}Ccpjsip2UZL2CR2VsWTH7aF0vWKHQ7jn',
...               'topsecret')
   True

Next I need to make sure I can generate one that OpenLDAP can use. So I used a script I wrote to create a user in ldap. This time I use the same password and get a different hash which is good.


>>> pw = "{SSHA}hq5ROYE8RoLA1Zcz6azgNQP5PkdETocx"

which OpenLDAP represents in LDIFs as base64 encoded


...
# tester, people, reedobrien.com
dn: uid=tester,ou=people,dc=reedobrien,dc=com
objectClass: top
objectClass: inetOrgPerson
uid: tester
cn: test test
sn: test
mail: test@example.com
userPassword:: e1NTSEF9MHhaMEdZc2Fob1JNeXZWR2FVdGszS0VwSFZTQnVLTlc=
...

So I try authenticating:


>>> userPasword = "e1NTSEF9MHhaMEdZc2Fob1JNeXZWR2FVdGszS0VwSFZTQnVLTlc="

>>> decode(userPassword) == pw
    True

>>> checkPassword(decode(userPassword), 'topsecret')
    True

Now I want to make sure it works through the ldap server itself
In python:



>>> import ldap

>>> con = ldap.initialize("ldap://127.0.0.1")
>>> con.simple_bind_s('uid=tester,ou=people,dc=reedbrien,dc=com',
                      'topsecret')
    (97, []) ## indicates success

And not using my stuff.


reedobrien$ ldapsearch -x -D "uid=tester,ou=people,dc=reedobrien,dc=com" -w topsecret
# extended LDIF
#
# LDAPv3
# base <> with scope sub
# filter: (objectclass=*)
# requesting: ALL
#

# search result
search: 2
result: 32 No such object

# numResponses: 1

How do they say, `w00t!`

So it was cool doing it and it made me feel smart for a minute. I am sure that there is a module or ten out there that already does this, but I couldn't find it. At least it was educational.

Unfortunately, after getting it to work; I don't I understand any better what it really does or how SSHA is better than plain old SHA.

Oh, well. Hopefully it helps some weary traveller out there one day.

Serving up a ZODB on demand from a repozo backup

2008-01-11T16:07:00.000-05:00

Since I said I would on the mailing list/forum.

Here it is:


import os
import shutil
import tarfile
import tempfile

def application(environ, start_response):
  status = '200 OK'

  tempdir  = tempfile.mkdtemp()
  datafile = '%s/Data.fs' % tempdir
  tarball = '%s.tar.bz2' % os.path.join(tempdir,
                                        os.path.basename(datafile))

  ## If I were smarter or had more time
  ## I would import from repozo or at
  ## least use subprocess
  os.system("/usr/local/zope/2.9.7/bin/repozo.py -v -z -R -r \
         /usr/local/zope/sites/2.9.7/msrd/zeo/var/backup/ -o %s" % datafile)

  out = tarfile.TarFile.open(tarball, 'w:bz2')
  os.chdir(tempdir)
  out.add('Data.fs')
  out.close()


  response_headers = [('Content-type', 'application/octet-stream'),
                      ('Content-Length', str(len(open(tarball).read()))),
                      ('Content-Disposition',
                       'inline; filename="Data.fs.tar.bz2"')]

  start_response(status, response_headers)
  try:
      return [open(tarball, 'rb').read()]
  finally:
      shutil.rmtree(tempdir)

I did it really quick. But it solved my problem very well. It gets a copy of the Data.fs from the last repozo backup. So all developers don't need shell access to the server...

A couple things I could see doing that would make this cooler are:

Not using os.system
implement a method to do HEAD requests with the time of the last repozo delta so I don't compress and transfer this beast if there are no changes...

If you have mod_wsgi installed and working you just need to add something like:

WSGIScriptAlias /Data.fs.tar.bz2 /usr/local/apps/getDataArchived.py

to the appropriate configuration file.

Migrate a Subversion Sub Repository

2007-12-16T22:19:00.000-05:00

Migrate SVN

Description:

Instructions to migrate a sub repository to it's own repository.

Get a copy of the repo:

    cp -r me@myserver:/usr/local/data/svn/myrepo .

Dump it:


svnadmin dump --quiet myrepo > myrepo.DUMP

Filter the hunk you want:


cat myrepo.DUMP | svndumpfilter include MySubProject > MySubProject.DUMP|more

copy it to the new server (omit this if doing it on the new server):


scp MySubProject.DUMP robrien@myotherserver:~/

Logon:


 ssh robrien@myotherserver

Create a new repo:


 sudo ./bin/svnadmin create /usr/local/repos/MySubProject
 sudo chown -R www:www /usr/local/repos/

Load the repo:

    sudo svnadmin load --quiet /usr/local/repos/MySubProject < /usr/home/robrien/MySubProject.DUMP

Done.

Build PIL on Mac OSX

2007-10-29T21:26:00.000-04:00

This entry attempts to give instructions for how to build the Python Imaging Library (PIL) on Mac OSX.

#download and untar the jpeg library
#see here for location http://www.ijg.org/

curl http://path/to/jpegsrc.v6b.tar.gz | tar zxf -

#change directory to

cd jpeg-6b

#run configure

./configure CFLAGS='-fPIC'

#run make

make

#Run make install

sudo make install

#run ranlib

ranlib libjpeg.a

#copy the lib file to lib

cp libjpeg.a /usr/local/lib

#copy headers to include

cp *.h /usr/local/include

#make this your real python

PYTHON_COMMAND="/path/to/your/python"

#get and untar the PIL source

curl http://path/to/Imaging-1.1.6.tar.gz | tar zxf -

# cd

cd Imaging-1.1.6

# copy setup out of it's way

mv setup.py setup.py.tmp

# set up a replacement command
# you can do edit setup.py by hand if you wish

SED_CMD="s|JPEG_ROOT = None|JPEG_ROOT = libinclude(\"/usr/local/lib\")|; s|ZLIB_ROOT = None|ZLIB_ROOT = libinclude(\"/usr/local/lib\")|;"

#replace the stock jpg locations with the new ones you made

cat setup.py.tmp | sed -e "$SED_CMD" > setup.py

#build it

$PYTHON_COMMAND setup.py build_ext -i

#test it

$PYTHON_COMMAND selftest.py

#install it

sudo $PYTHON_COMMAND setup.py install

Undoing transactions in a broken Plone/Zope

2007-10-24T14:34:00.000-04:00

Today there was a situation where someone made a permission change that made Plone inaccessible. The ZMI was also inaccessible (by design (not mine)). We could get access to the filesystem, however. So after a couple attempts at undos through in debug....There were issues with permissions and RESPONSE objects and all the other Zope machinery. So I tried mounting a ZODB directly and undoing transactions. The following worked great for me:


filename = "/Users/reedobrien/instances/zeo/msrd/var/Data.fs"
import ZODB.FileStorage
import ZODB.DB
storage = ZODB.FileStorage.FileStorage(filename)
DB = ZODB.DB(storage)
txs = DB.undoLog()[::-1]
DB.undo(id=txs[0]['id'])
import transaction; transaction.commit()

Maybe the wrong way and YMMV, but it stayed in one piece for me.

Passwordless Unix Login

2007-08-14T10:26:00.000-04:00

Create a key pair

mack:~ reedobrien$ ssh-keygen -t rsa -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/reedobrien/.ssh/id_rsa.
Your public key has been saved in /Users/reedobrien/.ssh/id_rsa.pub.
The key fingerprint is:
bd:6c:18:e3:1a:46:5f:3d:d9:95:ed:82:6b:ea:6b:ec

Add the public Key to the server where you want to authenticate

mack:~ reedobrien$ scp ~/.ssh/id_rsa.pub username@remoteserver:~/.ssh/id_rsa.pub
mack:~ reedobrien$ ssh -l username remoteserver
password: ********
remoteserver:~ username$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
remoteserver:~ username$ exit

NOTE: There are many more advanced features, configurations, identity management things that can be done not covered here.

Test

mack:~ reedobrien$ ssh -l robrien remoteserver
Last login: Mon Aug 13 22:28:10 2007 from 131.182.85.5
Welcome to A Better World!
remoteserver:~ robrien$

ipython in zope3 zopectl debug

2007-08-02T23:32:00.000-04:00

The solution in my earlier post doesn't fly in zope3. To get it to work in zope3 I modified:
~/instances/z3/bin/debugzope

if __name__ == '__main__':    db = startup()
     del startup
     from zope.app.debug import Debugger
     debugger = app = Debugger.fromDatabase(db)
     root = app.root()
     del db
     del Debugger
     import IPython
     IPython.Shell.IPShell(user_ns={'root': root,
                                     'app': app,                         
                                    'top' : app.root() 
                                     }).mainloop(sys_exit=1)

I haven't figured out much about how the debugger differs in Zope3. But it gives me top as app.root().


In [1]: print top.items()
<OOBTreeItems object at 0x355c598>

In [2]: app.root()['foo']
Out[2]: <zope.app.folder.folder.Folder object at 0x36dfef0>

In [3]: top
Out[3]: <zope.app.folder.folder.Folder object at 0x36a27f0>

In [4]: top['foo']
Out[4]: <zope.app.folder.folder.Folder object at 0x36df830>

In [5]: print [x for x in top.items()]
[(u'foo', <zope.app.folder.folder.Folder object at 0x36df830>)]

ipython in zopectl debug

2007-07-01T15:56:00.000-04:00

A long time ago (6mos? a year) I got tired of manually typing support for history and tab completion into the zope debugger. So I just hooked the debug command up to read my .pythonrc file by importing user in the do_debug method of zopectl.py.

def do_debug(self, arg):
   cmdline = self.get_startup_cmd(self.options.python + ' -i',
   "import Zope2; app=Zope2.app(); import user;")

Today I tried getting ipython working as described in the plone docs. I also tried vanrees notes

and a couple others. I could not get it going though. So I decided to return to the hack above and extend it to load ipython.

def do_debug(self, arg):
    cmdline = self.get_startup_cmd(self.options.python + ' -i',
    "import Zope2; app=Zope2.app(); import user;     
     ns={'__name__':'msrd','app':app}; import IPython;   
     IPython.Shell.IPShell(user_ns=ns).mainloop(sys_exit=1);")

Poof, now debug comes up with ipython. Yay... What does that mean you may ask?

In [2]: ??app.msrd
Type: ImplicitAcquirerWrapper
Base Class:
String Form:
Namespace: Interactive
Length: 1
Docstring [source file open failed]:
Make PloneSite subclass CMFSite and add some methods.
This will be useful for adding more things later on.

I know this is probably the wrong way to do this, but nyah nyah. It is working now. I do welcome feedback...

As a note: I also have import user in my args. This picks up stuff from your file defined in PYTHONSTARTUP. But then if you already use PYTHONSTARTUP you probably already know that.

Plone site broken? Fix ir through the zodb

2007-01-19T20:23:00.000-05:00

Today someone changed the ip on a dev box running an instance that had the ip as part of the url in CacheFu. So of course it won't load a page anymore. What to do? reinstall? nah. Let 's think for a second. Plone is a nice interface to Zope/CMF. Zope/CMF is a lovely interface to ZODB. ZODB is a persistence engine for Python objects. CacheFu's settings are persisted as atributes of a cache object in the ZODB. Let's just fix it directly....

/usr/local/{instance}/bin/zopectl debug
Starting debugger (the name "app" is bound to the top-level Zope object)
...{ 8< snip } ...
>>> import readline, rlcompleter, transaction
>>> readline.parse_and_bind("tab: complete")
# now we have tab completion
# bind the plone root to s
>>> s = app.itcd
# bind our (s)ites cache settings to cc
>>> cc = s.portal_cache_settings
>>> cc.getDomains()
('http://1.2.3.4:80',) ## <<== look there is our old value!!
# yup it needs to be this new value
>>> cc.setDomains("http://4.3.2.1:8080")
# and persisted. Otherwise it will be aborted at exit
>>> transaction.commit()

That was easy, no?

Quick and dirty regexen search for spamd & postfix logs

2007-01-05T19:51:00.000-05:00


#!/usr/bin/env python
#-------------------------------------------------------------
# Name:        finder.py
# Purpose:     This is a script to search through  Posfix 
#              and spamd logs for the last x days and return 
#              hits. Mostly it is a quick and dirty way to 
#              find entris regarding emails blocked by 
#              spamd/postfix   
# Author:      Reed L. O'Brien reed at reedobrien com
#                                                                                                           
# Created:     2007-01-05
# Modified:    2007-01-05
# Copyright:   (c) Reed L. O'Brien 2007
# License:     DWYWWI (improvements welcome)                         
#--------------------------------------------------------------                      


#Do the imports
import os, re, bz2, sys, time

# make sure there is a regex
try:
    # compile the regex
  regx = re.compile(sys.argv[1], re.IGNORECASE)
except IndexError:
  print """
  usage:
         finder  [days back to search]

         ex: finder foobar 2
         will find all occurences of 'foobar' in the last 2
         days of spamd and maillog files.\n\tThe number of days
         is optional and defaults to 1 if not given."""
  sys.exit(0)

# empty list to store hits in
found = []

# move to the log directory
os.chdir('/var/log')
# Start a counter for the number counted
s = 0

#get and set days
try:
 days = int(sys.argv[2])
except:
 days = 1
# Get a list of qualifying files NOTE: you may need stat(x).[st_ctime|st_mtime] depending on your OS
L = [f for f in os.listdir('.')
   if os.stat(f).st_birthtime > time.time() - (days * 86400)
   and (f.startswith('spamd') or f.startswith('maillog'))]
# get a count of how many files to search
n = len(L)

# start a loop on the list
for f in L:
    # If it is a bz2 open it as a bz2 object
if (f.startswith('spamd') or f.startswith('maillog')) and f.endswith('2'):
 # tell em what is happening
   sys.stdout.write("\rsearching: %2s  remain %s                   " % (f,n))
   sys.stdout.flush()
        # set a line count
   c = 1
        # get a handle on the file
        handle = bz2.BZ2File(f)
        # iterate through the lines
        for line in handle:
        # if the regex is found
       if regx.search(line):
           # append the filename, line count and line content to the found list
           found.append("%-10s : %s\n%s" % (c, f, line))
                # increment the line count
           c += 1
       else:
           # or just increment the count if no regex match 
           c += 1
                # decrement the number of files remaining
           n -= 1  
                # increment the number of files searched
           s += 1

    ## DO the same as above as a regular file object if not a bz2 file SEE NOTES FOR last loop
if (f.startswith('spamd') or f.startswith('maillog')) and not f.endswith('2'):
   sys.stdout.write("\rsearching: %2s  remain %s                     " % (f,n))
   sys.stdout.flush()
   c = 1
   handle = open(f)
   for line in handle:
       if regx.search(line):
           found.append("%-10s : %s\n%s" % (c, f, line))
           c += 1
       else:
           c += 1
           n -= 1
           s += 1
                ##make some space to overwrite the sys.stdout text 
print '\n\n\n\n\n'
print 'Searched:', s # Print how many files were searched

#print the results from the found list.
for x in found:
print x

Mostly functional email regex

2006-12-28T13:46:00.000-05:00

for word based delims in text:
em = re.compile(r"\b['A-Z0-9._%-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b",
   re.IGNORECASE)


for start and endline delims:
em = re.compile(r"^['A-Z0-9._%-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$",
    re.IGNORECASE)

less spam with spamd and pf

2006-12-14T15:02:00.000-05:00

untested but should be pretty close

##Assumes Freebsd built with postfix and pf:
##requires spamd

#Add line to /etc/fstab
# Device Mountpoint FStype Options Dump Pass
fdescfs /dev/fd fdecfs rw 0 0

# mount it
sudo mount -a

## build spamd
cd /usr/ports/mail/spamd && sudo make install

#answer yes
This system has no entry for spamd in /etc/services
Would you like to add it automatically? (y/n) [y]? y
This system has no entry for spamd-cfg in /etc/services
Would you like to add it automatically? (y/n) [y]? y

# setup spamd.conf
all:\
:spamhaus:spamhausDROP:whitelist:spews1:spews2:china:korea:becks:blacklist:

# Mirrored from http://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2
spamhaus:\
:black:\
:msg="SPAM. Your address %A is in the Spamhaus Block List\n\
See http://www.spamhaus.org/sbl and\
http://www.abuse.net/sbl.phtml?IP=%A for more details":\
:method=http:\
:file=www.openbsd.org/spamd/SBL.cidr.gz:

spamhausDROP:\
:black:\
:msg="SPAM. Your address %A is in the Spamhaus DROP List\n\
See http://www.spamhaus.org/sbl and\
http://www.abuse.net/sbl.phtml?IP=%A for more details":\
:method=http:\
:file=www.spamhaus.org/DROP/drop.lasso:

becks:\ # experimental, probably blocks some good ips
:black:\
:msg="SPAM. Your address %A has sent spam within the last 24 hours":\
:method=http:\
:file=www.openbsd.org/spamd/traplist.gz

# Mirrored from http://www.spews.org/spews_list_level1.txt
spews1:\
:black:\
:msg="SPAM. Your address %A is in the spews level 1 database\n\
See http://www.spews.org/ask.cgi?x=%A for more details":\
:method=http:\
:file=www.openbsd.org/spamd/spews_list_level1.txt.gz:

# Mirrored from http://www.spews.org/spews_list_level2.txt
spews2:\
:black:\
:msg="SPAM. Your address %A is in the spews level 2 database\n\
See http://www.spews.org/ask.cgi?x=%A for more details":\
:method=http:\
:file=www.openbsd.org/spamd/spews_list_level2.txt.gz:

# Mirrored from http://www.okean.com/chinacidr.txt
china:\
:black:\
:msg="SPAM. Your address %A appears to be from China\n\
See http://www.okean.com/asianspamblocks.html for more details":\
:method=http:\
:file=www.openbsd.org/spamd/chinacidr.txt.gz:

# Mirrored from http://www.okean.com/koreacidr.txt
korea:\
:black:\
:msg="SPAM. Your address %A appears to be from Korea\n\
See http://www.okean.com/asianspamblocks.html for more details":\
:method=http:\
:file=www.openbsd.org/spamd/koreacidr.txt.gz:

# Whitelists are done like this, and must be added to "all" after each
# blacklist from which you want the addresses in the whitelist removed.
#
whitelist:\
:white:\
:file=/var/mail/whitelist.txt:

blacklist:\
:black:\
:msg=/var/mail/blackmsg.txt:\
:method=file:\
:file=/var/mail/blacklist.txt:

# touch the spamd and chown it
sudo touch /var/db/spamd && sudo chown nobody:wheel /var/db/spamd

#create the inital log file
sudo touch /var/log/spamd

# make spamd log in it's own log in /etc/syslog.conf
!spamd
*.* /var/log/spamd

#restart it
sudo /etc/rc.d/syslogd restart

# make it rotate IN /etc/newsyslog.conf
/var/log/spamd 640 1000 * @T00 JC

#edit pf.conf
# macros
nic = "xl0"

tcp_services = "{ 22, 25, 8080}"
icmp_types = "echoreq"
udp_services = "{ 123, 53 }"
web_service = "{ 80 }"
mail_host = "127.0.0.1"

priv_nets = "{ 127.0.0.0/8, 172.16.0.0/12, 10.0.0.0/8 }"
my_nets = "{ 192.168.68.0/24}"

# options
set block-policy return
set loginterface $nic
set skip on lo0

# scrub
scrub in all

#########
## Spamd
#########

# grey host list
table persist

#white host list
table persist
table persist file "/var/mail/whitelist.txt"

# forward white listed ips
rdr pass on $nic inet proto tcp from to $nic port smtp -> 127.0.0.1 port 8025

rdr pass on $nic proto tcp from to $nic port smtp -> $mail_host port smtp

rdr pass on $nic proto tcp from to $nic port smtp -> $mail_host port smtp

# send all suspects to the spamd daemon

rdr pass on $nic inet proto tcp from ! to $nic port smtp -> 127.0.0.1 port 8025

rdr pass on $nic inet proto tcp from any to $nic port smtp -> $mail_host port smtp

# filter rules
block all

block drop in quick on $nic from $priv_nets to any
block drop out quick on $nic from any to $priv_nets

pass in on $nic inet proto tcp from any to $nic port smtp flags S/SA keep state

pass in on $nic inet proto tcp from $my_nets to $nic port $tcp_services flags S/SA keep state

pass in on $nic inet proto tcp from any to $nic port $web_service flags S/SA keep state

pass in on $nic inet proto udp from any to $nic port $udp_services

pass in on $nic inet proto udp from any to $nic port 123

pass in inet proto icmp all icmp-type $icmp_types keep state

pass out on $nic proto tcp all modulate state flags S/SA
pass out on $nic proto { udp, icmp } all keep state

#make /var/mail/whitelist.txt
#FDS
192.251.225.192/26
#apple
17.0.0.0/8
#aol.com
152.163.225.0/24
205.188.139.0/24
205.188.144.0/24
205.188.156.0/23
205.188.159.0/24
64.12.136.0/23
64.12.138.0/24
152.163.225.0/24
205.188.139.0/24
205.188.144.0/24
205.188.156.0/23
205.188.159.0/24
64.12.136.0/23
64.12.138.0/24
#amazon.com
207.171.160.0/19
87.238.80.0/21
72.21.196.0/24
72.21.208.0/24
207.171.160.32/28
207.171.180.176/28
207.171.164.32/28
207.171.190.0/28
87.238.80.24/29
87.238.84.24/29
72.21.196.0/24
72.21.208.0/24
#_spf.google.com
216.239.56.0/23
64.233.160.0/19
66.249.80.0/20
72.14.192.0/18
#spf-a.hotmail.com
209.240.192.0/19
65.52.0.0/14
131.107.0.0/16
157.54.0.0/15
157.56.0.0/14
157.60.0.0/16
167.220.0.0/16
204.79.135.0/24
204.79.188.0/24
204.79.252.0/24
207.46.0.0/16
199.2.137.0/24
#spf-b.hotmail.com
199.103.90.0/23
204.182.144.0/24
204.255.244.0/23
206.138.168.0/21
64.4.0.0/18
65.54.128.0/17
207.68.128.0/18
207.68.192.0/20
207.82.250.0/23
207.82.252.0/23
209.1.112.0/23
#spf-c.hotmail.com
209.185.128.0/23
209.185.130.0/23
209.185.240.0/22
216.32.180.0/22
216.32.240.0/22
216.33.148.0/22
216.33.151.0/24
216.33.236.0/22
216.33.240.0/22
216.200.206.0/24
204.95.96.0/20
#spf-d.hotmail.com
65.59.232.0/23
65.59.234.0/24
209.1.15.0/24
64.41.193.0/24
216.34.51.0/24
#_spf-a.microsoft.com
213.199.128.139
213.199.128.145
207.46.50.72
207.46.50.82
#_spf-b.microsoft.com
131.107.65.22
131.107.65.131
131.107.1.101
131.107.1.102
217.77.141.52
217.77.141.59
#_spf-c.microsoft.com
131.107.1.18
131.107.1.19
131.107.1.20
131.107.70.12
131.107.70.16
#s._spf.ebay.com.
66.135.209.192/27
66.135.197.0/27
64.4.240.64/27
64.4.244.64/27
#m._spf.ebay.com
66.135.215.224/27
216.33.244.96/27
216.33.244.84
#p._spf.ebay.com
67.72.99.26
206.165.246.83
206.165.246.84
206.165.246.85
206.165.246.86
64.127.115.252
194.64.234.129/27
#c._spf.ebay.com
12.155.144.75
62.22.61.131
63.104.149.126
64.68.79.253
64.94.204.222
66.135.215.134
67.72.12.29

#make /var/mail/blacklist.txt
1.2.3.4 My-black

#make /var/mail/blackmsg.txt
SPAM. Your address %A is in my blacklist

# in postfix main.cf
strict_rfc821_envelopes = yes
smtpd_helo_required = yes
smtpd_delay_reject = yes

smtpd_recipient_restrictions =
warn_if_reject reject_unknown_client,
reject_invalid_hostname,
reject_non_fqdn_hostname,
reject_non_fqdn_sender,
reject_non_fqdn_recipient,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
permit_mynetworks,
reject_unauth_destination,
reject_rhsbl_sender dsn.rfc-ignorant.org
reject_rhsbl_sender bogusmx.rfc-ignorant.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client sbl-xbl.spamhaus.org,
reject_rbl_client dnsbl.sorbs.net,
reject_rbl_client list.dsbl.org
reject_rbl_client relays.ordb.org,
permit

smtpd_data_restrictions =
reject_unauth_pipelining,
permit

relay_domains = $mydestination /usr/local/etc/postfix/relay_domains.txt
smtp_recipient_restrictions = permit_mynetworks reject_unauth_destination
relay_recipient_maps = hash:/usr/local/etc/postfix/relay_recipients
transport_maps = hash:/usr/local/etc/postfix/transport

#/usr/local/etc/postfix/relay_domains.txt
example.com
another.net
three.org

#/usr/local/etc/postfix/relay_recipients
@example.com x
@another.net x
justme@three.org x

#/usr/local/etc/postfix/transport
example.com relay:[mail.example.com]
another.net relay:[mail.example.com]
three.org relay:[mail.three.org]

#do
sudo postmap /usr/local/etc/postfix/relay_recipients
sudo postmap /usr/local/etc/postfix/transports

#put spamd into rc.conf
pfspamd_enable="YES"
pfspamd_flags="-v -4 -g -G25:4:864 -s10"

First Post

2006-09-02T15:23:00.000-04:00

Great more data smog.