Using a TCP proxy to bypass firewall

In the middle of a month long holiday in Africa, I finally am at a place where I can connect my notebook to the internet. I have been able to get to my mail via webmail, but I don't have all the filters I like set up on the webmail system, so it is still way to noisy to get past all the mailing lists...

IMAP is a blessing. Except when the provider firewall blocks port 993. In this particular case it doesn't block 10993 though, so I decided to set up a proxy which would get me my mail. It was really quite simple.

So this is an example of how to use balance on freebsd 7 to set up a proxy for a TCP service. In my case i am proxying imaps via an unprivileged port on my utility server to the imaps port on the actual mail server.

I connected to my freebsd box over ublocked ssh and...

reed@host $ cd /usr/ports/net/balance
reed@host $ sudo make install

I did a quick connection test like so:

reed@host $ sudo balance -fp 10993 mail.serverbox.com:993

and a quick telnet confirmed connection to the destination so I then added some config to rc.conf:

### balance

And I am now receiving and filtering email with Mail...



create an installer dmg of a program

Open the PackageMaker program.
Click "Assist me..."
Add the files you want to install and give them a destination on the destination drive.
Select files appropriate for a Welcome page, README, license, and conclusion.
Add a bg image if so desired.

Save and move to the directory in a terminal then

$hdiutil create Subversion-1.4.3 -srcFolder ~/projects/Subversion-1.4.3.mpkg


Aquamacs and Python

Aquamacs with Python completion and ipython

  1. Install Pymacs (python setup.py install)
  2. Make sure you have pymac-services and rebox in your PATH. They install where the python binaries are.
  3. Install python-mode.el in your site-lisp (/Library/Application\ Support/Emacs/site-lisp (create if necessary))

    1. from the python mode folder copy pycomplete.py to your python site-packages (or otherwise add it to your python path.) and the .el files to your site-lisp directory

  4. Install ipython
  5. edit your .emacs

(your path may differ, as well as your mileage)

(setq auto-mode-alist (cons '("\\.py$" . python-mode) auto-mode-alist))
(setq interpreter-mode-alist (cons '("python" . python-mode) interpreter-mode-alist))
(autoload 'python-mode "python-mode" "Python editing mode." t)
(autoload 'pymacs-load "pymacs" nil t)
(autoload 'pymacs-eval "pymacs" nil t)
(autoload 'pymacs-apply "pymacs")
(autoload 'pymacs-call "pymacs")
(require 'pycomplete)
(setq ipython-command "/Library/Frameworks/Python.framework/Versions/Current/bin/ipython")
(require 'ipython)


Mapping specific requests to a single ZEO client instance

As a follow up to Apache Round Robin for ZeoClients:

we needed to send session specific things to a single instance.

An example of why this is necessary is for collective.captcha. If the connections are being round robin-ed you get audio and image files from different instances. That is BAD. It doesn't work.

First make sure varnish doesn't cache the captcha images and audio or your captcha protected form page.
In varnish.vcl add:

# Do NOT cache captcha image, audio, request
if ( req.url ~ "/@@captcha"
|| req.url ~ "/contact-info"
) {

Next we need to map a server to do our captcha stuff.
In zopeservers.map add the line:

CAPTCHA localhost:6970

I will leave it to you to think of how to creatively balance requests to the other instances. Perhaps remove teh one that does some captcha and other special work from ALL, or maybe add the other instances more than once... OK I didn't leave much, but there are probably smarter ways to do it.

Finally we need to add a rule for mapping captcha requests to the correct instance.
To the appropriate apache.conf (http.conf) add:

RewriteRule ^/(@@captcha/(audio|image)|contact-info) \
http://${zopeservers:CAPTCHA}/VirtualHostBase/http/%{SERVER_NAME}:80/msrd/VirtualHostRoot/$1 [L,P]

Don't forget to restart apache and varnish!!

Thanks to Chris Shenton for suggesting the solution.


Error building psycopg custom egg

I was running a buildout and got this error:

/psycopg/config.h:119: error: static declaration of 'round' follows non-static declaration

I know little to nothing about C so I asked Chris Shenton. He said, "Sounds like it is already defined. Try commenting it out." It worked.

I assume this is new in FreeBSD 7 which I am running. So I think this should work:

--- psycopg/config.h 2008-02-26 21:47:43 +0000
+++ psycopg/config.h 2008-02-26 21:56:06 +0000
@@ -113,7 +113,7 @@
#define inline

-#if defined(__FreeBSD__) || (defined(_WIN32) && !defined(__GNUC__)) || defined(__sun__)
+#if defined(__FREEBSD__ < 7) || (defined(_WIN32) && !defined(__GNUC__)) || defined(__sun__)
/* what's this, we have no round function either? */
static double round(double num)

I went to file a bug at initd.org, but their system was down. So I put it here. Thanks for the clue Chris.


Apache Round Robin for ZeoClients

There is a common approach to running Plone/Zope behind some service that distributes requests across numerous ZeoClients. Here's how I have been doing it using Apache (2.2.x).

| _______ZeoClient(6969)

Everybody knows how to setup varnish, right? So I am only going to show the Apache bits here.

First you need a rewrite map.

ALL localhost:6968|localhost:6969|localhost:6970|localhost:6917

Of course there can be more or less hosts in the map and they don't have to be all localhost either.

Next you need to enter the right stuff in the apache configuration. As here in a VirtualHost.

<VirtualHost *:81>
RewriteMap zopeservers rnd:/usr/local/apache2/conf/zopeservers.map
RewriteRule ^/(.*) \
http://${zopeservers:ALL}/VirtualHostBase/http/%{SERVER_NAME}:80/ploneroot/VirtualHostRoot/$1 [L,P]

How does Emiril say... BAM!

Not so hard at all. A couple of notes:

1. I don't think apache knows how to detect if a client is down or broken. So it will serve up all the errors it gets.

2. Varnish doesn't cache https, so it needs to be handled wihtout varnish...


Seeded (Salted) SHA Passwords

This week I needed to work with passwords from an OpenLDAP database. I needed to create users and encode their passwords as SSHA. After much googling and reading of authentication code examples, this is what I came up with.

import hashlib
import os
from base64 import urlsafe_b64encode as encode
from base64 import urlsafe_b64decode as decode

def makeSecret(password):
salt = os.urandom(4)
h = hashlib.sha1(password)
return "{SSHA}" + encode(h.digest() + salt)

def checkPassword(challenge_password, password):
challenge_bytes = decode(challenge_password[6:])
digest = challenge_bytes[:20]
salt = challenge_bytes[20:]
hr = hashlib.sha1(password)
return digest == hr.digest()

Wow, python rules. So first I want to make sure I can validate a password against one generated by slappasswd

reedobrien$ slappasswd -s topsecret
And then see if I can validate it

>>> checkPassword('{SSHA}Ccpjsip2UZL2CR2VsWTH7aF0vWKHQ7jn',
... 'topsecret')

Next I need to make sure I can generate one that OpenLDAP can use. So I used a script I wrote to create a user in ldap. This time I use the same password and get a different hash which is good.

>>> pw = "{SSHA}hq5ROYE8RoLA1Zcz6azgNQP5PkdETocx"

which OpenLDAP represents in LDIFs as base64 encoded

# tester, people, reedobrien.com
dn: uid=tester,ou=people,dc=reedobrien,dc=com
objectClass: top
objectClass: inetOrgPerson
uid: tester
cn: test test
sn: test
mail: test@example.com
userPassword:: e1NTSEF9MHhaMEdZc2Fob1JNeXZWR2FVdGszS0VwSFZTQnVLTlc=

So I try authenticating:

>>> userPasword = "e1NTSEF9MHhaMEdZc2Fob1JNeXZWR2FVdGszS0VwSFZTQnVLTlc="

>>> decode(userPassword) == pw

>>> checkPassword(decode(userPassword), 'topsecret')

Now I want to make sure it works through the ldap server itself
In python:

>>> import ldap

>>> con = ldap.initialize("ldap://")
>>> con.simple_bind_s('uid=tester,ou=people,dc=reedbrien,dc=com',
(97, []) ## indicates success

And not using my stuff.

reedobrien$ ldapsearch -x -D "uid=tester,ou=people,dc=reedobrien,dc=com" -w topsecret
# extended LDIF
# LDAPv3
# base <> with scope sub
# filter: (objectclass=*)
# requesting: ALL

# search result
search: 2
result: 32 No such object

# numResponses: 1

How do they say, `w00t!`

So it was cool doing it and it made me feel smart for a minute. I am sure that there is a module or ten out there that already does this, but I couldn't find it. At least it was educational.

Unfortunately, after getting it to work; I don't I understand any better what it really does or how SSHA is better than plain old SHA.

Oh, well. Hopefully it helps some weary traveller out there one day.


Serving up a ZODB on demand from a repozo backup

Since I said I would on the mailing list/forum.

Here it is:

import os
import shutil
import tarfile
import tempfile

def application(environ, start_response):
status = '200 OK'

tempdir = tempfile.mkdtemp()
datafile = '%s/Data.fs' % tempdir
tarball = '%s.tar.bz2' % os.path.join(tempdir,

## If I were smarter or had more time
## I would import from repozo or at
## least use subprocess
os.system("/usr/local/zope/2.9.7/bin/repozo.py -v -z -R -r \
/usr/local/zope/sites/2.9.7/msrd/zeo/var/backup/ -o %s" % datafile)

out = tarfile.TarFile.open(tarball, 'w:bz2')

response_headers = [('Content-type', 'application/octet-stream'),
('Content-Length', str(len(open(tarball).read()))),
'inline; filename="Data.fs.tar.bz2"')]

start_response(status, response_headers)
return [open(tarball, 'rb').read()]

I did it really quick. But it solved my problem very well. It gets a copy of the Data.fs from the last repozo backup. So all developers don't need shell access to the server...

A couple things I could see doing that would make this cooler are:
  • Not using os.system
  • implement a method to do HEAD requests with the time of the last repozo delta so I don't compress and transfer this beast if there are no changes...

If you have mod_wsgi installed and working you just need to add something like:

WSGIScriptAlias /Data.fs.tar.bz2 /usr/local/apps/getDataArchived.py

to the appropriate configuration file.