Migrate a Subversion Sub Repository

Migrate SVN


Instructions to migrate a sub repository to it's own repository.

Get a copy of the repo:
    cp -r me@myserver:/usr/local/data/svn/myrepo .
Dump it:

svnadmin dump --quiet myrepo > myrepo.DUMP
Filter the hunk you want:

cat myrepo.DUMP | svndumpfilter include MySubProject > MySubProject.DUMP|more

copy it to the new server (omit this if doing it on the new server):

scp MySubProject.DUMP robrien@myotherserver:~/


ssh robrien@myotherserver

Create a new repo:

sudo ./bin/svnadmin create /usr/local/repos/MySubProject
sudo chown -R www:www /usr/local/repos/

Load the repo:

    sudo svnadmin load --quiet /usr/local/repos/MySubProject < /usr/home/robrien/MySubProject.DUMP



Build PIL on Mac OSX

This entry attempts to give instructions for how to build the Python Imaging Library (PIL) on Mac OSX.

#download and untar the jpeg library

#see here for location http://www.ijg.org/
curl http://path/to/jpegsrc.v6b.tar.gz | tar zxf -
#change directory to
cd jpeg-6b
#run configure
./configure CFLAGS='-fPIC'
#run make
#Run make install
sudo make install
#run ranlib
ranlib libjpeg.a
#copy the lib file to lib
cp libjpeg.a /usr/local/lib
#copy headers to include
cp *.h /usr/local/include
#make this your real python
#get and untar the PIL source
curl http://path/to/Imaging-1.1.6.tar.gz | tar zxf -
# cd
cd Imaging-1.1.6
# copy setup out of it's way
mv setup.py setup.py.tmp
# set up a replacement command
# you can do edit setup.py by hand if you wish
SED_CMD="s|JPEG_ROOT = None|JPEG_ROOT = libinclude(\"/usr/local/lib\")|; s|ZLIB_ROOT = None|ZLIB_ROOT = libinclude(\"/usr/local/lib\")|;"
#replace the stock jpg locations with the new ones you made
cat setup.py.tmp | sed -e "$SED_CMD" > setup.py
#build it
$PYTHON_COMMAND setup.py build_ext -i
#test it
$PYTHON_COMMAND selftest.py
#install it
sudo $PYTHON_COMMAND setup.py install


Undoing transactions in a broken Plone/Zope

Today there was a situation where someone made a permission change that made Plone inaccessible. The ZMI was also inaccessible (by design (not mine)). We could get access to the filesystem, however. So after a couple attempts at undos through in debug....There were issues with permissions and RESPONSE objects and all the other Zope machinery. So I tried mounting a ZODB directly and undoing transactions. The following worked great for me:

filename = "/Users/reedobrien/instances/zeo/msrd/var/Data.fs"
import ZODB.FileStorage
import ZODB.DB
storage = ZODB.FileStorage.FileStorage(filename)
DB = ZODB.DB(storage)
txs = DB.undoLog()[::-1]
import transaction; transaction.commit()

Maybe the wrong way and YMMV, but it stayed in one piece for me.


Passwordless Unix Login

Create a key pair

mack:~ reedobrien$ ssh-keygen -t rsa -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/reedobrien/.ssh/id_rsa.
Your public key has been saved in /Users/reedobrien/.ssh/id_rsa.pub.
The key fingerprint is:

Add the public Key to the server where you want to authenticate

mack:~ reedobrien$ scp ~/.ssh/id_rsa.pub username@remoteserver:~/.ssh/id_rsa.pub
mack:~ reedobrien$ ssh -l username remoteserver
password: ********
remoteserver:~ username$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
remoteserver:~ username$ exit

NOTE: There are many more advanced features, configurations, identity management things that can be done not covered here.


mack:~ reedobrien$ ssh -l robrien remoteserver
Last login: Mon Aug 13 22:28:10 2007 from
Welcome to A Better World!
remoteserver:~ robrien$


ipython in zope3 zopectl debug

The solution in my earlier post doesn't fly in zope3. To get it to work in zope3 I modified:

if __name__ == '__main__':    db = startup()
del startup
from zope.app.debug import Debugger
debugger = app = Debugger.fromDatabase(db)
root = app.root()
del db
del Debugger
import IPython
IPython.Shell.IPShell(user_ns={'root': root,
'app': app,
'top' : app.root()

I haven't figured out much about how the debugger differs in Zope3. But it gives me top as app.root().

In [1]: print top.items()
<OOBTreeItems object at 0x355c598>

In [2]: app.root()['foo']
Out[2]: <zope.app.folder.folder.Folder object at 0x36dfef0>

In [3]: top
Out[3]: <zope.app.folder.folder.Folder object at 0x36a27f0>

In [4]: top['foo']
Out[4]: <zope.app.folder.folder.Folder object at 0x36df830>

In [5]: print [x for x in top.items()]
[(u'foo', <zope.app.folder.folder.Folder object at 0x36df830>)]


ipython in zopectl debug

A long time ago (6mos? a year) I got tired of manually typing support for history and tab completion into the zope debugger. So I just hooked the debug command up to read my .pythonrc file by importing user in the do_debug method of zopectl.py.
def do_debug(self, arg):
cmdline = self.get_startup_cmd(self.options.python + ' -i',
"import Zope2; app=Zope2.app(); import user;")
Today I tried getting ipython working as described in the plone docs. I also tried vanrees notes
and a couple others. I could not get it going though. So I decided to return to the hack above and extend it to load ipython.
def do_debug(self, arg):
cmdline = self.get_startup_cmd(self.options.python + ' -i',
"import Zope2; app=Zope2.app(); import user;
ns={'__name__':'msrd','app':app}; import IPython;
Poof, now debug comes up with ipython. Yay... What does that mean you may ask?

In [2]: ??app.msrd
Type: ImplicitAcquirerWrapper
Base Class:
String Form:
Namespace: Interactive
Length: 1
Docstring [source file open failed]:
Make PloneSite subclass CMFSite and add some methods.
This will be useful for adding more things later on.
I know this is probably the wrong way to do this, but nyah nyah. It is working now. I do welcome feedback...

As a note: I also have import user in my args. This picks up stuff from your file defined in PYTHONSTARTUP. But then if you already use PYTHONSTARTUP you probably already know that.


Plone site broken? Fix ir through the zodb

Today someone changed the ip on a dev box running an instance that had the ip as part of the url in CacheFu. So of course it won't load a page anymore. What to do? reinstall? nah. Let 's think for a second. Plone is a nice interface to Zope/CMF. Zope/CMF is a lovely interface to ZODB. ZODB is a persistence engine for Python objects. CacheFu's settings are persisted as atributes of a cache object in the ZODB. Let's just fix it directly....

/usr/local/{instance}/bin/zopectl debug
Starting debugger (the name "app" is bound to the top-level Zope object)
...{ 8< snip } ...
>>> import readline, rlcompleter, transaction
>>> readline.parse_and_bind("tab: complete")
# now we have tab completion
# bind the plone root to s
>>> s = app.itcd
# bind our (s)ites cache settings to cc
>>> cc = s.portal_cache_settings
>>> cc.getDomains()
('',) ## <<== look there is our old value!!
# yup it needs to be this new value
>>> cc.setDomains("")
# and persisted. Otherwise it will be aborted at exit
>>> transaction.commit()

That was easy, no?


Quick and dirty regexen search for spamd & postfix logs

#!/usr/bin/env python
# Name: finder.py
# Purpose: This is a script to search through Posfix
# and spamd logs for the last x days and return
# hits. Mostly it is a quick and dirty way to
# find entris regarding emails blocked by
# spamd/postfix
# Author: Reed L. O'Brien reed at reedobrien com
# Created: 2007-01-05
# Modified: 2007-01-05
# Copyright: (c) Reed L. O'Brien 2007
# License: DWYWWI (improvements welcome)

#Do the imports
import os, re, bz2, sys, time

# make sure there is a regex
# compile the regex
regx = re.compile(sys.argv[1], re.IGNORECASE)
except IndexError:
print """
finder [days back to search]

ex: finder foobar 2
will find all occurences of 'foobar' in the last 2
days of spamd and maillog files.\n\tThe number of days
is optional and defaults to 1 if not given."""

# empty list to store hits in
found = []

# move to the log directory
# Start a counter for the number counted
s = 0

#get and set days
days = int(sys.argv[2])
days = 1
# Get a list of qualifying files NOTE: you may need stat(x).[st_ctime|st_mtime] depending on your OS
L = [f for f in os.listdir('.')
if os.stat(f).st_birthtime > time.time() - (days * 86400)
and (f.startswith('spamd') or f.startswith('maillog'))]
# get a count of how many files to search
n = len(L)

# start a loop on the list
for f in L:
# If it is a bz2 open it as a bz2 object
if (f.startswith('spamd') or f.startswith('maillog')) and f.endswith('2'):
# tell em what is happening
sys.stdout.write("\rsearching: %2s remain %s " % (f,n))
# set a line count
c = 1
# get a handle on the file
handle = bz2.BZ2File(f)
# iterate through the lines
for line in handle:
# if the regex is found
if regx.search(line):
# append the filename, line count and line content to the found list

found.append("%-10s : %s\n%s" % (c, f, line))
# increment the line count
c += 1
# or just increment the count if no regex match
c += 1
# decrement the number of files remaining
n -= 1
# increment the number of files searched
s += 1

## DO the same as above as a regular file object if not a bz2 file SEE NOTES FOR last loop
if (f.startswith('spamd') or f.startswith('maillog')) and not f.endswith('2'):
sys.stdout.write("\rsearching: %2s remain %s " % (f,n))
c = 1
handle = open(f)
for line in handle:
if regx.search(line):
found.append("%-10s : %s\n%s" % (c, f, line))
c += 1
c += 1
n -= 1
s += 1
##make some space to overwrite the sys.stdout text
print '\n\n\n\n\n'
print 'Searched:', s # Print how many files were searched

#print the results from the found list.
for x in found:
print x