Migrate a Subversion Sub Repository

Migrate SVN

Description:

Instructions to migrate a sub repository to it's own repository.

Get a copy of the repo:

    cp -r me@myserver:/usr/local/data/svn/myrepo .

Dump it:

svnadmin dump --quiet myrepo > myrepo.DUMP

Filter the hunk you want:

cat myrepo.DUMP | svndumpfilter include MySubProject > MySubProject.DUMP|more

copy it to the new server (omit this if doing it on the new server):

scp MySubProject.DUMP robrien@myotherserver:~/

Logon:

 ssh robrien@myotherserver

Create a new repo:

 sudo ./bin/svnadmin create /usr/local/repos/MySubProject
 sudo chown -R www:www /usr/local/repos/

Load the repo:

    sudo svnadmin load --quiet /usr/local/repos/MySubProject < /usr/home/robrien/MySubProject.DUMP

Done.

Build PIL on Mac OSX

This entry attempts to give instructions for how to build the Python Imaging Library (PIL) on Mac OSX.

#download and untar the jpeg library
#see here for location http://www.ijg.org/

curl http://path/to/jpegsrc.v6b.tar.gz | tar zxf -

#change directory to

cd jpeg-6b

#run configure

./configure CFLAGS='-fPIC'

#run make

make

#Run make install

sudo make install

#run ranlib

ranlib libjpeg.a

#copy the lib file to lib

cp libjpeg.a /usr/local/lib

#copy headers to include

cp *.h /usr/local/include

#make this your real python

PYTHON_COMMAND="/path/to/your/python"

#get and untar the PIL source

curl http://path/to/Imaging-1.1.6.tar.gz | tar zxf -

# cd

cd Imaging-1.1.6

# copy setup out of it's way

mv setup.py setup.py.tmp

# set up a replacement command
# you can do edit setup.py by hand if you wish

SED_CMD="s|JPEG_ROOT = None|JPEG_ROOT = libinclude(\"/usr/local/lib\")|; s|ZLIB_ROOT = None|ZLIB_ROOT = libinclude(\"/usr/local/lib\")|;"

#replace the stock jpg locations with the new ones you made

cat setup.py.tmp | sed -e "$SED_CMD" > setup.py

#build it

$PYTHON_COMMAND setup.py build_ext -i

#test it

$PYTHON_COMMAND selftest.py

#install it

sudo $PYTHON_COMMAND setup.py install

Undoing transactions in a broken Plone/Zope

Today there was a situation where someone made a permission change that made Plone inaccessible. The ZMI was also inaccessible (by design (not mine)). We could get access to the filesystem, however. So after a couple attempts at undos through in debug....There were issues with permissions and RESPONSE objects and all the other Zope machinery. So I tried mounting a ZODB directly and undoing transactions. The following worked great for me:

filename = "/Users/reedobrien/instances/zeo/msrd/var/Data.fs"
import ZODB.FileStorage
import ZODB.DB
storage = ZODB.FileStorage.FileStorage(filename)
DB = ZODB.DB(storage)
txs = DB.undoLog()[::-1]
DB.undo(id=txs[0]['id'])
import transaction; transaction.commit()

Maybe the wrong way and YMMV, but it stayed in one piece for me.

Passwordless Unix Login

Create a key pair

mack:~ reedobrien$ ssh-keygen -t rsa -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/reedobrien/.ssh/id_rsa.
Your public key has been saved in /Users/reedobrien/.ssh/id_rsa.pub.
The key fingerprint is:
bd:6c:18:e3:1a:46:5f:3d:d9:95:ed:82:6b:ea:6b:ec

Add the public Key to the server where you want to authenticate

mack:~ reedobrien$ scp ~/.ssh/id_rsa.pub username@remoteserver:~/.ssh/id_rsa.pub
mack:~ reedobrien$ ssh -l username remoteserver
password: ********
remoteserver:~ username$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
remoteserver:~ username$ exit

NOTE: There are many more advanced features, configurations, identity management things that can be done not covered here.

Test

mack:~ reedobrien$ ssh -l robrien remoteserver
Last login: Mon Aug 13 22:28:10 2007 from 131.182.85.5
Welcome to A Better World!
remoteserver:~ robrien$

ipython in zope3 zopectl debug

The solution in my earlier post doesn't fly in zope3. To get it to work in zope3 I modified:
~/instances/z3/bin/debugzope

if __name__ == '__main__':    db = startup()
     del startup
     from zope.app.debug import Debugger
     debugger = app = Debugger.fromDatabase(db)
     root = app.root()
     del db
     del Debugger
     import IPython
     IPython.Shell.IPShell(user_ns={'root': root,
                                     'app': app,                         
                                    'top' : app.root() 
                                     }).mainloop(sys_exit=1)

I haven't figured out much about how the debugger differs in Zope3. But it gives me top as app.root().

In [1]: print top.items()
<OOBTreeItems object at 0x355c598>

In [2]: app.root()['foo']
Out[2]: <zope.app.folder.folder.Folder object at 0x36dfef0>

In [3]: top
Out[3]: <zope.app.folder.folder.Folder object at 0x36a27f0>

In [4]: top['foo']
Out[4]: <zope.app.folder.folder.Folder object at 0x36df830>

In [5]: print [x for x in top.items()]
[(u'foo', <zope.app.folder.folder.Folder object at 0x36df830>)]

ipython in zopectl debug

A long time ago (6mos? a year) I got tired of manually typing support for history and tab completion into the zope debugger. So I just hooked the debug command up to read my .pythonrc file by importing user in the do_debug method of zopectl.py.

def do_debug(self, arg):
   cmdline = self.get_startup_cmd(self.options.python + ' -i',
   "import Zope2; app=Zope2.app(); import user;")

Today I tried getting ipython working as described in the plone docs. I also tried vanrees notes

and a couple others. I could not get it going though. So I decided to return to the hack above and extend it to load ipython.

def do_debug(self, arg):
    cmdline = self.get_startup_cmd(self.options.python + ' -i',
    "import Zope2; app=Zope2.app(); import user;     
     ns={'__name__':'msrd','app':app}; import IPython;   
     IPython.Shell.IPShell(user_ns=ns).mainloop(sys_exit=1);")

Poof, now debug comes up with ipython. Yay... What does that mean you may ask?

In [2]: ??app.msrd
Type: ImplicitAcquirerWrapper
Base Class:
String Form:
Namespace: Interactive
Length: 1
Docstring [source file open failed]:
Make PloneSite subclass CMFSite and add some methods.
This will be useful for adding more things later on.

I know this is probably the wrong way to do this, but nyah nyah. It is working now. I do welcome feedback...

As a note: I also have import user in my args. This picks up stuff from your file defined in PYTHONSTARTUP. But then if you already use PYTHONSTARTUP you probably already know that.

Plone site broken? Fix ir through the zodb

Today someone changed the ip on a dev box running an instance that had the ip as part of the url in CacheFu. So of course it won't load a page anymore. What to do? reinstall? nah. Let 's think for a second. Plone is a nice interface to Zope/CMF. Zope/CMF is a lovely interface to ZODB. ZODB is a persistence engine for Python objects. CacheFu's settings are persisted as atributes of a cache object in the ZODB. Let's just fix it directly....

/usr/local/{instance}/bin/zopectl debug
Starting debugger (the name "app" is bound to the top-level Zope object)
...{ 8< snip } ...
>>> import readline, rlcompleter, transaction
>>> readline.parse_and_bind("tab: complete")
# now we have tab completion
# bind the plone root to s
>>> s = app.itcd
# bind our (s)ites cache settings to cc
>>> cc = s.portal_cache_settings
>>> cc.getDomains()
('http://1.2.3.4:80',) ## <<== look there is our old value!!
# yup it needs to be this new value
>>> cc.setDomains("http://4.3.2.1:8080")
# and persisted. Otherwise it will be aborted at exit
>>> transaction.commit()

That was easy, no?

Quick and dirty regexen search for spamd & postfix logs

#!/usr/bin/env python
#-------------------------------------------------------------
# Name:        finder.py
# Purpose:     This is a script to search through  Posfix 
#              and spamd logs for the last x days and return 
#              hits. Mostly it is a quick and dirty way to 
#              find entris regarding emails blocked by 
#              spamd/postfix   
# Author:      Reed L. O'Brien reed at reedobrien com
#                                                                                                           
# Created:     2007-01-05
# Modified:    2007-01-05
# Copyright:   (c) Reed L. O'Brien 2007
# License:     DWYWWI (improvements welcome)                         
#--------------------------------------------------------------                      


#Do the imports
import os, re, bz2, sys, time

# make sure there is a regex
try:
    # compile the regex
  regx = re.compile(sys.argv[1], re.IGNORECASE)
except IndexError:
  print """
  usage:
         finder  [days back to search]

         ex: finder foobar 2
         will find all occurences of 'foobar' in the last 2
         days of spamd and maillog files.\n\tThe number of days
         is optional and defaults to 1 if not given."""
  sys.exit(0)

# empty list to store hits in
found = []

# move to the log directory
os.chdir('/var/log')
# Start a counter for the number counted
s = 0

#get and set days
try:
 days = int(sys.argv[2])
except:
 days = 1
# Get a list of qualifying files NOTE: you may need stat(x).[st_ctime|st_mtime] depending on your OS
L = [f for f in os.listdir('.')
   if os.stat(f).st_birthtime > time.time() - (days * 86400)
   and (f.startswith('spamd') or f.startswith('maillog'))]
# get a count of how many files to search
n = len(L)

# start a loop on the list
for f in L:
    # If it is a bz2 open it as a bz2 object
if (f.startswith('spamd') or f.startswith('maillog')) and f.endswith('2'):
 # tell em what is happening
   sys.stdout.write("\rsearching: %2s  remain %s                   " % (f,n))
   sys.stdout.flush()
        # set a line count
   c = 1
        # get a handle on the file
        handle = bz2.BZ2File(f)
        # iterate through the lines
        for line in handle:
        # if the regex is found
       if regx.search(line):
           # append the filename, line count and line content to the found list
           found.append("%-10s : %s\n%s" % (c, f, line))
                # increment the line count
           c += 1
       else:
           # or just increment the count if no regex match 
           c += 1
                # decrement the number of files remaining
           n -= 1  
                # increment the number of files searched
           s += 1

    ## DO the same as above as a regular file object if not a bz2 file SEE NOTES FOR last loop
if (f.startswith('spamd') or f.startswith('maillog')) and not f.endswith('2'):
   sys.stdout.write("\rsearching: %2s  remain %s                     " % (f,n))
   sys.stdout.flush()
   c = 1
   handle = open(f)
   for line in handle:
       if regx.search(line):
           found.append("%-10s : %s\n%s" % (c, f, line))
           c += 1
       else:
           c += 1
           n -= 1
           s += 1
                ##make some space to overwrite the sys.stdout text 
print '\n\n\n\n\n'
print 'Searched:', s # Print how many files were searched

#print the results from the found list.
for x in found:
print x