Apache Round Robin for ZeoClients

There is a common approach to running Plone/Zope behind some service that distributes requests across numerous ZeoClients. Here's how I have been doing it using Apache (2.2.x).

                                  ________Zeoclient(6968)
                                 | _______ZeoClient(6969)
client-->varnish(80)-->apache(81)|<_______ZeoClient(6970)
                                 |________ZeoClient(6971)

Everybody knows how to setup varnish, right? So I am only going to show the Apache bits here.

First you need a rewrite map.

#zopeservers.map
ALL     localhost:6968|localhost:6969|localhost:6970|localhost:6917

Of course there can be more or less hosts in the map and they don't have to be all localhost either.

Next you need to enter the right stuff in the apache configuration. As here in a VirtualHost.

<VirtualHost *:81>
...
RewriteMap  zopeservers rnd:/usr/local/apache2/conf/zopeservers.map
RewriteRule ^/(.*) \
    http://${zopeservers:ALL}/VirtualHostBase/http/%{SERVER_NAME}:80/ploneroot/VirtualHostRoot/$1 [L,P]
...
</VirtualHost>

How does Emiril say... BAM!

Not so hard at all. A couple of notes:

1. I don't think apache knows how to detect if a client is down or broken. So it will serve up all the errors it gets.

2. Varnish doesn't cache https, so it needs to be handled wihtout varnish...

Seeded (Salted) SHA Passwords

This week I needed to work with passwords from an OpenLDAP database. I needed to create users and encode their passwords as SSHA. After much googling and reading of authentication code examples, this is what I came up with.

import hashlib
import os
from base64 import urlsafe_b64encode as encode
from base64 import urlsafe_b64decode as decode

def makeSecret(password):
 salt = os.urandom(4)
 h = hashlib.sha1(password)
 h.update(salt)
 return "{SSHA}" + encode(h.digest() + salt)

def checkPassword(challenge_password, password):
 challenge_bytes = decode(challenge_password[6:])
 digest = challenge_bytes[:20]
 salt = challenge_bytes[20:]
 hr = hashlib.sha1(password)
 hr.update(salt)
 return digest == hr.digest()

Wow, python rules. So first I want to make sure I can validate a password against one generated by slappasswd

reedobrien$ slappasswd -s topsecret
{SSHA}Ccpjsip2UZL2CR2VsWTH7aF0vWKHQ7jn

And then see if I can validate it

>>> checkPassword('{SSHA}Ccpjsip2UZL2CR2VsWTH7aF0vWKHQ7jn',
...               'topsecret')
   True

Next I need to make sure I can generate one that OpenLDAP can use. So I used a script I wrote to create a user in ldap. This time I use the same password and get a different hash which is good.

>>> pw = "{SSHA}hq5ROYE8RoLA1Zcz6azgNQP5PkdETocx"

which OpenLDAP represents in LDIFs as base64 encoded

...
# tester, people, reedobrien.com
dn: uid=tester,ou=people,dc=reedobrien,dc=com
objectClass: top
objectClass: inetOrgPerson
uid: tester
cn: test test
sn: test
mail: test@example.com
userPassword:: e1NTSEF9MHhaMEdZc2Fob1JNeXZWR2FVdGszS0VwSFZTQnVLTlc=
...

So I try authenticating:

>>> userPasword = "e1NTSEF9MHhaMEdZc2Fob1JNeXZWR2FVdGszS0VwSFZTQnVLTlc="

>>> decode(userPassword) == pw
    True

>>> checkPassword(decode(userPassword), 'topsecret')
    True

Now I want to make sure it works through the ldap server itself
In python:

>>> import ldap

>>> con = ldap.initialize("ldap://127.0.0.1")
>>> con.simple_bind_s('uid=tester,ou=people,dc=reedbrien,dc=com',
                      'topsecret')
    (97, []) ## indicates success

And not using my stuff.

reedobrien$ ldapsearch -x -D "uid=tester,ou=people,dc=reedobrien,dc=com" -w topsecret
# extended LDIF
#
# LDAPv3
# base <> with scope sub
# filter: (objectclass=*)
# requesting: ALL
#

# search result
search: 2
result: 32 No such object

# numResponses: 1

How do they say, `w00t!`

So it was cool doing it and it made me feel smart for a minute. I am sure that there is a module or ten out there that already does this, but I couldn't find it. At least it was educational.

Unfortunately, after getting it to work; I don't I understand any better what it really does or how SSHA is better than plain old SHA.

Oh, well. Hopefully it helps some weary traveller out there one day.

Serving up a ZODB on demand from a repozo backup

Since I said I would on the mailing list/forum.

Here it is:

import os
import shutil
import tarfile
import tempfile

def application(environ, start_response):
  status = '200 OK'

  tempdir  = tempfile.mkdtemp()
  datafile = '%s/Data.fs' % tempdir
  tarball = '%s.tar.bz2' % os.path.join(tempdir,
                                        os.path.basename(datafile))

  ## If I were smarter or had more time
  ## I would import from repozo or at
  ## least use subprocess
  os.system("/usr/local/zope/2.9.7/bin/repozo.py -v -z -R -r \
         /usr/local/zope/sites/2.9.7/msrd/zeo/var/backup/ -o %s" % datafile)

  out = tarfile.TarFile.open(tarball, 'w:bz2')
  os.chdir(tempdir)
  out.add('Data.fs')
  out.close()


  response_headers = [('Content-type', 'application/octet-stream'),
                      ('Content-Length', str(len(open(tarball).read()))),
                      ('Content-Disposition',
                       'inline; filename="Data.fs.tar.bz2"')]

  start_response(status, response_headers)
  try:
      return [open(tarball, 'rb').read()]
  finally:
      shutil.rmtree(tempdir)

I did it really quick. But it solved my problem very well. It gets a copy of the Data.fs from the last repozo backup. So all developers don't need shell access to the server...

A couple things I could see doing that would make this cooler are:

• Not using os.system
• implement a method to do HEAD requests with the time of the last repozo delta so I don't compress and transfer this beast if there are no changes...
  

If you have mod_wsgi installed and working you just need to add something like:

WSGIScriptAlias /Data.fs.tar.bz2 /usr/local/apps/getDataArchived.py

to the appropriate configuration file.