Welcome

Welcome on my blog
I'll try to contribute my -modest- knowledge to the community.
Here you'll find samples of python twisted code but also some network configurations.

Hoping you'll find it usefull.

gracefully
cat 'tonthon\'s brain' > /dev/null
Tonthon

Tuesday, June 21, 2011

Install openerp v6

Last time I was looking for an Erp solution and I tried openerp.
The official installation guide is provided here : http://doc.openerp.com/v6.0/install/linux/
I wanted to install it from source on a basic ubuntu 11.04 machine.
Once more, an incomplete (or not up to date) documentation.

What a time spent by all those users who are looking to make it work, if they were contributing that time to something more usefull... what a wonderfull world would it be...

"I love clear and ready-to-copy-paste documentations !!!"

First thing to know : openerp 6 needs python 2.6 and not python 2.5 since the io module needed by dateutils isn't provided in python 2.5.

Context



Ubuntu server 11.04
Using the root account for convenience.

Database

(install postgress and add openerp user)

apt-get install postgresql
su postgres
createuser --createdb --username postgres --no-createrole --pwprompt openerp


Server installation


Installing dependencies

apt-get install python-lxml python-mako python-egenix-mxdatetime python-dateutil python-psycopg2 python-pychart
apt-get install python-pydot python-tz python-reportlab python-yaml python-vobject

Let's add python-setuptools :

apt-get install python-setuptools

Download

wget http://www.openerp.com/download/stable/source/openerp-server-6.0.2.tar.gz
tar -zxf openerp-server-6.0.2.tar.gz

Install, add a specific user and launch the server

cd openerp-server*
python setup.py install
adduser openerp
su openerp
openerp-server


Client installation


Open a new prompt since the server is already logging in the first one.
Dependencies (here python-xml is replaced with python-lxml)

apt-get install python-gtk2 python-glade2 python-matplotlib python-egenix-mxdatetime python-lxml python-tz
apt-get install python-hippocanvas python-pydot

Download and install

wget http://www.openerp.com/download/stable/source/openerp-client-6.0.2.tar.gz
tar -zxvf openerp-client-6.0.2.tar.gz
cd openerp-client*
python setup.py install

Add symlinks
You have to detect where your client has been installed. I like to use updatedb and locate commands to retrieve the good path. Finally you should get something like this :

/usr/local/lib/python2.7/dist-packages/openerp_client-6.0.2-py2.7.egg/

We have to add symlinks to make our client startup :

mkdir -p /usr/share/pixmaps/
ln -s /usr/local/lib/python2.7/dist-packages/openerp_client-6.0.2-py2.7.egg/share/openerp-client /usr/share/
ln -s /usr/local/lib/python2.7/dist-packages/openerp_client-6.0.2-py2.7.egg/share/pixmaps/openerp-client /usr/share/pixmaps/

openerp-client script
Now we have to correct openerp-client script to let it work:

vim /usr/local/bin/openerp-client
:2,2s/openerp-client/openerp_client-6.0.2-py2.7.egg\/openerp-client/c
:wq

NB : The two last commands are vim specific.

Launch
Now we are ready to launch our client :

openerp-client

Aaah !! Let's learn how it works, that will take much more time, but it will be usefull spent time ... or not ... "what does it meen?" ... no matter ... it's not intended to become a philosophical blog.

Tuesday, February 8, 2011

Ldaptor : ldap with twisted, client side

Introduction

Ldaptor is an ldap library build on top of python and twisted, it provides :

  • LDAP client logic

  • Separately-accessible LDAP and BER protocol message generation/parsing

  • ASCII-format LDAP filter generation and parsing

  • LDIF format data generation

  • Samba password changing logic



Here we will use the client-side tools to retrieve datas from our ldap server.
We will use the ldap server written there :
ldaptor : ldap with twisted, server-side


Quick overview

  • The connector
    A connector is used to wrap the ldap client, this connector looks up DNS SRV records.

    >>> from ldaptor.protocols.ldap.ldapconnector import LDAPClientCreator
    >>> from ldaptor.protocols.ldap.ldapclient import LDAPClient
    >>> from twisted.internet import reactor
    >>> clientBuilder = LDAPClientCreator(reactor, LDAPClient)
    >>> print clientBuilder
    <ldaptor.protocols.ldap.ldapconnector.LDAPClientCreator instance at 0xb753830c>
    >>>

    You can then build clients for different ldap servers.

  • The DN object

    >>> from ldaptor.protocols.ldap.distinguishedname import DistinguishedName as DN
    >>> dn = DN('dc=example,dc=com')
    >>> print dn
    dc=example,dc=com
    >>> print dn.getDomainName()
    'example.com'
    >>>

  • Bind to a remote server
    You then build a client for that given DN

    >>> binded_client = clientBuilder.connectAnonymously(dn, overrides={dn:('localhost', 8080)})
    >>> print binded_client
    <Deferred at 0xa44268c>
    >>>

    The connectAnonymously method connect to the remote host and binds automatically.
    Note : Here we override the remote's adress to avoid DNS lookup.

    To make authenticated bindings :

    >>> userdn = DN('uid=esteban,ou=people,dc=example,dc=fr')
    >>> client = clientBuilder.connect(dn, {dn:('localhost', 8080)})
    >>> client.addCallback(lambda client:LDAPEntry(client=client,
    dn=userdn).bind(password))
    >>>

    We first connect to the remote and then build an ldapentry with our userdn which we bind with the given password.


Here is a full example retrieving datas from the previous-described server.
Put all this stuff into a file you call client.py, run the server and launch a `python client.py`

#!/usr/bin/python
#-*-coding:utf-8-*-
"""
Simple Ldaptor client
"""
import sys

from twisted.python import log
from twisted.internet import reactor

from ldaptor.protocols.ldap.distinguishedname import DistinguishedName as DN
from ldaptor.ldapfilter import parseFilter
from ldaptor.protocols.ldap.ldapsyntax import LDAPEntry
from ldaptor.protocols.ldap.ldapconnector import LDAPClientCreator
from ldaptor.protocols.ldap.ldapclient import LDAPClient


class Client:
"""
Simple Ldap client to retrieve user's informations
"""
def __init__(self, dn, host, port):
dn = DN(dn)
self.config = dict(base=dn,
server={dn:(host, port)})
self.clientBuilder = LDAPClientCreator(reactor, LDAPClient)
def bind(self):
"""
returns a deferred anonymous connection
"""
return self.clientBuilder.connectAnonymously(
self.config['base'],
self.config['server'])
@staticmethod
def _unbindAndReturn(result, client):
"""
Unbind the binded client
"""
client.unbind()
return result
def buildBase(self, client):
"""
return the base ldap entry the search will
be run from
"""
return LDAPEntry(client=client, dn=self.config['base'])
def _search(self, client, login):
"""
Launch our search
"""
ldapFilter = parseFilter('(uid=%s)' % login)
base = self.buildBase(client)
return base.search(filterObject=ldapFilter).addBoth(
self._unbindAndReturn, client)
@staticmethod
def _formatRes(results, attributes):
"""
Format results to get only the attributes we wanted to keep
"""
for result in results:
yield dict([(attr, list(result.get(attr)))
for attr in attributes])
def getUser(self, login, attributes):
"""
Public method used to retrieve ldap user's infos
"""
deferred = self.bind()
deferred.addCallback(self._search, login)
deferred.addCallback(self._formatRes, attributes)
return deferred

def main():
log.startLogging(sys.stderr)
def printRes(results):
for res in results:
log.msg(res)
client = Client('dc=fr', '127.0.0.1', 8080)
datas = client.getUser('yoen', ('mail','sn'))
datas.addCallback(printRes)
reactor.callLater(3, reactor.stop)
reactor.run()

if __name__ == '__main__':
main()

Ldaptor : ldap with twisted, server side

Introduction
"Ldaptor is a set of pure-Python LDAP client programs, applications and a programming library." (Ldaptor on Tv's blog)

Here we will see how to build a simple pythonic ldap server using twisted and ldaptor.

The ldiftree

>>> from ldaptor.ldiftree import LDIFTreeEntry
>>> tree = LDIFTreeEntry('/tmp/mytemporaryldaptordir')
>>> print tree
dn:


>>>

You can then add childs to your root :

>>> com = tree.addChild('dc=com', {'objectClass':['dcObject'],'dc':['com']})
>>> print com
dn: dc=com
objectClass: dcObject
dc: com


>>>

Note : the addChild returns the added node so you can then add childs to it ...

First step : we need seeds to make our tree grow
The goal of this article isn't to provide a curse about ldap (you'll find hundreds of them googling).
Here is the basic tree structure we will use :

COUNTRY = ('dc=fr',
{'objectclass':['dcObject','country'],
'dc':['fr'],
'description':["French country 2 letters iso description"]})
COMPANY = ('dc=example',
{'objectclass':['dcObject','organization'],
'dc':['example'],
'description':["My organisation"],
'o':"Example, Inc"})
PEOPLE = ('ou=people',
{'ou':'people',
'description':'People from Example Inc',
'objectclass':'organizationalunit'})
USERS = [
('uid=yoen', {'objectClass': ['people', 'inetOrgPerson'],
'cn':['Yoen Van der Weld'],
'sn':['Van der Weld'],
'givenName':['Yoen'],
'uid':['yoen'],
'mailDir':['/home/yoen/mailDir'],
'userpassword': ['secret']}),
('uid=esteban', {'objectClass': ['people', 'inetOrgPerson'],
'cn':['Esteban Garcia Marquez'],
'sn':['Garcia Marquez'],
'givenName':['Esteban'],
'uid':['esteban'],
'mailDir':['/home/esteban/mailDir'],
'userpassword':['secret2']}),
('uid=mohamed', {'objectClass': ['people', 'inetOrgPerson'],
'cn':['Mohamed Al Ghâlib'],
'sn':['Al Ghâlib'],
'givenName':['mohamed'],
'uid':['mohamed'],
'mailDir':['/home/mohamed/mailDir'],
'userpassword':['secret3']}),
]

Country->Company->People populated with three accounts.
Each node is described with two datas its rdn (relative distinguished name) and its attributes.
We will not provide a schema entry for simplicity.

Second step : let the tree grow
Let's build our structure based on the scheme we defined here above :

import tempfile

from ldaptor.ldiftree import LDIFTreeEntry

class Tree:
def __init__(self, path='/tmp'):
dirname = tempfile.mkdtemp('.ldap', 'test-server', '/tmp')
self.db = LDIFTreeEntry(dirname)
self.init_db()
def init_db(self):
"""
Add subtrees to the top entry
top->country->company->people
"""
country = self.db.addChild(COUNTRY[0], COUNTRY[1])
company = country.addChild(COMPANY[0], COMPANY[1])
people = company.addChild(PEOPLE[0], PEOPLE[1])
for user in USERS:
people.addChild(user[0], user[1])


tree = Tree()

Note that we use the tempfile standard lib to avoid erasing existing datas.
A in-memory tree could also be used (see test_server.py)

Third step : Creating the factory used to serve
To understand what protocol and factories are in twisted read this : Writing Servers.

from twisted.internet.protocol import ServerFactory

from ldaptor.protocols.ldap.ldapserver import LDAPServer

class LDAPServerFactory(ServerFactory):
protocol = LDAPServer
def __init__(self, root):
self.root = root

Our factory simply store persistently the root (the tree in fact).

When the ldap protocol handle the ldap tree, it retrieves it from the factory adapting the factory to the IConnectedLDAPEntry interface. So we need to register an adapter for our factory to match the IConnectedLDAPEntry.
See Components: Interfaces and Adapters for further informations.

from twisted.python.components import registerAdapter

from ldaptor.interfaces import IConnectedLDAPEntry
registerAdapter(lambda x: x.root,
LDAPServerFactory,
IConnectedLDAPEntry)


We will say now that all is prepared to run our ldap server.

Fourth step : running our server

from twisted.application import service, internet
from twisted.internet import protocol, reactor

factory = LDAPServerFactory(tree.db)
application = service.Application("ldaptor-server")
myService = service.IServiceCollection(application)
reactor.listenTCP(8080, factory)
reactor.run()


Final step : let's try
To test that all, use a ldap client.
For example install the ldap-utils package and in another prompt, launch:

ldapsearch -x -h 127.0.0.1 -p 8080 'uid=esteban'


Our two files

schema.py :

#-*-coding:utf-8-*-
COUNTRY = ('dc=fr',
{'objectclass':['dcObject','country'],
'dc':['fr'],
'description':["French country 2 letters iso description"]})
COMPANY = ('dc=example',
{'objectclass':['dcObject','organization'],
'dc':['example'],
'description':["My organisation"],
'o':"Example, Inc"})
PEOPLE = ('ou=people',
{'ou':'people',
'description':'People from Example Inc',
'objectclass':'organizationalunit'})
USERS = [
('uid=yoen', {'objectClass': ['people', 'inetOrgPerson'],
'cn':['Yoen Van der Weld'],
'sn':['Van der Weld'],
'givenName':['Yoen'],
'uid':['yoen'],
'mail':['/home/yoen/mailDir'],
'userpassword': ['secret']}),
('uid=esteban', {'objectClass': ['people', 'inetOrgPerson'],
'cn':['Esteban Garcia Marquez'],
'sn':['Garcia Marquez'],
'givenName':['Esteban'],
'uid':['esteban'],
'mail':['/home/esteban/mailDir'],
'userpassword':['secret2']}),
('uid=mohamed', {'objectClass': ['people', 'inetOrgPerson'],
'cn':['Mohamed Al Ghâlib'],
'sn':['Al Ghâlib'],
'givenName':['mohamed'],
'uid':['mohamed'],
'mail':['/home/mohamed/mailDir'],
'userpassword':['secret3']}),
]

server.py :

#-*-coding:utf-8-*-
"""
Testing a simple ldaptor ldap server
"""
import tempfile, sys

from twisted.application import service, internet
from twisted.internet import reactor
from twisted.internet.protocol import ServerFactory
from twisted.python.components import registerAdapter
from twisted.python import log

from ldaptor.interfaces import IConnectedLDAPEntry
from ldaptor.protocols.ldap.ldapserver import LDAPServer
from ldaptor.ldiftree import LDIFTreeEntry

from schema import COUNTRY, COMPANY, PEOPLE, USERS


class Tree:
def __init__(self, path='/tmp'):
dirname = tempfile.mkdtemp('.ldap', 'test-server', '/tmp')
self.db = LDIFTreeEntry(dirname)
self.init_db()
def init_db(self):
"""
Add subtrees to the top entry
top->country->company->people
"""
country = self.db.addChild(COUNTRY[0], COUNTRY[1])
company = country.addChild(COMPANY[0], COMPANY[1])
people = company.addChild(PEOPLE[0], PEOPLE[1])
for user in USERS:
people.addChild(user[0], user[1])

class LDAPServerFactory(ServerFactory):
"""
Our Factory is meant to persistently store the ldap tree
"""
protocol = LDAPServer
def __init__(self, root):
self.root = root

if __name__ == '__main__':
# First of all, to show logging info in stdout :
log.startLogging(sys.stderr)
# We initialize our tree
tree = Tree()
# When the ldap protocol handle the ldap tree,
# it retrieves it from the factory adapting
# the factory to the IConnectedLDAPEntry interface
# So we need to register an adapter for our factory
# to match the IConnectedLDAPEntry
registerAdapter(lambda x: x.root,
LDAPServerFactory,
IConnectedLDAPEntry)

# Run it !!
factory = LDAPServerFactory(tree.db)
application = service.Application("ldaptor-server")
myService = service.IServiceCollection(application)
reactor.listenTCP(8080, factory)
reactor.run()


In a first prompt launch the server :

python server.py

Thursday, January 27, 2011

Twisted adbapi : asynchronous database access

Introduction

First of all, read the official twisted documentation.
""" It is a non-blocking interface to the standardized DB-API 2.0 API, which allows you to access a number of different RDBMSes."""

When I first used this module I faced some troubles I didn't expected, so I decided to build an example tutorial covering the basics.

Here we will build a simple sqlite database and access to it through a main DB class.
Let say we store user's informations into a database and we want to retrieve an Avatar object build from those infos.

First step : building our test database
We will use the sqlite3 library to build our database.

import tempfile
import sqlite3

USERS = (('yoen', 'Van der Veld', 'Yoen',),
('esteban', 'Garcia Marquez', 'Estebán',),
('mohamed', 'Al Ghâlib', 'Mohamed',),)

def setup_db():
""" Setup our sqlite database """
# Tempfile ensure us not to override an existing file
myfilename = tempfile.mktemp('.sqlite', 'test-bdd', '/tmp')
conn = sqlite3.connect(myfilename)
curs = conn.cursor()
curs.execute("Create table users (login text unique, name text, \
forname text)")
for login, name, forname in USERS:
curs.execute("INSERT INTO USERS VALUES (?, ?, ?)", (login,
name,
forname))
conn.commit()
curs.close()
return myfilename

dbname = setup_db()

At this point we've got a working database dbname with one table 'users' containing three fields : 'login' (unique), 'name', 'forname'.

Second step : the Avatar Object

class Avatar:
def __init__(self, login, name, forname):
self.login = login
self.name = name
self.forname = forname
def render(self):
msg = "Mr '%s %s' is connected under '%s'" % (self.forname,
self.name,
self.login,)
return msg


Nothing special to say here, don't need more for our example.

Third step : the base connector
This class will interact with the database and provide a get_user_avatar method.

from twisted.enterprise.adbapi import ConnectionPool

class DBPool:
"""
Sqlite connection pool
"""
def __init__(self, dbname):
self.dbname = dbname
self.__dbpool = ConnectionPool('sqlite3', self.dbname)
def shutdown(self):
"""
Shutdown function
It's a required task to shutdown the database connection pool:
garbage collector doesn't shutdown associated thread
"""
self.__dbpool.close()
def build_avatar(self, dbentries):
"""
Build avatar from dbentries
"""
login, name, forname = dbentries[0]
return Avatar(login, name, forname)
def get_user_avatar(self, login):
"""
Build associated avatar object
"""
query = 'SELECT * from `users` where login=?'
return self.__dbpool.runQuery(query, (login,)).addCallback(
self.build_avatar)


Now we have all we need to be able to retrieve avatars from our database.
Some special points to underline :
  • The ConnectionPool automatically start within twisted's standard reactor

  • The ConnectionPool has to be stopped after using it to avoid the "thread.error: can't start new thread" error


Last step : Let's try

from twisted.internet.defer import DeferredList
from twisted.internet import reactor
def printResult(result):
for r in result:
print(r[1])

# We initialize the db pool with our dbname retrieved in the first step.
dbpool = DBPool(dbname)
# DeferredList seems more adapted than chained callbacks in this sort of cases
ret_render = lambda avatar: avatar.render()
deferreds = [dbpool.get_user_avatar(login).addCallback(ret_render)
for login in ('yoen', 'esteban', 'mohamed',)]

dlist = DeferredList(deferreds)
dlist.addCallback(printResult)
# We ask our pool to shutdown all the initialized connections
dlist.addCallback(lambda _:dbpool.shutdown)
# Want to get the hand back :-)
reactor.callLater(4, reactor.stop)
reactor.run()