Author Archives: David

About David

A mostly professional code monkey, to contact me translate the following "direct from blog to email" at the website ominian.net. Replace spaces with underscores or dashes, whatever is valid. A script will pick up your email, scrub it against a white list, and if your not a spammer I will get an email.

metaclass service bus decorator concept

While playing around with some idea’s for making PyProxy completely overridable ( and also potentially undebuggable ) I started playing around with metaclasses.

Below is from my toxic directory [ gist url ]

from collections import defaultdict
from functools import wraps

Just some preliminary basic utilities

A simple bus implementation, decoratedCalls is a dictionary of service calls, with a list
of callbacks to each service call hook.

decoratedCalls = defaultdict(list)

def call(name, *args, **kwargs):
    print name, args, kwargs
    if name in decoratedCalls:
        for cb in decoratedCalls[name]:
            cb(*args, **kwargs)

Syntactic suger to add clarity later as to what functions are being bound to what. Adding in
debug/logging hooks here could allow for easier tracing of what is called for what methods.

class Subscribe(object):
    def __init__(self, name):
        self.name = name
        
    def __call__(self, f):
        decoratedCalls[self.name].append(f)
        return f
        

Here’s the actual bus decorator, very simple just wraps the decorated function with a pre and post bus calls.

class BusDecorator(object):
    def __init__(self, name):
        self.name = name
        
    def __call__(self, f):
        
        @wraps(f)
        def decorator(inst, *args, **kwargs):
            call("%s-pre" % self.name, inst, args, kwargs)            
            retval = f(inst, *args, **kwargs)
            call("%s-post" % self.name, inst, retval)            
            return retval
        return decorator

And here’s my Bus Metaclass that combines most of the above.

The ease of wrapping the target class is accomplished by cdict which is a dictionary of
every defined attribute of the target class. As you can see it’s trivial to spin
through and decorate every callable with the BusDecorator

class BusWrap(type):
    
    def __new__(mcs, clsname, bases, cdict):

        modName = cdict.get("__module__", "unknownclass")
        
        for name in cdict.keys():
            prefix = "%s.%s.%s" % ( modName, clsname, name)
            if callable(cdict[name]):                
                cdict[name] = BusDecorator(prefix)(cdict[name])
        
        return type.__new__(mcs, name, bases, cdict)
        

Now give a dirt simple class like

            
class Foo(object):
    __metaclass__ = BusWrap
    
    def __init__(self):
        print "init'd"
        
    def bar(self):
        print "bar"
        
    def blah(self):
        print "blah"
        
    def ich(self):
        print "ich"
        
    def ego(self):
        print "lego"
        
    def say(self, *args):
        print "Saying ", args
      

And two service handlers to pre Foo.bar being called and after Foo.ego is called

@Subscribe("__main__.Foo.bar-pre")        
def preBar(inst, *args, **kwargs):
    if not hasattr(inst, "ext_info"):
        inst.ext_info = "Here"
        
@Subscribe("__main__.Foo.ego-post")
def postEgo(inst, *args, **kwargs):
    if hasattr(inst, "ext_info"):
        print "Extended info is ", inst.ext_info

Our test shows….

x = Foo()
x.bar()
x.blah()
x.ich()
x.ego()
x.say("abba", "dabba")

this as output

__main__.Foo.__init__-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
init'd
__main__.Foo.__init__-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.bar-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
bar
__main__.Foo.bar-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.blah-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
blah
__main__.Foo.blah-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.ich-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
ich
__main__.Foo.ich-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.ego-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
lego
__main__.Foo.ego-post (<__main__.__init__ object at 0x02637890>, None) {}
Extended info is  Here
__main__.Foo.say-pre (<__main__.__init__ object at 0x02637890>, ('abba', 'dabba'), {}) {}
Saying  ('abba', 'dabba')
__main__.Foo.say-post (<__main__.__init__ object at 0x02637890>, None) {}


Plugin’s for python

As the #1 google result for “python plugin”, the linked to blog post is extremely valuable for jump starting research into implementing your own plugin system ( like me ) or finding one that is viable for implementation in your project ( possibly like me ).

These resources highlight one reason why there is not a standard Python plug-in framework: there are a variety of different capabilities that a user may want, and the complexity of the framework generally increases as these new capabilities are added

http://wehart.blogspot.com/2009/01/python-plugin-frameworks.html

TxWeb Alpha – A different spin on twisted.web

I’ve spent some more time on my current pet, txweb, and I think it’s pretty much at the as good as it gets stage.

Below is the source for the example.py

#App level
from txweb import Site, expose
#twisted
from twisted.web import server, resource
from twisted.internet import reactor
from twisted.web.static import File

from os.path import abspath, dirname, join

Mostly pretty standard imports for a twisted.web application.

Now here is the “Controllers”, they’re stripped down to bare-bones just to keep it simple

class PageOne(object):

    @expose
    def foo(self, request):
        return "Hello From PageOne Foo!"        
    
    @expose
    def delayed(self, request):
        def delayedResponse():
            request.write("I was delayed :( ")
            request.finish()
            
        reactor.callLater(5, delayedResponse)
        return server.NOT_DONE_YET
    
    
class PageTwo(object):

    @expose
    def index(self, request):
        """ /pagetwo/index """
        return "Hello From PageTwo index!"
        

rootFile = lambda filename : abspath(join(dirname(__file__), filename))
        
class Root(object):
    
    @expose
    def index(self, request):
        """
            Will handle both / and /index paths
        """
        return "Hello From Index!"
    
    @expose
    def __default__(self, request):
        """
            Unless overriden further down, this will catch all 404's
        """
        return "I Caught %s " % request.path
    
    pageone = PageOne()
    pagetwo = PageTwo()
        
    readme  = File(rootFile("README.md"))
    license = File(rootFile("txweb/LICENSE.txt"))

Basically a txWeb enabled twisted service converts a URL path to an Object path.

So /hello/world could resolve to root.hello.world() if such a construct was provided.

Much more importantly, with the above example, /license resolves to the local file txweb/LICENSE and /readme resolves to README.md !

In summary txweb doesn’t throw away the epic amount of work the Twisted developers and volunteers have put forth, it just presents it in another way.

A friendlier asynchronous twisted web, the ghetto monkey patch way

UPDATE to the UPDATE – A cleaned up and more coherent example of txweb is here
UPDATE – Github repo here

I like twisted, and I like Cherrypy, unfortunately just like my militant atheist friends and my more spiritual friends neither seems to get along with the other.

What to do? MONKEY PATCH + GHETTO HACKING to the rescue!

Note, this is just a mockup of CherryPy’s routing system and not a bridge or interface to CherryPy. There is no CherryPy to be had here, just ghetto py.

from twisted.web import server, resource
from twisted.internet import reactor


def expose(func):
    func.exposed = True
    return func

class PageOne(object):

    def foo(self, request):
        return "Hello From PageOne Foo!"        
    foo.exposed = True
    
    @expose
    def delayed(self, request):
        def delayedResponse():
            request.write("I was delayed :( ")
            request.finish()
            
        reactor.callLater(5, delayedResponse)
        return server.NOT_DONE_YET
    
    
class PageTwo(object):

    @expose
    def index(self, request):
        return "Hello From PageTwo index!"
        
        
class Root(object):
    
    @expose
    def index(self, request):
        return "Hello From Index!"
    
    @expose
    def __default__(self, request):
        return "I Caught %s " % request.path
    
    pageone = PageOne()
    pagetwo = PageTwo()
        
class OneTimeResource(resource.Resource):
    """
        Monkey patch to avoid rewriting more of twisted's lower web
        layer which does a fantastic job dealing with the minute details
        of receiving and sending HTTP traffic.
        
        func is a callable and exposed property in the Root OO tree
    """
    def __init__(self, func):
        self.func = func
        
    def render(self, request):
        #Here would be a fantastic place for a pre-filter
        return self.func(request)
        #ditto here for a post filter
        
        
class OverrideSite(server.Site):
    """
        A monkey patch that short circuits the normal
        resource resolution logic @ the getResourceFor point
        
    """
    def checkAction(self, controller, name):
        """
            On success, returns a bound method from the provided controller instance
            else it return None
        """
        action = None
        if hasattr(controller, name):
                action = getattr(controller, name)
                if not callable(action) or not hasattr(action, "exposed"):
                    action = None
        
        return action
        
                    
    def routeRequest(self, request):
        action = None
        response = None
        
        root = parent = self.resource
        defaultAction = self.checkAction(root, "__default__")
        
        path = request.path.strip("/").split("/")
        
         
        
        for i in range(len(path)):
            element = path[i]
            
            parent = root
            root = getattr(root, element, None)
            request.prepath.append(element)
            
            if root is None:                
                break
            
            if self.checkAction(root, "__default__"):
                #Check for a catchall default action
                defaultAction = self.checkAction(root, "__default__")
                
                
            if element.startswith("_"):
                #500 simplistic security check
                action = lambda request: "500 URI segments cannot start with an underscore"
                break
                
            if callable(root) and hasattr(root, "exposed") and root.exposed == True:
                action = root
                request.postpath = path[i:] 
                break
            
            
                
        else:
            if action is None:
                if root is not None and self.checkAction(root, "index"):
                    action = self.checkAction(root, "index")
                
                
        #action = OneTimeResource(action) if action is not None else OneTimeResource(lambda request:"500 Routing error :(")
        if action is None:
            if defaultAction:
                action = defaultAction
            else:            
                action = lambda request:"404 :("
                
        return OneTimeResource(action)         

                
                
        
    def getResourceFor(self, request):
        return self.routeRequest(request)
        
"""
    Twisted thankfully doesn't do any type checking, so a
    dumb OO graph is A-Okay here.  It will be assigned to
    site.resource
"""
dumb = OverrideSite(Root())

reactor.listenTCP(80, dumb )
reactor.run()

Slapped this together in about 30 minutes… so there is a HIGH probability that it is almost entirely edge cased! Still it does work ( for me ) and it doesn’t hijack too much of twisted’s core, so it could be viable with a lot of unit-testing love, some additional sanity checking logics, and maybe some well thought out refactoring.

The magic JS console thing

Source here – https://github.com/devdave/DevDave-s-miscellanious-bucket/tree/master/jsConsole

Just wanted to see what would be involved in making a terminal like UI for the web. Honestly there’s a few REALLY annoying bits, but with a bit of hackery it was pretty straight forward. I’ve found splitting up the keyCode/which between key press and key down events was a lot simpler then making some sort of jury rigged code to character map when trying to compute shifted keys.

Also, I put in a brutally ugly left to right parser that does auto-complete / tab completion for good measure. It’s got a very limited vocabulary but I could easily snap it into an asyncronous Ajax handler to bridge it with a much more powerful server side module.

Like all of my short term projects, no warranty or guarantee that it works is provided or implied.

Unasked question for stackoverflow, an idea for removing/minimizing a singleton

Problem
——-
I have two cooperative services in twisted, one is a HTTP proxy and the other a HTTP service to power a web console. At the start and end of every proxy request, an overloaded method copies the request ( headers, POST data, etc ) and also the response. Currently I have a singleton data.Store that receives new records and it works except its becoming unwieldy to unit-test.

Ideas
—–
One thought was to implement an in-process synchronous service bus in a module and make it near stateless.

bus.py

    channels = defaultdict(list)


    def register(name, callback):
        global channels
        channels[name].append(callback)

    def callbacks(name):
        global channels
        #use of generator to allow for the caller to apply exception handling logic
        for callback in channels[name]:
            yield callback

   def first(name, default = None):
        global channels
        return channels[0] if count(channels) > 0 else default

A simple example might be

main.py

store = data.Store()


bus.register("store.addRecord", store.AddRecord)

proxy.py

class ProxyClient(proxy.ProxyClient):
     ....
     def handleResponse(self, data):
         proxy.ProxyClient.handleResponse(self, data)
         bus.first("store.addRecord", lambda : return False)(data)

Does this seems sensible? bus.channels can be cleared by setUp logic while testing can use in testcase methods for mocking. Or is there already something like this backed into twisted ( which would be much more preferable as the twisted dev’s can usually be relied on to already have unit-testing )

By the time I got to here, I decided that I DID in fact like this solution. Should but may not update to see how this went.

PyOverlord is now PyProxy aka Proxy Project

https://github.com/devdave/PyProxy

ProxyProject

  • Author: David W
  • Status: Pre-Alpha

Primary goal

ProxyProject is initially meant to resolve one very common and VERY tedious part of Web development,
validating web forms. A developer would place Proxy Project between their browser and their development environment
then run through the process of populating and submitting a form. After the developer would return to the Proxy
Project control panel and click a button, in return a python mechanize script would be presented.

Secondary goals

Since the Proxy is already recording everything, additional features would be the ability to replay the response side
of requestes on demand ( basically caching the respons ).

The ability to mechanize scripts into “story” scripts, to make it easier to re-use and organize them for building up
front loaded system under-test scripts.

Could be nice goals

A tool to marry SQL statements stored in a MySQL general log file to individual responses, perhaps make it easier to add
additional DB level post mechanize

Adding new branch on github

Branch currently uncommitted changes into a new branch
#git checkout -b new_branch
Save your changes
#git commit OR git add & commit whatever
#git push origin new_branch

Done; somewhat odd that this simple task isn’t that well document or from googling has a lot of around the world methods to accomplish the above.