Category Archives: python

A friendlier asynchronous twisted web, the ghetto monkey patch way

UPDATE to the UPDATE – A cleaned up and more coherent example of txweb is here
UPDATE – Github repo here

I like twisted, and I like Cherrypy, unfortunately just like my militant atheist friends and my more spiritual friends neither seems to get along with the other.

What to do? MONKEY PATCH + GHETTO HACKING to the rescue!

Note, this is just a mockup of CherryPy’s routing system and not a bridge or interface to CherryPy. There is no CherryPy to be had here, just ghetto py.

 
from twisted.web import server, resource
from twisted.internet import reactor
 
 
def expose(func):
    func.exposed = True
    return func
 
class PageOne(object):
 
    def foo(self, request):
        return "Hello From PageOne Foo!"        
    foo.exposed = True
 
    @expose
    def delayed(self, request):
        def delayedResponse():
            request.write("I was delayed :( ")
            request.finish()
 
        reactor.callLater(5, delayedResponse)
        return server.NOT_DONE_YET
 
 
class PageTwo(object):
 
    @expose
    def index(self, request):
        return "Hello From PageTwo index!"
 
 
class Root(object):
 
    @expose
    def index(self, request):
        return "Hello From Index!"
 
    @expose
    def __default__(self, request):
        return "I Caught %s " % request.path
 
    pageone = PageOne()
    pagetwo = PageTwo()
 
class OneTimeResource(resource.Resource):
    """
        Monkey patch to avoid rewriting more of twisted's lower web
        layer which does a fantastic job dealing with the minute details
        of receiving and sending HTTP traffic.
 
        func is a callable and exposed property in the Root OO tree
    """
    def __init__(self, func):
        self.func = func
 
    def render(self, request):
        #Here would be a fantastic place for a pre-filter
        return self.func(request)
        #ditto here for a post filter
 
 
class OverrideSite(server.Site):
    """
        A monkey patch that short circuits the normal
        resource resolution logic @ the getResourceFor point
 
    """
    def checkAction(self, controller, name):
        """
            On success, returns a bound method from the provided controller instance
            else it return None
        """
        action = None
        if hasattr(controller, name):
                action = getattr(controller, name)
                if not callable(action) or not hasattr(action, "exposed"):
                    action = None
 
        return action
 
 
    def routeRequest(self, request):
        action = None
        response = None
 
        root = parent = self.resource
        defaultAction = self.checkAction(root, "__default__")
 
        path = request.path.strip("/").split("/")
 
 
 
        for i in range(len(path)):
            element = path[i]
 
            parent = root
            root = getattr(root, element, None)
            request.prepath.append(element)
 
            if root is None:                
                break
 
            if self.checkAction(root, "__default__"):
                #Check for a catchall default action
                defaultAction = self.checkAction(root, "__default__")
 
 
            if element.startswith("_"):
                #500 simplistic security check
                action = lambda request: "500 URI segments cannot start with an underscore"
                break
 
            if callable(root) and hasattr(root, "exposed") and root.exposed == True:
                action = root
                request.postpath = path[i:] 
                break
 
 
 
        else:
            if action is None:
                if root is not None and self.checkAction(root, "index"):
                    action = self.checkAction(root, "index")
 
 
        #action = OneTimeResource(action) if action is not None else OneTimeResource(lambda request:"500 Routing error :(")
        if action is None:
            if defaultAction:
                action = defaultAction
            else:            
                action = lambda request:"404 :("
 
        return OneTimeResource(action)         
 
 
 
 
    def getResourceFor(self, request):
        return self.routeRequest(request)
 
"""
    Twisted thankfully doesn't do any type checking, so a
    dumb OO graph is A-Okay here.  It will be assigned to
    site.resource
"""
dumb = OverrideSite(Root())
 
reactor.listenTCP(80, dumb )
reactor.run()

Slapped this together in about 30 minutes… so there is a HIGH probability that it is almost entirely edge cased! Still it does work ( for me ) and it doesn’t hijack too much of twisted’s core, so it could be viable with a lot of unit-testing love, some additional sanity checking logics, and maybe some well thought out refactoring.

PyProxy – Aka the development Helper proxy

Coming out of nothing and into supah doopa Alpha is finally a working proof of concept of my python web proxy. I don’t really want to talk about the asinine alternatives I’ve tried until I finally said “fuck it, time to go completely twisted!” Low and behold the actual proxy part is 4 lines of code, which is then expanded to maybe 20-30 to allow for overloading some lower level classes.

Originally a public announcement for this project would have been in August at the earliest, give me time to clean things up and go from proof of concept to working concept but apparently a lot of other people have similar thoughts and I figured it’s better to collaborate then compete.

So some quick notes:
The ultimate goal for PyProxy ( or whatever it ends up being named ) is to sit between a developer and a development server. The first and immediate idea for this was to automagically parse out Python mechanize scripts to replicate the traffic. These mechanize scripts could then be collected into a suite, marking other scripts as requirements ( example login process ). That alone would make it pretty easy to create full system under test unit-tests. The next idea was to add in regex or pattern based hooks that could allow a developer to dial in to a specific domain, or even a specific set of webpages.

After that, the idea was to just continually tack on support plugins and scripts, maybe tell PyProxy the name of the target application’s database, and if it’s MySQL, switch on the general log. This could allow for combining both mechanize scripts AND a SQLObject or SQLAlchemy powered unit-test suite to assert that the correct data was changed.

The final future idea was to make a Firefox/Chrome extension that would allow a developer to control some parts of the proxy from their browser and also see additional information. For Python and PHP web apps, imagine have a finalization plugin that appended a response header listing all File’s used to perform a request…. then imagine having a “click to edit” button that, if the dev. instance is workstation local, would have your favorite IDE open the specified file for editing.

All in all, I think these are really subtle idea’s that if combined together, would cut down some mudane parts of developing a web app.

GitHub repo (https://github.com/devdave/PyProxy) here

Komodo IDE auto-generating setters/getters for PHP

I’ve got about thirty auto-generated PHP Doctrine models that are missing their required Java style setters/getters. At class number three of typing in these accessory methods I snapped and said “There has to be a better way” and magically the universe smacked me upside the head and reminded me that I’m using Komodo IDE which just so happens to have a disgustingly powerful Python powered macro system.

Four minutes later, out came the copy & paste hacked together monstrosity below. It’s not perfect but it doesn’t need to be, just has to work well enough to save my sanity and my client’s time.

To use, follow Komodo’s help documentation for creating a new python Macro then copy and paste this code into the macro window OR a slightly easier way, make the macro then click the “edit macro” context menu property to open the macro source file as a new view in Komodo.

The macro uses some very simple rules. It’s only looking for private properties in a format of “^\s*private \$[a-zA-Z0-9_]$”, it collects all of these variable names and then appends them through the setter and getter templates to a string buffer. Finally the buffer is
inserted into the current document at the position of the cursor. The output is coherent but not whitespace friendly which isn’t too big of a deal to re-format. Note, the macro has no concept of PHP syntax, so if there is more then one class in a file, the results will not be desirable.

 
from xpcom import components
import re
 
viewSvc = components.classes["@activestate.com/koViewService;1"]\
    .getService(components.interfaces.koIViewService)
view = viewSvc.currentView.queryInterface(components.interfaces.koIScintillaView)
 
sm = view.scimoz
sm.currentPos   # current position in the editor
sm.text         # editor text
sm.selText      # the selected text
#sm.text = "Hello World!"
 
output = u"\n"
 
setterTemplate = """
    function set%s($value){
        $this->%s = $value;
    }
"""
 
getterTemplate = """
    /**
    *@return string
    */
    function get%s(){
        return $this->%s;
    }
"""
 
propertyTemplate = """
%s
 
%s
"""
 
prefixSize = len(u"private $")
 
def formalName(rawName):
    return u"%s" % "".join([part.title() for part in rawName.split("_")])
 
 
 
 
#todo find a better way to split lines, what if its Mac or Windows format?
for line in sm.text.split("\n"):
    if line.strip().startswith("private $"):
        #trim of the private $ and trailing semi-colon
        realName = line.strip()[prefixSize:-1]        
        output += propertyTemplate % ( setterTemplate %(formalName(realName), realName), getterTemplate % (formalName(realName), realName))        
 
 
 
sm.insertText(sm.currentPos, output)

Semi-critical OpenSSL memory issue shared with Node.js and Twisted Python

I found this ( http://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/ ) just a tad disturbing. The TLDR is that OpenSSL automatically sets compression on for SSL connections. That’s great if you’re trying to cut down on bandwidth… but not so great when thinking about both Memory and CPU utilization, definitely more so memory in this case.

Fortunately the post author and company created some partial duct tape solutions for anyone suffering under out of memory issues in their infrastructure.

Python pydoc module

Python documentation here

Example

$python -m pydoc pydoc

Talk about eating your own dogfood! This is exactly like using the help() function in the python command line interpreter… except accessible from your shell prompt…. but wait!

It’ gets better! Not only does it make julian fries ( may not for any implementation ) but it’s got a few versatile little secrets

$ python -m pydoc
pydoc - the Python documentation tool
 
pydoc.py <name> ...
    Show text documentation on something.  <name> may be the name of a
    Python keyword, topic, function, module, or package, or a dotted
    reference to a class or function within a module or module in a
    package.  If <name> contains a '/', it is used as the path to a
    Python source file to document. If name is 'keywords', 'topics',
    or 'modules', a listing of these things is displayed.
 
pydoc.py -k <keyword>
    Search for a keyword in the synopsis lines of all available modules.
 
pydoc.py -p <port>
    Start an HTTP server on the given port on the local machine.
 
pydoc.py -g
    Pop up a graphical interface for finding and serving documentation.
 
pydoc.py -w <name> ...
    Write out the HTML documentation for a module to a file in the current
    directory.  If <name> contains a '/', it is treated as a filename; if
    it names a directory, documentation is written for all the contents.

Now the graphical interface isn’t anything to write home about, but the -p option provides a no thrills web interface to
almost everything accessible to your python interpreter. This can make it slightly easier to troll through foreign modules
looking for undocumented sub modules and classes… or having an accessible reference doc for properly managed modules

Python urllib

Python documentation here

Unfortunately there is little or no documentation on the command line properties of urllib but it does recognize everything that urllib can handle. So
python -m urllib http://website.com will grab the specified url and print to std out

Note FTP works as well but you need to follow the pattern ftp://user:password@website.com if authentication is required

Python module dis

Python documentation here

Example

python -m dis myFile.py provides an interesting look into a python file’s guts
I could easily imagine this being part of some sort of static time inspect system where dis sits at the front and a parse
walks down the output lines, turning the data into a dependency and symbol graph. Unfortunately it doesn’t seem to provide anything more and is really just a test function most likely intended for unit-testing the python stdlib.

Python SimpleHTTPServer

When working on pure javascript applications ( canvas widgets & such ), I’ve found using the SimpleHTTPServer disgustingly useful as it serves the current working directly without much thrills.

Python documentation here

Usage

$ python -m SimpleHTTPServer 8081 0.0.0.0
Serving HTTP on 0.0.0.0 port 8081 ...

Note that it’s not necessary to set the 2nd argument to 0.0.0.0 if you want the service to listen on all routes. It normally will by default listen on everything… just habit for me to always append that.

Another useful part of this server is that it servers an apache directory style listing of all file’s present unless there is a valid index file like index.htm present.