Tag Archives: python

Flask CRUD with sqlalchemy and jinja2 contextfilters

Quick disclaimer, the Flask CRUD thing is not public domain yet and is very volatile.

The project is here
https://github.com/devdave/wfmastery/tree/revamp_1/wfmastery
And the outline for the crud thing is in this commit https://github.com/devdave/wfmastery/commit/e249895ddc53c0696f59d3def5718e76855af5b9

https://github.com/devdave/wfmastery/blob/revamp_1/wfmastery/crud.py
https://github.com/devdave/wfmastery/blob/revamp_1/wfmastery/views.py
https://github.com/devdave/wfmastery/blob/revamp_1/wfmastery/templates/equipment_list.j2.html

First is how the crud is currently constructed

class Equipment(CrudAPI):
 
    def populate(self):
 
        self.record_cls = db.Equipment
        self.identity = "equipment"
 
        self.template_form = "equipment_form.j2.html"
        self.template_list = "equipment_list.j2.html"
 
        self._listColumn("id")
        self._listColumn("hidden")
        self._listColumn("name", magic_field="magic-string")
        self._listColumn("pretty_name", magic_field="magic-string")
 
        self._addRelationship("category", "name", magic_field="magic-filter")
        self._addRelationship("subcategory", "name", magic_field="magic-filter")

both vars “template_form” and “template_list” are going to be preset once I am certain that the templates can stand on their own with the context vars provided. The “magic-” params and their use are very much magic (eg really toxic) and would recommend ignoring them.

From there the CrudAPI takes over. Skipping ahead to how this relates to context filters. I had this tag mess here in the template

-{%-      for column_name in origin.list_columns -%}
 -{%-          if column_name in origin.magic_columns -%}
 -        {{ cell("", column_name|title, classes=origin.magic_columns[column_name]) -}}
 -{%-          else -%}
 -        {{ cell("", column_name|title) -}}
 -{%          endif %}
 -{%-      endfor %}

and was really not happy with it. So I dived into Flask and Jinja2’s documentation and code to figure out if I could apply Python code inline.

The answer is yes via jinja2’s contextfilters which are not exposed to Flask but can still be used.

@App.template_filter("render_header")
def render_header(context, column_name, value="", **kwargs):
    result = ""
    if column_name in context['origin'].magic_columns:
        result = context['cell'](value, column_name.capitalize(), classes=context['origin'].magic_columns[column_name])
    else:
        result = context['cell'](value, column_name.capitalize())
 
 
    return result
 
render_header.contextfilter=True

The trick to going from filter to contextfilter is just applying `my_func.contextfilter = True` outside of your functions scope. From there you have access to almost everything (if not everything). The var “origin” is the CrudAPI’s instance passed to the template.

This has opened a lot more opportunities to do clean up. Taking

 
{% macro data_attributes(data_map, prefix="data-") -%}
 
    {%- for name, value in data_map.items() -%}
    {{" "}}{{prefix}}{{name}}="{{value}}"
    {%- endfor -%}
{%- endmacro %}
 
 
{% macro cell(name, value, classes=None, data_attrs={}) %}
        <span class="{{- ["cell",classes]|join(" ") if classes else "cell" -}}"{{data_attributes(data_attrs)}}>
      {{- caller() if caller else value -}}</span>
  {%- endmacro -%}

and condensing it down to

{% macro cell(name, value, classes=None, data_attrs={}) %}
        <span class="{{- ["cell",classes]|join(" ") if classes else "cell" -}}" {{data_attrs|dict2attrs("data")|safe}}>
    {{- caller() if caller else value -}}</span>
{%- endmacro -%}

via a simple non-context filter

@App.template_filter("dict2attrs")
def dict_to_attributes(attributes, prefix=None):
    results = []
    name2dash = lambda *x: "-".join(x)
    format_str = "%s-{}=\"{}\"" % prefix if prefix else "{}=\"{}\""
 
    for key, value in attributes.items():
        results.append(format_str.format(key, value))
 
    #TODO disable autoescape
    return " ".join(results)

Just note that at the moment output is still managed by Jinja’s autoescape and I’d rather not shut that off so calls MUST be suffixed with “|safe” as used above.

As for the Crud API, I feel like that is coming along nicely.

Flask list routes (rake equivalent).

While working on a pet project I really wanted a rails rake equivalent for Flask.

Googling led to http://flask.pocoo.org/snippets/117/ which gave me enough direction to figure out how to make that work with Python 3.5 and Flask 0.12.

The biggest problem I had with that snippet is that it didn’t account for URL variable rules.

/foo/<int:bar>

as it would blow up because werkzeug & Flask sanitize inputs to ensure they match the expected type.

I started doing some seriously crazy stuff like monkey patching the rules temporarily to make
ALL converters run through a String/Unicode converter. It’s at this point that I noticed in dbgp (symbolic debugger) that it was naturally converting the rules to strings.

@App.cli.command("list_routes")
def list_routes():
    """
        Roll through Flask's URL rules and print them out
        Thank you to Jonathan Tushman
            And Thank you to Roger Pence
            Sourced http://flask.pocoo.org/snippets/117/ "Helper to list routes (like Rail's rake routes)"
 
        Note that a lot has possibly changed since that snippet and rule classes have a __str__
            which greatly simplifies all of this
    """
 
 
    format_str = lambda *x: "{:30s} {:40s} {}".format(*x)#pylint: disable=W0108
    clean_map = defaultdict(list)
 
 
    for rule in App.url_map.iter_rules():
        methods = ",".join(rule.methods)
        clean_map[rule.endpoint].append((methods, str(rule),))
 
    print(format_str("View handler", "HTTP METHODS", "URL RULE"))
    print("-"*80)
    for endpoint in sorted(clean_map.keys()):
        for rule, methods in sorted(clean_map[endpoint], key=lambda x: x[1]):
            print(format_str(endpoint, methods, rule))

Example output

"HOSTED@LOCAL:5000"
View handler                   HTTP METHODS                             URL RULE
--------------------------------------------------------------------------------
Equipment                      /equipment/                              OPTIONS,POST
Equipment                      /equipment/                              HEAD,OPTIONS,GET
Equipment                      /equipment/<int:record_id>               HEAD,OPTIONS,GET,PUT,DELETE
index                          /                                        HEAD,OPTIONS,GET
static                         /static/<path:filename>                  HEAD,OPTIONS,GET

metaclass service bus decorator concept

While playing around with some idea’s for making PyProxy completely overridable ( and also potentially undebuggable ) I started playing around with metaclasses.

Below is from my toxic directory [ gist url ]

from collections import defaultdict
from functools import wraps

Just some preliminary basic utilities

A simple bus implementation, decoratedCalls is a dictionary of service calls, with a list
of callbacks to each service call hook.

decoratedCalls = defaultdict(list)
 
def call(name, *args, **kwargs):
    print name, args, kwargs
    if name in decoratedCalls:
        for cb in decoratedCalls[name]:
            cb(*args, **kwargs)

Syntactic suger to add clarity later as to what functions are being bound to what. Adding in
debug/logging hooks here could allow for easier tracing of what is called for what methods.

class Subscribe(object):
    def __init__(self, name):
        self.name = name
 
    def __call__(self, f):
        decoratedCalls[self.name].append(f)
        return f

Here’s the actual bus decorator, very simple just wraps the decorated function with a pre and post bus calls.

 
class BusDecorator(object):
    def __init__(self, name):
        self.name = name
 
    def __call__(self, f):
 
        @wraps(f)
        def decorator(inst, *args, **kwargs):
            call("%s-pre" % self.name, inst, args, kwargs)            
            retval = f(inst, *args, **kwargs)
            call("%s-post" % self.name, inst, retval)            
            return retval
        return decorator

And here’s my Bus Metaclass that combines most of the above.

The ease of wrapping the target class is accomplished by cdict which is a dictionary of
every defined attribute of the target class. As you can see it’s trivial to spin
through and decorate every callable with the BusDecorator

 
class BusWrap(type):
 
    def __new__(mcs, clsname, bases, cdict):
 
        modName = cdict.get("__module__", "unknownclass")
 
        for name in cdict.keys():
            prefix = "%s.%s.%s" % ( modName, clsname, name)
            if callable(cdict[name]):                
                cdict[name] = BusDecorator(prefix)(cdict[name])
 
        return type.__new__(mcs, name, bases, cdict)

Now give a dirt simple class like

 
class Foo(object):
    __metaclass__ = BusWrap
 
    def __init__(self):
        print "init'd"
 
    def bar(self):
        print "bar"
 
    def blah(self):
        print "blah"
 
    def ich(self):
        print "ich"
 
    def ego(self):
        print "lego"
 
    def say(self, *args):
        print "Saying ", args

And two service handlers to pre Foo.bar being called and after Foo.ego is called

 
@Subscribe("__main__.Foo.bar-pre")        
def preBar(inst, *args, **kwargs):
    if not hasattr(inst, "ext_info"):
        inst.ext_info = "Here"
 
@Subscribe("__main__.Foo.ego-post")
def postEgo(inst, *args, **kwargs):
    if hasattr(inst, "ext_info"):
        print "Extended info is ", inst.ext_info

Our test shows….

x = Foo()
x.bar()
x.blah()
x.ich()
x.ego()
x.say("abba", "dabba")

this as output

__main__.Foo.__init__-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
init'd
__main__.Foo.__init__-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.bar-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
bar
__main__.Foo.bar-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.blah-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
blah
__main__.Foo.blah-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.ich-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
ich
__main__.Foo.ich-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.ego-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
lego
__main__.Foo.ego-post (<__main__.__init__ object at 0x02637890>, None) {}
Extended info is  Here
__main__.Foo.say-pre (<__main__.__init__ object at 0x02637890>, ('abba', 'dabba'), {}) {}
Saying  ('abba', 'dabba')
__main__.Foo.say-post (<__main__.__init__ object at 0x02637890>, None) {}


Plugin’s for python

As the #1 google result for “python plugin”, the linked to blog post is extremely valuable for jump starting research into implementing your own plugin system ( like me ) or finding one that is viable for implementation in your project ( possibly like me ).

These resources highlight one reason why there is not a standard Python plug-in framework: there are a variety of different capabilities that a user may want, and the complexity of the framework generally increases as these new capabilities are added

http://wehart.blogspot.com/2009/01/python-plugin-frameworks.html

TxWeb Alpha – A different spin on twisted.web

I’ve spent some more time on my current pet, txweb, and I think it’s pretty much at the as good as it gets stage.

Below is the source for the example.py

 
#App level
from txweb import Site, expose
#twisted
from twisted.web import server, resource
from twisted.internet import reactor
from twisted.web.static import File
 
from os.path import abspath, dirname, join

Mostly pretty standard imports for a twisted.web application.

Now here is the “Controllers”, they’re stripped down to bare-bones just to keep it simple

class PageOne(object):
 
    @expose
    def foo(self, request):
        return "Hello From PageOne Foo!"        
 
    @expose
    def delayed(self, request):
        def delayedResponse():
            request.write("I was delayed :( ")
            request.finish()
 
        reactor.callLater(5, delayedResponse)
        return server.NOT_DONE_YET
 
 
class PageTwo(object):
 
    @expose
    def index(self, request):
        """ /pagetwo/index """
        return "Hello From PageTwo index!"
 
 
rootFile = lambda filename : abspath(join(dirname(__file__), filename))
 
class Root(object):
 
    @expose
    def index(self, request):
        """
            Will handle both / and /index paths
        """
        return "Hello From Index!"
 
    @expose
    def __default__(self, request):
        """
            Unless overriden further down, this will catch all 404's
        """
        return "I Caught %s " % request.path
 
    pageone = PageOne()
    pagetwo = PageTwo()
 
    readme  = File(rootFile("README.md"))
    license = File(rootFile("txweb/LICENSE.txt"))

Basically a txWeb enabled twisted service converts a URL path to an Object path.

So /hello/world could resolve to root.hello.world() if such a construct was provided.

Much more importantly, with the above example, /license resolves to the local file txweb/LICENSE and /readme resolves to README.md !

In summary txweb doesn’t throw away the epic amount of work the Twisted developers and volunteers have put forth, it just presents it in another way.

A friendlier asynchronous twisted web, the ghetto monkey patch way

UPDATE to the UPDATE – A cleaned up and more coherent example of txweb is here
UPDATE – Github repo here

I like twisted, and I like Cherrypy, unfortunately just like my militant atheist friends and my more spiritual friends neither seems to get along with the other.

What to do? MONKEY PATCH + GHETTO HACKING to the rescue!

Note, this is just a mockup of CherryPy’s routing system and not a bridge or interface to CherryPy. There is no CherryPy to be had here, just ghetto py.

 
from twisted.web import server, resource
from twisted.internet import reactor
 
 
def expose(func):
    func.exposed = True
    return func
 
class PageOne(object):
 
    def foo(self, request):
        return "Hello From PageOne Foo!"        
    foo.exposed = True
 
    @expose
    def delayed(self, request):
        def delayedResponse():
            request.write("I was delayed :( ")
            request.finish()
 
        reactor.callLater(5, delayedResponse)
        return server.NOT_DONE_YET
 
 
class PageTwo(object):
 
    @expose
    def index(self, request):
        return "Hello From PageTwo index!"
 
 
class Root(object):
 
    @expose
    def index(self, request):
        return "Hello From Index!"
 
    @expose
    def __default__(self, request):
        return "I Caught %s " % request.path
 
    pageone = PageOne()
    pagetwo = PageTwo()
 
class OneTimeResource(resource.Resource):
    """
        Monkey patch to avoid rewriting more of twisted's lower web
        layer which does a fantastic job dealing with the minute details
        of receiving and sending HTTP traffic.
 
        func is a callable and exposed property in the Root OO tree
    """
    def __init__(self, func):
        self.func = func
 
    def render(self, request):
        #Here would be a fantastic place for a pre-filter
        return self.func(request)
        #ditto here for a post filter
 
 
class OverrideSite(server.Site):
    """
        A monkey patch that short circuits the normal
        resource resolution logic @ the getResourceFor point
 
    """
    def checkAction(self, controller, name):
        """
            On success, returns a bound method from the provided controller instance
            else it return None
        """
        action = None
        if hasattr(controller, name):
                action = getattr(controller, name)
                if not callable(action) or not hasattr(action, "exposed"):
                    action = None
 
        return action
 
 
    def routeRequest(self, request):
        action = None
        response = None
 
        root = parent = self.resource
        defaultAction = self.checkAction(root, "__default__")
 
        path = request.path.strip("/").split("/")
 
 
 
        for i in range(len(path)):
            element = path[i]
 
            parent = root
            root = getattr(root, element, None)
            request.prepath.append(element)
 
            if root is None:                
                break
 
            if self.checkAction(root, "__default__"):
                #Check for a catchall default action
                defaultAction = self.checkAction(root, "__default__")
 
 
            if element.startswith("_"):
                #500 simplistic security check
                action = lambda request: "500 URI segments cannot start with an underscore"
                break
 
            if callable(root) and hasattr(root, "exposed") and root.exposed == True:
                action = root
                request.postpath = path[i:] 
                break
 
 
 
        else:
            if action is None:
                if root is not None and self.checkAction(root, "index"):
                    action = self.checkAction(root, "index")
 
 
        #action = OneTimeResource(action) if action is not None else OneTimeResource(lambda request:"500 Routing error :(")
        if action is None:
            if defaultAction:
                action = defaultAction
            else:            
                action = lambda request:"404 :("
 
        return OneTimeResource(action)         
 
 
 
 
    def getResourceFor(self, request):
        return self.routeRequest(request)
 
"""
    Twisted thankfully doesn't do any type checking, so a
    dumb OO graph is A-Okay here.  It will be assigned to
    site.resource
"""
dumb = OverrideSite(Root())
 
reactor.listenTCP(80, dumb )
reactor.run()

Slapped this together in about 30 minutes… so there is a HIGH probability that it is almost entirely edge cased! Still it does work ( for me ) and it doesn’t hijack too much of twisted’s core, so it could be viable with a lot of unit-testing love, some additional sanity checking logics, and maybe some well thought out refactoring.

PyProxy – Aka the development Helper proxy

Coming out of nothing and into supah doopa Alpha is finally a working proof of concept of my python web proxy. I don’t really want to talk about the asinine alternatives I’ve tried until I finally said “fuck it, time to go completely twisted!” Low and behold the actual proxy part is 4 lines of code, which is then expanded to maybe 20-30 to allow for overloading some lower level classes.

Originally a public announcement for this project would have been in August at the earliest, give me time to clean things up and go from proof of concept to working concept but apparently a lot of other people have similar thoughts and I figured it’s better to collaborate then compete.

So some quick notes:
The ultimate goal for PyProxy ( or whatever it ends up being named ) is to sit between a developer and a development server. The first and immediate idea for this was to automagically parse out Python mechanize scripts to replicate the traffic. These mechanize scripts could then be collected into a suite, marking other scripts as requirements ( example login process ). That alone would make it pretty easy to create full system under test unit-tests. The next idea was to add in regex or pattern based hooks that could allow a developer to dial in to a specific domain, or even a specific set of webpages.

After that, the idea was to just continually tack on support plugins and scripts, maybe tell PyProxy the name of the target application’s database, and if it’s MySQL, switch on the general log. This could allow for combining both mechanize scripts AND a SQLObject or SQLAlchemy powered unit-test suite to assert that the correct data was changed.

The final future idea was to make a Firefox/Chrome extension that would allow a developer to control some parts of the proxy from their browser and also see additional information. For Python and PHP web apps, imagine have a finalization plugin that appended a response header listing all File’s used to perform a request…. then imagine having a “click to edit” button that, if the dev. instance is workstation local, would have your favorite IDE open the specified file for editing.

All in all, I think these are really subtle idea’s that if combined together, would cut down some mudane parts of developing a web app.

GitHub repo (https://github.com/devdave/PyProxy) here

Python pydoc module

Python documentation here

Example

$python -m pydoc pydoc

Talk about eating your own dogfood! This is exactly like using the help() function in the python command line interpreter… except accessible from your shell prompt…. but wait!

It’ gets better! Not only does it make julian fries ( may not for any implementation ) but it’s got a few versatile little secrets

$ python -m pydoc
pydoc - the Python documentation tool
 
pydoc.py <name> ...
    Show text documentation on something.  <name> may be the name of a
    Python keyword, topic, function, module, or package, or a dotted
    reference to a class or function within a module or module in a
    package.  If <name> contains a '/', it is used as the path to a
    Python source file to document. If name is 'keywords', 'topics',
    or 'modules', a listing of these things is displayed.
 
pydoc.py -k <keyword>
    Search for a keyword in the synopsis lines of all available modules.
 
pydoc.py -p <port>
    Start an HTTP server on the given port on the local machine.
 
pydoc.py -g
    Pop up a graphical interface for finding and serving documentation.
 
pydoc.py -w <name> ...
    Write out the HTML documentation for a module to a file in the current
    directory.  If <name> contains a '/', it is treated as a filename; if
    it names a directory, documentation is written for all the contents.

Now the graphical interface isn’t anything to write home about, but the -p option provides a no thrills web interface to
almost everything accessible to your python interpreter. This can make it slightly easier to troll through foreign modules
looking for undocumented sub modules and classes… or having an accessible reference doc for properly managed modules

Python urllib

Python documentation here

Unfortunately there is little or no documentation on the command line properties of urllib but it does recognize everything that urllib can handle. So
python -m urllib http://website.com will grab the specified url and print to std out

Note FTP works as well but you need to follow the pattern ftp://user:password@website.com if authentication is required