Tag Archives: oh_snap

metaclass service bus decorator concept

While playing around with some idea’s for making PyProxy completely overridable ( and also potentially undebuggable ) I started playing around with metaclasses.

Below is from my toxic directory [ gist url ]

from collections import defaultdict
from functools import wraps

Just some preliminary basic utilities

A simple bus implementation, decoratedCalls is a dictionary of service calls, with a list
of callbacks to each service call hook.

decoratedCalls = defaultdict(list)
 
def call(name, *args, **kwargs):
    print name, args, kwargs
    if name in decoratedCalls:
        for cb in decoratedCalls[name]:
            cb(*args, **kwargs)

Syntactic suger to add clarity later as to what functions are being bound to what. Adding in
debug/logging hooks here could allow for easier tracing of what is called for what methods.

class Subscribe(object):
    def __init__(self, name):
        self.name = name
 
    def __call__(self, f):
        decoratedCalls[self.name].append(f)
        return f

Here’s the actual bus decorator, very simple just wraps the decorated function with a pre and post bus calls.

 
class BusDecorator(object):
    def __init__(self, name):
        self.name = name
 
    def __call__(self, f):
 
        @wraps(f)
        def decorator(inst, *args, **kwargs):
            call("%s-pre" % self.name, inst, args, kwargs)            
            retval = f(inst, *args, **kwargs)
            call("%s-post" % self.name, inst, retval)            
            return retval
        return decorator

And here’s my Bus Metaclass that combines most of the above.

The ease of wrapping the target class is accomplished by cdict which is a dictionary of
every defined attribute of the target class. As you can see it’s trivial to spin
through and decorate every callable with the BusDecorator

 
class BusWrap(type):
 
    def __new__(mcs, clsname, bases, cdict):
 
        modName = cdict.get("__module__", "unknownclass")
 
        for name in cdict.keys():
            prefix = "%s.%s.%s" % ( modName, clsname, name)
            if callable(cdict[name]):                
                cdict[name] = BusDecorator(prefix)(cdict[name])
 
        return type.__new__(mcs, name, bases, cdict)

Now give a dirt simple class like

 
class Foo(object):
    __metaclass__ = BusWrap
 
    def __init__(self):
        print "init'd"
 
    def bar(self):
        print "bar"
 
    def blah(self):
        print "blah"
 
    def ich(self):
        print "ich"
 
    def ego(self):
        print "lego"
 
    def say(self, *args):
        print "Saying ", args

And two service handlers to pre Foo.bar being called and after Foo.ego is called

 
@Subscribe("__main__.Foo.bar-pre")        
def preBar(inst, *args, **kwargs):
    if not hasattr(inst, "ext_info"):
        inst.ext_info = "Here"
 
@Subscribe("__main__.Foo.ego-post")
def postEgo(inst, *args, **kwargs):
    if hasattr(inst, "ext_info"):
        print "Extended info is ", inst.ext_info

Our test shows….

x = Foo()
x.bar()
x.blah()
x.ich()
x.ego()
x.say("abba", "dabba")

this as output

__main__.Foo.__init__-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
init'd
__main__.Foo.__init__-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.bar-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
bar
__main__.Foo.bar-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.blah-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
blah
__main__.Foo.blah-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.ich-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
ich
__main__.Foo.ich-post (<__main__.__init__ object at 0x02637890>, None) {}
__main__.Foo.ego-pre (<__main__.__init__ object at 0x02637890>, (), {}) {}
lego
__main__.Foo.ego-post (<__main__.__init__ object at 0x02637890>, None) {}
Extended info is  Here
__main__.Foo.say-pre (<__main__.__init__ object at 0x02637890>, ('abba', 'dabba'), {}) {}
Saying  ('abba', 'dabba')
__main__.Foo.say-post (<__main__.__init__ object at 0x02637890>, None) {}


A friendlier asynchronous twisted web, the ghetto monkey patch way

UPDATE to the UPDATE – A cleaned up and more coherent example of txweb is here
UPDATE – Github repo here

I like twisted, and I like Cherrypy, unfortunately just like my militant atheist friends and my more spiritual friends neither seems to get along with the other.

What to do? MONKEY PATCH + GHETTO HACKING to the rescue!

Note, this is just a mockup of CherryPy’s routing system and not a bridge or interface to CherryPy. There is no CherryPy to be had here, just ghetto py.

 
from twisted.web import server, resource
from twisted.internet import reactor
 
 
def expose(func):
    func.exposed = True
    return func
 
class PageOne(object):
 
    def foo(self, request):
        return "Hello From PageOne Foo!"        
    foo.exposed = True
 
    @expose
    def delayed(self, request):
        def delayedResponse():
            request.write("I was delayed :( ")
            request.finish()
 
        reactor.callLater(5, delayedResponse)
        return server.NOT_DONE_YET
 
 
class PageTwo(object):
 
    @expose
    def index(self, request):
        return "Hello From PageTwo index!"
 
 
class Root(object):
 
    @expose
    def index(self, request):
        return "Hello From Index!"
 
    @expose
    def __default__(self, request):
        return "I Caught %s " % request.path
 
    pageone = PageOne()
    pagetwo = PageTwo()
 
class OneTimeResource(resource.Resource):
    """
        Monkey patch to avoid rewriting more of twisted's lower web
        layer which does a fantastic job dealing with the minute details
        of receiving and sending HTTP traffic.
 
        func is a callable and exposed property in the Root OO tree
    """
    def __init__(self, func):
        self.func = func
 
    def render(self, request):
        #Here would be a fantastic place for a pre-filter
        return self.func(request)
        #ditto here for a post filter
 
 
class OverrideSite(server.Site):
    """
        A monkey patch that short circuits the normal
        resource resolution logic @ the getResourceFor point
 
    """
    def checkAction(self, controller, name):
        """
            On success, returns a bound method from the provided controller instance
            else it return None
        """
        action = None
        if hasattr(controller, name):
                action = getattr(controller, name)
                if not callable(action) or not hasattr(action, "exposed"):
                    action = None
 
        return action
 
 
    def routeRequest(self, request):
        action = None
        response = None
 
        root = parent = self.resource
        defaultAction = self.checkAction(root, "__default__")
 
        path = request.path.strip("/").split("/")
 
 
 
        for i in range(len(path)):
            element = path[i]
 
            parent = root
            root = getattr(root, element, None)
            request.prepath.append(element)
 
            if root is None:                
                break
 
            if self.checkAction(root, "__default__"):
                #Check for a catchall default action
                defaultAction = self.checkAction(root, "__default__")
 
 
            if element.startswith("_"):
                #500 simplistic security check
                action = lambda request: "500 URI segments cannot start with an underscore"
                break
 
            if callable(root) and hasattr(root, "exposed") and root.exposed == True:
                action = root
                request.postpath = path[i:] 
                break
 
 
 
        else:
            if action is None:
                if root is not None and self.checkAction(root, "index"):
                    action = self.checkAction(root, "index")
 
 
        #action = OneTimeResource(action) if action is not None else OneTimeResource(lambda request:"500 Routing error :(")
        if action is None:
            if defaultAction:
                action = defaultAction
            else:            
                action = lambda request:"404 :("
 
        return OneTimeResource(action)         
 
 
 
 
    def getResourceFor(self, request):
        return self.routeRequest(request)
 
"""
    Twisted thankfully doesn't do any type checking, so a
    dumb OO graph is A-Okay here.  It will be assigned to
    site.resource
"""
dumb = OverrideSite(Root())
 
reactor.listenTCP(80, dumb )
reactor.run()

Slapped this together in about 30 minutes… so there is a HIGH probability that it is almost entirely edge cased! Still it does work ( for me ) and it doesn’t hijack too much of twisted’s core, so it could be viable with a lot of unit-testing love, some additional sanity checking logics, and maybe some well thought out refactoring.

Komodo IDE auto-generating setters/getters for PHP

I’ve got about thirty auto-generated PHP Doctrine models that are missing their required Java style setters/getters. At class number three of typing in these accessory methods I snapped and said “There has to be a better way” and magically the universe smacked me upside the head and reminded me that I’m using Komodo IDE which just so happens to have a disgustingly powerful Python powered macro system.

Four minutes later, out came the copy & paste hacked together monstrosity below. It’s not perfect but it doesn’t need to be, just has to work well enough to save my sanity and my client’s time.

To use, follow Komodo’s help documentation for creating a new python Macro then copy and paste this code into the macro window OR a slightly easier way, make the macro then click the “edit macro” context menu property to open the macro source file as a new view in Komodo.

The macro uses some very simple rules. It’s only looking for private properties in a format of “^\s*private \$[a-zA-Z0-9_]$”, it collects all of these variable names and then appends them through the setter and getter templates to a string buffer. Finally the buffer is
inserted into the current document at the position of the cursor. The output is coherent but not whitespace friendly which isn’t too big of a deal to re-format. Note, the macro has no concept of PHP syntax, so if there is more then one class in a file, the results will not be desirable.

 
from xpcom import components
import re
 
viewSvc = components.classes["@activestate.com/koViewService;1"]\
    .getService(components.interfaces.koIViewService)
view = viewSvc.currentView.queryInterface(components.interfaces.koIScintillaView)
 
sm = view.scimoz
sm.currentPos   # current position in the editor
sm.text         # editor text
sm.selText      # the selected text
#sm.text = "Hello World!"
 
output = u"\n"
 
setterTemplate = """
    function set%s($value){
        $this->%s = $value;
    }
"""
 
getterTemplate = """
    /**
    *@return string
    */
    function get%s(){
        return $this->%s;
    }
"""
 
propertyTemplate = """
%s
 
%s
"""
 
prefixSize = len(u"private $")
 
def formalName(rawName):
    return u"%s" % "".join([part.title() for part in rawName.split("_")])
 
 
 
 
#todo find a better way to split lines, what if its Mac or Windows format?
for line in sm.text.split("\n"):
    if line.strip().startswith("private $"):
        #trim of the private $ and trailing semi-colon
        realName = line.strip()[prefixSize:-1]        
        output += propertyTemplate % ( setterTemplate %(formalName(realName), realName), getterTemplate % (formalName(realName), realName))        
 
 
 
sm.insertText(sm.currentPos, output)

Semi-critical OpenSSL memory issue shared with Node.js and Twisted Python

I found this ( http://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/ ) just a tad disturbing. The TLDR is that OpenSSL automatically sets compression on for SSL connections. That’s great if you’re trying to cut down on bandwidth… but not so great when thinking about both Memory and CPU utilization, definitely more so memory in this case.

Fortunately the post author and company created some partial duct tape solutions for anyone suffering under out of memory issues in their infrastructure.

JQuery dataTable plugin quick notes

Server-side processing

To return the names of the columns you want, you need to use aoColumndefs and provided individual objects with the property sName.

Below is most of the implementation for a jeditable Ajax driven dataTables, this is the Alpha version and still needs a lot of love in places, but the gist of it is here

var semesterTable = $("#semesterViewTable").dataTable({
                "bProcessing"   : true
            ,   "bServerSide"   : true
            ,   "sAjaxSource"   : "<?php echo url_for("roster/ajax/{$semester_id}"); ?>"
            ,   "fnRowCallback" : function( nRow, aData, iDisplayIndex, iDisplayIndexFull ) {
                                    //console.log(nRow, aData, iDisplayIndex, iDisplayIndexFull);
                                    var recordId = aData[0];
                                    $("td", nRow).data("recordId", recordId);
                                    return nRow;
                                }
            , "fnDrawCallback"  : function(){
                                    $("td.editableMajor")
                                        .editable("<?php echo url_for("roster/edit_record");?>",
                                                    {                                                        
                                                          loadurl: "<?php echo url_for("roster/distinct/major/{$semester_id}") ?>"
                                                        , type: "select"
                                                        , submit: "OK"
                                                        , method: "post"
                                                        , submitdata: function(){
                                                            console.log(this, arguments);
                                                            var $this = $(this);
                                                            return {
                                                                    recordId: $this.data("recordId")
                                                                    , classes: $this.attr("class")
                                                                }
                                                        }
                                                    });
 
                                }
            /* Seriously a meta-generator for this would be nice */
            ,"aoColumnDefs": [
                ,{ sName:"record_id",          aTargets:[00], bSearchable:false, bVisible:false }
                ,{ sName:"roster_date",         aTargets:[01], sClass:"roster_date"}
                ,{ sName:"roster_name_both",    aTargets:[02], sClass:"roster_name_both"}
                ,{ sName:"roster_student_id",   aTargets:[03], sClass:"roster_student_id"}
                ,{ sName:"roster_major1",       aTargets:[04], sClass:"roster_major1 editableMajor"}
                ,{ sName:"roster_major2",       aTargets:[05], sClass:"roster_major2 editableMajor"}
                ,{ sName:"roster_major3",       aTargets:[06], sClass:"roster_major3 editableMajor"}
                ,{ sName:"roster_major4",       aTargets:[07], sClass:"roster_major4 editableMajor"}
                ,{ sName:"roster_minor1",       aTargets:[08], sClass:"roster_minor1 editableMajor"}
                ,{ sName:"roster_minor2",       aTargets:[09], sClass:"roster_minor2 editableMajor"}
            ]
    });

For the tbody, just put something like

        <tbody><tr><td>Chill the fuck out, it's loading</td></tr></tbody>

Server side data contract

dataTables expects a JSON’ified construct like this

$responseBody = array(
		"sEcho" => intval($input['sEcho']),
		"iTotalRecords" => (int) $count,
		"iTotalDisplayRecords" => count($data),
		"aaData" => $data
            );
  • sEcho is a security variable used by dataTables, basically just send it back and don’t worry about it
  • iTotalRecords – is the MAXIMUM # of records in the data set without any filtering
  • iTotalDisplayRecords – is the MAXIMUM # of records in the data set WITH filtering
  • An array of numerically indexed arrays, dataTables doesn’t like associative arrays

Total and Display total records

If the blink tag still worked reliably, I’d put a neon sign here. Basically you need to do 3 queries per Ajax call… call 1 is to get a

 
   SELECT count(1) FROM sourceTABLE

call 2 is

 SELECT count(1) FROM sourceTable WHERE someCriteria 

and finally call 3 is

 SELECT yourColumnList FROM sourceTable WHERE someCriteria

With memcache or such, I’d advise caching the total count but no matter what you unfortunately need to make these three SQL calls :(.

Initial look at ExtJS 4

So… the graphing library look damn good, but focusing more on the nutts & bolts.

Previously ExtJS has had an on demand instantiation system vi anoymous objects with a property of xtype. It’s fairly likely that not every component of an ExtJS enhanced page is going to be used by a user or even looked at, so the xtype support allowed for avoiding costly and wasted object/component instantiations.

Now it looks like they’re taking that one step further. Looking at the Public Release 3 of ExtJS 4’s documentation, I noticed some new additions… specifically lazy class loading. If I’m correct in my assumptions, that means that instead of loading the ENTIRE ExtJS library and possibly only using a tiny bit of it, you can load up only what you need saving rendering time & bandwidth!

Ext.require([
    'widget.window',
    'widget.button',
    'layout.fit'
]);
 
Ext.onReady(function() {
    var window = Ext.widget('window', {
        width: 500,
        height: 300,
        layout: 'fit',
        items: {
            xtype: 'button',
            text: 'Hello World',
            handler: function() { alert(this.text) }
        }
    });
 
    window.show();
});

Authoritative source here

Stellar simulator thingy

Inspired by this
IF YOU CAN SEE THIS, YOUR BROWSER SUCKS!





Source code @ Github here

So… this basically covers elliptical paths ( which I want to play with more ), a rendering chain I’m happy with, and a few others things. What isn’t exactly right: speed, planetary/stellar surface animation is missing, and I’d like to have a fly by mechanism that shifts the Y ratio progressively over time.

What’s in a title?

With my latest contract, I will be celebrating almost seven years ( 6 continuous years ) as a senior consultant.  I’m comfortable with that title for some very simple reasons:  I started my first business ( onsite tech. support ) when I was 13 then got bought out by a larger firm and became a junior partner at 16.  Then after moving from Florida to Philadelphia ( trust me, official records say Philadelphia is a city but it’s really a state of it’s own ) I started college and multiple companies.  I know the cardinal rule of business, which is to provide less for more but appear to be providing more for less ) regardless of domain.

As a consultant, your technical know how usually plays second fiddle to finding ways for your client’s to be profitable and self-sustaining.  Not to say that you can be technically inept but I’ve worked with a few consultants that were/are.  It doesn’t really matter what the most elegant choice is sometimes if its going to take to long to ensure your client keeps the initiative in their business domain.  That really sucks, but its the truth.

That said, both my clients and my consultancy agencies I’ve worked for have thrown all sorts of crazy titles at me.  Chief engineer, chief sys. admin, architect, senior engineer, lead engineer, software developer, etc.  I’ve been somewhat dumbfounded by the naming choices as the reality doesn’t always match the title.

During the interview for my recent client, when they asked what grade/title I felt I was at I told them I was a competent software engineer with full stack experience.  It would have been quicker to just say “I’m the mother fucking man” but I refuse to do that because all it takes is one little estoric piece of knowledge I don’t know about to blow that card house down.  Somehow we got into a monologue where I explained my opinion on titles:

Apprentice

What’s the difference between a POST & GET based form?   I don’t know how to setup a development workstation.   Of course I know SOAP but why do we have to use all this XML?  Why do I need to use a quicksort when this works? ( emergent bubble sort ).

I don’t know what you’re saying.

Journeymen

Caching is stupid, it gets in the way.  Why should we use that open source framework when we can write our own?  Documentation sucks and we don’t have enough time to implement unit-testing!  Yo dawg, I herd you like patterns so I put a singleton factory in your observer so that I can command pattern inject your dependancies…. uhm no, I’ve got no idea how to fix that bug, let me get back to you in a couple days ( turns out to be an entire week).   But we’re going to need all this extra complexity because I’m bored and figured will need it someday soon.

I don’t know how to implement that and won’t tell you that until I’ve fucked things all to hell.

Craftsmen

I think I can get that done in the needed timeframe.  I hate this language/framework/database but will bear down and get this done while looking for opportunities to make things better.  I refactored feature X to use a factory pattern and will make notes to try and reinforce that change elsewhere.  I’m hesistent to add more complexity here and realize this will be a problem in the future so I’m going to add a quick unit-test to ensure integrity.

Here is how I would implement that, what do you think?

Master

No I won’t/can’t implement that right now until we get the core features down.  Why don’t you like technology X,Y,Z and how can we compromise?  I appreciate your input and will take that into consideration.  Hey boss, check out what minion 12 accomplished, I think he/she/it has a bright future.   It’s unacceptable to modify the system tier framework to resolve your issue, asking the framework mailing list/group for advice/criticism and if they accept it then submit a path.

I will delivery these features in this time frame and if I can’t I will tell you why and a better target date.

Summary

I’ve been writing software since I was 7 or 8, I’ve made a couple million dollars in revenue writing software over a decade or two, and my clients have made several more million using that software.  I’ve worked with people with decades of “experience” but no practical knowledge and inversely people with literally a year or two of experience but are masters at their trade.  I remember working with a Ruby on Rails developer a couple years back who was a goddamn bad ass combat developer knocking out reliable code and pulling epic 90 hours weeks..he learned to program the year prior.  The conclusion is that I think titles are a bit of crap.  Sticking around long enough with a company and you’ll end up being the lead/chief/senior company minion by default.  OR have enough clients you half-assed through and you can claim the title of senior… but neither is an indicator of competence.