Category Archives: Uncategorized

Google Discovery service

I am dumbfounded I haven’t heard of this service before, it’s pretty clever. Anyone familiar with WSDL or Soap is going to instantly grok this.

For example, here’s the Google Analytics API discovery document https://www.googleapis.com/discovery/v1/apis/analytics/v3/rest

With this data set, it’s relatively trivial to auto-generate an interface / command pattern instance to make the user feel like they’re actually using the API directly. ¬†Furthermore the discovery document provides a fairly good reference breakdown of required arguments, argument types, and basic validation type hints to developers.

A trivial example, here’s how to grab the first profile for the currently authenticated user in ¬†analytics:

analytics = client.discovered_api("analytics",'v3')

#Grab the first profile
result = client.execute(analytics.management.accounts.list)
print result.data.items[0]

Until I called discover on analytics, my client had no idea how to deal with the Analytics API, but now it not only knows how to work with the remote service but can provide basic validations, some advanced error handling ( bad/wrong input, connectivity issues) and in an interactive console in Ruby or Python I can explorer the api as needed.

My business DSL connection was out this evening, so I wrote up a quick script to track how long exactly it was out for.

import envoy
import time

roller = 0
Limit = 50
while Limit > 0:
    rs = envoy.run("ping google.com")
    if rs.status_code != 1:
        print "PING %s" % time.ctime()
        Limit -= 1
        time.sleep(10)        
    
    roller += 1
    if roller % 100:
        print "PONG %s" % time.ctime()

Envoy is probably one of my favorite utility libraries after the Http Request library. Github repo is here https://github.com/kennethreitz/envoy/blob/master/README.rst

Trial’s & Tribulations with Apple push notification

So far really not impressed with Apple’s APN service, especially it’s sandbox/development tier.

To get started, I recommend reading the Founder/CTO of Server Density’s recipe/guide to getting started with APN, here ( http://blog.serverdensity.com/2009/07/10/how-to-build-an-apple-push-notification-provider-server-tutorial/ )

Because of how many times I had to clean slate my dev. environment, I tried to streamline the process of converting the APN certificate and my private key. In the process of converting the private key to pem format, you do have to provide a pass phrase.

echo "Convert certificate"
openssl pkcs12 -clcerts -nokeys -out apns-dev-cert.pem -in apns-dev-cert.p12

echo "Converting key"
echo "You must provide a PEM phrase, it will be stripped out in next step"
openssl pkcs12 -nocerts -out apns-dev-key.pem -in apns-dev-key.p12

echo "Stripping off PEM phrase"
openssl rsa -in apns-dev-key.pem -out apns-dev-key-noenc.pem

echo "Concatenating keys"
cat apns-dev-cert.pem apns-dev-key-noenc.pem > apns-dev.pem

One SERIOUS word of warning about APN is a scenario where you *known* beyond a shadow of a doubt that your APN requests are properly formed and the TLS connection works as expected but nothing happens. To repeat, everything looks like it works but you don’t get any APN’s to your device. My advice is to delete your APN certificate from your keychain and re-download the APN certificate, convert it again, and try again.

To verify your key is correct, the command line openssl utility is somewhat invaluable

 openssl s_client -connect gateway.sandbox.push.apple.com:2195 -cert apns-dev-cert.pem -key apns-dev-key-noenc.pem

There should be no errors and the console should hang immediately after the — line until you type in some garbage.

Apple: Anti-developer

As I said earlier, I am really not impressed with APN for one specific reason. I lost about 4-5 hours of my life yesterday trying various different APN libraries for PHP, Ruby, Python, and objective-c. My payload to APN was correct, my token ID was triple verified, and I was sure I was sending the data down the wire. Meanwhile I got no error messages or warnings that my certificate was out of date/order and because of that my APN requests were being ignored.

The worst technology on the planet isn’t that what gives ambigious/cryptic error messages but that which just silently fails with no means to diagnose the problem. Hell it’s worse that the infamous “There was an error, somewhere, sometime.” joke.

Selective Dropbox sync for Ubuntu

  1. Find a good tutorial to setup the CLI Dropbox agent – This one is the most concise IMO
  2. Grab this http://www.dropbox.com/download?dl=packages/dropbox.py
  3. `dropbox.py exclude add` is the command you need

Say I’ve got 30+ directories in my dropbox but I only want “md”

  find -maxdepth 1 -type d | cut -b3 | grep -v md | xargs python md/dropbox.py exclude add

That will add everything else to the exclusion list AND clean out the local Dropbox path for you. One point to note is that .dropbox-cache/ gets hoovered into the exclusion list, which is redundant and doesn’t appear to cause any harm.

I’ve increasingly been using Dropbox as a low-security transfer protocol ( critical things like ssh keys, password dictionaries, and certificates are handled separately ) and now with the selective sync it’s becoming fairly trivial to keep my horde of virtual servers both in house and running on AWS synced. One thing I’ve been trying out is using the autostart functionality to kick on when I login to a box. I am not sure but if Dropbox could kick on only when I am logged in would be fairly nice.

PyCon 2012 Day 1 itinerary

Graph Processing in Python
Scalability at YouTube
Practical Machine Learning in Python
LUNCH
Code Generation in Python: Dismantling Jinja
The Magic of Metaprogramming
Throwing Together Distributed Services With Gevent
Putting Python in PostgreSQL

And I think that ends day 1 of the conference. Mostly I am hoping to combine what I learn in the Graph processing and Practical ML talks, while having an overview of code generation might be a solution ( or a new problem ) to my current client’s issues. As for metaprogramming, gevent, and postgreSQL; those just seem fun.

Off to PyCon 2012

Still trying to figure out what presentations I want to go to, but I am genuinely excited to be attending another PyCon, this time around in California. Also looking forward to seeing startup row to see what wonderful things are being built with Python this year around.

Update a Github Fork from the Original Repo

I have a few forked repo’s, either because I needed to make some unique and custom change to the project or because I wanted a safe snapshot in time of the project.

Occasionally I’d like to update these forks against their master and the Google has provided a very concise blog post here(http://bradlyfeeley.com/2008/09/03/update-a-github-fork-from-the-original-repo/). Kudo’s to Bradly Feeley.

A Tour de Force with Sencha Touch v2

The team I am working with just passed the 1,000 man hour mark on a flagship Mobile application using Sencha Touch version 2 pr3 and phone gap. Until the business people make a PR statement I can’t say much… so in the interim, things I’ve discovered.

  1. Don’t trust Ext.Loader to get it right – eventually using some cheap tricks I wrote an automatic dependancy handler that writes script tags in the correct order for all needed components. Holding @ 12K lines of JS and the “good” mobile devices are performent
  2. Ext.layout.Vbox will test your sanity – http://www.sencha.com/forum/showthread.php?140185-vbox-gives-components-0-height-hbox-gives-components-0-width Their is a provided solution but I’ve found it’s best to use a Ext.env or Ext.os controlled magic height/width variables instead of relying soley on the rendering logic to get it right
  3. I re-implemented this ExtJS 4 widget in Sencha Touch 2. My original estimate was 2.5 hours but that became 12 hours later to learn the Ext.component system to get it right. At this point I’ve read 60% of Sencha Touch’s source code, line by line, so I guess this was a positive serendipity tour. That said, you’re going to pay dearly to make exotic custom components until some more tools are ported from ExtJS4 to Sencha Touch
  4. Last, Android’s mobile rendering engine is the MSIE of Smart phones.

On a sadder note, I am finally breaking down and purchasing an Apple computing device of some sort as the jail break toolchain I’ve got isn’t exactly professional grade.

Pynarcissus parser post-parse analysis with a visitor like pattern

Someone recently asked me how the more time involved analyzer I wrote works, unfortunately I can’t post it as its targeted for a client project… but I can post a simpler mockup of it combined with the person’s function print logic.

Sure this loses the ability to step back up the chain of tokens, but for most cases like detecting functions or in my case line numbers to intrument, this construct works for me.

Working with & using pynarcissus to parse Javascript in Python

If you’re reading this, you’re either one of the 10-15 user-agents stalking my blog or one of the search engines sent you here because you’d like to analyze Javascript from inside of Python. In my cursory research phase, the best option seems to be pynarcissus.

As of 2012/01/6 there has been no commits or updates to the project which seemed disheartening until I gave it a try. From my tests, pynarcissus DOES WORK.

Below is proof of concept code I wrote to walk through the entire token tree of a ExtJS Sencha touch file which is probably the most extreme JS compatible test I could come up with, mostly because ExtJS code reliably and utterly confuses Komodo IDE and other IDE’s I use on a daily basis.

The codez:


from pynarcissus import parse
from collections import defaultdict

"""
    Syntax analysis done dirty

"""

#Linenums JSCoverage said were correct
targetnums = [28,44,51,53,57,59,60,68,69,70,77,78,79,86,87,88,95]

#Operands/tokens unique to these lines
injectionOPS = {'IF','CALL','VAR','RETURN'}
#operands/tokens that should be avoided
exclusionOPS = {''}

#test file
raw = open("./juggernaut/parser/test/test_original.js").read()

tokens = parse(raw,'test_original.js')

#Master list of linenums to TOKEN types


def findlines(node, pr_linenums = None, capture_all = False):
    """
        Walk over the hills and through the valleys and hope
        we hit every single token in the tree along the way.
    """
    linenums = defaultdict(set) if pr_linenums is None else pr_linenums

    if node.type == 'IF':
        dbgp = 1

    if node.type in injectionOPS or capture_all:
        linenums[node.lineno].add(node.type)

    for sub in node:
        if len(sub) > 0:
            linenums = findlines(sub, linenums, capture_all)

        for attr in ['thenPart', 'elsePart', 'expression', 'body' ]:
            child = getattr(sub, attr, None)
            if child:
                linenums = findlines(child, linenums, capture_all)


        if sub.type in injectionOPS or capture_all:
            linenums[sub.lineno].add(sub.type)

    return linenums

linenums = findlines(tokens, defaultdict(set), False)
injectionTargets = sorted(linenums.keys())

source = raw.split('\n')
from cStringIO import StringIO
buffer = StringIO()
for lineno, line in enumerate(source, 1):
    if lineno in injectionTargets:
        print >> buffer, "$_injectionjs['testfile'][%s] += 1;" % lineno

    print >> buffer, line

buffer.reset()
open('test_file.js','wb').write(buffer.read())

dbgp = 1

Briefly, the above code is a unstructured test to attempt to match what JSCoverage thinks are the correct lines to instrument and for the most part it works well. The most crucial needed to be able to blindly traverse the token node tree is the logic inside the findlines function.


    """
         the product of pynarcissus.parse is a Node object.
         Node's inherit lists, so for some JS structures you will
         iterate over it's children.  Other times, you might not be
         so lucky, hence the inner loop to detect Node elements.
         
         For instance, for Node.type == 'IF' the thenPart and elsePart
         properties will be populated alongside with 'expression'.
         
         Othertimes, specially for function/method nodes, there will be a body
         attribute.
         
        All of these attributes, if not None, are instances of pynarcissus's Node class.
        
    """
    for sub in node:
        if len(sub) > 0:
            linenums = findlines(sub, linenums, capture_all)

        for attr in ['thenPart', 'elsePart', 'expression', 'body' ]:
            child = getattr(sub, attr, None)
            if child:
                linenums = findlines(child, linenums, capture_all)


        if sub.type in injectionOPS or capture_all:
            linenums[sub.lineno].add(sub.type)

As you can see, there’s a linenums variable being passed around like the village bicycle, here is what it looks like.

pprint.pprint(linenums.items())
[(1, set(['SCRIPT'])),
 (28,
  set(['CALL',
       'DOT',
       'IDENTIFIER',
       'LIST',
       'OBJECT_INIT',
       'SEMICOLON',
       'STRING'])),
 (29, set(['IDENTIFIER', 'PROPERTY_INIT', 'STRING'])),
 (30, set(['ARRAY_INIT', 'IDENTIFIER', 'PROPERTY_INIT', 'STRING'])),
 (31, set(['IDENTIFIER', 'OBJECT_INIT'])),
 (32, set(['IDENTIFIER', 'NULL', 'PROPERTY_INIT'])),
 (33, set(['PROPERTY_INIT'])),
 (34, set(['IDENTIFIER', 'OBJECT_INIT'])),
 (35, set(['IDENTIFIER', 'PROPERTY_INIT', 'STRING'])),
 (36, set(['FALSE', 'IDENTIFIER', 'PROPERTY_INIT'])),
 (37, set(['IDENTIFIER', 'PROPERTY_INIT', 'TRUE'])),
 (38, set(['IDENTIFIER', 'PROPERTY_INIT', 'STRING'])),
 (39, set(['IDENTIFIER', 'PROPERTY_INIT', 'TRUE'])),
 (40, set(['IDENTIFIER', 'PROPERTY_INIT', 'STRING'])),
 (41, set(['PROPERTY_INIT'])),
 (43, set(['FUNCTION', 'IDENTIFIER', 'SCRIPT'])),
 (44, set(['BLOCK', 'IF'])),
 (45, set(['CALL', 'DOT', 'IDENTIFIER', 'LIST', 'SEMICOLON', 'STRING'])),
 (46, set(['IF'])),
 (52, set(['PROPERTY_INIT'])),
 (57, set(['FUNCTION', 'IDENTIFIER', 'SCRIPT'])),
 (58, set(['IDENTIFIER', 'VAR'])),
 (60, set(['IDENTIFIER', 'VAR'])),
 (64, set(['CALL', 'DOT', 'IDENTIFIER', 'LIST', 'SEMICOLON', 'THIS'])),
 (66,
  set(['CALL', 'DOT', 'IDENTIFIER', 'LIST', 'SEMICOLON', 'STRING', 'THIS'])),
 (67, set(['RETURN'])),
 (68, set(['PROPERTY_INIT'])),
 (74, set(['FUNCTION', 'IDENTIFIER', 'SCRIPT'])),
 (75, set(['IDENTIFIER', 'VAR'])),
 (76, set(['IDENTIFIER', 'VAR'])),
 (77, set(['RETURN'])),
 (78, set(['PROPERTY_INIT'])),
 (83, set(['FUNCTION', 'IDENTIFIER', 'SCRIPT'])),
 (84, set(['IDENTIFIER', 'VAR'])),
 (85, set(['IDENTIFIER', 'VAR'])),
 (86, set(['CALL', 'DOT', 'IDENTIFIER', 'LIST', 'SEMICOLON'])),
 (87, set(['PROPERTY_INIT'])),
 (92, set(['FUNCTION', 'IDENTIFIER', 'SCRIPT'])),
 (93, set(['IDENTIFIER', 'VAR'])),
 (94, set(['IDENTIFIER', 'VAR'])),
 (95, set(['CALL', 'DOT', 'IDENTIFIER', 'LIST', 'SEMICOLON'])),
 (96, set(['PROPERTY_INIT'])),
 (101, set(['FUNCTION', 'IDENTIFIER', 'SCRIPT'])),
 (102, set(['CALL', 'DOT', 'IDENTIFIER', 'LIST', 'SEMICOLON', 'STRING'])),
 (103, set(['PROPERTY_INIT']))]

This is enough information for me to be able to safely determine injection points for an instrumentation line. I am still working through some problems with this whole mess, specifically line 46 looks like:

       if(1 == 1){
$_injectionjs['testfile'][45] += 1;
            console.log('');
$_injectionjs['testfile'][46] += 1;
        }else if(2 == 3){
            console.log("boop");
        }else{
            console.log("bleep");
        }

after being instrumented. As you can see, I am skipping over the else if chain… which leads me to believe I am missing YET ANOTHER dynamically assigned Node property.