Tag Archives: utilities

Service accounts with Ruby and the google api client

  1. Goto https://code.google.com/apis/console
  2. Create a new project
  3. Under services, select Google Analytics
  4. Goto API Access of a new project
  5. generate a new oAuth credential, selecting “service account”.
  6. Download your p12 cert file

 

In Google analytics, use the service account email address and assign it as a new administrator to the desired GA profile.

On ALL servers destined to use the service account to GA, install and syncronized with ntp.  If your even 500ms off, you’re going to have a bad time with “Grant_invalid” messages.

In a testbed project, make 2 files: Gemfile & testbed1.rb ( you’re going to create a lot of these testbed files on your own ).

Gemfile

source "http://rubygems.org"
gem 'google-api-client'
gem 'pry'

testbed1.rb

require 'pry'
require 'rubygems'
require 'google/api_client'

keyFile = "SOME_KEY_GOES_HERE-privatekey.p12"

cID = "YOUR_SOOPA_SECRET_SA"

scope = 'https://www.googleapis.com/auth/analytics.readonly'
email = 'YOUR_EMAIL_GOES_HERE@developer.gserviceaccount.com'


#Normally it would be bad to put the passphrase here, but aftet talking to several dozen devs, everyone's pass phrase is not a secret
key = Google::APIClient::PKCS12.load_key(keyFile, "notasecret")

asserter = Google::APIClient::JWTAsserter.new(
   email,
   scope,
   key)

puts asserter.authorize()

binding.pry

If all goes well, the JWTAsserter instance will get a valid token and no exceptions/errors will be thrown. If you’ve followed all of the steps listed earlier and things are still breaking, troubleshooting paths are: Verify your machine time is correct, verify your service account email address is bound to google analytics, and lastly you might need to wait until Google platform catches up with the changes. For myself, it took 9 hours until my account finally authenticated through. Google has potentially a million or more servers organized into cells, the fact that they seem to cooperate well doesn’t mean they’re perfectly in sync at all times.

Otherwise if all else fails, I recommend stalking this guy Nick https://groups.google.com/d/msg/google-analytics-data-export-api/maa_fyjD2cM/sT8tDDh0wNsJ – It seems like he’s a Google employee on the service account dev/implementation team and generally stuff gets fixed if he says it’s getting fixed.

As I run into any more troubles, I will update and add more notes.

Plugin’s for python

As the #1 google result for “python plugin”, the linked to blog post is extremely valuable for jump starting research into implementing your own plugin system ( like me ) or finding one that is viable for implementation in your project ( possibly like me ).

These resources highlight one reason why there is not a standard Python plug-in framework: there are a variety of different capabilities that a user may want, and the complexity of the framework generally increases as these new capabilities are added

http://wehart.blogspot.com/2009/01/python-plugin-frameworks.html

PyProxy – Aka the development Helper proxy

Coming out of nothing and into supah doopa Alpha is finally a working proof of concept of my python web proxy. I don’t really want to talk about the asinine alternatives I’ve tried until I finally said “fuck it, time to go completely twisted!” Low and behold the actual proxy part is 4 lines of code, which is then expanded to maybe 20-30 to allow for overloading some lower level classes.

Originally a public announcement for this project would have been in August at the earliest, give me time to clean things up and go from proof of concept to working concept but apparently a lot of other people have similar thoughts and I figured it’s better to collaborate then compete.

So some quick notes:
The ultimate goal for PyProxy ( or whatever it ends up being named ) is to sit between a developer and a development server. The first and immediate idea for this was to automagically parse out Python mechanize scripts to replicate the traffic. These mechanize scripts could then be collected into a suite, marking other scripts as requirements ( example login process ). That alone would make it pretty easy to create full system under test unit-tests. The next idea was to add in regex or pattern based hooks that could allow a developer to dial in to a specific domain, or even a specific set of webpages.

After that, the idea was to just continually tack on support plugins and scripts, maybe tell PyProxy the name of the target application’s database, and if it’s MySQL, switch on the general log. This could allow for combining both mechanize scripts AND a SQLObject or SQLAlchemy powered unit-test suite to assert that the correct data was changed.

The final future idea was to make a Firefox/Chrome extension that would allow a developer to control some parts of the proxy from their browser and also see additional information. For Python and PHP web apps, imagine have a finalization plugin that appended a response header listing all File’s used to perform a request…. then imagine having a “click to edit” button that, if the dev. instance is workstation local, would have your favorite IDE open the specified file for editing.

All in all, I think these are really subtle idea’s that if combined together, would cut down some mudane parts of developing a web app.

GitHub repo (https://github.com/devdave/PyProxy) here

Python helping PHP development

Just a quick little utility script I wrote to make watching PHP error logs a tad easier:

Usage is

#tail -F /var/log/php/php_error.log | ~/bin/parsePHPErrors.py

Thanks to this Stackoverflow question, it even colorizes!

#!/usr/bin/python
import sys
import os
import re

input = os.fdopen(sys.stdin.fileno(),'r',100)
regex = re.compile("(.*?PHP (Warning|Error|Notice|):.*)")
line = input.readline()

class bcolors:
    HEADER = '\033[95m'
    OKBLUE = '\033[94m'
    OKGREEN = '\033[92m'    
    WARNING = '\033[93m'
    FAIL = '\033[91m'
    ENDC = '\033[0m'


myColor = bcolors.OKGREEN

while line:
    
    if regex.match(line) or line.find("TODO") > -1:
        
        isCorrect = True
        
        if line.find("Fatal error") > -1:
            myColor = bcolors.FAIL
            pass
        elif line.find("Error")  > -1:
            myColor = bcolors.FAIL
            #todo get console color code
            pass
        elif line.find("Warning")  > -1:
            myColor = bcolors.WARNING
            pass
        elif line.find("Notice")  > -1:
            myColor = bcolors.OKBLUE
            pass
        elif line.find("TODO") > -1:
            myColor = bcolors.OKBLUE
            pass
        
        if isCorrect:            
            print "#"*25, "@"*25, "#"*25
            print
    else:
        pass
        #myColor = bcolors.OKGREEN
        
        
    print myColor, line.strip(), bcolors.ENDC
    line = input.readline()

Output looks like:

######################### @@@@@@@@@@@@@@@@@@@@@@@@@ #########################

[28-Mar-2011 11:14:10] PHP Notice:  Undefined variable: sql in

basically breaking up individual error messages into visually easy to spot blocks

Ping, a canvas toolset, is nearing Alpha Alpha stage

So I’ve got a working Quadtree implementation that is relatively performent on Google Chrome 10

Demo here

GitHub repo here

So far bounding box detection isn’t but working on that, also documentation, and unit-tests are non-existent but I’m working on that now.

Once I hash those out, then this should be an useful collision detection/proximity check library for canvas tag games.

Deferreds for JQuery 1.5

Anyone familiar with async programming like Node.JS or Python’s twistd library will be familiar with deferreds. Its usually a simple nuisance to re-implement when needed, so it’s nice to hear that’s not an issue anymore with JQuery’s $.when & $.then methods

Really useful guide here

What does this mean?

Say you’ve got a compound action you want to complete. For example a user clicks on a leaf in a tree like directory
menu. Now you want to update both the menu to add any possible sub nodes of that leaf PLUS update a content panel and once its finished, maybe update the hash tag to the current page’s url.

   
  $.when( updateMenu(), updateContent() )
     .success( function(){
           console.log("Both menu & content panel have been updated!");
      })
     .fail( function(){
           console.log("Oh noes, we didn't finish... try again?");
      });

That’s pretty much the gist of deferreds in a nutshell. Furthermore it appears that the author of this fantastic addition to jQuery really grokked the concept of deferreds because you can chain one deferred result to multiple child deferred’s allowing for cascading events to fire in response to one event.

Return to canvas tag

Currently in between clients and waiting for the silent alpha of my delicious clone to wrap up….so its back to experiments with the canvas tag.

That means back to my unframework canvas library Ping. Some of the code still makes sense, but other bits are… well special. Currently there is an .ex object thats appended to the CanvasRenderingContext2D internal class and it works, but what I’m thinking of is re-working the code so that they’re added post-instantiation like a factory.


canvasMaker = function(elemId, options){
    var element = document.getElementById(elemId);
    var context = element.getContext("2d");
    //Here is where all the extension methods would be dynamically assigned to the 2d canvas instance
    wrappedContext = wrapContext(context);
    //Maybe add a singleton check around the bulk of this that keys on the elemId
    return wrappedContext;
}

The problem I’m trying to work around specifically is the need should ever arise to have n+1 managed tags on the same document. Currently there is so many hardwired references that it would be impossible to have the two cooperate without collisions and undesirable behavior.

Python pydoc module

Python documentation here

Example

$python -m pydoc pydoc

Talk about eating your own dogfood! This is exactly like using the help() function in the python command line interpreter… except accessible from your shell prompt…. but wait!

It’ gets better! Not only does it make julian fries ( may not for any implementation ) but it’s got a few versatile little secrets

$ python -m pydoc
pydoc - the Python documentation tool

pydoc.py  ...
    Show text documentation on something.   may be the name of a
    Python keyword, topic, function, module, or package, or a dotted
    reference to a class or function within a module or module in a
    package.  If  contains a '/', it is used as the path to a
    Python source file to document. If name is 'keywords', 'topics',
    or 'modules', a listing of these things is displayed.

pydoc.py -k 
    Search for a keyword in the synopsis lines of all available modules.

pydoc.py -p 
    Start an HTTP server on the given port on the local machine.

pydoc.py -g
    Pop up a graphical interface for finding and serving documentation.

pydoc.py -w  ...
    Write out the HTML documentation for a module to a file in the current
    directory.  If  contains a '/', it is treated as a filename; if
    it names a directory, documentation is written for all the contents.

Now the graphical interface isn’t anything to write home about, but the -p option provides a no thrills web interface to
almost everything accessible to your python interpreter. This can make it slightly easier to troll through foreign modules
looking for undocumented sub modules and classes… or having an accessible reference doc for properly managed modules

Python urllib

Python documentation here

Unfortunately there is little or no documentation on the command line properties of urllib but it does recognize everything that urllib can handle. So
python -m urllib http://website.com will grab the specified url and print to std out

Note FTP works as well but you need to follow the pattern ftp://user:password@website.com if authentication is required