for i in range(0, len(sys.path)):    
    if sys.path[i].find("~") > -1:        
        sys.path[i] = path.expanduser(sys.path[i])
        break
else:    
    sys.path.insert(1,path.expanduser("~/lib/python"))

More duct tape code to allow me to rely on ~/lib/python to store common code (ex. apache log parsing & analysis ) when PYTHONPATH might not be set or
set incorrectly.

Basically if ~ is found in sys.path, the break statement skips the else. Alternatively, if not found then else is executed.
Python compound statements: For loops

Meanwhile I used Doxygen a lot, its great for finding out how spread out calls are to individual libraries, has great call graphs, and if setup correctly can draw out the entire hierarchy of even the most complicated of projects.

What I’d like to do is create something that uses xhprof with documentation logic similar to doxygen, to make dynamically generated documentation that not only shows you how a class is defined, but how it’s defined, what depends on it, and what does it depend on.

UI:
I was thinking of something that was intelligent in the ways of MVC frameworks, recording a list of all urls called into a framework and then displaying them as a list broken down by URL components.

A simple example might be a MVC project with one controller called user
User has a standard CRUDE outline so the url’s might be
/user/create
/user/list
/user/edit
/user/save
/user/delete

So the first screen would be:

Entry points:
User – 5 sub points

The sub points part would be a anchor leading to a digest page listing # of recorded profiles to /user/ then maybe a digest of slowest call with the subcompontent ( say save ) to fastest ( delete ), highest / lowest memory consumption, a link to files common to this URI, and then links to progress further down the URL chain.

Clicking on list would lead to /usr/list – again showing the above, but now specific to this URL. At this point there shouldn’t be anymore child components in the URL ( GET arguments are automatically stripped OFF ) so an additional feature would be to click a link to show a visual graph similar to what stock xhprof UI provides.

The fun part would be when you clicked on one of the listed files. Immediately this would bring up a page similar to a doxygen product, but it would have some additional information:
# of dependant URLS that executed this file
# of classes, functions, and methods used in this file by this url path ( /user/list )
fastest and slowest execution speeds plus memory consumption percentage for /user/list.

Clicking on a class, method, or function would cause it to again drill down and show stats JUST for the specified scope.

Considering its taken me 3 months to get PyProxy to stable useful Alpha, this project is probably going to take me a year or more… but it should be fun.

<script>
    (function(){
        var original = document.write;
        document.write = function(arg1){
                        console.group("document.write");
                        console.log("doc.write: " + arg1);
                        console.groupEnd();
                        original.apply(document, arguments);
        };
    }());

    (function(){
       var truImage = Image;
       window.Image = function(width, height){
        try{

            this.root = new truImage(width, height);
            this.__defineSetter__("src", function(val){
                console.log("New Image @ " + val)
                this.root.src = val;

                });
        }catch(Err){
            console.log("New Image Err");
        }
       }
    }())
</script>

It’s abomination code yes, but it is also extremely useful for further illuminating what exactly is going on in the time prior to document.onLoad plus exposes image beacons generated using Javascript Image objects… something that normally doesn’t show up any where in the DOM, hence I don’t believe FireBug or Chrome inspector panels can report it.

Normally to get something like this into a website would require adding it into development, which then opens the risk of it slipping into production.

The functional beta of PyProxy should be released this weekend if I don’t go on a bender.

Works great when your webstack uses make to automagically compile multiple files then minify the whole mess down… but your constantly making tweaks and fixes.

update: No client needs to know about another client, each user belongs exclusively to one client

Vertical scaling:
Client data is stored in a normalized SQL based database. user credentials for all users is stored in client_users table.

Benefits
Shortest path of resistance for development
relatively easy to maintain integrity with
Easy to backup
Problem:
Because all client credentials are stored in this one table along with related data, to maintain or increase performance would require beefing up your database tier with more resources or investing in more slaves to this one master.

Horizontal scaling:
Each client exists on a prefixed table schema. client_users becomes client01_users

Benefits
Someone with intermediate level skills in DB administration could write a simple script to copy client#_* tables to a new DB server in about 5 minutes ( then another hour to sanity check/test/verify). In this way you can push your low traffic clients onto a overbooked server and profit from the infrastructure savings while charging your higher traffic clients for requiring dedicated hardware.

Problems
Maintenance/development time can extend into oblivion and paralyse development entirely if shared nothing techniques don’t include automation and schema change management systems.
A simple task like adding/dropping a column will take much longer to peform as you have to do it on multiple tables/machines in lockstep to software changes.
Backup’s get pretty interesting sometimes
Summary
If in the beginning I can see an opportunity for shared nothing, I will fight tooth and nail to get that implemented. For new client’s with scaling issues, after contracted my initial proposal will include refactoring to incorporate sharding or shared nothing principals. In my mind, the extra complexity can be managed if approached/handled correctly.

Attempt 1 went like:

 
//Simulated row
<td>
  <a href="#" onclick="$('#my_SubData<?= $currentRecordId ?>').slideToggle(); return false">Click me to show stuff!</a><br>
  <div id="my_SubData<?= $currentRecordId; ?>" style="display: none "> Blah blah blah....</div>
</td>

I hate messy code, but unfortunately this is PHP so there’s only so much one borderline pyschotic developer can accomplish. Or is there?

 
//prior to my table
            $('a.actionable').live('click',function(){                
                $($(this).attr('href')).slideToggle();
                return false;
            });
 
//Now 
<td>
  <a href="#my_SubData<?= $currentRecordId ?>')" class="actionable">Click me to show stuff!</a><br>
  <div id="my_SubData<?= $currentRecordId; ?>" style="display: none "> Blah blah blah....</div>
</td>

It’s almost elegant if you ignore the PHP inject. JQuery’s .live handler automatically routes all unhandled click events to the closest “.actionable” classed element, then inside the live event handler, you grab the anchor’s href value to get the element Id of what you want to edit.

class ArrayBuilder {
    public function __set($name, $value){
        return $this->$name = $value;
    }
 
    public function __call($name, $value){
        if(count($value) == 1){
            $this->$name = $value[0];
        }else{
            $this->$name = $value;
        }
        return $this;
    }
 
    public function toArray(){
        return get_object_vars($this);
    }
 
    public static function FACTORY(){
        return new ArrayBuilder();
    }
 
}

Usage:

  $x = ArrayBuilder::FACTORY()->hello("World")->digits(1,2,3,4,5)->foo("BaR?")->toArray();
  var_dump($x);
   array(3) {
  ["hello"]=>
  string(5) "World"
  ["digits"]=>
  array(5) {
    [0]=>
    int(1)
    [1]=>
    int(2)
    [2]=>
    int(3)
    [3]=>
    int(4)
    [4]=>
    int(5)
  }
  ["foo"]=>
  string(4) "BaR?"
}

Works great for factory scenarios and confusing the $*&! out of the unwary.

Now a few months down the road, I’ve been playing with a much simpler approach using Lighttpd. It’s not perfect for a couple reasons I will point out…but good enough for me, for now.

# default document-root
  var.baseDir = var.CWD
  server.document-root = var.baseDir + "/"
   index-file.names   = ( "index.php", "index.html", "index.htm", "default.htm" )

  # TCP port
  server.port = 8000

  # selecting modules
  server.modules = ("mod_rewrite", "mod_accesslog", "mod_fastcgi")

  #logs
  accesslog.filename = var.baseDir + "/access.log"
  server.errorlog = var.baseDir + "/error.log"

    server.dir-listing = "enable"
    mimetype.use-xattr = "enable"
    mimetype.assign = (
  ".html" => "text/html",
  ".txt" => "text/plain",
  ".jpg" => "image/jpeg",
  ".png" => "image/png",
  ".js" => "text/javascript"
  )

  static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
  fastcgi.server = (
  ".php" =>
  (( "host" => "127.0.0.1",
     "port" => 9000,
      "bin-path" => "/usr/bin/php5-cgi"
  )))

What seems like a lifetime ago, I was the database administrator for a very heavily trafficked web SaaS site. One thing I remembered vividly was that for a poorly configured development instance of MySQL, random logging and utility files would crop up in strange places. Since I was already spending a good portion of my life at that time with MySQL, I spent some extra time to figure out that these files were showing up because my.cnf was configured with relative file paths. Taking that memory alongside my habit of relaying on document relative paths in Apache, I wondered if I could do the same for lighttpd.

Turns out you can actually. So using the above template, I can host arbitrary locations in my /home directory by simply running “lighttpd -f lighttpd.conf -D” which translates out to: Lighttp’y run using this specific configuration file ( -F lighttpd.conf ) in the current PWD and stay resident to the console -D so I can shut you down later”

Problems

Unless you change the local config file to something else, you can’t run two instances of lighttp’y concurrently. First reason is that the second instance would try to listen to the same port for http traffic. This is pretty easy to fix, just change sever.port to some other port #. The second problem is that fastcgi.server also has to have it’s own port to listen for fastcgi traffic.

Also, it might not be to obvious, but if you ran the above config on a non-firewalled, internet homed box, someone could find your development environment and possibly cause havoc.

Solutions

The first problem is a doozy that won’t have an easy fix… and its debatable whether investing time to come up with a fix.

Second problem is stupid easy by adding:

   server.bind = "127.0.0.1"

to tell your instance to home on the loopback address.

I’ve been using SVN for several years now, since late 2005 or early 2006, and its done me well since then. But a new darling has entered my life and its name is Git. I like git for some very specific reasons: It’s stupid easy to work with and as good or better then SVN for reliability. Also it helps that my preferred IDE, Komodo, recognizes and works with Git as well.
That said, I use google code for hosting my public projects and it only supports SVN and HG. So one night I read up on git and noticed that it had a plugin/support for bridging to a SVN managed repo. So began my journey.

!Note – I am using a client side credentials manager, if access to your repo requires authentication credentials you will need to supply them where appropriate.

Initialization

Bridging a new Git Repo to a new directory in SVN
Step 1:

$svn mkdir https://devdave.googlecode.com/svn/trunk/javascript/canvastag/cnvLib

Committed revision 26.

Now I’ve got a fresh/empty svn directory

   $git svn init https://devdave.googlecode.com/svn/trunk/javascript/canvastag/cnvLib
   Initialized empty Git repository in ~/dev/javascript/cnvLib/.git/

Now I’ve got a local git repo that has most of the information I need for the bridge, to verify everything is hunky dory so far:

    $git svn fetch
    #r26 = a47a908ea96bec2d737f6646c47418778ce6ad32 (git-svn)

Final steps, the graft

Some tutorials out there say you should run git svn rebase or such, but doing so has gone poorly for me with results like

$ git svn rebase
      fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
      Use '--' to separate paths from revisions
      log --no-color --first-parent --pretty=medium HEAD: command returned error: 128

I believe the problem relates to lining up your local Git repository with the remote SVN repository. Unfortunately I was somewhat stuck in trying to figure out how the hell to fix this problem for a long time. Fortunately Google search is the modern day version of a genie, it can grant you the information you seek, but first you have to know what to ask. Which led me to finding this blog post.

I am not doing things “exactly” like the aforementioned blog, but close enough that I can get a similar result.

$ git show-ref
2a41e85e0dc264cfdb6c04b59547a3093c02f64f refs/heads/master
a47a908ea96bec2d737f6646c47418778ce6ad32 refs/remotes/git-svn
$ git log --pretty=oneline master
2a41e85e0dc264cfdb6c04b59547a3093c02f64f Initial import, copied from ping project
$ echo "2a41e85e0dc264cfdb6c04b59547a3093c02f64f a47a908ea96bec2d737f6646c47418778ce6ad32" >> .git/info/grafts

Since I am doing things a little odd/differently, there is no “trunk” branch available. But the idea is close enough to duplicate. I’ve already made some add/commit’s to my local Git repo but there isn’t a logical way to tell git/git-svn that this local commit also belongs to my remote SVN remote. A more human friendly version of git show-ref is
available via:

$ git branch -a
* master
  remotes/git-svn

So the concatenation of the two internal reference’s into .git/info/graft helps git-svn understand that the two separate repositories belong to each other. Sometimes it’s nice when a configuration structure does what it’s called.

That said, on to the next obvious step, actually pushing changes from a local Git repo to SVN.

$ git svn dcommit
Committing to https://devdave.googlecode.com/svn/trunk/javascript/canvastag/cnvLib ...
        A       app.canvas.js
        A       app.input.js
        A       app.main.js
        A       app.mainloop.js
        A       app.shapes.js
Committed r27
        A       app.canvas.js
        A       app.mainloop.js
        A       app.input.js
        A       app.shapes.js
        A       app.main.js
r27 = ae1cd1055deffe4209ab89d1316c97336c731c39 (git-svn)
No changes between current HEAD and refs/remotes/git-svn
Resetting to the latest refs/remotes/git-svn

What’s going on here?

As I am still learning the finer points of Git, it was important to me to understand what exactly .git/info/grafts does. So turning to my favorite resource for arbitrary questions… I found an answer on Stack overflow: What are git info grafts for? And a more official answer in the Git Wiki http://git.wiki.kernel.org/index.php/GraftPoint