Consultancy, the downsides

People outside of my vocation always marvel at my “flexible” life-style and for a much fewer number the amount of money I make in an hour or a week when employed to a client or project. These things are definitely nice, another nice perk is that I’ve been exposed to a much larger breath of technologies, idea’s, people, and I am probably some sort of interview guru as I can usually reverse engineer what the goal of the interview is. Then there is the fact of life that if I am being called into a project, it’s not because I am some sort of genius code monkey but because the problem is pretty terrible. I might also be thrown under an undeserving bus, months down the road due to things completely out of my control. And lastly there is no future in being a consultant.

With interviews, things usually go poorly when prospective client’s don’t know what the hell their goals are for an interview. Or worse if they just have a bucket list they’re trying to check off. In the first case I’ve politely ended interviews ( as the interviewee ) when presented with “What’s the difference between a inner & outer join” ( for that one I usually draw two circles and shade in relevant sections to illustrate every variation ( left, right, inner, outer) and then told I am wrong. or asked to implement something really crazy like a b-tree on a whiteboard. In the first case this is a red flag that the company is a semi-hostile environment and the interviewer is probably trying to protect their incompetence and the second it begs the question “Would I need to implement my own b-tree logic, what other wheel is being reinvented here?” I’ve seen a lot of the first “No you’re wrongs” in my career so far and they’re completely opposite of the second where the interviewer has good intentions but fails to find out if the interviewee has critical thinking and problem solving skills.
The other problem of interviews is the bucket list interviews, especially depending on who is doing it. Usually its headhunter’s going down a line and are completely ignorant of how insane they sound when they ask questions like “Do you have 10+ years of experience with Ruby on Rails?” That one took me off guard so I asked “Which Rails? Version 2 or 3 and is that with 3’s asset generation system or what?” to which the headhunter had no answer. Then that also begs the question of what is the goal? Technically I have 6-7 years of experience with Rails as I was first introduced to it in 2006 and my last working experience was the start of 2013… but I haven’t used Rails 100% of the time in between. If had to guess I might have 4000 hours of Rails experience which is 6000 hours short of being a master of that ever changing landscape.

Being a consultant doesn’t mean you’re always going to find world challenging work either. For an unnamed client we will call Acme, I was called in because an adjacent department knew me and knew that I was their go to guy for making something that actually worked. A lot of things had gone wrong for Acme, their project manager turned out to be a liar that sat on his feet while a “lead” contractor watched netflix and pecked out some code occasionally. I’ve seen a lot of convoluted solutions in my career and each one is special. For Acme’s case the lead was very good at aesthetics but had never read a book on design patterns or done any business software before. If pressed, I’d say the system was some sort of reversed MVP pattern but with stored procedure style application code ( implemented in the application tier ). Further the system could only be talked to via XHR via a global anchor tag handler that took the anchor tag’s href, wrapped it in XML, and sent that to the server. The server then built the view up, calling injects along the way until it reached the end and sent that back as a two element httpbody document, the first half was javascript to be executed and the second was html. It would eval the javascript xml element and then insert/overwrite the contents of a browsers main view-able content. There was a lot of problems with this system but fortunately the “lead” had been sacked alongside the original PM just before I arrived. Another consultant and I sat down over dinner and after flipping a coin as to whether we were running away, we delegated out goals, the other guy was going to focus on adding new “features” to the system while I worked on figuring out how to refactor the system into a MVC pattern… or at least bring in some sort of simple model system to get rid of the functional procedures crap. We pulled that off. Meanwhile a third “Real” project flow consultant pulled some serious smoke & glitter magic tricks and claimed whatever credit for several 220 hour long months of “fixing” the application. He did all this after we threatened to gang up on him if he got in the way of the refactoring process.

Also even if I do everything right… things can still go badly down the road. For the prior dept at Acme, I built a simple little MVC application with a data bridge from a proprietary database the name of which I can’t remember. It worked and handed it off to someone that knew how it worked, how to maintain it, and life was good. A year and change later the maintainer got fired with prejudice for insulting some executive or along those lines. I say with prejudice because he didn’t do a hand off for the application and all of the documentation for the application disappeared alongside any backups. The replacement then went ahead and corrupted the only intact copy of the application’s database, breaking the application. They didn’t call me up until a week before it was needed again and then I sat through a fairly nasty conference call where people were blaming me for “There’s no documentation, no backups, and it doesn’t work.” In a fairly neutral voice I explained my rate had just tripled and oh by the way here’s a signed receipt by the person vilifying me showing him signing off that “A) Product meets expectations B) Documentation provided in digital and printed format” with a “will notify” consultant of preexisting clause so that I could fix any bugs for free. Even if they had agreed to 3x my normal rate, I am pretty sure pointing out that the relevant work manager was slandering me to cover his incompetence pretty much ended any chance of that dept being a client ever again. I’ve worked in hostile environments before ( people screaming at each other, on the job substance abuse, or micro-managers that are more concerned about covering their ass then actually doing something ) and it’s really not worth it to be there. Maybe 30-40% of all contract/consultant opportunities I see are like this and it never really ends well because it ultimately comes down that the man or woman that could actually fix the problem is just some well meaning person who’s afraid of conflict ( so what was a minor leak in a pipe becomes an insane tempest flood ).

Lastly, being a consultant/code monkey for hire has no future. It will take you down the road a little bit more, cover your bills for a time, and it’s actually got some good experiences but generally a consultant is there to solve a problem while a contractor is there to give the company an emergency release valve ( oh no, we’re running out of money FIRE the contractors!). If a problem is never solved then you’re a terrible consultant and if you’re a contractor for several years in a company…well why buy the cow when you already got the milk? Sure consultants can team up into a consultancy to try and make more money, but that doesn’t scale and becomes unbelievably risky as you grow larger. Furthermore it becomes some sort of pyramid scheme where the company stops hiring experienced engineers & developers and begins hiring student intern/junior engineers but selling them as senior engineers. If done right, the client might not even realize that they’re overpaying by 70% but there will be hell to pay if an auditor shows up one day to find out how things looked below the covers.

Software consultancy/freelancing and dating

For almost ALL consultancies, they start at the bottom of the food chain taking small mom & pop shop deals and slowly work their way upward ( or out of the business ) by gaining relevant skills and reputation. Unfortunately this progression somewhat promotes some bad behaviors and a fatalistic outlook for clients.

A small company can’t afford a NASA space program setup ( unit-test coverage w/70% or greater, Jeeves, scrum/agile when it’s just one developer and one client ) so the task becomes “Get it done on time/budget” and hope the coder’s discipline is strong enough not to deliver a ball of mud. Talking about time/money, these client’s might cover 10, 20, or at max 40 man hours and that’s not going to be enough to cover someone’s basic fiscal needs.

It take’s time to develop and manage reputation, so you might do hundreds of these small client’s for years and that will eventually skew how you approach clients and god help you when a relatively meaningful client comes in. The first 40 or even 160 hour work cycle comes to an end and the client is happy, which is sadly great. Great because it means money and stability, sad because you didn’t have time to do unit-tests, the architecture is hardcoded, and now you’re gonna eat all that technical debt you ignored to “get it done”. It become’s a question of whether you can learn fast enough how to function in a meaningful medium to long term relationship.

At this point I’m pointing the finger at the code monkey consultant as the person to blame, but that’s not quite reality either.

On the other side are client’s that don’t want to commit to a long term relationship under any circumstance. Maybe their a media company or a Boiler room consultancy that doesn’t actually have any employees beyond it’s sales & marketing staff. For these two client types, from my experience they either pay terribly or pretty decently. The latter usually stop paying decently the first time they hire a naive smart kid who’s happy with 1/5th pay. Both company types will setup some sort of long term open relationship with smaller firms but it’s understood that if they can find better and or cheaper, you’re gone. Honestly I don’t mind them too much as atleast their honest about their intentions. Still you can’t build a life on these firms as who knows when they’re going to kick you to the curb or never call back?

Also, I’ve run into a lifetime supply of extremely dishonest employer’s but the real dishonesty lies in themselves and their blind faith in success. “I know we contracted you for 6 months, but we ran out of money and can’t pay your last invoice” has happened a few times to me and people I know. For me I usually ask them if I can take something of equivalent value “How much do you think this desk lamp is worth?” and then walk away.

How this all correlates to dating explicitly is that sometimes your first real relationship seems like “It’s the one” until it’s not. Other time’s the one’s you think are right for you turn out not to be “…Google’s awesome but I feel utterly useless and alone…” is something a friend voiced to me on his 1 year anniversary at the super star company. Then there is “Yeah they make me work long/strange hours but the pay is good” sounds like “Yeah, Baby might hurt me occasionally but I know they love me.” And then there’s always the opposite “Man that was such a great company, I don’t know why I didn’t take it serious” is pretty much like Mike + Mechanics “All I need is a Miracle”.

Also, if anyone’s still thinking “I think you’re stretching, strange grammatically challenged wordpress guy” think about job/proposal interviews. You might not flat out lie and say “I invented C when I was 6, that Ritche guy just copied from me.” and how many “Fantastic opportunities” can there be in the entire software industry? I’ve lost count of how many “We have an amazing idea, just sign this NDA” turns out to be a Facebook/Twitter/Whatever is shiny this year clone. We can’t all be super star’s and there is only so many green fields to be developed on before the VC money runs out and it’s murder-mystery software maintenance time. It’s just a fact that both side’s in the average blind date want you to see them at their best and deliberately downplay their downsides, “This sweater hide’s my man boob’s perfectly” or “This dress perfectly cover’s my love handles.”.

For me, I had success in dating by writing my own rules. One of the first few things I said to my now wife was that I had man boob’s so I should be qualified to have some weigh in on female conversational topics ( except I don’t wear a bra, so I still got excluded). Sadly in business, my experiments in being brutally honest/real in project proposals/interviews hasn’t paid out as well… it might also have played against me that the word “eccentric” has been liberally used by others to describe me.

Ubuntu ruby-rvm package is toxic

If you do

  apt-get install ruby-rvm

On any Ubuntu distro, you are going to have a really bad time.

Proper directions are on Stack overflow, where the RVM repository owner mentions that he wishes the Ubuntu ruby-rvm package would be dropped.

SO link http://stackoverflow.com/questions/9056008/installed-ruby-1-9-3-with-rvm-but-command-line-doesnt-show-ruby-v/9056395#9056395

Orthello – Atlas Data doesn’t populate

Slowly went batshit trying to figure this out.

There’s a fairly nice walk through on making 2d games with Unity3D but I ran into a weird problem. While trying to setup a SpriteAtlas, I could set the fields for Texture and Atlast Data, but nothing happened ( it’s supposed to populate the contents of the sprite atlas… but doesn’t).

The images below show the initial import of an Orthello SpriteAtlas, after dragging the Texture and atlas data to their appropriate spots, AND last bit occurred after I typed in an arbitrary number into the “size” field. After that the widgets internal logic kicked into gear and populated the atlas as desired.

Also, if reading just ain’t your thing OR you need a sanity check on getting started with Unity3D, this great video tutorial of making a 2D game was really helpful.

Google Discovery service

I am dumbfounded I haven’t heard of this service before, it’s pretty clever. Anyone familiar with WSDL or Soap is going to instantly grok this.

For example, here’s the Google Analytics API discovery document https://www.googleapis.com/discovery/v1/apis/analytics/v3/rest

With this data set, it’s relatively trivial to auto-generate an interface / command pattern instance to make the user feel like they’re actually using the API directly.  Furthermore the discovery document provides a fairly good reference breakdown of required arguments, argument types, and basic validation type hints to developers.

A trivial example, here’s how to grab the first profile for the currently authenticated user in  analytics:

analytics = client.discovered_api("analytics",'v3')

#Grab the first profile
result = client.execute(analytics.management.accounts.list)
print result.data.items[0]

Until I called discover on analytics, my client had no idea how to deal with the Analytics API, but now it not only knows how to work with the remote service but can provide basic validations, some advanced error handling ( bad/wrong input, connectivity issues) and in an interactive console in Ruby or Python I can explorer the api as needed.

Follow up on Google API with service accounts and ruby

refer to the previous post on this subject

Using analytics.data.qa.get doesn’t work as desired, so instead you need to do something like:

visitCount = client.execute(analytics.to_h["analytics.data.ga.get"], {
                              :ids=>"ga:" + YOUR_PROFILE_NUMBER,
                              "start-date" => "2012-06-12",
                              "end-date" => "2012-07-01",
                              :metrics => "ga:visits"}
                            )

Note the part of using to_h[‘analytics.data.ga.get”] to get the right signature to pass onto the APIClient execute method. That will work and works well. To make it easier to formulate queries against GA, I’d recommend using Google’s query explorer tool here http://ga-dev-tools.appspot.com/explorer/

Service accounts with Ruby and the google api client

  1. Goto https://code.google.com/apis/console
  2. Create a new project
  3. Under services, select Google Analytics
  4. Goto API Access of a new project
  5. generate a new oAuth credential, selecting “service account”.
  6. Download your p12 cert file

 

In Google analytics, use the service account email address and assign it as a new administrator to the desired GA profile.

On ALL servers destined to use the service account to GA, install and syncronized with ntp.  If your even 500ms off, you’re going to have a bad time with “Grant_invalid” messages.

In a testbed project, make 2 files: Gemfile & testbed1.rb ( you’re going to create a lot of these testbed files on your own ).

Gemfile

source "http://rubygems.org"
gem 'google-api-client'
gem 'pry'

testbed1.rb

require 'pry'
require 'rubygems'
require 'google/api_client'

keyFile = "SOME_KEY_GOES_HERE-privatekey.p12"

cID = "YOUR_SOOPA_SECRET_SA"

scope = 'https://www.googleapis.com/auth/analytics.readonly'
email = 'YOUR_EMAIL_GOES_HERE@developer.gserviceaccount.com'


#Normally it would be bad to put the passphrase here, but aftet talking to several dozen devs, everyone's pass phrase is not a secret
key = Google::APIClient::PKCS12.load_key(keyFile, "notasecret")

asserter = Google::APIClient::JWTAsserter.new(
   email,
   scope,
   key)

puts asserter.authorize()

binding.pry

If all goes well, the JWTAsserter instance will get a valid token and no exceptions/errors will be thrown. If you’ve followed all of the steps listed earlier and things are still breaking, troubleshooting paths are: Verify your machine time is correct, verify your service account email address is bound to google analytics, and lastly you might need to wait until Google platform catches up with the changes. For myself, it took 9 hours until my account finally authenticated through. Google has potentially a million or more servers organized into cells, the fact that they seem to cooperate well doesn’t mean they’re perfectly in sync at all times.

Otherwise if all else fails, I recommend stalking this guy Nick https://groups.google.com/d/msg/google-analytics-data-export-api/maa_fyjD2cM/sT8tDDh0wNsJ – It seems like he’s a Google employee on the service account dev/implementation team and generally stuff gets fixed if he says it’s getting fixed.

As I run into any more troubles, I will update and add more notes.

My business DSL connection was out this evening, so I wrote up a quick script to track how long exactly it was out for.

import envoy
import time

roller = 0
Limit = 50
while Limit > 0:
    rs = envoy.run("ping google.com")
    if rs.status_code != 1:
        print "PING %s" % time.ctime()
        Limit -= 1
        time.sleep(10)        
    
    roller += 1
    if roller % 100:
        print "PONG %s" % time.ctime()

Envoy is probably one of my favorite utility libraries after the Http Request library. Github repo is here https://github.com/kennethreitz/envoy/blob/master/README.rst

Ubuntu 11.10 Google native client dependancies

sudo apt-get install ia32-libs
sudo apt-get install gcc-multilib g++-multilib libsdl1.2-dev texinfo libcrypto++-dev libssl-dev lib32ncurses5-dev m4 libelf-dev

This is about 300Mb of stuff, some of it can be culled but I didn’t feel like messing around with that.

Allows for pepper 19 examples to be compiled successfully.