Cassandra 1.2.x – Wide rows are freaky on AWS m1.xlarge

CQL3 is a very nice abstraction to Cassandra but its important to pay attention to what it is doing.

In SQL land, 1 record == 1 row. In Cassandra 1 record == 1 row, but 2+ records can ALSO be on the same row. This has to do with CQL’s partition and primary keys. Your partition key is what decides which row a record belongs to while the primary key is where the record is in a row. If you only have a primary key and no partition key, 1 record == 1 row, but if you have a composite ( partition key, primary key) every record where partition key is the same is going on the same row.

I had a few rows that were ~30GB in size which put stress on nodes using m1.xlarge ( 8GB heap, 300MB new heap size ) with epic Compaction cycles of doom.

quick setup Datastax Cassandra for Ubuntu 12.04

curl -L http://debian.datastax.com/debian/repo_key | sudo apt-key add -
sudo sh -c 'echo "deb http://debian.datastax.com/community/ stable main" >> /etc/apt/sources.list.d/datastax.list'
 
sudo apt-get update
sudo apt-get install cassandra

Really wish there was a cassandra-core and cassandra-server divide as more often than not I just need the c* tools and libraries and not so much the server itself.

Also 1.2.5 is TOXIC as all hell for high read/write environments.

AWS + python + new datastax driver for python

These directions will setup openvpn ( you need that unless your dev’ing inside AWS )
http://sysadminandnetworking.blogspot.com/2012/12/openvpn-on-ec2aws.html

New Python cassandra driver is here https://github.com/datastax/python-driver
It’s really new so it has sharp edges.

Howto: Make a HTML desktop app with PySide

Given a project structure like:

my_project
 main.py
 www/
    index.html
    scripts/ 
        jquery-1.9.1.min.js

main.py looks like:

import sys
from os.path import dirname, join
 
#from PySide.QtCore import QApplication
from PySide.QtCore import QObject, Slot, Signal
from PySide.QtGui import QApplication
from PySide.QtWebKit import QWebView, QWebSettings
from PySide.QtNetwork import QNetworkRequest
 
 
 
web     = None
myPage  = None
myFrame = None
 
class Hub(QObject):
 
    def __init__(self):
        super(Hub, self).__init__()
 
 
    @Slot(str)
    def connect(self, config):
        print config
        self.on_client_event.emit("Howdy!")
 
    @Slot(str)
    def disconnect(self, config):
        print config
 
    on_client_event = Signal(str)
    on_actor_event = Signal(str)
    on_connect = Signal(str)
    on_disconnect = Signal(str)
 
 
myHub = Hub()
 
class HTMLApplication(object):
 
    def show(self):
        #It is IMPERATIVE that all forward slashes are scrubbed out, otherwise QTWebKit seems to be
        # easily confused
        kickOffHTML = join(dirname(__file__).replace('\\', '/'), "www/index.html").replace('\\', '/')
 
        #This is basically a browser instance
        self.web = QWebView()
 
        #Unlikely to matter but prefer to be waiting for callback then try to catch
        # it in time.
        self.web.loadFinished.connect(self.onLoad)
        self.web.load(kickOffHTML)
 
        self.web.show()
 
    def onLoad(self):
        if getattr(self, "myHub", False) == False:
            self.myHub = Hub()
 
        #This is the body of a web browser tab
        self.myPage = self.web.page()
        self.myPage.settings().setAttribute(QWebSettings.DeveloperExtrasEnabled, True)
        #This is the actual context/frame a webpage is running in.  
        # Other frames could include iframes or such.
        self.myFrame = self.myPage.mainFrame()
        # ATTENTION here's the magic that sets a bridge between Python to HTML
        self.myFrame.addToJavaScriptWindowObject("my_hub", myHub)
 
        #Tell the HTML side, we are open for business
        self.myFrame.evaluateJavaScript("ApplicationIsReady()")
 
 
#Kickoff the QT environment
app = QApplication(sys.argv)
 
myWebApp = HTMLApplication()
myWebApp.show()
 
sys.exit(app.exec_())

and index.html looks like:

<html>
    <head>
        <!-- Oh hell no, I can access anything on my computer via this -->
        <script src="./scripts/jquery-1.9.1.min.js"></script>
 
        <script language="javascript" type="text/javascript">
 
            function callback(){
                console.log("callback!");
                alert("called!");
            }
 
 
            function sendStuff() {
                console.log("sendStuff!");
                my_hub.connect("hello?");
            }
 
            function ApplicationIsReady(){
                console.log("ApplicationIsReady!");
                my_hub.on_client_event.connect(callback);
            }
 
        </script>
    </head>
    <body>
        Tell the hub to connect
        <button onclick=sendStuff()>No, click me!</button>
    </body>
    <script>
        alert(typeof jQuery);
    </script>
</html>

Basically class Hub is a bridge/interface pattern between Python and the HTML Javascript engines. It’s probably best to pass things between the two as JSON as that’s a language both sides are readily prepared to deal with.

Stuff I didn’t deal with yet: The Main window title ( self.web.setWindowTitle), setting a window Icon ( no clue on this one ).

Otherwise this gives me an OS agnostic UI, I can leverage my HTML skillset without having to learn QML or QT Designer’s UI interface, and I can hopefully recycle some logic. Additionally I can go full fledged with the bridge logic, split it between the two, or shove all the complexity into JS and have basic Python endpoints exposed.

Second thoughts on the ZMQInline logic

Initially I was going to set the experiment with ZMQInline aside but I am starting to realize that it does have some merits.

Below is a burnout test I did using dotGraph ( current name candidate )

@ZMQInline()
    def do_kickstart(self):
        self.log.debug("do_kickstart")
        for i in range(1, 100):
            try:
                try:
                    self.log.debug("Process Loop {0}".format(i))
                    response = yield Request(self.control, RequestKickstart(), timeout = 5000)
                except TimedoutError:
                    self.log.exception("Timedout")
 
                except Exception:
                    self.log.exception("Unexpected exception")
                else:
                    self.log.debug("Got "+ repr(response))
 
                    if RespondWithKickstart.Matches(response):
                        self.log.debug("Got Kickstart!")
            except Exception as e:
                self.log.error("Caught an exception leak {0}".format(e))
 
        self.loop.stop()

In Dotgraph, Kickstart is the first task both client’s and services need to accomplish and it’s basically “Hello I am here, who do I talk to?”. Depending on the Kickstart response, clients and services might be sent to a completely different host, port set. The idea is that if DotGraph needs to grow, it can have an array of slave hubs and then delegate new client’s to the most underbooked.

One thing I realized is that the above structure would perfect for doing standoff logic ( attempt, time out…wait, attempt N seconds later). Of course it begs the question of WHY the other party hasn’t responded in time ( are they down or just backlogged? ). Other problem is how to do unit-testing cleanly? Still, I haven’t thrown it away yet so maybe I will find a use for it in DotGraph.

FYI on pyzmq and IOLoop

A funny little quirk has gotten me twice now ( hopefully it won’t be three times ).

In Twisted, once the generator has been called ( even inadvertently ) it becomes the defacto instance.

In PyZMQ, calling ioloop.IOLoop() creates a new loop ( and blows the old one away )… The correct approach is to always call it via

    ioloop.IOLoop.Instance()

I don’t want to talk about how much time I wasted working out how to make a pyzmq like interface similar to Twisted’s inlineCallback, but it was enough!

On a lighter note: how to make my co-worker do XXX

From stack exchange – http://programmers.stackexchange.com/questions/185923/how-can-i-deal-with-a-team-member-who-dislikes-making-comments-in-code/185926#185926

I think anyone who has written software alongside anyone else has had at least one grievance with their peers work. Sometimes it can be benign like

if(foo == bar): do_stuff

versus

if(foo == bar) {
   do_stuff
}

And other times it can a tad more annoying like

class Foo { function Foo(arg1,arg2){ this.arg1=arg1, this.arg2=arg2} }

To get a co-worker to change that style, from what I’ve seen it’s not going to happen by asking them to use more XXX and formatting politely. I worked with someone like this and they got immediately defensive and said “I just don’t like wasting space.” which kind of took me off guard as it didn’t really make any sense. For that case I was able to pull rank and kickback their commit’s which probably isn’t the best solution. I’ve seen other solutions like implementing a CI that stage one was an automatic style checker… which is basically what I did but automated and probably a worse solution as sometimes you need to write something really dirty. Maybe a better solution is code reviews but it is fairly rare to see a company institute those. 9 out of 11[SIC] companies, the guy writing the checks for the software engineers isn’t an engineer themselves and in the short term they’re going to see that one of their senior engineers isn’t directly doing work that correlates to profits. In that last case, when it comes up in a review the lead can justify the expenses by pointing out that they’re minimizing the wasted time other staff have to spend just to comprehend something.

Finally at the end of the day, while it’s important that a person finds reward in their job… it’s still a job and not your private playpen. Instead it’s more like software survivor island, the majority has to work together to keep their company/product going and if one goof ball insists on writing software in a conflicting manner then the majority it doesn’t really matter if they are “right”, they need to conform or GTFO.

For scenario’s where you ARE that goofball, you need to get political and I don’t mean selling the highest non-technical person on your island but demonstrating to your team the merits of an idea and working with their feedback. For real world example, I joined a consultancy as the goofball and started with the company architect and development manager. I didn’t sell them on the idea that I was right but on the idea until it became their “right” idea. From there I prodded momentum until one day the idea stopped being a minority opinion and a coup was staged. A week later the company made a fairly dramatic shift in how they did development. That’s a nice happy ending scenario but it doesn’t always happen. Twice I’ve failed to sell the virtues of unit-testing ( one case they assigned me the task of writing unit-tests which was just a means of getting me to shut up about them… but completely negated the idea) and twice I walked away because I got tired of having last minute “Oh my god everything is fucked up” hack sessions to fix prod. That’s life, sometimes you lose and sometimes you win.

Consultancy, the downsides

People outside of my vocation always marvel at my “flexible” life-style and for a much fewer number the amount of money I make in an hour or a week when employed to a client or project. These things are definitely nice, another nice perk is that I’ve been exposed to a much larger breath of technologies, idea’s, people, and I am probably some sort of interview guru as I can usually reverse engineer what the goal of the interview is. Then there is the fact of life that if I am being called into a project, it’s not because I am some sort of genius code monkey but because the problem is pretty terrible. I might also be thrown under an undeserving bus, months down the road due to things completely out of my control. And lastly there is no future in being a consultant.

With interviews, things usually go poorly when prospective client’s don’t know what the hell their goals are for an interview. Or worse if they just have a bucket list they’re trying to check off. In the first case I’ve politely ended interviews ( as the interviewee ) when presented with “What’s the difference between a inner & outer join” ( for that one I usually draw two circles and shade in relevant sections to illustrate every variation ( left, right, inner, outer) and then told I am wrong. or asked to implement something really crazy like a b-tree on a whiteboard. In the first case this is a red flag that the company is a semi-hostile environment and the interviewer is probably trying to protect their incompetence and the second it begs the question “Would I need to implement my own b-tree logic, what other wheel is being reinvented here?” I’ve seen a lot of the first “No you’re wrongs” in my career so far and they’re completely opposite of the second where the interviewer has good intentions but fails to find out if the interviewee has critical thinking and problem solving skills.
The other problem of interviews is the bucket list interviews, especially depending on who is doing it. Usually its headhunter’s going down a line and are completely ignorant of how insane they sound when they ask questions like “Do you have 10+ years of experience with Ruby on Rails?” That one took me off guard so I asked “Which Rails? Version 2 or 3 and is that with 3’s asset generation system or what?” to which the headhunter had no answer. Then that also begs the question of what is the goal? Technically I have 6-7 years of experience with Rails as I was first introduced to it in 2006 and my last working experience was the start of 2013… but I haven’t used Rails 100% of the time in between. If had to guess I might have 4000 hours of Rails experience which is 6000 hours short of being a master of that ever changing landscape.

Being a consultant doesn’t mean you’re always going to find world challenging work either. For an unnamed client we will call Acme, I was called in because an adjacent department knew me and knew that I was their go to guy for making something that actually worked. A lot of things had gone wrong for Acme, their project manager turned out to be a liar that sat on his feet while a “lead” contractor watched netflix and pecked out some code occasionally. I’ve seen a lot of convoluted solutions in my career and each one is special. For Acme’s case the lead was very good at aesthetics but had never read a book on design patterns or done any business software before. If pressed, I’d say the system was some sort of reversed MVP pattern but with stored procedure style application code ( implemented in the application tier ). Further the system could only be talked to via XHR via a global anchor tag handler that took the anchor tag’s href, wrapped it in XML, and sent that to the server. The server then built the view up, calling injects along the way until it reached the end and sent that back as a two element httpbody document, the first half was javascript to be executed and the second was html. It would eval the javascript xml element and then insert/overwrite the contents of a browsers main view-able content. There was a lot of problems with this system but fortunately the “lead” had been sacked alongside the original PM just before I arrived. Another consultant and I sat down over dinner and after flipping a coin as to whether we were running away, we delegated out goals, the other guy was going to focus on adding new “features” to the system while I worked on figuring out how to refactor the system into a MVC pattern… or at least bring in some sort of simple model system to get rid of the functional procedures crap. We pulled that off. Meanwhile a third “Real” project flow consultant pulled some serious smoke & glitter magic tricks and claimed whatever credit for several 220 hour long months of “fixing” the application. He did all this after we threatened to gang up on him if he got in the way of the refactoring process.

Also even if I do everything right… things can still go badly down the road. For the prior dept at Acme, I built a simple little MVC application with a data bridge from a proprietary database the name of which I can’t remember. It worked and handed it off to someone that knew how it worked, how to maintain it, and life was good. A year and change later the maintainer got fired with prejudice for insulting some executive or along those lines. I say with prejudice because he didn’t do a hand off for the application and all of the documentation for the application disappeared alongside any backups. The replacement then went ahead and corrupted the only intact copy of the application’s database, breaking the application. They didn’t call me up until a week before it was needed again and then I sat through a fairly nasty conference call where people were blaming me for “There’s no documentation, no backups, and it doesn’t work.” In a fairly neutral voice I explained my rate had just tripled and oh by the way here’s a signed receipt by the person vilifying me showing him signing off that “A) Product meets expectations B) Documentation provided in digital and printed format” with a “will notify” consultant of preexisting clause so that I could fix any bugs for free. Even if they had agreed to 3x my normal rate, I am pretty sure pointing out that the relevant work manager was slandering me to cover his incompetence pretty much ended any chance of that dept being a client ever again. I’ve worked in hostile environments before ( people screaming at each other, on the job substance abuse, or micro-managers that are more concerned about covering their ass then actually doing something ) and it’s really not worth it to be there. Maybe 30-40% of all contract/consultant opportunities I see are like this and it never really ends well because it ultimately comes down that the man or woman that could actually fix the problem is just some well meaning person who’s afraid of conflict ( so what was a minor leak in a pipe becomes an insane tempest flood ).

Lastly, being a consultant/code monkey for hire has no future. It will take you down the road a little bit more, cover your bills for a time, and it’s actually got some good experiences but generally a consultant is there to solve a problem while a contractor is there to give the company an emergency release valve ( oh no, we’re running out of money FIRE the contractors!). If a problem is never solved then you’re a terrible consultant and if you’re a contractor for several years in a company…well why buy the cow when you already got the milk? Sure consultants can team up into a consultancy to try and make more money, but that doesn’t scale and becomes unbelievably risky as you grow larger. Furthermore it becomes some sort of pyramid scheme where the company stops hiring experienced engineers & developers and begins hiring student intern/junior engineers but selling them as senior engineers. If done right, the client might not even realize that they’re overpaying by 70% but there will be hell to pay if an auditor shows up one day to find out how things looked below the covers.

Software consultancy/freelancing and dating

For almost ALL consultancies, they start at the bottom of the food chain taking small mom & pop shop deals and slowly work their way upward ( or out of the business ) by gaining relevant skills and reputation. Unfortunately this progression somewhat promotes some bad behaviors and a fatalistic outlook for clients.

A small company can’t afford a NASA space program setup ( unit-test coverage w/70% or greater, Jeeves, scrum/agile when it’s just one developer and one client ) so the task becomes “Get it done on time/budget” and hope the coder’s discipline is strong enough not to deliver a ball of mud. Talking about time/money, these client’s might cover 10, 20, or at max 40 man hours and that’s not going to be enough to cover someone’s basic fiscal needs.

It take’s time to develop and manage reputation, so you might do hundreds of these small client’s for years and that will eventually skew how you approach clients and god help you when a relatively meaningful client comes in. The first 40 or even 160 hour work cycle comes to an end and the client is happy, which is sadly great. Great because it means money and stability, sad because you didn’t have time to do unit-tests, the architecture is hardcoded, and now you’re gonna eat all that technical debt you ignored to “get it done”. It become’s a question of whether you can learn fast enough how to function in a meaningful medium to long term relationship.

At this point I’m pointing the finger at the code monkey consultant as the person to blame, but that’s not quite reality either.

On the other side are client’s that don’t want to commit to a long term relationship under any circumstance. Maybe their a media company or a Boiler room consultancy that doesn’t actually have any employees beyond it’s sales & marketing staff. For these two client types, from my experience they either pay terribly or pretty decently. The latter usually stop paying decently the first time they hire a naive smart kid who’s happy with 1/5th pay. Both company types will setup some sort of long term open relationship with smaller firms but it’s understood that if they can find better and or cheaper, you’re gone. Honestly I don’t mind them too much as atleast their honest about their intentions. Still you can’t build a life on these firms as who knows when they’re going to kick you to the curb or never call back?

Also, I’ve run into a lifetime supply of extremely dishonest employer’s but the real dishonesty lies in themselves and their blind faith in success. “I know we contracted you for 6 months, but we ran out of money and can’t pay your last invoice” has happened a few times to me and people I know. For me I usually ask them if I can take something of equivalent value “How much do you think this desk lamp is worth?” and then walk away.

How this all correlates to dating explicitly is that sometimes your first real relationship seems like “It’s the one” until it’s not. Other time’s the one’s you think are right for you turn out not to be “…Google’s awesome but I feel utterly useless and alone…” is something a friend voiced to me on his 1 year anniversary at the super star company. Then there is “Yeah they make me work long/strange hours but the pay is good” sounds like “Yeah, Baby might hurt me occasionally but I know they love me.” And then there’s always the opposite “Man that was such a great company, I don’t know why I didn’t take it serious” is pretty much like Mike + Mechanics “All I need is a Miracle”.

Also, if anyone’s still thinking “I think you’re stretching, strange grammatically challenged wordpress guy” think about job/proposal interviews. You might not flat out lie and say “I invented C when I was 6, that Ritche guy just copied from me.” and how many “Fantastic opportunities” can there be in the entire software industry? I’ve lost count of how many “We have an amazing idea, just sign this NDA” turns out to be a Facebook/Twitter/Whatever is shiny this year clone. We can’t all be super star’s and there is only so many green fields to be developed on before the VC money runs out and it’s murder-mystery software maintenance time. It’s just a fact that both side’s in the average blind date want you to see them at their best and deliberately downplay their downsides, “This sweater hide’s my man boob’s perfectly” or “This dress perfectly cover’s my love handles.”.

For me, I had success in dating by writing my own rules. One of the first few things I said to my now wife was that I had man boob’s so I should be qualified to have some weigh in on female conversational topics ( except I don’t wear a bra, so I still got excluded). Sadly in business, my experiments in being brutally honest/real in project proposals/interviews hasn’t paid out as well… it might also have played against me that the word “eccentric” has been liberally used by others to describe me.