Python dataclass database (DCDB)

Why

While I do use sqlalchemy and to some extent peewee for my projects, I slowly got tired of having to relearn how to write SQL when I’ve known SQL since the mid-90’s.

DCDB’s design is also aiming for simplicity and minimal behind the scenes automagical behaviors. Instead complexity should be added voluntarily and in such a way that it can be traced back.

Example

import dataclasses as dcs
import dcdb 

@dcs.dataclass()
class Foo:
    name:str
    age:int

db = dcdb.DBConnection(":memory:") # alternatively this can be a file path
db.bind(Foo)
"""
   Bind doesn't change Foo in the local scope but instead
   it creates a new class DCDB_Foo which is stored to the DBConnection in it's 
   table registry.

   Behind the scenes, a table `Foo` is created to the connected database.  No changes to the name are made (eg pluralization). How you wrote your bound dataclasses is almost exactly how it is stored in the sqlite database.

   An exception is that a .id instance property along with DB methods like: update/save, Create, Get, and Select are added to the class definition.

"""
record = db.t.Foo(name="Bob", age="44")
assert record.name == "Bob"
same_record = db.t.Foo.Get("name=?", "Bob")
assert record.age == 44
assert record.id == same_record.id

record.age = 32
record.save()

same_record = db.t.Foo.Get("age=?", 32)
assert record.id == same_record.id
assert same_record.age == 32

same_record.delete()

"""
Note it is important to notice that currently same_record and 
record have the same .id # property but they are different 
instances and copies of the same record with no shared reference.   
Changes to one copy will not reflect with the other.

"""

Github DCDB

Hacking around a hack

So google friendly search terms

Pexpect, subprocess.POpen. How to suppress quotes from Popen with Windows. Answer: You cannot BUT you can hack around it.

Note I am using delegator.py BUT it is just a very nice wrapper around PExpect which in turn is a really nice wrapper around Python’s subprocess.Popen which is a really nice wrapper around a ball of shit called CreateProcess. Pedants asside, you can’t make a ball of shit shine no matter how much abstraction you throw at it.

Solution:

fix = f"some_exec -arg abc -arg 123 -arg3=foo"
cmd_line = ["cmd", "/c", fix ]
result = delegator.run(cmd_line)

This looks goofy and it is. The problem I ran into is with a POSIX compliant app that wasn’t prepared for Windows compliant shenanigans https://docs.python.org/3/library/subprocess.html#converting-argument-sequence

Illustration

$some_exec -arg abc -arg2 123 -arg3=foo

That’s what I expected of crafting a subprocess call with an argument like

delegator.run(["some_exec", "-arg abc", "-arg 123", "-arg3=foo"])

What I got was

"some_exec", "-arg abc", "-arg 123", "-arg3=foo"

Which is hilarious because `some_exec` was so not prepared for that and wasn’t able to parse the command line arguments. How we go from a list of command arguments to a quoted text string is all thanks to this https://github.com/python/cpython/blob/3.7/Lib/subprocess.py#L485

Unless you want to write your own standard library version of subprocess.py just so you can work around one single (likely many) “non-complaint” Windows console executables, you need to abuse the cmd.exe shell’s own quirks.

To demonstrate what I mean

    fix = f"some_exec -arg abc -arg 123 -arg3=foo"
    cmd_line = ["echo", "cmd", "/c",  fix ]
    result = delegator.run(cmd_line)

Outputs `cmd /c “some_exec -arg abc -arg2 123 -arg3=foo` which is in turn executed by cmd.exe without quoting every single argument to `some_exec`.

I walked downward from delegator.py’s run() through pexpect to get to subprocess.py’s POpen and this is it, there wasn’t a cheaper fix then this.

For more info on why this works – tldr “cmd /c argument” is run inside of its own temporary shell and all of that sanitizing code in subprocess is skipped because it see’s the `fix` part of the sequence as one gigantic argument to `cmd`. If you try it any other way, say you try to skip out on cmd, you will get `some_exec “-arg abc -arg2 123 -arg3=foo”` which put me right BACK to having my some_exec blow up as it doesn’t know to strip the quotes out (or even to look inside for the arguments).

Summary: I didn’t log a bug report to some_exec’s maintainer because I needed this to work right the hell now and I didn’t have the time or energy to push through a bug report, wait for a fix, and I don’t have time to even wade into some_exec’s code base to figure out what to patch. Nevermind if it is written in C or C++ which I could do but I haven’t used either language in about a year so what I did write would be some ghastly buffer overflow exploit cesspool. I don’t have time to do that. Besides honestly if I was the developer/maintainer of some_exec I would be skeptical if I even wanted to deal with this bullshit on top of everything else that is more pressing.

WSL GNU/Debian Python 3.6.5

Relevant date: 2018 May 30th
This works today but there is no guarantee this is the right way in a month and especially a year later.

After install WSL and then Debian linux, the first thing I did was something recommended from the reviews:

sudo apt update && sudo apt full-upgrade && sudo apt install aptitude
sudo aptitude install ~pstandard ~prequired ~pimportant -F%

Googling led to some goofy answers like using a PPA (which isn’t a bad thing) but you can screw up your environment if not careful

So….. time to go old school and build from source.

sudo apt-get install -y make build-essential libssl-dev zlib1g-dev   
sudo apt-get install -y libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm 
sudo apt-get install -y libncurses5-dev  libncursesw5-dev xz-utils tk-dev

Those are the basic requirements BUT I am fairly certain somethings like lxml (for Beautiful Soup) will be missing. Not an issue if you don’t intend to use lxml AND they are not required for building the python 3.6 interpreter.

wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz
tar xvf Python-3.6.5.tgz
cd Python-3.6.5
./configure --enable-optimizations --with-ensurepip=install
make
sudo make altinstall

NOTE! you can do

make -j8

for a faster build time BUT ideally using

-j# where the # is the number of cores on your CPU

is better

NOTE! if you have a faster computer, the actual compile process is fairly quick but than it runs a post build unit testing suite that is absolutely massive because of the `–enable-optimizations` flag. If you need speed for a commercial environment AND you are 100% certain your environment across all machines is the same, just retar up the source directory with the pre-built binaries, send it to each machine, and then run `make altinstall`.

From there you can run

user@machine:~/$python3.6.5

some people recommend fiddling with

alternatives

so you can just do

user@machine:~/$python3.6.5

but generally I use virtualenv and explicitly make a new environment with my version of python to keep me sane.

Python 3 – abusing annotations

Since my retirement a few years ago my habit of trying out sometimes useless or convoluted ideas has gone up a few notches.

Latest discovery is that `inspect.signature()` passes parameter annotations straight through. With a bit of function decorator hackery, you can get positional/keyword instrumented transformers.


@magic_params
def some_func(always_str:str):
   print(f"always_str is {type(always_str)} with a value of {repr(always_str)}")

>>some_func(123)
always_str is  with a value of "'123'"

another example


def reversed_str(raw):
    if not isinstance(raw, str):
        raw = str(raw)

    return raw[::-1]

@magic_decorator
def goofy_func(bizarro:reversed_str):
    return bizarro

assert goofy_func("Hello World") == "dlroW olleH"

A working proof of concept

In one of my pet projects, I have a method with a signature like `def process_request(self, action:SomeClass.FromDict)` which takes a dictionary for the `action` parameter and passes that to SomeClass.FromDict which then returns a instance of `SomeClass`.

In another case, when dealing with Twisted in Python3 and that all strings are type `` I used something like the magic_decorator above and a transformer `SafeStr` (ex. def do_something(name:SafeStr)` to ensure that the name parameter is ALWAYS of type str. Anecdotally Python3 blows up if you try to do something like .startswith()`.

Grand scheme I think this is an interesting quirk but if my comments and wording isn’t clear, I would prescribe caution if using this in revenue generating code (or code intended to make you wealthy or at least provide money for pizza & beer).

Flask CRUD with sqlalchemy and jinja2 contextfilters

Quick disclaimer, the Flask CRUD thing is not public domain yet and is very volatile.

The project is here
https://github.com/devdave/wfmastery/tree/revamp_1/wfmastery
And the outline for the crud thing is in this commit https://github.com/devdave/wfmastery/commit/e249895ddc53c0696f59d3def5718e76855af5b9

https://github.com/devdave/wfmastery/blob/revamp_1/wfmastery/crud.py
https://github.com/devdave/wfmastery/blob/revamp_1/wfmastery/views.py
https://github.com/devdave/wfmastery/blob/revamp_1/wfmastery/templates/equipment_list.j2.html

First is how the crud is currently constructed

class Equipment(CrudAPI):

    def populate(self):

        self.record_cls = db.Equipment
        self.identity = "equipment"

        self.template_form = "equipment_form.j2.html"
        self.template_list = "equipment_list.j2.html"

        self._listColumn("id")
        self._listColumn("hidden")
        self._listColumn("name", magic_field="magic-string")
        self._listColumn("pretty_name", magic_field="magic-string")

        self._addRelationship("category", "name", magic_field="magic-filter")
        self._addRelationship("subcategory", "name", magic_field="magic-filter")

both vars “template_form” and “template_list” are going to be preset once I am certain that the templates can stand on their own with the context vars provided. The “magic-” params and their use are very much magic (eg really toxic) and would recommend ignoring them.

From there the CrudAPI takes over. Skipping ahead to how this relates to context filters. I had this tag mess here in the template

-{%-      for column_name in origin.list_columns -%}
 -{%-          if column_name in origin.magic_columns -%}
 -        {{ cell("", column_name|title, classes=origin.magic_columns[column_name]) -}}
 -{%-          else -%}
 -        {{ cell("", column_name|title) -}}
 -{%          endif %}
 -{%-      endfor %}

and was really not happy with it. So I dived into Flask and Jinja2’s documentation and code to figure out if I could apply Python code inline.

The answer is yes via jinja2’s contextfilters which are not exposed to Flask but can still be used.

@App.template_filter("render_header")
def render_header(context, column_name, value="", **kwargs):
    result = ""
    if column_name in context['origin'].magic_columns:
        result = context['cell'](value, column_name.capitalize(), classes=context['origin'].magic_columns[column_name])
    else:
        result = context['cell'](value, column_name.capitalize())


    return result

render_header.contextfilter=True

The trick to going from filter to contextfilter is just applying `my_func.contextfilter = True` outside of your functions scope. From there you have access to almost everything (if not everything). The var “origin” is the CrudAPI’s instance passed to the template.

This has opened a lot more opportunities to do clean up. Taking


{% macro data_attributes(data_map, prefix="data-") -%}

    {%- for name, value in data_map.items() -%}
    {{" "}}{{prefix}}{{name}}="{{value}}"
    {%- endfor -%}
{%- endmacro %}
  

{% macro cell(name, value, classes=None, data_attrs={}) %}
        
      {{- caller() if caller else value -}}
  {%- endmacro -%}

and condensing it down to

{% macro cell(name, value, classes=None, data_attrs={}) %}
        
    {{- caller() if caller else value -}}
{%- endmacro -%}

via a simple non-context filter

@App.template_filter("dict2attrs")
def dict_to_attributes(attributes, prefix=None):
    results = []
    name2dash = lambda *x: "-".join(x)
    format_str = "%s-{}=\"{}\"" % prefix if prefix else "{}=\"{}\""

    for key, value in attributes.items():
        results.append(format_str.format(key, value))

    #TODO disable autoescape
    return " ".join(results)

Just note that at the moment output is still managed by Jinja’s autoescape and I’d rather not shut that off so calls MUST be suffixed with “|safe” as used above.

As for the Crud API, I feel like that is coming along nicely.

Flask list routes (rake equivalent).

While working on a pet project I really wanted a rails rake equivalent for Flask.

Googling led to http://flask.pocoo.org/snippets/117/ which gave me enough direction to figure out how to make that work with Python 3.5 and Flask 0.12.

The biggest problem I had with that snippet is that it didn’t account for URL variable rules.

/foo/<int:bar>

as it would blow up because werkzeug & Flask sanitize inputs to ensure they match the expected type.

I started doing some seriously crazy stuff like monkey patching the rules temporarily to make
ALL converters run through a String/Unicode converter. It’s at this point that I noticed in dbgp (symbolic debugger) that it was naturally converting the rules to strings.

@App.cli.command("list_routes")
def list_routes():
    """
        Roll through Flask's URL rules and print them out
        Thank you to Jonathan Tushman
            And Thank you to Roger Pence
            Sourced http://flask.pocoo.org/snippets/117/ "Helper to list routes (like Rail's rake routes)"

        Note that a lot has possibly changed since that snippet and rule classes have a __str__
            which greatly simplifies all of this
    """


    format_str = lambda *x: "{:30s} {:40s} {}".format(*x)#pylint: disable=W0108
    clean_map = defaultdict(list)


    for rule in App.url_map.iter_rules():
        methods = ",".join(rule.methods)
        clean_map[rule.endpoint].append((methods, str(rule),))

    print(format_str("View handler", "HTTP METHODS", "URL RULE"))
    print("-"*80)
    for endpoint in sorted(clean_map.keys()):
        for rule, methods in sorted(clean_map[endpoint], key=lambda x: x[1]):
            print(format_str(endpoint, methods, rule))

Example output

"HOSTED@LOCAL:5000"
View handler                   HTTP METHODS                             URL RULE
--------------------------------------------------------------------------------
Equipment                      /equipment/                              OPTIONS,POST
Equipment                      /equipment/                              HEAD,OPTIONS,GET
Equipment                      /equipment/               HEAD,OPTIONS,GET,PUT,DELETE
index                          /                                        HEAD,OPTIONS,GET
static                         /static/                  HEAD,OPTIONS,GET

PyQT5 QMediaPlaylist documentation snafu

QMediaPlaylist has a method

addMedia

with the signature

bool QMediaPlaylist::addMedia(const QMediaContent &content)

http://doc.qt.io/qt-5/qmediaplaylist.html#addMedia

And the documentation suggests for c++ that

playlist->addMedia(QUrl("http://example.com/movie1.mp4"));

should work.
http://doc.qt.io/qt-5/qmediaplaylist.html#details

BUT in PyQT5

playlist.addMedia(QUrl.fromLocalFile("some/path/music.mp3")

errors out with unexpected QUrl.

So….. you have to do

QMediaContent(QtCore.QUrl.fromLocalFile("filePath/fileName.mp3"))

in python. A bit unwieldy but PySide2 appears to have stalled from an outsiders perspective.

Initially I suspect that QUrl and QMediaContent inherited from some sort of common base class and were split off and its perhaps an untested use case of using QUrl was lost. I ended up making a bandaid in python with

MakePath = lambda x: QtM.QMediaContent(QtCore.QUrl.fromLocalFile(x))

where QtM is `from PyQt5 import QtMultimedia as QtM`

Hotkeys for CKEditor format/styles

Sourced from
http://stackoverflow.com/a/14425430/9908

function bib_onKey(event){
            var styles = [], config = this.config,
                codex = { }, format_name = undefined;
            codex[CKEDITOR.ALT + 49] = "format_h1";
            codex[CKEDITOR.ALT + 50] = "format_h2";
            codex[CKEDITOR.ALT + 51] = "format_h3";

            format_name = codex[event.data.keyCode];

            if (format_name !== undefined) {
                this.fire("saveSnapshot"); #save state for undo
                //Format codes are not stored anywhere I could find
                // so build them.
                var style = new CKEDITOR.style(config[format_name]),
                    elementPath = this.elementPath();
                    #Honestly no idea but its needed to work
                    style._.enterMode = config.enterMode;
                #if format is live, kill it or vice versa
                this[style.checkActive( elementPath )
                    ? "removeStyle"
                    : "applyStyle"
                ]( style );
                #another hit on the undo logic.
                this.fire( 'saveSnapshot' );
            }
        }

put that in the call to CKEditor as a custom config parameter

CKEDITOR.inline("your tag name", { on:{key:bib_onKey}})

os.walk for pathlib.Path

I needed the pathlib equivalent to os.walk so I went to the source code for os.walk and reimplemented it to use pathlib instead

def path_walk(top, topdown = False, followlinks = False):
    """
         See Python docs for os.walk, exact same behavior but it yields Path() instances instead
    """
    names = list(top.iterdir())

    dirs = (node for node in names if node.is_dir() is True)
    nondirs =(node for node in names if node.is_dir() is False)

    if topdown:
        yield top, dirs, nondirs

    for name in dirs:
        if followlinks or name.is_symlink() is False:
            for x in path_walk(name, topdown, followlinks):
                yield x

    if topdown is not True:
        yield top, dirs, nondirs

yes there is Path.glob(“**”) but I like how os.walk is structured and this passes my unit-tests.

dirs and nondirs are generators out of habit. List comprehensions would likely be better and it’s almost impossible to avoid the overhead memory cost so having them as is, is just personal preference.

From a discussion on a python chat board – How to deal with criticism

The poster asked “This may be offtopic but i really want to share this.
Did you ever faced criticism with your code. How to overcome this criticism ?”

And this was my response.

Every so often I go back to something I wrote 10 years ago and I laugh my ass off. Now if 10 years ago someone had done that, it would have been somewhat of a painful experience. People have already mentioned “you are not you’re code” which is mostly true BUT at same time you are your code at this moment in time. When a peer criticizes your code, you need to quickly detach yourself and hear out what they have to say. Sometimes their input is going to be asinine ( “I prefer naming variable after my kid’s” ) but hopefully its going to real value ( “You should implement dash or camel casing, does ‘expertsexchange’ mean Expert Sex change or Experts exchange? “)

Before I give whatever advice on filtering poisonous vs constructive criticism, going to address why you want criticism. If you are a small person, you will be stuck in a small world and only realize the situation when people are selling cars and you’re still crafting buggy whips to a doomed industry ( eg COBOL & mainframes vs Java & server farms ). To improve your craft, yes you need challenging work but you also need to be exposed to different idea’s or you will not grow professionally.

Now as far as handling criticism and deciding if it has value. #1 is that you need to check your emotions and listen. #2 identify the problem they have with your work and clarify so you have concrete examples of what is and is not the problem. #3 Evaluate the value of fixing the problem “Does this improve things for my team and our success” or “Does this make my product better and or more maintainable?”

If the person cannot give concrete examples of #2, tell them you don’t understand. If they cannot do this without resorting to verbal abuse, conversation is over and escalate to supervision or discontinue discourse.

If the person cannot demonstrate #3 ( eg how does using their children’s names make things “better”? ) escalate or discontinue, telling them that you don’t seen an advantage to their proposal.

Finally, emotion’s get you into a fight you may not be able to win or will have costs to your career down the line. One example is a person I found immensely influential to my career and I thought he was the bee’s knees. This person was outspoken and somewhat vitriolic but he was generally right ( or appeared to be ). 5 years down the road, he seems like a “has been” that is constantly verballing threatening to beat up people when they criticize his code or his behavior “I am a MMA fighter, I will kick your ass”. For the most part I think that guy’s career is on its way out as who wants to collaborate or associate with him? Also I am not talking about Linus Torvald… he’s a completely different kind of crazy with a completely different problem.