daml 0.1.4 Release, A Python Markup Language for the Web

Since my initial post outlining daml as a haml-for-python, a lot has changed. Now it has become something more and plays a new role in a bigger project I am working on. Let’s look at some of the new syntax changes.

CSS Selector Style Attributes

Declaring attributes now follows css selectors appended to a tag hash. For example,

%html
    %body
        #content.main[attr=val][apples=yes] some content

Inlining Tag Hashes

One can now inline tag hashes. The functionality is still being worked on and is broken beyond a single tier. The intent is to handle use cases such as,

%html
    %body
        %ul#nav
            %li %a[href=/] Home
            %li %a[href=/contact] Contact

        #wrapper #container
            #top
            #content

Implicit Python Declaration

Since my last post, all python code had to be preceded with a colon (:). Now, this is only necessary when embedding a function call in plain text (or calling a filter). For Example,

def greet(s):
    return 'Hello, {0}.'.format(s)

nav = ['www', 'www2', 'www3']

%html
    %body
        %p A greeting for you. :greet('Daniel')
        %ul for x in nav:
            %li {x}

The trade off to the above is explicit line breaks for multiline text as similarly done in python,

%html
    %body This is some text \
        that spans multiple \
        lines.

I am still putting alot of thought into this portion of the syntax and it will most likely see some changes in a number of months. Explicit line breaks can become rather confusing when handling complex mixed contents, but again, tools such as daml aren’t particularly suited for writing page content, but rather establishing layout.

New Filters

I added two new filters as a test, these may be removed in future releases (and migrated to my personal set of daml extensions). They are for declaring css and js files. I think the syntax speaks for itself in function,

%html
    %head
        :css /css/
            main.css
            extra.css
            lib/that_lib.css

        :js /js/lib/
            utils.js
            js.js
            jquery.min.js
            etc.js
    %body
        %h1 I do enjoy filters!

I think such filters could be upgraded to support appending additional files, but again, I dont really have a place for such things that aren’t a necessity to a fully featured template engine.

In Summary

This release requires cython >= 0.13 and lxml. I use this regularly on Windows 7 x64 and linux. The 0.1.4 release can be found on pypi, http://pypi.python.org/pypi/dmsl/0.2 and you can follow development at github, https://github.com/dasacc22/dmsl

Advertisements

Python implementation of HAML

UPDATE Dmsl is an active project on github and I am looking for active developers that can submit any bugs they may come across. I have been using it in production environments for nearly one year as of 12/2011. Refer to the repository README for an in depth look at current development, http://github.com/dasacc22/dmsl/blob/master/README

So it has been a busy past few days. During which I have written up an implementation of HAML in python. The closest thing I saw to this before was GRHML, whose site seems non-functional and I had a hard time finding information on its status. So without further ado, let me introduce DAML, my HAML implementation.

:title = 'Hello World!'

%html
    %head
        %title {title}
    %body
        :include('header.html')

        :greeting = lambda x: '%p Hello, {0}'.format(x)

        :greeting('user')
        :greeting('me')
        :greeting('world')

        :items = ['a', 'b', 'c', 'd']

        :def is_last(i):
        :   return len(items) is i+1 and '(class=last)' or ''

        %h2 Loop Method 2 (python list comprehension)
        %ul
            :['%li{0} {1}'.format(is_last(i), x) for i, x in enumerate(items)]

        :include('footer.html')

Ok, now this isn’t all it does but let me talk about a couple things first. For one, this project, as of today, is only 3 or 4 days old. Its hard for me to recall b/c I’ve poured a lot of time into trying to make this run as fast as possible. Today on the other hand has produced a lot of slow downs and hackish code for implementing features like Django code blocks. For you XSL people (which would be me) that means a way to name and call templates from other templates. So anyway, a lot of this code from today (and yesterday) needs some love. With that said, lets go over some differences between HAML and DAML when it comes to marking up HTML.

You’ll notice right away that declaring tags is straight forward and precisely the same. I’ve never actually used HAML so I dont know its exact syntax, but I spent a lot of time perusing the documentation for HAML. Now the actual text processor in DAML is quite fast for building documents. At one point during development, when i was still testing speeds in comparison to HAML, I had results along the lines of 0.21ms processing time for DAML versus 2.4ms processing time for HAML. This was for plain-jane HTML declaration. Lets look at some of that now with DAML.

%html
    %head
        %title Good Stuff

    %body
        #header Some stuff here
            and indentions of plain text
            will all be part of this div

            .span while this child div whose class="span"
                will be embedded within the above div

            while this text is tailed
            and multilined too

        %p and heres some random content too
            %strong that can be played
            however you like.

        %p one thing worth noting is that new tags
            %strong need
            to be on
            %em new lines
            so keep that in mind.

All of this renders fine of course. But I still have a TODO list for handling stuff unrelated to python expression evaluating. Namely comments, escaping, whitespace control. This is pretty much in line with HAML thus far although I often see blank lines in HAML appended with equals-sign and I cant recall what for. Regardless, the next thing to note is the use of variables. Which loosely follows the string.Formatter (and would fully support it if not for speed issues at the moment, but soon hopefully). Lets go over an example of setting and using variables.

:title = 'Hello World!'
%h1 {title}

Simple, eh? Now here’s the deal. There is a sand-boxed python eval going on. All lines starting with :colons are getting added to an evaluation queue that gets compiled and eval’d. Currently you would tag variables just like using string.Formatter and in the future, this will support all the goodies associated with it. Currently though whats going on is its simply accessing the variable declared (versus string.Formatter being setup with the documents namespace and getting called). Never-the-less, this works fine so far. Lets look at some other things we can do with “:”

:l = []
:for x in range(20):
:    l.append(x)

%div
    :['%p {0}'.format(x+1) for x in l]

Notice first, that you can basically freely declare normal python code blocks. You can also declare functions to make use of as shown with the initial intro document. I would say, ideally, the syntax I would want to go for in the majority of cases is something like (for-loop with plain-text block not yet functional)

#body
    :items = ['a', 'b', 'c', 'd']
    :is_last = lambda i: len(items) is i+1 and '(class=last)' or ''
    %ul
        :for i, x in enumerate(items):
            %li:is_last(i) {x}

    #div.class(attr=val, attr2=val2) oh and here's this too

So this is something I am working towards, but you can totally declare :func(*args) inline right now and it works. Notice, going back to the plain-text that you can freely declare %tag#id.class(attr=val) in almost any order, the only except is you can not start a line with (attr=val) nor can you declare (attr=val)(attr=val) though the latter may be added. Part of my TODO list is to have the ability to span attributes across multiple lines.

Ill touch on one last bit here, I implemented something similar to django blocks (at its most basic level). Its a wee bit limited until i implement multiline filter options as part of the preprocessor I *just* started working on not but 4 hours ago. But let me show what the final syntax would look like

:extends('template.daml')

:block header
    %h1 {title} OVERRIDE!

Now the above doesn’t work until i finish a bit on this preprocessor but to show you how that will work, let me show you how the above does currently work

:extends('template.daml')

:block('header\n    %h1 {title} OVERRIDE!')

Now that does work, and as you can see, my preprocessor is basically going to go through for indented portions of text under colon directive calls and push it into the related function thats in the sandboxed globals. This basically means that you can write all sorts of text filters and everything really really easy b/c then back on your end you may have a need to do “blank” with “blah. So..

my_python_code.py
-------------------------
def my_filter(s):
    s = s.splitlines()
    for x in s:
        pass # do blank with blah
    return s

I’ve basically tried to unify a number of features of HAML here so that anything can be easily overridden or customized and extended. Nothing thats just built-in and untouchable. There’s even more to come surely but I am honestly way burnt out on this.

Let me lastly explain just how FAST this works currently. Now before I went in today/yesterday adding in all these crazy bits and pieces, I grabbed the bench suite from genshi that benchmarks all the most popular python templating engines. I wrote my template to do everything to spec as the others were doing and these were my results.

Mako: 0.38ms
DAML: 0.44ms
Cheetah: 0.66ms
Genshi-Text: 0.96ms

and after that it just trailed off into really big numbers with Genshi coming in at 1.5ms?? and django at 2.4ms?? So this thing has the potential to be FAST. Now since the hackery I’ve pulled in the past number of hours I’ve actually almost doubled my processing time and the above test for DAML now runs at 0.85ms but keep in mind i have done no optimization/refactoring/code-cleanup/obvious-bug-fixes-needed to the codebase i just recently committed. So that portion needs some love and I’d love to get the speed back down to what it was just early yesterday morning.

Now, if your interested in the project, pleeease check it out at github, http://github.com/dasacc22/dmsl and play around with it. I’m going to slow my development down to a crawl comparitively as I’m really burnt out. I wont be implementing any new features and I’ll be cleaning up my code base yet again and trying to write beautiful code to everything ive hacked together today/last-nite.

And please feel free to contact me by email or comments or however regarding this project. I would like to get the code cleaned up so that I can collaborate on different things to really bring this project up to par.

Thanks, and enjoy
http://github.com/dasacc22/dmsl

Motionbuilder from AutoDesk and OpenCV in Harmony

For anyone ever curious, recently I have delved into the world of motion capture, image recognition and the likes for the past week. The task, build a headset with a mounted usb camera that tracks the eye and moves it accordingly to a model in motionbuilder. Just to get this out there, I know nothing about 3D art, modelling or anything of the likes, including motionbuilder. This would be the first time I’ve ever even used the software.

That aside, I asked someone for a model with a moveable eye and what i got was basically a head with the eyes fixed to a null point. when the point moves, the eyes move. So next was to get the data I had been receiving from opencv to motionbuilder. Just to clarify, currently I am simply using a default haarcascade for face detection so as to track something with motion on the screen. Im taking the center point of the face and using it as proof of concept to move the null. Books are expected Monday so i can delve into tracking the pupil.

The thing about motionbuilder though is its python implementation is hooorrible. As in the worst of the worst. If i were to say its barely usable, i might be hitting the head on the nail for some, but it still feels like a bit of an overstatement. That said, its still great that it has python support so kudos for someone atleast trying to implement it. Im sure its a daunting task for the type of project.

Let me briefly describe the limitations for any that might be unfamiliar. For one, the python version used is 2.4.1 and it comes with pyfbsdk library and nothing else. Considering the lack of documentation for the python module, i would typically resort to something like,

import inspect
for x in inspect.getmembers(FBSystem().Scene):
    print x

but as i said no libraries. So the first thing i did was go to the python site and download 2.4.1. Its not actually listed there but just click on whatever the latest 2.4.x is and then change the revision number to 2.4.1 in your address bar. Download, install, then copy over from your c:\python24\Lib directory all the .py’s to your program\ files\\autodesk\\python\\lib folder. Now you can you do some basic stuff like inspect.getmembers. Secondly, the python console in motionbuilder is horrid. You can type in no more than one line at a time. Syntax error’s have crashed the console. I cant up-arrow to previous commands. The output is limited to whatever the last command was that you ran, aaaaand the text of the output isn’t selectable. So that means no copy and paste of all the methods of whatever after you inspect.getmembers… aaargh! I still have the screenshot on my desktop somewhere …

But fear not! b/c there is telnet. From the python console from within motionbuilder, there is a tab that lets you enable telnet. So you can open up a telnet client and go to addy 127.0.0.1 port 4242 and hopefully you should be presented with a python console. I say hopefully b/c i didn’t have the best of luck the first few times b/c of motionbuilder (and/or my own) quirkiness.

And finally, the real bugger in it all is that if you write a python script that takes some time to run, all of motionbuilder locks until the script is finished. So this means no script thats running in the background waiting to receive data. Instead, this data needs to be sent via the telnet link to motionbuilder. I found some great resources, but mainly ill list this particular one,

http://chrisevans3d.com/tutorials/mbui.htm

He’s got some great sample code for integrating a seperate wxpython script into motionbuilder and breaks down a number of things as well. Unfortunately, his code for making use of telnetlib from within a seperate python instance to issue commands to motionbuilder didn’t work out so hot, which is precisely what prompted me to write about this. The code he had listed seemed a bit cryptic with these read_until’s with params of 0.1 and 0.01, and i didn’t see anything of the sort mentioned in python docs (barely looking of course) so I wrote my class for doing this and saved it in a mbpipe.py and it reads as follows

import telnetlib

class MBPipeline:
    def __init__(self, host="127.0.0.1", port="4242"):
        self.tn = telnetlib.Telnet(host, port)
        self.tn.read_until('>>> ')

    def call(self, command):
        self.tn.write(command + '\n')
        r = self.tn.read_until('>>> ')[:-6]
        try:
            return eval(r)
        except:
            return str(r)

Now from my script where i am doing opencv stuff (or simply from your console) I can do a

from mbpipe import MBPipeline
mb = MBPipeline()
mb.call('FBSystem().Scene.Components[216].PropertyList.Find("Lcl Translation")')

and what it returns is the actual tuple from motionbuilder. In any case where the string from the telnet session can be eval’d, you’ll receive the object. Otherwise just the string.

Just thought I’d share. 😀

I may comment later on my experiences with opencv, which so far have been great. QueryFrame, haarcascade, convert to image and push over to pyglet and render to screen my live video (note, opencv has its own windowing and controls which most will probably find useful).

IBM Sliding Puzzle Contest

So someone passed onto me a pdf for an IBM sliding puzzle contest. Basically, it consists of a 3 row by 3 column puzzle with one empty space, you know the ones, and you have to write a piece of software that solves for the answer. The instructional pdf suggests that while its acceptable for your answer to be over 20 moves, it should optimally be about 20 or less and be relatively quick.

At first I had no clue about how to do something like this but found the idea very interesting. Four hours later I had a python script that solves for all possible solutions up to how ever many moves you choose it too. Turns out the shortest answer is 12 moves, according to my script. I haven’t validated the 12 move answer, but i did validate a 14 move answer by hand with success (which was actually a twelve move answer with a repeat move making it 14), and i see little reason for the 12 move answer to be wrong.

Anyway, here it is in all its glory

puzzle = ['0', '4', '2', '5', '8', '3', '1', '7', '6']

answers = []

possible_moves = [
    [lambda p: swap(p, 1, 0), lambda p: swap(p, 3, 0)],
    [lambda p: swap(p, 0, 1), lambda p: swap(p, 2, 1), lambda p: swap(p, 4, 1)],
    [lambda p: swap(p, 1, 2), lambda p: swap(p, 5, 2)],
    [lambda p: swap(p, 0, 3), lambda p: swap(p, 4, 3), lambda p: swap(p, 6, 3)],
    [lambda p: swap(p, 1, 4), lambda p: swap(p, 3, 4), lambda p: swap(p, 5, 4), lambda p: swap(p, 7, 4)],
    [lambda p: swap(p, 2, 5), lambda p: swap(p, 4, 5), lambda p: swap(p, 8, 5)],
    [lambda p: swap(p, 3, 6), lambda p: swap(p, 7, 6)],
    [lambda p: swap(p, 4, 7), lambda p: swap(p, 6, 7), lambda p: swap(p, 8, 7)],
    [lambda p: swap(p, 5, 8), lambda p: swap(p, 7, 8)]
]

def serialize(p):
    return ''.join(p)

def swap(L, m, t):
    if L[t] is '0':
        L[t] = L[m]
        L[m] = '0'
        return serialize(L)
    
def no_dups(S):
    if len(S.split("-")) != len(set(S.split("-"))):
        return False
    else:
        return True

def search(tree, index=0):
    if index < 12:
        for each in tree:
            for i, val in enumerate(list(each.split("-")[-1])):
                if val is '0':
                    generation = []
                    for move in possible_moves[i]:
                        result = each+"-"+move(list(each.split("-")[-1]))
                        if "123456780" in result:
                            answers.append(result)
                        elif no_dups(result):
                            generation.append(result)
                    search(generation, index+1)

search(["-"+serialize(puzzle)])
shortest_answer = sorted(answers)[0]
print "=========="
print "Shortest Answer: " + str(len(shortest_answer.split("-"))-2) + " Moves"
print "++++++++++"
print shortest_answer

Edit: I just thought i would add,
yeah so the thing that was the turning point for solving this thing was how you look at solving the puzzle. Ive always looked at those puzzles as, ok what can i move into the empty space and rotate these pieces around and this and that but thats totally the wrong viewpoint. The way to visualize solving the answer is to look at the empty space as your focus and to move the empty space around, pushing the other numbers around into place. The puzzle suddenly becomes eaiser to solve by hand on your own and thats how the program solves for the answer too, by moving the empty space around, not trying to calculate how to get a particular number to its destination

Project Eulier and XSLT

Well when i came across Project Euler and saw the first problem, i naturally did what anyone in their right mind would do … solve it using xslt

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="text"/>
    <xsl:variable name="iterations" select="1000"/>
    
    <xsl:template name="sum_multiples">
        <xsl:param name="i">0</xsl:param>
        <xsl:param name="incrementer"></xsl:param>
        <xsl:param name="result">0</xsl:param>
        
        <xsl:choose>
            <xsl:when test="$i < $iterations">
                <xsl:call-template name="sum_multiples">
                    <xsl:with-param name="i" select="$i + $incrementer"/>
                    <xsl:with-param name="result" select="$result + $i"/>
                    <xsl:with-param name="incrementer" select="$incrementer"/>
                </xsl:call-template>
            </xsl:when>
            
            <xsl:otherwise>
                <xsl:value-of select="$result"/>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:template>
    
    <xsl:template match="/">
    <xsl:param name="three">
        <xsl:call-template name="sum_multiples">
            <xsl:with-param name="incrementer">3</xsl:with-param>
        </xsl:call-template>
    </xsl:param>
    <xsl:param name="five">
        <xsl:call-template name="sum_multiples">
            <xsl:with-param name="incrementer">5</xsl:with-param>
        </xsl:call-template>
    </xsl:param>
    <xsl:param name="dups">
        <xsl:call-template name="sum_multiples">
            <xsl:with-param name="incrementer">15</xsl:with-param>
        </xsl:call-template>
    </xsl:param>
    <xsl:value-of select="$three + $five - $dups"/>
    </xsl:template>
</xsl:stylesheet>

the same thing in python could go something like

three = [x for x in range(1000) if x % 3 is 0]
five = [x for x in range(1000) if x % 5 is 0]
print sum(set(three + five))

Edit: Im still learning python, well after posting this and on the ride home i realized i could define my list with one line.

print sum([x for x in range(1000) if x % 3 is 0 or x % 5 is 0])

CherryPy, SQLAlchemy, & URI to SQL method

Ok some basic info first. I have sqlalchemy objects defined. I use cherrypy and have been writing methods for doing queries to my tables in the various means ive learned how over the past couple years. that is to say, /show_actor?id=4&production_id=2

I keep looking at this code and thinking, this just isn’t right, i was even recently looking at the turbogears2 wiki example (getting ideas for decorator usage and such) and seeing similar code, Again thinking, this just isn’t right. Then somewhere between thinking and writing some code, i ended up halfway through the following and dont know how i got there. Basically its a uri to sql method. I make use of cherrypy’s default method to traverse the uri that gets passed as *args … well traverse it as in translate it into traversing my sql tables. Its all very short and concise i think.

I got the first draft of my uri to sql done, it goes something like this,

@expose
def default(self, *args, **kwargs):
    for n in args:
        if n in globals():
            query=Session.query(globals()[n])
    for k,v in make_filters(args):
        query=query.filter(globals()[k].name==v)
    yield json(query.all())

and uses this outside function

def make_filters(uri):
    for n in range(len(uri)):
        if not n%2 and n+1 < len(uri):
            yield [uri[n], uri[n+1]]

Basically now, the args passed to my default method let me traverse my sql tables like it was a directory tree. So you might say, i can open my folder Productions and find “aa” in there, and then i can open my Capture data related to “aa” and find MC001, or go to my Actor folder and get all my actor files such as “Jon” all accessible via

/show/Production/aa/Capture/MC001
or
/show/Production/aa/Actor/Jon

, then using keyword arguments like

/show/Production/aa/Actor/Jon?type=json

i can specify special conditions or whatever and ive totally eliminated this really lame reiterative code of multiple methods for calling different data like /show_actor?prod_id=1, or calls with multiple if statements for handling what im asking for and its totally recursive no matter how deeply nested my tables are.

Im sure theres tons of problems and some shortsightedness here, and im still a relative newb to python (going on 6 months of use now i think) but anyway, i really dig this. Now I can start coding my ajax app to grab this or grab that and as I define tables through sqlalchemy, i will automatically be able to access my data.

Some things I need/want to do now is to provide custom filter options via keyword arguments and some sort of deliver data as type .. w/e, json, xml, yaml, yadda yadda yadda

Also to note, i call a json method in my code, thats part of a (probably overly complicated) generator i wrote that takes an sqlalchemy result object and produces a usable dict that gets dumped to json. It needs some extra work I think to be more useful but does what i need it to for now. Heres the code for that

def gmap(obj):
    for item in obj.__dict__.items():
        if item[0][0] is '_':
            continue
        if isinstance(item[1], unicode):
            yield [item[0], str(item[1])]
        else:
            yield item

def json(obj):
    if isinstance(obj, list):
        return simplejson.dumps( map(lambda x: dict(x), map(lambda x: gmap(x), obj)) )
    else:
        return simplejson.dumps( dict(gmap(obj)) )

Thanks to google’s advanced python talk on video.google for breaking down some key concepts on generators and such.

CherryPy, SndObj, and SVG

Ok so sometime back when i first started tinkering with pysndobj, i was toying with some ideas for a user interface. Primarily i come from a web background and i decided to toy with the idea of a web frontend to a sndobj thread(s) that would write to an output stream that multiple people could connect to to work collaboratively together semi-realtime. Yeah, it sounds a bit amibitious but anyway I decided to take the time to toy around with SVG as well, which ive never worked with, to see what I might come up with.

Ultimately, I got this, cpsndobj.

This is the source code to my simple cherrypy/sndobj/svg demo. Basically I have an svg knob that i made that you can manipulate by clicking and dragging the mouse down on it. Unfortunately with a browser you cannot lock the mouse down in place so it can seem a bit odd if you reach the edge of your screen. But anyways, tar -xzf this bad boy and ./python cpsndobj.py and you will find a local webserver at http://127.0.0.1:8080/. Of course you need to sudo easy_install-2.5 cherrypy if you wanna use it. The rest is javascript(jquery if i remember correctly(been a while since i looked(i think this is why i like python, avoiding all these tags(woot!)))). And I believe I have it set to connect to a jack server so you’ll want to change it to SndRTIO if you want otherwise or start your jack first. After you get it up and running, visit the local address /on to turn on the modulating frequency the visit the root of the site to display the svg knob. Now click and drag it. and youll see it do its thing. Things to note, if your using internet explorer, you’ll need to install the adobe svg viewer (though ive not tested it in IE). Here in firefox land, we apparently like things to run slow so if youd like to experience a smooth svg knob .. experience, then get opera installed and notice the difference.

Uh but yet, i dont use opera past playing with an svg knob and playing flash movies that dont lock up my browser in linux.

One might say, ultimately i basically have a tinker toy thats ok (woot!).

–Edit: I suddenly realized the fallacy of my “i like python” statement above, python still has paranthesis.. duh..

SndObj, Jack, and Inputs

Ok so I have been tinkering with pysndobj on and off for a while. One thing I have been wanting to do is get it setup where i have a thread using SndJackIO and doing a line in from my edirol ua-25 usb soundcard with my guitar. I couldn’t find any documentation on how to do it with SndJackIO for a while, though there was plenty for SndRTIO. But then i noticed a reference to core audio on mac and doing inputs. Effectively, there is a single SndJackIO for both input and output. So when instantiating for such just do a

outp = sndobj.SndJackIO(“MyName”)
inp = outp

Anyway, here it is in all its glory

import sndobj

jack = sndobj.SndJackIO(‘test5’)
inp = jack

snd = sndobj.SndIn(inp, 1)
cmb = sndobj.Comb(0.001, 0.001, snd)

jack.SetOutput(1, cmb)

thread = sndobj.SndThread()
thread.AddObj(snd)
thread.AddObj(cmb)
thread.AddObj(inp, sndobj.SNDIO_IN)
thread.AddObj(jack, sndobj.SNDIO_OUT)

thread.ProcOn()

–Edit: Above, cmb is a filter that is not needed. You can simply SetOutput to snd instead of cmb and bypass that.