Predictability

In the last one and a half year, I've been working as a backend engineer for a small startup. I engineer a RESTful API, which implements the business logic of the application, and is consumed by our web client.

Initially I had a large code base to study, and get familiar with the basic concepts of what we had up to that point, how and why it was built, and how we wanted to extend it. It was by itself a difficult task. It took me a while to, firstly, comprehend the big picture, get an intuition of the architecture, and then dive into the details. It was months later that I understood everything. And even this point I keep exploring it and discovering things in retrospect.

The API was initially built using a couple of open source packages. The combination of these packages, was indeed a very stable API framework. However it suffered from a few things, the most important of which was that the responses were often unpredictable. A very common case was the following:

Imagine an HTTP method like DELETE, on an API handler that doesn't allow that method. Depending on the resource that the request would try to access, the handler would yield either a 405 Method Not Allowed, or a 410 Gone response.  

Wait a minute... If the handler doesn't even allow the method, why don't we always get a 405 Method Not Allowed? 

Well, It had to do with an implementation detail of one of the tools we used, which was initially checking the existence of a resource, and then whether the HTTP method was allowed on the handler. So if for example the resource was inaccessible, a 410 Gone response was returned. If the resource was accessible, then a 405 Method Not Allowed response was returned.

And this is only a single example of this behavior.

I did try to modify the internals of the API framework, and get it to behave more like I wanted it to. But I found it very difficult. There was a lot of monkey patching in the code, and a lot of long methods with multiple nested levels. It was just too much. For this reason I eventually decided to develop my own API framework, django-icetea. I had studied the behavior of the API frameworks I was already using, I knew the weaknesses, and I had a vision of what I needed.

It was simple. I needed an API framework that would be predictable. I needed an API framework that given a certain API handler, and a certain HTTP method, would have a very clear, intuitive behavior, and hence response, that I could reason about. The framework should make it easy for anyone who disagrees with the default behavior, to extend or modify it and tailor it down to his preferences.
For this reason, the code should be maintainable, clear and easy to study.

Whether it's RESTful, RESTless, REST-kind-of-sort-of, I don't really care. What I care about is being able to predict its behavior and built my application with that in mind and no unpleasant surprises. A predictable piece of software can be tested and verified. I can't even stress how important this is for any task.

PS. django-icetea is still under development, but already rock solid.

Django and ORM

It's been quite a few months now that I've been involved in a large Django project. One of the things that immediately impressed me, is Django's ORM (Object Relational Mapping) system. In a few words, this is an an abstraction that allows developers to avoid the use of SQL queries, and rather use Python objects to communicate with the database. It's simply a layer over the database, which in reality completely integrates database access with the programming language.

A big advantage of this, is that developers don't need to directly invoke SQL code. I don't like SQL. I consider it non-intuitive and unfriendly. Moreover it is extremely easy to screw up everything using SQL queries, not only because of the nature of the language, but also because of the many security edge case it has.

On the other hand, Django's ORM is much more intuitive and easy to use, integrates seemlessly with Python code, and deals with a lot of the security flaws of SQL.

There is however one problem, that I, for one, often face. Using the ORM, I am often not aware of the  database interaction that I have. When using SQL, I would explicitly declare my queries, and I would know when, what kind of, and how, an SQL query would be executed. Django's ORM integrates so seemlessly with the Python way of doing this, that I often execute database queries without noticing. I simply interact with objects, which behind the scenes, execute database queries to retrieve their data.

Moreover, when invoking methods of the ORM,  I am often not aware of the nature of database queries that are executed. Simple ORM methods, can very well execute 2 or 3 database queries. Other ORM methods are translated to complex SQL joins, which can easily kill performance.

Therefore, using ORMs doesn't come at no cost. Developers need to be very cautious, and very aware of what happends behind the scenes, Relying simply on the abstractions of the ORM is dangerous. The abstraction is leaky.

Django model validation

There are a few things that have been confusing me on Django, and have only cleared out just lately.


When defining a model's fields, what do arguments blank and null do exactly? I would expect that whatever it is that they do, has to do with model level properties. Apparently this is not completely true. 
For some reason, Django assumes that the only interface with the database is Forms, and places controls and validations only over data coming from forms. This leaves the models unprotected from other interfaces, say a REST API, or even access from the interactive python shell, which are obvious alternative ways of interacting with the database.
Let's take a closer look.

---

Django defines blank as: 
If True, the field is allowed to be blank. Default is False.

Note that this is different than null. null is purely database-related, whereas blank is validation-related. If a field has blank=True, validation on Django’s admin site will allow entry of an empty value. If a field has blank=False, the field will be required.

So what is blank really? If True, it says that when you enter data for that field, through a form, you can leave the field blank. If False, then you have to enter a value for the field. The blank argument has no sense when interacting with the database in another way. Say, when you use the interactive python shell, you can add whatever value you want for that specific field, and blank sets no limitation over that. No exception whatsoever will be raised.

---

null is defined as:

If True, Django will store empty values as NULL in the database. Default is False. 

Note that empty string values will always get stored as empty strings, not as NULL. Only use null=True for non-string fields such as integers, booleans and dates. For both types of fields, you will also need to set blank=True if you wish to permit empty values in forms, as the null parameter only affects database storage (see blank). 

This parameter does affect model level properties. It defines how the empty value for the field will be saved. Whether it will be saved as NULL, or whether it will use the field's default value.

---

Then what about model level validations? It seems that Django provides only form level validations. However method full_clean() provides us with the way to enforce validations on model level.
The Django documentation here, mentions: "Note that full_clean() will not be called automatically when you call your model’s save() method, nor as a result of ModelForm validation. You’ll need to call it manually when you want to run model validation outside of a ModelForm.".
Therefore, in order to force model level validations, we will have to overwrite the model's save() method and call function full_clean(). The save() method will call full_clean(), which in turn will call the 3 validation methods clean_fields(), clean() and validate_unique(). Depending on the kind of validation we want, we may override function clean() to provide custom validations.


All in all, the model/form validation is for me, one of the most annoying Django features. I don't understand why Django assumes that data will only be entered in the database through forms. Even so, does it makes sense to provide model definition atrributes/methods that refer to a form instead of the model itself? I don't know if this is some legacy feature that's kept for backward compatibility, but I consider it a major flaw.

Update 22/12/2011: Be aware that model validations often involve invoking database queries. For example the clean_fields() method, which does the field validation prior to saving a model instance. If the model has foreign keys, then clean_fields() will query the database in order to make sure that the foreign keys we have set correspond to existing database records. For a model with 2 foreign keys, there will be 2 database queries before the actual INSERT query. So, be aware of this extra overhead.
When developing always check the query log, and make sure you understand what each and every query does and why it's been invoked.

A song you don't know

Watch out for the bat. A song by Roger Glover, the bass player of Deep Purple, with vocals by John Gustafson.

From Roger Glover's solo album The Butterfly Ball and the Grasshopper's Feas. A great album that I discovered lately. With guest vocals by greats, such as Ronnie James Dio, Glenn Hughes and David Coverdale.

Amsterdam: Urban ripoffs

The following story happened in Amsterdam in the early days of 2011. 

Last few days, a friend had problems with her landlord.
To make a long story short, her contract expired and she moved to another place. She informed the landlord on time. Everything normal so far. The landlord charged her some arbitrary amount of money for cleaning and fixing things that he claims were not in good condition. In addition he claimed that she had to pay him some 380 euros for municipality bills not addressed to her. Of course he wanted to take this money from the deposit. Well, since he has the deposit, he makes the calculations, he keeps as much as he wants. End of story. So he thought. Apparently the landlord thought he was dealing with some foreigners who don't know anything, and he could just rip them off like that.

To make things worse, he was aggressive and extremely rude... towards a girl.

I advised my friend to ask for her whole deposit back. If the landlord would ask for 100 euro off the deposit, then she would need to ask for receipts of the 100 euro expenses for the house.
I went with her, since she was terrified of his aggressiveness. Of course he went crazy when I mentioned receipts. He just demanded a shitload of money off the deposit, with no arguments. I told him we are leaving and that we would advice our lawyers, and let him know.

As I came back home, I did a small research on the internet, checking forums about the rights of tenants and landlords. While I knew that my friend was totally right and she was rightfully asking her deposit back, it was great to see that the Dutch legal system favours tenants incredibly, towards landlords. The best thing of my research though, was discovering Wijksteunpunt Wonen. This is an institute which offers help, support and consultancy to tenants. Searching for a place? Need information on housing? Have problems with your landlord? Want to know your rights as a tenant? This is your place.

Eventually we went to this place. I was hesitant at first. First impression? Very polite people. After a few minutes of waiting we talked to a lawyer describing him the whole situation. Soon enough he assured us that our claims were completely reasonable. He explained to us what we have to do and soon enough he told us that he will act as our representative. He prepared a letter addressed to the landlord letting him know that he needs to pay the deposit back, otherwise all the necessary legal procedures will follow.
An hour ago we were hopeless. Now, out of nowhere we found a great consultant, who informed us about all our rights, and acts as our representative. Free of charge.

So, what do we want from the landlord? As far as I'm concerned, he can keep as much from the deposit as possible. But aggression and rudeness will not be tolerated. If you were more polite and didn't try to rip off a girl, there would be no problems. You would even get to keep your part of the deposit, that you claim will be kept for "damages". Damages that never occured. You never even described them to anyone. Heck, you even did the inspection on your own. And you know what? Such an inspection is not valid legally. That's the fact.

You tried to rip off a girl. It was supposed to be easy. But some of those foreigners are sometimes not that stupid. Now she has a legal advisor. Let's not make this even worse than it already is for you.

The point? We have many rights that we are not aware of. We were lucky enough to find an institute specialized to our situation. It's always worth spending a few minutes researching about a situation, and try to get as much help as possible. Surely, someone else has probably had a similar or even worse story.
Share your story and help the next in line!

Django's Template Tags

Lately I decided to learn Django. It's a framework on which I have read a lot, and I decided that it was time to test it myself.

I went through a few tutorials, but I would always end up to something which was poorly explained. Google searching and the official documentation sometimes didn't help at all, so I would end up banging my head on the wall.

A couple of days ago, I discovered ericflo's great screencast tutorial on Django. On video 7, the screencasts introduces us to template tags. I struggled a lot with them and after watching the video several times and doing a lot of researching, I can finally say that I have somewhat figured out what is going on. 

I will try here to explain my findings, using code snippets from the aforementioned tutorial.

So, what are Template tags? They are custom made tags used in Django templates. No, Really! This mechanism enables developers to define their own tags and call them from within the templates. These tags might as well take arguments, just like normal template tags. If the word tags is confusing, think of commands. So tags are custom made commands for Django templates.

And why would we need this? In cases where we have defined a couple of views which return the same kind of objects, this most likely means that also the templates corresponding to these views have similar code. Why not have a more generic template then, which (at least partially) represents both views? That's where we need template tags!
Personally I hate HTML! I think HTML code is ugly and hacky, and anything that helps me reduce it is more than welcome.

When a template tag is called, a corresponding python function is invoked. The function defines the template tag, and returns a dictionary of key-value pairs. A decorator wrapped around the function uses its output to render a mini-template, and produce some HTML code. This HTML code is injected in the original template that called the custom template tag. This is the logic behind the whole story.
The mini-template I mention, includes the common HTML code that would have appeared in many templates earlier.

How is it done? Let's see a short example that creates a template tag called event.

1. We create a folder called templatetags within our Django app folder. In that folder we create the __init__.py file (standard Python procedure) and another file, say events_tags.py where we define our template tags.

2. Open events_tags.py.

from django import template

register = template.Library()

@register.inclusion_tag(mini_template, takes_context = True)

def event(context, e)

    ...

    ...

The first 2 lines are necessary.

The decorator is the wrapper I talked about earlier. The mini_template argument, is the mini-template that renders the return dictionary of event(), and the takes_context = True parameter tells the wrapper to use the context of the request. That means that any values passed in the POST request, are also available to the mini-template.

event() is the function that produces and returns a dictionary which will be used by the wrapper to render the HTML code from the mini-template. The context is again the context of the HTTP request and e is an argument given when the template tag was invoked. Note that context is a mandatory argument.

3. We can now include the template tag event in any of our Django app's templates. This is done by calling 

{% load events_tags %}

from within our template. This loads the tags defined in the file events_tags.py of the templatetags folder of our app.

Once this is done, we can then call

{% event e %}

from our template. Of course we should make sure that e is a semantically appropriate object for the event() function.

Phew...

Student gig

I was still a student in Greece. Must have been 2005. I had nothing better to do that evening so I went to a local bar where a student band would be playing classic rock. You know Highway Star, Cocaine, Paranoid and the like. 

At some point, I couldn't believe my ears... I couldn't believe what I was listening to. I had never heard it before, but It was the most beautiful and moving melody I ever listened to. I kept having this melody in my head for a long time but couldn't find out the song or the artist.

A few months later, I was listening to an album I had borrowed from a friend. Rising by Rainbow. I was a big fan of Deep Purple and Ritchie Blackmore, and l was quite enthusiastic about his next project. 

The rest is history. That song was Stargazer. From that point, I became a huge Rainbow fan. Rising and Stargazer, are the product of a guitar genious and an unbelievable vocalist when both were at top of their game.

The amazing cover of this masterpiece

Funny how a stupid student gig can change so much in someone's life. I don't think the guys playing that night would ever think so. It's these small things though, that make life so exciting. Yea, they rarely happen. But when they do, you get to remember them forever.

Still, years later, when listening to:

"Time is standing still

He give back my will"

I get goosebumps. I'm grateful for that.