Context Based Drop Down list - Django Admin

Problem : We have a django admin forms with Department and Employee models ,where a Department is a Foreign Key to each employee ( Each employee belongs to one department) . In one scenario we want  to edit a department , and while editing the department model we want to see  only those Employees in the drop down list having relevance to the department / model being edited. So the use case will be

1. List of Department to edit.
2. Click Department and Department page opens
3. Here only the Employees applicable to the department being edited and not the super set.  Normally django will show all the objects linked to the Model and not only the ones linked to the one being edited

Solution: Overide the formfield_for_foreignkey function in admin.py under DepartmentAdmin. Most of the example like the one in django official tutorial shows how one can select things based on the logged in user , but how about selecting things based on Model being edited ?  So lets override the "formfield_for_foreignkey" for populating the employees based on the department being edited, as django shows the id or primary key of the object being edited in the request path , we will get this id and filter the records based on the this id;

Now when we edit we will see only the employees relevant to the department we are editing. ....With this we are good to go !

Crash Course in Django - Part 2 : Step by Step Guide to Make Django App Real Quick

Introduction :  In  first part we covered the basics of creating objects , saving and retrieving them from database. In this series we will try and extend it further with a user interface and setting it all up -Lets try and create one employee listing ( Refer part 1 for the Employee Problem ) interface where all the companies employee are listed and can be searched by name. This will a step by step tutorial to make a django application rapidly and will also try and integrate various other technologies like Jquery , Twitter bootstrap etc with Django Templates. Lets start again with the bullet points:

Tip : 

• There are two ways to handle this ride
  • You can get the code from here and start walking through each point  to make it real quick. With all files in place you can just cruise through 
  • Or you can start step by step with thought process and start creating your files and verifying the changes made with mock ups and gist file changes posted after every essential step
• Now which step to choose depends - again on how quick is quick. 
• Last , please don't forget this is not a series on Jquery , User Interface or html layouts etc. they are just there to fit the purpose 

Get Geared: 

• Lets create a mock up first  - how our interface will look like 

• For creating this form we would be using twitter bootstrap  and the corresponding html will look as shown below

• In real world application I would remove the side pane of employees listing with link button for detail as we can show all the employees present with click of search button with no parameters. The side pane is included in this article to show some of the Django template language features. 
• Lets quickly start creating the project structure 
  • Create Project : $ django-admin.py startproject demoapp
  • Add Postgres option in settings.py as database
  • Add demo app as in INSTALLED_APPS section of setting.py
  • Add models.py file to demo app sub directory and add the Employee class we discussed in part one. The screen shot below shows the setting.py changes we made
    

    
    
  • From the command prompt in the Project directory where manage.py file is present call
    $ python manage.py syncdb and we will see the message like  which will also create a table in the database. When we call syncdb for the first time django asks us to create one admin login - create one. How to use admin login we will cover later in the series for  now lets focus on the task at hand
    

    
    

    

    
    

    
    
• Now if we want to check if  things are OK , lets learn a new utility from the command prompt type the following: $ python manage.py shell . This will open a command prompt in context of the current project. So if want to check the database connection type 
  • $ from django.db import connection
  • $ from django.db import connection & cursor = connection.cursor(). If no error then we are able to connect to the database
• We can similarly use shell to import Employee object and ORM functions to check the status of our code. For example
  • $ from demoapp.models import Employee will import Employee
  • Employee.objects.all() will return blank as we don't have and employee yet
• Lets add some Employees from the console itself
  • emp = Employee(employeeid=124,name='Mathew',dob='1980-1-11',salary='5423.4') & emp.save()
  • emp = Employee(employeeid=1004,name='James Holmes',dob='1970-10-21',salary='7000') & emp.save()
  • emp = Employee(employeeid=14,name='Monica Hilton',dob='1986-07-10',salary='3000') & emp.save()
  • Employee.objects.all() will now return a set of 3 employees
• We can do a batch save but lets leave it for advance use cases

Lets try and get our html page up & running now with our back end almost ready lets hit on layout part

• What kind of URL we want to expose to our Employees - something like below will do for now
  • url we need : http://127.0.0.1:8000/employees
  • Lets add  url(r'^employees/',get_all_employees) the URLS.py file  and corresponding get_all_employees view to views.py file. The changes will look as shown below for both the files 

                    

• But before we load our html some changes we need to make to settings.py file and we need to have some understanding of how Django Template Engine work

• As we know Django view returns a HttpResponse object and we also tried passing string and JSON to it.  How about giving one well constructed html ? We can do that using Templates . So create a html file and replace the dynamic content - like user name , in our case employee name with variables with double braces {{variable_name}}
• For example my Html says <p>Welcome James !<p>  where James is the name of logged in user and will keep on changing for each logged in user we assign a variable to this and change it like  <p>Welcome {{USER_NAME}} !<p>  and we will pass o a USER_NAME variable to the Template from our view using render_to_response function. render_to response is standard django function that takes the name of the template - html file as first argument and dictionary as second argument. In the above example the render_to_response will be like render_to_response('User.html',{'USER_NAME':Computed User Value})  and the  Django Engine when renders it will replace USER_NAME in html to the value that we have passed. If you want to read more on this please refer here .
•  For our rapid learning purpose whatever  values we want to pass or show in html pass them via render_to_response django function as KEY-VALUE pairs . Use the KEYS on html file which will be replaced by values while rendering. This html file is our Template. 
• Two more things that can come handy while creating Templates - if / else statements on the passed variable can be used within the html file like
  • {% if VARIABLE_1 == TEST_VALUE  %} <p> Good Results </p> {% else %} <p>  Bad Result </p> {% endif %}. Django Will now selectively render html based on the TEST Value and VARIABLE_1 comparison which we pass from backend
  • {% for item in item_list} <p> item.attribute_name </p> {%endfor%}. This will be used to parse list of object and accessing the value inside that object by dot operator. As previously said we will pass this item list also from backend
• So in our case we will pass employee list from backend like render_to_response('Employee.html',{'employees_list':employees_list})  and parse it in the html file for creating the employee list. The side pane of employees we see in the html gets generated like: 

       

• Please note the way we are passing Employee Id to Javascript function - This will be used to fetch employee detail once employee is clicked
• Last setup change - how will Django know where to pick the template from - there is a TEMPLATE_DIRS  tag in settings.py and there we will give the path of the folder where we will keep our Employee.html  , which is  TEMPLATE_DIRS = "demoapp/templates" . And  for loading the static file one needs to provide STATIC_URL = '/static/'  and append the <script> </script> , link style sheet  load  directives in html as below. Don't forget to create one static folder in demoapp folder just at the level we have template folder and put the bootstrap folder ( js, css, img folders inside it). So django engine will look for static folder as defined in STATIC_DIR and append the paths to load the javascript , css and other resources 
  • Also add load static files directive to tell Django Engine to do so {%load staticfiles%}
  • <script src="{% static "bootstrap/js/bootstrap.js"%}"></script> 
  • <script src="{% static "bootstrap/js/jquery-1.7.2.js"%}"></script>
  •  <link rel="stylesheet" type="text/css" href="{% static "bootstrap/css/bootstrap.css"%}"/>
• So create on templates folder inside the demo app folder and place the Employee.html file there
• Try running the server with python manage.py runserver command ( runs the application in default development server )and type http://127.0.0.1:8000/employees/  on the browser . The html we made will show up. Along with the list of Employees we have in the system

         

• We now have two problems to solve - 
  • On click of employee Employee detail should render on the right pane
  • Implementing employee search
• For first we will implement get_employee_by_id view on  the server. Map this view with corresponding URL and call this URL via jquery GET function and then parsing the returned data to be placed in the form. ( As this is a Django Series -Not talking much about the JQuery bit here , please Google about JQuery Ajax calls. Please note how we mapped the URL in Jquery Ajax call to the urls.py file. See onEmployeeClick function below for javascript code 
• For second - Searching employee by id is same as getting name of employee and other details we discussed above via employeeid. I will talk more about Search by name because searching by Id will always return one employee but seaching by name might return more so search functionality will try and place result in row order using twitter bootstrap. Please refer the code below for the same.See onEmployeeSearch function below for javascript code
• And now the mapped url and views changes :
• Now we can download the whole code and compare the changes  from the repository here
• Some files  come by default when django creates an app like init.py and wsgi.py. The later one is used in deployment scenarios that we will cover later ..

This brings us to end of this part and I guess we are good to go ....

Crash Course in Django - Part 1 : Django Basics

Introduction :  This is first  lecture for people who want to learn Django rapidly. This assumes some familiarity with Programming of Web based application but not mandatory. Here I will try to be as generic as possible and will try and redirect the reader to specific as and when needed.  And as true for all the matters of interest - there is Google. So lets start bullet point approach to quickly get us going for Django based web development.

Gear Up : 

• Install Django :  Install Django as mentioned in the Documentation here. The installation is fairly straight forward both on Windows and Linux based OS.
• You need a Data Base - Django comes with SQLLite Database for practice purpose but I suggest you either install MySql or Postgres. If you are not using the inbuilt SQLLite database you also need to install connectors for same. Find more about connecting to various databases here. I am not repeating it here because all this is fairly well documented
• When you are up and running with basic setup of Django start by creating a new project by typing the following command on the command line:  

django-admin.py startproject demoapp

• Demo App is a name of the directory and will contain manage.py file. This is the file that is used by Django to deliver its magic. ( More on this later). The command will also create a directory structure as 

demoapp/
    manage.py
    demoapp/
        __init__.py
        settings.py
        urls.py
        wsgi.py
        models.py ( This is file we will add later. Not by default)
        views.py ( This is file we will add later. Not by default)   

• Django Follows MVC pattern to achieve separation of concerns. Like the way we have Controllers in the MVC pattern to take care of logic, Django call the controllers as views. These views are collated in one file named as views.py. So any function you can think of like addition of two integer or a function to sort and return certain data set, will be written in views. View is nothing but a function that takes certain argument -request and givens certain output - HttpResponse
• The constrain with view is that it should always return HttpResponse type of object. So even if we want to return a string or JSON, put it inside a HttpResponse object like 
• As shown above view has a default argument as request - this is nothing but a HTTP request object which can be used to access variables from URL or from the html form being submitted. Also it returns HTTP Response. In a nutshell use the request object to get the data out of html form. Use the HttpResponse to render data on the form. Django comes with a template language which can be used to render calculated variables on the form. Will be covering Django Templates in later part of this series
• Next file of interest is urls.py , this file is nothing but a collection or a map of url and function ( Django view ). Its a way to tell the Django Engine - if a URL of specified type ( this also includes regular expressions) comes as a request on the server , redirect it to the view specified in the map. From there on the view takes care of processing the HttpRequest object and rendering a HttpResponse

• Essentially URL.py is a file where pattern in URL is matched and appropriate function is called from the views.py file. So if the incoming request is like http://127.0.0.1:8000/get_units , Django Engine will match it to  first URL entry and will call get_units function from the view which will in-turn return the JSON object we talked about earlier
• So till now - views.py contains  logical bundling of function that operate on Incoming Request  and urls.py  helps that request to reach the suitable function. Question is where and how we get and manipulate data from ? After all its not only about rendering the incoming request to a Django Template
• Comes in the frame models.py file - This is the file that contains all the data holders we need to get and manage data from database system. A model is in-program representation of Data Tables in the Database. Lets take an example on this - consider the application for creating a Employee List
• Employee List application will  store employees detail in the database - lets assume our employee has four attributes and attribute type

Attribute Name	Attribute Type
Employee Id	Integer
Name	string
Date of Birth	Date Time
Salary	Float

• In the Database this will be employee table & in django this will map to employee model as 
• For legacy databases - use db_column attribute to map each and every table column with Django Model and use Meta tag to map the table as shown above. For fresh implementations call python manage.py syncdb and Django will create all the tables for us

• Once the model is created we can use Django ORM to directly work with Employee Objects , which in-turn will interact with database. For Example if we would like to do following operations on our Employee from our python view ( remember all the logic and action is in views.py) import the employee model in the view and call the CURD functionality directly using Django ORM. 

Creating Employee	emp = Employee(employeeid=124,name='Mathew',dob='1980-1-11',salary='5423.4')
Saving	emp.save()
Get All employees	Employee.objects.all()
Get Employee By id	Employee.objects.get(employeeid=124)

• Once we have reached so far we can easily manipulate on data sets in data base using Django ORM to put it all together lets try and see how one will implement a URL -  http://127.0.0.1:8000/get_employee?EMPID=124  API using whatever we have discussed where EMPID is passed a Query string parameter to the view
• Last - settings.py is the file where one keeps all the configuration info regarding the web application - like data base settings . logging , application to include , admin module etc. 
• This brings us to the end of this part. Hope we are good to go ! 

Business Rule Engine in Python

Problem:  This a typical business case – we have a lot of business rules to define a bill for certain Business Entity. These business rules were changing every now and then. Just to give a brief example, what are the kind of business rules we are talking about:

Sample rule set:

Charge $ X if quantity P is between A & B	Charge $ Z if time of day is between D & E
Charge $ Y if quantity Q is between B & C	Give DISCOUNT if R is less than F

These Rules will repeat in sets for various other business scenarios and Categories for example

Category 1	Rule 1
	Rule 2
Category 2	Rule 1
	Rule 2
	Rule 3
	Rule 4

One way to solve this issue is a lengthy business analysis cycle, where a Business Analyst sits for 2-3 month and write a complete set of business rules for each and every case. The problem with this approach was the Category and Rule keeps on changing, so by the time the Analysis is documented and given to development team the rule itself might have changed. Still let’s assume this will not happen so the code that will come out will be something like this:

Now its clear that the values a_multiplier_1 ,v_1, v_2 will be changed  for the sake of business rules frequently and code will have lot of if and else - Such code is error prone and hard to maintain , patchy stuff and don't forget the business problem of collecting these requirement

Solution:  

Well one way to solve this as above -With python we can solve this a bit differently by using eval(expression) built in function.  But first as defined in Python Docs: 

eval : The expression argument is parsed and evaluated as a Python expression (technically speaking, a condition list) using the globals and locals dictionaries as global and local namespace

>>> x = 1
>>> print eval('x+1')
2

Steps below uses the logic above to accomplish the solution for the problem

• Lets hand over the power of writing & changing the business rules to Analysts team, let them update the values using forms - using Django its a trivial job.
• The form will allow the Business Analysts to create variables and assign them values 
• These variables will be of two type - Constants & Expression as shown below

      

• Then allow user to write expression like  -

Variable_1	100
Variable_2	200
Variable_3	Variable_1 + Variable_2
Variable_4	IF Variable_3 > 1000 RETURN VARIABLE_3* VARIABLE_1 END IF Variable_3 < 1000 RETURN VARIABLE_3* VARIABLE_2 END

• Based on the order of the calculations  write the code that stores these variables and passes the variables of type expression ( Variable_3 & Variable_4) to the eval function to be processed as Python expression. 
• Lets think of Logic to Calculate Variable_4 ,assuming the function parsing the variables has already stored Variable_1   , Variable_2 and Variable_3 , say in a dictionary with Key as variable name and value as calculated values                                        

• Return all the values calculated as JSON , which will get consumed by any other function as Key Value pair.  

And with this we are good to go - writing code to write no code ! 

General: Many people will like to call it as Domain Specific Language where in Domain Related Parser can  be written in English Language as shown above and the code can parse it for domain specific purpose. It makes one think how will languages like Java and C# handle these dynamic expressions. That might be a  topic for some other day I guess. 

Setting up VIM for Python

Background : I thought I should try Python on Linux for development  of Python App.  Having only introductory level knowledge of Linux makes it quite difficult to set up the environment for development in Linux.  The reason I plan to use VIM for development is because that's readily available - ctrl+alt+T and vim will get me vim . And using VIM is cool :) . Although I have a decent background in programming and IDEs - read Visual Studio right from 2003 version of it  and I am quite a big fan of Visual Studio as IDE, setting up vim for python was difficult

Problem : Setting up vim for python development.

Solution :

• Vim gets installed in /etc/vim directory for all the system user and there there you have on vimrc file. vimrc file contains the configuration for the vim setup
• A .vimrc file is created in user home directory to provide for vim configuration setting and as a dot appended file is hidden file, one will not see it like other files on typing $ls on home directory even if it exists
• So for a raw set up we will not find a .vimrc file in home directory of the user.
• To override the behaviour of vimrc file one need to create a .vimrc file in its home directory and anything you write here will be used as setting for the vim editor for the logged in user. To create a .vimrc file:  vim .vimrc in your home directory will be sufficient. Anything we write in this file will be used as vim settings
• Now get some .vimrc settings  for python development and the link that we will use is this well written blog explaining the plugins and setting. The link itself is sufficient for us to go up an running , it took me sometime to get the .vimrc thing correct.  Follow the instruction mentioned there.
• Remove   filetype off  from .vimrc file
• Might have to also use call pathogen#incubate() instead of call Pathogen#runtime_append_all_bundles() as in my case it was throwing an error with the message to do above replacement , might be any upgrade to the versions
• Peace of advice don't just jump points on the link above and open vim now and then . Follow through every point  and once all the points are covered start using vim - we will have our VIM set for fun. Got this link a month back but git and so many commands on the blog made it look very complex so I left it
• And now after a month when I had some cool had installed everything in 20 minutes I guess.
• Also read here  for a good overview of linux file structure as and when time permits 

And with this we are good to go ! 

                                                          

 

List Comprehension Use Case

Problem : 

We have a list of  key , value pair and we just need to sum up the value part of the list. One way in languages like C# or Java was to loop through the whole loop and sum up the value part of the KeyValue object.

Solution:

    class KeyValue():
   
    def __init__(self,key,value):
    self.key = key
        self.value = value
        def __unicode__(self):
        return self.key+"~"self.value
So typical C# or Java code for the key value object mentioned above will look like -

    def sum_up_list():
    # Lets create List
    key_value_list = []
    key_value_list.append(KeyValue('V1',1))
    key_value_list.append(KeyValue('V2',2))
    # What if to the sum the list
    sum_of_value = sum([x.value for x in key_value_list])  ----------One line code

       foreach obj in key_value_list {sum = sum + obj.value }; -----Same effect using loop in C#

And we are good to go

Make dictionary out of two lists

Problem : Now that we have two lists - One list of keys and other list of values and we want to make dictionary out of this. How to do this using Python

Solution : Use zip command to merge the two lists. If the list count is not equal the zip command will truncate the longer one. So I will try to put a length check before merging.

So if keyList and valueList are the two lists we need to merge and make dictionary , lets make one :


newDict ={}
if len(keyList) == len(valList):
    newDict = dict(zip(keyList,valList))

And we are good to go :)

Generating a Time Interval Based String

Problem :  Need to generate time divided string for time series representation of data. The data can be in 1 minute interval or can be 15 minute interval or can be 1 hour interval. 

Sample Output: 

• if  timeInterval in 1 min : '01/01/2001 00:00~01/01/2001 00:01'............23:58,23:59,24:00
• if  timeInterval in 15 min : '01/01/2001 00:00~01/01/2001 00:15'.......,
• if  timeInterval in 30 min : 01/01/2001 00:00~01/01/2001 00:30'...........

Solution:

Lets think , we need to travel through the day based on time interval - can be of 1 min , 15 min , 30 min , 60 min till the end of day. 

So start condition is : 'startDate 00:00'

timeInterval : deltaT 

end Condition when the day gets over so = startDate+ 1 day

Below is the python code that does just that :

from datetime import timedelta,datetime   timeGrain = 15 # Can be 30 or so   deltaT= timedelta(minutes=int(timeGrain))  startDate = datetime.datetime.strptime(inputDateTime,'%Y-%m-%d %H:%M:%S') # Convert  String to Date time  endDate= (startDate+timedelta(days=1))  # End Condition  while startDate != endDate:     var = startDate     nextDateTime = startDate+ deltaT     keyList.append(str(var)+'~'+str(nextDateTime))     startDate =nextDateTime

And the output : ['2013-03-01 00:00:00~2013-03-01 00:15:00', '2013-03-01 00:15:00~2013-03-01 00:3 0:00', '2013-03-01 00:30:00~2013-03-01 00:45:00', '2013-03-01 00:45:00~2013-03-0 1 01:00:00', '2013-03-01 01:00:00~2013-03-01 01:15:00', '2013-03-01 01:15:00~201 3-03-01 01:30:00', '2013-03-01 01:30:00~2013-03-01 01:45:00', '2013-03-01 01:45: 00~2013-03-01 02:00:00', '2013-03-01 02:00:00~2013-03-01 02:15:00', '2013-03-01 02:15:00~2013-03-01 02:30:00', '2013-03-01 02:30:00~2013-03-01 02:45:00', '2013- 03-01 02:45:00~2013-03-01 03:00:00', '2013-03-01 03:00:00~2013-03-01 03:15:00']

So we are good to go with timedelta object one can iterate over seconds , minutes and days

Community for Development

Well ,

I am really impressed by various community tools present for any Open Source Developer . Will I be wrong in saying .... One Open stack developer knows and learns more that one Enterprise ( Read .Net  Microsoft Platform ) and can use lot of code and best practices and need not to write lot of stuff for things that are already done and made and perfected  , but in this case the open source . ( Read Linux OS on Apache Web Server ) end up having issues with integration , getting things working and moving , downloading and installing packages after packages to make the stuff moving , Unlike Microsoft's one click and done process ! ...Still to have a opinion on this.