Friday, December 26, 2014

How to make Ajax calls from jquery client side to django view

Problem - We need to make Ajax call from client side browser to the server posting some data. Instead of posting the whole form to the server we need to use ajax to post only the required data back to the server. The problem arises in how to provide URL parameter for the post call which includes the server root and the app route as well.

Solution - 

  • Let the form include the csrf token as provided by django framework by placing {% csrf_token %}
  • The javascript function that will make the required call at :

          var host = document.location.host; // host + port
          var protocol = document.location.protocol;
          var sr_origin = '//' + host;
          var origin = protocol + sr_origin;
          $.ajax({
                         type:"POST",
                         url :origin+"/appName/post_view",
                         data:{
                                     "DATA_X": x,
                                     "DATA_Y" : y,
                                      csrfmiddlewaretoken:$("input[name=csrfmiddlewaretoken]").val() 
                                   },
            datatype:"json",
            error:function(data){ alert('Error:');},
            success:function(data){ alert("Success");}

          });
  • On the view side we need to provide a map for the URL : /appName/post_view mentioned above as : url (r'^post_view$',views.post_view,name='post_view') in the url.py file located within the appName folder [appName is the name of django app from which we want to make the call to view function]
Once this mapping is done we are good to make our ajax call ! 


Tuesday, December 9, 2014

Get Last N Recent Django Model Instances

Problem:
We have our good old Django employee application and we want to show 5 recent joiners in the team on a Django template.

Solution:

Lets take the case of Employee instance having a created_date as one of the attribute that will define the datetime when the employee entered the system. We will follow the following approach:

  1. Get the queryset of django instance sorted in decending by created_date attribute
  2. Pull out top 5 or n from this list
  3. Please note the method in the sis a static method as it is not dependent on the instance but the Class itself
Code Sample:



 



Thursday, November 27, 2014

Deleting a Foreign Key Model Instance

Problem:

Consider a scene where you have a django model with a Foreign Key reference to another Model Instance.  Now when you try and update the Foreign Key Instance , one need to remove the already existing instance. While trying to do that django also delete the model instance hosting the Foreign Key reference. So how to delete the Foreign Key Reference object without deleting the host instance.


Solution:

The solution is to update the model with on_delete attribute  set as models.SET_NULL. Django by default assumes cascading behavior while deleting any model object.

Not knowing this costed me some precious time to figure this out.




Tuesday, November 18, 2014

How to get GIS coordinates from the address using Google Geo Location API

Problem : One of the requirement to enter data for client on boarding might be to enter the location data for the customer. The location data can then be used to generate specific insights based on client locations.The client form consists of typical form fields like - Name,Age,Email and Address. The Address as you might have seen is various applications is divided into 3-4 fields like

Address:
  • Address Line 1 
  • Locality 
  • City 
  • State 
  • Country
  • Zip Code
The problem now is to use this address to get latitude & longitude details of the address using the Google Map API. Once we have latitude & longitude data we can plot this point on map and make the analytics more interactive.

Solution:

  • Parse the form to get the values out from the form field - Lets store them in variables as: addressLine,locality,city,state,country, zip
  • Create a location string by adding all the above variables separated by  "+" sign as:
       loc = addressLine+ '+' + locality + '+' + city + '+' + country + '+' + zip
       (above expression assumes that we have converted all the variable to string)
  • Declare a variable to point to geo location Google API and a dictionary to pass the encode location arguments
        GEOLOC_URL = 'http://maps.googleapis.com/maps/api/geocode/json'
        GEO_ARGS = {}
        GEO_ARGS.update({'address': loc })
        Geolocation API needed 'sensor': "false"/ "true" attribute in earlier versions
        indicating the use of sensors to collect data
  • Appended the GEO_ARGS to GEOLOC_URL to create new URL
        url = GEOLOC_URL + '?' + urllib.urlencode(GEO_ARGS)
  • Once the url is encoded we now need to send the request for which we will import urllib and get the result back which is a dictionary with all the details
        result = json.load(urllib.urlopen(url))
        ( import urllib & json before calling the above line of code )
  • If everything went well one can check the status using result['status'] which will be OK
  • Now our Lat / Long Values be 
  • latitude =  result['results'][0]['geometry']['location']['lat']
  • longitude =  result['results'][0]['geometry']['location']['lng']
  • If the API is able to find the location accurately it returns only one results element and if the results are ambiguos it returns a list of probable elements so checking this might be important while parsing the JSON
  • The code below shows summarizes what we discussed above
Now we can store these values in our database to be retrieved later to be plotted on Google Maps

Sunday, November 2, 2014

Use C# .Net to Load & Analyze Excel Sheet Data

Problem : This is something that repeatedly keeps on coming and I am putting it for my reference here. This snippet is of good use for anyone wanting to do data analysis on Excel Data Sheet by LINQ or by SQL like queries.

Solution : Following things to be noted in the code below:

  1. On windows you need to provide path with double backward slash (\\)
  2. The file that you are trying to open for analysis - As the connection string changes for the xls and xlsx file
  3. The name of the worksheet should end with $ sign
  4. Now once you have data table loaded in the memory one can start using Rows and Columns set to analyze data via C# Code


Best  case will be to analyze Stock Market data  through C# code as most of the Open Data is available in csv / excel format , but I will take that up some other time.


Saturday, October 4, 2014

Django ORM - Composite Key Problem - Special Case Solution

Problem: Recently had an assignment to pull data out from client database and expose the data tables as dashboards to be previewed by the clients. So the solution was to expose the tables as Django models via admin interface, simple !  The problem was that the client had a legacy Oracle database with read only access to us, so any changes to table like adding one ID column was ruled out. Initial analysis of database showed that tables were simple flat data files with no foreign key relations but having composite key as Primary Key for each of the table. Therefore Django ORM could not map objects - one to one , meaning thereby objects.get(id =pk) will throw an exception. And hence django admin will not work as easily as I thought.

Solution: Django does not support composite key by default for the reason as mentioned in documentation but my case was - I wanted to have only a read only view , with not even selecting a  single object to edit or delete or update. For the case in hand I only wanted the django's select all view - default dashboard where all the objects of particular type gets listed ( rows of the database table ).  So here is the solution
  1. So made any one field as Primary key - irrespective of what the field is, as in case of django primary key is needed for selecting one object. 
  2. Remove the hyperlink for selecting the single django object 
  3. Remove the default add button that comes with django admin to avoid inserts in the database by a primary key
  4. Remove the Action tool bar to remove the delete action - In case if you want to have some custom action like -Export Data functionality on your read only view , you have to rely on raw django queries and not the ORM
  5. The whole attempt is to not to let user select one particular record / object from the dashboard and hence avoiding the need to django ORM to call objects.get(id=pk)
Sample Code Snippet:


Summary: The above technique is only applicable for cases where no direct selection of object is required via django ORM. This is quite a bit of constrain for anyone thinking of using django for legacy database with default ORM. Thinking of evaluating other ORMs and how well they integrate with django ecosystem.  Suggestions ?  SQLAlchemy ??

But for now we are good to go ! 

Friday, March 21, 2014

File Location of Running Python Script

Problem : During development phase I was saving files with file.open("filename"), for this the python interpretor tries and search the file in the directory where the script is located.  For the development part this was good but once deployed to production and where  corntab script was suppose to execute it , the interpretor was not looking for these files in the directory of the script resulting in code break.

Solution: Instead of absolute file names or path provide path relative to the script or python file that is running


  1. Get Path of the directory where current python script is running
  2. import os
  3. filepath  = os.path.dirname(os.path.realpath(__file__)) + "/"+"filename"
  4. Now the file path will have the file you want to open
  5. file = open(filepath)
And with this we are good to go !


Timezone Conversion in Python

Problem : We have an application where timezone is not saved and only timestamp is stored.  Problem arises when one tries to convert UTC timezone to local timezone and then converting it to time stamp . To be stored in Timezone Agnostic database.

Solution :

  1. import iso8601 module
  2. timestamp_iso8601_string = '2014-03-18T04:19:05-00:00'
  3. tz_aware_date = iso8601.parse_date(timestamp_iso8601_string)
  4. import pytz
  5. ist_timezone =pytz.timezone('YOUR TIMEZONE')
  6. your_localtimewithout_tz = tz_aware_date.astimezone(ist_timezone).replace(tzinfo=None)
  7. your_localtimewithout_str = str(your_localtimewithout_tz)
This will do the required conversion. And we are good to go.

Thursday, February 27, 2014

Upgrading Postgres

Problem : This took me quite a while to figure. I had Postgresql version 9.2 on my machine & I wanted to upgrade it to latest 9.3.  Tried most of the links here and there on internet but was getting wierd errors - mostly because of the configuration issues. Most of the Postgres documentation is good and revolves around using pg_upgrade and can be used actually as is

Solution :

Step 1 - sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2 \
              postgresql-9.3 postgresql-contrib-9.3 postgresql-server-dev-9.3

Note - I skipped this one and got few weird errors. So install the contribs for both the old and new version we are upgrading from 9.2 to 9.3

Step 2 - sudo /etc/init.d/postgresql stop 

Note - This will stop the Postgres instances running in the cluster

Step 3 - cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.3/bin/pg_upgrade \  -b /usr/lib/postgresql/9.2/bin \
   -B /usr/lib/postgresql/9.3/bin \   -d /var/lib/postgresql/9.2/main \    -D /var/lib/postgresql/9.3/main \
   -o ' -c config_file=/etc/postgresql/9.2/main/postgresql.conf' \    -O ' -c config_file=/etc/postgresql/9.3/main/postgresql.conf'

Note - Do not forget $ cd /tmp here otherwise the system gives error as "cannot write to log file pg_upgrade_internal.log"    


Step 4 - Now after step 3 one might get error as

Performing Consistency Checks
-----------------------------
Checking current, bin, and data directories                 ok

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.
Failure, exiting

To overcome this one stop the postmaster service that get started via init.d  try the following :

sudo /etc/init.d/postgresql stop

Final Note - The upgraded server will be running on port 5433 and not the default 5432. When connecting via pgadmin, make sure we put in the correct port

After this the upgrade should go smoothly as it happened in my case.


Reference : http://wiki.postgresql.org/wiki/Using_pg_upgrade_on_Ubuntu/Debian