Wednesday, February 6, 2013

INSTALLING SCRAPY ON WINDOWS 8 (Web Crawling)

Scrapy is framework to crawl website and extract structured data from their web pages. To install scrapy on Win 8 their are some dependencies that needs to be installed first. This post will list them accordingly. Keep in mind that i have python 2.7 installed on my system and win 8. So if you have a different version of python select packages accordingly

After installing Python, follow these steps before installing Scrapy:
Add the C:\python27\Scripts and C:\python27 folders to the system path by adding those directories to the PATH environment variable. You can do this by control panel > system and security > system > advanced system settings and scroll down to PATH variable, click edit and add the path to the end after adding a semicolon ( ; )

  1. Install OpenSSL by following these steps:
  2. Go to Win32 OpenSSL page


  1. Download Visual C++ 2008 redistributables for your Windows and architecture
  2. Download OpenSSL for your Windows and architecture (the regular version, not the light one)
  3. Add the c:\openssl-win32\bin (or similar) directory to your PATH, the same way you added python27 in the first step`` in the first step
  4. easy install for python in windows. http://pypi.python.org/pypi/setuptools
Use setuptools for installing lxml and zope-interaface. Just got command line, go into directory where packages are placed and then type easy_install "package_name.egg". This will install the file into the windows.

 Some binary packages that Scrapy depends on (like Twisted, lxml and pyOpenSSL) require a compiler available to install, and fail if you don’t have Visual Studio installed. You can find Windows installers for those in the following links.




And its done. Now follow the scrapy tutorial and make spiders to crawl the websites. 
Happy Crawling

Tuesday, February 5, 2013

MULTIPLICATION AND DIVISION IN DJANGO TEMPLATES

Django only provides with the built-in add tag. It means if you have to subtract something you have to use the same add tag. Generally it is recommended that you should do maths operations in the views. but if you want to do it in templates then there are two ways
  1.  Write you own template tag that should receive arguments and return results. That is also easy you have to have a template tag directory in you apps and settings file. To find more on that follow this link.
  2. The other is to follow this post. It will let you know how to multiply and divide without creating your own template tags.
 Lets start with a simple add example.
{{ list.item |add:"10" }}
will return the result by adding item and 100
item + 10 is equivalent to {{ list.item |add:"10" }}
If you want to subtract then 
{{ list.item |add:"-10" }}
will return the result by adding -100 to item
item - 10 is equivalent to {{ list.item |add:"-10" }} 
Now comes the tricky part,  if you want to multiply or divide then???

Its easy. You need to take a look at the widthratio tag. It takes 3 arguments. Now you have to play around with this tag.
Normally it is like
{% widthratio X Y Z %}
How it is evaluated?
         It is evaluated like X/Y*Z. Now if you want to Multiply supply Y as 1 so it would be X/1 * Z that evaluates to X*Z. Similarly if you want to divide than supply Z as 1 it would evaluate to X/Y * 1 which has no effect on the result.
In short
To Compute A*B:
{% widthratio X 1 Z %}
To Compute A/B:

{% widthratio A B 1 %}
In templates you usually have lists upon which you are iterating or any other data structure. so you need to replace the variable with you variables or items and example of that would be
{% for list in temp_list %}
        {% widthratio list.price 1 list.books %} 
{% endfor %} 
This will multiply the price with the books by iterating over this list.

To demonstrate the combined use use of add and multiplication the example is demonstrated below:
compute (P+Q) * (R+S): {% widthratio P|add:Q 1 R|add:S %}









Monday, February 4, 2013

ITERATING OR DISPLAYING DICTIONARY ITEMS IN DJANGO TEMPLATES

Iterating over a dictionary in python templates is what we usually come across. In this section i have described how to iterate a dictionary and also what happens if its values are a tuple.

{% for key, val in dict.items %}
    <td>{{ key }}</td>
    <td>{{ val }}</td>
{%  endfor  %}

If the dictionary value contains a Tuple: 
For example if your dictionary is like

{'title' : ('book','play'), title2' : ('book2','play2')}

Then play with the indices of tuples
for instance the book, book2 and play,play2 are at indices 0 and 1 respectively, so we can access their values in templates like described under

 {% for key, val in dict.items %}
    <td>{{ key }}</td>
    <td>{{ val.0 }}</td>
    <td>{{ val.1 }}</td>
{%  endfor  %}
This will display the key and against the key the values of tuples

Now if you want to match a key by converting it in lowercase so use built-in template tag for this purpose
if you have a template variable name then it can be converted to lowercase by using lowerfilter
{{ name | lower }}
There are many built-in templates for django which can be found at
https://docs.djangoproject.com/en/dev/ref/templates/builtins/?from=olddocs
You can also make your custom template tags. I will be publishing a post in near future guiding you about making the custom template tags