What's new in Django community blogs?

Continuously rebuild your project

[Archived Version] □ Published at Writing on django | David Winterbottom

New developers joining a project will often find that the project won't build cleanly on their machine, and hours of time will be sunk into setting up the project so work can start. This is sad and expensive for all concerned.

This is a particular menace in agencies (or anywhere with lots of small projects) where a large team of developers need to jump between projects. Tools like Vagrant and Docker can help but aren't the panacea they first seem to be [*].

Counter this by using continuous integration to build your project from scratch. Then any changes that break the build process (such as database schema changes not applying correctly) will be spotted early. New team members will ...


Get Data From WMS Layers Using GDAL and Python

[Archived Version] □ Published at Ivan Pasic Blog posts

Hello everyone, I haven't been writing for quite some time because I've been very busy last months but from now on I'll try to write as often as I can.

Today I'd like to talk a little bit more about GDAL library, WMS service and how to extract different type of images from WMS layers. 
Before you keep reading please let me tell you that ever since I started learning more about GIS I've always prefered vector data before raster and I've honestly never used GDAL commands before.
However, now I came up with some project idea where I should find a way to handle some orthophoto imagery and do some raster image processing.
That's why I decided to learn more about it and to share with you some things that I've learned while playing with it.

Let's first give short explanation of terms that will be used here.

GDAL (Geospatial Data Abstraction Library) is a library for reading and writing raster geospatial data formats.                              It's released by the OSGEO (Open Source Geospatial Foundaiton) under X/MIT type of Open Source license. OGR library is part of the GDAL source tree and it provides similair capability for vector data. You can read more about GDAL/OGR on its official site.

If you want to try by yourself examples described in this post then you should first install GDAL. You are free to choose the way to do it but if you are to lazy to figure out how to do it then this should do the job for you:

cd /usr/local/src
wget http://download.osgeo.org/gdal/gdal-1.9.0.tar.gz
tar xvfz gdal-1.9.2.tar.gz
cd gdal-1.9.0
./configure --with-python
make
sudo make install

WMS (Web Map Service) is OGC standard for serving georeferenced map images over the Internet. There are many WMS Servers that can serve WMS layers but probably the most often used are Geoserver and Mapserver.

WMS requests support this operations:
GetCapabilities - retrieves metadata about the service (e.g. map image format, WMS version, map bounding box, CORS, ..)
GetMap - retrieves map image for defined area and content
GetFeatureInfo (optional) - retrieves geometry and attribute data for specific location on map
GetLegendGraphic (optional) - returns map legend 
DescribeLayer (optional) 

To help you get better idea of one simple GetMap request we can try it on some real world example. For this purposes I will use WMS layer which contains DOF (Digital Orthopohoto) imagery of my country (Croatia). 

http://ganimed.geoportal.dgu.hr/cwms?service=WMS&request=GetMap&version=1.1.1&layers=DOF&srs=EPSG:3765&format=image/jpeg&width=256&height=256&bbox=399609,4887306,400326,4888023

If you just copy and paste this url in your browser you should see small image (256x256 pixels) in left upper corner of the screen. It's response for your GetMap request (if you don't see any image then it's probably because that server is not in function anymore, in that case please write a comment below this post so I can change it).

If you take a closer look at the defined url you can see that we set different parameters there. Here is short description for each of them.
service - service name
request - operation name
version - service version
layers - layers to display
srs - Spatial Reference System for map output
format - format for the map output
width - width of map output (in pixels)
height - height of map output (in pixels)
bbox - bounding box for map extent (minx,miny,maxx,maxy)

Those are just some of parameters you can define for WMS GetMap request but for purpose of this example there is no need for additional parameters. 

So, as you can see, we defined set of parameters to get jpeg image of specific area within some WMS layer. Although this can often be very useful and interesting sometimes we need better way to access some WMS service and its data. 
For example, fif you need to save response images then you probably don't want to do it manually by typing url in your browser every time and then saving it. 

As you can assume it can easily be done with Python.

One way for it is to use urllib library and then directly call a WMS and write the response out to a file. However, I'd like to present you another approach this time which provides you much more possibilites.

Accessing web image services using OWSLib

If you haven't found it already, there is great library called OWSLib.
As it says in its description - "OWSLib is a Python package for client programming with Open Geospatial Consortium (OGC) web service (hence OWS) interface standards, and their related content models."
It can be used for some other OGC services too but in this example we will use it just for working with WMS service.

Here is simple code where you can see how to do previous GetMap request using OWSLib library. 

from owslib.wms import WebMapService

wms = WebMapService('http://ganimed.geoportal.dgu.hr/cwms', version='1.1.1')

img = wms.getmap(layers=['DOF'],
        srs = 'EPSG:3765',
        bbox = (399609, 4887306, 400326, 4888023),
        size = (256,256),
        format = 'image/jpeg',
        transparent = True
    )

out = open('dof_img.jpg', 'wb')
out.write(img.read())
out.close() 

This can be very useful when you develop some web applications and need to make different WMS requests based on your user inputs (e.g. bounding box) and similair. 
However, please visit OWSLib official documentation to get better idea of what you can all do with this library.

Accessing web image services using WMS in GDAL

Beside previously described approach, accessing several different types of web image services is also possible using the WMS format in GDAL.
To do that, you first need to create local service description XML file. 
For example, if you want to get same image as before (when we used Python) then your XML file should look somehow like this:

gdal_wms_example.xml

<GDAL_WMS>
    <Service name="WMS">
        <Version>1.1.1</Version>
        <ServerUrl>http://ganimed.geoportal.dgu.hr/cwms</ServerUrl>
        <SRS>EPSG:3765</SRS>
        <CRS>CRS:HTRS96</CRS> 
        <ImageFormat>image/jpeg</ImageFormat>
        <Layers>DOF</Layers>
    </Service>
    <DataWindow>
        <UpperLeftX>399609</UpperLeftX>
        <UpperLeftY>4887306</UpperLeftY>
        <LowerRightX>400326,</LowerRightX>
        <LowerRightY>4888023</LowerRightY>
        <TileLevel>19</TileLevel>
    <TileCountX>1</TileCountX>
    <TileCountY>1</TileCountY>
    </DataWindow>
    <Projection>EPSG:3765</Projection>
    <BlockSizeX>256</BlockSizeX>
    <BlockSizeY>256</BlockSizeY>
    <BandsCount>3</BandsCount>
</GDAL_WMS>

Now when you have XML file created you can run GDAL commands on it. 
For the beginning let's try to get information about our file:

gdalinfo path/to/gdal_wms_example.xml

You should see output with set of different informations (e.g. driver, size, coordinate system, ..)

However, we are probably more interesting in doing something more specific with our file.
Here you can find all utility programs that are distributed with GDAL (http://www.gdal.org/gdal_utilities.html)
Probably the most often used command is gdal_translate. We will use it here to get image output (.jpeg file) of our defined area based on previously created XML file. 

gdal_translate -of JPEG -outsize 256 256 gdal_wms_dgu.xml dof_img2.jpg

As you can probably guess this command gives us JPEG image output (256x256 pixels) based on our XML file.
If you compare it to our dgu_dof.jpg image that we got by running our python script then you should see that they are the same.
Please note that you need to explicity define image output because by default it is set to GeoTIFF (GTiff) so instead of .jpg file you would get .tif file.
Actually, let's try to do that because if you are working with map images and GIS applications then you would probably need GeoTIFF images before jpeg. 

gdal_translate -of GTiff -outsize 256 256 gdal_wms_dgu.xml dof_geotiff.tif

Now if you want you can get more info about our output image by running:

gdalinfo dof_geotiff.tif

To find out more possible operations on raster data (like resampling and rescaling pixels, setting ground control points on output images,...) please visit http://www.gdal.org/gdal_translate.html

At the end, I'd just like to mention one more thing about running GDAL commands within Python.
First, it's important to mention that you can use GDAL within Python if you install it as Python Package https://pypi.python.org/pypi/GDAL/.
Honestly as I've just started exploring this raster image processing with GDAL I didn't have time to test how to use the Python GDAL/OGR API but I definitely willl and maybe I'll write new post then.
However, I'd like to remember you that Python has subprocess module which can be used for accessing system commands.

So if you want to run previous GDAL command within python script you can always do it like this:

import subprocess

subprocess.call(["gdal_translate", "-of", "GTiff", "-outsize", "250", "250", "gdal_wms_dgu.xml", "dof_img2.tif"])


That would be all for this post, I hope some of you will find it useful or at least it will help you get an idea for some of you future tasks and projects. I will continue studying this field and I hope I will write more about it.

Feel free to comment and ask anything you want below.


Query and Filter Leaflet Map Layers

[Archived Version] □ Published at Ivan Pasic Blog posts

Most of the web maps today are developed using same philosophy. They have different type of data organized in layers to achieve better and easier visualization of those data. Using simple controls they allow users to control what layers they want to see on map.

However, although it's very good way to provide better and easier visualization of map data it's often not enough.

Sometimes we want to query each of those layers and filter data within each layer to get only results we are interested in.
For example, if you have layer cities maybe you would like to display only cities that are larger then 1000m2 or that have more then 100 000 citizens and so on.

In this tutorial I will try to explain you how to filter map data based on their attributes.
To make this more easier to understand we'll use some real world example. So, let's say that in our DB we have table ApartmentsNY with those fields:
- name
- city
- price
- wifi
- breakfast

We'd probably like to allow user to choose which apartments they want to see on map based on some of those attributes. 
For example, someone could probably just want to see apartments that are located in NYC and whose price is not over 100$ per night. Then a user could be even more specific and say that she wants only apartments that have wifi network. 

So now when we know some possible use case let's try to implement it. 
For the purpose of this example I will use technologies I've already been writing about - LeafletJS, PostGIS, (Geo)Django. 
Although I'd really like to explain all possible steps here I just can't because it takes to much time. So I assume that you already know how to set up a Django project and create simple map using LeafletJS. If you don't then please check some basic LeafletJS tutorials and read one of my previous articles http://ipasic.com/article/let-user-add-point-map-geodjango-leaflet/ to figure out how to set up Django project with PostGIS database.
You can also find complete source code of this simple application on my github repository .

So let's first create model ApartmentsNY

models.py

from django.contrib.gis.db import models
from django.contrib.gis import admin
from django.utils import timezone
from autoslug import AutoSlugField
from easy_thumbnails.fields import ThumbnailerImageField

def image_upload_folder(instance, filename):
    return "apartments_images/%s" % (filename)

choices_city = (        
    ('New York City', 'NYC'),
    ('Syracuse', 'Syracuse'),
    ('Buffalo', 'Buffalo'),
    ('Rochester', 'Rochester'),
    ('Yonkers', 'Yonkers')
)

class ApartmentsNY(models.Model):
    name = models.CharField("Name of the apartment", max_length=50)
    slug = AutoSlugField(populate_from='name', unique=True) 
    city = models.CharField("City", max_length=70, choices=choices_city)
    price = models.IntegerField("Price per night [$]")
    wifi = models.BooleanField("WiFi", default=False)
    breakfast = models.BooleanField("Breakfast", default=False)
    image = ThumbnailerImageField(upload_to=image_upload_folder, blank=True)

    geom = models.PointField(srid=4326)

    def __unicode__(self):
        return self.name

    class Meta:
        verbose_name_plural = "Apartments in NY"

admin.site.register(ApartmentsNY, admin.OSMGeoAdmin)


After you've created model ApartmentsNY go to your admin interface and add few test data. I've added 6 apartments for purpose of this example (you can add as many as you want). 

Now when we have added few apartments let's create simple map where we will first display them all.

**Note: I won't explain each part of code here but if you have problems with understanding it then please ask for help in comments or send me an email. Also, please visit github repo to see complete code and getter better understanding of it.

map.js

/* Define base layers */
var cycleURL='http://{s}.tile.thunderforest.com/cycle/{z}/{x}/{y}.png';
var cycleAttrib='Map data © OpenStreetMap contributors';
var opencyclemap = new L.TileLayer(cycleURL, {attribution: cycleAttrib}); 

var osmUrl='http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png';
var osmAttrib='Map data © openstreetmap contributors';
var osm = new L.TileLayer(osmUrl, {attribution: osmAttrib}); 

/* create new layer group */
var layer_apartments = new L.LayerGroup();
var array_markers = new Array();

/* create custom marker which will represent apartments in layer 'layer_apartments' */
customMarker = L.Marker.extend({
   options: { 
      title: 'Name of the apartment',
   }
});

/* define function which adds markers from array to layer group */
function AddPointsToLayer() {
    for (var i=0; i<array_markers.length; i++) {
        array_markers[i].addTo(layer_apartments);
    }
} 

/* Get all apartments from DB and add them to layer:_apartments */
$.ajax({
    url: '/map/get-apartments/',
    type: 'GET',
    success: function(response) {
        $.each(eval(response), function(key, val) {      
            //fields in JSON that was returned          
            var fields = val.fields; 

            // parse point field to get values of latitude and longitued
            var regExp = /\(([^)]+)\)/;
            var matches = regExp.exec(fields.geom);
            var point = matches[1];
            var lon=point.split(' ')[0];
            var lat=point.split(' ')[1];

            //function which creates and adds new markers based on filtered values
            marker = new customMarker([lat, lon], {
                title: fields.name,
                opacity: 1.0  
            }); 
            marker.bindPopup("<strong>Name: </strong>" + fields.name + "<br><strong>City: </strong>"
                + fields.city + "<br><strong>Price: </strong>"+ fields.price);
            marker.addTo(map);
            array_markers.push(marker);
        });

        // add markers to layer and add it to map
        AddPointsToLayer();
    }
});

/* create map object */
var map = L.map('map', {
    center: [41.75, -74.98],
    zoom: 7,
    fullscreenControl: true,
    fullscreenControlOptions: {
        position: 'topleft'
    },
    layers: [osm, layer_apartments]
});

var baseLayers = {
    "OpenCycleMap": opencyclemap,
    "OpenStreetMap": osm
};

var overlays = {
    "Apartments in NYC": layer_apartments
};

L.control.layers(baseLayers, overlays).addTo(map);


As you can see, I've added 2 base layers to map (OpenStreetMap and OpenCycleMap) and one overlay - layer_apartments which contains all apartments from our database. To get all apartments from our database we use AJAX call to send get request to our server. Received data is in JSON format so we first need to parse that data and then create map markers from it. Please have in mind that to make this work you first had to define view in you views.py which is responsible for serializing data and retrieveing it in JSON format.
I always use DjangoRestFramework when I need to implement some kind of REST API so if you want follow this example then please install it in your virtualenv (all dependencies for this app can be found in 'requirements.txt' on my github repo)

views.py

from rest_framework import status
from rest_framework.decorators import api_view
from rest_framework.response import Response
from django.core import serializers
from website.models import ApartmentsNY

@api_view(['GET'])
def get_apartments(request):
    result = ApartmentsNY.objects.all()
    data = serializers.serialize('json', result)
    return Response(data, status=status.HTTP_200_OK, content_type='application/json')


So, now when we have created our map (with all apartments displayed on it) let's try to implement simple map filter.

Although it doesn't matter how and where on your map you will create your 'div' element with select options for filtering, I really like Leaflet sidebar plugin . It's easy to add on your map and beside it, it has nice responsive design so it's really great to use. 

After you have added sidebar to your map and defined all select options within it let's see how filtering works.
It really doesn't make sense to copy and paste here all my code so please check it on github repo to get better idea how it works. I will show you only 2 most important things here. 
First, let's see function which is called each time when user filters map objects using sidebar filter.

/* getResult function is called every time when user filters map objects using sidebar filter */
function getResult() {
    // fetch value of all filter fields
    var selected_city = $("#select_city").val();
    var selected_price = $("#slider_price").data("value");
    var boolean_wifi = $("#select_wifi").val();
    var boolean_breakfast = $("#select_breakfast").val();

    // get fields where value is not 'all' so that you later filter only those fields
    var fields = new Array();

    if (selected_city !== 'all') {
        fields.push("city");
    }

    if (boolean_wifi !== 'all') {
        fields.push("wifi");
    }
    if (boolean_breakfast !== 'all') {
        fields.push("breakfast");
    }

    // price field doesn't have value 'all' so it will be filtered in any case
    fields.push("price");

    /* ajax call to get all apartments with defined filter values */
    $.ajax({
        url: '/map/apartments/filter/',
        type: 'GET',
        data: "city=" + selected_city + "&price=" + selected_price + "&wifi=" 
        + boolean_wifi + "&breakfast=" + boolean_breakfast+ "&fields=" + fields,
        success: function(response) {
            // first delete all markers from layer apartments
            array_markers.length=0;
                layer_apartments.clearLayers();

                   $.each(eval(response), function(key, val) {      
                    //fields in JSON that was returned          
                    var fields = val.fields; 

                    // parse point field to get values of latitude and longitued
                    var regExp = /\(([^)]+)\)/;
                    var matches = regExp.exec(fields.geom);
                    var point = matches[1];
                    var lon=point.split(' ')[0];
                    var lat=point.split(' ')[1];

                    //function which creates and adds new markers based on filtered values
                    marker = new customMarker([lat, lon], {
                    title: fields.name,
                    opacity: 1.0  
                }); 
                    marker.addTo(map);
                    array_markers.push(marker);
                });

                // add markers to layer and add it to map
                AddPointsToLayer();
            }
    });
}


So, the basic idea is to first get values of all fields and then make get request to server (using AJAX) to get only apartments filtered by those values. Then we simply delete all previous map markers and display again only the ones which we just received.

Here is view function which is used to filter model objects and retrieve them as a JSON back to the client.

views.py

from rest_framework import status
from rest_framework.decorators import api_view
from rest_framework.response import Response
from django.core import serializers
from website.models import ApartmentsNY

@api_view(['GET'])
def apartments_filter(request):
    request_data = request.QUERY_PARAMS
    filtered_fields = request_data['fields']

    kwargs = {}

    if "city" in filtered_fields:
        kwargs['city'] = request_data['city']
    if "price" in filtered_fields:
        price = request_data['price'] # e.g (150, 400) 
        price_values = price[1:][:-1].split(',')
        min_price = price_values[0]
        max_price = price_values[1]
        kwargs['price__range'] =  (min_price, max_price)
    if "wifi" in filtered_fields:
        kwargs['wifi'] = request_data['wifi']
    if "breakfast" in filtered_fields:
        kwargs['breakfast'] = request_data['breakfast']

    try:
        result = ApartmentsNY.objects.filter(**kwargs)
        data = serializers.serialize('json', result)
        return Response(data, status=status.HTTP_200_OK, content_type='application/json')
    except:
        return Response(status=status.HTTP_400_BAD_REQUEST)

As you can see, we used Python kwargs to dynamically build queries and filter our model objects. At the end we serialized data and returned it in response as JSON data.

One good thing about all this is that if you already use GeoDjango then you can do many spatial queries too using built-in filters. For example, you can geolocate user and then look for the closest apartment to his current location. However, that would probably be too much for this blog post, maybe I will write about it some other time.

I hope you understand the basic idea here. I really wished if I could go into more details and explain it somehow better but it takes too much time so if you have any questions about this please feel free to ask in the comment section below.
Also don't forget to check github repo with all source code. Feel free to clone the code and play with it.

 


Serve Map Tiles Using Tile Map Service (TMS) in Ol3

[Archived Version] □ Published at Ivan Pasic Blog posts

In this blog post I will try to explain you what is Tile Map Service (TMS) and how does it work. To get better idea of it I will show it how you can use it on some real example with OpenLayers3.

What is TMS?

On OpenStreetMap Wiki you can find this definition:
"TMS (Tile Map Service) is a protocol for serving maps as tiles i.e. splitting the map up into a pyramid of images at multiple zoom levels."

So, the basic idea is that you have many different images (tiles) which are loaded when they come into view. On smallest scale level there is only one tile and when you zoom in one level there are 4 tiles and so on. So for each area which is overlayed by an image there are 4 more higher resolution images which are displayed when user zoom in that area.
This approach is used by GoogleMaps, Bing and OpenStreetMap too.

                                                         
Image taken from: R.García, J.Pablo de Castro, E.Verdú, M.J.Verdú and L.M.Regueras: Web Map Tile Services for Spatial Data Infrastructures

Maybe you think "why should I use TMS when I already know how to serve my maps using WMS"Well although WMS is great way to server you map, in some cases it's just not the best solution.

Main advantage os TMS is that it's very fast and that's because tiles are pre-rendered on the server side. Often times this can reduce waiting time for data and your map is delivered to visitors much faster. Beside it, it doesn't require any extra software (like Geoserver or Mapserver), it provides better reliability and beside it, scaling will be much easier.
However, I'm not saying it's perfect solution and I don't advise you to use it for every use case. For example, disadvantage of TMS is that you will probably have large data to store and beside it, if your map data is often changing it would be harder to update it.

Anyway, enough talking, let's try to see how it works on our example.

TMS Layer Example

For this purpose we will use data from NaturalEarthData so please go to this link and download zip file with Natural Earth I data (large size). 
When you unzip it you should see 5 files within it but probably most important for us is NE1_HR_LC.tif file. It's 21,600x10,800 pixels image which we will use to create our map tiles.

Now, before you continue please first make sure that you have GDAL installed, otherwise you won't be able to follow this tutorial.
GDAL is our friend and you will probably use it most of the time when you work with raster data.

So, first let's get some info about our image with 'gdalinfo'

gdalinfo NE1_HR_LC.tif 

If everything is well you should see output with information about image size, coordinate system, pixel size, ...

As you can see coordinate system of this image is WGS84 (EPSG code is 4326).
Before we continue let's first try to reproject it to Spherical Mercator projection (EPSG code is 3857) which is used by most web mapping applications.
It can be done using GDAL command 'gdalwarp' which is used for image reprojection.

Important note! Gdalwarp sometimes creates output images which has much bigger size then the input images.

So, to avoid that we will gdalwarp to VRT first and then after that run gdal_translate with -co compress=lzw option

gdalwarp -t_srs EPSG:3857 -of vrt NE1_HR_LC.tif natural_earth.vrt
gdal_translate -co compress=LZW natural_earth.vrt natural_earth.tif

Now when we have our natural_earth.tif image in new coordinate system let's generate our map tiles from it.

To create our map tiles we will use GDAL2Tiles utility which generates directory with small tiles and metadata following the OSGeo Tile Map Service Specification.
If you don't like your command line so much then you can also do it using MapTiler application. It's very nice product developed by Petr Pridal from Klokan Technologies. It also uses gdal2tiles but it has nice GUI which makes you whole process easier. However, free version of MapTiler doesn't support tiling of large images and beside it, it has some other small disadvantages so please try to follow this example using command line tools.

So, let's try to create our directories with map tiles by running:

gdal2tiles.py natural_earth.tif 

*Please be patient, this process can last longer then you may expect (especially if size of the input image is very high)

If everything went well you should now see newly created directory with name natural_earth.

Now let's explain that directory structure.
There are 6 main directoris (0-5) where name of the directory is actually the number of the zoom level. In each of those folder there is one or more directories which name is column number of the tile. Inside each of that folder you can find one or more images which name is row number of that tile.
So, for example, tile that has this path natural_earth/5/4/3.jpg is the tile that is 4th from the left and 3rd from the bottom for zoom level 5

Important note!
Tiles which are generated using GDAL2Tiles are always created following TMS schema. However, if you use MapTiler to generate your map tiles then there is a difference because MapTiler creates XYZ/WMTS tiles instead of TMS tiles.
Please visit this link http://www.maptiler.org/google-maps-coordinates-tile-bounds-projection/ to get better idea of how those two types differ.
You will notice that number of column is same for both schemas but row number (y) is different. 
So the only difference between the two is a flipped y coordinate.
You can express that relation using this formula:  y = (2^z) - y - 1

Now when you now how your tiles are organized we can create simple ol3 map with TMS layer with Natural Earth data.

map.html

<!DOCTYPE html>
<html>
  <head>
    <title>TMS Layer example in ol3</title>
    <meta charset="utf-8"/>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no"/>
    <script src="http://ol3js.org/en/master/build/ol.js" type="text/javascript"></script>
    <link rel="stylesheet" href="http://ol3js.org/en/master/css/ol.css" type="text/css">
    <style>
      html, body, #map {width:100%; height:100%; margin:0; padding:0;}
    </style>
  </head>
  <body>
    <div id="map" class="map"></div>
    <script> 
      var extent = ol.proj.transform([-180,-85.051129,179.976804,85.051129],
                                     'EPSG:4326', 'EPSG:3857');
      var center = ol.proj.transform([-0.011598000000006436, 0],
                                     'EPSG:4326', 'EPSG:3857');
      var map = new ol.Map({
        layers: [
          new ol.layer.Tile({
            source: new ol.source.XYZ({
              tileUrlFunction: function(coordinate) {
          if(coordinate == null){
            return "";
          } 
          var z = coordinate.a;
          var x = coordinate.x;
          var y = (1 << z) - coordinate.y - 1;
          return 'natural_earth/' + z + '/' + x + '/' + y + '.png';
        },
              extent: extent,
              minZoom: 0,
              maxZoom: 5
            })
          })
        ],
        renderer: 'canvas',
        target: 'map',
        view: new ol.View({
          projection: 'EPSG:3857',
          center: center,
          zoom: 1
        })
      });
      map.getView().fitExtent(extent, map.getSize());
    </script>
  </body>
</html>


Django Blog Tutorial - the Next Generation - Part 8

Aug 31 2014 [Archived Version] □ Published at Matthew Daly's Blog

Hello again! In our final instalment, we’ll wrap up our blog by:

  • Implementing a sitemap
  • Optimising and tidying up the site
  • Creating a Fabric task for easier deployment

I’ll also cover development tools and practices that can make using Django easier. But first there’s a few housekeeping tasks that need doing…

Don’t forget to activate your virtualenv - you should know how to do this off by heart by now!

Upgrading Django

At the time of writing, Django 1.7 is due any day now, but it’s not out yet so I won’t cover it. The biggest change is the addition of a built-in migration system, but switching from South to this is well-documented. When Django 1.7 comes out, it shouldn’t be difficult to upgrade to it - because we have good test coverage, we shouldn’t have much trouble catching errors.

However, Django 1.6.6 was recently released, and we need to upgrade to it. Just enter the following command to upgrade:

pip install Django --upgrade

Then add it to your requirements.txt:

pip freeze > requirements.txt

Then commit your changes:

git add requirements.txt
git commit -m 'Upgraded Django version'

Implementing a sitemap

Creating a sitemap for your blog is a good idea - it can be submitted to search engines, so that they can easily find your content. With Django, it’s pretty straightforward too.

First, let’s create a test for our sitemap. Add the following code at the end of tests.py:

class SitemapTest(BaseAcceptanceTest):
    def test_sitemap(self):
        # Create a post
        post = PostFactory()

        # Create a flat page
        page = FlatPageFactory()

        # Get sitemap
        response = self.client.get('/sitemap.xml')
        self.assertEquals(response.status_code, 200)

        # Check post is present in sitemap
        self.assertTrue('my-first-post' in response.content)

        # Check page is present in sitemap
        self.assertTrue('/about/' in response.content)

Run it, and you should see the test fail:

$ python manage.py test blogengine
Creating test database for alias 'default'...
...........................F
======================================================================
FAIL: test_sitemap (blogengine.tests.SitemapTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 847, in test_sitemap
    self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200

----------------------------------------------------------------------
Ran 28 tests in 6.873s

FAILED (failures=1)
Destroying test database for alias 'default'...

Now, let’s implement our sitemap. The sitemap application comes with Django, and needs to be activated in your settings file, under INSTALLED_APPS:

    'django.contrib.sitemaps',

Next, let’s think about what content we want to include in the sitemap. We want to index our flat pages and our blog posts, so our sitemap should reflect that. Create a new file at blogengine/sitemap.py and enter the following text:

from django.contrib.sitemaps import Sitemap
from django.contrib.flatpages.models import FlatPage
from blogengine.models import Post

class PostSitemap(Sitemap):
    changefreq = "always"
    priority = 0.5

    def items(self):
        return Post.objects.all()

    def lastmod(self, obj):
        return obj.pub_date


class FlatpageSitemap(Sitemap):
    changefreq = "always"
    priority = 0.5

    def items(self):
        return FlatPage.objects.all()

We define two sitemaps, one for all the posts, and the other for all the flat pages. Note that this works in a very similar way to the syndication framework.

Next, we amend our URLs. Add the following text after the existing imports in your URL file:

from django.contrib.sitemaps.views import sitemap
from blogengine.sitemap import PostSitemap, FlatpageSitemap

# Define sitemaps
sitemaps = {
    'posts': PostSitemap,
    'pages': FlatpageSitemap
}

Then add the following after the existing routes:

    # Sitemap
    url(r'^sitemap\.xml$', sitemap, {'sitemaps': sitemaps},
            name='django.contrib.sitemaps.views.sitemap'),

Here we define what sitemaps we’re going to use, and we define a URL for them. It’s pretty straightforward to use.

Let’s run our tests:

$ python manage.py test blogengine
Creating test database for alias 'default'...
............................
----------------------------------------------------------------------
Ran 28 tests in 6.863s

OK
Destroying test database for alias 'default'...

And done! Let’s commit our changes:

git add blogengine/ django_tutorial_blog_ng/settings.py
git commit -m 'Implemented a sitemap'

Fixing test coverage

Our blog is now feature-complete, but there are a few gaps in test coverage, so we’ll fix them. If, like me, you’re using Coveralls.io, you can easily see via their web interface where there are gaps in the coverage.

Now, our gaps are all in our view file - if you take a look at my build, you can easily identify the gaps as they’re marked in red.

The first gap is where a tag does not exist. Interestingly, if we look at the code in the view, we can see that some of it is redundant:

class TagPostsFeed(PostsFeed):
    def get_object(self, request, slug):
        return get_object_or_404(Tag, slug=slug)

    def title(self, obj):
        return "RSS feed - blog posts tagged  %s" % obj.name

    def link(self, obj):
        return obj.get_absolute_url()

    def description(self, obj):
        return "RSS feed - blog posts tagged %s" % obj.name

    def items(self, obj):
        try:
            tag = Tag.objects.get(slug=obj.slug)
            return tag.post_set.all()
        except Tag.DoesNotExist:
            return Post.objects.none()

Under the items function, we check to see if the tag exists. However, under get_object we can see that if the object didn’t exist, it would already have returned a 404 error. We can therefore safely amend items to not check, since that try statement will never fail:

class TagPostsFeed(PostsFeed):
    def get_object(self, request, slug):
        return get_object_or_404(Tag, slug=slug)

    def title(self, obj):
        return "RSS feed - blog posts tagged  %s" % obj.name

    def link(self, obj):
        return obj.get_absolute_url()

    def description(self, obj):
        return "RSS feed - blog posts tagged %s" % obj.name

    def items(self, obj):
        tag = Tag.objects.get(slug=obj.slug)
        return tag.post_set.all()

The other two gaps are in our search view - we never get an empty result for the search in the following section:

def getSearchResults(request):
    """
    Search for a post by title or text
    """
    # Get the query data
    query = request.GET.get('q', '')
    page = request.GET.get('page', 1)

    # Query the database
    if query:
        results = Post.objects.filter(Q(text__icontains=query) | Q(title__icontains=query))
    else:
        results = None

    # Add pagination
    pages = Paginator(results, 5)

    # Get specified page
    try:
        returned_page = pages.page(page)
    except EmptyPage:
        returned_page = pages.page(pages.num_pages)

    # Display the search results
    return render_to_response('blogengine/search_post_list.html',
                              {'page_obj': returned_page,
                               'object_list': returned_page.object_list,
                               'search': query})

So replace it with this:

def getSearchResults(request):
    """
    Search for a post by title or text
    """
    # Get the query data
    query = request.GET.get('q', '')
    page = request.GET.get('page', 1)

    # Query the database
    results = Post.objects.filter(Q(text__icontains=query) | Q(title__icontains=query))

    # Add pagination
    pages = Paginator(results, 5)

    # Get specified page
    try:
        returned_page = pages.page(page)
    except EmptyPage:
        returned_page = pages.page(pages.num_pages)

    # Display the search results
    return render_to_response('blogengine/search_post_list.html',
                              {'page_obj': returned_page,
                               'object_list': returned_page.object_list,
                               'search': query})

We don’t need to check whether query is defined because if q is left blank, the value of query will be an empty string, so we may as well pull out the redundant code.

Finally, the other gap is for when a user tries to get an empty search page (eg, page two of something with five or less results). So let’s add another test to our SearchViewTest class:

    def test_failing_search(self):
        # Search for something that is not present
        response = self.client.get('/search?q=wibble')
        self.assertEquals(response.status_code, 200)
        self.assertTrue('No posts found' in response.content)

        # Try to get nonexistent second page
        response = self.client.get('/search?q=wibble&page=2')
        self.assertEquals(response.status_code, 200)
        self.assertTrue('No posts found' in response.content)

Run our tests and check the coverage:

coverage run --include="blogengine/*" --omit="blogengine/migrations/*" manage.py test blogengine
coverage html

If you open htmlcov/index.html in your browser, you should see that the test coverage is back up to 100%. With that done, it’s time to commit again:

git add blogengine/
git commit -m 'Fixed gaps in coverage'

Remember, it’s not always possible to achieve 100% test coverage, and you shouldn’t worry too much about it if it’s not possible - it’s possible to ignore code if necessary. However, it’s a good idea to aim for 100%.

Using Fabric for deployment

Next we’ll cover using Fabric, a handy tool for deploying your changes (any pretty much any other task you want to automate). First, you need to install it:

pip install Fabric

If you have any problems installing it, you should be able to resolve them via Google - most of them are likely to be absent libraries that Fabric depends upon. Once it’s installed, add it to your requirements.tzt:

pip freeze > requirements.txt

Next, create a file called fabfile.py and enter the following text:

#!/usr/bin/env python
from fabric.api import local

def deploy():
    """
    Deploy the latest version to Heroku
    """
    # Push changes to master
    local("git push origin master")

    # Push changes to Heroku
    local("git push heroku master")

    # Run migrations on Heroku
    local("heroku run python manage.py migrate")

Now, all this file does is push our changes to Github (or wherever else your repository is hosted) and to Heroku, and runs your migrations. It’s not a terribly big task anyway, but it’s handy to have it in place. Let’s commit our changes:

git add fabfile.py requirements.txt
git commit -m 'Added Fabric task for deployment'

Then, let’s try it out:

$ fab deploy

There, wasn’t that more convenient? Fabric is much more powerful than this simple demonstration indicates, and can run tasks on remote servers via SSH easily. I recommend you take a look at the documentation to see what else you can do with it. If you’re hosting your site on a VPS, you will probably find Fabric indispensable, as you will need to restart the application every time you push up a new revision.

Tidying up

We want our blog application to play nicely with other Django apps. For instance, say you’re working on a new site that includes a blogging engine. Wouldn’t it make sense to just be able to drop in this blogging engine and have it work immediately? At the moment, some of our URL’s are hard-coded, so we may have problems in doing so. Let’s fix that.

First we’ll amend our tests. Add this at the top of the tests file:

from django.core.urlresolvers import reverse

Next, replace every instance of this:

        response = self.client.get('/')

with this:

response = self.client.get(reverse('blogengine:index'))

Then, rewrite the calls to the search route. For instance, this:

        response = self.client.get('/search?q=first')

should become this:

        response = self.client.get(reverse('blogengine:search') + '?q=first')

I’ll leave changing these as an exercise for the reader, but check the repository if you get stuck.

Next, we need to assign a namespace to our app’s routes:

from django.conf.urls import patterns, include, url

from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    # Examples:
    # url(r'^$', 'django_tutorial_blog_ng.views.home', name='home'),
    # url(r'^blog/', include('blog.urls')),

    url(r'^admin/', include(admin.site.urls)),

    # Blog URLs
    url(r'', include('blogengine.urls', namespace="blogengine")),

    # Flat pages
    url(r'', include('django.contrib.flatpages.urls')),
)

We then assign names to our routes in the app’s urls.py:

from django.conf.urls import patterns, url
from django.views.generic import ListView, DetailView
from blogengine.models import Post, Category, Tag
from blogengine.views import CategoryListView, TagListView, PostsFeed, CategoryPostsFeed, TagPostsFeed, getSearchResults
from django.contrib.sitemaps.views import sitemap
from blogengine.sitemap import PostSitemap, FlatpageSitemap

# Define sitemaps
sitemaps = {
    'posts': PostSitemap,
    'pages': FlatpageSitemap
}

urlpatterns = patterns('',
    # Index
    url(r'^(?P<page>\d+)?/?$', ListView.as_view(
        model=Post,
        paginate_by=5,
        ),
        name='index'
        ),

    # Individual posts
    url(r'^(?P<pub_date__year>\d{4})/(?P<pub_date__month>\d{1,2})/(?P<slug>[a-zA-Z0-9-]+)/?$', DetailView.as_view(
        model=Post,
        ),
        name='post'
        ),

    # Categories
    url(r'^category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryListView.as_view(
        paginate_by=5,
        model=Category,
        ),
        name='category'
        ),


    # Tags
    url(r'^tag/(?P<slug>[a-zA-Z0-9-]+)/?$', TagListView.as_view(
        paginate_by=5,
        model=Tag,
        ),
        name='tag'
        ),

    # Post RSS feed
    url(r'^feeds/posts/$', PostsFeed()),

    # Category RSS feed
    url(r'^feeds/posts/category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryPostsFeed()),

    # Tag RSS feed
    url(r'^feeds/posts/tag/(?P<slug>[a-zA-Z0-9-]+)/?$', TagPostsFeed()),

    # Search posts
    url(r'^search', getSearchResults, name='search'),

    # Sitemap
    url(r'^sitemap\.xml$', sitemap, {'sitemaps': sitemaps},
            name='django.contrib.sitemaps.views.sitemap'),
)

You also need to amend two of your templates:

``` html blogengine/templates/blogengine/includes/base.html {% raw %}<!DOCTYPE html>

{% block title %}My Django Blog{% endblock %}

    <!-- Place favicon.ico and apple-touch-icon.png in the root directory -->

    {% load staticfiles %}
    <link rel="stylesheet" href="{% static 'bower_components/html5-boilerplate/css/normalize.css' %}">
    <link rel="stylesheet" href="{% static 'bower_components/html5-boilerplate/css/main.css' %}">
    <link rel="stylesheet" href="{% static 'bower_components/bootstrap/dist/css/bootstrap.min.css' %}">
    <link rel="stylesheet" href="{% static 'bower_components/bootstrap/dist/css/bootstrap-theme.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/main.css' %}">
    <link rel="stylesheet" href="{% static 'css/code.css' %}">
    <script src="{% static 'bower_components/html5-boilerplate/js/vendor/modernizr-2.6.2.min.js' %}"></script>
</head>
<body>
    <!--[if lt IE 7]>
        <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
    <![endif]-->

    <!-- Add your site or application content here -->

    <div id="fb-root"></div>
    <script>(function(d, s, id) {
        var js, fjs = d.getElementsByTagName(s)[0];
        if (d.getElementById(id)) return;
            js = d.createElement(s); js.id = id;
            js.src = "//connect.facebook.net/en_GB/all.js#xfbml=1";
            fjs.parentNode.insertBefore(js, fjs);
        }(document, 'script', 'facebook-jssdk'));</script>

    <div class="navbar navbar-static-top navbar-inverse">
        <div class="container-fluid">
            <div class="navbar-header">
                <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#header-nav">
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                </button>
                <a class="navbar-brand" href="{% url 'blogengine:index' %}">My Django Blog</a>
            </div>
            <div class="collapse navbar-collapse" id="header-nav">
                <ul class="nav navbar-nav">
                    {% load flatpages %}
                    {% get_flatpages as flatpages %}
                    {% for flatpage in flatpages %}
                    <li><a href="{{ flatpage.url }}">{{ flatpage.title }}</a></li>
                    {% endfor %}
                    <li><a href="http://matthewdaly.co.uk/feeds/posts/">RSS feed</a></li>

                    <form action="/search" method="GET" class="navbar-form navbar-left">
                        <div class="form-group">
                            <input type="text" name="q" placeholder="Search..." class="form-control"></input>
                        </div>
                        <button type="submit" class="btn btn-default">Search</button>
                    </form>
                </ul>
            </div>
        </div>
    </div>

    <div class="container">
        {% block header %}
            <div class="page-header">
                <h1>My Django Blog</h1>
            </div>
        {% endblock %}

        <div class="row">
            {% block content %}{% endblock %}
        </div>
    </div>

    <div class="container footer">
        <div class="row">
            <div class="span12">
                <p>Copyright &copy; {% now "Y" %}</p>
            </div>
        </div>
    </div>

    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
    <script>window.jQuery || document.write('<script src="{% static 'bower_components/html5-boilerplate/js/vendor/jquery-1.10.2.min.js' %}"><\/script>')</script>
    <script src="{% static 'bower_components/html5-boilerplate/js/plugins.js' %}"></script>
    <script src="{% static 'bower_components/bootstrap/dist/js/bootstrap.min.js' %}"></script>

    <!-- Google Analytics: change UA-XXXXX-X to be your site's ID. -->
    <script>
        (function(b,o,i,l,e,r){b.GoogleAnalyticsObject=l;b[l]||(b[l]=
        function(){(b[l].q=b[l].q||[]).push(arguments)});b[l].l=+new Date;
        e=o.createElement(i);r=o.getElementsByTagName(i)[0];
        e.src='//www.google-analytics.com/analytics.js';
        r.parentNode.insertBefore(e,r)}(window,document,'script','ga'));
        ga('create','UA-XXXXX-X');ga('send','pageview');
    </script>
</body>

{% endraw %} ```

{% raw %}{% extends "blogengine/includes/base.html" %}

    {% load custom_markdown %}

    {% block content %}
        {% if object_list %}
            {% for post in object_list %}
            <div class="post col-md-12">
            <h1><a href="{{ post.get_absolute_url }}">{{ post.title }}</a></h1>
            <h3>{{ post.pub_date }}</h3>
            {{ post.text|custom_markdown }}
            </div>
            {% if post.category %}
            <div class="col-md-12">
            <a href="{{ post.category.get_absolute_url }}"><span class="label label-primary">{{ post.category.name }}</span></a>
            </div>
            {% endif %}
            {% if post.tags %}
            <div class="col-md-12">
            {% for tag in post.tags.all %}
            <a href="{{ tag.get_absolute_url }}"><span class="label label-success">{{ tag.name }}</span></a>
            {% endfor %}
            </div>
            {% endif %}
            {% endfor %}
        {% else %}
            <p>No posts found</p>
        {% endif %}

        <ul class="pager">
        {% if page_obj.has_previous %}
        <li class="previous"><a href="{% url 'blogengine:search' %}?page={{ page_obj.previous_page_number }}&q={{ search }}">Previous Page</a></li>
        {% endif %}
        {% if page_obj.has_next %}
        <li class="next"><a href="{% url 'blogengine:search' %}?page={{ page_obj.next_page_number }}&q={{ search }}">Next Page</a></li>
        {% endif %}
        </ul>

    {% endblock %}{% endraw %}

Let’s run our tests:

$ python manage.py test blogengine/
Creating test database for alias 'default'...
.............................
----------------------------------------------------------------------
Ran 29 tests in 10.456s

OK
Destroying test database for alias 'default'...

And commit our changes:

git add .
git commit -m 'Now use named routes'

Debugging Django

There are a number of handy ways to debug Django applications. One of the simplest is to use the Python debugger. To use it, just enter the following lines at the point you want to break at:

import pdb
pdb.set_trace()

Now, whenever that line of code is run, you’ll be dropped into an interactive shell that lets you play around to find out what’s going wrong. However, it doesn’t offer autocompletion, so we’ll install ipdb, which is an improved version:

pip install ipdb
pip freeze > requirements.txt

Now you can use ipdb in much the same way as you would use pdb:

import ipdb
ipdb.set_trace()

Now, ipdb is very useful, but it isn’t much help for profiling your application. For that you need the Django Debug Toolbar. Run the following commands:

pip install django-debug-toolbar
pip freeze > requirements.txt

Then add the following line to INSTALLED_APPS in your settings file:

    'debug_toolbar',

Then, try running the development server, and you’ll see a toolbar on the right-hand side of the screen that allows you to view some useful data about your page. For instance, you’ll notice a field called SQL - this contains details of the queries carried out when building the page. To actually see the queries carried out, you’ll want to disable caching in your settings file by commenting out all the constants that start with CACHE.

We won’t go into using the toolbar to optimise queries, but using this, you can easily see what queries are being executed on a specific page, how long they take, and the values they return. Sometimes, you may need to optimise a slow query - in this case, Django allows you to drop down to writing raw SQL if necessary.

Note that if you’re running Django in production, you should set DEBUG to False as otherwise it gives rather too much information to potential attackers, and with Django Debug Toolbar installed, that’s even more important.

Please also note that when you disable debug mode, Django no longer handles static files automatically, so you’ll need to run python manage.py collectstatic and commit the staticfiles directory.

Once you’ve disabled debug mode, collected the static files, and re-enables caching, you can commit your changes:

git add .
git commit -m 'Installed debugging tools'

Optimising static files

We want our blog to get the best SEO results it can, so making it fast is essential. One of the simplest things you can do is to concatenate and minify static assets such as CSS and JavaScript. There are numerous ways to do this, but I generally use Grunt. Let’s set up a Grunt config to concatenate and minify our CSS and JavaScript.

You’ll need to have Node.js installed on your development machine for this. Then, you need to install the Grunt command-line interface:

sudo npm install -g grunt-cli

With that done, we need to create a package.json file. You can create one using the command npm init. Here’s mine:

{
  "name": "django_tutorial_blog_ng",
  "version": "1.0.0",
  "description": "Django Tutorial Blog NG =======================",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/matthewbdaly/django_tutorial_blog_ng.git"
  },
  "author": "Matthew Daly <matthew@matthewdaly.co.uk> (http://matthewdaly.co.uk/)",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/matthewbdaly/django_tutorial_blog_ng/issues"
  },
  "homepage": "https://github.com/matthewbdaly/django_tutorial_blog_ng"
}

Feel free to amend it as you see fit.

Next we install Grunt and the required plugins:

npm install grunt grunt-contrib-cssmin grunt-contrib-concat grunt-contrib-uglify --save-dev

We now need to create a Gruntfile for our tasks:

module.exports = function (grunt) {
    'use strict';

    grunt.initConfig({
        concat: {
            dist: {
                src: [
                    'blogengine/static/bower_components/bootstrap/dist/css/bootstrap.css',
                    'blogengine/static/bower_components/bootstrap/dist/css/bootstrap-theme.css',
                    'blogengine/static/css/code.css',
                    'blogengine/static/css/main.css',
                ],
                dest: 'blogengine/static/css/style.css'
            }
        },
        uglify: {
            dist: {
                src: [
                    'blogengine/static/bower_components/jquery/jquery.js',
                    'blogengine/static/bower_components/bootstrap/dist/js/bootstrap.js'
                ],
                dest: 'blogengine/static/js/all.min.js'
            }
        },
        cssmin: {
            dist: {
                src: 'blogengine/static/css/style.css',
                dest: 'blogengine/static/css/style.min.css'
            }
        }
    });

    grunt.loadNpmTasks('grunt-contrib-concat');
    grunt.loadNpmTasks('grunt-contrib-uglify');
    grunt.loadNpmTasks('grunt-contrib-cssmin');
    grunt.registerTask('default', ['concat', 'uglify', 'cssmin']);
};

You’ll also need to change the paths in your base HTML file to point to the minified versions:

``` html blogengine/templates/blogengine/includes/base.html {% raw %}<!DOCTYPE html>

{% block title %}My Django Blog{% endblock %}

    <!-- Place favicon.ico and apple-touch-icon.png in the root directory -->

    {% load staticfiles %}
    <link rel="stylesheet" href="{% static 'css/style.min.css' %}">
</head>
<body>
    <!--[if lt IE 7]>
        <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
    <![endif]-->

    <!-- Add your site or application content here -->

    <div id="fb-root"></div>
    <script>(function(d, s, id) {
        var js, fjs = d.getElementsByTagName(s)[0];
        if (d.getElementById(id)) return;
            js = d.createElement(s); js.id = id;
            js.src = "//connect.facebook.net/en_GB/all.js#xfbml=1";
            fjs.parentNode.insertBefore(js, fjs);
        }(document, 'script', 'facebook-jssdk'));</script>

    <div class="navbar navbar-static-top navbar-inverse">
        <div class="container-fluid">
            <div class="navbar-header">
                <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#header-nav">
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                </button>
                <a class="navbar-brand" href="{% url 'blogengine:index' %}">My Django Blog</a>
            </div>
            <div class="collapse navbar-collapse" id="header-nav">
                <ul class="nav navbar-nav">
                    {% load flatpages %}
                    {% get_flatpages as flatpages %}
                    {% for flatpage in flatpages %}
                    <li><a href="{{ flatpage.url }}">{{ flatpage.title }}</a></li>
                    {% endfor %}
                    <li><a href="http://matthewdaly.co.uk/feeds/posts/">RSS feed</a></li>

                    <form action="/search" method="GET" class="navbar-form navbar-left">
                        <div class="form-group">
                            <input type="text" name="q" placeholder="Search..." class="form-control"></input>
                        </div>
                        <button type="submit" class="btn btn-default">Search</button>
                    </form>
                </ul>
            </div>
        </div>
    </div>

    <div class="container">
        {% block header %}
            <div class="page-header">
                <h1>My Django Blog</h1>
            </div>
        {% endblock %}

        <div class="row">
            {% block content %}{% endblock %}
        </div>
    </div>

    <div class="container footer">
        <div class="row">
            <div class="span12">
                <p>Copyright &copy; {% now "Y" %}</p>
            </div>
        </div>
    </div>

    <script src="{% static 'js/all.min.js' %}"></script>

    <!-- Google Analytics: change UA-XXXXX-X to be your site's ID. -->
    <script>
        (function(b,o,i,l,e,r){b.GoogleAnalyticsObject=l;b[l]||(b[l]=
        function(){(b[l].q=b[l].q||[]).push(arguments)});b[l].l=+new Date;
        e=o.createElement(i);r=o.getElementsByTagName(i)[0];
        e.src='//www.google-analytics.com/analytics.js';
        r.parentNode.insertBefore(e,r)}(window,document,'script','ga'));
        ga('create','UA-XXXXX-X');ga('send','pageview');
    </script>
</body>

{% endraw %} ```

Now, run the Grunt task:

grunt

And collect the static files:

python manage.py collectstatic

You’ll also want to add your node_modules folder to your gitignore:

venv/
*.pyc
db.sqlite3
reports/
htmlcov/
.coverage
node_modules/

Then commit your changes:

git add .
git commit -m 'Optimised static assets'

Now, our package.json will cause a problem - it will mean that this app is mistakenly identified as a Node.js app. To prevent this, create the following file:

package.json

Then commit your changes and push them up:

git add .slugignore
git commit -m 'Added slugignore'
fab deploy

If you check, your site should now be loading the minified versions of the static files.

That’s our site done! As usual I’ve tagged the final commit with lesson-8.

Sadly, that’s our final instalment over with! I hope you’ve enjoyed these tutorials, and I look forward to seeing what you create with them.


Setting up OpenSprinkler

Aug 28 2014 [Archived Version] □ Published at Nerdy Dork under tags  technology

I set up my OpenSprinkler last night. It was a pretty painless endeavor. I’ll walk through it in this post so I can explain what I did. The Purchase There are a few different models and options to choose from on RaysHobby.net. Time seems to be my greatest commodity at this stage in life so […]

The post Setting up OpenSprinkler appeared first on Dustin Davis.


eGenix PyRun - One file Python Runtime 2.0.1 GA

Aug 27 2014 [Archived Version] □ Published at eGenix.com News & Events

Introduction

eGenix PyRun is our open source, one file, no installation version of Python, making the distribution of a Python interpreter to run based scripts and applications to Unix based systems as simple as copying a single file.

eGenix PyRun's executable only needs 11MB for Python 2 and 13MB for Python 3, but still supports most Python application and scripts - and it can be compressed to just 3-4MB using upx, if needed.

Compared to a regular Python installation of typically 100MB on disk, eGenix PyRun is ideal for applications and scripts that need to be distributed to several target machines, client installations or customers.

It makes "installing" Python on a Unix based system as simple as copying a single file.

eGenix has been using the product internally in the mxODBC Connect Server since 2008 with great success and decided to make it available as a stand-alone open-source product.

We provide both the source archive to build your own eGenix PyRun, as well as pre-compiled binaries for Linux, FreeBSD and Mac OS X, as 32- and 64-bit versions. The binaries can be downloaded manually, or you can let our automatic install script install-pyrun take care of the installation: ./install-pyrun dir and you're done.

Please see the product page for more details:

    >>> eGenix PyRun - One file Python Runtime

News

This is a patch level release of eGenix PyRun 2.0. The major new feature in 2.0 is the added Python 3.4 support.

New Features

  • Upgraded eGenix PyRun to work with and use Python 2.7.8 per default.

Enhancements / Changes

  • Fixed a bug in the license printer to show the correct license URL.

install-pyrun Quick Install Enhancements

eGenix PyRun includes a shell script called install-pyrun, which greatly simplifies installation of PyRun. It works much like the virtualenv shell script used for creating new virtual environments (except that there's nothing virtual about PyRun environments).

With the script, an eGenix PyRun installation is as simple as running:

./install-pyrun targetdir

This will automatically detect the platform, download and install the right pyrun version into targetdir.

We have updated this script since the last release:

  • Updated install-pyrun to default to eGenix PyRun 2.0.1 and its feature set.

For a complete list of changes, please see the eGenix PyRun Changelog.

Please see the eGenix PyRun 2.0.0 announcement for more details about eGenix PyRun 2.0.

Downloads

Please visit the eGenix PyRun product page for downloads, instructions on installation and documentation of the product.

More Information

For more information on eGenix PyRun, licensing and download instructions, please write to sales@egenix.com.

Enjoy !

Marc-Andre Lemburg, eGenix.com


Django Blog Tutorial - the Next Generation - Part 7

Aug 25 2014 [Archived Version] □ Published at Matthew Daly's Blog

Hello once again! In this instalment we’ll cover:

  • Caching your content with Memcached to improve your site’s performance
  • Refactoring and simplifying our tests
  • Implementing additional feeds
  • Creating a simple search engine

Don’t forget to activate your virtualenv:

source venv/bin/activate

Now let’s get started!

Memcached

If you frequent (or used to frequent) social media sites like Reddit, Slashdot or Digg, you may be familiar with something called variously the Digg or Slashdot effect, whereby if a page gets submitted to a social media site, and subsequently becomes popular, it can be hit by a huge number of HTTP requests in a very short period of time, slowing it down or even taking the server down completely.

Now, as a general rule of thumb, for most dynamic websites such as blogs, the bottleneck is not the web server or the programming language, but the database. If you have a lot of people hitting the same page over and over again in quick succession, then you’re essentially running the same query over and over again and getting the same result each time, which is expensive in terms of processing power. What you need to be able to do is cache the results of the query in memory for a given period of time so that the number of queries is reduced.

That’s where Memcached comes in. It’s a simple key-value store that allows you to store values in memory for a given period of time so that they can be retrieved without having to query the database. Memcached is a very common choice for caching, and is by far the fastest and most efficient type of cache available for Django. It’s also available on Heroku’s free tier.

Django has a very powerful caching framework that supports numerous types of cache in addition to Memcached, such as:

  • Database caching
  • Filesystem caching
  • Local memory caching

There are also third-party backends for using other caching systems such as Redis.

Now, the cache can be used in a number of different ways. You can cache only certain parts of your site if you wish. However, because our site is heavily content-driven, we should be pretty safe to use the per-site cache, which is the simplest way to set up caching.

In order to set up Memcached, there’s a couple of Python libraries we’ll need. If you want to install them locally, however, you’ll need to install both memcached and libmemcached (on Ubuntu, the packages you need are called memcached and libmemcached-dev) on your development machine. If you don’t want to do this, then just copy and paste these lines into requirements.txt instead:

django-pylibmc-sasl==0.2.4
pylibmc==1.3.0

If you are happy to install these dependencies locally, then run this command once memcached and libmemcached are installed:

pip install pylibmc django-pylibmc-sasl

With that done let’s configure Memcached. Open up the settings file and add the following at the bottom:

def get_cache():
  import os
  try:
    os.environ['MEMCACHE_SERVERS'] = os.environ['MEMCACHIER_SERVERS'].replace(',', ';')
    os.environ['MEMCACHE_USERNAME'] = os.environ['MEMCACHIER_USERNAME']
    os.environ['MEMCACHE_PASSWORD'] = os.environ['MEMCACHIER_PASSWORD']
    return {
      'default': {
        'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',
        'TIMEOUT': 300,
        'BINARY': True,
        'OPTIONS': { 'tcp_nodelay': True }
      }
    }
  except:
    return {
      'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'
      }
    }

CACHES = get_cache()
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_SECONDS = 300
CACHE_MIDDLEWARE_KEY_PREFIX = ''

Then add the following to MIDDLEWARE_CLASSES:

    'django.middleware.cache.UpdateCacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',

That’s it! The first section configures the application to use Memcached to cache the content when running on Heroku, and sets some configuration parameters, while the second section tells Django to use the per-site cache in order to cache all the site content.

Now, Heroku doesn’t include Memcached by default - instead it’s available as an add-on called Memcachier. To use add-ons you need to set up a credit card for billing. We will set it up to use the free developer plan, but if you outgrow this you can easily switch to a paid plan. To add Memcachier, run this command:

heroku addons:add memcachier:dev

Please note that Memcachier can take a few minutes to get set up, so you may want to leave it a little while between adding it and pushing up your changes. Now we’ll commit our changes:

git add requirements.txt django_tutorial_blog_ng/settings.py
git commit -m 'Implemented caching with Memcached'

Then we’ll push them up to our remote repository and to Heroku:

git push origin master
git push heroku master

And that’s all you need to do to set up Memcached. In addition to storing your query results in Memcached, enabling the caching framework in Django will also set various HTTP headers to enable web proxies and browsers to cache content for an appropriate length of time. If you open up your browser’s developer tools and compare the response headers for your homepage on the latest version of the code with the previous version, you’ll notice that a number of additional headers appear, including Cache-Control, Expires and Last-Modified. These tell web browsers and web proxies how often to request the latest version of the HTML document, in order to help you reduce the bandwidth used.

As you can see, for a site like this where you are the only person adding content, it’s really easy to implement caching with Django, and for a blog there’s very little reason not to do it. If you’re not using Heroku and are instead hosting your site on a VPS, then the configuration will be somewhat different - see here for details. You can also find information on using other cache backends on the same page.

That isn’t all you can do to speed up your site. Heroku doesn’t seem to be very good for serving static files, and if your site is attracting a lot of traffic you might want to host your static files elsewhere, such as on Amazon’s S3 service. Doing so is outside the scope of this tutorial, but for that use case, you should check out django-storages.

Clearing the cache automatically

There is one issue with this implementation. As it is right now, if you view the home page, add a post, then reload the page, you may not see the new post immediately because the cache will continue serving the old version until it has expired. That behaviour is less than ideal - we would like the cache to be cleared automatically when a new post gets added so that users will see the new version immediately. That response will still be cached afterwards, so it only means one extra query.

This is the ideal place to introduce signals. Signals are a way to carry out a given action when an event takes place. In our case, we plan to clear the cache when a post is saved (either created or updated).

Note that as we’ll be testing the behaviour of the cache at this point, you’ll need to install Memcached on your local machine, and we’ll need to change the settings to fall back to our local Memcached instance:

def get_cache():
  import os
  try:
    os.environ['MEMCACHE_SERVERS'] = os.environ['MEMCACHIER_SERVERS'].replace(',', ';')
    os.environ['MEMCACHE_USERNAME'] = os.environ['MEMCACHIER_USERNAME']
    os.environ['MEMCACHE_PASSWORD'] = os.environ['MEMCACHIER_PASSWORD']
    return {
      'default': {
        'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',
        'TIMEOUT': 300,
        'BINARY': True,
        'OPTIONS': { 'tcp_nodelay': True }
      }
    }
  except:
    return {
      'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211'
      }
    }

CACHES = get_cache()
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_SECONDS = 300
CACHE_MIDDLEWARE_KEY_PREFIX = ''

If you don’t want to install Memcached locally, you can skip this step, but be aware that the test we write for clearing the cache will always pass if you do skip it.

Then we’ll run our tests to make sure nothing has been broken:

$ python manage.py jenkins blogengine
Creating test database for alias 'default'...
.......................
----------------------------------------------------------------------
Ran 23 tests in 6.164s

OK

Let’s commit:

git add django_tutorial_blog_ng/settings.py
git commit -m 'Now use Memcached in development'

Now we’ll add a test for clearing the cache to PostViewTest:

    def test_clear_cache(self):
        # Create the category
        category = Category()
        category.name = 'python'
        category.description = 'The Python programming language'
        category.save()

        # Create the tag
        tag = Tag()
        tag.name = 'perl'
        tag.description = 'The Perl programming language'
        tag.save()

        # Create the author
        author = User.objects.create_user('testuser', 'user@example.com', 'password')
        author.save()

        # Create the site
        site = Site()
        site.name = 'example.com'
        site.domain = 'example.com'
        site.save()

        # Create the first post
        post = Post()
        post.title = 'My first post'
        post.text = 'This is [my first blog post](http://127.0.0.1:8000/)'
        post.slug = 'my-first-post'
        post.pub_date = timezone.now()
        post.author = author
        post.site = site
        post.category = category
        post.save()
        post.tags.add(tag)

        # Check new post saved
        all_posts = Post.objects.all()
        self.assertEquals(len(all_posts), 1)

        # Fetch the index
        response = self.client.get('/')
        self.assertEquals(response.status_code, 200)

        # Create the second post
        post = Post()
        post.title = 'My second post'
        post.text = 'This is [my second blog post](http://127.0.0.1:8000/)'
        post.slug = 'my-second-post'
        post.pub_date = timezone.now()
        post.author = author
        post.site = site
        post.category = category
        post.save()
        post.tags.add(tag)

        # Fetch the index again
        response = self.client.get('/')

        # Check second post present
        self.assertTrue('my second blog post' in response.content)

This should be fairly self-explanatory. We create one post, and request the index page. We then add a second post, request the index page again, and check for the second post. The test should fail because the cached version is returned, rather than the version in the database.

Now we have a test in place, we can implement a fix. First, add this to the top of your models.py:

from django.db.models.signals import post_save
from django.core.cache import cache

Then add the following at the bottom of the file:


# Define signals
def new_post(sender, instance, created, **kwargs):
    cache.clear()

# Set up signals
post_save.connect(new_post, sender=Post)

This is fairly straightforward. What we’re doing is first defining a function called new_post that is called when a new post is created. We then connect it to the post_save signal. When a post is saved, it calls new_post, which clears the cache, making sure users are seeing the latest and greatest version of your site immediately.

Let’s test it:

$ python manage.py jenkins blogengine
Creating test database for alias 'default'...
........................
----------------------------------------------------------------------
Ran 24 tests in 8.473s

OK
Destroying test database for alias 'default'...

There are a number of signals available, and when you create one, you have access to the created object via the instance parameter. Using signals you can implement all kinds of functionality. For instance, you could implement the functionality to send an email when a new post is published.

If you’re using Travis CI, you’ll also need to update the config file:

language: python
python:
- "2.7"
services: memcached
before_install:
    - sudo apt-get install -y libmemcached-dev
# command to install dependencies
install: "pip install -r requirements.txt"
# command to run tests
script: coverage run --include="blogengine/*" --omit="blogengine/migrations/*" manage.py test blogengine
after_success:
    coveralls

Time to commit:

git add blogengine/ .travis.yml
git commit -m 'Now clear cache when post added'

Formatting for RSS feeds

Now, we want to offer more than one option for RSS feeds. For instance, if your blog is aggregated on a site such as Planet Python, but you also blog about JavaScript, you may want to be able to provide a feed for posts in the python category only.

If you have written any posts that use any of Markdown’s custom formatting, you may notice that if you load your RSS feed in a reader, it isn’t formatted as Markdown. Let’s fix that. First we’ll amend our test:

class FeedTest(BaseAcceptanceTest):
    def test_all_post_feed(self):
        # Create the category
        category = Category()
        category.name = 'python'
        category.description = 'The Python programming language'
        category.save()

        # Create the tag
        tag = Tag()
        tag.name = 'python'
        tag.description = 'The Python programming language'
        tag.save()

        # Create the author
        author = User.objects.create_user('testuser', 'user@example.com', 'password')
        author.save()

        # Create the site
        site = Site()
        site.name = 'example.com'
        site.domain = 'example.com'
        site.save()

        # Create a post
        post = Post()
        post.title = 'My first post'
        post.text = 'This is my *first* blog post'
        post.slug = 'my-first-post'
        post.pub_date = timezone.now()
        post.author = author
        post.site = site
        post.category = category

        # Save it
        post.save()

        # Add the tag
        post.tags.add(tag)
        post.save()

        # Check we can find it
        all_posts = Post.objects.all()
        self.assertEquals(len(all_posts), 1)
        only_post = all_posts[0]
        self.assertEquals(only_post, post)

        # Fetch the feed
        response = self.client.get('/feeds/posts/')
        self.assertEquals(response.status_code, 200)

        # Parse the feed
        feed = feedparser.parse(response.content)

        # Check length
        self.assertEquals(len(feed.entries), 1)

        # Check post retrieved is the correct one
        feed_post = feed.entries[0]
        self.assertEquals(feed_post.title, post.title)
        self.assertTrue('This is my <em>first</em> blog post' in feed_post.description)

Don’t forget to run the tests to make sure they fail. Now, let’s fix it:

from django.shortcuts import render
from django.views.generic import ListView
from blogengine.models import Category, Post, Tag
from django.contrib.syndication.views import Feed
from django.utils.encoding import force_unicode
from django.utils.safestring import mark_safe
import markdown2

# Create your views here.
class CategoryListView(ListView):
    def get_queryset(self):
        slug = self.kwargs['slug']
        try:
            category = Category.objects.get(slug=slug)
            return Post.objects.filter(category=category)
        except Category.DoesNotExist:
            return Post.objects.none()


class TagListView(ListView):
    def get_queryset(self):
        slug = self.kwargs['slug']
        try:
            tag = Tag.objects.get(slug=slug)
            return tag.post_set.all()
        except Tag.DoesNotExist:
            return Post.objects.none()


class PostsFeed(Feed):
    title = "RSS feed - posts"
    link = "feeds/posts/"
    description = "RSS feed - blog posts"

    def items(self):
        return Post.objects.order_by('-pub_date')

    def item_title(self, item):
        return item.title

    def item_description(self, item):
        extras = ["fenced-code-blocks"]
        content = mark_safe(markdown2.markdown(force_unicode(item.text),
                                               extras = extras))
        return content

All we’re doing here is amending the item_description method of PostsFeed to render it as Markdown. Now let’s run our tests again:

$ python manage.py jenkins
Creating test database for alias 'default'...
........................
----------------------------------------------------------------------
Ran 24 tests in 9.370s

OK
Destroying test database for alias 'default'...

With that done, we’ll commit our changes:

git add blogengine/
git commit -m 'Fixed rendering for post feed'

Refactoring our tests

Now, before we get into implementing the feed, our tests are a bit verbose. We create a lot of items over and over again - let’s sort that out. Factory Boy is a handy Python module that allows you to create easy-to-use factories for creating objects over and over again in tests. Let’s install it:

pip install factory_boy
pip freeze > requirements.txt
git add requirements.txt
git commit -m 'Installed Factory Boy'

Now let’s set up a factory for creating posts. Add this at the top of the test file:

import factory.django

Then, before your actual tests, insert the following:

# Factories
class SiteFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Site
        django_get_or_create = (
            'name',
            'domain'
        )

    name = 'example.com'
    domain = 'example.com'

Now, wherever you call Site(), add its attributes, and save it, replace those lines with the following:

        site = SiteFactory()

Much simpler and more concise, I’m sure you’ll agree! Now, let’s run the tests to make sure they aren’t broken:

$ python manage.py test blogengine
Creating test database for alias 'default'...
........................
----------------------------------------------------------------------
Ran 24 tests in 7.482s

OK
Destroying test database for alias 'default'...

Let’s commit again: bash git add blogengine/tests.py git commit -m 'Now use Factory Boy for site objects'

Let’s do the same thing with Category objects:

class CategoryFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Category
        django_get_or_create = (
            'name',
            'description',
            'slug'
        )

    name = 'python'
    description = 'The Python programming language'
    slug = 'python'

Again, just find every time we call Category() and replace it with the following:

        category = CategoryFactory()

Now if we run our tests, we’ll notice a serious error:

$ python manage.py test blogengine
Creating test database for alias 'default'...
EE..EE.EE.EE...E.EEE..E.
======================================================================
ERROR: test_create_category (blogengine.tests.PostTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 42, in test_create_category
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_create_post (blogengine.tests.PostTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 80, in test_create_post
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_create_post (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 339, in test_create_post
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_create_post_without_tag (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 378, in test_create_post_without_tag
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_delete_category (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 245, in test_delete_category
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_delete_post (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 467, in test_delete_post
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_edit_category (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 221, in test_edit_category
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_edit_post (blogengine.tests.AdminTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 410, in test_edit_post
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_all_post_feed (blogengine.tests.FeedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 810, in test_all_post_feed
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_category_page (blogengine.tests.PostViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 640, in test_category_page
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_clear_cache (blogengine.tests.PostViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 753, in test_clear_cache
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_index (blogengine.tests.PostViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 518, in test_index
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create


    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

======================================================================
ERROR: test_post_page (blogengine.tests.PostViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 576, in test_post_page
    category = CategoryFactory()
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__
    return cls.create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create
    return cls._generate(True, attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate
    obj = cls._prepare(create, **attrs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare
    return cls._create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create


    return cls._get_or_create(model_class, *args, **kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create
    obj, _created = manager.get_or_create(*args, **key_fields)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
TypeError: save() got an unexpected keyword argument 'force_insert'

----------------------------------------------------------------------
Ran 24 tests in 5.162s

FAILED (errors=13)
Destroying test database for alias 'default'...

Thankfully, this is easy to fix. We just need to amend the custom save() method of the Category model:

    def save(self, *args, **kwargs):
        if not self.slug:
            self.slug = slugify(unicode(self.name))
        super(Category, self).save(*args, **kwargs)

That should resolve the issue:

$ python manage.py jenkins
Creating test database for alias 'default'...
........................
----------------------------------------------------------------------
Ran 24 tests in 7.749s

OK
Destroying test database for alias 'default'...

Let’s commit again:

git add blogengine/
git commit -m 'Category now uses Factory Boy'

Now let’s do the same thing for tags:

class TagFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Tag
        django_get_or_create = (
            'name',
            'description',
            'slug'
        )

    name = 'python'
    description = 'The Python programming language'
    slug = 'python'

And replace the sections where we create new Tag objects:

        tag = TagFactory()

Note that some tags have different values. We can easily pass different values to our TagFactory() to override the default values:

        tag = TagFactory(name='perl', description='The Perl programming language')

The Tag model has the same issue as the Category one did, so let’s fix that:

    def save(self, *args, **kwargs):
        if not self.slug:
            self.slug = slugify(unicode(self.name))
        super(Tag, self).save(*args, **kwargs)

We run our tests again:

$ python manage.py test blogengine/
Creating test database for alias 'default'...
........................
----------------------------------------------------------------------
Ran 24 tests in 7.153s

OK
Destroying test database for alias 'default'...

Time to commit again:

git add blogengine/
git commit -m 'Now use Factory Boy for tags'

Next we’ll create a factory for adding users. Note that the factory name doesn’t have to match the object name, so you can create factories for different types of users. Here we create a factory for authors - you could, for instance, create a separate factory for subscribers if you wanted:

class AuthorFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = User
        django_get_or_create = ('username','email', 'password',)

    username = 'testuser'
    email = 'user@example.com'
    password = 'password'

And as before, replace those sections where we create users with the following:

        author = AuthorFactory()

Run the tests again:

$ python manage.py test blogengine
Creating test database for alias 'default'...
........................
----------------------------------------------------------------------
Ran 24 tests in 5.808s

OK
Destroying test database for alias 'default'...

We commit our changes:

git add blogengine/
git commit -m 'Now use Factory Boy for creating authors'

Now we’ll create a flat page factory:



class FlatPageFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = FlatPage
        django_get_or_create = (
            'url',
            'title',
            'content'
        )

    url = '/about/'
    title = 'About me'
    content = 'All about me'

And use it for our flat page test: python blogengine/tests.py page = FlatPageFactory()

Check the tests pass:

$ python manage.py test blogengine
Creating test database for alias 'default'...
........................
----------------------------------------------------------------------
Ran 24 tests in 5.796s

OK
Destroying test database for alias 'default'...

And commit again:

git add blogengine/
git commit -m 'Now use Factory Boy for flat page test'

Now we’ll create a final factory for posts:

class PostFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Post
        django_get_or_create = (
            'title',
            'text',
            'slug',
            'pub_date'
        )

    title = 'My first post'
    text = 'This is my first blog post'
    slug = 'my-first-post'
    pub_date = timezone.now()
    author = factory.SubFactory(AuthorFactory)
    site = factory.SubFactory(SiteFactory)
    category = factory.SubFactory(CategoryFactory)

This factory is a little bit different. Because our Post model depends on several others, we need to be able to create those additional objects on demand. By designating them as subfactories, we can easily create the associated objects for our Post object.

That means that not only can we get rid of our Post() calls, but we can also get rid of the factory calls to create the associated objects for Post models. Again, I’ll leave actually doing this as an exercise for the reader, but you can always refer to the GitHub repository if you’re not too sure.

Make sure your tests still pass, then commit the changes:

git add blogengine/
git commit -m 'Now use Factory Boy for testing posts'

Using Factory Boy made a big difference to the size of the test file - I was able to cut it down by over 200 lines of code. As your application gets bigger, it gets harder to maintain, so do what you can to keep the size down.

Additional RSS feeds

Now, let’s implement our additional RSS feeds. First, we’ll write a test for the category feed. Add this to the FeedTest class:

    def test_category_feed(self):
        # Create a post
        post = PostFactory(text='This is my *first* blog post')

        # Create another post in a different category
        category = CategoryFactory(name='perl', description='The Perl programming language', slug='perl')
        post2 = PostFactory(text='This is my *second* blog post', title='My second post', slug='my-second-post', category=category)

        # Fetch the feed
        response = self.client.get('/feeds/posts/category/python/')
        self.assertEquals(response.status_code, 200)

        # Parse the feed
        feed = feedparser.parse(response.content)

        # Check length
        self.assertEquals(len(feed.entries), 1)

        # Check post retrieved is the correct one
        feed_post = feed.entries[0]
        self.assertEquals(feed_post.title, post.title)
        self.assertTrue('This is my <em>first</em> blog post' in feed_post.description)

        # Check other post is not in this feed
        self.assertTrue('This is my <em>second</em> blog post' not in response.content)

Here we create two posts in different categories (note that we create a new category and override the post category for it). We then fetch `/feeds/posts/category/python/‘ and assert that it contains only one post, with the content of the first post and not the content of the second.

Run the tests and they should fail:

$ python manage.py test blogengine
Creating test database for alias 'default'...
................F........
======================================================================
FAIL: test_category_feed (blogengine.tests.FeedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 716, in test_category_feed
    self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200

----------------------------------------------------------------------
Ran 25 tests in 5.804s

FAILED (failures=1)
Destroying test database for alias 'default'...

Because we haven’t yet implemented that route, we get a 404 error. So let’s create a route for this:

from django.conf.urls import patterns, url
from django.views.generic import ListView, DetailView
from blogengine.models import Post, Category, Tag
from blogengine.views import CategoryListView, TagListView, PostsFeed, CategoryPostsFeed

urlpatterns = patterns('',
    # Index
    url(r'^(?P<page>\d+)?/?$', ListView.as_view(
        model=Post,
        paginate_by=5,
        )),

    # Individual posts
    url(r'^(?P<pub_date__year>\d{4})/(?P<pub_date__month>\d{1,2})/(?P<slug>[a-zA-Z0-9-]+)/?$', DetailView.as_view(
        model=Post,
        )),

    # Categories
    url(r'^category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryListView.as_view(
        paginate_by=5,
        model=Category,
        )),

    # Tags
    url(r'^tag/(?P<slug>[a-zA-Z0-9-]+)/?$', TagListView.as_view(
        paginate_by=5,
        model=Tag,
        )),

    # Post RSS feed
    url(r'^feeds/posts/$', PostsFeed()),

    # Category RSS feed
    url(r'^feeds/posts/category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryPostsFeed()),
)

Note that the category RSS feed route is similar to the post RSS feed route, but accepts a slug parameter. We will use this to pass through the slug for the category in question. Also note we import the CategoryPostsFeed view. Now, we need to create that view. Fortunately, because it’s written as a Python class, we can extend the existing PostsFeed class. Open up your views file and amend it to look like this:

from django.shortcuts import get_object_or_404
from django.views.generic import ListView
from blogengine.models import Category, Post, Tag
from django.contrib.syndication.views import Feed
from django.utils.encoding import force_unicode
from django.utils.safestring import mark_safe
import markdown2

# Create your views here.
class CategoryListView(ListView):
    def get_queryset(self):
        slug = self.kwargs['slug']
        try:
            category = Category.objects.get(slug=slug)
            return Post.objects.filter(category=category)
        except Category.DoesNotExist:
            return Post.objects.none()


class TagListView(ListView):
    def get_queryset(self):
        slug = self.kwargs['slug']
        try:
            tag = Tag.objects.get(slug=slug)
            return tag.post_set.all()
        except Tag.DoesNotExist:
            return Post.objects.none()


class PostsFeed(Feed):
    title = "RSS feed - posts"
    description = "RSS feed - blog posts"
    link = '/'

    def items(self):
        return Post.objects.order_by('-pub_date')

    def item_title(self, item):
        return item.title

    def item_description(self, item):
        extras = ["fenced-code-blocks"]
        content = mark_safe(markdown2.markdown(force_unicode(item.text),
                                               extras = extras))
        return content


class CategoryPostsFeed(PostsFeed):
    def get_object(self, request, slug):
        return get_object_or_404(Category, slug=slug)

    def title(self, obj):
        return "RSS feed - blog posts in category %s" % obj.name

    def link(self, obj):
        return obj.get_absolute_url()

    def description(self, obj):
        return "RSS feed - blog posts in category %s" % obj.name

    def items(self, obj):
        return Post.objects.filter(category=obj).order_by('-pub_date')

Note that many of our fields don’t have to be explicitly defined as they are inherited from PostsFeed. We can’t hard-code the title, link or description because they depend on the category, so we instead define methods to return the appropriate text.

Also note get_object() - we define this so that we can ensure the category exists. If it doesn’t exist, then it returns a 404 error rather than showing an empty feed.

We also override items() to filter it to just those posts that are in the given category.

If you run the tests again, they should now pass:

$ python manage.py test blogengine
Creating test database for alias 'default'...
.........................
----------------------------------------------------------------------
Ran 25 tests in 5.867s

OK
Destroying test database for alias 'default'...

Let’s commit our changes:

git add blogengine/
git commit -m 'Implemented category RSS feed'

Now, we can get our category RSS feed, but how do we navigate to it? Let’s add a link to each category page that directs a user to its RSS feed. To do so, we’ll need to create a new template for category pages. First, let’s add some code to our tests to ensure that the right template is used at all times. Add the following to the end of the test_index method of PostViewTest:

        # Check the correct template was used
        self.assertTemplateUsed(response, 'blogengine/post_list.html')

Then, add this to the end of test_post_page:


        # Check the correct template was used
        self.assertTemplateUsed(response, 'blogengine/post_detail.html')

Finally, add this to the end of test_category_page:

        # Check the correct template was used
        self.assertTemplateUsed(response, 'blogengine/category_post_list.html')

These assertions confirm which template was used to generate which request.

Next, we head into our views file:

class CategoryListView(ListView):
    template_name = 'blogengine/category_post_list.html'

    def get_queryset(self):
        slug = self.kwargs['slug']
        try:
            category = Category.objects.get(slug=slug)
            return Post.objects.filter(category=category)
        except Category.DoesNotExist:
            return Post.objects.none()

    def get_context_data(self, **kwargs):
        context = super(CategoryListView, self).get_context_data(**kwargs)
        slug = self.kwargs['slug']
        try:
            context['category'] = Category.objects.get(slug=slug)
        except Category.DoesNotExist:
            context['category'] = None
        return context

Note that we first of all change the template used by this view. Then, we override get_context_data to add in additional data. What we’re doing is getting the slug that was passed through, looking up any category for which it is the slug, and returning it as additional context data. Using this method, you can easily add additional data that you may wish to render in your Django templates.

Finally, we create our new template:

{% extends "blogengine/includes/base.html" %}

    {% load custom_markdown %}

    {% block content %}
        {% if object_list %}
            {% for post in object_list %}
            <div class="post col-md-12">
            <h1><a href="{{ post.get_absolute_url }}">{{ post.title }}</a></h1>
            <h3>{{ post.pub_date }}</h3>
            {{ post.text|custom_markdown }}
            </div>
            {% if post.category %}
            <div class="col-md-12">
            <a href="{{ post.category.get_absolute_url }}"><span class="label label-primary">{{ post.category.name }}</span></a>
            </div>
            {% endif %}
            {% if post.tags %}
            <div class="col-md-12">
            {% for tag in post.tags.all %}
            <a href="{{ tag.get_absolute_url }}"><span class="label label-success">{{ tag.name }}</span></a>
            {% endfor %}
            </div>
            {% endif %}
            {% endfor %}
        {% else %}
            <p>No posts found</p>
        {% endif %}

        <ul class="pager">
        {% if page_obj.has_previous %}
        <li class="previous"><a href="http://matthewdaly.co.uk/{{ page_obj.previous_page_number }}/">Previous Page</a></li>
        {% endif %}
        {% if page_obj.has_next %}
        <li class="next"><a href="http://matthewdaly.co.uk/{{ page_obj.next_page_number }}/">Next Page</a></li>
        {% endif %}
        </ul>

        <a href="http://matthewdaly.co.uk/feeds/posts/category/{{ category.slug }}/">RSS feed for category {{ category.name }}</a>

    {% endblock %}

Note that the category has been passed through to the template and is now accessible. If you run the tests, they should now pass:

$ python manage.py jenkins
Creating test database for alias 'default'...
.........................
----------------------------------------------------------------------
Ran 25 tests in 7.232s

OK
Destroying test database for alias 'default'...

With that done. we can commit our changes:

git add templates/ blogengine/
git commit -m 'Added link to RSS feed from category page'

Next up, let’s implement another RSS feed for tags. First, we’ll implement our test:

      def test_tag_feed(self):
          # Create a post
          post = PostFactory(text='This is my *first* blog post')
          tag = TagFactory()
          post.tags.add(tag)
          post.save()

          # Create another post with a different tag
          tag2 = TagFactory(name='perl', description='The Perl programming language', slug='perl')
          post2 = PostFactory(text='This is my *second* blog post', title='My second post', slug='my-second-post')
          post2.tags.add(tag2)
          post2.save()

          # Fetch the feed
          response = self.client.get('/feeds/posts/tag/python/')
          self.assertEquals(response.status_code, 200)

          # Parse the feed
          feed = feedparser.parse(response.content)

          # Check length
          self.assertEquals(len(feed.entries), 1)

          # Check post retrieved is the correct one
          feed_post = feed.entries[0]
          self.assertEquals(feed_post.title, post.title)
          self.assertTrue('This is my <em>first</em> blog post' in feed_post.description)

          # Check other post is not in this feed
          self.assertTrue('This is my <em>second</em> blog post' not in response.content)

This is virtually identical to the test for the categroy feed, but we adjust it to work with the Tag attribute and change the URL. Let’s check that our test fails:

$ python manage.py test blogengine
Creating test database for alias 'default'...
.................F........
======================================================================
FAIL: test_tag_feed (blogengine.tests.FeedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 757, in test_tag_feed
    self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200

----------------------------------------------------------------------
Ran 26 tests in 5.760s

FAILED (failures=1)
Destroying test database for alias 'default'...

As before, we create a route for this:

from django.conf.urls import patterns, url
from django.views.generic import ListView, DetailView
from blogengine.models import Post, Category, Tag
from blogengine.views import CategoryListView, TagListView, PostsFeed, CategoryPostsFeed, TagPostsFeed

urlpatterns = patterns('',
    # Index
    url(r'^(?P<page>\d+)?/?$', ListView.as_view(
        model=Post,
        paginate_by=5,
        )),

    # Individual posts
    url(r'^(?P<pub_date__year>\d{4})/(?P<pub_date__month>\d{1,2})/(?P<slug>[a-zA-Z0-9-]+)/?$', DetailView.as_view(
        model=Post,
        )),

    # Categories
    url(r'^category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryListView.as_view(
        paginate_by=5,
        model=Category,
        )),

    # Tags
    url(r'^tag/(?P<slug>[a-zA-Z0-9-]+)/?$', TagListView.as_view(
        paginate_by=5,
        model=Tag,
        )),

    # Post RSS feed
    url(r'^feeds/posts/$', PostsFeed()),

    # Category RSS feed
    url(r'^feeds/posts/category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryPostsFeed()),

    # Tag RSS feed
    url(r'^feeds/posts/tag/(?P<slug>[a-zA-Z0-9-]+)/?$', TagPostsFeed()),
)

Next, we create our view:

class TagPostsFeed(PostsFeed):
    def get_object(self, request, slug):
        return get_object_or_404(Tag, slug=slug)

    def title(self, obj):
        return "RSS feed - blog posts tagged  %s" % obj.name

    def link(self, obj):
        return obj.get_absolute_url()

    def description(self, obj):
        return "RSS feed - blog posts tagged %s" % obj.name

    def items(self, obj):
        try:
            tag = Tag.objects.get(slug=obj.slug)
            return tag.post_set.all()
        except Tag.DoesNotExist:
            return Post.objects.none()

Again, this inherits from PostsFeed, but the syntax for getting posts matching a tag is slightly different because they use a many-to-many relationship.

We also need a template for the tag pages. Add this to the end of the test_tag_page method:


        # Check the correct template was used
        self.assertTemplateUsed(response, 'blogengine/tag_post_list.html')

Let’s create that template:

{% extends "blogengine/includes/base.html" %}

    {% load custom_markdown %}

    {% block content %}
        {% if object_list %}
            {% for post in object_list %}
            <div class="post col-md-12">
            <h1><a href="{{ post.get_absolute_url }}">{{ post.title }}</a></h1>
            <h3>{{ post.pub_date }}</h3>
            {{ post.text|custom_markdown }}
            </div>
            {% if post.category %}
            <div class="col-md-12">
            <a href="{{ post.category.get_absolute_url }}"><span class="label label-primary">{{ post.category.name }}</span></a>
            </div>
            {% endif %}
            {% if post.tags %}
            <div class="col-md-12">
            {% for tag in post.tags.all %}
            <a href="{{ tag.get_absolute_url }}"><span class="label label-success">{{ tag.name }}</span></a>
            {% endfor %}
            </div>
            {% endif %}
            {% endfor %}
        {% else %}
            <p>No posts found</p>
        {% endif %}

        <ul class="pager">
        {% if page_obj.has_previous %}
        <li class="previous"><a href="http://matthewdaly.co.uk/{{ page_obj.previous_page_number }}/">Previous Page</a></li>
        {% endif %}
        {% if page_obj.has_next %}
        <li class="next"><a href="http://matthewdaly.co.uk/{{ page_obj.next_page_number }}/">Next Page</a></li>
        {% endif %}
        </ul>

        <a href="http://matthewdaly.co.uk/feeds/posts/tag/{{ tag.slug }}/">RSS feed for tag {{ tag.name }}</a>

    {% endblock %}

This is virtually identical to the category template. You’ll also need to apply this template in the view for the tag list, and pass the tag name through as context data:

class TagListView(ListView):
    template_name = 'blogengine/tag_post_list.html'

    def get_queryset(self):
        slug = self.kwargs['slug']
        try:
            tag = Tag.objects.get(slug=slug)
            return tag.post_set.all()
        except Tag.DoesNotExist:
            return Post.objects.none()

    def get_context_data(self, **kwargs):
        context = super(TagListView, self).get_context_data(**kwargs)
        slug = self.kwargs['slug']
        try:
            context['tag'] = Tag.objects.get(slug=slug)
        except Tag.DoesNotExist:
            context['tag'] = None
        return context

Let’s run our tests:

$ python manage.py test blogengine
Creating test database for alias 'default'...
..........................
----------------------------------------------------------------------
Ran 26 tests in 5.770s

OK
Destroying test database for alias 'default'...

You may want to do a quick check to ensure your tag feed link works as expected. Time to commit:

git add blogengine templates
git commit -m 'Implemented tag feeds'

Moving our templates

Before we crack on with implementing search, there’s one more piece of housekeeping. In Django, templates can be applied at project level or at app level. So far, we’ve been storing them in the project, but we would like our app to be as self-contained as possible so it can just be dropped into future projects where we need a blog. That way, it can be easily overridden for specific projects. You can move the folders and update the Git repository at the same time with this command:

git mv templates/ blogengine/

We run the tests to make sure nothing untoward has happened:

$ python manage.py test
Creating test database for alias 'default'...
..........................
----------------------------------------------------------------------
Ran 26 tests in 5.847s

OK
Destroying test database for alias 'default'...

And we commit:

git commit -m 'Moved templates'

Note that git mv updates Git and moves the files, so you don’t need to call git add.

Implementing search

For our final task today, we will be implementing a very simple search engine. Our requirements are:

  • It should be in the header, to allow for easy access from anywhere in the front end.
  • It should search the title and text of posts.

First, we’ll write our tests:

class SearchViewTest(BaseAcceptanceTest):
    def test_search(self):
        # Create a post
        post = PostFactory()

        # Create another post
        post2 = PostFactory(text='This is my *second* blog post', title='My second post', slug='my-second-post')

        # Search for first post
        response = self.client.get('/search?q=first')
        self.assertEquals(response.status_code, 200)

        # Check the first post is contained in the results
        self.assertTrue('My first post' in response.content)

        # Check the second post is not contained in the results
        self.assertTrue('My second post' not in response.content)

        # Search for second post
        response = self.client.get('/search?q=second')
        self.assertEquals(response.status_code, 200)

        # Check the first post is not contained in the results
        self.assertTrue('My first post' not in response.content)

        # Check the second post is contained in the results
        self.assertTrue('My second post' in response.content)

Don’t forget to run the tests to make sure they fail:

 python manage.py test
Creating test database for alias 'default'...
..........................F
======================================================================
FAIL: test_search (blogengine.tests.SearchViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 819, in test_search
    self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200

----------------------------------------------------------------------
Ran 27 tests in 6.919s

FAILED (failures=1)
Destroying test database for alias 'default'...

With that done, we can add the search form to the header:

“` html blogengine/templates/blogengine/includes/base.html <!DOCTYPE html>

{% block title %}My Django Blog{% endblock %}

    <!-- Place favicon.ico and apple-touch-icon.png in the root directory -->

    {% load staticfiles %}
    <link rel="stylesheet" href="{% static 'bower_components/html5-boilerplate/css/normalize.css' %}">
    <link rel="stylesheet" href="{% static 'bower_components/html5-boilerplate/css/main.css' %}">
    <link rel="stylesheet" href="{% static 'bower_components/bootstrap/dist/css/bootstrap.min.css' %}">
    <link rel="stylesheet" href="{% static 'bower_components/bootstrap/dist/css/bootstrap-theme.min.css' %}">
    <link rel="stylesheet" href="{% static 'css/main.css' %}">
    <link rel="stylesheet" href="{% static 'css/code.css' %}">
    <script src="{% static 'bower_components/html5-boilerplate/js/vendor/modernizr-2.6.2.min.js' %}"></script>
</head>
<body>
    <!--[if lt IE 7]>
        <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
    <![endif]-->

    <!-- Add your site or application content here -->

    <div id="fb-root"></div>
    <script>(function(d, s, id) {
        var js, fjs = d.getElementsByTagName(s)[0];
        if (d.getElementById(id)) return;
            js = d.createElement(s); js.id = id;
            js.src = "//connect.facebook.net/en_GB/all.js#xfbml=1";
            fjs.parentNode.insertBefore(js, fjs);
        }(document, 'script', 'facebook-jssdk'));</script>

    <div class="navbar navbar-static-top navbar-inverse">
        <div class="container-fluid">
            <div class="navbar-header">
                <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#header-nav">
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                </button>
                <a class="navbar-brand" href="http://matthewdaly.co.uk/">My Django Blog</a>
            </div>
            <div class="collapse navbar-collapse" id="header-nav">
                <ul class="nav navbar-nav">
                    {% load flatpages %}
                    {% get_flatpages as flatpages %}
                    {% for flatpage in flatpages %}
                    <li><a href="{{ flatpage.url }}">{{ flatpage.title }}</a></li>
                    {% endfor %}
                    <li><a href="http://matthewdaly.co.uk/feeds/posts/">RSS feed</a></li>

                    <form action="/search" method="GET" class="navbar-form navbar-left">
                        <div class="form-group">
                            <input type="text" name="q" placeholder="Search..." class="form-control"></input>
                        </div>
                        <button type="submit" class="btn btn-default">Search</button>
                    </form>
                </ul>
            </div>
        </div>
    </div>

    <div class="container">
        {% block header %}
            <div class="page-header">
                <h1>My Django Blog</h1>
            </div>
        {% endblock %}

        <div class="row">
            {% block content %}{% endblock %}
        </div>
    </div>

    <div class="container footer">
        <div class="row">
            <div class="span12">
                <p>Copyright &copy; {% now "Y" %}</p>
            </div>
        </div>
    </div>

    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
    <script>window.jQuery || document.write('<script src="{% static 'bower_components/html5-boilerplate/js/vendor/jquery-1.10.2.min.js' %}"><\/script>')</script>
    <script src="{% static 'bower_components/html5-boilerplate/js/plugins.js' %}"></script>
    <script src="{% static 'bower_components/bootstrap/dist/js/bootstrap.min.js' %}"></script>

    <!-- Google Analytics: change UA-XXXXX-X to be your site's ID. -->
    <script>
        (function(b,o,i,l,e,r){b.GoogleAnalyticsObject=l;b[l]||(b[l]=
        function(){(b[l].q=b[l].q||[]).push(arguments)});b[l].l=+new Date;
        e=o.createElement(i);r=o.getElementsByTagName(i)[0];
        e.src='//www.google-analytics.com/analytics.js';
        r.parentNode.insertBefore(e,r)}(window,document,'script','ga'));
        ga('create','UA-XXXXX-X');ga('send','pageview');
    </script>
</body>

“`

Now we’ll actually implement our search. Implementing search using Django’s generic views can be fiddly, so we’ll write our search view as a function instead. First, amend the imports at the top of your view file to look like this:

from django.shortcuts import get_object_or_404, render_to_response
from django.core.paginator import Paginator, EmptyPage
from django.db.models import Q
from django.views.generic import ListView
from blogengine.models import Category, Post, Tag
from django.contrib.syndication.views import Feed
from django.utils.encoding import force_unicode
from django.utils.safestring import mark_safe
import markdown2

Next, add the following code to the end of the file:

def getSearchResults(request):
    """
    Search for a post by title or text
    """
    # Get the query data
    query = request.GET.get('q', '')
    page = request.GET.get('page', 1)

    # Query the database
    if query:
        results = Post.objects.filter(Q(text__icontains=query) | Q(title__icontains=query))
    else:
        results = None

    # Add pagination
    pages = Paginator(results, 5)

    # Get specified page
    try:
        returned_page = pages.page(page)
    except EmptyPage:
        returned_page = pages.page(pages.num_pages)

    # Display the search results
    return render_to_response('blogengine/search_post_list.html',
                              {'page_obj': returned_page,
                               'object_list': returned_page.object_list,
                               'search': query})

As this is the first time we’ve written a view without using generic views, a little explanation is called for. First we get the values of the q and page parameters passed to the view. q contains the query text and page contains the page number. Note also that our page defaults to 1 if not set.

We then use the Q object to perform a query. The Django ORM will AND together keyword argument queries, but that’s not the behaviour we want here. Instead we want to be able to search for content in the title or text, so we need to use a query with an OR statement, which necessitates using the Q object.

Next, we use the Paginator object to manually paginate the results, and if it raises an EmptyPage exception, to just show the last page instead. Finally we render the template blogengine/search_post_list.html, and pass through the parameters page_obj for the returned page, object_list for the objects, and search for the query.

We also need to add a route for our new view:


    # Search posts
    url(r'^search', 'blogengine.views.getSearchResults'),

Finally, let’s create a new template to show our results:

{% extends "blogengine/includes/base.html" %}

    {% load custom_markdown %}

    {% block content %}
        {% if object_list %}
            {% for post in object_list %}
            <div class="post col-md-12">
            <h1><a href="{{ post.get_absolute_url }}">{{ post.title }}</a></h1>
            <h3>{{ post.pub_date }}</h3>
            {{ post.text|custom_markdown }}
            </div>
            {% if post.category %}
            <div class="col-md-12">
            <a href="{{ post.category.get_absolute_url }}"><span class="label label-primary">{{ post.category.name }}</span></a>
            </div>
            {% endif %}
            {% if post.tags %}
            <div class="col-md-12">
            {% for tag in post.tags.all %}
            <a href="{{ tag.get_absolute_url }}"><span class="label label-success">{{ tag.name }}</span></a>
            {% endfor %}
            </div>
            {% endif %}
            {% endfor %}
        {% else %}
            <p>No posts found</p>
        {% endif %}

        <ul class="pager">
        {% if page_obj.has_previous %}
        <li class="previous"><a href="http://matthewdaly.co.uk/search?page={{ page_obj.previous_page_number }}&q={{ search }}">Previous Page</a></li>
        {% endif %}
        {% if page_obj.has_next %}
        <li class="next"><a href="http://matthewdaly.co.uk/search?page={{ page_obj.next_page_number }}&q={{ search }}">Next Page</a></li>
        {% endif %}
        </ul>

    {% endblock %}

Let’s run our tests:

$ python manage.py test
Creating test database for alias 'default'...
...........................
----------------------------------------------------------------------
Ran 27 tests in 6.421s

OK
Destroying test database for alias 'default'...

Don’t forget to do a quick sense check to make sure it’s all working as expected. Then it’s time to commit:

git add blogengine/
git commit -m 'Implemented search'

And push up your changes:

git push origin master
git push heroku master

And that’s the end of this instalment. Please note this particular search solution is quite basic, and if you want something more powerful, you may want to look at Haystack.

As usual, you can get this lesson with git checkout lesson-7 - if you have any problems, the repository should be the first place you look for answers as this is the working code base for the application, and with judicious use of a tool like diff, it’s generally pretty easy to track down most issues.

In our next, and final instalment, we’ll cover:

  • Tidying everything up
  • Implementing an XML sitemap for search engines
  • Optimising our site
  • Using Fabric to make deployment easier

Hope to see you then!


Chaussette and Circus as a new way to deploy your Django applications

Aug 24 2014 [Archived Version] □ Published at RkBlog - Python, Linux, Astronomy under tags  django web framework tutorials

Many Python web applications run on production with the help of gunicorn or uwsgi. There are however other solutions that may turn out to be even better than the currently popular solutions. One of alternatives is chaussette and circus. Chaussette is a WSGI server that can serve WSGI applications like your Django project. Circus is an application to control and monitor processes and sockets. It can manage chaussette, celery or other sockets and processes making it a manager of all project services.


Password reset for mobile application with Django and Tastypie

Aug 24 2014 [Archived Version] □ Published at michalcodes4life under tags  django programming python rest tastypie

Django offers ready password reset views for web applications. With a couple of tweaks it can be converted into an API and used in your mobile applications. Form This PasswordResetForm is essentialy a copy of django.contrib.auth.forms.PasswordResetForm with a few changes (outlined in the comments). This code is from Django 1.7, previous versions slightly differ. Email […]


What's New With WhisperGifts

Aug 24 2014 [Archived Version] □ Published at Ross Poulton under tags  django whispergifts

It's been a busy few months for WhisperGifts, my bridal gift registry service built with Django. This post rounds up a few recent additions and changes, including technical details for those who want to do something similar. Highlights include weather forecasting, REST APIs, and JavaScript bookmarklets.


A thank you to the companies that support Django's infrastructure

Aug 23 2014 [Archived Version] □ Published at The Django weblog

It's not surprising that a project developing an open source web framework will, from time to time, require server resources. When a project gets to be the size of Django, the server resources needed to support the project can get expensive.

Luckily, we don't have to cover these costs ourselves. We have a lot of very generous companies in our community that donate server resources to the Django Software Foundation (DSF) so that Django can keep running.

There are two companies in particular that deserve to be pointed out for their generous contributions supporting the Django project's infrastructure.

Firstly, Rackspace donates server instances that we use to run the main djangoproject.com web site, plus docs.djangoproject.com. These sites generate a lot of traffic - millions of hits per month - and they're the most visible parts of Django's public presence. In addition, Rackespace donates servers to run Django's Continuous Integration (CI) servers. These servers run the full Django test suite on every supported database and platform to make sure that we don't ship a release with a known problem on a supported platform.

Secondly, Heroku recently donated $1500 in service credits to the DSF. This matches a similar grant they gave the DSF last year. We use Heroku to host such services as dashboard.djangoproject.com, people.djangoproject.com, djangosnippets.org, and a Sentry instance.

A huge thanks to Rackspace and Heroku for their generous contributions to the Django community!

UPDATE 25 August 7:00 CDT: Another company that we neglected to mention is Divio. Django's CI infrastructure has only recently been moved to Rackspace; for several years prior to this move, Divio provided the hosting for our CI tools. Although Divio is no longer providing this resource, it was invaluable for the several years that we were using it. A belated Thank You to goes to Divio for their generous contribution.


Productivity Tools & Processes

Aug 23 2014 [Archived Version] □ Published at Nerdy Dork under tags  programming &amp; internet

Last week as I was driving to a user group meeting, I listed to a podcast episode. The subject was managing email. Since then I have cleaned up my inbox and I have been at inbox zero or close to it all week. Between my 5 email accounts I have 1 message in my inbox […]

The post Productivity Tools & Processes appeared first on Dustin Davis.


Django REST framework Sprint at DjangoCon US 2014

Aug 21 2014 [Archived Version] □ Published at José Padilla under tags  django djangocon

image

This’ll be my first time attending and speaking at DjangoCon US. I’ll be talking about JSON Web Tokens, Django, and Django REST Framework.

I’ve been talking about working on a Sprint for Django REST Framework with Tom Christie on Twitter. Tom pointed out a list of decent candidates to work on if we get some people together. We’ll have a tagged list of obvious bug candidates before the sprint. He’s also planning on helping us out remotely those days.

It seems like a great idea to celebrate the successful Django REST framework 3 campaign on Kickstarter and with the timing of DjangoCon, “kickstart” the development of upcoming versions.

So, I’m recruiting anyone who’d be interested and available in working together. Everyone’s welcome to join in - if you use Django REST framework and would like to become more familiar with its internals while working together with other developers with various levels of expertise. Sprints will be held Friday and Saturday Sep 5-6. 
If you’re interested please send me an email so we can start getting organized. If you know someone who might be interested please share this with them.


PyDDF Python Sprint 2014

Aug 21 2014 [Archived Version] □ Published at eGenix.com News & Events

The following text is in German, since we're announcing a Python sprint in Düsseldorf, Germany.

Ankündigung

Der erste PyDDF Python Sprint 2014 in Düsseldorf:

Samstag, 27.09.2014, 10:00-18:00 Uhr
Sonntag, 28.09.2014, 10:00-18:00 Uhr
Seminarraum 25.41.00.45
(Gebäude 25.41, Erdgeschoss, Raum 45) des
ZIM der HHU Düsseldorf

Informationen

Das Python Meeting Düsseldorf (PyDDF) veranstaltet zusammen mit dem ZIM der Heinrich-Heine-Universität Düsseldorf ein Python Sprint Wochenende im September.

Der Sprint findet am Wochenende 27/28.09.2013 im Seminarraum 25.41.00.45 (Gebäude 25.41, Erdgeschoss, Raum 45) des ZIM der HHU Düsseldorf stattfinden:
Folgende Themengebiete haben wir als Anregung angedacht:
  • Openpyxl
Openpyxl ist eine Python Bibliothek, mit der man Excel 2010 XLSX/XLSM Dateien lesen und schreiben kann.

Charlie ist Co-Maintainer des Pakets und würde gerne an folgenden Themen arbeiten:

- ElementTree Implementation des lxml.etree.xmlfile Moduls (context manager)
- Co-Routines für die Serialisierung
- Python Code-Object-Generierung anhand des Schemas
  • HTTP Audio Streaming für Mopidy
Mopidy ist ein MPD Musikserver, der viele Internet-Streaming-Dienste abonnieren kann, diese jedoch nur über lokale Audiogeräte ausgibt.

Es wäre schön, wenn man auch Internetradios anschließen könnte, wie z.B. die Squeezebox. Es gibt dazu schon ein Ticket, auf dem man vermutlich aufbauen könnte:

- https://github.com/mopidy/mopidy/issues/56

Ziel wäre es, eine Mopidy Extension zu schreiben, die dieses Feature umsetzt.
Natürlich kann jeder Teilnehmer weitere Themen vorschlagen, z.B.
  • Kivy (Python auf Android/iOS)
  • RaspberryPi (wir werden ein paar davon mitbringen)
  • FritzConnection (Python API für die Fritzbox)
  • OpenCV (Bilder von Webcams mit Python verarbeiten)
  • u.a.
Alles weitere und die Anmeldung findet Ihr auf der Sprint Seite:
Teilnehmer sollten sich zudem auf der PyDDF Liste anmelden, da wir uns dort koordinieren:
Wir haben nur begrenzten Platz im Seminarraum, daher wäre es gut, wenn wir die ungefähre Anzahl Teilnehmer schon in Vorfeld einplanen könnten. Platz ist für max. 30 Teilnehmer.

Über das Python Meeting Düsseldorf

Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet.

Einen guten Überblick über die Vorträge bietet unser PyDDF YouTube-Kanal, auf dem wir Videos der Vorträge nach den Meetings veröffentlichen.

Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf.

Viel Spaß !

Marc-Andre Lemburg, eGenix.com

Published: 2013-03-20


django-planet aggregates posts from Django-related blogs. It is not affiliated with or endorsed by the Django Project.

Social Sharing

Feeds

Tag cloud

admin administration advanced ajax apache api app appengine app engine apple aprendiendo python architecture articles asides audrey authentication automation backup bash basics bitbucket blog blog action day book books buildout business cache celery celerycrawler challenges cherokee choices class-based-views cliff cloud code coding command configuration couchdb css d data database databases debian deploy deployment developers development digitalocean django djangocon django-rest-framework django templates documentation dojango dojo dreamhost dughh eclipse education email encoding error events extensions fabric facebook family fashiolista fedora field filter fix flash flask form forms friends gae gallery games geek general gentoo gis git github gnome google google app engine gunicorn hackathon hacking hamburg heroku holidays hosting howto how-tos html http i18n image install intermediate internet ios iphone java javascript jobs journalism jquery json justmigrated linear regression linkedin linode linux mac machine learning math memcached mercurial meta migration mirror misc model models mod_wsgi mongodb mozilla mvc mysql nelenschuurmans newforms news nginx nosql ogólne open source open-source orm osx os x ottawa paas performance philosophy php pi pil pinax pip piston planet plone plugin pony postgres postgresql ppoftw private programmieren programming programming &amp; internet project projects pycon pygrunn pyladies pypi pypy python python3 quick tips quora rabbitmq rails rant ratnadeep debnath redis release request resolutions rest review rtnpro ruby science script security server setup simple smiley snaking software software development south sql ssh ssl static storage supervisor svn sysadmin tag talk nerdy to me tastypie tdd techblog technical technology template templates test testing tests tip tools tornado transifex travel tumbles tutorial twitter twoscoops typo3 ubuntu uncategorized unicode unix usergroup uwsgi uxebu virtualenv virtualenvwrapper web web 2.0 web design &amp; development webdev web development webfaction whoosh windows wordpress work