Python for web services

Why python?

Python's home: http://python.org

There are several good programming languages for creating web sites. For certain tasks, one language may be better than another. However, fluency in languages is the only way to evaluate this.

Programming is mostly learning by doing.

Features of Python

Example: a8e.py

A programming for making long works indecipherably short

Problem: how to make abbreviations for words? Several terms have evolved from taking the first and last letter of a long word and substituting the number of letters in the middle: 

#!/usr/bin/env python

import sys
import urllib2

def a8e(text):
  text = text.split()
  retval = []
  for word in text:
    if len(word) < 4:
      retval.append(word)
    else:
      retval.append(word[0] + '%d' % (len(word) - 2) + word[-1])
  return ' '.join(retval)

def main(args=sys.argv[1:]):
  if len(args) == 1 and (args[0].startswith('http://')
                         or args[0].startswith('https://')):
    text = urllib2.urlopen(args[0]).read()
  else:
    text = ' '.join(args)
  # TODO: read from stdin if no args
  print a8e(text)

if __name__ == '__main__':
  main()
http://k0s.org/hg/config/file/tip/python/a8e.py

Constructs in a8e.py

Let's walk through the a8e.py program and see what's going on:

    import sys
    import urllib2

import the modules you need

    def a8e(text):
declare a function. this function takes exactly one argument 
      text = text.split()
split the passed-in string by whitespace (spaces, tabs, and newlines) 
  retval = []
declare a variable, retval, as an empty list; this will be modified to be the value that is returned from the a8e() function 
  for word in text:
for each word in the text ... 
    if len(word) < 4:
      retval.append(word)
if the word has less than four characters, just append it to the list retval 
    else:
      retval.append(word[0] + '%d' % (len(word) - 2) + word[-1])
otherwise, take the first letter of the word, the (string of) the length of the word, and the last letter of the word and append it to the list retval 
  return ' '.join(retval)
make a string by joining the words in reval with a single space and return it

Good Development Practices

Using Python Interactively

Run python from the command line:

> python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Limitations:

Navigating Python

help()
Tells you about an object, if the programmer was nice enough to include docstrings
dir()
Tells you what is in an "object"
__file__
The location of a module on disk (when available)

Basic Data Types in Python

Modules
import sys; from webob import Request
Strings
'a foo bar'
Lists
[1,2,3,4,'five']
Tuples
very similar to lists, but cannot be changed in place: (1,2,3,4,'five')
Dictionaries
Known as hashes in other languages: {1: 'one', 2: 'two', 3: 'three', 4: 'four'}

Mutable and Immutable Types

Some objects (instances of data types) may be changed in place. These are called mutable.

Other objects cannot be changed in place. These are called immutable.

Mutable Types

Immutable Types

Usually functions that operate on immutable types return a new instance of the same type.

Example: string.translate returns a new copy of the string

Note also in passing that dictionary keys must be immutable.

Importing Python Modules

Importing is done with the import command: 

import sys
print sys.platform

You may also import individual attributes from modules: 

from sys import platform
print platform

Your module must exist on some component of sys.path to be importable (e.g. in the current directory)

Debugging Python Code

pdb is your friend

import pdb; pdb.set_trace()

Example:

def foo(bar):
  return bar*bar

print foo(2)

# uncomment the next line to activate pdb and fix the problem!
#import pdb; pdb.set_trace()

print foo([2,3,4])
print foo(3)

Note: help() does not work inside a pdb (but dir() does)

What is virtualenv?

virtualenv is a virtual environment to isolate python software. virtualenv allows you to isolate a development environment.

Installing virtualenv

- with easy_install:

easy_install virtualenv

virtualenv is one of the few things you might want to install in your global site-packages

- in development mode:

hg clone http://bitbucket.org/ianb/virtualenv
cd virtualenv
python setup.py develop

Alternatively, you can run virtualenv.py directly out of the checkout directory

Installing python software: a quick lesson

Usually, you'll want to make a virtualenv for a project or when trying out software

virtualenv ${project}
. ${project}/bin/activate 
# on windows: ${project}/Scripts/activate.bat

For software on http://pypi.python.org:

easy_install ${package}

For software in development mode:

  1. Make a src directory in your virtualenv.
    Example: mkdir src
  2. Get the source.
    Example: hg clone http://k0s.org/hg/webob_view/
  3. Set up the source for development mode. (I hope you're in a virtualenv so you don't hose your global site-packages!)
    Example: cd webob_view; python setup.py develop

For this class, you'll want to easy_install paste, webob, and various other packages you might want along the way.

WSGI

Python's Web Server Gateway Interface

python's Web Server Gateway Interface is a long name for the standard that software should conform to. This way, a developer may write software that can work with multiple python web servers.

WSGI provides a contract so python developers can make use of the same basic structure for request processing. This allows python code to be portable between server implementations if they implement the WSGI spec.

Basic WSGI application:

def application(environ, start_response):
  text = 'hello world'
  start_response("200 OK", [('Content-Type', 'text/plain'),
                            ('Content-Length', str(len(text)))])
  return [text]

basic_wsgi.py

Here, application is a function, but it may be anything callable (e.g. an object) that takes environ, start_response

WebOb

WebOb is a simplerequest-response object. It is used by Pylons amongst other frameworks and libraries.

http://pythonpaste.org/webob/ ← read this webpage! (homework)

Getting query string parameters with WebOb:

from webob import Request

...

  def __call__(self, environ, start_response):
    request = Request(environ)
    value = request.GET.get('key') 
    # for the url http://example.com/?key=elephants this will return 'elephants'

Getting POST data with WebOb:

request.POST.get('key')

Getting headers with WebOb:

request.headers.get('Content-Length')

Other functionality:

request.path_info
the PATH_INFO string Example: '/'
request.cookies
dictionary of all cookies sent by the client
request.method
string of the request method Example: 'GET'
request.remote_user
string identifier of the authenticated user (or None if none)

and lots more! when you have a working webapp, you can import pdb; pdb.set_trace() and use dir() to look around your request

WebOb makes web programming easy

The "Hello world!" web service

from webob import Request, Response

def hello(environ, start_response):
  request = Request(environ)
  response = Response(content_type='text/plain',
                      body='Hello world!')
  return response(environ, start_response)

from paste.httpserver import serve
serve(hello)
helloworld.py

Serving web with python

Because of WSGI, it shouldn't (*) matter much what web server you use for testing, as your web service code should work with any WSGI server

There is an excellent comparison of WSGI server performance complete with how you would deploy a sample application (you should read!):

Frameworks vs. web services

Python web frameworks:

Web Frameworks vs. Web Servers

What is the difference between a web framework and a web server?

http://wiki.python.org/moin/WebFrameworks describes Paste as a web framework whereas here I describe it as a server. Which is it?

Web framework is a vague term and generally describes any toolkit on which you can build web sites.

Turning a8e.py into a web service

#!/usr/bin/env python

import sys
import urllib2
from webob import Request, Response

def a8e(text):
  text = text.split()
  retval = []
  for word in text:
    if len(word) < 4:
      retval.append(word)
    else:
      retval.append(word[0] + '%d' % (len(word) - 2) + word[-1])
  return ' '.join(retval)

def a8eweb(environ, start_response):
  request = Request(environ)
  text = ' '.join(request.GET.keys())
  response = Response(content_type='text/plain',
                      body=a8e(text))
  return response(environ, start_response)

def main():
  from paste.httpserver import serve
  serve(a8eweb)

if __name__ == '__main__':
  main()
ae8web.py

Redirecting with WebOb

from webob import exc

# ...

  def __call__(environ, start_response):

    # ... do some stuff
    
    raise exc.HTTPSeeOther(location='http://example.com/')
    # XXX does not work until python 2.5!

    # on older python do:
    response = exc.HTTPSeeOther(location='http://example.com/')
    return response(environ, start_response)

What is paste?

Paste is a lot of things:

But we're mostly going to use the webserver (Feel free to play with other web servers! Paste isn't commonly used in production)

paste .ini files

paste .ini files tell you how to assemble a website from WSGI applications 

#!/usr/bin/env paster

[DEFAULT]
debug = true
email_to = jhammel@mozilla.com
smtp_server = localhost
error_email_from = paste@localhost

[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 7654

[composite:main]
use = egg:Paste#urlmap
/ = hello

set debug = false

[app:hello]
paste.app_factory = hello.factory:factory
hello.ini

Breaking this down:

There are several sections to this .ini file:

[DEFAULT]
section for variables to be used in the file
[server:main]
sever configuration
[composite:main]
how the web site is laid out
[app:hello]
configuration for the hello webapp
#!/usr/bin/env paster ← she-bang to run this file with

[DEFAULT]
debug = true
email_to = jhammel@mozilla.com
smtp_server = localhost
error_email_from = paste@localhost

[server:main]
use = egg:Paste#http  ← use the Paste web server
host = 0.0.0.0        ← serve on this IP address (the local machine)
port = 7654           ← serve on port 7654

[composite:main]
use = egg:Paste#urlmap ← use this to map URLS
/ = hello              ← map the root use to the hello application

set debug = false ← don't use debug in production!

[app:hello]
paste.app_factory = hello.factory:factory ← factory to create the hello application

paste factories

paste factories tell how to compose a WSGI application

In order to use a paste factory, it must be available on your path

Storage

What do you do if you need to store data?

the type of storage should be appropriate to the problem