Nick Carlson's Blog A Programmer's Dystopia

Code dump

Crappilly written code, but I need to post it to get it out of my head

import socket
from urllib2 import Request, urlopen, URLError, HTTPError
from httplib import BadStatusLine
from Queue import Queue
import threading
from random import randint
 
socket.setdefaulttimeout(5)
 
class UrlChecker(threading.Thread):
    def __init__(self, url):
        self.url = url
        self.result = None
        threading.Thread.__init__(self)
 
    def GetResult(self):
        return self.result
 
    def run(self):
        req = Request(self.url)
        #print "Request dispatched on: " + self.url
        try:
            response = urlopen(req)
            print self.url + " Is ok!"
            return response
        except HTTPError, e:
            #print 'The server couldn't fulfill the request.'
            print 'Error code: ', e.code
            return e.code
        except URLError, e:
            #print 'We failed to reach a server.'
            #print 'Reason: ', e.reason
            return e.reason
        except BadStatusLine, e:
            #print "!! Status bad !! "
            return "bad status"
        else:
            print "All good"
            # everything is fine
 
def get_files(files):
    def producer(q, files):
        for url in files:
            thread = UrlChecker(url)
            thread.start()
            q.put(thread, True)
 
    finished = []
    def consumer(q, total_files):
        while len(finished) < total_files:
            thread = q.get(True)
            thread.join(3)
            finished.append(thread.GetResult())
 
    q = Queue(200)
    prod_thread = threading.Thread(target=producer, args=(q, files))
    cons_thread = threading.Thread(target=consumer, args=(q, len(files)))
    prod_thread.start()
    cons_thread.start()
    prod_thread.join(3)
    cons_thread.join(3)
 
files = []
for i in range(1000):
    files.append("http://" + str(randint(1,255)) + "." + str(randint(1,255)) + "." + 
                str(randint(1,255)) + "." + str(randint(1,255)))
 
get_files(files)
 
 
[/code]


November 8th, 2008 / Trackback

Howto Install Webpy

My new favorite web framework runs on python. It's called Web.py. And I'm not the only one who likes it. The guys over at Reddit seem to have taken an interest in it as well. In this post, I'm going to show you how to set up a development environment for Web.py.

First, you'll want to make sure you have python installed. Webpy will run on older versions of python, but I recommend 2.5x.

Next we'll install MySql. You have a few options here:

  • If you're on a Mac, download Mysql and install it via package.
  • If you're on linux, run apt-get or yum to install mysql.
  • If you're on windows, just download and install Xampp.

No matter what OS you're running, go ahead and install Xampp. It's the easiest way to get pypmyadmin up and running. Next, head over to webpy's site and download the latest version (0.23 as of this writing). Extract the archive's files and run the following command.

python setup.py install


Since we'll be using mysql with webpy, lets install a popular mysql library for python. You can download it here. If you're on windows, you can go ahead and download the exe. If you're on linux, run the following command:

sudo easy_install MySQL-python


If you're on a mac, there's a bit more work involved. As of this writing, I'm using MySQL version 5.0.51b and MySQL-python version 1.2.2. You'll need to extract the files from the MySQL-python tarball and edit setup_posix.py. Replace the following line:

mysql_config.path = "mysql_config"


with...

mysql_config.path = "/usr/local/mysql-5.0.51b-osx10.5-x86/bin/mysql_config"


The actual path may vary if you are using a different version of mysql, so be aware of that. Once that is done, run the following command in the MySQL-python directory:

sudo python setup.py build


If all goes as planned, MySQL-python will be built and copied over to your python installation.

Congratulations! Web.py is officially installed. In my next post I'll show you how to make a simple Web.py application. If you can't wait, head over to webpy's official site and check out their tutorial.

June 10th, 2008 / Tags: webpy, web.py, tutorial, python, framework / Trackback

A Simple Webpy Tutorial

Today we'll be writing a simple todo list application. My goal is not to show you the finer points of todo lists, but rather to show you how to properly set up a webpy project for small to medium sized applications.

First, let's look at a very simple "hello world" example.

import web
 
urls = ('/', 'index')
 
class index:
    def GET(self):
        print "Hello, world!"
 
if __name__ == "__main__": web.run(urls, globals())


To run this, save the code in a file. Any name will do, but we'll call it application.py. Then run the file from the command line with:

python application.py


Then point your browser to http://localhost:8080.

The important points to take away from the code above are as follows:

URLs are routed via the urls tuple at the top of the file. The first value indicates the exposed URL. Regular expressions can be used here. If you've had any experience with routing URLs via htaccess files, you'll be right at home. The second value indicates the class to which the request will be routed. Take this example:

urls = (
'/',                        'index',
# a request to http://localhost:8080 will be routed to to the index
# class within this file
'/blog/post/(d+)/',         'app.controllers.posts.read',
# a request to http://localhost:8080/blog/post/28/ will be routed to the
# posts class with the value 28 being passed. According to our urls tuple,
# the posts class is located in the /apps/controllers directory.
 
# /
# ..application.py
# ..app/
# ....controllers/
# ......posts.py
)


A Todo List

Now lets write an actual application using web.py. Go ahead and download the application and have a look at the files and directory structure. Here's a screenshot of what the application looks like.

Within this application, we'll do the following.

  • Separate our code using a traditional MVC split
  • Establish a "public" folder for serving static files (images, javascript, etc)
  • Connect to a MySql database and use web.py's internal db abstraction layer
  • Use a traditional CRUD approach to our controller design
  • Expose both POST and GET for RESTful URLs

First, some formalities: the config file and database schema.

config.py

...
# connect to the database
web.config.db_parameters = dict(dbn='mysql', db='todo', user='root',
pw='')
...
# set our global base template
base = web.template.render('app/views/base/', cache=cache)
...


Connecting to a mysql database is pretty simple. In this case, our database name is "todo". The base variable set's the global base template.

schema.sql

CREATE TABLE IF NOT EXISTS `items` (
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `body` varchar(255) NOT NULL,
    PRIMARY KEY  (`id`)
);


Our application.py file from above has changed a bit. Remember, this is the file that's called when instantiating the application.

application.py

...
import app.controllers
 
urls = (
# This first entry is for our public files such as css, js, and images
'/(public)/.*',                         'app.controllers.public.public',
# our site's root maps to the index function within the items controller
'/',                                    'app.controllers.items.index',
'/items/add/',                          'app.controllers.items.create',
'/items/(d+)/',                         'app.controllers.items.read',
'/items/(d+)/edit/',                    'app.controllers.items.update',
'/items/delete/',                       'app.controllers.items.destroy'
)
...


You'll notice that we're now using regular expressions for our URL routing. Also notice that only one URL exists for updating an item. We're able to achieve this since web.py allows for the exposure of POST and GET. Take a look at the items controller:

items.py (controller)

...
from app.models import item
 
# set our template path, read cache setting from config file (false)
view = web.template.render('app/views/items/', cache=config.cache)
 
# CRUD in full effect (Create, Read, Update, Destroy)
 
class index:
    def GET(self):
        # "readall" is a function in our "item" model
        item_list = item.readall()
        # webpy allows for nest templates.
        # item_list is passed to /views/items/index.html, which is then
        # passed to /views/base/layout.html
        print config.base.layout(view.index(item_list))
 
class create:
    # Here we are exposing HTTP's POST method rather than GET.
    def POST(self):
        i = web.input()
        item.create(i.body)
        # seeother() is webpy's standard redirect function
        web.seeother('/')
 
class read:
    def GET(self, id):
        this_item = item.read(id)
        print config.base.layout(view.read(this_item))
 
class update:
    def GET(self, id):
        this_item = item.read(id)
        print config.base.layout(view.edit(this_item))
 
    def POST(self, id):
        i = web.input()
        item.update(id, i.body)
        web.seeother('/')
 
class destroy:
    def POST(self):
        i = web.input()
        item.destroy(i.id)
        web.seeother('/')


item.py (model)

...
# Webpy comes with a simple database abstraction layer. The standard
# insert, select, update and delete calls are available.
 
def create(body):
    return web.insert('items', body = body)
 
def update(id, body):
    return web.update('items', where = 'id = ' +id, body=body)
 
def destroy(id):
    return web.delete('items', where = 'id = ' + id)
 
def read(id):
    this_item = web.select('items', where = 'id = ' + id, limit=1)
    if this_item:
        # return the first (and only) record
        return this_item[0]
    else:
        # There's a problem :(
        return None
 
def readall():
    return web.select('items')


Aside from the exposure of POST and GET, this controller is pretty basic. It has the CRUD methods that we've all come to know and love. Notice that the templates can be nested. For example, within the index class, we're sending the item_list variable to the index.html template. Once that template is processed, the results are then sent to the layout.html template.

Here's the index.html template. You'll also notice the basic presentation logic that's possible within the templates.

$def with (items)
<h1>Todo List</h1>
$if items:
<ul id="todo_list"> $for item in items:
    <li> $item.body
<a href="/items/$item.id/">permalink</a>
<a href="/items/$item.id/edit/">edit</a>
<a onclick="return confirmDelete($item.id)" href="#">delete</a></li>
</ul>
$else:
<em>Go ahead. Add an item.</em>
 
<form action="/items/delete/" method="post"> <input name="id" type="hidden" /> </form>
<h2>Add Item</h2>
<form action="/items/add/" method="post"> <input id="item" name="body" type="text" /> <input type="submit" value="Add Item" /> </form>


And finally, our layout.html template.

$def with (page)
 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"&gt;
 
Todo List
 
<script src="/public/js/delete.js"></script>
<div id="content">$:page
 
</div>


Hopefully this little intro app has been helpful. If you have any questions, feel free to post them here.

Downloads: todo.tar.gz

June 10th, 2008 / Tags: webpy, web.py, tutorial, python, framework, todo / Trackback