Here were some DjangoCon 2010 highlights for me, along with talk titles so you can find the full presentation videos online.

Creating better apps

Both Eric Florenzano (“Why Django Sucks, and How We Can Fix It”) and Alex Gaynor (“Rethinking the Reusable Application Paradigm”) had many useful things to say about creating better and more reusable apps.  The videos are worth watching, here are some highlights:

• Narrower abstractions would make apps suck less — such as not directly exposing models (create an API) so you can swap out model or implementation.  This was also a big part of Alex’s talk.
• Class-based views are a nice way of creating an API and allow implementations to change over time.  There are various ways of doing class-based views and there isn’t an official blessed way, but class-based views would be an improvement (and offer more flexibility) than current function-based views in Django.

Databases other than SQL

With the growth in non-relational databases out there, I enjoyed starting to play with djangon-nonrel with MongoDB.  This project also supports Google App Engine.

After hearing experiences from the “NoSQL and Django Panel” it’s obvious that these are new technologies (relative to SQL) and that they are quite varied in what they support and the problems they solve.  There are also pros and cons with schemaless databases that you should be aware of when picking your solution.

Andrew Godwin (“Step Away From That Database”) also presented a nice range of databases — Document databases (MongoDB, CouchDB), Key-value stores (Cassandra, Redis), Message Queues (AMQP, Celery), Graph databases — and reminds us that filesystems are also key-value stores, and that version-control systems can provide a versioned storage.

Data Migration to Django

Brian Luft’s talk (“Data Herding: How to Shepherd Your Flock Through Valleys of Darkness”) will resonate with anyone looking for better methods of moving from a legacy system to Django. He also highlights the opportunity and flexibility that Django framework provides for clients in moving them off of their desktop and legacy systems.

Website Security

Adam Baldwin (“Pony Pwning”) gave a great talk on website security and things to be thinking about while building Django sites.

Cross-Site Scripting (“xss”) vulnerability are alive and well. Though Django offers protections, these can be turned off or it’s easy to make a mistake in the template layer and open yourself up. You can learn more and play with xss vulnerabilities: http://owasp-esapi-python-swingset.appspot.com/xss/django

Adam suggests considering OWASP ESAPI, auditing templates, auditing reusable snippets, and educating designers.

For REST services, good to consider django-piston rather than writing your own.

Other things include checking upload extensions to avoid running arbitrary code, e.g. Django ImageField doesn’t check extensions and could possibly run embedded code.  Apache “mod_security” as a nice monitoring tool that can let you know what’s happening, as well as prevent some issues.

An Exclusionary Establishment?

Eric Florenzano hit on some issues on the social engineering side of Django.  He provided examples of situations that can send the messages of “your contributions aren’t important” out to the larger Django community — enhancements that went through all the steps but didn’t make it into core (eg truncatechars), key contributors not given committer rights (eg Alex Gaynor), no non-core developer code accepted into django.contrib.

James Bennett’s (“Topics of Interest”) reflected some of this too in the fact that there are very few core committers (14) and even fewer than understand the ORM and that they’re having a hard time growing core-committers.  Eric’s mention of Guido van Rossum’s quote (creator of Python) “give out more commit privileges sooner”.

Participation could be easier and more inviting if more core devs are added and there is also less concern about breaking trunk.

Something I’ll add: My understanding is that too many talks were submitted for DjangoCon so of course some were turned away, yet some presenters had more than 1 slot.  Many of the talks were very good, but I think it would be healthy to open up more opportunities for community members to present.

Improving Django and Becoming Core Developer

Given the concern about lack of core committers, Russell Keith-Magee (“So you want to be a core developer?”) was timely. In addition to going through the process of contributing code, he also mentioned that contributing to the Django community in the form of help (django-user list, stack overflow, etc) and writing new docs (tutorials, howtos, elaborating on existing docs) is key.

Interesting Apps and Projects

The lightning talks and Eric Holscher’s talk (“Large Problems in Django, Mostly Solved”) were a good source of Django apps to try if you’re not already using them.

Eric also shares his measure of what makes a solid reusable app — needs to have an easy setup, a good upgrade path, good documentation and well-tested.

Apps he mentioned were Haystack (search), Sphinx (documentation), South (db migration), Celery (delayed execution), Fabric (deployment), gunicorn.org (async server), pip and virtualenv (packaging), TastyPie and Piston (RESTful), Taggit (tagging), Hudson (continuous integration tool), django-filter (django admin filtering), djangopackages.com (for finding new apps).

Some interesting things from the lightning talks:

• http://djangopackages.com (reusable apps, sites, tools — a Django Dash project. Nice work Daniel Greenfeld!)
• http://readthedocs.org  (hosted help documents — a Django Dash project)
• logbook – an easier way to add logging to Django

Performance

Frank Wiles’ talk (“Alice in Performanceland — Down the Rabbit Hole”) was full of a lot of tasty morsels to help you “do less” and “remove pressure on your server” that ultimately increases the performance and responsiveness of your site.

Improving your Code Quality

Human code review is best says Peter Baumgartner in his talk (“Monitoring Code Quality in Your Django Project”). There are also many great code quality tools out there that many of us use: test suites, code coverage, link/pep8, profiling, code complexity metrics, value — and pulling this together via Hudson.  Also, creating a build in one step is offers new developers a fast way to get started on a project.

Managed Django Hosting

I missed Nate Aune’s lightning talk on his DjangoZoom.com cloud deployment solution, but looks interesting. http://djangozoom.com/ponyexpress/

One of the lightning talks discussed the django-servee project: http://www.servee.com/

Thanks DjangoCon!

It was nice meeting new people using Django in Seattle and in other areas of Washington State, such as in Richland, Wenatchee and Spokane — and reconnecting with friends from the Plone and greater Python communities.

Thank you to the organizers, presenters and supporters for another great DjangoCon!

We’ll be discussing “What did we learn at DjangoCon?” at this Thursday’s Django Seattle. See http://www.djangoseattle.org for more details.

In the meantime, here are some high-level take-aways:

• Should JavaScript and RESTful services be part of the Django core?  JS is even more useful/powerful with the latest fast JS engines in Chrome, Firefox and Safari/Webkit. Competitor Rails builds in RESTful features – some promising ones for Django include django-piston and django-roa.  I liked how Ted Leung talked about “science experiments” and posed many ideas on what we may want to experiment with to get right before approaching Django core.
• Git – Though this doesn’t directly relate to Django, DVCS systems like Git and Mercurial are in wide use.  SVN is a given, but now feel I need to know Git and Mercurial well – since popular projects are using these.  I also wanted to pick a “pet” DVCS to use as my default too.  I’ve chosen Git (mainly because of git-svn and GitHub), but will be using Mercurial as well.
• Django Tips and Tricks – Many to pick from, but I liked Query.as_sql() method to show the SQL the Django ORM generates on your behalf, the flexibility of using “signals” to loosely couple functionality (see django-signals-ahoy on bithub), reusing other Python WSGI middleware (such as repoze.bitblt, repoze.squeeze, repoze.profile), pylint/djangolint, class-based views, db schema migrations with South, much faster test speeds in Django 1.1, various test utilities floating around, talks on performance, etc.

* Django jobs are growing, and Django also a popular platform for Start-ups.

* Check out the DjangoCon2009 Wiki for slides and presentations.

NOTE: This post is for installing Python 2.4 on a brand new Snow Leopard Instance. If upgrading on top of Leopard, you may have to update easy_install, macports, etc.  More Googling around may be required.

Though I had to create two buildouts to get this to work — is there a way to get this into one buildout?

I first tried to create one buildout by combining  Florian Schulze’s buildout recipe with a standard Zope recipe — but since initial bootstrap was run by Python 2.5, I couldn’t get the Zope instance to use the new Python 2.4. So I first ran a buildout to build Python 2.4 (using OSX-installed Python 2.5), then used that new Python 2.4 to run bootstrap.py on the Zope 2.8.x buildout.

Here’s the recipe I used to just build Python 2.4 (requires Florian’s buildout, see Alexander Limi’s comment below for where to find this):

[buildout]
#extends = src/snowleopard.cfg     # no longer required as Joe mentions below
python-buildout-root = ${buildout:directory}/src
parts -=
   ${buildout:python25-parts}
   ${buildout:python26-parts}

[install-links]
prefix = /opt/local

Then I ran a simple Zope 2.8 buildout to see if it would compile (using new Python 2.4 to bootstrap), and it did!

[buildout]
parts =
   zope2
   instance

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.8.9.1/Zope-2.8.9.1-final.tgz

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
verbose-security = on

Here is my initial post:

I have to say — most everything I’ve installed on a fresh Snow Leopard install has worked flawlessly and swiftly — except for (the minor inconvenience of) iStat not working.  UPDATE: iStat 2.0 is available for Snow Leopard now. There’s a new beta of MenuMeters too for Snow Leopard.

There are also nice subtle improvements, see Mac Life’s 100 Top Snow Leopard Tips, Trick and Features for improvements to Preview, Expose, Stacks, etc.  I’ve very happy with the upgrade.

Now for the bad news for those like myself who depend on Python 2.4 for Plone, since many versions of Zope require Python 2.4.  I also use Python for Django, though that should run fine on Python that shipped with Snow Leopard.

You can read many of the initial details around the web, but here’s what I’ve experienced and have been able to put together:

• Note that these details are for a fresh Snow Leopard install – there are a different set of issues if you’re upgrading over your existing Leopard.  NEW: “Clark’s Tech Blog” has a nice write-up about upgrading Python after upgrading Leopard to Snow Leopard.
• Snow Leopard ships with Python 2.5.4, and this runs as a 32-bit application.
• I also need 2.4 branches of Python too, so I tried rolling my own (as usual) and it didn’t compile.  I then followed that thread for awhile.
• I then thought I pulled a fast one when I compiled from MacPorts and everything ran great!
• … but then I compiled Zope, and attempted to run an instance.  I saw a mysterious “No such file or directory” error.  Hmmm, I can navigate to that file, but running the script with my new Python interpretor was causing this error.
• After digging around with Activity Monitor, I discovered that the Python I built from scratch was running as a 64-bit app — while the Python that comes with Snow Leopard was only running 32-bit — which is telling, since most everything else on Snow Leopard is running 64-bit.
• Guessing that the the mysterious “No such file or directory” (when the file and directory did indeed exist) was due to a weird cocktail of 32-bit pieces living with 64-bit pieces.
• My latest theory was that I needed to figure out how to build Python as 32-bit.  I played with Macports and various architecture settings to hardwire this, but long-story-short — the architecture override isn’t used everywhere — so parts still compile natively as 64-bit on Snow Leopard.
• The best thread on the topic (that’s steadily growing) is here: http://bugs.python.org/issue6802 with msg92153 left today, which basically offers some additional settings for compiling Python as a 32-bit app (for Python 2.6).  Also mentions that Snow Leopard did some magic to get Python 2.5 working as a 32-bit app.
  
• My hope is that once “32-bit” Python 2.4 happens, the rest of the Zope install, etc, will be back to the good ol’ days in Leopard.

Plan B’s:

Otherwise, to save some headache, I’m wondering about installing a small Linux distro on VMWare as a local mini web-server where I can easily install Python and Zope — though that’s a bit of a pain too.

Luckily I also have my old Leopard in a separate partition (see my Extra life for my MacBook Pro with Snow Leopard and inexpensive hardware blog entry) and can boot that if necessary to work on various Zope/Plone sites (that required Python 2.4) while this is all being sorted out.

Now time to see if I can get 32-bit Python 2.4.6 compiled and installed, while waiting for more patches and information to appear…

Thanks to 14 Sprinters who came together on July 25, we now have a Django Seattle Website at http://www.djangoseattle.org

Some of our Django Seattle Sprinters

There was a lot of infrastructure work done at the sprint which is still in development and didn’t make it to the live site yet — but the experience of getting to know each other, and learning/sharing Django knowledge was another fine Sprint accomplishment.

Here’s a brief summary of what people worked on:

• Integrated in Blogging, Profile and Calendar functionality from django-basic-apps. Initially Pinax was explored, but had a lot of dependencies and seemed better for creating specific sites genres, but was challenging to incorporate into our existing site.
• Created a Twitter portlet that shows live #djangoseattle Tweets.
• Setup Flatpages for core content, and creating a database-driven menu
• Created a logo and initial site design and templates
• Setup Django on live server
• Used the Django Debug Toolbar while developing the site
• Some were playing with Django for the first time
• Some floated around to help diagnose problems and help those new to Django

Functionality brainstorm

Thank you Sprinters

Our sprinters (in alphabetical order) were: Andrew Beyer, Jon Callahan, Jesse Franceschini, Doug, Brian Gershon, Johann Heller, Paul Pham, Micah Ransdell, Leo Shklovskii, Trevor Smith, Jesse Snyder, Alex Tokar, Ragan Webber, Ben Wilber

Thank you Sponsors

Also a Big Thank You to our sponsors, hosts and organizers.

• Jon Callahan at Mazama Science treated all 14 of us to a tasty PCC Natural Markets lunch, coffee, drinks and snacks.
• Michael Kim at Grapevyn brought in Top Pot Doughnuts.
• Paul Pham hosted us at his coworking space Office Nomads which was a great place to have a sprint.
• Leo Shklovskii at Evoworx and Brian Gershon at Web Collective had a great time organizing the sprint.

We look forward to our next sprint!

Optimization Tutorial

I then got on my geek at “Faster Python Programs through Optimization” (Mike Müller of Python Academy), where we dove deeper into profiling and tips on improving speed or saving memory.

Some paraphrased guidelines to consider before you start optimizing (which were also reinforced in the “Real World Django” tutorial which I’ll chat about next):

• Make sure your program is really too slow – could be other factors like network traffic, database, etc.
• Don’t optimize as you go – might ultimately not need to spend that time.  Also working code is always important first.
• Only consider realistic use cases and user experience

We played with the profiling tools (profile, cProfile, time, pystone, heapy) and used them to compare various techniques:

• xrange and also Generators shaved off time by not having to allocate memory for large data sets.
• use built-in types as much as possible (including some newer collection classes)
• iterating and appending strings by first appending to lists, then using a join statement to create large strings (versus building strings via += and loops)
• One new one for me was converting lists to Sets before testing for membership of an item in the list, which is fast due to Set optimizations.
• The tutorial also covered pysco, processing and numpy modules, as well as caching techniques.

Real-world Django Tutorial

This very aptly named presentation by Jacob Kaplan-Moss and James Bennett was excellent for those of us who develop and deploy Django websites.  The full skinny (with link to slides) is here: http://jacobian.org/speaking/2009/real-world-django/

Some highlights for me included:

• Focus on tight Django Applications that promote reuse while also breaking a website into components. Benefits of also leveraging packaging up your own components.
• Gain flexibility by leveraging Django Managers, and they help encapsulate behavior behind an API.
• Can extend models via new (in Django 1.1) Proxy subclasses.
• Lots of discussion and recommendations for testing — from unit testing, through functional testing, and then browser-based functional testing. Yep, you need them all. I’d like to play more with Twill and Windmill.
• Automating deployment – including options like virtualenv (and virtualenvwrapper), Ian Bicking’s pip (“pip installs packages”), zc.buildout, and Fabric.  zc.buildout’s power was emphasized (with its recipes, etc) was a bit overshadowed by comments on lack of documentation. I’d like to give pip and Fabric a try.
• Apache + mod_wsgi is now a preferred platform for server Django sites (or at least much more consistent performance and memory-usage wise than Apache + mod_python).
• Definitely flip through the session slides!  These were just some highlights for me out of 189 slides of useful information.

Various tidbits for the next few days here at PyCon:

• Open Space sessions come highly recommended
• There’s a heavy testing thread throughout conference (10 sessions worth!)
• Friday 11am: Using Windmill
• Saturday 4:15p: Ian Bicking’s session (creator of PIP and virtualenv, among many other topics)
• Sunday 10:35a: Panel: Functional Testing Tools in Python
• … though it will ultimately be tough to pick and choose from all the great topics!

Time for some sleep… more tutorials tomorrow, then 3 days of conference, then 4 days of sprints!

ps: It’s been great to see familiar faces from the Zope and Plone communities, which is often where I “get my Python on”.  Lately I’m also doing a lot of Django, so enjoying all the synergy around Python here at PyCon2009!

After Web Collective just finished a Django website that needed to scale to 12,000 simultaneous peak users, Flickr architect Cal Henderson’s keynote was timely and very entertaining – particularly his commentary on “serious” frameworks and essentially a comic roasting of all of the major ones.

Since most of us are building “houses and not skyscrapers” there is some question as to whether Django should introduce all of Cal’s suggestions (who is a “scale” geek working with on a very large website), but how can you argue with his three major focus areas of scale, performance and “being awesome“.

Discussions of “scale” and “Django” remind me to play with Google App Engine soon. I also just discovered Django Debug Toolbar and other interesting things reading through Cal’s blog.

Learning from other Python Frameworks (Django and Zope)

With Zope (and Plone) being the platform I’ve used for some time now, I found Mark Ramm’s perspective on how to learn from all the smart folks in various Python (and other) frameworks a good 10,000 ft reminder.

He highlights that Zope3, Turbogears, and Pylons have learned some of the lessons of old Zope2 in that they are frameworks that strive to use community components that work with each other in multiple contexts.  They also are trying not to isolate themselves from the Python web community by building functionality specific to just one platform.

He wasn’t saying Django was far afloat in these areas, but rather that Django could make some of its innovations available more easily to other frameworks (e.g. Django’s ORM, and Django’s template engine), and that it should maybe use some of the Python components from other frameworks (e.g. Django middleware and WSGI middleware compatibility).

I look forward to poking around the rest of the DjangoCon 2008 videos, such as Guido van Rossum’s keynote and others on YouTube’s DjangoCon 2008 Session’s playlist.

Most were familiar with just one platform or the other, so it was a great opportunity to ask questions and learn about the strengths of each platform.

http://presentations.webcollective.coop/django-plone