text:
Gordon Tillman has a wiki page up on how to mix Django content into a CherryPy site. It's easy and probably works, but please don't do it anymore.
We're officially going to deprecate the wsgiapp Tool because 1) it doesn't conform to the WSGI spec (and cannot be fixed to do so), and 2) there's a better way to mix content in a CherryPy site:
tree.graft(app, script_name) method is the proper way to add Django or other WSGI content to an existing CherryPy site. Instead of nesting the two frameworks, we branch instead. To take Gordon's example, instead of:
class DjangoApp(object):
_cp_config = {
'tools.wsgiapp.on': True,
'tools.wsgiapp.app': AdminMediaHandler(WSGIHandler()),
cherrypy.tree.mount(DjangoApp(), '/')
You should always write this instead:
cherrypy.tree.graft(AdminMediaHandler(WSGIHandler()), '/')
Look, if you nest the one inside the other, CherryPy's going to do an awful lot of HTTP request parsing that is going to be completely redundant, since Django's going to do it again anyway. And this code is not very fast. Your site is going to crawl. That's strike one for nesting.
Strike two is the "always on" nature of nesting as opposed to branching. When you write your request/response cycle like an onion, every component which could possibly play a part in the request has to be called, even if just to reply "I'm not involved in this one". Given the slowness of Python function calls, this is rarely a good thing. If you thought your site was crawling before... This was a major design flaw of CherryPy 2, and is a major reason CherryPy 3 is 3x faster: the old Filters were called all the time, even if you didn't need them; the new Tools are only called when they're applicable.
Strike three against the nested approach is that it's always easier to traverse a tree of siblings than it is to traverse a nested set; programmers, for some reason, like to hide information from you, including how their site components go together. The branched version will be much easier to reason about, statically analyze, and write inspection tools for.
So please, use
tree.graft, and stop using the wsgiapp Tool in CherryPy 3. We're going to formally deprecate it soon.
I was asked last week how many lines of code some of my projects are, and didn't have an answer handy. Fortunately, it's easy to write a LOC counter in Python:
"""Calculate LOC (lines of code) for a given package directory."""
import os
import re
def loc(path, pattern="^.*\.py$"):
"""Return the number of lines of code for all files in the given path.
If the 'pattern' argument is provided, it must be a regular expression
against which each filename will be matched. By default, all filenames
ending in ".py" are analyzed.
lines = 0
for root, dirs, files in os.walk(path):
for name in files:
if re.match(pattern, name):
f = open(os.path.join(root, name), 'rb')
for line in f:
line = line.strip()
if line and not line.startswith("#"):
lines += 1
return lines
I've added the above to my company's public-domain
misc package at http://projects.amor.org/misc/. Here are the results for my high-priority projects (some are proprietary):
>>> from misc import loc
>>> loc.loc(r"C:\Python24\Lib\site-packages\raisersedge")
>>> loc.loc(r"C:\Python24\Lib\site-packages\dejavu")
>>> loc.loc(r"C:\Python24\Lib\site-packages\geniusql")
>>> loc.loc(r"C:\Python24\Lib\site-packages\cherrypy")
>>> loc.loc(r"C:\Python24\Lib\site-packages\endue")
>>> loc.loc(r"C:\Python24\Lib\site-packages\mcontrol")
>>> loc.loc(r"C:\Python24\Lib\site-packages\misc")
~= 61 kloc. Pretty hefty for a single in-house web app stack. :/ But, hey, nobody said integration projects were easy.
WSGI has enabled an ecosystem where site deployers can, in theory, mix multiple applications from various frameworks into a single web site, served by a single HTTP server. And that's great. But there are several areas where WSGI is purposefully silent, where there is still room for standards-based collaboration:
Most frameworks address all of the above already, to varying degrees; however, they still tend to do so in a very monolithic manner. Paste is notable for attempting to provide some of them in discrete pieces (especially WSGI graph construction and a config format tailor-made for it).
But I'm going to focus here on just two of these issues: process state and site-wide services. I believe we can separate these two from the rest of the pack and provide a simple, common specification for both, one that's completely implementable in 100 lines of code by any framework.
One of the largest issues when combining multiple frameworks in a single process is answering the question, "who's in control of the site as a whole?" Multiple frameworks means multiple code bases who all think they should provide:
...and they often disagree about those behaviors. Throw Apache or lighttpd into the mix and you've got some serious deployment issues.
The typical solution to this is to have each component provide a means of shutting off each process-controlling feature. For example, CherryPy 3 obeys the config entry
engine.autoreload_on = False, while
django-admin.py takes a
--noreload command-line arg. But these are different for each framework, and difficult to coordinate as the number of components grows. Since, for example, only one autoreloader is needed per site, a more usable solution would be to selectively turn on just one instead of turning off all but one.
For a worse example, let's look at handling SIGTERM. Currently, we have the following:
OK, Django doesn't actually provide a SIGTERM handler, but you get the idea. If several components register a SIGTERM handler, only one of them will "win" by virtue of being the last one to register. And chances are, the winning handler will shut down its component cleanly and then exit the process, leaving other components to fend for themselves.
In fact, there's a whole list of negatives for the monolithic approach to process control and site services:
We could attempt to solve this with a Grand Unified Site Container, but that would most likely:
The Web Site Process Bus uses a simple publish/subscribe architecture to loosely connect WSGI components with site services. Here's our SIGTERM example, implemented with a WSPBus:
The singleton Bus object does three things:
Each method which changes the state also has an equivalent named channel. Any framework, server, or other component may register code as a listener on any channel. For example, a web framework can register database-connection code to be run when the "start" method is called, and disconnection code for the "stop" method:
bus.subscribe("start", orm.connpool.start)
bus.subscribe("stop", orm.connpool.stop)
Any channel which has no listeners will simply ignore all published messages. This allows component code to be much simpler; callers do not need to know whether their actions are appropriate--they are appropriate if a listener is subscribed to that channel.
In addition to the builtin state-transition channels, components are free to define their own pub/sub channels. CherryPy's current implementation, for example, defines the additional channels
start_thread and
stop_thread, and registers channels for signals, such as "SIGTERM", "SIGHUP", and "SIGUSR1" (which then typically call bus methods like "restart" and "exit"). Some of these could be standardized. Other custom channels would be more naturally tightly-coupled, requiring awareness on the part of callers and callees.
Since WSPB state-changing method calls are expected to be sporadic, and often fundamentally serial (e.g., "autoreload"), their execution is synchronous. Subscribers (mostly of custom channels), however, are free to return immediately, and continue their operation asynchronously.
The WSPB cleanly solves all of the problems outlined above. The various components are no longer in competition over process state; instead, there is a single race-free state machine. However, no single component has to know whether or how many other components are deployed in the same site.
Frameworks and servers can provide a subset of all site services, with a common, imperative-Python API for deployers to add or substitute their own. However, the WSPB doesn't define a config syntax, so each framework can continue to provide its own unique layer to translate config into that API. A deployer of a combined Pylons/Zope website could choose a Pylons startup script and config syntax to manage the lifecycle of the Zope components.
The WSPB doesn't try to instantiate or compose WSGI components (server -> middleware -> app) either. So there's even room for site daemons which provide no traditional web app functionality; instead, they specialize in providing tools to compose WSGI component graphs via a config file or even a GUI.
It also "plays nice" with
mod_wsgi, FastCGI, and SCGI. Those who develop WSGI gateways for these will have a clear incentive to consolidate their ad-hoc startup and shutdown models into the WSPB. For example, a modpython gateway can use apache.register_cleanup to just call bus.stop() instead of providing custom cleanup-declaration code.
Best of all, the WSPB can be defined as a specification which any framework can provide in a small amount of code. Rather than attempt to draft the specification here (that can be hashed out on Web-SIG, since this is by no means complete), I'm just going to provide an example:
except NameError:
from sets import Set as set
import sys
import threading
import time
import traceback as _traceback
# Use a flag to indicate the state of the bus.
class _StateEnum(object):
class State(object):
states = _StateEnum()
states.STOPPED = states.State()
states.STARTING = states.State()
states.STARTED = states.State()
states.STOPPING = states.State()
class Bus(object):
"""Process state-machine and messenger for HTTP site deployment."""
states = states
state = states.STOPPED
def __init__(self):
self.state = states.STOPPED
self.listeners = dict([(channel, set()) for channel
in ('start', 'stop', 'exit',
'restart', 'graceful', 'log')])
self._priorities = {}
def subscribe(self, channel, callback, priority=None):
"""Add the given callback at the given channel (if not present)."""
if channel not in self.listeners:
self.listeners[channel] = set()
if priority is None:
priority = getattr(callback, 'priority', 50)
self._priorities[(channel, callback)] = priority
def unsubscribe(self, channel, callback):
"""Discard the given callback (if present)."""
listeners = self.listeners.get(channel)
if listeners and callback in listeners:
del self._priorities[(channel, callback)]
def publish(self, channel, *args, **kwargs):
"""Return output of all subscribers for the given channel."""
if channel not in self.listeners:
return []
exc = None
output = []
items = [(self._priorities[(channel, listener)], listener)
for listener in self.listeners[channel]]
for priority, listener in items:
# All listeners for a given channel are guaranteed to run even
# if others at the same channel fail. We will still log the
# failure, but proceed on to the next listener. The only way
# to stop all processing from one of these listeners is to
# raise SystemExit and stop the whole server.
output.append(listener(*args, **kwargs))
except (KeyboardInterrupt, SystemExit):
self.log("Error in %r listener %r" % (channel, listener),
exc = sys.exc_info()[1]
if exc:
return output
def start(self):
"""Start all services."""
self.state = states.STARTING
self.log('Bus starting')
self.state = states.STARTED
def restart(self):
"""Restart the process (may close connections)."""
self.log('Bus restart')
def graceful(self):
"""Advise all services to reload."""
self.log('Bus graceful')
def block(self, state=states.STOPPED, interval=0.1):
"""Wait for the given state, KeyboardInterrupt or SystemExit."""
while self.state != state:
except (KeyboardInterrupt, IOError):
# The time.sleep call might raise
# "IOError: [Errno 4] Interrupted function call" on KBInt.
self.log('Keyboard Interrupt: shutting down bus')
except SystemExit:
self.log('SystemExit raised: shutting down bus')
def stop(self):
"""Stop all services."""
self.state = states.STOPPING
self.log('Bus stopping')
self.state = states.STOPPED
def exit(self, status=0):
"""Stop all services and exit the process."""
self.log('Bus exit')
def log(self, msg="", traceback=False):
if traceback:
exc = sys.exc_info()
msg += "\n" + "".join(_traceback.format_exception(*exc))
self.publish('log', msg)
From sucks-rocks.com:
No, really. It rocks. Rocks, rocks, rocks.
(Thanks, jamwt!)
PyCon 2007 is nearing a close; here are some notes on how it affected CherryPy:
Chad Whitacre (author of Aspen) herded several cats into a room on Sunday and forced us to discuss the various issues surrounding Python web application deployment. This is hinted at in the WSGI spec:
Finally, it should be mentioned that the current version of WSGI does not prescribe any particular mechanism for "deploying" an application for use with a web server or server gateway. At the present time, this is necessarily implementation-defined by the server or gateway. After a sufficient number of servers and frameworks have implemented WSGI to provide field experience with varying deployment requirements, it may make sense to create another PEP, describing a deployment standard for WSGI servers and application frameworks.
There were three basic realms where the participants agreed we could try to collaborate/standardize:
Process control: stop, start, restart, daemonization, signal handling, socket re-use, drop privileges, etc. If you're familiar with CherryPy 3, you'll recognize this list as 95% of the current cherrypy.engine object. The CherryPy team has already been discussing ways of breaking up the Engine object; this may facilitate that (and vice-versa). Joseph Tate volunteered to look at socket re-use issues specifically, but the general consensus seemed to be that much of this would be hashed out on Web-SIG.
WSGI stack composition: Jim Fulton proposed that we could all agree on Paste Deploy (at least a good portion of the API) to manage this in a cross-framework manner. Most heads nodded, "yes". Jim also proposed that each of the framework authors take the next week to refamiliarize themselves with Deploy, and then start pestering Ian Bicking with specific API issues. Ian suggested that he should fork Paste Deploy into another project specifically for this. For CherryPy, this would first mean offering standard egg entry points. [Personally, I'd like to standardize on a pure-Python API for deploy, not a config file format API. In other words, make the config file format optional, so that users of CP-only apps could avoid having to learn a distinct config file format for deployment. It should be possible to transform various config file formats into the same Python object(s).]
Benchmarks: Jim also suggested we create a standard WSGI HTTP server benchmark suite, with various test applications and concurrency scenarios. This would compare various WSGI HTTP servers, as opposed to CherryPy's existing benchmark suite which compares successive versions of the full CP stack. Ian volunteered to begin work on that project (with the expectation that others would contribute substantial use cases, etc).
Others who were present for at least a portion of the long discussion: me, Mark Ramm, Kevin Dangoor, Ben Bangert, Jonathan Ellis, Matt Good, Brian Beck, and Calvin Hendryx-Parker.
After some discussion with Mark (and he with Ian and Ben), we agreed that CherryPy could do more in the WSGI-middleware-authoring department. There is a continuous pressure to simply re-use or fix up the existing CherryPy request object to fill this need; however, there are some fundamental problems with that approach (such as the use of threadlocals to manage context, and the difficulty of streaming WSGI output through a CherryPy app). At the moment, I'm leaning toward adding a new API to CherryPy which would be similar to the application API, but specifically targeted at middleware authoring.
I've been a proper fool (and might still be). I've been telling everyone that CherryPy 3 is much easier to learn and use because it's been tailored to be help()-ful. What I meant was that you could open an interactive interpreter, type
help(cherrypy.<thing>) and have at least some idea of what it does. I spent quite a bit of time honing the top-level namespace down to as few components as possible (and some of the component namespaces, too) in order to make
help() easier to read.
This is harder to do than you might think. Unlike simple linear scripts or libraries, the most important objects when CherryPy is "live" don't exist at an interactive prompt. The Request, Response, and Session objects are all heavily dependent on the context of a real HTTP conversation. They're hard to create in a vacuum. And although there's one of each per thread while the system is running, they are implemented as thread local objects so that the CherryPy programmer can treat each of them as if there were only one: a global.
Thread locals are a great invention, but they suffer from one serious drawback when used in a threaded framework: they allow anyone to add attributes to them. If the framework re-uses the same thread for multiple requests, it becomes difficult to reliably clean out all of those attributes between requests.
CherryPy's solution to that was to add a container in 2.1; instead of a separate thread local for the Request, Response, and Session objects, there is a single, hidden thread local called
cherrypy._serving, and the Request, Response, and Session objects for each thread are attributes of the "serving" object. This makes it easy for cleanup code: it just calls
cherrypy._serving.__dict__.clear() when the request ends. (Aside: this technique also allows the Request, Response and Session types to be overridden).
However, pushing those objects into a container means they're no longer so easy to reference. CherryPy code would become uglier and more difficult if, instead of:
...you had to write:
So a
_ThreadLocalProxy class was introduced to allow CherryPy code to keep writing the nicer, shorter syntax. In short, it passes
__getattr__ (and other double-underscore methods) through to a wrapped object. So
cherrypy.request became a proxy object to a wrapped Request object. Ditto for response and session.
That was fine for CherryPy 2, but one of the goals for version 3.0 is better IDE support. Most IDE's at least provide calltips for code completion, but there aren't usually any HTTP requests coming in as you're writing code! CP 2's thread local proxies didn't have a request object in the main thread (or any thread that wasn't started by the HTTP server), so typing
cherrypy.request. couldn't result in a calltip as you coded. The solution for CherryPy 3 was to have the proxy's
__getattr__ and friends wrap a default object if a live object could not be found. And the default objects' attributes are true defaults; if they're not overridden (in config or code), they won't change when the system goes live. This makes interactive exploration even easier; you can forget all about the threading and pretend you're looking at live, global objects.
But there's another catch: one of the few problems with using a proxy object in pure Python is that it's no longer of the same type as the wrapped object. Unfortunately for us, Python's builtin
help function uses pydoc, and pydoc calls
type(obj) quite a bit.
You can certainly call
help(cherrypy.request.run) and get the correct docstring, because "run" is an attribute of
cherrypy.request, the proxy calls
__getattr__ first, and then
type() is called on the attribute, not the request object/proxy. But if you attempt
help(cherrypy.request), you're in for some confusion, because the proxy implementation leaks out.
Or rather, it did leak out until just now. I took the plunge and CherryPy now monkeypatches pydoc, so that it "passes the
help() call through the proxy". Monkeypatching the standard library is of course a huge no-no, but the alternative was to essentially copy and paste most of pydoc and distribute the result with CherryPy. Now,
help(cherrypy.response) at least prints:
>>> help(cherrypy.response)
Help on Response in module cherrypy._cprequest object:
class Response(__builtin__.object)
| An HTTP Response, including status, headers, and body.
| Application developers should use Response.headers (a dict) to
| set or modify HTTP response headers. When the response is finalized,
| Response.headers is transformed into Response.header_list as
| (key, value) tuples.
| Methods defined here:
| __init__(self)
| check_timeout(self)
| If now > self.time + self.timeout, set self.timed_out.
| This purposefully sets a flag, rather than raising an error,
| so that a monitor thread can interrupt the Response thread.
| collapse_body(self)
| finalize(self)
| Transform headers (and cookies) into self.header_list.
| ----------------------------------------------------------------------
| Data and other attributes defined here:
| __dict__ = <dictproxy object>
| dictionary for instance variables (if defined)
| __weakref__ = <attribute '__weakref__' of 'Response' objects>
| list of weak references to the object (if defined)
| body = <cherrypy._cprequest.Body object>
| The body of the HTTP response (the response entity).
| cookie = <SimpleCookie: >
| header_list = []
| headers = {}
| status = ''
| stream = False
| time = None
| timed_out = False
| timeout = 300
But there's a further flaw with the above output of
help(); none of the data members of the Response class are documented! A few of them are mentioned in the class docstring, to be sure, but hardly to a truly useful extent. The Request object is an even poorer state, since it has so many more data members.
The solution for that issue is somewhat complicated, as well. It turns out that there are plenty of good documentation generators for Python code (that emit HTML or text; epydoc and pudge spring to mind), but no serious helpers for making
help() more informative. This is a real shame; I would almost always rather have
help() be truly helpful than go read a book or search online docs.
So I proposed a (small!) metaclass to help alleviate the problem for CherryPy. When you look at CherryPy source code, now, you might see something like this:
class Request(object):
"""An HTTP request."""
__metaclass__ = cherrypy._AttributeDocstrings
prev = None
prev__doc = """
The previous Request object (if any). This should be None
unless we are processing an InternalRedirect."""
# Conversation/connection attributes
local = http.Host("localhost", 80)
local__doc = \
"An http.Host(ip, port, hostname) object for the server socket."
remote = http.Host("localhost", 1111)
remote__doc = \
"An http.Host(ip, port, hostname) object for the client socket."
_AttributeDocstrings metaclass does one thing: finds class members whose names look like
<attrname>__doc, takes their
str value, formats it, and folds it into the class docstring. Here's a snippet of the resulting
help() output:
Help on Request in module cherrypy._cprequest object:
class Request(__builtin__.object)
| An HTTP request.
| local [= http.Host('localhost', 80, 'localhost')]:
| An http.Host(ip, port, hostname) object for the server socket.
| prev [= None]:
| The previous Request object (if any). This should be None
| unless we are processing an InternalRedirect.
| remote [= http.Host('localhost', 1111, 'localhost')]:
| An http.Host(ip, port, hostname) object for the client socket.
Christian's first question was, "why not just write it yourself by hand in the docstring?" Here's the long answer. The metaclass:
I chose the naming convention because it allows the attribute name and the
attribute__doc name to line up horizontally (it doesn't matter which comes first; I prefer to put the doc after the attribute). It also looks similar to the conventions in Python's C code, where doc variable names look like
module_attribute__doc__ or sometimes just
Hopefully these two improvements, although more awkward than I like implementation-wise, will make using CherryPy much easier and faster. Feel free to
help() us out by writing a few data member docstrings!
A couple of months ago, in response to someone else's speed claims, I posted a comment that CherryPy's built in WSGI server could serve 1200 simple requests per second. The demo used Apache's "ab" tool to test ("-k -n 3000 -c %s"). In the last few days before the release of CherryPy 3.0 final, I've done some further optimization of cherrypy.wsgiserver, and now get 2000+ req/sec on my modest laptop.
threads | Completed | Failed | req/sec | msec/req | KB/sec |
10 | 3000 | 0 | 2170.79 | 0.461 | 358.18 |
20 | 3000 | 0 | 2080.34 | 0.481 | 343.26 |
30 | 3000 | 0 | 1920.31 | 0.521 | 316.85 |
40 | 3000 | 0 | 2051.84 | 0.487 | 338.55 |
50 | 3000 | 0 | 2051.84 | 0.487 | 338.55 |
The improvements are due to a variety of optimizations, including:
I want to make it clear that the benchmark does not exercise any part of CherryPy other than the WSGI server. I used a very simple WSGI application (not the full CherryPy stack):
def simple_app(environ, start_response):
"""Simplest possible application object"""
status = '200 OK'
response_headers = [('Content-type','text/plain'),
start_response(status, response_headers)
return ['My Own Hello World!']
The full stack of CherryPy includes the WSGI application side as well, and consequently takes more time. But that has risen from about 380 requests per second in October to:
Client Thread Report (1000 requests, 14 byte response body, 10 server threads):
threads | Completed | Failed | req/sec | msec/req | KB/sec |
10 | 1000 | 0 | 536.86 | 1.863 | 85.36 |
20 | 1000 | 0 | 509.47 | 1.963 | 81.01 |
30 | 1000 | 0 | 499.28 | 2.003 | 79.39 |
40 | 1000 | 0 | 491.90 | 2.033 | 78.21 |
50 | 1000 | 0 | 504.32 | 1.983 | 80.19 |
Average | 1000.0 | 0.0 | 508.366 | 1.969 | 80.832 |
If you want to benchmark the full CherryPy stack on your own, just install CherryPy and run the script at
Here's the other script for the "bare server" benchmarks:
import re
import sys
import threading
import time
from cherrypy import _cpmodpy
APACHE_PATH = "apache"
class ABSession:
"""A session of 'ab', the Apache HTTP server benchmarking tool."""
parse_patterns = [('complete_requests', 'Completed',
r'^Complete requests:\s*(\d+)'),
('failed_requests', 'Failed',
r'^Failed requests:\s*(\d+)'),
('requests_per_second', 'req/sec',
r'^Requests per second:\s*([0-9.]+)'),
('time_per_request_concurrent', 'msec/req',
r'^Time per request:\s*([0-9.]+).*concurrent requests\)$'),
('transfer_rate', 'KB/sec',
r'^Transfer rate:\s*([0-9.]+)'),
def __init__(self, path=SCRIPT_NAME + "/", requests=3000, concurrency=10):
self.path = path
self.requests = requests
self.concurrency = concurrency
def args(self):
assert self.concurrency > 0
assert self.requests > 0
return ("-k -n %s -c %s <a href="http://localhost:%s%s"">http://localhost:%s%s"</a> %
(self.requests, self.concurrency, PORT, self.path))
def run(self):
# Parse output of ab, setting attributes on self
args = self.args()
self.output = _cpmodpy.read_process(AB_PATH or "ab", args)
for attr, name, pattern in self.parse_patterns:
val = re.search(pattern, self.output, re.MULTILINE)
if val:
val = val.group(1)
setattr(self, attr, val)
setattr(self, attr, None)
safe_threads = (25, 50, 100, 200, 400)
if sys.platform in ("win32",):
# For some reason, ab crashes with > 50 threads on my Win2k laptop.
safe_threads = (10, 20, 30, 40, 50)
def thread_report(path=SCRIPT_NAME + "/", concurrency=safe_threads):
sess = ABSession(path)
attrs, names, patterns = zip(*sess.parse_patterns)
rows = [('threads',) + names]
for c in concurrency:
sess.concurrency = c
rows.append([c] + [getattr(sess, attr) for attr in attrs])
return rows
def print_report(rows):
widths = []
for i in range(len(rows[0])):
lengths = [len(str(row[i])) for row in rows]
for row in rows:
for i, val in enumerate(row):
print str(val).rjust(widths[i]), "|",
if __name__ == '__main__':
def simple_app(environ, start_response):
"""Simplest possible application object"""
status = '200 OK'
response_headers = [('Content-type','text/plain'),
start_response(status, response_headers)
return ['My Own Hello World!']
from cherrypy import wsgiserver as w
s = w.CherryPyWSGIServer(("localhost", PORT), simple_app)
I played around with this as a potential hack for CherryPy 3. It's WSGI middleware for adding almost-transparent "internal redirect" capabilities to any WSGI application.
My operating theory was that anyone writing a WSGI app that does not already have an internal-redirect feature was probably using HTTP redirects (302, 303, or 307) to do nearly the same thing. This middleware simply waits for a 307 response status and performs the redirection itself within the same request, without informing the user-agent.
This should be OK because 307 isn't normally cacheable anyway, and some versions of IE don't bother to ask the user as the spec requires already, so it just duplicates an existing browser bug. I could have used a custom HTTP code like 399, but if that ever leaked out to the UA (because someone forgot to enable the middleware) then the UA should fall back to "300 Multiple Choices", which didn't seem like a good fit. At least by using 307, the fallback should be appropriate, if not graceful.
Here's the code, which could probably use some improvements:
"""WSGI middleware which performs "internal" redirection."""
import StringIO
class _Redirector(object):
def __init__(self, nextapp, recursive=False):
self.nextapp = nextapp
self.recursive = recursive
self.location = None
self.write_proxy = None
self.status = None
self.headers = None
self.exc_info = None
self.seen_paths = []
def start_response(self, status, headers, exc_info):
if status[:3] == "307":
for name, value in headers:
if name.lower() == "location":
self.location = value
self.status = status
self.headers = headers
self.exc_info = exc_info
return self.write
def write(self, data):
# This is only here for silly apps which call write.
if self.write_proxy is None:
self.write_proxy = self.sr(self.status, self.headers, self.exc_info)
def __call__(self, environ, start_response):
self.sr = start_response
nextenv = environ.copy()
curpath = nextenv['PATH_INFO']
if nextenv.get('QUERY_STRING'):
curpath = curpath + "?" + nextenv['QUERY_STRING']
while True:
# Consume the response (in case it's a generator).
response = [x for x in self.nextapp(nextenv, self.start_response)]
if self.location is None:
# No redirection required; complete the response normally.
self.sr(self.status, self.headers, self.exc_info)
return response
# Start with a fresh copy of the environ and start altering it.
nextenv = environ.copy()
nextenv['REQUEST_METHOD'] = 'GET'
nextenv['CONTENT_LENGTH'] = '0'
nextenv['wsgi.input'] = StringIO.StringIO()
nextenv['redirector.history'] = self.seen_paths[:]
# "The [Location response-header] field value
# consists of a single absolute URI."
path, params,
nextenv["QUERY_STRING"], frag) = urlparse(self.location)
if frag:
raise ValueError("Illegal #fragment in Location response "
"header %r" % self.location)
if params:
path = path + ";" + params
# Assume 'path' is already unquoted according to
# <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2">http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2</a>
if path.lower().startswith(environ['SCRIPT_NAME'].lower()):
nextenv["PATH_INFO"] = path[len(environ['SCRIPT_NAME']):]
raise ValueError("Location response header %r does not "
"match current SCRIPT_NAME %r"
% (self.location, environ['SCRIPT_NAME']))
# Update self.seen_paths and check for recursive calls.
curpath = nextenv['PATH_INFO']
if nextenv.get('QUERY_STRING'):
curpath = curpath + "?" + nextenv['QUERY_STRING']
if curpath in self.seen_paths:
raise RuntimeError("redirector visited the same URL twice: %r"
% curpath)
# Reset self for the next iteration
self.location = None
self.write_proxy = None
self.status = None
self.headers = None
self.exc_info = None
def redirector(nextapp, recursive=False):
"""WSGI middleware which performs "internal" redirection.
Whenever the next application sets a response status of 307 and
provides a Location response header, this component will not pass
that response on to the user-agent; instead, it parses the URI
provided in the Location response header and calls the same
application again using that URI. The following entries in the
WSGI environ dict may be modified when redirecting: wsgi.url_scheme,
SERVER_NAME, PATH_INFO, QUERY_STRING. REQUEST_METHOD is always
set to 'GET', so any desired parameters must be supplied as
query string arguments in the Location response header.
The wsgi.input entry will always be reset to an empty StringIO,
and CONTENT_LENGTH will be set to 0.
If 'recursive' is False (the default), each new target URI will be
checked to see if it has already been visited in the same request;
if so, a RuntimeError is raised. If 'recursive' is True, no check
is made and therefore no such errors are raised.
def redirect_wrapper(environ, start_response):
ir = _Redirector(nextapp, recursive)
return ir(environ, start_response)
return redirect_wrapper
...you should know that CherryPy 3 (soon to be released) includes a Routes dispatcher:
class City:
def __init__(self, name):
self.name = name
self.population = 10000
def index(self, **kwargs):
return "Welcome to %s, pop. %s" % (self.name, self.population)
def update(self, **kwargs):
self.population = kwargs['pop']
return "OK"
d = cherrypy._cprequest.RoutesDispatcher()
d.connect(name='hounslow', route='hounslow', controller=City('Hounslow'))
d.connect(name='surbiton', route='surbiton', controller=City('Surbiton'),
action='index', conditions=dict(method=['GET']))
d.mapper.connect('surbiton', controller='surbiton',
action='update', conditions=dict(method=['POST']))
conf = {'/': {'request.dispatch': d}}
cherrypy.tree.mount(root=None, config=conf)
cherrypy.config.update({'environment': 'test_suite'})
You tell CherryPy you want to use Routes dispatching in your app config with "
request.dispatch = <obj>". The astute reader will note this means:
To make your own dispatcher:
__call__ method that takes a
path_info argument.
*args and GET/POST parameters as
**kwargs, but if that's not what your dispatch style requires, do something else. It's completely customizable. You can even set request.handler to None if you don't want anything called at that point. Note also that HTTPRedirect and HTTPError (including NotFound) can be used as handlers; when called, they
raise self.
path_info argument above. The default CherryPy dispatcher does a lot of work to correctly allow config file entries to override
_cp_config entries on the CherryPy object tree. But if your dispatch style doesn't use a tree, you don't need to do all that.
Grab the latest trunk and start playing!
I probably waited too long, but today I upgraded both CherryPy (3.0alpha/trunk) and Dejavu (1.5alpha/trunk) to Python 2.5. The moves were surprisingly easy:
There were three changes in all:
[P.S. I've noticed CP 3 is about 3% slower in 2.5 than 2.4, even with the zombie frames and other optimizations. Hmmm.]
Amazingly, even though Dejavu makes extensive use of bytecode hacks, there was only one real change! The "logic" module needed an upgrade to the "comparison" function, which produces an Expression via the
types.CodeType() constructor. Apparently, function args are no longer included in
co_names, and
co_consts no longer includes a leading 'None' value (except when there are cell references?). Finally,
co_flags for "normal" functions now includes
CO_NESTED by default. These changes also forced some parallel upgrades to the test suite.
While fixing the above, however, I noticed a long-standing bug in Dejavu's LambdaDecompiler. Python 2.4 used
ROT_THREE before a
STORE_SUBSCR, and this worked in Dejavu; but Python 2.5 uses
ROT_TWO before
STORE_SUBSCR, which showed me I had the stack-popping backwards in both functions. Bah. Fixed now.
Both packages needed a good bit of work changing some relative import statements into absolute ones. Not really hard, just boring.
Thanks to the Python core devs for a very smooth transition!
|<< <
|> >>


text:
Sweet Potato & Avocado Salsa
1. Preheat oven to 450 degrees. On a rimmed baking sheet, toss together 1 large sweet potato (1 pound), peeled and diced small, 1 red onion, diced small, and 1 tablespoon extra-virgin olive oil. Roast until sweet potato is tender and browned in spots, about 20 minutes. Transfer to a large bowl and let cool completely.
2. Add 1 medium tomato, diced small, 1 avocado, diced small, 2 tablespoons chopped fresh cilantro, and 1/4 cup fresh lime juice (from 2 limes). Season with coarse salt and toss to combine.
Quick ways to enjoy:
- service with crackers or a toasted baguette for a snack
- scatter over chopped lettuce for a simple salad
- stir into couscous to create an easy side dish
- use as a topping for chicken tacos or nachos
- fold into scrambled eggs at breakfast
(original recipe - multiple delicious)
(original recipe - multiple delicious)

subset: commoncrawl


text:
A couple weeks ago Kaleb and Aiden had their very first camping trip. For three days and two nights they slept under the stars, swam in the reservoir, drove their cars around in the dirt, tried out the jet-ski for the first time and ate pizza mountain pies. We had a blast! Plus, Keith and his brother picked one of the best sites we've ever had! It was SO pretty and a short distance to the kybo. Day two K&A could run the trails themselves and knew exactly where to go.
We shared a tent with the boys and everyone slept on air mattresses. It was so fun and I know they'll be really excited to do it again soon! It felt great to relax and eat s'more after s'more huddled around the campfire. I sang kon-o-kwee songs non-stop and loved waking up to sweatshirt weather and excited toddlers.


text:
While I was waiting for my ordered fabric to arrive, I went back to my Mary Wigham. I'm on to the last motif at the bottom left corner, that means I'm 1/3 done on this piece – well, almost… For some reason this motif is giving me so much trouble!!!! I can't remember how many times I've frogged it, and last night I had to fix the tail of the bird 3 times and still didn't get in right!!! I was sooo hoping to finish this before the fabrics arrive so I can go back to my quilting!!
And of course these beautiful fabrics arrived today, which made me feel much better when I saw them. I did a little web surfing last week and was so excited when I found these fabric from the same 1974 series of my tumbler quilt~~ So I'm going to set Mary aside for the moment and work on my quilt again, just to change the scene a little bit and hope the frog will be gone when I get back to it.
Did you see Jennifer's awesome finds at market? She is generously giving away some stash on her blog, be sure to go check it out!!


text:
Very, very exciting news! I just led my first ever professional development at our county PD Day! Back in November, my AP and Curriculum Specialist approached me and the other 5th grade Math/Science teacher about possibly presenting Math Workshop/Guided Math, first to our staff and then to the county. Of course, we said yes!
Trial Run
Right before Winter Break, we presented a trial run, abbreviated version (20 minutes) to our staff. Really, we just presented an overview of what we do, why we do it, and answered questions. I created a cheat sheet to help guide us and for everyone to take with them. Click on the image to download for FREE from my TPT account.
The Big Day!
Whew! Four 90 minute sessions is no joke! At least we got to enjoy adult lunch. :)
We used a PowerPoint to guide our discussion with activities thrown in. We had such great feedback. After our first session, teachers were telling their colleagues to come to our session so people were just randomly showing up. I loved it! It was nice to be around a different group of teachers and to hear what is going on in other schools. Our county is all over the place. Some schools use small group instruction in reading and some teachers were told they are not allowed to use small group centers (huh?).
|Pros/Cons of Whole Group Instruction
Teachers were e-mailing their principals asking for permission to come observe our math classes in action! How cool is that?! A lot of times we leave PD sessions annoyed and tired. It was great to see people (except the 2 party poopers) leaving re-energized and excited about teaching.
Anywho, that's enough bragging on myself for today. Off to finish report card grades!
PS Loved this quote hung in the high school today: