text:
tag:blogger.com,1999:blog-86599129599760795542018-09-17T07:58:31.934+01:00swl10Steve firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-44555696550580567062016-01-03T14:54:00.001+00:002016-01-03T14:54:51.844+00:00sphinx: ignoring sys.path?
I'm a big fan of using Sphinx for documentation and I use it for my Pyslet Python module. For some time now I've had an infuriating problem with generating documentation that I've been putting off solving on the assumption that it is caused by some weird and hard to discover configuration issue.

The problem is that whenever I run sphinx to build my package documentation it uses the currently installed version of the package and not the local copy I'm working on. This is particularly annoying because I have various versions of Python installed on my Mac (going back some years) and sphinx always seems to run with the native Python that came with XCode (Python 2.7.10 at the time of writing) whereas typing "python" at the terminal uses one of my older custom builds. I guess I'd always thought there was something weird going on and just relied on the workaround which was to run:

$/usr/bin/python setup.py install

...every time I wanted to rebuild the docs. That forced the latest version of the package to be installed into the Python interpreter that was going to be run by sphinx.

I had already edited my sphinx conf.py file as follows:

sys.path.insert(0, os.path.abspath('..'))

That should have been enough to put the correct path to the working copy in the search order before anything that was installed in the site packages but, alas, it just didn't seem to work. I checked that sys.path was correct, I even started the interpreter on the command line to ensure the working copies were getting loaded after this modification. They were, it seemed inexplicable!

I finally solved the mystery today, and thought I'd blog the answer in case anyone else does the same stupid thing I did.

Earlier today I added a new module to my package and running the docs gave me an import error. Previously I'd thought that some strange site-specific import hook was causing sys.path to be ignored but an ImportError suggested that sys.path was being completely ignored. I got so frustrated that I edited the autodoc.py package to check that sys.path was set correctly just before it calls __import__ and then I set a breakpoint to see if I could see what was happening. I tried setting sys.path to a single directory, the one with the working version of my package in it. Still the import brought up the version installed in site packages. How is this possible?

When you import a module with a name like "package.modA" you're actually (sort of) importing the package first and not the module. As a result, if the package has already imported and you execute import package.modB Python will ignore sys.path because it already knows where the package is. This was exactly what was going wrong for me...

The Solution

My package contains a trivial module called 'info' which contains a few strings that I use to describe the package both within the package itself and also in setup.py. At some point I must have gone in to the conf.py I use for Sphinx and optimised away some of the redundant text by importing the strings directly from my package...

#
# conf.py
#
import pyslet.info as info

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('..'))

# ... [snip]

# General information about the project.
project = info.title_name
copyright = info.copyright

Notice that conf.py imports my 'info' module before I change sys.path, as a result it gets imported from site packages and every future import from my pyslet package will come from site packages too. The solution is simple, and sanity has been restored:

#
# conf.py
#

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('..'))
import pyslet.info as info

# ... [snip]

# General information about the project.
project = info.title_name
copyright = info.copyright

And now my documentation builds correctly from the working copy. I cannot believe that I'm the first person to trip over this issue. A bit of Googling (after the fact) reveals this module, for example: CodeChat conf.py which uses a similar technique to mine.

Happy New Year!
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-53095688751208662662015-10-11T17:15:00.000+01:002015-10-11T17:15:37.948+01:00Pyslet goes https
After months of being too busy to sort this out I have finally moved the Pyslet website to SSL. This is a quick post to explain how I've done this.

Firstly, I've wanted to do this for a while because I want to use the above website to host a web version of the QTI migration tool but encouraging users to upload their precious assessment materials to a plain old HTTP URL should (hopefully would) have proved a challenge. I saw an advert for free SSL certificates for open source projects from GlobalSign so in a rush of enthusiasm I applied and got my certificate. There's a checklist of rules that the site must comply with to be eligible (see previous link) which I'll summarise here:

OSI license: Pyslet uses the BSD 3-Clause License: check!

Actively maintained: well, Pyslet is a spare-time activity but I'm going to give myself a qualified tick here.

Not used for commercial purposes: the Pyslet website is just a way of hosting demos of Pyslet in action, no adverts, no 'monetization' of any kind: check!

Must get an A rating with GlobalSign's SSL Checker...

That last one is not quite as easy as you might think. Here's what I did to make it happen, I'll assume you have already dome some openssl magic, applied for and received your crt file.

Download the intermediate certificate chain file from GlobalSign here, the default one for SHA-256 Orders was the correct one for me.

Put the following files into /var/www/ssl (your location may vary):

www.pyslet.org.key
www.pyslet.org.crt
globalsign-intermediate.crt

The first one is the key I originally created with:

openssl genrsa -des3 -out www.pyslet.org.key.encrypted 2048
openssl req -new -key www.pyslet.org.key.encrypted -out www.pyslet.org.csr
openssl rsa -in www.pyslet.org.key.encrypted -out www.pyslet.org.key

The second file is the certificate I got from GlobalSign themselves. The third one is the intermediate certificate I downloaded above.

Set permissions (as root):
chown -R root:root /var/www/ssl/*.key
chmod 700 /var/www/ssl/*.key

Add a virtual host to Apache's httpd.conf (suitable for Apache/2.2.31):
Listen 443

<VirtualHost *:443>
 ServerName www.pyslet.org
 SSLEngine on

 SSLCertificateFile /var/www/ssl/www.pyslet.org.crt
 SSLCertificateKeyFile /var/www/ssl/www.pyslet.org.key
 SSLCertificateChainFile /var/www/ssl/globalsign-intermediate.crt

 SSLCompression off
 SSLProtocol all -SSLv3 -SSLv2
 SSLCipherSuite AES128+EECDH:AES128+EDH
 SSLHonorCipherOrder on

# Rest of configuration goes here....

</VirtualHost>

This is a relatively simple configuration designed to get an A rating while not worrying too much about compatibility with really old browsers.
Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-62908269731892171032015-02-16T09:24:00.000+00:002015-02-16T09:24:50.934+00:00Accessing the ESA Sentinel Mission Data with Python and OData
I've had a couple of enquiries now about how to access the OData feeds on the ESA Sentinel mission science data hub. Sentinel 1 is the first of a new group of satellites in the Copernicus programme to monitor the Earth. That's about all I know I'm afraid. This data is not pretty desktop pictures (though doubtless there are some pretty pictures buried in there somewhere) but raw scientific data from instruments currently orbiting the Earth.

The source code described here is available in the samples directory on GitHub, you must be using the latest Pyslet from master for this script to enable the metadata override technique used here.

The data hub advertises access to the data through OData (version 1) but my Python library, Pyslet, was not able to access the feeds properly: hence the enquiries.

Turns out that the data feeds use a concept called containment in OData. The model of OData is one of entity sets (think SQL tables) with relations between them modelled by navigation properties. There's one particular use case that doesn't work very well in this scenario but seems popular. Given an entity (think table row or record) people want to add arbitrary key-value pairs. The ESA's data model does this by creating 'sub-tables' which define collections of attributes that hang off of each entity. The attribute name is the key in these collections. This doesn't really work in OData v1 (or v2) because these attribute values should still be entities in their own right and therefore they need a unique key and an entity set definition to contain them.

This isn't the only schema I've seen that attempts to do something like this either, SAP have published a similar schema suggesting that some early Java tools exposed OData this way.

The upshot is that you get nasty errors when you try and load these services with Pyslet. It complains of a rather obscure condition concerning (possibly multiple) unbound principals. When I wrote that error message, I didn't expect anyone to ever actually see it.

There's a proper way to do containment in earlier versions of OData, described in Containment is Coming with OData v4 which explains how to use composite keys. As the name of the article suggests though, this is written with hindsight after a better solution has been found for this use case in OData v4.

The fix for the ESA data feed is to download and edit a copy of the advertised metadata to get around the errors reported by Pyslet and then to initialise your OData client using this modified schema instead. It isn't a perfect fix, as far as Pyslet knows those attributes really are unique and do reside in their own entity set but it doesn't really matter for the purposes of using the OData client. You can navigate and formulate queries without tripping over data inconsistencies.

I've written a little script that I've added to Pyslet's sample code directory to illustrate the technique, along with a fixed up metadata file. The result is a little UNIX-style utility for downloading products from the ESA data hub:

$ ./download.py --help
Usage: download.py [options]

Options:
 -h, --help show this help message and exit
 -u USER, --user=USER user name for basic auth credentials
 -p PASSWORD, --password=PASSWORD
 password for basic auth credentials
 -v increase verbosity of output up to 3x
 -c, --cert download and trust site certificate

The data is available via https and requires a user name and password (you'll have to register on the data hub site but it's free to do so). To make it easier to set up the trust aspect I've added a -c option to download the site certificate and store it. If you don't have the site certificate you'll get an error like this:

ERROR:root:scihub.esa.int: closing connection after error failed to build secure connection to scihub.esa.int

Subsequent downloads verify that the site certificate hasn't changed: a bit like the way ssh offers to store a fingerprint the first time you connect to a remote host. Only use the -c option if you trust the network you are running on (you can use Firefox or some other 'trusted' browser to download the certificate too of course).

The password is optional, if you don't provide it you'll be prompted to enter it using Python's getpass function for privacy.

You pass the product identifiers as command line arguments, here is an example of a successful first-time run:

$ ./download.py -c -u swl10 8bf64ff9-f310-4027-b31f-8e95dd9bbf82
Password:
ERROR:root:Entity set Attributes has more than one unbound principal
dropping mutliplicity of Attribute_Node to 0..1. Continuing
ERROR:root:Entity set Attributes has more than one unbound principal
dropping mutliplicity of Attribute_Product to 0..1. Continuing
S1A_EW_GRDM_1SDH_20150207T084156_20150207T084218_004515_0058AE_3051 150751068

After running this command I had a scihub.esa.int.crt file (from the -c option) and a 150MB zip file downloaded to the current directory.

If you run with -vv to provide a bit more information you can see the OData magic in operation:

./download.py -vv -u swl10 8bf64ff9-f310-4027-b31f-8e95dd9bbf82
Password:
INFO:root:Sending request to scihub.esa.int
INFO:root:GET /dhus/odata/v1/ HTTP/1.1
INFO:root:Connected to scihub.esa.int with DHE-RSA-AES256-SHA, TLSv1/SSLv3, key length 256
INFO:root:Finished Response, status 401
INFO:root:Resending request to: https://scihub.esa.int/dhus/odata/v1/
INFO:root:Sending request to scihub.esa.int
INFO:root:GET /dhus/odata/v1/ HTTP/1.1
INFO:root:Connected to scihub.esa.int with DHE-RSA-AES256-SHA, TLSv1/SSLv3, key length 256
INFO:root:Finished Response, status 200
WARNING:root:Entity set Attributes has an unbound principal: Nodes
WARNING:root:Entity set Attributes has an unbound principal: Products
ERROR:root:Entity set Attributes has more than one unbound principal
dropping multiplicity of Attribute_Node to 0..1. Continuing
ERROR:root:Entity set Attributes has more than one unbound principal
dropping multiplicity of Attribute_Product to 0..1. Continuing
INFO:root:Sending request to scihub.esa.int
INFO:root:GET /dhus/odata/v1/Products('8bf64ff9-f310-4027-b31f-8e95dd9bbf82') HTTP/1.1
INFO:root:Connected to scihub.esa.int with DHE-RSA-AES256-SHA, TLSv1/SSLv3, key length 256
INFO:root:Finished Response, status 200
S1A_EW_GRDM_1SDH_20150207T084156_20150207T084218_004515_0058AE_3051 150751068
INFO:root:Sending request to scihub.esa.int
INFO:root:GET /dhus/odata/v1/Products('8bf64ff9-f310-4027-b31f-8e95dd9bbf82')/$value HTTP/1.1
INFO:root:Connected to scihub.esa.int with DHE-RSA-AES256-SHA, TLSv1/SSLv3, key length 256
INFO:root:Finished Response, status 200

As you can see, the fixed up metadata still generates error messages but these are no longer critical and the client is able to interact with the service.

I was given this product identifier as an example of something small to test with. I haven't researched what the data actually represents but the resulting zip file does contain a 'quick_look' image:

Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-39939429899568177192015-01-19T10:05:00.001+00:002015-01-19T10:05:43.884+00:00Yosemite Spotlight issues with HP drivers: check your console
I recently imported a bunch of email into Outlook for OS X and was disappointed that I was unable to search its contents. Outlook uses Apple's native spotlight search so, in theory, all I need to do is wait for spotlight to churn through the new material and I should be done. Hours passed and nothing seemed to happen.

The first thing I tried was to simply force a re-index of my hard-disk. I just added my main drive to the list of places to exclude from spotlight searching and then after waiting a minute or two (for superstitious reasons) I removed that item again and sat back waiting for the inevitable slowdown as mdworker launches into life and starts scanning all my data.

Nothing.

The next step I took was to try to figure out how to see if Spotlight was actually doing any indexing at all. There's no simple control panel or dashboard view of the indexing process. The only way I could find was to press command-space and search for something. It should show the indexing progress-bar if it is indexing (but if it is complete you'll see nothing). I was still getting nowhere and now I'd lost the ability to search for anything.

Check the Console

In these situations it is always worth checking the console. I don't mean the termainal, just the console utility app that spools system messages onto your screen and allows you to see what is happening on your Mac. There's a handy search box (which doesn't use Spotlight!) at the top which filters the current day's logs. Putting just 'md' in that box was enough to filter all the other stuff enabling me to see a constant stream of output from Spotlight's indexing application: mdworker.

Here's a sample:

Jan 12 23:06:45 LernaBookPro com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[24329]): Could not find uid associated with service: 0: Undefined error: 0 1422
Jan 12 23:06:45 LernaBookPro com.apple.xpc.launchd[1] (com.apple.mdworker.bundles): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.

You don't need to be an expert to see that this is some type of unexpected condition and the second line tells me that the resulting process exited out straight away. True to its word, every 10 seconds I got a pair of lines like this in my console log. Interestingly, the problem had been going on for some time, when I searched back through my logs I realised that Spotlight had probably not been indexing properly for ages. I have a feeling that things like new mail that arrives in your inbox gets indexed through some other mechanism and this can mask the fact that the long running indexer has not made a proper index of your hard disk. But that's just a hunch.

Check the web

Armed with this information I was able to find a thread on the internet that clearly dealt with the problem I was having: [...] com.apple.mdworker.bundles pollute logs with errors. Clearly the person who posted this thread didn't think (or perhaps realise) that Spotlight had actually failed and was chasing up unexpected slowness.

Interestingly, from this thread it is clear that the last number in the log entry is a numeric user id. In the case of the poster this was 502 which is typical of the range Apple uses for real users you add to your machine from the control panel. I think they start at 501. If you delete a user from your machine but leave lots of data lying around that is owned by that user then Yosemite seems to be having trouble and it is killing the Spotlight indexer.

The user id causing trouble for me is 1422, which is outside of this range so although the remedy might be similar the origin of my problem is different. I put 1422 into my search and found this thread: 27 inch iMac suddenly running very slowly where the person erases their disk and re-installs (yeah, that fixed it!).

Now use the Terminal

With the clues in the first thread it seems like I need to find some files on my disk owned by user id 1422. The terminal can do that using the Unix find command:

cd /
sudo find . -user 1422

This type of thing takes ages, especially if you have a large backup drive.

The Culprit

Turns out that the files owned by user 1422 are all part of the HP printer drivers. These may have been inherited from a previous installation, I'm not that sure, but either way there was information in /Library and inside the Application bundles themselves from HP that were identified this way. I had to use chown -R to change those to root ownership instead (I'm not sure what they are supposed to be).

You'll also find a few files in /private/var/ with paths similar to:

/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.Spotlight/1422

The exact names will be different but the critical thing is the end part, which is a directory created just for spotlight. It is named after the missing user and it has its ownership set to this non-existent user. It is actually these latter files which are causing the problem, Spotlight is trying to create an index for that user but is surprised when it finds the user doesn't exist. Just removing this directory isn't enough though because Spotlight will re-index your disk and as soon as it finds a file owned by 1422 again it will create this folder in /private/var and grind to a halt again. You must remove or re-own everything that spotlight might see: a real hassle if you have a large backup drive because of the way Time Machine works. I've solved that problem by just excluding my backup drive from Spotlight.

FWIW, unix systems are usually very tolerant of non-existent user ids. Many archive programs will restore files from other machines and systems and, if run as root, will update their ownership to match the original ownership before the files were archived. On networks of Unix workstations that share a user directory this is useful because you can tar up files on one machine and transfer them to another and all the ownership information comes across too. On a personal machine this is less useful and perhaps even dangerous, hence the '--insecure' option on tar.

I considered removing and reinstalling the HP software but I'm not convinced that it isn't a problem with the installer itself. It works fine on the machine I upgraded from 10.8, through 10.9 to 10.10 (after these fixes) but I noticed that on a different machine that came with Mavericks and was upgraded to Yosemite I had to re-install the driver even though I used the migration assistant to set it up from its predecessor (running 10.8) which did have the HP drivers installed. I struggled to find the right download for my HP Officejet 6310 and perhaps now I know why!

Restart

I couldn't find any way of telling mdworker to give up on user id 1422. Removing it's special directory didn't seem to help so I assume that somewhere a process has a cache of that information and the only way I could figure to get it going again was a restart.

Conclusion

If you are using Yosemite and have an HP printer or scanner check your console just in case Spotlight has died for you too. Do battle with the terminal. Restart. Enjoy Spotlight indexing and smoother performance from your Mac.
 Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-75825084579075020802014-11-24T01:06:00.000+00:002014-11-24T08:12:41.210+00:00LTI Tools: Putting Cookies in the Frame
Last year I ran into a problem with IMS LTI in some versions of Internet Explorer. It turns out that my LTI tool was assuming it was OK to save cookies in the browser but IE was assuming it wasn't. Chuck has written about this briefly on his blog and the advice given there is basically now best practice and documented in the specification itself. See LTI, Frames and Cookies – Oh MY!

So I just spent the day trying to implement something like this ready for my QTI migration tool to become an LTI tool rather than a desktop application and it was much harder than I thought it would be.

Firstly, why is this happening? Well if your tool is launched in a frame inside the tool consumer then you are probably falling in to the definition of 'third party' as far as cookies are concerned. There is clearly a balance to be had between privacy and utility here. If you think about it, framed pages probably know who framed them and so something innocuous like an advert for an online store could use cookies to piece together a browser history of every site you visited that included their ad. That's why when you visit your favourite online store after browsing some informational site to do product research the store seems to already know what you want to buy.

Cynics will see a battle between companies like Google and Amazon who make money by encouraging you to find products online (and buy them) and Microsoft who are more into the business of selling software and services, especially to businesses who may not appreciate having their purchasing department gamed. Perhaps it is no wonder that Amazon found themselves in court in Seattle having to defend their technical implementation of P3P.

It turns out that IE can/could be persuaded to accept your tool's cookie if your tool publishes a privacy policy which IE isn't able to parse. I don't want to appear too dismissive but I simply cannot understand how decisions about privacy can be codified into policies that computers can then read and execute with any degree of usefulness. Stackoverflow has some advice on how to do it 'properly' if you are that way inclined, however.

For the rest of us, we're going to have to get used to the idea that cookies may not be accepted and follow Chuck's advice to workaround the issue by opening a new window. It isn't just IE now anyway, by default Safari will block cookies in this situation too. Yes, the solution is horrible, but how horrible?

Some time ago I wrote some Python classes to help implement basic LTI. That felt like a good starting point for my new LTI tool. Here's Safari, pointing at the IMS LTI test harness which I've just primed with the URL of my tool: http://localhost:8081/launch

The page at: http://www.imsglobal.org/developers/LTI/test/v1p1/lms.php

In the next shot, I've scrolled down to where the action is going to take place. I've opened Safari's developer tools so we can see something of what is happening with cookies.

So what happens when I hit "Press to Launch"? It all happens rather quickly but the frame POSTs to my launch page, which then redirects to a cookie test page. That page realises that the cookies didn't stick and shows a form which autosubmits, popping up a new window with my tool content in. The content is not very interesting but here's how my browser looked immediately after:

There's a real risk that this window will be blocked by a pop-up blocker. Indeed, Chrome does seem to block this type of pop-up window. In fact, Chrome allows the original frame to use cookies so on that browser the pop-up doesn't need to fire. I only tripped the blocker during simulated tests. Still, it is clear that you do need to consider that LTI tools may neither have access to cookies nor be able to pop-up a new window automatically. To cover this case my app has a button that allows you to open the window manually instead, if I change tabs back to the Tool Consumer page you'll see how my pop-up sequence has left the original page/frame.

Notice that Safari now shows my two cookies. These cookies were set by the new window, not by the content of this frame, even though they show up in this view. They show me that my tool has not only displayed its content but has successfully created a cookie-trackable session.

The code required to make this happen is more complex than I thought. I've tried to represent my session construction logic in diagramatic form. Each box represents a request. The top line contains the incoming cookie values ('-' means missing) and the page name. The next line indicates the values of some key query parameters used to pass Session information, Return URL and whether or not a pop-up Window has been opened. The final line shows the out-going cookies. If you follow the right hand path from top to bottom you see a hit to the home page of the tool (aka launch page) with no cookies. It sets the values 0/A and redirects to the test page which confirms that cookies work, outputs a new session identifier (B) and redirects to the content. The left-turns in the sequence show the paths taken when cookies that were set are blocked by the browser.

Why does the session get rewritten from value A to B during the sequence? Just a bit of paranoia. Exposing session identifiers in URLs is considered bad form because they can be easily leaked. Having taken the risk of passing the initial session identifier through the query string we regenerate it to prevent any form of session hijacking or fixation attacks. This is quite a complex issue and is related to other weaknesses such as cross-site request forgery. I found Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet very useful reading!

I will check the code for this into my Pyslet GitHub project shortly, if you want a preview please get in touch or comment on this post. The code is too complex to show in the blog post itself.
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-77454641951195640992014-11-14T07:30:00.000+00:002017-05-24T07:01:46.931+01:00Basic Authentication, SSL and Pyslet's HTTP/OData client
Pyslet is my Python package for Standards in Learning Education and Training and represents a packaging up of the core of my QTI migration script code in a form that makes it easier for other developers to use. Earlier this year I released Pyslet to PyPi and moved development to Github to make it easier for people to download, install and engage with the source code.

Note: this article updated 2017-05-24 with code correction (see comments for details).

Warning: The code in this article will work with the latest Pyslet master from Github, and with any distribution on or later than pyslet-0.5.20141113. At the time of writing the version on PyPi has not been updated!

A recent issue that came up concerns Pyslet's HTTP client. The client is the base class for Pyslet's OData client. In my own work I often use this client to access OData feeds protected with HTTP's basic authentication but I've never properly documented how to do it. There are two approaches...

The simplest way, and the way I used to do it, is to override the client object itself and add the Authorization header at the point where each request is queued.

from pyslet.http.client import Client

class MyAuthenticatedClient(Client):

 # add an __init__ method to set some credentials
 # in the client

 def queue_request(self, request):
 # add in the authorization credentials
 if (self.credentials is not None and
 not request.has_header("Authorization")):
 request.set_header('Authorization',
 str(self.credentials))
 super(MyAuthenticatedClient, self).queue_request(request)

This works OK but it forces the issue a bit and will result in the credentials being sent to all URLs, which you may not want. The credentials object should be an instance of pyslet.http.auth.BasicCredentials which takes care of correctly formatting the header. Here is some sample code to create that object:

from pyslet.http.auth import BasicCredentials
from pyslet.rfc2396 import URI

credentials = BasicCredentials()
credentials.userid = "email@example.com"
credentials.password = "secretPa$$word"
credentials.protectionSpace = URI.from_octets(
 'https://www.example.com/mypage').get_canonical_root()

With the above code, str(credentials) returns the string: 'Basic dXNlckBleGFtcGxlLmNvbTpzZWNyZXRQYSQkd29yZA==' which is what you'd expect to pass in the Authorization header.

To make this code play more nicely with the HTTP standard I added some core-support to the HTTP client itself, so you don't need to override the class anymore. The HTTP client now has a credential store and an add_credentials method. Once added, the following happens when a 401 response is received:

The client iterates through any received challenges

Each challenge is matched against the stored credentials

If matching credentials are found then an Authorization header is added and the request resent

If the request receives another 401 response the credentials are removed from the store and we go back to (1)

This process terminates when there are no more credentials that match any of the challenges or when a code other than 401 is received.

If the matching credentials are BasicCredentials (and that's the only type Pyslet supports out of the box!), then some additional logic gets activated on success. RFC 2617 says that for basic authentication, a challenge implies that all paths "at or deeper than the depth of the last symbolic element in the path field" fall into the same protection space. Therefore, when credentials are used successfully, Pyslet adds the path to the credentials using BasicCredentials.add_success_path. Next time a request is sent to a URL on the same server with a path that meets this criterium the Authorization header will be added pre-emptively.

If you want to pre-empt the 401 handling completely then you just need to add a suitable path to the credentials before you add them to the client. So if you know your credentials are good for everything in /website/~user/ you could continue the above code like this:

credentials.add_success_path('/website/~user/')

That last slash is really important, if you leave it off it will add everything in '/website/' to your protection space which is probably not what you want.

SSL

If you're going to pass basic auth credentials around you really should be using https. Python makes it a bit tricky to use HTTPs and be sure that you are using a trusted connection. Pyslet tries to make this a little bit easier. Here's what I do.

With Firefox, go to the site in question and check that SSL is working properly

Export the certificate from the site in PEM format and save to disk, e.g., www.example.com.crt

Repeat for any other sites I want my python script to work with.

Concatenate the files together and save them to, say, 'certificates.pem'

Pass this file name to the HTTP (or OData) client constructor.

from pyslet.http.client import Client

my_client = Client(ca_certs='certificates.pem')
my_client.add_credentials(credentials)

In this code, I've assumed that the credentials were created as above. To be really sure you are secure here, try grabbing a file from a different site or, even better, generate a self-signed certificate and use that instead. (The master version of Pyslet currently has such a certificate ready made in unittests/data_rfc2616/server.crt). Now pass that file for ca_certs and check that you get SSL errors! If you don't, something is broken and you should proceed with caution, or you may just be on a Mac (see notes in Is Python's SSL module correctly validating certificates... for details). And don't pass None for ca_certs as that tells the ssl module not to check at all!

If you don't like messing around with the certificates, and you are using a machine and network that is pretty trustworthy and from which you would happily do your internet banking then the following can be used to proxy for the browser method:

import ssl, string
import pyslet.rfc2396 as uri

certs = []
for s in ('https://www.example.com', 'https://www.example2.com', ):
 # add other sites to the above tuple as you like
 url = uri.URI.from_octets(s)
 certs.append(ssl.get_server_certificate(url.get_addr(),
 ssl_version=ssl.PROTOCOL_TLSv1))
 with open('certificates.pem', 'wb') as f:
 f.write(string.join(certs,''))

Passing the ssl_version is optional above but the default setting in many Python installations will use the discredited SSLv3 or worse and your server may refuse to serve you, I know mine does! Set it to a protocol you trust.

Remember that you'll have to do this every so often because server certificates expire. You can always grab the certificate authority's certificate instead (and thereby trust a whole slew of sites at once) but if you're going that far then there are better recipes for finding and re-using the built-in machine certificate store anyway. The beauty of this method is that you can self-sign a server certificate you trust and connect to it securely with a Python client without having to mess around with certificate authorities at all, provided you can safely courier the certificate from your server to your client that is! If you are one of the growing number of people who think the whole trust thing is broken anyway since Snowden then this may be an attractive option.

With thanks to @bolhovsky on Github for bringing the need for this article to my attention.
 Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-76832834746303959202014-11-04T08:28:00.001+00:002014-11-04T08:28:43.807+00:00VMWare Fusion, Windows 8 and British Keyboards
I've been running VMWare Fusion for years to run a dual monitor setup with one screen Mac and the other screen 'PC'. I recently upgraded to VMWare Fusion 7 and my Windows 7 virtual machine kept working just fine. However I recently had to start from scratch with a new VM to install Windows 8.1. I realise that there may be upgrade routes via Windows 8 but, after two hours on the phone with Microsoft technical support for an unrelated issue I personally decided that a clean install would be the way to go.

The problem is, the keyboard just doesn't work properly in the default setup. It is years since I had to play with these things and it took me a while to figure out how to put mappings in place that make my British keyboard do the right thing. There may actually be a problem with Fusion 7 if this forum thread is anything to go by.

The symptoms are that the '@' symbol, which is typed with shift-2 on the Mac keyboard was coming out as double-quote like it would on a PC keyboard and try as I might I could not figure out how to type a back-slash at first at all.

The solution is to go to the VM's settings, duplicate the profile you are using (VMWare created a Windows 8 profile for me) and then add some custom mappings. I found this a bit counter intuitive because you need to start by opening something like Notepad in the VM and experimenting until you find the characters you want, note down the key combination you pressed to get them and then switch to the profile screen and add a mapping from the keys you want to press to the key combinations you just discovered work. A picture will help...

Shift-'

This is the double quote on my keyboard but without the custom profile it types '@', a simple switch with Shift-2 that almost all Mac users are probably already familiar with.

Shift-2

The reverse mapping of the above

Option-3

Turns out that the backslash key types a hash/number/sharp sign on the PC so we map Option-3 to that if, like me, you have become habituated to the Apple way.

\

Turns out that the backslash is now typed by the section sign (that's the curly thing that looks like a tiny spiral-arm galaxy).

Shift-`

Lastly, you may have to hunt for the '~', but it gets typed when you hit shift-backslash on the PC. On the Mac it appears over the back-quote or back-tick symbol.

That's the most important ones. Note that shift-3 correctly types the pound-sterling sign '£' so I didn't have to touch that at all. I have no idea how one types a section sign or the combined plus-minus '±' sign on the PC but in my line of work there is very little call for them. I warmly invite readers to add their own suggested mappings as comments to this post!
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-15586947893642074572014-05-26T11:56:00.000+01:002014-05-26T11:56:13.456+01:00Adding OData support to Django with Pyslet: First Thoughts
A couple of weeks ago I got an interesting tweet from @d34dl0ck, here it is:

This got me thinking, but as I know very little about Django I had to do a bit of research first. Here's my read-back of what Django's data layer does in the form of a concept mapping from OData to Django. In this table the objects are listed in containment order and the use case of using OData to expose data managed by a Django-based website is assumed. (See below for thoughts on consuming OData in Django as if it were a data source.)

OData Concept	Django Concept	Pyslet Concept

DataServices	Django website: the purpose of OData is to provide access to your application's data-layer through a standard API for machine-to-machine communication rather than through an HTML-based web view for human consumption.	Instance of the DataServices class, typically parsed from a metadata XML file.

Schema	No direct equivalent. In OData, the purpose of the schema is to provide a namespace in which definitions of the other elements take place. In Django this information will be spread around your Python source code in the form of class definitions that support the remaining concepts.	Instance of the Schema class, typically parsed from a metadata XML file.

EntityContainer	The database. An OData service can define multiple containers but there is always a default container - something that corresponds closely with the way Django links to multiple databases. Most OData services probably only define a single container and I would expect that most Django applications use the default database. If you do define custom database routers to map different models to different databases then that information would need to be represented in the corresponding Schema(s).	In Pyslet, an EntityContainer is defined by an instance of the EntityContainer class but this instance is handed to a storage layer during application startup and this storage layer class binds concrete implementations of the data access API to the EntitySets it contains.

EntitySet	Your model class. A model class maps to a table in the Django database. In OData the metadata file contains the information about which container contains an EntitySet and the EntityType definition in that file contains the actual definitions of the types and field names. In contrast, in Django these are defined using class attributes in the Python code.	Pyslet sticks closely to the OData API here and parses definitions from the metadata file. As a result an EntitySet instance is created that represents this part of the model and it is up to the object responsible for interfacing to the storage layer to provide concrete bindings.

Entity	An instance of a model class.	An instance of the Entity object, typically instantiated by the storage object bound to the EntitySet.

Where do you start?

Step 1: As you can see from the above table, Pyslet depends fairly heavily on the metadata file so a good way to start would be to create a metadata file that corresponds to the parts of your Django data model you want to expose. You have some freedom here but if you are messing about with multiple databases in Django it makes sense to organise these as separate entity containers. You can't create relationships across containers in Pyslet which mirrors the equivalent restriction in Django.

Step 2: You now need to provide a storage object that maps Pyslet's DAL onto the Django DAL. This involves creating a sub-class of the EntityCollection object from Pyslet. To get a feel for the API my suggestion would be to create a class for a specific model initially and then, once this is working, consider how you might use Python's built-in introspection to write a more general object.

To start with, you don't need to do too much. EntityCollection objects are just like dictionaries but you only need to override itervalues and __getitem__ to get some sort of implementation going. There are simple wrappers that will (inefficiently) handle ordering and filtering for you to start with so itervalues can be very simple...

def itervalues(self):
 return self.OrderEntities(
 self.ExpandEntities(
 self.FilterEntities(
 self.entityGenerator())))

All you need to do is write the entityGenerator method (the name is up to you) and yield Entity instances from your Django model. This looks pretty simple in Django, something like Customer.objects.all() where Customer is the name of a model class would appear to return all customer instances. You need to yield an Entity object from Pyslet's DAL for each customer instance and populate the property values from the fields of the returned model instance.

Implementing __getitem__ is probably also very easy, especially when you are using simple keys. Something like Customer.objects.get(pk=1) and then a similar mapping to the above seems like it would work for implementing basic resource look up by key. Look at the in-memory collection class implementation for the details of how to check the filter and populate the field values, it's in pyslet/odata2/memds.py.

Probably the hardest part of defining an EntityCollection object is getting the constructor right. You'll want to pass through the Model class from Django so that you can make calls like the above:

def __init__(self,djangoModel,**kwArgs):
 self.djangoModel=djangoModel
 super(DjangoCollection,self).__init__(**kwArgs)

Step 3: Load the metadata from a file, then bind your EntityCollection class or classes to the EntitySets. Something like this might work:

import pyslet.odata2.metadata as edmx
doc=edmx.Document()
with open('DjangoAppMetadata.xml','rb') as f:
 doc.Read(f)
customers=doc.root.DataServices['DjangoAppSchema.DjangoDatabase.Customers']
# customers is an EntitySet instance
customers.Bind(DjangoCollection,djangoModel=Customer)

The Customer object here is your Django model object for Customers and the DjangoCollection object is the EntityCollection object you created in Step 2. Each time someone opens the customers entity set a new DjangoCollection object will be created and Customer will be passed as the djangoModel parameter.

Step 4: Test that the model is working by using the interpreter or a simple script to open the customers object (the EntitySet) and make queries with the Pyslet DAL API. If it works, you can wrap it with an OData server class and just hook the resulting wsgi object to your web server and you have hacked something together.

Post hack

You'll want to look at Pyslet's expression objects and figure out how to map these onto the query objects used by Django. Although OData provides a rich query syntax you don't need to support it all, just reject stuff you don't want to implement. Simple queries look like they'd map to things you can pass to the filter method in Django fairly easily. In fact, one of the problems with OData is that it is very general - almost SQL over the web - and your application's data layer is probably optimised for some queries and not others. Do you want to allow people to search your zillion-record table using a query that forces a full table scan? Probably not.

You'll also want to look at navigation properties which map fairly neatly to the relationship fields. The Django DAL and Pyslet's DAL are not miles apart here so you should be able to create NavigationCollection objects (equivalent to the class you created in Step 2 above) for these. At this point, the power of OData will begin to come alive for you.

Making it Django-like

I'm not an expert on what is and is not Django like but I did notice that there is a Feed concept for exposing RSS in Django. If the post hack process has left you with a useful implementation then some sort of OData equivalent object might be a useful addition. Given that Django tends to do much of the heavy lifting you could think about providing an OData feed object. It probably isn't too hard to auto-generate the metadata from something like class attributes on such an object. Pyslet's OData server is a wsgi application so provided Django can route requests to it you'll probably end up with something that is fairly nicely integrated - even if it can't do that out of the box it should be trivial to provide a simple Django request handler that fakes a wsgi call.

Consuming OData

Normally you think of consuming OData as being easier than providing it but for Django you'd be tempted to consider exposing OData as a data source, perhaps as an auxiliary database containing some models that are externally stored. This would allow you to use the power of Django to create an application which mashed up data from OData sources as if that data were stored in a locally accessible database.

This appears to be a more ambitious project: Django non-rel appears to be a separate project and it isn't clear how easy it would be to intermingle data coming form an OData source with data coming from local databases. It is unlikely that you'd want to use OData for all data in your application. The alternative might be to try and write a Python DB API interface for Pyslet's DAL and then get Django treating it like a proper database. That would mean parsing SQL, which is nasty, but it might be the lesser of two evils.

Of course, there's nothing stopping you using Pyslet's builtin OData client class directly in your code to augment your custom views with data pulled from an external source. One of the features of Pyslet's OData client is that it treats the remote server like a data source, keeping persistent HTTP connections open, managing multi-threaded access and and pipelining requests to improve throughput. That should make it fairly easy to integrate into your Django application.
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-50018645618160455632014-05-12T00:12:00.000+01:002014-05-12T00:12:40.525+01:00A Dictionary-like Python interface for OData Part III: a SQL-backed OData Server
This is the third and last part of a series of three posts that introduce my OData framework for Python. To recap:<?p>

In Part I I introduced a new data access layer I've written for Python that is modelled on the conventions of OData. In that post I validated the API by writing a concrete implementation in the form of an OData client.

In Part II I used the same API and wrote a concrete implementation using a simple in-memory storage model. I also introduced the OData server functionality to expose the API via the OData protocol.

In this part, I conclude this mini-series with a quick look at another concrete implementation of the API which wraps Python's DB API allowing you to store data in a SQL environment.

As before, you can download the source code from the QTIMigration Tool & Pyslet home page. I wrote a brief tutorial on using the SQL backed classes to take care of some of the technical details.

Rain or Shine?

To make this project a little more interesting I went looking for a real data set to play with. I'm a bit of a weather watcher at home and for almost 20 years I've enjoyed using a local weather station run by a research group at the University of Cambridge. The group is currently part of the Cambridge Computer Laboratory and the station has moved to the William Gates building.

The Database

The SQL implementation comes in two halves. The base classes are as close to standard SQL as I could get and then a small 'shim' sits over the top which binds to a specific database implementation. The Python DB API takes you most of the way, including helping out with the correct form of parameterisation to use. For this example project I used SQLite because the driver is typically available in Python implementations straight out of the box.

I wrote the OData-style metadata document first and used it to automatically generate the CREATE TABLE commands but in most cases you'll probably have an existing database or want to edit the generated scripts and run them by hand. The main table in my schema got created from this SQL:

CREATE TABLE "DataPoints" (
 "TimePoint" TIMESTAMP NOT NULL,
 "Temperature" REAL,
 "Humidity" SMALLINT,
 "DewPoint" REAL,
 "Pressure" SMALLINT,
 "WindSpeed" REAL,
 "WindDirection" TEXT,
 "WindSpeedMax" REAL,
 "SunRainStart" REAL,
 "Sun" REAL,
 "Rain" REAL,
 "DataPointNotes_ID" INTEGER,
 PRIMARY KEY ("TimePoint"),
 CONSTRAINT "DataPointNotes" FOREIGN KEY ("DataPointNotes_ID") REFERENCES "Notes"("ID"))

To expose the database via my new data-access-layer API you just load the XML metadata, create a SQL container object containing the concrete implementation and then you can access the data in exactly the same way as I did in Part's I and II. The code that consumes the API doesn't need to know if the data source is an OData client, an in memory dummy source or a full-blown SQL database. Once I'd loaded the data, here is a simple session with the Python interpreter that shows you the API in action.

>>> import pyslet.odata2.metadata as edmx
>>> import pyslet.odata2.core as core
>>> doc=edmx.Document()
>>> with open('WeatherSchema.xml','rb') as f: doc.Read(f)
...
>>> from pyslet.odata2.sqlds import SQLiteEntityContainer
>>> container=SQLiteEntityContainer(filePath='weather.db',containerDef=doc.root.DataServices['WeatherSchema.CambridgeWeather'])
>>> weatherData=doc.root.DataServices['WeatherSchema.CambridgeWeather.DataPoints']
>>> collection=weatherData.OpenCollection()
>>> collection.OrderBy(core.CommonExpression.OrderByFromString('WindSpeedMax desc'))
>>> collection.SetPage(5)
>>> for e in collection.iterpage(): print "%s: Max wind speed: %0.1f mph"%(unicode(e['TimePoint'].value),e['WindSpeedMax'].value*1.15078)
...
2002-10-27T10:30:00: Max wind speed: 85.2 mph
2004-03-20T15:30:00: Max wind speed: 82.9 mph
2007-01-18T14:30:00: Max wind speed: 80.6 mph
2004-03-20T16:00:00: Max wind speed: 78.3 mph
2005-01-08T06:00:00: Max wind speed: 78.3 mph

Notice that the container itself isn't needed when accessing the data because the SQLiteEntityContainer __init__ method takes care of binding the appropriate collection classes to the model passed in. Unfortunately the dataset doesn't go all the way back to the great storm of 1987 which is a shame as at the time I was living in a 5th floor flat perched on top of what I was reliably informed was the highest building in Cambridge not to have some form of structural support. I woke up when the building shook so much my bed moved across the floor.

Setting up a Server

I used the same technique as I did in Part II to wrap the API with an OData server and then had some real fun getting it up and running on Amazon's EC2. Pyslet requires Python 2.7 but EC2 Linux comes with Python 2.6 out of the box. Thanks to this blog article for help with getting Python 2.7 installed. I also had to build mod_wsgi from scratch in order to get it to pick up the version I wanted. Essentially here's what I did:

# Python 2.7 install
sudo yum install make automake gcc gcc-c++ kernel-devel git-core -y
sudo yum install python27-devel -y
# Apache install
# Thanks to http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-LAMP.html
sudo yum groupinstall -y "Web Server"
sudo service httpd start
sudo chkconfig httpd on

And to get mod_wsgi working with Python2.7...

sudo bash
cd
yum install httpd-devel -y
mkdir downloads
cd downloads
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.4.tar.gz
tar -xzvf mod_wsgi-3.4.tar.gz
cd mod_wsgi-3.4
./configure --with-python=/usr/bin/python2.7
make
make install
# Optional check to ensure that we've got the correct Python linked
# you should see the 2.7 library linked
ldd /etc/httpd/modules/mod_wsgi.so
service httpd restart

To drive the server with mod_wsgi I used a script like this:

#! /usr/bin/env python

import logging, os.path
import pyslet.odata2.metadata as edmx
from pyslet.odata2.sqlds import SQLiteEntityContainer
from pyslet.odata2.server import ReadOnlyServer

HOME_DIR=os.path.split(os.path.abspath(__file__))[0]
SERVICE_ROOT="http://odata.pyslet.org/weather"

logging.basicConfig(filename='/var/www/wsgi-log/python.log',level=logging.INFO)

doc=edmx.Document()
with open(os.path.join(HOME_DIR,'WeatherSchema.xml'),'rb') as f:
 doc.Read(f)

container=SQLiteEntityContainer(filePath=os.path.join(HOME_DIR,'weather.db'),
 containerDef=doc.root.DataServices['WeatherSchema.CambridgeWeather'])

server=ReadOnlyServer(serviceRoot=SERVICE_ROOT)
server.SetModel(doc)

def application(environ, start_response):
 return server(environ,start_response)

I'm relying on the fact that Apache is configured to run Python internally and that my server object persists between calls. I think by default mod_wsgi serialises calls to the application method but a smarter configuration with a multi-threaded daemon would be OK because the server and container objects are thread safe. There are limits to the underlying SQLite module of course so you may not gain a lot of performance this way but a proper database would help.

Try it out!

If you were watching carefully you'll see that the above script uses a public service root. So let's try the same query but this time using OData. Here it is in Firefox:

Notice that Firefox recognises that the OData feed is an Atom feed and displays the syndication title and updated date. I used the metadata document to map the temperature and the date of the observation to these (you can see they are the same data points as above by the matching dates). The windiest days are never particularly hot or cold in Cambridge because they are almost always associated with Atlantic storms and the sea temperature just doesn't change that much.

The server is hosted at http://odata.pyslet.org/weather
Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-24119399253310869372014-02-24T21:43:00.003+00:002014-02-24T21:43:37.099+00:00A Dictionary-like Python interface for OData Part II: a Memory-backed OData Server
In my previous post, A Dictionary-like Python interface for OData I introduced a new sub-package I've added to Pyslet to implement support for OData version 2. You can download the latest version of the Pyslet package from the QTI Migration Tool & Pyslet home page.

To recap, I've decided to set about writing my own data access layer for Python that is modelled on the conventions of OData. I've validated the API by writing a concrete implementation in the form of an OData client. In this post I'll introduce the next step in the process which is a simple alternative implementation that uses a different underlying storage model, in other words, an implementation which uses something other than a remote OData server. I'll then expose this implementation as an OData server to validate that my data access layer API works from both perspectives.

Metadata

Unlike other frameworks for implementing OData services Pyslet starts with the metadata model, it is not automatically generated from your code, you must write it yourself. This differs from the object-first approach taken by other frameworks, illustrated here:

This picture is typical of a project using something like Microsoft's WCF. Essentially, there's a two-step process. You use something like Microsoft's entity framework to generate classes from a database schema, customise the classes a little and then the metadata model is auto-generated from your code model. Of course, you can go straight to code and implement your own code model that implements the appropriate queryable interface but this would typically be done for a specific model.

Contrast this with the approach taken by Pyslet where the entities are not model-specific classes. For example, when modelling the Northwind service there is no Python class called Product as there would be in the approach taken by other frameworks. Instead there is a generalised implementation of Entity which behaves like a dictionary. The main difference is probably that you'll use supplier['Phone'] instead of simply supplier.phone or, if you'd have gone down the getter/setter route, supplier.GetPhone(). In my opinion, this works better than a tighter binding for a number of reasons, but particularly because it makes the user more mindful of when data access is happening and when it isn't.

Using a looser binding also helps prevent the type of problems I had during the development of the QTI specification. Lots of people were using Java and JAXB to autogenerate classes from the XML specification (cf autogenerating classes from a database schema) but the QTI model contained a class attribute on most elements to allow for stylesheet support. This class attribute prevented auto-generation because class is a reserved word in the Java language. Trying to fix this up after auto-generation would be madness but fixing it up before turns out to be a little tricky and this glitch seriously damaged the specification's user-experience. We got over it, but I'm wary now and when modelling OData I stepped back from a tighter binding, in part, to prevent hard to fix glitches like the use of Python reserved words as property names.

Allocating Storage

For this blog post I'm using a lightweight in-memory data storage implementation which can be automatically provisioned from the metadata document and I'm going to cheat by making a copy of the metadata document used by the Northwind service. Exposing OData the Pyslet way is a little more work if you already have a SQL database containing your data because I don't have a tool that auto-generates the metadata document from the SQL database schema. Automating the other direction is easy, but more on that in Part III.

I used my web browser to grab a copy of http://services.odata.org/V2/Northwind/Northwind.svc/$metadata and saved it to a file called Northwind.xml. I can then load the model from the interpreter:

>>> import pyslet.odata2.metadata as edmx
>>> doc=edmx.Document()
>>> f=open('Northwind.xml')
>>> doc.Read(f)
>>> f.close()

This special Document class ensures that the model is loaded with the special Pyslet element implementations. The Products entity set can be looked up directly but at the moment it's empty!

>>> productSet=doc.root.DataServices['ODataWeb.Northwind.Model.NorthwindEntities.Products']
>>> products=productSet.OpenCollection()
>>> len(products)
0
>>> products.close()

This isn't surprising, there is nothing in the metadata model itself which binds it to the data service at services.odata.org. The model isn't linked to any actual storage for the data. By default, the model behaves as if it is bound to an empty read-only data store.

To help me validate that my API can be used for something other than talking to real OData services I've created an object that provisions storage for an EntityContainer (that's like a database in OData) using standard Python dictionaries. By passing the definition of an EntityContainer to the object's constructor I create a binding between the model and this new data store.

>>> from pyslet.odata2.memds import InMemoryEntityContainer
>>> container=InMemoryEntityContainer(doc.root.DataServices['ODataWeb.Northwind.Model.NorthwindEntities'])
>>> products=productSet.OpenCollection()
>>> len(products)
0

The collection of products is still empty but it is now writeable. I'm going to cheat again to illustrate this by borrowing some code from the previous blog post to open an OData client connected to the real Northwind service.

>>> from pyslet.odata2.client import Client
>>> c=Client("http://services.odata.org/V2/Northwind/Northwind.svc/")
>>> nwProducts=c.feeds['Products'].OpenCollection()

Here's a simple loop to copy the products from the real service into my own collection. It's a bit clumsy in the interpreter but careful typing pays off:

>>> for nwProduct in nwProducts.itervalues():
... product=collection.CopyEntity(nwProduct)
... product.SetKey(nwProduct.Key())
... collection.InsertEntity(product)
...
>>> len(collection)
77

To emphasise the difference between my in-memory collection and the live OData service I'll add another record to my copy of this entity set. Fortunately most of the fields are marked as Nullable in the model so to save my fingers I'll just set those that aren't.

>>> product=collection.NewEntity()
>>> product.SetKey(100)
>>> product['ProductName'].SetFromValue("The one and only Pyslet")
>>> product['Discontinued'].SetFromValue(False)
>>> collection.InsertEntity(product)
>>> len(collection)
78

Now I can do everything I can with the OData client using my copy of the service, I'll filter the entities to make it easier to see:

>>> import pyslet.odata2.core as core
>>> filter=core.CommonExpression.FromString("substringof('one',ProductName)")
>>> collection.Filter(filter)
>>> for p in collection.itervalues(): print p.Key(), p['ProductName'].value
...
21 Sir Rodney's Scones
32 Mascarpone Fabioli
100 The one and only Pyslet

I can access my own data store using the same API that I used to access a remote OData service in the previous post. In that post, I also claimed that it was easy to wrap my own implementations of this API to expose it as an OData service.

Exposing an OData Server

My OData server class implements the wsgi protocol so it is easy to link it up to a simple http server and tell it to handle a single request.

>>> from pyslet.odata2.server import Server
>>> server=Server("http://localhost:8081/")
>>> server.SetModel(doc)
>>> from wsgiref.simple_server import make_server
>>> httpServer=make_server('',8081,server)
>>> httpServer.handle_request()

My interpreter session is hanging at this point waiting for a single HTTP connection. The Northwind service doesn't have any feed customisations on the Products feed and, as we slavishly copied it, the Atom-view in the browser is a bit boring so I used the excellent JSONView plugin for Firefox and the following URL to hit my service:

http://localhost:8081/Products?$filter=substringof('one',ProductName)&$orderby=ProductID desc&$format=json

This is the same filter as I used in the interpreter before but I've added an ordering and specified my preference for JSON format. Here's the result.

As I did this, Python's simple server object logged the following output to my console:

127.0.0.1 - - [24/Feb/2014 11:17:05] "GET /Products?$filter=substringof(%27one%27,ProductName)&$orderby=ProductID%20desc&$format=json HTTP/1.1" 200 1701
>>>

The in-memory data store is a bit of a toy, though some more useful applications might be possible. In the OData documentation I go through a tutorial on how to create a lightweight memory-cache of key-value pairs exposed as an OData service. I'm not really suggestion using it in a production environment to replace memcached. What this implementation is really useful for is developing and testing applications that consume the DAL API without needing to be connected to the real data source. Also, it can be wrapped in the OData Server class as shown above and used to provide a more realistic mock of an actual service for testing that your consumer application still works when the data service is remote. I've used it in Pyslet's unit-tests this way.

In the third and final part of this Python and OData series I'll cover a more interesting implementation of the API using the SQLite database.
Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-52753483905925472942014-02-12T22:19:00.000+00:002014-02-12T22:19:25.429+00:00A Dictionary-like Python interface for OData
Overview

This blog post introduces some new modules that I've added to the Pyslet package I wrote. Pyslet's purpose is providing support for Standards for Learning, Education and Training in Python. The new modules implement the OData protocol by providing a dictionary-like interface. You can download pyslet from the QTIMigration Tool & Pyslet home page. There is some documentation linked from the main Pyslet wiki. This blog article is as good a way as any to get you started.

The Problem

Python has a database API which does a good job but it is not the whole solution for data access. Embedding SQL statements in code, grappling with the complexities of parameterization and dealing with individual database quirks makes it useful to have some type of layer between your web app and the database API so that you can tweak your code as you move between data sources.

If SQL has failed to be a really interoperable standard then perhaps OData, the new kid on the block, can fill the vacuum. The standard is sometimes referred to as "ODBC over the web" so it is definitely in this space (after all, who runs their database on the same server as their web app these days?).

My Solution

To solve this problem I decided to set about writing my own data access layer that would be modeled on the conventions of OData but that used some simple concepts in Python. I decided to go down the dictionary-like route, rather than simulating objects with attributes, because I find the code more transparent that way. Implementing methods like __getitem__, __setitem__ and itervalues keeps the data layer abstraction at arms length from the basic python machinery. It is a matter of taste. See what you think.

The vision here is to write a single API (represented by a set of base classes) that can be implemented in different ways to access different data sources. There are three steps:

An implementation that uses the OData protocol to talk to a remote OData service.

An implementation that uses python dictionaries to create a transient in-memory data service for testing.

An implementation that uses the python database API to access a real database.

This blog post is mainly about the first step, which should validate the API as being OData-like and set the groundwork for the others which I'll describe in subsequent blog posts. Incidentally, it turns out to be fairly easy to write an OData server that exposes a data service written to this API, more on that in future posts.

Quick Tutorial

The client implementation uses Python's logging module to provide logging. To make it easier to see what is going on during this walk through I'm going to turn logging up from the default "WARN" to "INFO":

>>> import logging
>>> logging.basicConfig(level=logging.INFO)

To create a new OData client you simply instantiate a Client object passing the URL of the OData service root. Notice that, during construction, the Client object downloads the list of feeds followed by the metadata document. The metadata document is used extensively by this module and is loaded into a DOM-like representation.

>>> from pyslet.odata2.client import Client
>>> c=Client("http://services.odata.org/V2/Northwind/Northwind.svc/")
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/ HTTP/1.1
INFO:root:Finished Response, status 200
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/$metadata HTTP/1.1
INFO:root:Finished Response, status 200

Client objects have a feeds attribute that is a plain dictionary mapping the exposed feeds (by name) onto EntitySet objects. These objects are part of the metadata model but serve a special purpose in the API as they can be opened (a bit like files or directories) to gain access to the (collections of) entities themselves. Collection objects can be used in the with statement and that's normally how you'd use them but I'm sticking with the interactive terminal for now.

>>> products=c.feeds['Products'].OpenCollection()
>>> for p in products: print p
...
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/Products HTTP/1.1
INFO:root:Finished Response, status 200
1
2
3
... [and so on]
...
20
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/Products?$skiptoken=20 HTTP/1.1
INFO:root:Finished Response, status 200
21
22
23
... [and so on]
...
76
77

The products collection behaves like a dictionary, iterating through it iterates through the keys in the dictionary. In this case these are the keys of the entities in the collection of products in Microsoft's sample Northwind data service. Notice that the client logs several requests to the server interspersed with the printed output. That's because the server is limiting the maximum page size and the client is following the page links provided. These calls are made as you iterate through the collection allowing you to iterate through very large collections without loading everything in to memory.

The keys alone are of limited interest, let's try a similar loop but this time we'll print the product names as well:

>>> for k,p in products.iteritems(): print k,p['ProductName'].value
...
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/Products HTTP/1.1
INFO:root:Finished Response, status 200
1 Chai
2 Chang
3 Aniseed Syrup
...
...
20 Sir Rodney's Marmalade
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/Products?$skiptoken=20 HTTP/1.1
INFO:root:Finished Response, status 200
21 Sir Rodney's Scones
22 Gustaf's Knäckebröd
23 Tunnbröd
...
...
76 Lakkalikööri
77 Original Frankfurter grüne Soße

Sir Rodney's Scones sound interesting, we can grab an individual record just as we normally would from a dictionary, by using its key.

>>> scones=products[21]
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/Products(21) HTTP/1.1
INFO:root:Finished Response, status 200
>>> for k,v in scones.DataItems(): print k,v.value
...
ProductID 21
ProductName Sir Rodney's Scones
SupplierID 8
CategoryID 3
QuantityPerUnit 24 pkgs. x 4 pieces
UnitPrice 10.0000
UnitsInStock 3
UnitsOnOrder 40
ReorderLevel 5
Discontinued False

The scones object is an Entity object. It too behaves like a dictionary. The keys are the property names and the values are one of SimpleValue, Complex or DeferredValue. In the snippet above I've used a variation of iteritems which iterates only through the data properties, excluding the navigation properties. In this model, there are no complex properties. The simple values have a value attribute which contains a python representation of the value.

Deferred values (navigation properties) can be used to navigate between Entities. Although deferred values can be opened just like EntitySets, if the model dictates that at most 1 entity can be linked a convenience method called GetEntity can be used to open the collection and read the entity in one call. In this case, a product can have at most one supplier.

>>> supplier=scones['Supplier'].GetEntity()
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/Products(21)/Supplier HTTP/1.1
INFO:root:Finished Response, status 200
>>> for k,v in supplier.DataItems(): print k,v.value
...
SupplierID 8
CompanyName Specialty Biscuits, Ltd.
ContactName Peter Wilson
ContactTitle Sales Representative
Address 29 King's Way
City Manchester
Region None
PostalCode M14 GSD
Country UK
Phone (161) 555-4448
Fax None
HomePage None

Continuing with the dictionary-like theme, attempting to load a non existent entity results in a KeyError:

>>> p=products[211]
INFO:root:Sending request to services.odata.org
INFO:root:GET /V2/Northwind/Northwind.svc/Products(211) HTTP/1.1
INFO:root:Finished Response, status 404
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/Library/Python/2.7/site-packages/pyslet/odata2/client.py", line 165, in __getitem__
 raise KeyError(key)
KeyError: 211

Finally, when we're done, it is a good idea to close the open collection. If we'd used the with statement this step would have been done automatically for us of course.

>>> products.close()

Limitations

Currently the client only supports OData version 2. Version 3 has now been published and I do intend to update the classes to speak version 3 at some point. If you try and connect to a version 3 service the client will complain when it tries to load the metadata document. There are ways around this limitation, if you are interested add a comment to this post and I'll add some documentation.

The client only speaks XML so if your service only speaks JSON it won't work at the moment. Most of the JSON code is done and tested so adding it shouldn't be a big issue if you are interested.

The client can be used to both read and write to a service, and there are even ways of passing basic authentication credentials. However, if calling an https URL it doesn't do certificate validation at the moment so be warned as your security could be compromised. Python 2.7 does now support certification validation using OpenSLL so this could change quite easily I think.

Moving to Python 3 is non-trivial - let me know if you are interested. I have taken the first steps (running unit tests with "python -3Wd" to force warnings) and, as much as possible, the code is ready for migration. I haven't tried it yet though and I know that some of the older code (we're talking 10-15 years here) is a bit sensitive to the raw/unicode string distinction.

The documentation is currently about 80% accurate and only about 50% useful. Trending upwards though.

Downloading and Installing Pyslet

Pyslet is pure-python. If you are only interested in OData you don't need any other modules, just Python 2.7 and a reasonable setuptools to help you install it. I just upgraded my machine to Mavericks which effectively reset my Python environment. Here's what I did to get Pyslet running.

Installed setuptools

Downloaded the pyslet package tgz and unpacked it (download from here)

Ran python setup.py install

Why?

Some lessons are hard! Ten years or so ago I wrote a migration tool to convert QTI version 1 to QTI version 2 format. I wrote it as a Python script and used it to validate the work the project team were doing on the version 2 specification itself. Realising that most people holding QTI content weren't able to easily run a Python script (especially on Windows PCs) my co-chair Pierre Gorissen wrote a small Windows-wrapper for the script using the excellent wxPython and published an installer via his website. From then on, everyone referred to it as "Pierre's migration tool". I'm not bitter, the lesson was clear. No point in writing the tool if you don't package it up in the way people want to use it.

This sentiment brings me to the latest developments with the tool. A few years back I wrote (and blogged about) a module for writing Basic LTI tools in Python. I did this partly to prove that LTI really was simple (I wrote the entire module on a single flight to the US) but also because I believed that the LTI specification was really on to something useful. LTI has been a huge success and offers a quick route for tool developers to gain access to users of learning management systems. It seems obvious that the next version of the QTI Migration Tool should be an LTI tool but moving from a desktop app to a server-based web-app means that I need a data access layer that can persist data and be smarter about things like multiple threads and processes.
Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-75463097280485802172014-02-05T13:47:00.000+00:002014-02-05T13:47:12.869+00:00Deleting from iCalendar Without Notifying the Organizer - At Last!Being a bit of a laggard I only just upgraded my Mac to Mavericks a few days ago. If only I'd known they had fixed the number one most annoying thing about iCalendar in OS X I'd have upgraded ages ago.

Yes, you can now delete an event from your calendar without notifying the organizer. This has caused me serious pain in the past. Events sometimes arrive to the wrong email address or get accidentally put in the wrong calendar and previously you had to just leave them there for fear of sending a stupid "Steve declined your event" type email.

Thank you!Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-13254796593999716542013-09-01T14:22:00.000+01:002013-09-02T09:23:52.646+01:00Transforming QTI v2 into (X)HTML 5
At a recent IMS meeting I again mentioned that transforming QTI v2 into HTML shouldn't be too difficult and that I'd already made a start on this project many years ago. I even mentioned it in an earlier blog post: Semantic Markup in HTML. To my shame, I got a comment on that post which called my bluff and I haven't posted my work - until now! I won't bore you with the excuses, day job, etc. I should also warn you that these files are in no way complete. However, they do solve most of the hard problems in my opinion and could be built out to cover the rest of the interaction types fairly easily.

If you want to sing along with this blog post you should look at the files in the following directory of the QTI migration source repository: https://code.google.com/p/qtimigration/source/browse/#svn%2Ftrunk%2Fqti2html. In there you'll find a collection of XSL files.

qti2html.xsl

The goal of this project was to see how easy it would be to transform QTI version 2 files into HTML 5 in such a way that the HTML 5 was an alternative representation of the complete QTI information model. The goal was not to create a rendering which would work in an assessment delivery engine but to create a rendering that would store all the information about an item and render it in a sensible way, perhaps in a way suitable for a reviewer to view. I was partly inspired by a comment from Dick Bacon, of SToMP fame. He said it would be nice to see everything from the question all in one go, including feedback that is initially hidden and so on. It sort of gave me the idea to do the XSL this way.

Let's see what this stylesheet does to the most basic QTI v2 example, here's the command I ran on my Mac:

xsltproc qti2html.xsl choice.xml > choice.xhtml

And here's how the resulting file looks in Firefox:

The first thing you'll notice is that there is no HTML form in sight. You can't interact with this page, it is static text. But remember the goal of this stylesheet is to represent the QTI information completely. Let's look at the generated HTML source:

 <title>Unattended Luggage</title>
 <meta name="qti.identifier" content="choice"/>
 <meta name="qti.adaptive" content="false"/>
 <meta name="qti.timeDependent" content="false"/>

To start with, the head element contains some useful meta tags with names starting "qti.", this allows us to capture basic information that would normally be in the root element of the item.

Unattended Luggage

Look at the text in the picture.

What does it say?

You
 must stay with your luggage at all times.

Do not let someone else
 look after your luggage.

Remember your luggage
 when you leave.

I've abbreviated the body here but you'll see that the item body maps in to a div with a appropriate class name (this time prefixed with qti- to make styling easier). The HTML copies across pretty much unchanged but the interesting part is the div with a class of "qti-choiceInteraction". Here we've mapped the choiceInteraction in the original XML into a div and used HTML5 style data attributes to add information about the behaviour. Cardinality, etc. In essence, this div performs the role of both interaction and response variable declaration.

I chose to map the choices themselves on to an unordered list in HTML, again, using HTML5 data- attributes to provide the additional information required by QTI.

html2qti.xsl

So can this format be converted back to QTI v2? Yes it can. The purpose of this stylesheet is to take the XHTML5 representation created by the previous stylesheet and turn it back in to valid QTIv2. This gives us the power to work in either representation. If we want to edit the HTML we can!

xsltproc html2qti.xsl choice.xhmtl > choice-new.xml

The output file is not identical to the input file, there are some changes to comments, white space and the order in which attributes are expressed but it is valid QTI v2 and is the same in every important respect.

qti.xsl

Once we have an XHTML5 representation of the item it seems like it should be easy to make something more interactive. A pure HTML/JS delivery engine for QTIv2 is an attractive proposition, especially if you can use an XSL transform to get there from the original QTI file. One of the main objections to QTI I used to hear from the SCORM crowd back in the early days was that there was no player for QTI files that were put in SCORM packages. But given that most modern browsers can do XSL this objection could just disappear, at least for formative quizzes where you don't mind handing out the scoring rules embedded in the HTML. You could even tie up such an engine with calls to the SCORM API to report back scores.

Given this background, the qti.xsl file takes a first step to transforming the XHTML5 file produced with the first XSL into something that is more like an interactive question.

xsltproc qti.xsl choice.xhtml > choice_tryout.xhtml

This is what the output looks like in Firefox:

This page is interactive, if you change your choice and click OK it re-scores the question. Note that I currently have a score of 1 as I have the correct choice selected.

Future Work

These XSL files will handle some of the other variants of multi-choice, including multi-response (yes, even when interacting) but are in no way a complete method of using XSL to transform QTI. For me, the only compelling argument for using XSL is if you want to embed it in a package to force the client to do the transformation or if you are using one of those cocoon like tools that uses XSL pipelines at runtime. Realistically, writing XSL is a pig and so much time has elapsed since I wrote it that it would take an hour or two to familiarise myself with it again if I wanted to make changes.

But the key idea is the transforms enabled by the first two, which present an alternative binding for the QTI information model.
Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-2049020507407065622013-08-29T08:17:00.001+01:002013-08-29T08:17:35.042+01:00In historic vote, New Zealand bans software patents | Ars Technica
In historic vote, New Zealand bans software patents | Ars Technica

In my personal opinion, this is great news. I have had to sift through patent applications from software vendors that have clearly been created by simply sending a bunch of interface files to a lawyer to be translated in to patentese. You know the sort of thing, an API call like Student.GetName(id) becomes a 'claim' in which a string representation of a student's name is obtained from a system with a stored representation of a student's registration information, etc, etc. If we carry on as we are, someone will have to write an Eclipse plug-in that generates the patent application every time you build your software.

So this news is a ray of hope. It isn't a blanket "IP law is bad" bill but a measured way of enshrining the basic principle that software 'as such' is not an invention. There will always be the odd counterexample where something is no longer patentable that we might feel should be (and vice versa!) but ever since I've been following this debate it seems to me that the system has stayed the same not for lack of suggestions of how to make it better but because of FUD around change. Thank you NZ for being bolder.

Initially 'via Blog this'
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-41014527013354545962013-08-25T15:46:00.000+01:002013-08-25T15:49:14.335+01:00What's in a name? Tin Can, Experience and the Tea Party
On Friday I was at e-Assessment Scotland, getting ready for Fiona Leteney's closing keynote "Anyone need a Can Opener?" when I tweeted:

Getting ready for the #easc13 session on the Experience API adlnet.gov/tla/experience… - anyone got an experience opener?

I tweeted this because I wanted to ensure that the twitter-savvy audience at #easc13 understood that there is an issue with the name of the API and that, in my opinion, it is important. Fiona followed up on twitter and, I tried to put this in to 140 characters but it just wouldn't fit. Before I go on though, a couple of disclaimers...

This blog post represents my personal views, not the views of my employer or of any other organization with which I may be affiliated. In the interests of full disclosure, I work for Questionmark who have presented work related to this API at previous conferencess, including at DevLearn. Also, I'm not a lawyer! I am interested in the world of open source and in some of the legal issues that new technology throws up, particularly when they relate to intellectual property law.

So What's in a name?

The title of @fionaleteney's talk clearly makes reference to the "Tin Can API" but she did touch on the naming issue in the presentation and introduced the newer "Experience API" name with the observation that "Tin Can" is likely to remain the recognisable brand.

Many people in the community probably think the name doesn't matter and that arguing over a name is an unimportant distraction. Indeed, it all feels a bit like the Monty Python film, The Life of Brian, in which followers argue over whether the shoe or the gourd is the correct symbol to represent their community. I believe it is important, in short because I don't think an API like this can succeed if it gets this type of thing wrong. The argument goes like this:

The purpose of the API is to make it easier for activity statements to flow between the tools that generate or initially record activities and tools that aggregate this information.

Therefore, the API is essentially an interoperability specification that will be more successful if a wide range of tools adopt it.

To get a wide range of tools you need a wide range of tool suppliers to invest in adding support.

To get a wide range of suppliers to invest you need a level playing field and trust within the supplier community

To get trust, you need good stewardship of the API.

I've honed in on one particular prerequisite for success here. There are plenty of other challenges that an API like this faces, including other issues related to IP law. After the session one delegate expressed concern about the ownership of information communicated using the API, and I have heard privacy issues voiced too. These are important things to get right, even more reason not to distract the suppliers' legal teams with branding issues when they should be advising their clients on how to use the API to improve the learner's experience from the right side of the law.

You'll have had your tea

One aspect of good stewardship for an API is the branding. In practice, that means control of the trademarks associated with an API. To people in the technology world this often translates directly into domain names but that is only one way in which trademarks are used. What about Google Ad words? Google has a policy for that. And don't forget logos.

Of course, the big guns of the IT world get this type of thing sorted out before they even embark on a joint project but legal issues are often the last thing a small learning technology community thinks about. We're all well meaning, what could possibly go wrong? Why doesn't everyone trust me? What do you mean, I'm not allowed to team up with a small group of my competitors to gain an advantage over the rest of the marketplace? And so on... I have to thank Ed Walker for drawing my attention to the importance of getting this type of thing right when he was in his role as CEO of the IMS Global Learning consortium. He was particularly clear on the last point.

Thinking of Ed brings me (via Boston) to an interesting article about a similar problem in a very different domain. In Trademarking the Tea Party, an article from 2010, Beth Hutchens touches on an issue which resonates with the problem experienced by the Experience API community. In this case, during a dispute over the identity of a political movement one group seeks trademark protection for the Tea Party name and all the other groups cry foul!

The problem is that the Tea Party is more of a conglomerate of different groups of people based loosely around a set of common goals, and not a collective. This could be problematic in gaining trademark protection.

This could well describe the sort of informal grouping that tends to build up around an API. The author goes on to say...

It makes sense to have at least some sort of structure to keep squabbles like this from coming up in the future.

The suggested solution is set yourselves up as a collective movement with a distinctive name so that you can register a collective mark. As with anything law related, it makes sense to put a little work in up-front to ensure you don't have to spend a lot of time later figuring out who (if anyone) has the right to say how the name can and can't be used. The rest of the article is worth a read by the way, if only for its use of the word antidisestablishmentarianism in a justifiable context. But more seriously, it is a great introduction for normal people on what a collective trademark is.

Kicking the Can

To understand why the naming thing has become an issue for this API it is best to read the discussion We Call It Tan Can. There are two issues being debated here so it is a bit confusing.

Which is the better name: Experience API or Tin Can API? - this blog post isn't about this question, for what it's worth I voted for Experience API but I could live with Tin Can if we got the second issue straight.

What is the appropriate legal way for the community to manage and control use of the API's name?

It is clear that the community recognises the problem. The team that originally led the development of the API registered a trademark with a view to handing it over to the project sponsor, the ADL. ADL is part of the US government and it seems like handing a trademark to a government department has turned out to be trickier than expected. The US Government does have a succinct page on the issue of government owned IP though: Copyright and Other Rights Pertaining to U.S. Government Works. On this page it makes it clear that you can't use a government owned trademark without permission (no big surprise there) and clears up the confusion of which bits of government IP are in the public domain and which aren't. (An issue which has fascinating implications for the makers of model aircraft, but we digress.)

In the above thread, Mike Rustici goes on to say:

Since we're on the topic of trademarks, another significant issue to consider in this debate is the fact that "experience api" is not trademarked. If ADL is unable or unwilling to secure one, that is very problematic for the future of this spec.
...
Anybody could claim to support or own "experience api" rendering the spec (or at least the label) meaningless.

So who should own the mark? The ADL seems to be struggling to take ownership and, even it did, how would it determine the rules under which permission should be granted to members of the API community? If one member abused the mark, would the US government pursue them on behalf of the other members? It isn't clear that the ADL has the desire to fill this role on their behalf.

Let's take the very particular issue of domain names, though Google Adwords do raise similar concerns. Policies like ICANN's Unified Domain Name Dispute Resolution Policy make it clear that trademarks are an important part of determining whether complaints will be upheld. So if the API's name was owned by a neutral party would that group invoke the policy to ensure that names like experienceapi.com or tincanapi.com were used in a way that ensures there will be no confusion? You'd hope so. Right now, both tincanapi.com and experienceapi.com point to basically the same site controlled by just one member of the community and are used to promote their services over and above those of other community members. As far as this blog post is concerned, both names are problematic now.

With the benefit of Experience

It's no surprise that people have had this type of problem before and that there are legal patterns that a community can follow to help ensure that their IP, including any trademarks, have the sort of stewardship which helps attract members and build the community. Creating a new membership organization just for this API would seem onerous but the problem with choosing an existing one is that they'll have already established a way of dealing with the IP and it is unlikely to be a perfect match. Still, this seems to be the solution being explored by ADL, quoting again from the main thread: <q>Plans are already being made to transfer ownership of the spec to an open standards body</q> - this can't come too soon.

One final word of caution here. One of the grim duties that falls on the owner of a trademark is to ensure that it is being upheld and is not just becoming a generic term for a bunch of similar products from a variety of suppliers (hoover, biro etc). I recall the magazine Private Eye getting letters from lawyers when they used the word Portakabin in one of their articles. If confusion takes over around this API then none of the marks will be enforceable and we'll have to start all over again.
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-43726302357858548422013-03-15T21:00:00.000+00:002013-03-15T21:00:02.248+00:00RSS Readers: in the dog house
So farewell Google Reader, I will miss you.

This week's announcement of the demise of Google Reader as part of the Second Spring of Cleaning seems to be an important milestone for the internet.

There's a lot of new blog articles lamenting its demise (to some extent, this is one of them) but we shouldn't be too shocked. The original concept behind RSS has been under threat for some time, in fact if you Google "War on RSS" you'll see an established idea that companies that have a powerful influence on the way we use the internet have been deprecating RSS for some time.

Perhaps the most interesting of these contribution comes from @vambenepe who wrote The war on RSS in February last year. It's a good overview of the way RSS reading features are going missing in systems we use to access the internet and contains this worrying quote:

Google has done a lot for RSS, but as a result it has put itself in position to kill it, either accidentally or on purpose. [...snip...] [... If] Google closed Reader, would RSS survive? Doubtful.

This particular commentator is interesting because since writing this he has moved on to become "Product Manager on Google Cloud Platform". Don't expect a follow up article but he did tweet yesterday:

"1 year ago, I asked: "If Google closed Reader, would RSS survive?" http://stage.vambenepe.com/archives/1932 We'll now find out but I won't be able to comment."

One of the takeaways here is that we're not just talking about RSS specifically. When we say RSS we can include Atom and readers of this blog will know that I'm a fan of Atom and the emerging OData standard that is based upon it. But let's not get carried away. This war is not on the protocol but on the use of RSS as a way of end users discovering content on the internet. The emergence of OData (based on the Atom Publishing Protocol, not the read-only RSS) as a protocol that sits between the web app and the data source is likely to get even stronger.

Even HTTP has changed. This blog post uses HTTP in an old fashioned way. I'm writing an article, inserting anchors that form hypertext links to other resources on the internet. I'm banking on the idea that these resources won't go away and that this article will join a persistant web of information. If you're reading this you're probably thinking, duh, that's what the internet is. In the early days this was true but the internet is no longer like this for the majority of users. HTTP sits as a protocol behind the web apps we use to check Twitter, Facebook and iTunes but the concept behind the way most people consume information on the internet bears no relation to the classic hypertext visions we used to cite when we were all researchers working in universities in the early 90s.

Go back and read the seminal As we may think or review the goals of Ted Nelson's Xanadu Project and you won't recognise the origins of iTunes, on-demand TV, micro-blogging or ad-supported social networks. From a UK point of view, we didn't even have commercial broadcast television until 1955 (when ITV was launched) which is 10 years after As we may think was published. The existence of these modern uses of the internet do not preclude the research use envisaged by these information scientists, it just relegates it to a niche.

The problem for people like you and me, who occupy this niche, is that the divergence of consumer internet technology from the original research oriented web is eventually going to make it more expensive. There's no law that says that Google has to provide an RSS reading tool for free (or a blogging service for that matter). In fact, the withdrawal of this service may actually provide a shot in the arm for the makers of RSS readers who have been starved by people like me who use the freebie Google Reader instead of their more tailored offerings. Yes, I would be prepared to pay to have something like Google Reader that stays in sync across my tablet, phone and laptop.

Ad, ad, ad...

While I'm on the subject of money, I do want to draw your attention to Xanadu's rule 9:

Every document can contain a royalty mechanism at any desired degree of granularity to ensure payment on any portion accessed, including virtual copies ("transclusions") of all or part of the document.

I really think it is time that technology providers started to look again at this goal. In the early days of the internet this was considered unrealistic. In fact, I remember sitting through meetings in which people responsible for creating the infrastructure that made the internet possible were highly doubtful that traffic accounting would ever be possible. The growth in internet traffic would always outpace the ability of switching gear and routers to count bits and report on usage. That prediction turned out to be wrong. I think they underestimated the strength of the business case behind bit-counting, which is routine on mobile platforms. My cheap router counts my own internet usage and I know my service provider has realtime stats too, if only to enforce their acceptable usage policy.

There have been a lot of haters for charging based on consumption of bits and this, in my opinion, has distorted the business models available to service providers towards ad-based services and away from the Xanadu-like micro payments.

Most of the rhetoric about the demise of Google Reader is taken from the point of view of the consumer, not the information publisher. Of course I want to consume content for free using free technology over an unlimited internet connection. But none of these things are really free. We've all heard the adage that if something is free then you're the product. As an RSS consumer, my costs just outstripped my marketable value to Google. I'm not a cash cow anymore, I'm a dog.

From Reader to Blogger

But as I type, I'm not just consuming the content I used to research it. I'm also publishing content of my own. At the moment for free. I don't want to enable ads on this blog but the technology doesn't yet make it easy for me, or anyone between me and you, to collect revenue and experiment with pricing. It's more complicated than you might think.

Rule 8 of Xanadu reads "Permission to link to a document is explicitly granted by the act of publication." Early internet sites seriously considered violating this principal. Content providers considered themselves to be so valuable that someone creating a site that aggregated links to their gems were somehow cheating the system. This has been turned completely on its head now, these days information providers are hungry for links and when those links result in product sales they are prepared to pay real money to the aggregator. This is the basis on which all the market comparison sites are run.

If content publishers got revenue from people viewing their materials (Xanadu style) then linking to someone's content becomes a valuable lead. How would payments trickle back to the owner of the  tag?

We know that the ad-model works. YouTube generates huge revenues for people like PSY. But for people outside the mainstream who occupy this niche, typified by users of Google Reader, we need another way to solve the money problem. Perhaps the new technology that emerges to take the place of Reader will come up with a creative way to address this issue. Especially if they start getting paid by their users.
 Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-70242987829028007572013-01-30T11:42:00.000+00:002013-01-30T11:42:05.350+00:00OData: Open for Comments at OASIS
Browsing around this morning I noticed that on Friday (25th Jan) there was a test posting to a new mailing list set up by the OASIS technical committee that is taking forward the OData specification.

To recap, OData is a specification that extends the popular Atom Publishing Protocol (APP) with conventions that make it easy to expose data sources (think relational databases) in a standard way. OData has been driven by Microsoft and is now at version 3, but it seems to be making the transition to a work item at OASIS where it seems likely that a more open specification process will be observed.

I've written about OData before but the best way to play with it is to look at some sample feeds, the Netflix database is one I tend to use for my examples because the data is real and something that is widely understood.

With the work now at a more formal standards body I hope that some of the rough edges of the existing specification can be knocked off. This type of thing is important if OData is to make the transition from a specification which works well if you have client and server libraries from the same vendor to one which can be truly interoperable.

For example, the current specification makes a mess of defining the simple concept of a string-literal parsed from a URL. As a result, it is impossible to make a conforming URI which will get you information about an actor like Peter O'Toole. Here's a URL that a naive user might construct:

http://odata.netflix.com/catalog/People?$filter=Name%20eq%20'Peter%20O'Toole'

Notice that the single-quote character in O'Toole terminates the literal and, sure enough, Netflix returns an error.

Syntax error at position 22.

In fact, there is an undocumented way to get around the problem, using the SQL-convention of doubling the quote character:

http://odata.netflix.com/catalog/People?$filter=Name%20eq%20'Peter%20O''Toole'

I've posted a comment to highlight this issue to the new OData comment list, let's see what happens! It's a public forum so anyone can join though the work of the technical committee itself is behind closed doors (OASIS is a subscription-based membership organization).

I'm a fan of what the basic OData specification is trying to do so getting things like this fixed is important. Just looking at the XML file you get back from the above URI immediately opens up the wonderful world of linked data, giving me relative links like People(69540)/TitlesActedIn from which you can see details of all the films Peter O'Toole has acted in. Don't like XML? Just add ?$format=json to the URL and you can consume the list directly into your web-page.

Last year I gave a lightning talk at a CETIS event in which I encouraged people who were creating REST-based protocols as part of their technical standards development process to have a really close look at OData. Building new specifications using existing protocols can dramatically save time when drafting and make it much easier for people to implement afterwards. And even if OData is not for you, if your application is a good fit for a REST-based approach why not just use APP as it is? Forgot the additional complications of things like WADL, you don't need them. What's more, if you use APP then you can take advantage of existing implementations in web browsers to provide basic and easy to consume views of your data.
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-28467405252995865722012-12-05T21:00:00.000+00:002012-12-05T21:00:05.184+00:00Writing a stream to a zipfile in Python, harder than you think!
So here's the problem, you have a stream (a file-like object) in Python and you want to spool the contents of it into a zip archive. Sounds like a common requirement? It turns out to be very hard. I propose a solution here with hooks.

There are two methods for writing data to a zip file in the Python zipfile module.

ZipFile.write(filename[, arcname[, compress_type]])

and

ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type])

The first takes the name of a file, opens it and spools the contents in to the archive in 8K chunks. Sounds like a good fit for what I want except that I have a file-like object, not a file name, and ZipFile.write won't accept that. I could create a temporary file on disk and write my data to that, then pass the name of the file instead but that supposes (a) that I have access to the file system for writing and (b) I don't mind spooling the data twice, once to the disk and once back out again for storage in the archive.

Before you protest, the ZipFile object only requires a file-like object with support for seek and tell, it doesn't actually have to be a file in the file system so (a) is still a valid scenario. We will have to ditch any clever ideas of spooling a zip file directly over network connections though. A closer look at the implementation shows us that once the data has been compressed and written out to the archive the stream is wound back to the archive entry's header to update information about the compressed and uncompressed sizes. Still, even if you are buffering the output at least you are dealing with the smaller compressed data and not the original uncompressed source.

So if ZipFile.write doesn't work for streams what about using ZipFile.writestr instead? This takes the data as a string of bytes (in memory). For larger files this is unlikely to be practicable. I did wonder about tricking this method with a string-like object but even if I could do this the method will still attempt to create an ordinary string with the entire compressed data which won't work for large streams.

Solution 1

The first solution is taken from a suggestion on StackOverflow. The idea is to wrap the ZipFile object and write a new method. Clearly that would be something good for the module maintainers to consider but it requires considerable copying of code. If I'm going to be so dependent on the internals of the ZipFile object implementation I might as well look to see if there is a better way.

Solution 2

Looking at the ZipFile implementation the write method is clearly very close to what I want to do. If only it would accept a file-like object! A closer look reveals that it only does two things with the passed filename. It calls os.stat and then, shortly afterwards, calls open to get a file-like object.

This got me thinking whether or not I could trick the write method in to accepting something other than the name of a file. I created an object (which I called a VirtualFilePath) and gave it a stat and open method. The implementation is not important, but this object essentially wraps my file-like object simulating these two operating system functions.

Unfortunately, I can't pass a VirtualFilePath to the operating system open function. I'll get an error that it wasn't expecting an instance. The same goes for os.stat. However, I can write hooks to intercept these calls and redirect the calls to my methods if the argument is a VirtualFilePath. This is basically what my solution looks like:

import os,__builtin__

stat_pass=os.stat
open_pass=__builtin__.open

def stat_hook(path):
 if isinstance(path,VirtualFilePath):
 return path.stat()
 else:
 return stat_pass(path)

def open_hook(path,*params):
 if isinstance(path,VirtualFilePath):
 return path.open(*params)
 else:
 return open_pass(path,*params)

class ZipHooks(object):
 hookCount=0

 def __init__(self):
 if not ZipHooks.hookCount:
 os.stat=stat_hook
 __builtin__.open=open_hook
 ZipHooks.hookCount+=1

 def __enter__(self):
 return self

 def __exit__(self, type, value, traceback):
 self.Unhook()

 def Unhook(self):
 ZipHooks.hookCount-=1
 if not ZipHooks.hookCount:
 os.stat=stat_pass
 __builtin__.open=open_pass

This code adds hooks which detect my VirtualFilePath object when it is passed to open or stat and redirects those calls. To make it easier to manage the hooks we create a ZipHooks object with __enter__ and __exit__ methods allowing it to be used in a 'with' statement like this:

with ZipHooks() as zh:
 # add stuff to an archive using VirtualFilePath here

There's one final detail to clear up. stat is supposed to return the size of the file but what if I don't know it because I'm reading data from a stream? In fact, closer inspection of the ZipFile.write method's implementation shows that it doesn't really rely on the size returned by stat as it monitors both compressed and uncompressed sizes and re-stuffs the header when it back-tracks.

The only other bits of stat that ZipFile.write is interested in is the modification date of the file and the mode (which it uses to determine if the file is really a directory). So if your file-like object isn't very file-like at all it won't matter too much because you only have to fake these fields in the stat result.
 Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-71979091499273870942012-08-14T10:45:00.000+01:002012-08-14T10:45:18.268+01:00Thomson Routers from plusnet: problems using Gmail over wifi?
Summary

If you came to this page because of a Google search for this problem then here is my advice in a nutshell:

Turn off the "Web Browsing Interception" feature in your router by setting it to "disabled".

Set your wifi to b/g operation only to reduce data speeds - you probably don't need more than 54Mb/s if you are on ADSL

Power the router off, count to 30 and power it back on.

If the problem returns, repeat step 3.

If the problem comes back frequently consider using a cheap timer plug to power cycle the router automatically in the middle of the night.

Oh, while you're here, could you just look at my PC?

Everyone who works in any job that is vaguely computer related will know that sometimes, when visiting a friend or relation's house the conversation will come around to some little problem they are having with their PC or home network. I had just such an experience at the weekend. I would normally have attempted to post the technical workaround to the plusnet forum where the particular issue I was debugging is being discussed but you have to be a plusnet customer to use the forums and I'm just a visiting techie so I'm posting what I know here.

The Symptoms

The scenario is a simple home network with a Thomson TG585 v8 ADSL router supplied by plusnet and an ordinary Dell laptop. I actually got a report of this problem earlier in the year. The main symptom was that email with attachments couldn't be sent. As the laptop owner spoke to me on a cordless wireless phone I could hear some interference on the line each time the laptop tried to send the email.

The First Diagnosis

This network is in a fairly isolated spot with only one other house within wifi range so interference from other wifi networks seemed unlikely. However, the house does have a cordless phone with a signal booster to reach to outbuildings. As is often the case with wifi issues you have to suspect interference, some older cordless phones are simply unusable alongside 2.4GHz routers. The obvious test is to instruct the user to plug their laptop directly in to one of the ethernet ports on the back of the router. Sure enough, the problem went away.

So is wifi really incompatible with this make of phone? The phone is of a modern design and is supposed to be compatible with wifi but perhaps that booster is the problem? To make the cable-based workaround easier the router was moved nearer the laptop. This took it further away from the phone and the booster - I had hoped that this might improve wifi connectivity enough for attachments to work but the problem remained. Needless to say, next time I visited the house the subject came up!

The problem gets worse

I must confess, I had forgotten about the problem completely until I arrived at the house and attempted to send an email via their wifi network from my own laptop. I was able to receive all my email no problem, web browsing was fast but I could not send messages through either my work email or through my Gmail account. Could this be related to the problem with sending attachments? Here is someone asking a similar question: Gmail, Windows Live Mail and Plusnet problems?

The problem appears so black-and-white I began to suspect that it must be a block at the ISP until I read this thread on plusnet's community forum: Mail cannot send attachments. The thread ends with an answer from a plusnet employee but the original problem from someone using Apple Mail (as I do) goes undiagnosed.

Eventually one of my smaller email messages managed to sneak out of my laptop at the Nth time of trying. Intermittent problems may be the worst to diagnose but at least this showed me that it is just a really bad connectivity problem.

I'm not alone

At this point I went back to my first diagnosis. I tried every channel, I powered off every other wireless device I could find (including the phones) and I still had the same trouble. This is beginning to look like a bug in the router. I checked the firmware version (22.214.171.124) and found a new thread that matched my symptoms: Thomson TG585V8 poor upload speed over wifi and then the more general: Wireless connectivity on the Thomson V8. The second of these has 13 pages of discussion and brings the problem right up to date: but with no resolution!

The solution

To recap, we have a router that is perfectly capable of sending (downloading) data over wifi at high speed but is very bad at receiving it during uploads. I started suspecting some type of buffer overflow or error in one of the protocol stacks but restricting it to b or b/g operation didn't fix the issue either. I then started looking at other tools and settings in the router and came upon "Web Browsing Interception". I have no idea what this means but it was set to automatic. I found one post on a DSL forum which was enough to raise suspicion: Web Browsing Interception?. I disabled it and applied the settings, almost immediately my email started going through as normal.

The same feature is cited in this thread relating to different Thomson hardware (which I found later): "Solved: Thomson Speedtouch Modem, disconnects often and slows down browsing".

So did this solve the problem with attachments? No. At least not directly. Turning this setting off stopped the router interfering with mail clients making outbound connections to Gmail and Microsoft mail products. But I was still getting larger attachments stuck at about the 100Kb mark. It may be that all I have done is free up enough resource (RAM or CPU) to send small messages by turning off a feature of the router software. In which case, it seems likely that a resource leak of some kind is to blame in these routers. To test the theory I powered the router on and off, waited ages for it to restart (it really is slow) and then tried again. Everything worked fine - all attachments sending without error, even large ones.

So is this Web Browsing Interception feature to blame? Possibly, the router has been powered on and off before without fixing the problem so it seems likely that this feature is either causing or exacerbating a resource leak in the unit.
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-23253190966518495102012-07-08T13:36:00.001+01:002012-07-08T13:42:30.829+01:00The demise of iGoogle: is this the beginning of the end for widgets?What's happening to iGoogle? - Web Search Help:

So I woke up this morning and found a warning on my home page. iGoogle is going away it calmly announced. November 2013, but earlier than that if you are using a 'mobile' device.

So that means my home page is going away. I can finally give up on the idea that Google will come up with a decent iGoogle experience on the iPad, or even on my Android phone. I may have just upgraded to a slightly larger 15" MacBook but it is still portable - it must be, I've just carried it from the UK to New Orleans ready for OSCELOT and Blackboard's annual 'world' conference. In fact, thanks to United Airlines' upgrade programme I was even able to plug it in and use it for most of the flight.

So what am I going to miss about my home page? I'll miss the "Lego men" theme but I can probably live without Expedia reminders which count me down to my next trip. In fact, if they would only just say something more appropriate than "Enjoy your holiday" I might miss that gadget a bit more. I have quick access to Google Reader but I increasingly find myself reading blogs on my phone these days - it just seems like the right thing to do on the tube (London's underground railway). I'm not sure about Google bookmarks - I assume they're staying so I might have to make them my home page instead. Finally, I'm not sure I've ever actually chatted to anyone through iGoogle.

So I'm over iGoogle, not as easily as Bookmark lists but nothing I can't handle.

 Is this the end of widgets?

iGoogle was one of the more interesting widget platforms when it launched. You can write your own widgets, get them published to their gadget list so that other users can download them and install them on their iGoogle page. They're small, simpler than full-blown computer applications on the desktop, simpler and smaller than complete websites. iGoogle is a platform which reduces the barrier to entry for all sorts of cool little apps. It is particularly good for apps that allow you to access information stored on the web, especially if it is accessible by JSON or similar web services.

You may notice a strong resemblance to mobile apps. Google certainly have and this is the main reason why iGoogle is going away. It is no longer the right platform. People with 15" screens organizing lots of widgets on their browser-based home page are an anachronism. These days people organize their apps on 'home screens', flipping between them with gestures. They don't need another platform. Apple have already seen this coming, in fact, they are having a significant influence on the future. There is already convergence in newer versions of Mac OS.

There's a lot of engineering involved in persuading browsers to act as a software platform. The browser doesn't do it all for you (and plugins do not seem like the solution because they are browser specific). There are a number of widget toolkits available for would-be portal engineers but the most popular portals tend to have their own widget implementations (just look at your LMS or Sharepoint).

For many years I've been watching the various widget specifications emerge from the W3C, lots of clever engineers are involved (those I know I hold in high regard) but I'm just beginning to get that sickening feeling that it is all going to have been a learning experience. At the end of the process we may have a technically elegant, well-specified white elephant.

As someone who has spent many years developing software in the e-Learning sector I've always found it hard to draw the line between applied research which is solely for the purposes of education and applied research which is more generally applicable but is being driven by the e-Learning sector. As a community, we often stretch the existing platforms and end up building out new frameworks only to have to throw stuff away as the market changes underneath us. The web replaced HyperCard and Toolbook in just this way - some of the content got migrated but the elaborate courseware management systems (as we used to call them) all had to be thrown away.

'via Blog this'Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-66034497111724493182012-07-02T15:16:00.001+01:002012-07-02T15:16:38.996+01:00QTI Pre-Conference Workshop: next week!Sadly I won't be able to make this event next week but I thought I'd pass on a link to the flyer in case there is anyone still making travel plans.

http://caaconference.co.uk/wp-content/uploads/CAA-2012-Pre-Conference-Workshop.pdf

I'm still making the odd change to the QTI migration tool - and the integration with the PyAssess library is going well. This will bring various benefits like the ability to populate the correct response for most item types when converting from v1. So if you have a v1 to v2 migration question coming out of the workshop please feel free to get in touch or post them here.

'via Blog this'Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-48298964454008940112012-06-19T15:35:00.000+01:002012-06-19T15:35:05.606+01:00What is ipsative assessment and why would I use it? | Getting Results -- The Questionmark BlogWhat is ipsative assessment and why would I use it? | Getting Results -- The Questionmark Blog:

Having recently registered for the eAssessment Scotland conference I was reminded that last year I learnt a new word there: ipsative assessment.

This is a nice summary on the subject from John Kleeman.

'via Blog this'Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-17813071891599620832012-06-15T10:13:00.000+01:002012-06-15T10:13:04.048+01:00Viewing OData $metadata with XSLT
In a recent post I talked about work I've been doing on OData. See Atom, OData and Binary Blobs for a primer on Atom and some examples of OData feeds.

In a nutshell, OData adds a definition language for database-like Entities (think SQL tables) to Atom and provides conventions for representing their properties (think SQL columns) in XML. Their definition language is based on ADO.NET, yes they could have chosen different bindings which would have made more work for their engineers, less for the rest of us and improved the chances of widespread adoption. But it is what it is (a phrase I seem to be hearing a lot of recently).

One of the OData-defined conventions is that data services can publish a metadata document which describes the entities that you are likely to encounter in the Atom feeds it publishes. This can be useful if you want to POST a new entry and you don't know what type of data property X is supposed to have. To get the metadata document you just get a special URL, for the sample data service published by Microsoft:

http://services.odata.org/OData/OData.svc/$metadata

To see what this might look like in the real world you can also look at the $metadata document published as part of the Netflix OData service I used as the source of my examples last time.

http://odata.netflix.com/v2/Catalog/$metadata

Wouldn't it be nice to have a simple documented form of this information? The schema on which it is based even allows annotations and Documentation elements. I browsed the web a bit but couldn't find anyone who had done this so I wrote a little XSLT myself. Here is a picture of part of the output from transforming the Netflix feed

Now there is an issue here. One of the things that I've commented on before is the annoying habit of specification writers to change the namespace they use when a new version is published. I can see why some people might do this but when 90% of the spec is the same it just causes frustration as tools which look for one namespace have to have significant revisions just to work with a minor addition of some optional elements.

As a result, I've published two versions of the XSLT that I used to create the above picture. If somebody out there knows how I can do all of this in one XSL file without lots of copying and pasting I'd love to know.

The first xslt uses the namespace from CSDL 1.1 and can be used to transform the metadata from the sample OData service published by Microsoft. The second xslt uses the namespace from CSDL 2.0 and must be used to transform the metadata from Netflix. If you get a blank HTML file with just a heading then try the other one. When clicking on these links your browser may attempt to render them as HTML, you can "View Source" to see the XSLT as plain text.

odata2html-v1p1.xsl

odata2html-v2.xsl

Here is how I've used these files on my Mac:

$ curl http://services.odata.org/OData/OData.svc/\$metadata > sample.xml
$ xsltproc odata2html-v1p1.xsl sample.xml > sample.html

$ curl http://odata.netflix.com/v2/Catalog/\$metadata > netflix.xml
$ xsltproc odata2html-v2.xsl netflix.xml > netflix.html

This transform could obviously be improved a lot, it only shows you the Entities, Complex Types and Associations though I am rather proud of the hyperlinking between them.
Steve Layhttps://email@example.com:blogger.com,1999:blog-8659912959976079554.post-60895568006412946582012-06-07T08:14:00.000+01:002012-06-07T08:14:04.138+01:00Launching learning activities with AICC CMI-5: GET or POST?
With the AICC working on a new specification to update the existing use of the CMI data model and replace the ageing (but still popular!) HACP it seems like a good opportunity to fix the concept of launch via a web browser to harmonise learning technology standards with best practice in mainstream applications. By the way, if you are curious HACP is defined in the 1993 document: [CMI001] CMI Guidelines for Interoperability AICC.

The Learning Management System (LMS) typically does the launch by creating a special page that is delivered to the user's web browser containing a form (or a naked hyperlink with query parameters) to trigger the activity in a remote system. There are some subtle things going on here because best practice in HTML has been based on the assumption that the server that generated the form is the one that will receive the submission whereas our community tends to rely on cross-site form submission (XSFS perhaps?).

Anyway, you can read my recommendation to the AICC in the open forum created by them to discuss the CMI-5 draft specification. And of course, if you haven't had a look yet I'd encourage you to have a quick look over the specification yourself. The more people review and comment on these documents the better.

Link: Topic: 8.1 Web (Browser) Environment

The topic I just started with my new proposal on how the LMS should launch a learning activity in future. I'm recommending that the POST method is used with GET supported for backwards compatibility only so if you support AICC standards please read and review my proposal because you might have to do some work on your launch code to support CMI-5 if my proposal is accepted.
Link: CMI-5 Forum Home Page

The home page of the AICC CMI-5 public forum, see all threads about the proposed specification here. Note that only registered users can see the interactive specification viewer in the forums but there is a general public download on the main CMI-5 page:

Link: CMI-5 Home Page

You can download the full specification as a PDF from here.
Steve Layhttps://firstname.lastname@example.com:blogger.com,1999:blog-8659912959976079554.post-41825664243936854352012-06-05T13:24:00.001+01:002012-06-05T13:24:33.616+01:00Lion, wxPython and py2app: targeting Carbon & Cocoa
About this time last year I wrote a blog entry on installing wxPython and py2app on my Mac running Snow Leopard. Well I have since upgraded to Lion and the same installation seemed to keep working just fine. This weekend I've actually upgraded to a new Mac and I thought this would be an excellent opportunity to grapple with this installation again and perhaps advance my understanding of what I'm doing.

Apple's Migration Assistant (a.k.a. Setup Assistant) had other ideas. You have to hand it to them, it took about 2-3 hours with the two machines plugged together and everything was copied across and working without any intervention. They really do make it easy to buy a new Mac and get productive straight away.

So is this blog post the Python equivalent of the "pass" statement? Just a no-op?

Well not quite. At the moment, I'm building my QTI migration tool using the Carbon version of wxPython which means forcing Python to work in 32bit mode. That's getting a bit out-dated now and I surely can't take advantage of my new 8GB Mac? I need to embrace 64bit builds, I need to figure out how to build for the Cocoa version of wxPython and I need to figure out how to do this while retaining my ability to create the 32-bit build for older hardware/versions of Mac OS.

So here is how I now recommend doing this...

Step 1: Install Python

The lesson from last time was that you can't rely on the python versions installed by Apple to do these builds for you. If you run python on a clean Lion install you'll get a 64-bit version of python 2.7.1:

$ which python
/usr/bin/python
$ python
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys;"%X"%sys.maxsize
'7FFFFFFFFFFFFFFF'

Thanks to How to tell if my python shell is executing in 32bit or 64bit mode? for the tip on printing maxsize.

I want to keep python pointing here because the command-line version of the migration tool, and the supporting Pyslet package (which can be used independently) need to work 'out-of-the-box'. The Migration Assistant had helpfully copied over my .bash_profile which included modifications made by my custom Mac binary install of Python 2.7 last year. The modifications are well commented and help to explain the different paths we'll be dealing with:

# Setting PATH for Python 2.7
# The orginal version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH}"
export PATH

Firstly, note that the Mac binaries install in /Library/Frameworks/ whereas the pre-loaded Apple installations are all in /System/Library/Frameworks/. This is a fairly subtle difference so a little bit of concentration is required to prevent mistakes. Anyway, as per the instructions above I restored my .bash_profile from .bash_profile.pysave and confirmed (as above) that I was getting the Apple python.

It seems like 2.7.3 is the latest version available as a Mac binary build from the main python download page. This will make it a bit easier to check I'm running the right interpreter! So I downloaded the dmg from the following link and ran the installer: http://www.python.org/ftp/python/2.7.3/python-2.7.3-macosx10.6.dmg. For me this was an upgrade rather than a clean install. The resulting binaries are put on the path in /usr/local/bin. By default, the interpreter runs in 64bit mode but it can be invoked in 32-bit mode too:

$ /usr/local/bin/python
Python 2.7.3 (v2.7.3:70274d53c1dd, Apr 9 2012, 20:52:43)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys;"%X"%sys.maxsize
'7FFFFFFFFFFFFFFF'

$ which python-32
/usr/local/bin/python-32
$ python-32
Python 2.7.3 (v2.7.3:70274d53c1dd, Apr 9 2012, 20:52:43)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys;"%X"%sys.maxsize
'7FFFFFFF'

This might be enough, but the wxPython home page does recommend that you use different python installations if you want to run both the Carbon and Cocoa versions. So I'll repeat the installation with a python 2.6 build. The current binary build is 2.6.6, this is missing some important security fixes that have been included in 2.6.8 but it looks safe for the migration tool. I downloaded the 2.6 installer from here. When I ran the installer I made sure I only installed the framework, I don't want everything else getting in the way.

I now have a python 2.6 installation in /Library/Frameworks/Python.framework/Versions/2.6

$ /Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys;"%X"%sys.maxsize
'7FFFFFFF'

Notice that this is a 32-bit build. Apple actually ship a 64-bit build of 2.6.7 so care will be needed, typing python2.6 at the terminal will not bring up this new installation.

To make life a little bit easier I always create a bin directory in my home directory and add it to the path in my .bash_profile using lines like these:

PATH=~/bin:${PATH}
export PATH

This is going to come in very handy in the next step.

Step 2: setuptools and easy_install

setuptools is required by lots of Python packages, it is designed to make your life very easy but it takes a bit of fiddling to get it working with these custom installations. It's an egg which means it runs magically from the command line, I'll show you the process of installing it on python 2.6 but the instructions for putting it in 2.7 are almost identical (it's just a different egg).

I downloaded the egg from here: http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg#md5=bfa92100bd772d5a213eedd356d64086 and then took a peek at the top of the script:

$ head -n 8 setuptools-0.6c11-py2.6.egg
#!/bin/sh
if [ `basename $0` = "setuptools-0.6c11-py2.6.egg" ]
then exec python2.6 -c "import sys, os; sys.path.insert(0, os.path.abspath('$0')); from setuptools.command.easy_install import bootstrap; sys.exit(bootstrap())" "$@"
else
 echo $0 is not the correct name for this egg file.
 echo Please rename it back to setuptools-0.6c11-py2.6.egg and try again.
 exec false
fi

I've only shown the top 8 lines here, the rest is binary encoded gibberish. The thing to notice is that, on line 3 it invokes python2.6 directly so if I want to control which python installation setuptools is installed for I need to ensure that python2.6 invokes the correct interpreter. That's where my local bin directory and path manipulation comes in handy.

$ cd ~/bin
$ ln -s /Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6 python2.6
$ ln -s /Library/Frameworks/Python.framework/Versions/2.6/bin/python2.7 python2.7

Now for me, and anyone who inherits my $PATH, invoking python2.6 will start my custom MacPython install.

$ sudo -l python2.6
/Users/swl10/bin/python2.6

Fortunately sudo is configured to inherit my environment. It was worth checking as this is configurable. I can now install setuptools from the egg:

$ sudo sh setuptools-0.6c11-py2.6.egg
Password: [I had to type my root password here]
Processing setuptools-0.6c11-py2.6.egg
Copying setuptools-0.6c11-py2.6.egg to /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
...

I did the same for python 2.7 (note that you need a different egg) and then added links to easy_install to my bin directory:

$ cd ~/bin
ln -s /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install easy_install-2.7
ln -s /Library/Frameworks/Python.framework/Versions/2.6/bin/easy_install easy_install-2.6

Step 3: wxPython

wxPython is a binary installer tied to a particular python version. However, I believe it uses a scatter gun approach to search for python installations and will install itself everywhere with a single click. That is the reason why it is better to run completely different versions of python if you want completely different versions of wxPython. In fact, if you find yourself with multiple installs there is a wiki page that explains how to switch between versions. But a little playing reveals that this refers to Major.Minor version numbers. It can't cope with the subtlety of switching between builds or switching between Carbon and Cocoa as far as I can tell so this won't help us.

My plan is to install the Carbon wxPython (which is 32bit only) for python 2.6 and the newer Cocoa wxPython for python 2.7. The wxPython download page has a stable and unstable version but to get Cocoa I'll need to use the unstable version. The stability referred to is that of the API, rather than the quality of the code. Being cautious I downloaded the stable 2.8 (Carbon) installer for python 2.6 and the unstable 2.9 Cocoa installer for python 2.7. Installation is easy but look out for a useful script on the disk image which allows you to review and manage your installations. To invoke the script you can just run it from the command line:

$ /Volumes/wxPython2.9-osx-22.214.171.124-cocoa-py2.7/uninstall_wxPython.py

When I was done with the installations it reported the following configurations as being present:

 1. wxPython2.8-osx-unicode-universal-py2.6 188.8.131.52
 2. wxPython2.9-osx-cocoa-py2.7 126.96.36.199

(If, like me, you are upgrading from previous installations you may have to clean up older builds here.) At this point I tested my wxPython based programs and confirmed that they were working OK. I was impressed that the Cocoa version seems to work unchanged.

Step 4: py2app

With the groundwork done right, the last step is very simple. The symlinks we put in for easy_install make it easy to install py2app.

$ sudo easy_install-2.6 -U py2app
Password: [type your root password here]
Searching for py2app
Reading http://pypi.python.org/simple/py2app/
Reading http://undefined.org/python/#py2app
Reading http://bitbucket.org/ronaldoussoren/py2app
Best match: py2app 0.6.4

...[snip]...

Installed /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/altgraph-0.9-py2.6.egg
Finished processing dependencies for py2app

The plot is very similar for python 2.7.

I've now added the following to my setup script:

import wx
if "cocoa" in wx.version():
 suffix="-Cocoa"
else:
 suffix="-Carbon"

The suffix is then appended to the name passed to the setup call itself. The following command results in a 32bit, Carbon binary, compatible with OS X 10.3 onwards.

$ python2.6 setup.py py2app

While this command creates a Cocoa based 64bit binary for 10.5 and later.

$ python2.7 setup.py py2app

And that is how to target both Carbon and Cocoa in your wxPython projects.
Steve Layhttps://email@example.com