Shuffl sprint 6 progress review

Progress during this sprint has been fairly poor, largely due to distraction from both personal and non-project affairs. The total amount of effort spent was 3.5 days against planned effort of 8 days. A further factor impacting progress was that reorganizing the code to enable graph label-row selection in a data table took about a day longer than expected.

On the positive side, I did hold a second review meeting with Chris Holland, and the current development is being conducted very much in response to his feedback, to make the data graphing display more useful to him.

Also on the positive side, the mechanisms for linking data between cards seem to be working nicely (e.g. when I reload new data into a table card, or change the label row, an associated graph card updates immediately; but I do need to capture this relationship when I save and restore a workspace).

Some thoughts about sprint planning: for this sprint, having fewer than 4 days actual effort expended, the sprint planning and review process seems to lack sufficient data to be meaningful. In setting sprint duration, it would seem reasonable to take account of the total amount of effort being applied rather than simply the number of elapsed days. I don't plan to change the duration of the remaining two sprints for this project, but for future projects, planning sprints with fewer than 10-15 total days of effort may be something to avoid.

Accessing files from browser-based Javascript

Designing a user-friendly mechanism for accessing files from Javascript has proven somewhat elusive. My original plan was to create a temporary form in a card using an <input type="file"> element to upload the file to the AtomPub server. Unfortunately, I could find no way to control the way that data is uploaded, and I would be forced to bypass the AtomPub module, leading to problems and fragility when I implement support for different back-ends. Next, I thought to use an <input type="file"> element to allow me to browse the file system and select a file, then to some behind-the-scenes mapping from the selected file path to the corresponding AtomPub URI, so that the Javascript code can read the file via an Ajax GET request. This approach was foiled by the browsers refusal to let me access the full path name of the file: only the file name is returned from the form (in Firefox, at least). A little Googling suggests this is a browser security "feature" (though I have to say that it smells like security by obscurity). So my final solution is to simply ask the user to enter the URI of the file - I separate the base URI of the AtomPub-served file area from the rest of the filename so that a reasonable default can be provided. It's not as user-friendly as I'd like, but I'm not prepared to invest a lot of time in this at the moment, so I'm leaving it at that. I'm hoping that I may come across a better solution in due course. Fortunately, my experience with research users is that they're prepared to deal with a little clunkiness if it doesn't require too much additional thought and it helps them to get a result they want. I'm writing this up in the hope that someone else may be able to offer a solution. So far, the most likely option seems to be to upload to a temporary file area in the AtomPub server domain and return the URI thus allocated, but that would force me to make too many assumptions about the back-end service than I'm comfortable with right now.

Sprint 3 review

Sprint 3 is finished, and progress notes can be viewed here.

The main points to note are that I've had to re-assess my plan for implementing workspace persistence, and have backed up to implement a new layer of test cases. Progress has been made, but workspace saving/persistence is still not yet complete.

The other main feature of the sprint has been lots of meetings: the JISCRI meeting, the London linked web data, and a VoCamp in Bristol. All of these have contributed, in various ways, to my thoughts for taking Shuffl to users, and have provided opportunities to discuss ideas and use-cases.

I've tagged this progress note as a FAIL as the project is slipping compared with plan: AtomPub has proved rather more of a handful to master than expected. Progress is being made, but more slowly than hoped. I'm hoping to be able to get back to user-visible functionality before too much longer.

DELETE on AtomPub media resource

I read somewhere that deleting an AtomPub media resource associated with a feed item (using HTTP DELETE) should also delete the feed item. It turns out that the eXist AtomPub implementation does not support this, claiming (erroneously) that the resource does not exist.

Examining the AtomPub spec provides little support (though on first reading I actually thought this did support what I want to do):

To delete a Member Resource, a client sends a 

DELETE request to its Member URI, as specified 

in [RFC2616].  The deletion of a Media Link

Entry SHOULD result in the deletion of the 

corresponding Media Resource.

-- RFC 5023, section 9.4

This is unfortunate for Shuffl, as I had been hoping to not get too bogged down in feed items vs resources. Now I might need to include a feed item URI along with card data if I'm to be able to delete the card data. Or maybe I'll just delete the feed and recreate it without the missing data?

More generally, I am beginning to wonder if AtomPub was the best choice for persisting Shuffl data. I estimate the complexities of dealing with AtomPub have cost me a week or so in slippage, and that I would be better off with WebDAV. I'll stick with it for now, but I'm thinking about a plug-in framework for Shuffl back-end support so I can support alternative protocols.

AtomPub with jQuery implementation done

WIN and FAIL: the AtomPub protocol handler has been developed and tested, but has taken more than twice the effort originally estimated. The failure here is, I think more a failure of estimation than indicative of a technical problem. For details, see http://shuffl-announce.blogspot.com/2009/08/implementing-atompub.html.

eXist and Jetty configuration

My initial experiments to get eXist installed and running as an AtomPub server went very smoothly.

But I've just spent a day thrashing around with the server configuration, trying to get it to serve static files from the Shuffl project test directory so that I can run tests more easily, without falling foul of the Javascript "same origin" restriction. This has been complicated by a number of factors:

• Jetty is natively configured by Java object dependency injection, for which XML configuration files are a shim interface
• the Jetty documentation isn't very approachable, especially as the XML configuration sections don't actually tell you how to configure the server - for that, you have to dig into the various servlet classes to understand the values that need to be injected
• some configuration options can alternatively be applied through the servlet container configuration (web.xml), which is a completely different format
• the eXist installation runs a specially tailored configuration of Jetty that doesn't immediately make it easy to find the configuration options
• default security settings in Jetty do not permit following symbolic links when serving static files

Most of these are not necessarily bad things, but in combination they create a system whose configuration is about as user-friendly as a cornered rat. For me, the breakthrough came when I found a line in the eXist documentation "The Jetty configuration can be found in tools/jetty/etc/jetty.xml" (http://exist-db.org/deployment.html, section 3). This is relative to the eXist installation directory.

Editing this file, I can change (almost the last line in the file):

<Set class="org.mortbay.util.FileResource" name="checkAliases" type="boolean">true</Set>

to

<Set class="org.mortbay.util.FileResource" name="checkAliases" type="boolean">false</Set>

Now I can create a symbolic link shuffl in ${EXIST_HOME}/webapp, linking to my shuffl project directory, and by pointing my browser at http://localhost:8080/exist/shuffl/ I can browse my project directories and run the test files via eXist. Phew!