Monday 28 September 2009

Framework for testing Shuffl card plugins

Card plugins are looking to be a powerful extension point for Shuffl, allowing as they do for arbitrary additional logic to be associated with a card. This extra logic may encompass data handling, user interactions and external system interactions.
This presents a challenge for testing, especially as it has proven tricky to simulate user interaction with other jQuery plugins, such as editable text values, without resorting to external testing tools like Selenium. (I have used Selenium in the past, and it has been very useful, but it does involve working with a whole new framework, and in my experience it can be unreliable when testing event-intensive browser code. I prefer to stick with the simple jQuery testing framework for unit testing, though I do expect to eventually adopt a UI testing framework as well.)
The approach I'm taking is based on what we did with the FlyWeb project: there, we adopted a pattern for widgets based on the Model-View-Controller (MVC) pattern. Essentially, all updates to the browser display and user interactions were decoupled from the main system logic through a generic "Model" facility with which code interacts using a publish/subscribe (or "Observer") pattern.
I've looked at replicating this for Shuffl, and was able to create a simple jQuery plug-in that provides a model and model-listener capability for any jQuery object, all in little more than 20 lines of code! It turns out that jQuery already has most of the logic required, between its data() method, and its event bind, unbind and trigger capabilities. I've simplified the pattern from FlyWeb by also using the model interface to also serve as a controller interface for a Shuffl card - that is, external interactions with a card are by setting, getting and listening for changes to model values.
With this in place, I have refactored the existing Shuffl card plugin code to use browser DOM accessing methods only for rendering the display, and otherwise top use the model as the primary interface for a card. The biggest changes have been that user input is recorded by updating the model, with listeners reflecting those changes to the browser DOM, and card serialization is now performed entirely from the model, where previously it involved reading values from the DOM.
This allows test code to be written that updates the model and looks for appropriate changes in other model values and/or in the browser DOM. Code that is not covered by this testing regime is reduced to those areas that respond to user inputs and generate appropriate callbacks, and generation of graphical displays; so far, such untested logic is almost entirely within off-the-shelf jQuery plugins.
Making these changes, and in particular adding new test cases for card interactions, has allowed me to discover and correct a number of bugs in the existing code. I count this a win, even though I spent 2.3 days on an activity budgeted for just half a day. The overrun is mostly caused by retrofitting the new MVC-derived pattern to existing widgets, and creating additional test cases for these.

Tuesday 22 September 2009

Accessing files from browser-based Javascript

Designing a user-friendly mechanism for accessing files from Javascript has proven somewhat elusive. My original plan was to create a temporary form in a card using an <input type="file"> element to upload the file to the AtomPub server. Unfortunately, I could find no way to control the way that data is uploaded, and I would be forced to bypass the AtomPub module, leading to problems and fragility when I implement support for different back-ends. Next, I thought to use an <input type="file"> element to allow me to browse the file system and select a file, then to some behind-the-scenes mapping from the selected file path to the corresponding AtomPub URI, so that the Javascript code can read the file via an Ajax GET request. This approach was foiled by the browsers refusal to let me access the full path name of the file: only the file name is returned from the form (in Firefox, at least). A little Googling suggests this is a browser security "feature" (though I have to say that it smells like security by obscurity). So my final solution is to simply ask the user to enter the URI of the file - I separate the base URI of the AtomPub-served file area from the rest of the filename so that a reasonable default can be provided. It's not as user-friendly as I'd like, but I'm not prepared to invest a lot of time in this at the moment, so I'm leaving it at that. I'm hoping that I may come across a better solution in due course. Fortunately, my experience with research users is that they're prepared to deal with a little clunkiness if it doesn't require too much additional thought and it helps them to get a result they want. I'm writing this up in the hope that someone else may be able to offer a solution. So far, the most likely option seems to be to upload to a temporary file area in the AtomPub server domain and return the URI thus allocated, but that would force me to make too many assumptions about the back-end service than I'm comfortable with right now.

Sunday 20 September 2009

Shuffl: basic workspace persistence using AtomPub works

I've now got Shuffl persistence working against eXist. The UI is a bit clunky (well, a lot clunky really), and the error handling is poor, but it does seem to work. I must properly document how to set it up.

Roughly:

  1. install eXist
  2. configure eXist/Jetty to serve static files from some location. (See: http://shuffl-announce.blogspot.com/2009/08/exist-and-jetty-configuration.html)
  3. copy the shuffl files to that static location (I just link to my Eclipse workspace, so short-circuit this bit)
  4. start eXist
  5. Run the demo application from file shuffl-demo.xhtml (e.g. browse to http://localhost:8080/exist/shuffl/static/demo/shuffl-demo.xhtml).
  6. Use the shuffl menu (click on the logo) "Save as new workspace...", and change the URI in the dialog to something like "http://localhost:8080/exist/atom/edit/shuffl-test/" and click OK. Note the location at the bottom of the window changes to something like "http://localhost:8080/exist/atom/edit/shuffl-test/shuffl-test.json" - any filename in the URI entered is not used.
  7. Make changes to the UI and use the menu option "Save workspace"

Later, to restore the saved data, start shuffl then use menu option "Open workspace...", and change the Atom feed path to the value used previously.

Not yet implemented:

  • saving card size
  • delete cards
  • just about anything else you want to do

It's a small WIN, but I do count this as a WIN. It was a bit of a struggle getting to this point.

Wednesday 16 September 2009

Sprint 4 plan

The plan for sprint 4 has been prepared.
This will be a 3 week / 12 day sprint.
Focus for this sprint will be (a) to complete the workspace persistence via AtomPub, and (b) to start looking at card collections and structures to support the visualization use case raised in discussion with Chris Holland. I'll probably focus on that to the exclusion of planned basic user authentication during this sprint, since the user emphasis here is on visualization rather than sharing/publication.
I have also configured this announcement blog so that postings are sent automatically the the project discussion group. (It came up in discussion with Ross Gardler that there is a tension here between JISC reporting requirements that require a syndication feed of tagged project announcements, and common open source practice of using a mailing list for project announcements and discussions. I think this approach, if it works as hoped, nicely meets these requirements, and provides a place to view announcements that us uncluttered by chit-chat.)

Initial SWOT analysis posted

An initial SWOT analysis for the Shuffl project has been posted here.
It is far from complete, and I intend to use this as a place to collect additional notes as I go along.

Tuesday 15 September 2009

Sprint 3 review

Sprint 3 is finished, and progress notes can be viewed here.

The main points to note are that I've had to re-assess my plan for implementing workspace persistence, and have backed up to implement a new layer of test cases. Progress has been made, but workspace saving/persistence is still not yet complete.

The other main feature of the sprint has been lots of meetings: the JISCRI meeting, the London linked web data, and a VoCamp in Bristol. All of these have contributed, in various ways, to my thoughts for taking Shuffl to users, and have provided opportunities to discuss ideas and use-cases.

I've tagged this progress note as a FAIL as the project is slipping compared with plan: AtomPub has proved rather more of a handful to master than expected. Progress is being made, but more slowly than hoped. I'm hoping to be able to get back to user-visible functionality before too much longer.

Wednesday 9 September 2009

DELETE on AtomPub media resource

I read somewhere that deleting an AtomPub media resource associated with a feed item (using HTTP DELETE) should also delete the feed item. It turns out that the eXist AtomPub implementation does not support this, claiming (erroneously) that the resource does not exist.
Examining the AtomPub spec provides little support (though on first reading I actually thought this did support what I want to do):
To delete a Member Resource, a client sends a 
DELETE request to its Member URI, as specified 
in [RFC2616].  The deletion of a Media Link
Entry SHOULD result in the deletion of the 
corresponding Media Resource.
-- RFC 5023, section 9.4
This is unfortunate for Shuffl, as I had been hoping to not get too bogged down in feed items vs resources. Now I might need to include a feed item URI along with card data if I'm to be able to delete the card data. Or maybe I'll just delete the feed and recreate it without the missing data?
More generally, I am beginning to wonder if AtomPub was the best choice for persisting Shuffl data. I estimate the complexities of dealing with AtomPub have cost me a week or so in slippage, and that I would be better off with WebDAV. I'll stick with it for now, but I'm thinking about a plug-in framework for Shuffl back-end support so I can support alternative protocols.

Wednesday 2 September 2009

Relearning the agile lessons

My coding has been getting a bit bogged down recently, and I've finally stepped back a little to try and figure out why.
One of the lessons of agile development is that things go more smoothly if one proceeds by small steps. I realize I've been trying to implement too much at once, and as a result the code has been getting away from me.
So how did this come about? In-browser unit testing can take quite a bit of organization to put in place, and I've been so flushed by the ease of throwing stuff together with jQuery that I've neglected to create sufficient unit tests, and instead relying upon manual operation testing of the user interface. For the highly-visible user interface functions, and proceeding in small steps, I was getting away with this. But my current task is to save a shuffl workspace using AtomPub. The AtomPub handlers I've developed do have fairly good unit test coverage, but the additional logic needed to assemble the workspace to be saved is also rather complex, and it is here that I'm getting bogged down.
So I'm suspending that line of development, and reorganizing the existing code into modules around which I can put in place some decent unit tests. Strangely, I now feel that things are once again moving forward at a respectable pace! This is all work that needs to be done sooner or later, and sooner seems best.

jQuery with Firefox gotcha: use .html, not .xhtml

I've been tearing hair out over this error message from Firefox when trying to run my test cases from the local file system.
uncaught exception: [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIDOMNSHTMLElement.innerHTML]" nsresult: "0x80004003 (NS_ERROR_INVALID_POINTER)" location: "JS frame :: file:///Users/graham/workspace/googlecode_shuffl/jQuery/js/jquery-1.3.2.js :: anonymous :: line 911" data: no]
Line 0
According to a random page I found on the web, it turns out to be caused by a subtle bug in Firefox and XHTML parsing. The file's DTD declaration calls for transitional, loose XHTML. The file was being served with a .xhtml extension. Changing the file extension to just .html solves the problem.