JSON for linked data - migrating to RDF

Shuffl [1] is the tool we're planning to use to annotate ADMIRAL [9] datasets and submissions.  For ease of implementation Shuffl was initially written to use JSON data formats for persisting data.  It was always my intention that Shuffl should use RDF, and I've recently being working through some of the changes required to support this for ADMIRAL.

Recently, my attention was drawn to a blog post [2] by Sandro Hawke [8] titled From JSON to RDF in Six Easy Steps with JRON, which starts out with the observation "Sometimes, if you stand in the right place and squint, JSON and RDF line up perfectly". It goes on to describe a strategy for representing RDF in JSON that aims to retain the convenience of JSON while being capable of representing full gamut of valid RDF.  It happens that the format described is fundamentally similar to the format I've been using to serialize Shuffl data.  This leads me to consider that JRON could be a great way to make RDF programmer-friendly, and provide a migration path for all the web applications that offer and consume JSON data serialization to participate in the web of linked data.

I've also been experimenting with Jeni Tennison's rdfquery [3] plugin for jQuery [4]. This was a natural choice for use with Shuffl, which is also extensively based on jQuery. It turns out that rdfquery is a gem of a library that seems to be not widely used.  It cleverly manages to bridge the conceptual gap between jQuery-style document access and SPARQL-style dataset queries.

My most recent development has been to implement a JRON-to-rdfquery mapping [5][6][7] as a jQuery plugin.  The implementation is mostly-complete with respect to the JRON specification [2] - for more details of its status see [6].  In the process of creating this implementation, a couple of issues have been raised concerning the design of JRON:

• representing RDF statements about multiple "root" subjects, and
• handling "bare" URIs when a default (i.e. blank) prefix is defined

The first of these is not yet resolved, though I'm in discussion with Sandro.  I've resolved the second by introducing an N3-like option to wrap full URIs in angle brackets.

So How does this relate to migrating a JSON application? My migration plan for Shuffl, as of late July 2010 is in two phases, the first of which is essentially a fairly complex refactoring:

• create a plugin framework for Shuffl to use alternative data serialization formats (done).
• create a plugin to support the current JSON serialization (done)
• modify Shuffl to use the plugin framework (not done)

The second phase introduces the new RDF-related functionality into Shuffl:

• implement a mapping between JRON [2] and rdfquery databank [3] (done)
• modify Shuffl to use a JRON-compatible JSON format (not done)
• implement a serialization plugin to map Shuffl/JRON to RDF, via an rdfquery databank (not done)

at which point, Shuffl should be capable of reading and writing both JRON and RDF/XML

My original plan was to use this approach to convert Shuffl to using mainly RDF, but looking back I can see some attraction in promoting JRON as a first-class citizen in the linked data world.  Compared with RDF, it's really easy to read and write for common data objects.  The existence of relatively simple syntax-transformation software to map between JRON and "proper" RDF shows that interoperability need not be hard to achieve  (my implementation took about 3-4 days, and much of that was building up the test suite). And a format for RDF that is close to "native" JSON means that it is much easier to write simple browser mashup-style applications that can consume and do cool things with this data.

[1] http://code.google.com/p/shuffl/

[2] http://decentralyze.com/2010/06/04/from-json-to-rdf-in-six-easy-steps-with-jron/

[3] http://code.google.com/p/rdfquery/

[4] http://jquery.com/

[5] http://code.google.com/p/shuffl/wiki/JRON_implementation_notes (early design notes)

[6] http://code.google.com/p/shuffl/source/browse/trunk/docs/JRON-for-rdfquery-implementation-notes.txt (post-implementation notes)

[7] http://code.google.com/p/shuffl/source/browse/trunk/src/rdfquery.jron.js (jQuery plugin source)
http://code.google.com/p/shuffl/source/browse/trunk/src/test/test-rdfquery.jron.js (test suite)
http://code.google.com/p/shuffl/source/browse/trunk/static/test/test-rdfquery.jron.html
(test runner - references some other files from http://code.google.com/p/shuffl/source/browse/)

[8] http://www.w3.org/People/Sandro/

[9] http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL

File browsing interfaces working

The previous post hinted at a more usable interface for file selection for selecting workspace and data files.

The additional work has now been completed to provide a point-and-click style interface when opening files - as when opening a new workspace or loading a data file into a worksheet data card - when the underlying storage is accessed using webDAV.  I feel this is an important step forward for Shuffl, and paves the way for building some properly usable applications on the underlying framework.

There do remain some usability problems that need to be resolved, such as improved diagnostics in the face of I/O errors, providing some mechanism to remove cards from a workspace, and improving the back-end storage handler configuration, but these are relatively modest developments compared with the file-browsing capabilities.

For testing, I have been using XAMPP with mod_dav enabled.  A relatively modest subset of DAV capabilities is being used.

Shuffl WebDAV test cases all running (WIN)

We have successfully implemented a WebDAV storage module for Shuffl that supports all storage interface functions, including listing the contents of a collection.

This paves the way for a more user-friedly interface for loading and saving workspaces.

Sustaining Shuffl: the view ahead

Sustainability and related matters

This posting constitutes my reflections at the end of the Shuffl JISCRI project about matters relating to open source sustainability and near-term future development plans.

Sustainability plan updated

The sustainability plan at http://code.google.com/p/shuffl/wiki/OpenSourceSustainabilityPlan has been updated to reflect my views on the sustainability issues having reached the end of the project. This is not a fully-fledged, long-term sustainability plan, but I think it does indicate a direction of travel, and I feel this covers as much detail as is meaningful at this stage.

Due to funding of the ADMIRAL project (http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL), a level of continuing Shuffl development is expected to continue until 2011, during which time we intend to flesh out some of the functionality and focus more on the original goals of data curation.

CLA adopted

An Individual Contributor Licence Agreement, based on one approved by Oxford University for the Simal project, has been adopted for Shuffl: http://code.google.com/p/shuffl/wiki/Individual_Contributor_License_Agreement.

At some stage, we may wish to consider reviewing this if responsibility for actually publishing the software is transferred to an organization or person other than University of Oxford.

Changing requirements

During the course of this Rapid Innovation project, the requirements focus has shifted from data acquisition and curation to data visualization, in response to comments from a potential research user. This has seemed the right thing to do for a number of reasons, mainly to do with maximizing the potential for early user engagement, but it has meant that some of the original headline goals have been sidelined.

I have also not yet made direct contact with other research users, to gather their (possibly different) view of requirements. Jun Zhao and I have arranged to visit Helen White-Cooper in the very near future to resume our discussions with her about data habdling requirements and tools, at which time I plan to demonstrate Shuffl, and discuss with her the possibilities for an image annotation version, and other possible applications.

In its current state, Shuffl does appear to be a solid base for development of new functionality, with many (though not all) of the fundamental architectural features having been tested and stabilized.

Active deployment and ADMIRAL environment

As noted above, it is our intention that development the data curation aspects of Shuffl will be intensified for the ADMIRAL project, as a number of the work packages have been based on developments of Shuffl.

To this end, we will continue to work with Chris Holland, as well as other researchers, to create a deployed system including Shuffl that they will use on a day-to-day basis.

For ADMIRAL, we have the advantage that we will be creating an infrastructure service that can be enhanced with features we can use to overcome some of the usability limitations of the current browser-only version of Shuffl. For example, we will be able to provide an HTTP request proxy service so that data can be read from and written to servers other than the one from which the Shuffl application has been loaded. We can also tailor the storage services to better support file browsing for locating data files and Shuffl workspaces. We can also look to host useful data visualization and handling services,

Not YAAS (Yet Another Application Server)?

It has been noted on the Shuffl discussion list that some of the data visualization work, and the porting of a FlyWeb widget, might be seen as creating just another application server running in the browser. Such a direction could leave behind some more distinctive aspects of the original Shuffl concept, and end up reinventing facilities that are provided by other developments such as W3C widgets served by a Wookie widget server. Scott Wilson implemented an early proof-of-concept of a Wookie widget running in a Shuffl card, an idea that has so far not been followed through. One reason for this was a reluctance to get involved in implememting specialized server-side support as part of the original Shuffl project.

With the ADMIRAL project, we can review the option of using a Wookie server as part of the data sharing infrastructure to serve up standard data visualization widgets, rather than re-implementing them as Shuffl cards. We would need to understand enough about the Wookie API to hook it into the model API used by Shuffl to link information across cards. In this way, we may be able to reap the advantages of data visualization for user engagement without having to put effort into implementing them.

Shuffl sprint 8 - progress review

The updated sprint plan is at http://code.google.com/p/shuffl/wiki/SprintPlan_8.

This was the final sprint, so effort was largely directed mainly towards leaving the project in a clean state for further work.

The demonstration application running from Google Code SVN has been enhanced to represent the current state of development, and the underlying framework has been significantly improved and stabilized. While the delivered functionality is less than had been intended, the project is in fairly good shape for ongoing development as part of the Shuffl project, and the basic ideas have been somewhat vindicated.

I did not make time to install a copy of the system for Chris Holland, but still hope to do this soon. We will be continuing to work with Chris in the ADMIRAL project.

See also the final progress report at http://code.google.com/p/shuffl/wiki/20091130_JISCRI_ProjectFinalProgressReport.

Shuffl final progress report

The final progress report for the Shuffl JISCRI project has been published in the project wiki, at 20091130_JISCRI_ProjectFinalProgressReport.

Shuffl sprint 8 plan

The plan for sprint 8 has been posted at http://code.google.com/p/shuffl/wiki/SprintPlan_8.

This will be the final sprint to be conducted under the initial Shuffl JISCRI project umbrella, but work on Shuffl will continue as part of our ADMIRAL project.

The main goals for this sprint are:

• to complete the back-end storage interface work started in the previous sprint
• to improve the online prototype demonstrator for project evaluation (mainly, create a new Shuffl workspace with instructions displayed for doing the various things that Shuffl can do). Also, notes for installing eXist and Shuffl on a new system
• install a copy of Shuffl and eXist on a machine our research user's workgroup
• final reporting and project wrap-up, including tidying up some matters to do with project sustainability

Shuffl sprint 7 - progress review

The updated sprint plan is at http://code.google.com/p/shuffl/wiki/SprintPlan_7.

The high point of this sprint was the enthusiastic reception of the visualization interface, even though this has been completed at the expense of some of the other more curation-oriented features. If this truly helps us achieve better engagement for the ADMIRAL project, I judge this will have been a good trade-off, but some cautionary notes raised on the discussion group about maintaining the right project focus need to be borne in mind. The mechanisms for working directly with spreadsheet data went some way to reducing the pressure for some of the features not yet implemented. I have agreed with Chris Holland to install a copy of Shuffl on a system where he can use it directly with his own data, which will hopefully provide a powerful point of engagement for continuing work on ADMIRAL.

It may be worth noting that I don't feel the focus on visualization has been entirely at the expense of the original goals of Shuffl; i.e., to provide a lightweight tool for capturing and sharing annotations and data. Many of the fundamental capabilities have been demonstrated, but in different combinations: user-editable semi-structured data, card linking, and a flexible, pluggable framework for introducing new card structures. On the down side, some of the intended work on containers (e.g. stacks of cards) has not been addressed, and the card serialization format currently deployed is JSON and not RDF.

The testing framework has been extremely valuable. The full test suite now performs in excess of 2000 individual tests (though many of these are repetitious). Areas which have proved more challenging to debug have been exactly those parts of the user interaction code that are not covered by unit tests. I have resisted taking time to implement a UI test framework (e.g. based on Selenium), but rather have tried to move logic out of the user interaction code into unit-testable functions. This is a debatable strategy, but in the limited time available I didn't feel the benefits of deploying a full UI test suite would get me further forward. When I get time, I'd like to evaluate the Windmill framework (http://www.getwindmill.com/), as my past experience with Selenium has been somewhat mixed.

With work on Shuffl planned to continue as part of the ADMIRAL project, I feel my top priority is to implement as much as possible of the features desired by the actively engaged researcher - which is to improve the interface for saving and loading workspaces. Other than that, I need to continue the steps taken to promote sustainability of the outputs, including creation of a more approachable demonstration prototype. These two strands of effort would ideally come together in a back-end storage plugin that works with the Google Data API - if the opportunity presents, I'd rather like to do a mini-hackathon with someone who is familiar with the details of the Google Data API.

"I can really see myself using this"

Big win !!!

As the Shuffl project draws to a close, I have been having some doubts about the amount of progress made, and have been asking myself whether it was the right decision to spend effort on data visualization within Shuffl (which has been time consuming). But I've just had a real boost.

During a brief demonstration to Chris Holland, I showed him his own spreadsheet data loaded and plotted as graphs in Shuffl, eliciting the response:

"I can really see myself using this"

Vindication indeed!

Chris described the ability to quickly draw up plots and add annotations without having to use four different programs as a real winner for him. Working with Chris' own data, I have seen the need for and implemented (a) selecting data blocks from within a worksheet, and (b) supporting a mixture of linear and logarithmic data plots, both of which I believe to have been contributory to Chris' response.

We have agreed that:

• Label editing (requested previously) is not an immediate priority
• Plot colour selection would still be nice
• I shall focus my remaining efforts on improving the user interface for workspace saving and loading (which will require some reworking of the storage interface, but will in any case prepare the ground for continuing Shuffl work in the ADMIRAL project), and
• I shall arrange to install a copy of eXist on a computer accessible to Chris so he can try using Shuffl in his own environment.

The main new feature that Chris requested during the demonstration was the ability to print a Shuffl workspace, or save it as an image for incorporation into a paper or document. I think this is a reasonable and do-able goal, but it won't be implemented within the current project. Maybe as part of ADMIRAL? For now, there is screen capture and printing.

There is also an interaction here with discussions I've had with Scott Wilson (Wookie Widgets) and Ross Gardler (OSS Watch), in which I have been wisely cautioned against trying to make Shuffl into a generic application server. If data visualization is to prove a draw for engaging with researchers then, looking forward to the ADMIRAL project, I think I should look seriously into the possibility of incorporating a Wookie server into the planned ADMIRAL Data Store server (LSDS). I suspect that a real win here would be if Wookie can serve a widget for displaying raw spreadsheet content (by "raw", I mean here without export to CSV format). I look forward to more interesting discussion and exploration.

Shuffl sprint 7 plan

The plan for Sprint 7 has been posted at http://code.google.com/p/shuffl/wiki/SprintPlan_7.

The main targets for this sprint will to continue work not completed in the previous sprint:

• Visualization of data: after getting feedback from Chris Holland, and to obtaining some representative sample data, I have a number of user interface elements to complete.
• Improve error handling when loading/saving workspaces.
• Improve usability of the interface for importing data and loading/saving workspaces.

Shuffl sprint 6 progress review

Progress during this sprint has been fairly poor, largely due to distraction from both personal and non-project affairs. The total amount of effort spent was 3.5 days against planned effort of 8 days. A further factor impacting progress was that reorganizing the code to enable graph label-row selection in a data table took about a day longer than expected.

On the positive side, I did hold a second review meeting with Chris Holland, and the current development is being conducted very much in response to his feedback, to make the data graphing display more useful to him.

Also on the positive side, the mechanisms for linking data between cards seem to be working nicely (e.g. when I reload new data into a table card, or change the label row, an associated graph card updates immediately; but I do need to capture this relationship when I save and restore a workspace).

Some thoughts about sprint planning: for this sprint, having fewer than 4 days actual effort expended, the sprint planning and review process seems to lack sufficient data to be meaningful. In setting sprint duration, it would seem reasonable to take account of the total amount of effort being applied rather than simply the number of elapsed days. I don't plan to change the duration of the remaining two sprints for this project, but for future projects, planning sprints with fewer than 10-15 total days of effort may be something to avoid.

Shuffl sprint 6 plan

The plan for Sprint 6 has been posted at http://code.google.com/p/shuffl/wiki/SprintPlan_6.

The main targets for this sprint will be to work towards improved usability in two areas:

• Visualization of data: after demonstrating the data graphing interface to Chris Holland, I hope to find out what he really wants, and to get some representative sample data.
• Improve usability of the interface for importing data and loading/saving workspaces.

Shuffl sprint 5 progress review

The plan for sprint 5 has been updated with progress and review notes:

• http://code.google.com/p/shuffl/wiki/SprintPlan_5

This was a sprint of highs and lows - there were significant wins, but also more distractions, and the direction of progress seems to be diverging more from the original plan. A user demonstration session was postponed, to take place early in the next sprint. I have also spent a fair amount of time cleaning up existing code in various ways.

On the positive side, a demonstrable drag-and-drop data graphing interface has been completed (as planned), and I worked with Jun Zhao to successfully implement a card containing a FlyUI gene finder widget (not planned).

I have started to look at making the data access modules more robust - currently, an I/O error (such as accessing a non-existent web resource) can leave the application broken.

Encouragingly, plans are now being laid for projects that will continue aspects of Shuffl beyond the current Rapid Innovation project.

Task completion velocity is down significantly from the previous sprint (tasks completed/effort spent: 4.5/7.4 = 0.61), which reflects the various distractions from the intended development plan.

FlyUI GeneFinder widget in Shuffl card

I recently spent a day working with my colleague Jun Zhao to successfully implement a Shuffl card containing a FlyUI (http://code.google.com/p/flyui/) gene finder widget.

This achievement is an important confidence builder, as it further demonstrates the power and flexibility of the Shuffl card plugin model: we were able to embed quite complex FlyUI widget logic unmodified into a Shuffl card. The FlyUI widgets are built using the Yahoo YUI libraries, so this success also demonstrates that Shuffl (based on jQuery) can coexist with YUI code.

The remaining challenge is to find easy ways to deploy such applications. To overcome the Browsers' "same origin" restriction on Ajax calls, the Shuffl+GeneFinder application was deployed via an Apache server configured to redirect and proxy requests to the SPARQL endpoint for FlyBase data hosted at OpenFlydata.org. In this, Shuffl is playing out quite nicely as a flexible user interface toolkit for application developers, but it is not yet clear how to deploy Shuffl as a general purpose, user-configurable data viewing and annotation tool. Maybe we should think about deploying a Shuffl application server? Could Google Apps, or similar, be recruited for this purpose?

Shuffl drag-and-drop data visualization

An initial cut of data visualization in Shuffl is working. A "data table card" card can be dropped on a "data graph card" to display graphs of the data in the table. Thanks to jQuery, the drag-and-drop interface was a breeze to code, and the model pattern discussed previously makes it very easy to link the UI functions to underlying data logic. I'm using a jQuery plugin called 'flot' (http://code.google.com/p/flot/) to do the actual graphing.

Instructions for running the demonstration are at http://code.google.com/p/shuffl/wiki/Shuffl_Demonstration.

I intend to demonstrate this to one of my research collaborators to help determine how this should be carried forward.

I've also updated the project front-page (http://code.google.com/p/shuffl/) to reflect the current state of the project, and added a screenshot and updated links to the demonstration application and Shuffl-based brief introduction.

MVC pattern and mock widgets

I've just discovered that the MVC approach I've adopted (http://shuffl-announce.blogspot.com/2009/09/framework-for-testing-shuffl-card.html) makes it really easy to create mock shuffl cards (in the sense of mock objects: http://en.wikipedia.org/wiki/Mock_object) for testing.

Here's one I used today to test the drag-and-drop drop-target logic in a data visualization card:

// Instantiate mock table card

var tc = jQuery("<div/>");

tc.model('shuffl:labels',carddatagraph_labels1);

tc.model('shuffl:series',carddatagraph_series1);

The jQuery.model plugin I mentioned previously provides all the additional logic needed to ensure the mock card object responds as required to requests for information.

JISCRI reporting tags

I've decided to vary slightly the tags I use for reporting-via-blog. Updated details are noted at http://code.google.com/p/shuffl/wiki/ProjectPlanOutline_200906_200911.

In summary:

• some progress reports are neither WINs nor FAILs, just reports.
• and there was no tag set for ongoing planning, which I think is required for an agile project structure - goals change and new plans are created for each sprint/iteration - none of the existing proposed tag sets really fit this.

Shuffl sprint 5 plan

The plan for Sprint 5 has been posted at http://code.google.com/p/shuffl/wiki/SprintPlan_5.

Responding to research user desiderata, project activities are now diverging quite significantly from the original outline plan (other than user engagement still being a key activity at this stage).

The primary focus for this sprint will be to complete the initial data graphing display for the visualization use case, to demonstrate this to the researcher who requested this, and use that session to gather more detailed requirements for this aspect of Shuffl.

Secondary foci will be: improving aspects of the user interface, especially for workspace loading/saving, and creating some user documentation.

I also have an ongoing discussion with Ross Gardler about long-term sustainability and CLAs (contributor licence agreements) - see http://code.google.com/p/shuffl/issues/detail?id=9 for details.

Shuffl sprint 4 progress review

The plan for sprint 4 has been updated with progress and review notes:

• http://code.google.com/p/shuffl/wiki/SprintPlan_4

In outline, the sprint activity was:

• week 1: complete workspace persistence (initial cut)
• week 2: UI for data access; expand testing framework; card refactoring to MVC pattern
• week 3: tabular data loading and display; started on data graphing

The sprint has been characterized by good progress and technical compromises. Many of the completed tasks are not implemented completely as I would like or had originally intended, so I feel I may be accumulating some "technical debt" that will need to be addressed before the project can be truly usable by unsupported researchers and others. The user interface for saving/restoring workspaces is a case in point. See http://code.google.com/ p/shuffl/wiki/ TechnicalDebt for more details.

There has been less community engagement than planned, though some new use-cases have been noted (FlyKit, e-learning).

For the first time, I have calculated a "velocity"figure (planned effort for completed tasks/total sprint effort): 11.0/11.4 = 0.96. This figure was a surprise to me: I was expecting it to be lower on account of the unplanned time spent on admin, etc. Total sprint effort here does not include the small amount of unplanned additional time spent on other projects. Prediction of progress remains challenging with so many competing demands on limited time.

Shuffl data visualization use-case updated

I now have permission to publicly post my full notes from a meeting with Chris Holland, a researcher for whom I am targetting an application of Shuffl (and one of the supporters of our bid). Notes of our meeting are here:

• http://code.google.com/p/shuffl/wiki/20090902_UseCase_DataVisualization