Prep notes for NDF2011 demonstration

I didn't really have a presentation for my demonstration at the NDF, but the event team have asked for presentations, so here are the notes for my practice demonstration that I did within the library. The notes served as an advert to attract punters to the demo; as a conversation starter in the actual demo and as a set of bookmarks of the URLs I wanted to open.

---

Depending on what people are interested in, I'll be doing three things

*) Demonstrating basic editing, perhaps by creating a page from the requested articles at http://en.wikipedia.org/wiki/Wikipedia:WikiProject_New_Zealand/Requested_articles

*) Discussing some of the quality control processes I've been involved with (http://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion and http://en.wikipedia.org/wiki/New_pages_patrol)

*) Discussing how wikipedia handles authority control issues using redirects (https://secure.wikimedia.org/wikipedia/en/wiki/Wikipedia:Redirect ) and disambiguation (https://secure.wikimedia.org/wikipedia/en/wiki/Wikipedia:Disambiguation )

I'm also open to suggestions of other things to talk about.

Metadata vocabularies LODLAM NZ cares about

At today's LODLAM NZ, in Wellington, I co-hosted a vocabulary schema / interoperability session. I kicked off the session with a list of the metadata schema we care about and counts of how many people in the room cared about it. Here are the results:

8 Library of Congress / NACO Name Authority List
7 Māori Subject Headings
6 Library of Congress Subject Headings
5 SONZ
5 Linnean
4 Getty Thesauri
3 Marsden Research Subject Codes / ANZRSC Codes
3 SCOT
3 Iwi Hapū List
2 Australian Pictorial Thesaurus
1 Powerhouse Object Names Thesaurus
0 MESH

This straw poll naturally only reflects on the participants who attended this particular session and counting was somewhat haphazard (people were still coming into the room), but is gives a sample of the scope.

I don't recall whether the heading was "Metadata we care about" or "Vocabularies we care about," but it was something very close to that.

Unexpected advice

During the NDF2011 today I was in "Digital initiatives in Māori communities" put on the the talented Honiana Love and Claire Hall from the Te Reo o Taranaki Charitable Trust about their work on He Kete Kōrero. At the end I asked a question "Most of us [the audience] are in institutions with te Reo Māori holdings or cultural objects of some description. What small thing can we do to help enable our collections for the iwi and hapū source communities? Use Māori Subject Headings? The Iwi / Hapū list? Geotagging? ..." Quick-as-a-blink the response was "Geotagging." If I understood the answer (given mainly by Honiana) correctly, the point was that geotagging is much more useful because it's much more likely to be done right in contexts like this. Presumably because geotagging lends itself to checking, validation and visualisations that make errors easy to spot in ways that these other metadata forms don't; it's better understood by those processing the documents and processing the data.

I think it's fabulous that we're getting feedback from indigenous groups using information systems in indigenous contexts, particularly feedback about previous attempts to cater to their needs. If this is the experience of other indigenous groups, it's really important.

Goodbye 'social-media' world

You may or may not have noticed, but recently a number of 'social media' services have begun looking and working very similarly. Facebook is the poster-child, followed by google+ and twitter. Their modus operandi is to entice you to interact with family-members, friends and acquaintances and then leverage your interactions to both sell your attention advertisers and entice other members of you social circle to join the service.

There are, naturally, a number of shiny baubles you get for participating it the sale of your eyeballs to the highest bidder, but recently I have come to the conclusion that my eyeballs (and those of my friends, loved ones and colleagues) are worth more.

I'll be signing off google plus, twitter and facebook shortly. I my return for particular events, particularly those with a critical mass the size of Jupiter, but I shall not be using them regularly. I remain serenely confident that all babies born in my extended circle are cute, I do not need to see their pictures.

I will continue using other social media as before (email, wikipedia, irc, skype, etc) as usual. My deepest apologies to those who joined at least party on my account.

Recreational authority control

Over the last week or two I've been having a bit of a play with Ngā Ūpoko Tukutuku / The Māori Subject Headings (for the uninitiated, think of the widely used Library of Congress Subject Headings, done Post-Colonial and bi-lingually but in the same technology) the main thing I've been doing is trying to munge the MSH into Wikipedia (Wikipedia being my addiction du jour).

My thinking has been to increase the use of MSH by taking it, as it were, to where the people are. I've been working with the English language Wikipedia, since the Māori language Wikipedia has fewer pages and sees much less use.

My first step was to download the MSH in MARC XML format (available from the website) and use XSL to transform it into a wikipedia table (warning: large page). When looking at that table, each row is a subject heading, with the first column being the the te reo Māori term, the second being permutations of the related terms and the third being the scope notes. I started a discussion about my thoughts (warning: large page) and got a clear green light to create redirects (or 'related terms' in librarian speak) for MSH terms which are culturally-specific to Māori culture.

I'm about 50% of the way through the 1300 terms of the MSH and have 115 redirects in the newly created Category:Redirects from Māori language terms. That may sound pretty average, until you remember that institutions are increasingly rolling out tools such as Summon, which use wikipedia redirects for auto-completion, taking these mappings to the heart of most Māori speakers in higher and further education.

I don't have a time-frame for the redirects to appear, but they haven't appeared in Otago's Summon, whereas redirects I created ~ two years ago have; type 'jack yeates' and pause to see it at work.

Thoughts on "Letter about the TEI" from Martin Mueller

Thoughts on "Letter about the TEI" from Martin Mueller

Note: I am a member of the TEI council, but this message is should be read as personal position at the time of writing, not a council position, nor the position of my employer.

Reading Martin's missive was painful. I should have responded earlier, I think perhaps I was hoping someone else could say what I wanted to say and I could just say "me too." They haven't so I've become the someone else.

I don't think that Martin's "fairly radical model" is nearly radical enough. I'd like to propose a significantly more radical model as strawman:

---

1) The TEI shall maintain a document called the 'The TEI Principals.' The purpose of The TEI is to advance The TEI Principals.

2) Institutional membership of The TEI is open to groups which publish, collect and/or curate documents in formats released by The TEI. Institutional membership requires members acknowledge The TEI Principals and permits the members to be listed at http://www.tei-c.org/Activities/Projects/ and use The TEI logos and branding.

3) Individual membership of The TEI is open to individuals; individual membership requires members acknowledge The TEI Principals and subscribe to The TEI mailing list at http://listserv.brown.edu/?A0=TEI-L.

4) All business of The TEI is conducted in public. Business which needs be conducted in private (for example employment matters, contract negotiation, etc) shall be considered out of scope for The TEI.

5) Changes to the structure of The TEI will be discussed on the TEI mailing list and put to a democratic vote with a voting period of at least one month, a two-thirds majority of votes cast is required to pass a motion, which shall be in English.

6) Groups of members may form for activities from time-to-time, such as members meetings, summer schools, promotions of The TEI or collective digitisation efforts, but these groups are not The TEI, even if the word 'TEI' appears as part of their name.

---

I'll admit that there are a couple of issues not covered here (such as who holds the IPR), but it's only a straw man for discussion. Feel free to fire it as necessary.

unit testing framework for XSL transformations?

I'm part of the TEI community, which maintains an XML standard which is commonly transformed to HTML for presentation (more rarely PDF). The TEI standard is relatively large but relatively well documented, the transformation to HTML has thus far been largely piecemeal (from a software engineering point of view) and not error free.

Recently we've come under pressure to introduce significantly more complexity into transformations, both to produce ePub (which is wrapped HTML bundled with media and metadata files) and HTML5 (which can represent more of the formal semantics in TEI). The software engineer in me sees unit testing the a way to reduce our errors while opening development up to a larger more diverse group of people with a larger more diverse set of features they want to see implemented.

The problem is, that I can't seem to find a decent unit testing framework for XSLT. Does anyone know of one?

Our requirements are: XSLT 2.0; free to use; runnable on our ubuntu build server; testing the transformation with multiple arguments; etc;

We're already using: XSD, RNG, DTD and schematron schemas, epubcheck, xmllint, standard HTML validators, etc. Having the framework drive these too would be useful.

The kinds of things we want to test include:

1. Footnotes appear once and only once
2. Footnotes are referenced in the text and there's a back link from the footnote to the appropriate point in the text
3. Internal references (tables of contents, indexes, etc) point somewhere
4. Language encoding used xml:lang survives from the TEI to the HTML
5. That all the paragraphs in the TEI appear at least once in the HTML
6. That local links work
7. Sanity check tables
8. Internal links within parallel texts
9. ....

Any of many languages could be used to represent these tests, but ideally it should have a DOM library and be able to run that library across entire directories of files. Most of our community speak XML fluently, so leveraging that would be good.

Is there a place for readers' collectives in the bright new world of eBooks?

The transition costs of migrating from the world of books-as-physical-artefacts-of-pulped-tree to the world of books-as-bitstreams are going to be non-trivial.

Current attempts to drive the change (and by implication apportion those costs to other parties) have largely been driven by publishers, distributors and resellers of physical books in combination with the e-commerce and electronics industries which make and market the physical eBook readers on which eBooks are largely read. The e-commerce and electronics industries appear to see traditional publishing as an industry full of lumbering giants unable to compete with the rapid pace of change in the electronics industry and the associated turbulence in business models, and have moved to poach market-share. By-and-large they've been very successful. Amazon and Apple have shipped millions of devices billed as 'eBook readers' and pretty much all best-selling books are available on one platform or another.

This top tier, however, is the easy stuff. It's not surprising that money can be made from the latest bodice-ripping page-turner, but most of the interesting reading and the majority of the units sold are outside the best-seller list, on the so-called 'long tail.'

There's a whole range of books that I'm interested in that don't appear to be on the business plan of any of the current eBook publishers, and I'll miss them if they're not converted:

1. The back catalogue of local poetry. Almost nothing ever gets reprinted, even if the original has a tiny print run and the author goes on to have a wonderfully successful career. Some gets anthologised and a few authors are big enough to have a posthumous collected works, when their work is no longer cutting edge.
2. Some fabulous theses. I'm thinking of things like: http://ir.canterbury.ac.nz/handle/10092/1978, http://victoria.lconz.ac.nz/vwebv/holdingsInfo?bibId=69659 and http://otago.lconz.ac.nz/vwebv/holdingsInfo?bibId=241527
3. Lots of te reo Māori material (pick your local indigenous language if you're reading this outside New Zealand)
4. Local writing by local authors.

Note that all of these are local content---no foreign mega-corporation is going to regard this as their home-turf. Getting these documents from the old world to the new is going to require a local program run by (read funded by) locals.

Would you pay for these things? I would, if it gave me what I wanted.


What is it that readers want?

We're all readers, of one kind or another, and we all want a different range of things, but I believe that what readers want / expect out of the digital transition is:

1. To genuinely own books. Not to own them until they drop their eReader in the bath and lose everything. Not to own them until a company they've never heard of goes bust and turns off a DRM server they've never heard of. Not to own them until technology moves on and some new format is in use. To own them in a manner which enables them to use them for at least their entire lifetime. To own them in a manner that poses at least a question for their heirs.
2. A choice of quality books. Quality in the broadest sense of the word. Choice in the broadest sense of the word. Universality is a pipe-dream, of course, but with releasing good books faster than I can read them.
3. A quality recommendation service. We all have trusted sources of information about books: friends, acquaintances, librarians or reviewers that history have suggested have similar ideas as us about what a good read is.
   
4. To get some credit for already having bought the book in pulp-of-murdered-tree work. Lots of us have collections of wood-pulp and like to maintain the illusion that in some way that makes us well read.
5. Books bought to their attention based on whether they're worth reading, rather than what publishers have excess stock of. Since the concept of 'stock' largely vanishes with the transition from print to digital this shouldn't be too much of a problem.
6. Confidentially for their reading habits. If you've never come across it, go and read the ALA's The Freedom to Read Statement
   

A not-for-profit readers' collective

It seems to me that the way to manage the transition from the old world to the new is as a not-for-profit readers' collective. By that I mean a subscription-funded system in which readers sign up for a range of works every year. The works are digitised by the collective (the expensive step, paid for up-front), distributed to the subscribers in open file formats such as ePub (very cheap via the internet) and kept in escrow for them (a tiny but perpetual cost, more on this later).

Authors, of course, need to pay their mortgage, and part of the digitisation would be obtaining the rights to the work. Authors of new work would be paid a 'reasonable' sum, based on their statue as authors (I have no idea what the current remuneration of authors is like, so I won't be specific). The collective would acquire (non-exclusive) the rights to digitise the work if not born digital, to edit it, distribute it to collective members and to sell it to non-members internationally (i.e. distribute it through 'conventional' digital book channels). In the case of sale to non-members through conventional digital book channels the author would get a cut. Sane and mutually beneficial deals could be worked out with libraries of various sizes.

Generally speaking, I'd anticipate the rights to digitise and distribute in-copyright but out-of-print poetry would would be fairly cheap; the rights to fabulous old university theses cheaper; and rights to out-of-copyright materials are, of course, free. The cost of rights to new novels / poetry would hugely depend on statue of the author and the quality of the work, which is where the collective would need to either employ a professional editor to make these calls or vote based on sample chapters / poems or some combination of the two. Costs of quality digitisation is non-trivial, but costs are much lower in bulk and dropping all the time. Depending on the platform in use, members of the collective might be recruited as proof-readers for OCR errors.

That leaves the question of how to fund the the escrow. The escrow system stores copies of all the books the collective has digitised for the future use of the collectives' members and is required to give efficacy to the promise that readers really own the books. By being held in escrow, the copies survive the collective going bankrupt, being wound up, or evolving into something completely different, but requires funding. The simplest method of obtaining funding would be to align the collective with another established consumer of local literature and have them underwrite the escrow, a university, major library, or similar.

The difference between a not-for-profit readers' collective and an academic press?

Of hundreds of years, major universities have had academic presses which publish quality content under the universities' auspices. The key difference between the not-for-profit readers' collective I am proposing and an academic press is that the collective would attempt to publish the unpublished and out-of-print books that the members wanted rather than aiming to meet some quality criterion. I acknowledge a popularist bias here, but it's the members who are paying the subscriptions.

Which links in the book chain do we want to cut out?

There are some links in the current book production chain which we need to keep, there are others wouldn't have a serious future in a not-for-profit. Certainly there is a role for judgement in which works to purchase with the collective's money. There is a role for editing, both large-scale and copy-editing. There is a role for illustrating works, be it cover images or icons. I don't believe there is a future for roles directly relating to the production, distribution, accounting for, sale, warehousing or pulping of physical books. There may be a role for the marketing books, depending on the business model (I'd like to think that most of the current marketing expense can be replaced by combination of author-driven promotion and word-of-month promotion, but I've been known to dream). Clearly there is an evolving techie role too.

The role not mentioned above that I'd must like to see cut, of course, is that of the multinational corporation as gatekeeper, holding all the copyrights and clipping tickets (and wings).