Open Source Exile

Prep notes for NDF2011 demonstration

2011-12-02T14:11:00.003+13:00

I didn't really have a presentation for my demonstration at the NDF, but the event team have asked for presentations, so here are the notes for my practice demonstration that I did within the library. The notes served as an advert to attract punters to the demo; as a conversation starter in the actual demo and as a set of bookmarks of the URLs I wanted to open.

Depending on what people are interested in, I'll be doing three things

*) Demonstrating basic editing, perhaps by creating a page from the requested articles at http://en.wikipedia.org/wiki/Wikipedia:WikiProject_New_Zealand/Requested_articles

*) Discussing some of the quality control processes I've been involved with (http://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion and http://en.wikipedia.org/wiki/New_pages_patrol)

*) Discussing how wikipedia handles authority control issues using redirects (https://secure.wikimedia.org/wikipedia/en/wiki/Wikipedia:Redirect ) and disambiguation (https://secure.wikimedia.org/wikipedia/en/wiki/Wikipedia:Disambiguation )

I'm also open to suggestions of other things to talk about.

Metadata vocabularies LODLAM NZ cares about

2011-12-01T20:20:00.004+13:00

At today's LODLAM NZ, in Wellington, I co-hosted a vocabulary schema / interoperability session. I kicked off the session with a list of the metadata schema we care about and counts of how many people in the room cared about it. Here are the results:

8 Library of Congress / NACO Name Authority List
7 Māori Subject Headings
6 Library of Congress Subject Headings
5 SONZ
5 Linnean
4 Getty Thesauri
3 Marsden Research Subject Codes / ANZRSC Codes
3 SCOT
3 Iwi Hapū List
2 Australian Pictorial Thesaurus
1 Powerhouse Object Names Thesaurus
0 MESH

This straw poll naturally only reflects on the participants who attended this particular session and counting was somewhat haphazard (people were still coming into the room), but is gives a sample of the scope.

I don't recall whether the heading was "Metadata we care about" or "Vocabularies we care about," but it was something very close to that.

Unexpected advice

2011-11-30T21:08:00.002+13:00

During the NDF2011 today I was in "Digital initiatives in Māori communities" put on the the talented Honiana Love and Claire Hall from the Te Reo o Taranaki Charitable Trust about their work on He Kete Kōrero. At the end I asked a question "Most of us [the audience] are in institutions with te Reo Māori holdings or cultural objects of some description. What small thing can we do to help enable our collections for the iwi and hapū source communities? Use Māori Subject Headings? The Iwi / Hapū list? Geotagging? ..." Quick-as-a-blink the response was "Geotagging." If I understood the answer (given mainly by Honiana) correctly, the point was that geotagging is much more useful because it's much more likely to be done right in contexts like this. Presumably because geotagging lends itself to checking, validation and visualisations that make errors easy to spot in ways that these other metadata forms don't; it's better understood by those processing the documents and processing the data.

I think it's fabulous that we're getting feedback from indigenous groups using information systems in indigenous contexts, particularly feedback about previous attempts to cater to their needs. If this is the experience of other indigenous groups, it's really important.

Goodbye 'social-media' world

2011-11-26T21:58:00.000+13:00

You may or may not have noticed, but recently a number of 'social media' services have begun looking and working very similarly. Facebook is the poster-child, followed by google+ and twitter. Their modus operandi is to entice you to interact with family-members, friends and acquaintances and then leverage your interactions to both sell your attention advertisers and entice other members of you social circle to join the service.

There are, naturally, a number of shiny baubles you get for participating it the sale of your eyeballs to the highest bidder, but recently I have come to the conclusion that my eyeballs (and those of my friends, loved ones and colleagues) are worth more.

I'll be signing off google plus, twitter and facebook shortly. I my return for particular events, particularly those with a critical mass the size of Jupiter, but I shall not be using them regularly. I remain serenely confident that all babies born in my extended circle are cute, I do not need to see their pictures.

I will continue using other social media as before (email, wikipedia, irc, skype, etc) as usual. My deepest apologies to those who joined at least party on my account.

Recreational authority control

2011-11-06T21:24:00.004+13:00

Over the last week or two I've been having a bit of a play with Ngā Ūpoko Tukutuku / The Māori Subject Headings (for the uninitiated, think of the widely used Library of Congress Subject Headings, done Post-Colonial and bi-lingually but in the same technology) the main thing I've been doing is trying to munge the MSH into Wikipedia (Wikipedia being my addiction du jour).

My thinking has been to increase the use of MSH by taking it, as it were, to where the people are. I've been working with the English language Wikipedia, since the Māori language Wikipedia has fewer pages and sees much less use.

My first step was to download the MSH in MARC XML format (available from the website) and use XSL to transform it into a wikipedia table (warning: large page). When looking at that table, each row is a subject heading, with the first column being the the te reo Māori term, the second being permutations of the related terms and the third being the scope notes. I started a discussion about my thoughts (warning: large page) and got a clear green light to create redirects (or 'related terms' in librarian speak) for MSH terms which are culturally-specific to Māori culture.

I'm about 50% of the way through the 1300 terms of the MSH and have 115 redirects in the newly created Category:Redirects from Māori language terms. That may sound pretty average, until you remember that institutions are increasingly rolling out tools such as Summon, which use wikipedia redirects for auto-completion, taking these mappings to the heart of most Māori speakers in higher and further education.

I don't have a time-frame for the redirects to appear, but they haven't appeared in Otago's Summon, whereas redirects I created ~ two years ago have; type 'jack yeates' and pause to see it at work.

Thoughts on "Letter about the TEI" from Martin Mueller

2011-08-16T19:46:00.003+12:00

Thoughts on "Letter about the TEI" from Martin Mueller

Note: I am a member of the TEI council, but this message is should be read as personal position at the time of writing, not a council position, nor the position of my employer.

Reading Martin's missive was painful. I should have responded earlier, I think perhaps I was hoping someone else could say what I wanted to say and I could just say "me too." They haven't so I've become the someone else.

I don't think that Martin's "fairly radical model" is nearly radical enough. I'd like to propose a significantly more radical model as strawman:

1) The TEI shall maintain a document called the 'The TEI Principals.' The purpose of The TEI is to advance The TEI Principals.

2) Institutional membership of The TEI is open to groups which publish, collect and/or curate documents in formats released by The TEI. Institutional membership requires members acknowledge The TEI Principals and permits the members to be listed at http://www.tei-c.org/Activities/Projects/ and use The TEI logos and branding.

3) Individual membership of The TEI is open to individuals; individual membership requires members acknowledge The TEI Principals and subscribe to The TEI mailing list at http://listserv.brown.edu/?A0=TEI-L.

4) All business of The TEI is conducted in public. Business which needs be conducted in private (for example employment matters, contract negotiation, etc) shall be considered out of scope for The TEI.

5) Changes to the structure of The TEI will be discussed on the TEI mailing list and put to a democratic vote with a voting period of at least one month, a two-thirds majority of votes cast is required to pass a motion, which shall be in English.

6) Groups of members may form for activities from time-to-time, such as members meetings, summer schools, promotions of The TEI or collective digitisation efforts, but these groups are not The TEI, even if the word 'TEI' appears as part of their name.

I'll admit that there are a couple of issues not covered here (such as who holds the IPR), but it's only a straw man for discussion. Feel free to fire it as necessary.

unit testing framework for XSL transformations?

2011-06-23T22:02:00.002+12:00

I'm part of the TEI community, which maintains an XML standard which is commonly transformed to HTML for presentation (more rarely PDF). The TEI standard is relatively large but relatively well documented, the transformation to HTML has thus far been largely piecemeal (from a software engineering point of view) and not error free.

Recently we've come under pressure to introduce significantly more complexity into transformations, both to produce ePub (which is wrapped HTML bundled with media and metadata files) and HTML5 (which can represent more of the formal semantics in TEI). The software engineer in me sees unit testing the a way to reduce our errors while opening development up to a larger more diverse group of people with a larger more diverse set of features they want to see implemented.

The problem is, that I can't seem to find a decent unit testing framework for XSLT. Does anyone know of one?

Our requirements are: XSLT 2.0; free to use; runnable on our ubuntu build server; testing the transformation with multiple arguments; etc;

We're already using: XSD, RNG, DTD and schematron schemas, epubcheck, xmllint, standard HTML validators, etc. Having the framework drive these too would be useful.

The kinds of things we want to test include:

Footnotes appear once and only once
Footnotes are referenced in the text and there's a back link from the footnote to the appropriate point in the text
Internal references (tables of contents, indexes, etc) point somewhere
Language encoding used xml:lang survives from the TEI to the HTML
That all the paragraphs in the TEI appear at least once in the HTML
That local links work
Sanity check tables
Internal links within parallel texts
....

Any of many languages could be used to represent these tests, but ideally it should have a DOM library and be able to run that library across entire directories of files. Most of our community speak XML fluently, so leveraging that would be good.

Is there a place for readers' collectives in the bright new world of eBooks?

2011-03-23T20:13:00.002+13:00

The transition costs of migrating from the world of books-as-physical-artefacts-of-pulped-tree to the world of books-as-bitstreams are going to be non-trivial.

Current attempts to drive the change (and by implication apportion those costs to other parties) have largely been driven by publishers, distributors and resellers of physical books in combination with the e-commerce and electronics industries which make and market the physical eBook readers on which eBooks are largely read. The e-commerce and electronics industries appear to see traditional publishing as an industry full of lumbering giants unable to compete with the rapid pace of change in the electronics industry and the associated turbulence in business models, and have moved to poach market-share. By-and-large they've been very successful. Amazon and Apple have shipped millions of devices billed as 'eBook readers' and pretty much all best-selling books are available on one platform or another.

This top tier, however, is the easy stuff. It's not surprising that money can be made from the latest bodice-ripping page-turner, but most of the interesting reading and the majority of the units sold are outside the best-seller list, on the so-called 'long tail.'

There's a whole range of books that I'm interested in that don't appear to be on the business plan of any of the current eBook publishers, and I'll miss them if they're not converted:

The back catalogue of local poetry. Almost nothing ever gets reprinted, even if the original has a tiny print run and the author goes on to have a wonderfully successful career. Some gets anthologised and a few authors are big enough to have a posthumous collected works, when their work is no longer cutting edge.
Some fabulous theses. I'm thinking of things like: http://ir.canterbury.ac.nz/handle/10092/1978, http://victoria.lconz.ac.nz/vwebv/holdingsInfo?bibId=69659 and http://otago.lconz.ac.nz/vwebv/holdingsInfo?bibId=241527
Lots of te reo Māori material (pick your local indigenous language if you're reading this outside New Zealand)
Local writing by local authors.

Note that all of these are local content---no foreign mega-corporation is going to regard this as their home-turf. Getting these documents from the old world to the new is going to require a local program run by (read funded by) locals.

Would you pay for these things? I would, if it gave me what I wanted.

What is it that readers want?

We're all readers, of one kind or another, and we all want a different range of things, but I believe that what readers want / expect out of the digital transition is:

To genuinely own books. Not to own them until they drop their eReader in the bath and lose everything. Not to own them until a company they've never heard of goes bust and turns off a DRM server they've never heard of. Not to own them until technology moves on and some new format is in use. To own them in a manner which enables them to use them for at least their entire lifetime. To own them in a manner that poses at least a question for their heirs.
A choice of quality books. Quality in the broadest sense of the word. Choice in the broadest sense of the word. Universality is a pipe-dream, of course, but with releasing good books faster than I can read them.
A quality recommendation service. We all have trusted sources of information about books: friends, acquaintances, librarians or reviewers that history have suggested have similar ideas as us about what a good read is.
To get some credit for already having bought the book in pulp-of-murdered-tree work. Lots of us have collections of wood-pulp and like to maintain the illusion that in some way that makes us well read.
Books bought to their attention based on whether they're worth reading, rather than what publishers have excess stock of. Since the concept of 'stock' largely vanishes with the transition from print to digital this shouldn't be too much of a problem.
Confidentially for their reading habits. If you've never come across it, go and read the ALA's The Freedom to Read Statement

A not-for-profit readers' collective

It seems to me that the way to manage the transition from the old world to the new is as a not-for-profit readers' collective. By that I mean a subscription-funded system in which readers sign up for a range of works every year. The works are digitised by the collective (the expensive step, paid for up-front), distributed to the subscribers in open file formats such as ePub (very cheap via the internet) and kept in escrow for them (a tiny but perpetual cost, more on this later).

Authors, of course, need to pay their mortgage, and part of the digitisation would be obtaining the rights to the work. Authors of new work would be paid a 'reasonable' sum, based on their statue as authors (I have no idea what the current remuneration of authors is like, so I won't be specific). The collective would acquire (non-exclusive) the rights to digitise the work if not born digital, to edit it, distribute it to collective members and to sell it to non-members internationally (i.e. distribute it through 'conventional' digital book channels). In the case of sale to non-members through conventional digital book channels the author would get a cut. Sane and mutually beneficial deals could be worked out with libraries of various sizes.

Generally speaking, I'd anticipate the rights to digitise and distribute in-copyright but out-of-print poetry would would be fairly cheap; the rights to fabulous old university theses cheaper; and rights to out-of-copyright materials are, of course, free. The cost of rights to new novels / poetry would hugely depend on statue of the author and the quality of the work, which is where the collective would need to either employ a professional editor to make these calls or vote based on sample chapters / poems or some combination of the two. Costs of quality digitisation is non-trivial, but costs are much lower in bulk and dropping all the time. Depending on the platform in use, members of the collective might be recruited as proof-readers for OCR errors.

That leaves the question of how to fund the the escrow. The escrow system stores copies of all the books the collective has digitised for the future use of the collectives' members and is required to give efficacy to the promise that readers really own the books. By being held in escrow, the copies survive the collective going bankrupt, being wound up, or evolving into something completely different, but requires funding. The simplest method of obtaining funding would be to align the collective with another established consumer of local literature and have them underwrite the escrow, a university, major library, or similar.

The difference between a not-for-profit readers' collective and an academic press?

Of hundreds of years, major universities have had academic presses which publish quality content under the universities' auspices. The key difference between the not-for-profit readers' collective I am proposing and an academic press is that the collective would attempt to publish the unpublished and out-of-print books that the members wanted rather than aiming to meet some quality criterion. I acknowledge a popularist bias here, but it's the members who are paying the subscriptions.

Which links in the book chain do we want to cut out?

There are some links in the current book production chain which we need to keep, there are others wouldn't have a serious future in a not-for-profit. Certainly there is a role for judgement in which works to purchase with the collective's money. There is a role for editing, both large-scale and copy-editing. There is a role for illustrating works, be it cover images or icons. I don't believe there is a future for roles directly relating to the production, distribution, accounting for, sale, warehousing or pulping of physical books. There may be a role for the marketing books, depending on the business model (I'd like to think that most of the current marketing expense can be replaced by combination of author-driven promotion and word-of-month promotion, but I've been known to dream). Clearly there is an evolving techie role too.

The role not mentioned above that I'd must like to see cut, of course, is that of the multinational corporation as gatekeeper, holding all the copyrights and clipping tickets (and wings).

HOWTO: Deep linking into the NZETC site

2010-11-20T10:20:00.003+13:00

As the heaving mass of activity that is the mixandmash competition heats up, I have come to realise that I should have better documented a feature of the NZETC site, the ability to extract the TEI xml annotated with the IDs for deep linking.

Our content's archival form is TEI xml, which we massage for various output formats. There is a link from the top level of every document to the TEI for the document, which people are welcome to use in their mashups and remixes. Unfortunately, between that TEI and our HTML output is a deep magic that involves moving footnotes, moving page breaks, breaking pages into nicely browsable chunks, floating marginal notes, etc., and this makes it hard to deep link back to the website from anything derived from that TEI.

There is another form of the TEI available which is annotated with whether or not each structural element maps 1:1 to an HTML: nzetc:has-text and what the ID of that page is: nzetc:id This annotated XML is found by replacing the 'tei-source' in the URL with 'etexts'

Thus for The Laws of England, Compiled and translated into the Māori language at http://www.nzetc.org/tm/scholarly/tei-GorLaws.html there is the raw TEI at http://www.nzetc.org/tei-source/GorLaws.xml and the annotated TEI at http://www.nzetc.org/etexts/GorLaws.xml

Looking in the annotated TEI at http://www.nzetc.org/etexts/GorLaws.xml we see for example:

<div xml:id="t1-g1-t1-front1-tp1" xml:lang="en" rend="center" type="titlePage" nzetc:id="tei-GorLaws-t1-g1-t1-front1-tp1" nzetc:depth="5" nzetc:string-length="200" nzetc:has-text="true">

This means that this div has it's own page (because it has nzetc:has-text="true" and that the ID of that page is tei-GorLaws-t1-g1-t1-front1-tp1 (because of the nzetc:id="tei-GorLaws-t1-g1-t1-front1-tp1"). The ID can be plugged into: http://www.nzetc.org/tm/scholarly/<ID>.html to get a URL for the HTML. Thus the URL for this div is http://www.nzetc.org/tm/scholarly/tei-GorLaws-t1-g1-t1-front1-tp1.html This process should work for both text and figures.

Happy remixing everyone!

ePubs and quality

2009-11-08T20:36:00.002+13:00

You may have heard news about the release of "bookserver" by the good folks at the Internet Archive. This is a DRM-free ePub ecosystem, initially stocked with the prodigious output of Google's book scanning project and the Internet Archive's own book scanning project.

To see how the NZETC stacked up against the much larger (and better funded) collection I picked one of our Maori Language dictionaries. Our Maori and Pacifica dictionaries month-after-month make up the bulk of our top five must used resources, so they're in-demand resources. They're also an appropriate choice because when they were encoded by the NZETC into TEI, the decision was made not to use full dictionary encoding, but a cheaper/easier tradeoff which didn't capture the linguistic semantics of the underlying entries, but treated them as typeset text. I was interested in how well this tradeoff was wearing.

I did my comparison using the new firefox ePub plugin, things will be slightly different if you're reading these ePubs on an iPhone or Kindle.

The ePub I looked at was A Dictionary of the Maori Language by Herbert W. Williams. The NZETC has the 1957 sixth edition. There are two versions of the work on bookserver. A 1852 second edition scanned by Google books (original at the New York Public library) and a 1871 third edition scanned by the Internet Archive in association with Microsoft (original in the University of California library system). All the processing of both works appear to be been done in the U.S. The original print used macrons (NZETC), acutes (Google) and breves (Internet Archive) to mark long vowels. Find them here.

Lets take a look at some entries from each, starting at 'kapukapu':

NZETC:

kapukapu. 1. n. Sole of the foot.

2. Apparently a synonym for kaunoti, the firestick which was kept steady with the foot. Tena ka riro, i runga i nga hanga a Taikomako, i te kapukapu, i te kaunoti (M. 351).

3. v.i. Curl (as a wave). Ka kapukapu mai te ngaru.

4. Gush.

5. Gleam, glisten. Katahi ki te huka o Huiarau, kapukapu ana tera.

Kapua, n. 1. Cloud, bank of clouds. E tutakitaki ana nga kapua o te rangi, kei runga te Mangoroa e kopae pu ana (P.).

2. A flinty stone. = kapuarangi.

3. Polyprion oxygeneios, a fish. = hapuku.

4. An edible species of fungus.

5. Part of the titi pattern of tattooing.

Kapuarangi, n. A variety of matā, or cutting stone, of inferior quality. = kapua, 2.

Kāpuhi, kāpuhipuhi, n. Cluster of branches at the top of a tree.

Kāpui, v.t. 1. Gather up in a bunch. Ka kapuitia nga rau o te kiekie, ka herea.

2. Lace up or draw in the mouth of a bag.

3. Earth up crops, or cover up embers with ashes to keep them alight.

kāpuipui, v.t. Gather up litter, etc.

Kāpuka, n. Griselinia littoralis, a tree. = papauma.

Kapukiore, n. Coprosma australis, a shrub. = kanono.

Kāpuku = kōpuku, n. Gunwale.

Google Books:

Kapukapu, s. Sole of the foot,

Eldpukdpu, v. To curl* as a

wave.

Ka kapukapu mai te ngaru; The wave curls over.

Kapunga, v. To take up with both hands held together,

Kapungatia he kai i te omu; Take up food from the oven.

(B. C,

Kapura, s. Fire, -' Tahuna he kapura ; Kindle a fire.

Kapurangi, s. Rubbish; weeds,

Kara, s. An old man,

Tena korua ko kara ? How are you and the old man ?

Kara, s> Basaltic stone.

He kara te kamaka nei; This stone is kara.

Karaha, s. A calabash. ♦Kardhi, *. Glass,

Internet Archive:

kapukapu, n. sole of the foot.

kapukapu, v. i. 1. curl (as a wave). Ka kapukapu mai te ngaru. 2. gush.

kakapii, small basket for cooked food.

Kapua, n. cloud; hank of clouds,

Kapunga, n. palm of the hand.

kapunga, \. t. take up in both hands together.

Kapiira, n. fire.

Kapiiranga, n. handful.

kapuranga, v. t. take up by hand-fuls. Kapurangatia nga otaota na e ia. v. i. dawn. Ka kapuranga te ata.

Kapur&ngi, n. rubbish; uveds.

I. K&r&, n. old man. Tena korua ko kara.

II. K&r&, n. secret plan; conspiracy. Kei te whakatakoto kara mo Te Horo kia patua.

k&k&r&, D. scent; smell.

k&k&r&, a. savoury; odoriferous.

k^ar&, n. a shell-iish.

Unlike the other two, the NZETC version has accents, bold and italics in the right place. It' the only one with a workable and useful table of contents. It is also edition which has been extensively revised and expanded. Google's second edition has many character errors, while the Internet Archive's third edition has many 'á' mis-recognised as '&.' The Google and Internet Achive versions are also available as PDFs, but of course, without fancy tables of contents these PDFs are pretty challenging to navigate and because they're built from page images, they're huge.

It's tempting to say that the NZETC version is better than either of the others, and from a naïve point of it is, but it's more accurate to say that it's different. It's a digitised version of a book revised more than a hundred years after the 1852 second edition scanned by Google books. People who're interested in the history of the language are likely to pick the 1852 edition over the 1957 edition nine times out of ten.

Technical work is currently underway to enable third parties like the Internet Archive's bookserver to more easily redistribute our ePubs. For some semi-arcane reasons it's linked to upcoming new search functionality.

What LibraryThing metadata can the NZETC reasonable stuff inside it's CC'd epubs?

2009-11-08T14:56:00.002+13:00

This is the second blog following on from an excellent talk about librarything by LibraryThing's Tim given the VUW in Wellington after his trip to LIANZA.

The NZETC publishes all of it's works as epubs (a file format primarily aimed at mobile devices), which are literally processed crawls of it's website bundled with some metadata. For some of the NZETC works (such as Erewhon and The Life of Captain James Cook), LibraryThing has a lot more metadata than the NZETC, becuase many LibraryThing users have the works and have entered metadata for them. Bundling as much metadata into the epubs makes sense, because these are commonly designed for offline use---call-back hooks are unlikely to be avaliable.

So what kinds of data am I interested in?
1) Traditional bibliographic metadata. Both LT and NZETC have this down really well.
2) Images. LT has many many cover images, NZETC has images of plates from inside many works too.
3) Unique identification (ISBNs, ISSNs, work ids, etc). LT does very well at this, NZETC very poorly
4) Genre and style information. LT has tags to do fancy statistical analysis on, and does. NZETC has full text to do fancy statistical analysis on, but doesn't.
5) Intra-document links. LT has work as the smallest unit. NZETC reproduces original document tables of contents and indexes, cross references and annotations.
6) Inter-document links. LT has none. NZETC captures both 'mentions' and 'cites' relationships between documents.

While most current-generation ebook readers, of course, can do nothing with most of this metadata, but I'm looking forward to the day when we have full-fledged OpenURL resolvers which can do interesting things, primarily picking the best copy (most local / highest quality / most appropiate format / cheapest) of a work to display to a user; and browsing works by genre (LibraryThing does genre very well, via tags).

Interlinking of collections: the quest continues

2009-10-15T21:36:00.004+13:00

After an excellent talk today about LibraryThing by LibraryThing's Tim, I got enthused to see how LibraryThing stacks up against other libraries for having matches in it's authority control system for entities we (the NZETC) care about.
The answer is averagely.
For copies of printed books less than a hundred years old (or reprinted in the last hundred years), and their authors, LibraryThing seems to do every well. These are the books likely to be in active circulation in personal libraries, so it stands to reason that these would be well covered.
I tried half a dozen books from our Nineteenth-Century Novels Collection, and most were missing, Erewhon, of course, was well represented. LibraryThing doesn't have the "Treaty of Waitangi" (a set of manuscripts) but it does have "Facsimiles of the Treaty of Waitangi." It's not clear to me whether these would be merged under their cataloguing rules.
Coverage of non-core bibliographic entities was lacking. Places get a little odd. Sydney is "http://www.librarything.com/place/Sydney,%20New%20South%20Wales,%20Australia" but Wellington is "http://www.librarything.com/place/Wellington" and Anzac Cove appears to be is missing altogether. This doesn't seem like a sane authority control system for places, as far as I can see. People who are the subjects rather than the authors of books didn't come out so well. I couldn't find Abel Janszoon Tasman, Pōtatau Te Wherowhero or Charles Frederick Goldie, all of which are near and dear to our hearts.

Here is the spreadsheet of how different web-enabled systems map entities we care about.

Correction: It seems that the correct URL for Wellington is http://www.librarything.com/place/Wellington,%20New%20Zealand which brings sanity back.

eBook readers need OpenURL resolvers

2009-09-19T21:57:00.002+12:00

Everyone's talking about the next generation of eBook readers having larger reading area, more battery life and more readable screen. I'd give up all of those, however, for an eBook reader that had an internal OpenURL resolver.

OpenURL is the nifty protocol that libraries use to find the closest copy of a electronic resources and direct patrons to copies that the library might have already licensed from commercial parties. It's all about finding the version of a resource that is most accessible to the user, dynamically.

Say I've loaded 500 eBooks into my eBook reader: a couple of encyclopedias and dictionaries; a stack of books I was meant to read in school but only skimmed and have been meaning to get back to; current block-busters; guidebooks to the half-dozen countries I'm planning on visiting over the next couple of years; classics I've always meant to read (Tolstoy, Chaucer, Cervantes, Plato, Descartes, Nietzsche); and local writers (Baxter, Duff, Ihimaera, Hulme, ...). My eBooks by Nietzsche are going to refer to books by Descartes and Plato; my eBooks by Descartes are going to refer to books by Plato; my encyclopaedias are going to refer to pretty much everything; most of the works in translation are going to contain terms which I'm going to need help with (help which theencyclopedias and dictionaries can provide).

Ask yourself, though, whether you'd want to flick between works on the current generation of readers---very painful, since these devices are not designed for efficient navigation between eBooks, but linear reading of them. You can't follow links between them, of course, because on current systems links must point either with the same eBook or out on to the internet---pointing to other eBooks on the same device is verboten. OpenURL can solve this by catching those URLs and making them point to local copies of works (and thus available for free even when the internet is unavailable) where possible while still retaining their

Until eBook readers have a mechanism like this eBooks will be at most a replacement only for paperback novels---not personal libraries.

Thoughts on koha

2009-09-15T21:30:00.001+12:00

The Koha community is currently undergoing a spasm, with a company apparently forking the code.
As a result a bunch of people are looking at where the community should go from here and how it should be led. In particular the idea of a not-for-profit foundation has been floated and is to be discussed at a meeting early tomorrow morning .
My thoughts on this issue are pretty simple:

A not-for-profit is a fabulous idea
Reusing one of the existing software not-for-profit (Apache, Software in the Public Interest, etc) introduces a layer of non-library complexity. Libraries are have a long history with consortia, but tend to very much flock together with their own kind, I can see them being leary of a non-library entity.
A clear description of a forward-looking plan written in plain language that everyone can understand is vital to communicate the vision of the community, particularly to those currently on the fringes

Data and data modelling and underlying assumptions

2009-09-01T18:44:00.004+12:00

I feel that there was a huge disconnect between some groups of participants at #opengovt (http://groups.google.co.nz/group/nzopengovtbarcamp) in Wellington last weekend. This is my attempt to illuminate the gaps.

The gaps were about data and data modelling and underlying assumptions that the way one person / group / institution viewed a kind of data was the same as the way others viewed it.

This gap is probably most pronounced in geo-location.

There's a whole bunch of very bright people doing wonderful mashups in geo-location using a put-points-on-a-map model. Typically using google maps (or one of a small number of competitors) they give insights into all manner of things by throwing points onto maps, street views, etc, etc. It's a relatively new field and every time I look they seem to have a whizzy new toy. Whizzy thing of the day for me was http://groups.google.com/group/digitalnz/browse_thread/thread/b5b0c96ce08ca441 . Unfortunately the very success of the 'data as points' model encourages the view that location is a lat / long pair and the important metric is the number of significant digits in the lat / long.

In the GLAM (Galleries, Libraries, Archives and Museums) sector, we have a tradition of using thesauri such as the Getty Thesaurus of Geographic Names. Take all look at the entry for The Wellington region:http://www.getty.edu/vow/TGNFullDisplay?find=wellington&place=&nation=New+Zealand&prev_page=1&english=Y&subjectid=7000512

Yes, if has a lat and a long (with laughable precision), but the lat and long are arguably the least important information on the page. There's a faceted hierarchy, synonyms, linked references and type data. Te Papa have just moved to Getty for place names in their new site (http://collections.tepapa.govt.nz/) and frankly, I'm jealous. They paid a few thousand dollars for a licence to thesaurus and it's a joy to use.

The idea of #opengovt is predicated on institutions and individuals speaking the same languages, being able to communicate effectively, and this is clearly a case where we're not. Learning to speak each others languages seems like it's going to be key to this whole venture.

As something of a worked example, here's something that I'm working on at the moment. It's a page from The Manual of the New Zealand Flora by Thomas Frederick Cheeseman, a core text in New Zealand botany, see http://www.nzetc.org/tm/scholarly/tei-CheManu-t1-body1-d22-d5.html The text is live on our website, but it's not yet fully marked up. I've chosen it because it illustrates two separate kinds of languages and their disparities.

What are the geographic locations on that page?

* Nelson-Mountains flanking the Clarence Valley
* Marlborough—Kaikoura Mountains
* Canterbury—Kowai River
* Canterbury—Coleridge Pass
* Otago—Mount St. Bathan's

The qualifier "2000–5000 ft" (which I believe is an elevation range at which these flourish) applies across these. Clearly we're going to struggle to represent these with a finite number of lat/long points, no matter how accurate. In all likelihood, I'll not actually mark up these locations, since the because no one's working with complex locations, the cost benifit isn't within sight of being worth it.

Te Papa and the NZETC have a small-scale binomial name exercise underway, and for that I'll be scripting the extraction of the following names from that page:

* Notospartium carmichœliœ (synonym Notospartium carmichaeliae)
* Notospartium torulosum

There were a bunch of folks at the #opengovt barcamp who're involved in the "New Zealand Organisms Register" (http://www.nzor.org.nz/) project. As I understand it, they want me to expose the following names from that page:

* Notospartium carmichœliœ, Hook. f.
* Notospartium torulosum, Hook. f.

Of course the name the public want is:

* New Zealand Pink Broom
* ? (Notospartium torulosum appears not to have a common name)

Note that none of these taxonomic names actually appear in full on the page...

Yes is, clearly, an area where the best can be the good and visa versa, but the good needs to at least be aware of the best.

Learning XSLT 2.0 Part 1; Finding Names

2009-07-27T22:31:00.002+12:00

We mark up a lot of names, so one of the first things I decided to do was to build an XSLT stylesheet that takes a list of names and tags those names when they occur in a separate XSLT file. To make things easier and clearer, I've ignored little things like namespaces, conformant TEI, etc, etc.

First up, the list of names, these are multi-word names. Notice the simple structure, this could easily be built from a comma seperated list or similar:

<?xml version="1.0" encoding="UTF-8"?>
<names>
<name>Papaver argemone</name>
<name>Papaver dubium</name>
<name>Papaver Rhceas</name>
<name>Zanthoxylum novæ-zealandiæ</name>
</names>

Next, some sample text:

<?xml version="1.0" encoding="UTF-8"?>
<doc>
 There are several names Papaver argemone in this document Papaver argemone
 Some of them are the same as others (Papaver Rhceas Papaver rhceas P. rhceas)
 Non ASCII characters shouldn't cause a problem in names like Zanthoxylum novæ-zealandiæ AKA Zanthoxylum novae-zealandiae
</doc>

Finally the stylesheet. It consists of three parts: the regexp variable that builds a regexp from the names in the file; a default template for everything but text(); and a template for text()s that applies the rexexp.

<?xml version="1.0"?>
<xsl:stylesheet version="2.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" >


<xsl:variable name="regexp">
<xsl:value-of select="concat('(',string-join(document('name-list.xml')//name/text(), '|'), ')')"/>
</xsl:variable>


<xsl:template match="@*|*|processing-instruction()|comment()">
<xsl:copy>
<xsl:apply-templates select="@*|*|processing-instruction()|comment()|text()"/>
</xsl:copy>
</xsl:template>


<xsl:template match="text()">
<xsl:analyze-string select="." regex="{$regexp}">
<xsl:matching-substring>
<name type="taxonomic" subtype="matched">
<xsl:value-of select="regex-group(1)"/>
</name>
</xsl:matching-substring>
<xsl:non-matching-substring>
<xsl:value-of select="."/>
</xsl:non-matching-substring>
</xsl:analyze-string>
</xsl:template>

</xsl:stylesheet>

The output looks like:

<?xml version="1.0" encoding="UTF-8"?><doc>
 There are several names <name type="taxonomic" subtype="matched">Papaver argemone</name> in this document <name type="taxonomic" subtype="matched">Papaver argemone</name>
 Some of them are the same as others (<name type="taxonomic" subtype="matched">Papaver Rhceas</name> Papaver rhceas P. rhceas)
 Non ASCII characters shouldn't cause a problem in names like <name type="taxonomic" subtype="matched">Zanthoxylum novæ-zealandiæ</name> AKA Zanthoxylum novae-zealandiae
</doc>

As you may notice, I've not yet worked out the best way to handle the 'æ'

Legal Māori Archive

2009-06-06T23:22:00.002+12:00

Now that the Legal Māori Archive is live, I thought I'd highlight a couple of my favourite texts from the corpus.

The first is a great example of reinforcing cultural confusion. "The Laws of England, Compiled and translated into the Māori language" by judge Francis Dart Fenton is a bi-lingual compendium of the laws of England, but extraordinarily uses bible quotes as examples.

The second example is actaully a collection of texts, the works of Rev. Henry Hanson Turton, who compiled thousands of pages of land deeds and associated documents into six volumes. I can see these seeing a lot of use by Treaty researchers.

Why card-based records aren't good enough

2009-05-05T21:54:00.004+12:00

Card catalogs have a long tradition in librarianship, dating back, I'm told, to the book stock-take in the French revolution. Librarians understand card catalogs in a deep way that comes from generations of librarians having used them as a core professional tool all their professional lives. Librarians understand card catalogs in ways that I, as a computer scientist, never will. I still recall on one of my first visits to a university library, I asked a librarian where I might find books by a particular author, they found the work for me arguably as fast as I can now find works with the new wizzy electronic catalog.

It is natural, when faced with something new, to understand it in terms of what we already know and already understand. Unfortunately, understanding the new by analogy to the old can lead to form of the old being assumed in the new. It was true that when libraries digitized their card catalogs in the 1970s and 1980s, they were more or less exactly digital versions of the card catalog predecessors, because their content was limited to old data from the cards and new data from cataloging processes (which were unchanged from the card catalog era) and because librarians and users had come to equate a library catalog with a card catalog---it was what they expected.

MARC is a perfect example of this kind of thing. As a data format to directly replace a card catalog of printed books, it can hardly be faulted.

Unfortunately, digital metadata has capabilities undreamt of at the time of the French revolution, and card catalogs and MARC do a poor job of handling these capabilities.

A whole range of people have come up with criticisms of MARC that involve materials and methodologies not routinely held in libraries at the time of the French revolution (digital journal subscriptions and music, for example), but I view these as postdating card catalogs and thus the criticism as unfair.

So what was held in libraries in 1789 that MARC struggle with? Here's a list:

Systematically linking discussion of particular works with instances of those works
Systematically linking discussion of particular instances with those instances ("Was person X the transcriber of manuscript Y?")
Handling ambiguity ("This play may have been written by Shakespeare. It might also have been a later forgery by Francis Bacon, Christopher Marlowe or Edward de Vere")

All of these relate to core questions which have been studed in libraries for centuries. They're well understood issues, which changed little in the hundred years until the invention of the computer (which is when all the usually-cited issues with MARC began).

The real question is why we're still expecting an approach that didn't solve the problems two hundred years ago to solve our problems now? Computers are not magic in this area they just seem to be helping us do the wrong things faster, more reliably and for larger collections.

We need a new approach to bibliographic metadata, one which is not ontologically bound to little slips of paper. There are a whole range of different alternatives out there (including a bevy of RDF vocabularies), but I've yet to run into one which both allowed clear representation of existing data (because lets face it, I'm not going to re-enter worldcat, and neither are you, not in our lifetimes) and admitting non-card-based metadata as first class elements.

</rant>

LoC gets semantic

2009-05-01T09:34:00.001+12:00

This morning, the Library of Congress launched http://id.loc.gov/authorities/, their first serious entry into the semantic web.

The site makes the Library of Congress Subject Headings available as defererenable URLs. For example http://id.loc.gov/authorities/sh90005545.

NDHA demo and the National Library

2009-02-04T21:11:00.002+13:00

This morning I went to the NDHA demonstration where a National Library techie talked us through the NDHA ingest tools. The tools are the most visible piece of the NDHA infrastructure, and are designed to unify the ingest of digital documents, whether they are born-digital documents physically submitted (i.e. producers mail in CDs/DVDs etc); born-digital documents electronically submitted (i.e. producers upload content via wizzy web tools); or digital scans of current holdings produced as part of the on-going digitisation efforts. The tools have a unified system with different workflows for unpublished material (=archive) and published material (=libarary). The unification of library and archival functionality seemed like futile ground for miscommunication.

The infrastructure is (correctly) embedded across the library, and uses all the current tools for collection maintenance, searching and access.

As a whole it looks like the system is going to save time and money for large content producers and capture better metadata for small donors of content, which is great. By moving the capture of metadata closer to the source (while still allowing professional curatorial processes for selection and cataloguing), it looks like more context is going to be captured, which is fabulous.

A couple of things struck me as odd:

The first feedback to the producer/uploader is via a human. Despite having an elaborate validation suite, the user wasn't told immediately "that .doc file looks like a PDF, would you like to try again?" or "Yep, that *.xml file is valid and conforming XML" Time and again studies have shown that immediate feedback to allow people to correct their mistakes immediately is important and effective.
The range of metadata fields available for tagging content was very limited. For example there was no Iwi/Hapu list, no Maori Subject Headings, no Gazetteer of Official Geographic Names. When I asked about these I was told "that's the CMS's role" (=Collection Management Software, i.e. those would be added by professional cataloguers later), but if you're going to move the metadata collection to as close to content generation, it makes sense to at least have the option of proper authority control over it.

Or that's my take, anyway. Maybe I'm missing something.

Report from the NDHA's International Perspectives on Digital Preservation

2009-02-02T21:40:00.001+13:00

NOTE: I'm a computer scientist by training and this was largely librarian/archivist gig, so it's entirely possibly I've got the wrong end of the stick on one or more point in the summary below. It's also my own summary, and not the position of my employer, even though I was on work time during the event.

The NDHA is about to announce that the NDHA project has been completed on time and under budget. This is particularly pleasing in light of the poor history of government IT failures over the course of the last 30 years and a tribute to all concerned. Indeed, when I was taking undergraduate courses in software engineering a contemporary national library project was used as a text-book example of how not to run a software development undertaking. It's good to see how far they've come.

The event itself was a one-day event in the national library auditorium, with a handful of overseas speakers. I'm not entirely certain that a handful of foreigners counts as "international," but maybe that's just me being a snob. Certainly there was a fine turn-out of locals, including many from the National Library, the Ministry of Culture and Heritage and from VUW, including a number of students, who couldn't possibly have been there for the free food.

There seemed to be an underlying tension between librarianship and archivistship running through the event. I see this as being a really crazy turfwar, personally, since I see the chances of libraries and archives existing as separate entities and disciplines in fifty years seems pretty slim. The separation between the two, the "uniqueness" of objects in an archive seems to be to be obliterated by the free-duplication of digital objects. I've heard people say that archives also work access controls and embargoes for their depositors, but then so can libraries, particularly those in the military and those working with classified documents.

It seemed to me that the word "reliability" was used in a confusing number different of ways by different people. Without naming the guilty parties:

reliability as the truthfulness of the documents in the library/archive. This is the old problem of ingestors having to determine the absolute veracity of documents
reliability as getting the same metadata every time. This seems odd to me, since systems with audit control give _different_ results every time, because information on the previous accesses is included in the metadata of subsequent accesses
reliability as the degree to which the system conformed to a standard/specification

On reflection this may have been a symptom of the different vocabulary used by librarians and archivists. Whatever the cause, if we're wanting to spend public money, we have to be able to explain to the public what we're doing, and this isn't helping.

The organisers told us the presentations would be up by tonight (the evening of the presentation), but you won't find them on google if you go looking, because they tell google to please f**k off. I guess this is what someone was referring to when they said we had to work to make content accessible to google. The link is http://ndha-wiki.natlib.govt.nz/ndha/pages/IPoDP%2009%20Presentations and most were up at the time of writing.

I was hugely encouraged by the number of pieces of software that seemed to be being open sourced, as I see this as being a much better economic model than paying vendors for custom software, particularly since it's potentially scalable out from the national and top-tier libraries/archives/museums out to the second and third tier libraries/archives/museums, which by dint of their much larger numbers actually serve the most users and have the most content. It was unfortunate that the national library hasn't looked beyond propriety software for non-specialist software but continues to use AbodePhotoshop / Microsoft Windows, which are available only for limited periods of time on certain platforms (which will inevitably become obsolete), rather than openoffice, GIMP, etc, which are cross platform and licensed under perpetual licences which include the right to port the software from one platform to another. I guessPhotoshop / Windows is what their clients and funders know and use.

With a number of participants I had conversations about preservation. Andrew Wilson in his presentation used the quote:

“traditionally, preserving things meant keeping them unchanged; however our digital environment has fundamentally changed our concept of preservation requirements. If we hold on to digital information without modifications, accessing the information will become increasingly difficult, if not impossible” Su-Sing Chen, “The Paradox of Digital Preservation”, Computer, March 2001, 2-6

If you think about what intellectual objects we have from the Greeks (which is were us Westerners traditionally trace our intellectual history from), the majority fall into two main classes: (a) art works, which have survived primarily through roman copies and (b) texts, which have survived by copying, including a body of mathematics which were kept alive in the Arabic translation during a period when we Westerners were burning the works in Latin and Greek and claiming that the bible was the only book we needed. I'll grant you that a high-quality book will last maybe 500 years in a controlled environment, maybe even 1000, but for real permanence, you just can't get past physical ubiquity. If we have things truly worthy of long-term preservation, we should be striking deals with the Warehouse to get them into every home in the country, and setting them as translation exercises in our language learning courses.

I had some excellent conversations with other participants at the event, including Phillipa Tocker from Museums Aotearoa / Te Tari o Nga Whare Taonga o te Motu who told me about the http://www.nzmuseums.co.nz/ site they put together for their members.

Looking at the site I'm struck by how similar the search functionality is to http://www.nram.org.nz/. I'm not sure whether their relative similarity is a good thing (because it enables non-experts to search the holdings) or a bad thing (because by lowering themselves to the lowest common denominator they've devalued their uniqueness). While I'm certain that these websites have vital roles in the museums and archives community respectively, I can't help but feel that from an end-users perspective have two sites rather than one seems redundant, and the fact that they don't seem to reference/suggest any other information sources doesn't help. I can't imagine a librarian/archivist not being forth-coming with a suggestion of where to look next if they've run out of local relevant content---why should our websites be any different?

I recently changed the NZETC to point to likely-relevant memory institutions when a search returns no results (or when a user pages through to the end of any list of results).

I also talked to some chaps from Te Papa about the metadata they're using to to represent places names (Getty Thesaurus of Geographic Names) and species names (ad-hoc). At the NZETC we have many place names marked up (in NZ, Europe and the Pacific), but are not currently syncing with an external authority. Doing so would hugely enable interoperability. Ideally we'd be using the shiny new New Zealand Gazetteer of Official Geographic Names, but it doesn't yet have enough of the places we need (it basically only covers places mentioned in legislation or treaty settlements). It does have macrons in all the right places though, which is an excellent start. We currently don't mark up species names, but would like to, and again an external authority would be great.

It might have been useful if the day had included an overview of what the NDHA actually was and what had been achieved (maybe I missed this?).

flickr promoting the commons / creative commons

2009-02-01T11:38:00.004+13:00

flickr is promoting photos in what it calls "The Commons", but only to logged in users. Normal users (who can't comment / tag the photos in the commons anyway) don't get an obvious link to them (except via about a billion third party sites such as blogs and google search). The page also shows how the national library's choice not to include text in their logo has come up trumps.

The confusingly similarly named "The Commons" and "Creative Commons" parts of the website apparently don't reference each other. Odd.

Excellent stuff from New Zealand Geographic Board Ngā Pou Taunaha o Aotearoa

2009-01-09T21:00:00.002+13:00

A while ago, motivated by the need for an authoritative list of New Zealand place names for our with at the NZETC, I criticised the NZGB fairly roundly.
While they haven't produced what I/we want/need, in the last couple of months they've made huge progress in an unambiguously right direction.
Their primary work is the New Zealand Gazetteer of Official Geographic Names, a list of all official place names in New Zealand. It uses have a peculiar definition of "official" (= mentioned in legislation or a Treaty of Waitangi settlement), they have very few names of inhabited places (and no linking with the much larger ones maintained by official bodies such as the police and fire service), They have no elevation data for mountains and pass (which are defined by their height) and they define some things as points when they appear to be areas (such as Arthur Pass National Park), but it's much better than the New Zealand Place Names Database since:

It has a statutory reference for every place, given the source of the officialness of the name
It fully support Macrons
It has a machine readable-list of DoC administered lands --- I can imagine this being used for all sorts of interesting things, getting people out in other scenic and marine reserves.

NZGB sent around an email in which they explicitly addressed some of the points I'd earlier raised (I'm sure I wasn't the only one):

It should be noted that some of the naming practices of the past will have to be lived with, despite inconsistencies. Moving forward, the rules of nomenclature followed by the NZGB are designed to promote standardisation, consistency, and non-ambiguity. The modern format for dual names is '<Maori name> / <non-Maori name', which the NZGB has applied for the past 10 years, though Treaty settlement dual names sometimes deviate from this convention, because the decision is ultimately made by the Minister for Treaty of Waitangi Negotiations. Older forms of dual names, with brackets, will remain depicted as such until changed through the statutory processes of the NZGB Act 2008. These are not generally regarded as alternative names.
Macrons in Maori names have posed problems for electronic databases. Nevertheless they are part of the orthography, recommended by the Maori Language Commission, and the Board endorses their use. The Gazetteer will include macrons where they are formalised as part of the official name. When Section 32 of the new Act comes into force, official documents will be required to show official names, and these will need to include macrons where they have been included as part of the official name (unless the proviso is used). A list of those official names which have macrons is at http://www.linz.govt.nz/placenames/researching-place-names/macrons/index.aspx . LINZ's Customer Services has some solutions for showing macrons in LINZ's own databases and on published maps and charts, and is currently investigating how bulk data extracts might include information about macrons, for the customer's benefit.

Despite the name, it isn't clear in my mind exactly what's official and what isn't. Is the content of the "coordinates" column official? For railway lines this is a reference to the description, which in the cases of railways is usually of the form "From X to Y", where X and Y are place names, frequently place names that aren't on the list, so are thus presumably not official. Unless I'm going blind there is also no indication of accuracy on the physical measurements.

fuzzziness

2008-10-09T01:04:00.002+13:00

I've been using topic maps in my day job, so I decided to try out http://www.fuzzzy.com/, a social bookmark engine that uses an underlying topic map engine.
I tried to approach fuzzzy with an open mind, but the increasingly stumbling on really annoying (mis-)features.

This is the first bookmark engine I've ever used hat doesn't let users migrate their bookmarks with them. This is perhaps the biggest single feature fuzzzy could add to attract new users, since it seems that most people who're likely to use a bookmark engine have already played with another one long enough to have dozens or hundreds of bookmarks they'd like to bring with them. I know this is non-ideal from the point of view of the social bookmark engine they're migrating too, since it makes it hard to do things completely differently, but users have baggage.
While it'd possible to vote up or vote down just about everything (bookmarks, tags, bookmark-tags, users, etc), very little is actually done with these votes. If I've viewed a bookmark once and voted it down, why is it added to my "most used Bookmarks"? Surely if I've indicated I don't like it the bookmark should be hidden from me, not advertised to me.
For all the topic map goodness on the site, there is no obvious way to link from the fuzzzy topic map to other topic maps.
There doesn't seem to be much in the way of interfacing with other semantic web standards (i.e. RDF).
The help isn't. Admittedly this may be partly because many of the key participants have English as a second language.
There's a spam problem. But then everywhere has a spam problem.
It's not obvious that I can export my bookmarks out of fuzzzy in a form that any other bookmark engine understands.

These (mis-)features are a pity, because at NZETC we use topic maps for authority (in the librarianship sense), and it would be great to have a compatible third party that could be used for non-authoritative stuff and which would just work seamlessly.

Place name inconsistencies

2008-10-05T12:41:00.005+13:00

I've been looking at the "Dataset of New Zealand Geographic Place Names" from LINZ. This appears to be as close as New Zealand comes to an Official list of place names. I've been looking because it would be great to use as an authority in the NZETC.

Coming to the data I was aware of a number of issues:

Unlike most geographical data users, I'm primarily interested in the names rather than the relative positions
New Zealand is currently going through an extended period of renaming of geographic features to their original Māori names
The names in the dataset are primarily map labels and are subject to cartographic licence

What I didn't expect was the insanity in the names. I know that there are some good historical reasons for this insanity, but that doesn't make it any less insane.

Names can differ only by punctuation. There is a "No. 1 Creek" and a "No 1 Creek".
Names can differ only by presentation. There is a "Crook Burn or 8 Mile Creek", an "Eight Mile Creek or Boundary Creek" and an "Eight Mile Creek" (each in a different province).
There is no consistent presentation of alternative names. There is "Saddle (Mangaawai) Bivouac", "Te Towaka Bay (Burnside Bay)", "Queen Charlotte Sound (Totaranui)", "Manawatawhi/Three Kings Islands", "Mount Hauruia/Bald Rock", "Crook Burn or 8 Mile Creek" and "Omere, Janus or Toby Rock"
There is no machine-readable source of the Māori place names with macrons, and the human readable version has contains subtle difference to the machine-readable database (which contains no non-ASCII characters). For example "Franz Josef Glacier/Kā Roimata o Hine Hukatere (Glacier)" and "Franz Josef Glacier/Ka Roimata o Hine Hukatere" differ by more than the macrons. There appears to be no information on which are authoritative.

Right now I'm find finding this rather frustrating.

(grammar edit)

Does anyone publish the Dataset of New Zealand Geographic Place Names already in XML form?

2008-09-02T21:10:00.006+12:00

I've been playing with the Dataset of New Zealand Geographic Place Names which is a set of CSV files published by Toitū te whenua / Land Information New Zealand (LINZ). The data takes quite a bit of massaging, and I was wondering whether anyone else had already done the work of making acceptable XML out of the data rather than doing all the work myself.

I've attached the script I have so far, but it's not perfect. In particular:

It doesn't include place names with Macrons
It makes lots of ASCII-type assumptions
Many of the element names are poorly named and map non-obviously to fields in the CSV files.
The script isn't very generic and does little or no checking

Anyway, here's he script, hopefully it's successfully escaped. The basics are that it creates an sqlite database and streams the CSV files into it direct from the zip (which it expects to have been downloaded into the current directory). It then streams each point out using awk to transform it to XML.


#!/bin/bash
# script to import data from
# http://www.linz.govt.nz/placenames/search/place-names-dataset-download/index.aspx
# into an XML file.
# this script licensed under the GPL/BSD/Apache 2 licences

echo \(re\)creating the database, expect DROP errors the first time you run this 
sqlite nzgeonames.db << EOF
DROP TABLE name;
CREATE TABLE  name (id, name, east, north, pdescription, district, sheet, lat, long);

DROP TABLE district;
CREATE TABLE  district (district, description);


DROP TABLE pdescription;
CREATE TABLE  pdescription (pdescription, short, description);


DROP TABLE sheet;
CREATE TABLE  sheet (edition, map, sheet);

VACUUM;
EOF

echo importing the names
unzip -p nznames_6Aug08.zip namedata.txt  | sed 's/\r//'  |  sed 's/`/","/g' | awk -F^ '{print "INSERT INTO name VALUES (\"" $0 "\");"}' | sqlite nzgeonames.db 

echo importing the districts
unzip -p nznames_6Aug08.zip landdist.txt  | sed 's/\r//'  |  sed 's/`/","/g' | awk -F^ '{print "INSERT INTO district VALUES (\"" $0 "\");"}' | sqlite nzgeonames.db

echo importing the point descriptions \(expect two lines of errors\) 
unzip -p nznames_6Aug08.zip pointdes.txt  | sed 's/\r//'  |  sed 's/`/","/g' | awk -F^ '{print "INSERT INTO pdescription VALUES (\"" $0 "\");"}' | sed 's/:/","/' | sqlite nzgeonames.db

echo importing the sheet names
unzip -p nznames_6Aug08.zip sheetnam.txt  | sed 's/\r//'  |  sed 's/`/","/g' | awk -F^ '{print "INSERT INTO sheet VALUES (\"" $0 "\");"}' | sqlite nzgeonames.db 


# pick up the ugly duckling
sqlite nzgeonames << EOF
 INSERT INTO pdescription VALUES ("MRFM","MARINE ROCK FORMATION","Marine Rock Formation");
EOF

echo exporting points as xml
echo "<document source=\"Sourced from Land Information New Zealand, [date]. Crown copyright reserved.\">" > nzgeonames.xml
sqlite nzgeonames.db "SELECT name.id, name.name, name.east, name.north, name.pdescription, name.district, name.sheet, name.lat, name.long, district.description, pdescription.short, pdescription.description AS descriptionA, sheet.edition, sheet.map FROM name, district, pdescription, sheet WHERE name.district = district.district AND name.pdescription = pdescription.pdescription AND name.sheet = sheet.sheet;" | awk -F\| '{print "<point><id>" $1 "</id><name>" $2 "</name><east>" $3 "</east><north>" $4 "</north><pdescription>" $5 "</pdescription><district>" $6 "</district><sheet>" $7 "</sheet><lat>" $8 "</lat><long>" $9 "</long><description>" $10 "</description><short>" $11 "</short><descriptionA>" $12 "</descriptionA> <edition>" $13 "</edition> <map>" $14 "</map>  </point>"}' | sed 's/&/&amp;/' >> nzgeonames.xml 
echo "</document>" >> nzgeonames.xml

echo formatting the points nicely
xmllint --format nzgeonames.xml  > nzgeonames-formatted.xml

Library of Congress flickr experiment

2008-09-02T18:08:00.002+12:00

Native troops, Cuba (LOC)
Originally uploaded by The Library of Congress

While processing the photos from my parent's ruby wedding anniversary, I ran into the Library of Congress's flickr experiment.

I probably shouldn't have been, but I was astounded. It looks the the bastion of old-school cataloguing is coming to bathe in the fountain of social tagging.

This is part of a larger effort described at http://www.flickr.com/commons/

Saxon joy!

2008-08-26T22:14:00.003+12:00

I've just moved to saxon from libxml for some XSLT stuff I'm doing, and I'm really loving it.

Not only does saxon run take much less memory, it also speaks XSLT 2.0.

kowhai flowers at waikanae

2008-08-10T08:42:00.001+12:00

kowhai flowers at waikanae
Originally uploaded by Stuart Yeates

moving back to google reader from bloglines

2008-08-05T21:54:00.005+12:00

A couple of months ago I migrated to bloglines from google reader, not because I was necessarily unhappy with google reader, but because I was interested in seeing what else was available and how it might differ. I've just moved back to google reader.

OMPL just worked. I was able to move my RSS "reading list" from google reader to bloglines and back again with no fuss, no hassle and no duplication.

The advantages of google reader over bloglines are:

AJAX - whereas bloglines marks all items on a page as read when you browse to it, google reader marks them as read when you scroll past them.
Ordering - google entwines items from all feeds in time order, bloglines presents items feed by feed
Better integration with other services

The advantages of bloglines over google reader are:

Fast scanning of voluminous feeds
Fast browsing (it seems _much_ faster when there are thousands of items)
Less integration with other services

You'll notice that better integration is both a positive and a negative.

The fact that I have several google accounts and and only one of them is tied to my RSS reading means that there are tasks I can't multi-task between, even at the coarsest of levels and also means that contacts from the google account almost never get forwarded articles I discover via RSS.

The fact that my blogger.com account and my google reader accounts magically know about each other is great, as is being able to sign in once to a whole suite of tools.

In the end the reason for changing back was ordering. I read too many RSS feeds that cover the same topic for reading them out of order to make sense.

I've also just culled some of my RSS feeds, with the a prime criterion being the quality of their RSS. A number of web comics require one to click a link to read the strip and I no longer read them, but I still read Unshelved, which has the strip (and an ad) in the RSS.

Decent editor for blogger.com?

2008-08-04T21:45:00.002+12:00

Can someone recommend a decent replacement for the default editor for blogger.com?

Before it drives me insane...

KDE/Gnome Māori localisation on the rocks?

2008-08-04T19:55:00.004+12:00

It looks like Maori localisation has been removed from the KDE 4.0 repository:

stuartyeates@stuartyeates:~/tmp/mi$ svn co svn://anonsvn.kde.org/home/kde/trunk/l10n-kde4/mi/messages
svn: URL 'svn://anonsvn.kde.org/home/kde/trunk/l10n-kde4/mi/messages' doesn't exist
stuartyeates@stuartyeates:~/tmp/mi$ svn co svn://anonsvn.kde.org/home/kde/trunk/l10n-kde4/mi/docmessages
svn: URL 'svn://anonsvn.kde.org/home/kde/trunk/l10n-kde4/mi/docmessages' doesn't exist
stuartyeates@stuartyeates:~/tmp/mi$ svn co svn://anonsvn.kde.org/home/kde/branches/stable/l10n-kde4/mi/messages
svn: URL 'svn://anonsvn.kde.org/home/kde/branches/stable/l10n-kde4/mi/messages' doesn't exist

Things don't look good for the upcoming 4.* releases, with the stats for translation at 0%: http://l10n.kde.org/stats/gui/trunk-kde4/team/

Gnome Māori localisation is not much better: stable at 1%: http://l10n.gnome.org/teams/mi

In the medium/long term there is hope that much of this localisation can be bootstrapped by application-centric localisation that appears to be thriving, particularly with respect to firefox, thunderbird and OOo.

Leaving catalyst :( joining NZETC :)

2008-08-03T18:54:00.004+12:00

Last week I gave notice at my current employer (Catalyst.net.nz) and accepted a job at Victoria University's New Zealand Electronic Text Centre. The NZETC is primarily a TEI/XSLT/Cocoon-house which publishes digital versions of culturally significant works. It also runs a number of other digital services for the university library (into which it is currently being integrated). As such it's significantly closer to what I've been doing previously in terms of environment, content and technology.
Exciting things about the NZETC from my point of view:

We Will Not Cease by Archibald Baxter, a first hand account of conscientious objection in the first world war by the father of New Zealand's greatest poet
Works by John Cawte Beaglehole, including letters that mention him meeting my grandfather ("Jack Yeates") at Cambridge
Work with Learning Media, (who I've previously published poetry with) on their publishing process
Extensive use of the New Zealand Creative Commons Attribution Share-Alike License (CC BY-CA) license for works

The commute to work will be slightly longer, with me either getting off the bus one stop earlier and catching the cablecar up the hill, or getting off at my current stop and walking up. I'm hoping to do mainly the later.

Who should I nominate for the NZ Open Source Awards?

2008-07-09T21:11:00.003+12:00

So nominations are open for the New Zealand Open Source Awards and I have to decide who I should nominate. There doesn't seem to be anything stopping me nominating several, but picking one contender and throwing my weight behind them seems like the right thing to do. The ideas I've come up with so far are:

Kiharoa Dear for excellent work in getting firefox, thunderbird and open office working in Māori contexts:

http://kiharoa.dear.maori.nz/

Standards New Zealand for sanity control in the OOXML fiasco:

http://www.standards.co.nz/news/Media+releases/NZ+maintains+negative+vote+on+OOXML+Standard.htm

Hagley Community College for rolling out Ubuntu in a secondary school:

http://computing.hagley.school.nz/about/opensource

Who should I nominate? Is there someone I've missed?

I'm confused about hardy heron and default applications

2008-06-30T20:13:00.001+12:00

Back in the day you told your linux system which applications you wanted to use with environmental variables things like:

export EDITOR=/usr/bin/emacs

Then along came the wonderful debianness of the apt-family and the alternatives system.

update-alternatives --config vi

Now this system too is being undermined by various systems, leaving me uncertain where to set things. What I'm trying to do is:

have Sound Juicer and not Music Player (RhythmBox) launched when a CD is inserted. There is an entry for "Multimedia" under the "Preferred Applications" menu option, but this seems to be about opening files, not responding to newly-mounted media and Sound Juicer is not listed as an option. There doesn't seem to be anything about CDs under the "Removable Drives and Media Preferences" (although this is where the setting are that automatically load F-Spot when I attach my camera, which seems like the same kind of thing).
configure which applications I can launch on the .cr2/TIFF/Canon RAW files produced by my digital camera I want the same applications to appear in both the file browser and F-Spot (which look like they're presenting the same interface but apparently aren't). ufraw seems to be the tool of choice here (either standalone or as a gimp plugin), but I'd like to pass it some command line args. I can find no entry or this under the "Preferred Applications" menu option.

There are lots of menus with a "Help" as an option, but very few of them seem to be.

Mike O'Connor at Friday drinks

2008-06-30T16:34:00.002+12:00

Mike O'Connor
Originally uploaded by Stuart Yeates

I took some photos at Friday drinks, trying to do the whole wide-aperture-to-isolate-visual-elements thing. I wasn't really aware of just how much it is dependent on the relative position of the photographer, subject and background.

Some of them turned out better than others.

What should the ohloh homepage look like?

2008-06-29T13:01:00.002+12:00

In a previous post I criticised ohloh homepage for being completely useless for current users of the site. This was somewhat unfair, since I provided no concrete constructive suggestions as to what should be on the page. This blog post, hopefully, fixes that.

To my mind there are two classes of information that should be on the homepage: (a) things that lots of users are confused about and (b) things that are 'new' (think customised rss feeds) (c) combinations of both.

Finding out what people are confused about is easy, just look in the forums, where people are most confused about:

changes in their kudos
why their enlistment hasn't been updated
why version control system of choice isn't supported

The list of 'new' things is:

new / updated projects
new / updated users
new / updated enlistments
new forum posts
new RSS items in projects RSS feeds

These (1), (3) and (5) can be filtered by the users connection to the project (contributor/user/none).

So the trick now is to find combinations which help users understand what's going on and encourage users to engage with ohloh and the projects.

Idea X: A feed of updated enlistments a user is a contributor or user of:

Project A's enlistment at http://example.com/git updated 24st June 2008 at 24:50 GMT. A, B and C are the biggest commiters to this project, which is in Java and XML. Last updated 1st Feb 1970.
Project B's enlistment at http://example.com/git updated 24st June 2008 at 24:50 GMT. D, E and F are the biggest commiters to this project, which is in C and shell script. Last updated 1st Feb 1970.
Project C's enlistment at http://example.com/git failed at or about revision 12345. Click here for instrustions on what to do about this. Last updated 1st Feb 1970.
...

This not only tells user the status of their projects, but that enlistments are being processed, the expected time between each processing of enlistments, that some processing fails and that there's a link to find out more information. Such a feed also focuses attention on the processing of enlistments---which is the heart of ohloh and the key differentiating factor that seperates ohloh from 15 billion other open source sites.

Idea Y: A mixed feed of upstream bugs that effect ohloh performance and functionality:

Ticket "support for .xcu file format" updated in ohcount by user "batman"
Post "jabber message length" updated in Help! forum by user "someone else"
Ticket "svn branch support" updated in ohloh by user "robin"
Ticket "bzr support in ohloh" created in ohloh by user "joker"
Post "jabber message length" created in Help! forum by user "someone"

This lets people keep up with the status of ohloh progress on issues such as the implementation branch support for svn and support for hg.

New ohloh look and feel

2008-06-23T18:27:00.000+12:00

ohloh have changed their look and feel, and I've got to say I hate it.

Once you're logged in, almost nothing above the scroll cut on the front page is useful---we already know what ohloh is and don't need bandwidth-hogging ads to tell us. What we need are deep links into new stuff---projects, users and forum posts.

How about logged in users see content rather than ads on the homepage?

Kernel Hell and what to do about it

2008-06-15T11:07:00.003+12:00

I've been in kernel hell with my home system for the past couple of days. What I want to build is a custom kernel that'll do xen, vserver, vmware, selinux, support both my wireless chipsets and support my video chipset. Ideally it should be built the Debian/Ubuntu way, so it just works on my Ubuntu Hardy Heron system.

So far I've had various combinations of four or five out of six working at once.

I'm not a kernel hacker, but I have a PhD in computer science, so I should be able to at least make progress on this, and the fact that I can't is very frustrating. At work I grabbed a kernel off a co-worker, but it wasn't built the Debian/Ubuntu way.

Standing back and looking at the problem, there seem to be two separate contributing factors:

There are a huge number of organically-grown structural layers. I count git, the kernel build scripts, make, Linus's release system, the Debian kernel building system and the ubuntu kernel building system. I won't deny that each of these service a purpose, but that's is different points a which each of the six different things I'm trying to make work can begin their explanation of how to make them work and six different places for things to go wrong.
There are about many Linux distributions and each of the things I'm trying to get working caters to a different set of them.

In many ways the distribution kernel packagers are victims of their own success, most Ubuntu, debian and RedHat kernels just work because they're packagers keep adding more and more features and more and more drivers to the default kernels. With the default kernels working for so many people, fewer and fewer people build their own kernels and the pool of knowledge shrinks. The depth of that knowledge increases too, with the each evolution of the collective build system.

Wouldn't it be great if someone (ideally under the auspices of the OSDL) stepped in and said "This is insane, we need a system to allow users to build their own kernels from a set of <git repository, tag> pairs and a set of flags (a la the current kernel config system). It would download the git repositories and sync to the tags and then compile to the set of flags. Each platform can build their own GUI and their own backend so it works with their widget set and their low level architecture, but here's a prototype."

The system would take the set of repositories and tags in those repositories and download the sources with git, merge the results, use the flags to configure the build and build the kernel. Of course, sometimes the build won't work (in which case the system sends a copy of the config and the last N lines of output to a central server) and sometimes it will (in which case the system sends a copy of the config and an md5 checksum of the kernel to a central server and optionally uploads the kernel to a local repository), but more than anything it'll make it easy and safe for regular users to compile their own kernels. The system would supplant "building kernels the Debian way" or "building kernels the RedHat way" and enable those projects working at the kernel level to provide meaningful support and help to their users on distributions other than slackware.

Potential benefits I can see are:

increasing the number of crash-tolerant users willing to test the latest kernel features (better testing of new kernels and new features, which is something that's frequently asked for on lkml)
easing the path of new device drivers (users get to use their shiny new hardware on linux faster)
increasing the feedback from users to developers, in terms of which features people are using/interested in (better, more responsive, kernel development)
reduce the reliance on linux packagers to release kernels in which an impossible-to-test number of features work flawlessly (less stressed debian/ubuntu/redhat kernel packages)
ease the path to advanced kernel use such as virtualisation

You know the great thing about that list? Everyone who would need to cooperate gets some benefit, which means that it might just happen...

Macrons and URLs

2008-06-15T10:38:00.003+12:00

Macrons are allowed in the path part of URLs, but not currently in the machine-name part (or at least, not yet), so http://extensions.services.openoffice.org/project/māori-papakupu is good, but http://www.taiuru.Māori.nz/ is not (use http://www.taiuru.maori.nz/).

A review of how lots of programs handle macrons is at http://research.elabs.govt.nz/macron-support-in-open-source-web-applications/.

Exporting firefox 3.0 history to selenium

2008-06-14T09:06:00.005+12:00

In the new firefox 3.0 they've completely changed he way history is recorded, using a SQL engine to record it (details here).

I wrote a quik hack to export the history as a series of selenium tests:

sqlite3 .mozilla/firefox/98we5tz3.default/places.sqlite 'select * from moz_places' | awk -F\| '{print "<tr><td>open</td><td>"$2"</td><td></td></tr>" }'

sqlite3 locks the file, so you'll need to close firefox (or take a copy of the file) first. Cut and paste the results into an empty selenium test.

Obviously, your profile will have a different name.

OpenID - Everybody Wants To Go To Heaven, Nobody Wants To Die

2008-05-04T18:23:00.004+12:00

I've been looking recently at OpenId a little more closely. It's a great system, that I've been using for a number of years.

My first free openid provider (http://stuartyeates.videntity.org/) went belly up for reasons I'm still not too sure about, but the life expectancy of internet startups and free services has never been that long. I have a new identity (http://stuartyeates.myopenid.com/) which does me just fine.

One thing I've been noticing is that while a host of the internet services that I use want to be my identity provider, very few of them want to let me login using an openid provided by another identity provider. From a business point of view I can completely understand this: they want to know everything about the user so they can (a) off better, more integrated services and (b) see more customised advertising. I have no problem with (a), but (b) is one of the reasons I'm eying up openid in the first place.

They don't understand that to get to openid heaven, they're going to have to die. By giving away the user-authentication-and-give-us-your-personal-information step, they can drive significantly more logins and significantly deeper interaction with their website and with their content. Sure, they can't necessarily get access to the users email address and whatever fake personal details they submitted at sign up, but I've yet to see this used particularly well anyway. The personal details certainly don't seem to be used to anywhere near the same effect as analysis of user behaviour.

What would be really good is an openid identity provider with (a) shared no information with any of the large advertising groups or dominant internet companies, (b) really clear information about privacy expectations (and I'm not talking here about a page of legalese titled "privacy policy," but discussion and disclosure of things which jurisdictions user data is stored in, steps taken to avoid collection of user data, etc) and (c) a clear sustainability model to prevent it being bought-out for the user data and to ensure it's continuity (I can imagine paying a subscription for it).

I can imagine that this is the kind of thing that google or yahoo might sponsor in their efforts to promote their next generation of web authentication standards. The existance of such an outfit would hugely boost the reputation of such a standard and lacking the huge advertising and existing lockin it would hardly challenge their own identity providers while answering a whole range of independence, privacy and monopoly questions.

Tinkering with ohloh

2008-05-04T10:04:00.001+12:00

I've been tinkering with www.ohloh.net, which is probably best described as a web 2.0 freshmeat. Rather than tracking manually-updated releases it relies on automatically detected updates to version control repositories, RSS and geo-urls. It relies on wiki like editing rather than the strict ownership rules of freshmeat. ohloh does automatic language detection and source code analysis based on the version control repository and attributes individual commits to specific developers and their ohloh user account.

I've added and am actively curating a group of go/baduk projects. The overall goal is to encourage reuse and reduce the willingness of hackers to rewrite go/baduk systems from scratch.

My next step on the technical side is to write some GRDDL (Gleaning Resource Descriptions from Dialects of Languages) to transform the XML returned by the API into RDF, which I can them import into simal.

My next step on the social side is to mention what I'm doing in some of the go/baduk mailing lists, but I want to wait until I've got something concrete to provide that Sensei's Library (the current repository of information about go/baduk programs) hasn't already got.

Happiness is tests passing

2008-05-02T22:46:00.002+12:00

All tests passed
Originally uploaded by Stuart Yeates

2 tests failing

I've just (re-)attained the happy state of the unit tests passing in the key package of jgogears. I've invested a surprising amount of time and energy in the unit tests, mainly as a form of requirements analysis, and I'm really pleased with the result.

Sadly, the core package that contains much of the code has some outstanding, longstanding failures which are going to be challenging to fix. They are represent failures in jgogears' ability to round-trip board states between GNU-Go ASCII and SGF files and will require a bug-for-bug reimplementation of the GNU-Go ASCII board printer.

jgogears has reached the stage where it now plays games that are obviously meant to be go, but is not yet a serious contender.

Resetting vmware networking

2008-04-18T12:10:00.003+12:00

I had a hardware glitch (apparently chained failure of on-board network card, auto-negotiation on switch, auto-negotiation on larger switch) which left networking on my ubuntu edgy box somewhat randomised. Ubuntu recovered after a reboot, successfully finding the new network card (but calling it eth11). vmware-server didn't recover, and no amount of fiddling with the magic of vmware-config-network.pl would cause vmware to forget all it knew about networking and look only at the current state of play.

In an attempt to erase it's memory I removed /etc/vmware/locations but this caused vmware-config-network.pl not to run at all.

Eventually I worked out that the trick was to remove everything in /etc/vmware/locations other than the lines near the start telling it where to find directories and executables and rerun vmware-config-network.pl selecting all the defaults.

Suddenly everything worked.

Quality explanations of technical issues

2008-03-20T08:28:00.002+13:00

I'm a big fan of clear explanations. If you want to explain something to someone (and given the alternative is letting everyone learn from their own mistakes, this has got to be good), clear explanations are really important. I've tutored computer science at uni and I've explained open source concepts to a whole range of people as part of my work at OSS Watch and come to learn that an analogy can be very useful.

Imagine my pleasure at reading this analogy of a really rather complex compiler / interrupt issue.

For the record, I know nothing about the x86 interrupts, since we were taught interrupts using the much simpler RISC SPARC system.

Tinkering with suffix-trees and algorithms

2008-03-18T22:47:00.001+13:00

I've been tinkering with learning algorithms for my computer-go player, jgogears.

It linearises board positions and then uses classic string processing techniques, principally a large suffix-tree. Suffix-trees are widely used in information processing, information theory and compression fields of computer science. I also used them extensively in my recent Ph.D.

Currently I'm training with about 200 go games (~40k moves), giving me about 950K nodes in my suffix tree.

I've just switched my linearisation method from a strict distance measure to one which capitalises on adjacency much better.

There are a number of tuning parameters for the rate at which I grow the tree. I'll be tinkering with them as I increase the number of boards I'm using for training.

RedHat's MugShot and lockin

2008-03-14T12:09:00.001+13:00

I've been playing with MugShot, the RedHat's venture into the social networking sphere. I was initially impressed by the site, which has builds a dynamic (if HTML-only) website while asking for really very little information and not asking for any confidential information (i.e. no asking of gmail passwords to pester friends to join up). The website is slick and glossy, if a little heavy on my dialup connection.

The more I use MugShot, however, the more I see it supporting rather than undermining lockin in the social networking sphere.

1 Login with Gnome account or local username/password

While many of MugShot's Gnome account initial target audiance may have had Gnome accounts suitable for logging into to MugShot, I'm guessing that a really small proportion of people they knew did. Login using openid or similar would be great.

2 Supporting only a small number of services, and not flexibly

It's great that mugshot supports delicious. However, by only supporting a single service of this kind, not only are my magnolia bookmarks unsupported, but the current delicious control over the social bookmarking market is strengthened. I understand that the differences between the delicious and magnolia APIs are largely cosmetic, and find it hard to believe that much effort would have been required to support it. Similarly, there are a number of exciting competitors to youtube, amazon, facebook and other MugShot supported services which could be supported with very little time and effort on the part of MugShot.

The services that are added, use a one-size-fits all approach, when I know of no two long-term users of (for example) facebook who use it in the same way.

3 No machine-readable export

There is (that I can see) no constructive machine readable export of any kind from MugShot. Two basic RSS feeds (which aren't advertised in the GUI). No RDF/FOAF, no blogroll for integration into the next generation of social networking and web services. MugShot may see itself as the top of the social networking heap, but until that's evident to the rest of us, they need to play nicely with third parties, both below them and above them on the heap. Failing to export anything useful in a machine-readable format is not playing nicely.

4 No "deep connection" back to the data sources

Having configured MugShot, I can see lots of books, photos and links that those in my network find interesting. But even though Mugshot can tell the difference between books, photos and links when it's displaying them and knows about the accounts I have on services for bookmarking, favouriting or commenting on books, photos and links it doesn't offer functionality to bookmark, foavourite or comment on them. By not driving content and information back to the underlying datasources, MugShot undermines the underlying services and devalues them, in so doing it also (a) devalues the commitment and investment I've made to those systems and (b) reduces the likelihood that those services will go out of their way to help MugShot.

5 "Invitation to tinker" only covers look-and-feel

MugShot has a great system for creating skins for music players, and several hundred people appear to have accepted this invitation to tinker and made come great skins. Unfortunately only cosmetic changes are possible. What I would like to see is a generic feed subscription creator which let me add new services that MugShot could listen to. Just like the music player skins, only a very small proportion users would bother, and most of those who did would not produce anything to shout about, but with a feed subscription creator, each success would lead to a whole new service that Mugshot could access and channel to their users. Such a effort would completely alleviate issue problems with the small number of supported services.

MugShot has an invitation to tinker in the traditional open source web 1.0 sense, with a wiki and version control system, but it doesn't have an invitation to tinker in the web 2.0 sense in which users can scratch their itch (and make the project better) from within their browser in the way the yahoo pipes, for example, does.