Programming

unAPI format and Semantic Web

2009-05-24T04:59:00.000-07:00

Ross Singer's recent post about "One Data Format Identifier (and Registry) to Rule Them All" caused some interesting discussions. I didn't read all of them, and don't have much to add, but I do have a chance of reflecting how unAPI format wiki was created at the first place.

It turned out that I always have trouble of understanding why Semantic Web is so obsessed with URI and Ontology. I am not Semantic Web expert, but to me the strict URI approach seems to directly conflict with other "convention over configuration" approaches like tagging, wiki, twitter. And it doesn't seem everyone will have the time to learn each other's ontology anyhow. As a result, any sufficiently large RDF file always make my head spin because of all these long URIs, and I am not sure how much bandwidth were used to carry them on the Internet.

So why don't just use a word as the identifier, and let everyone pickup a dictionary and find out its semantic. I guess Oxford dictionary is a better agreed-upon ontology. In retrospect, perhaps this is how unAPI format differs from SRU/OpenURL, we can choose a name and hopefully it will work. Does it work? I don't know.

Since multiple copies keep stuff safe, I also paste here the unAPI format retrieved from Internet Archive.

name type example desc doc
amazon application/xml opa a convention used by OPA
asn1 text/plain opa Abstract Syntax Notation One asn.1
bibtex text/plain hubmed bibtex bibtex
dc text/plain opa unqualified Dublin Core
didl application/xml TODO MPEG-21 DIDL didl
endnote text/plain refbase endnote endnote
latex application/x-latex refbase latex latex
marcxml application/xml Technosophia MAchine Readable Cataloging in XML marcxml
markdown text/plain refbase markdown markdown
mods application/xml Technosophia Metadata Object Description Schema mods
html text/html refbase HTML HTML
oai_citeseer application/xml opa
oai_dc application/xml Technosophia unqualified OAI Dublin Core oai_dc
pdf application/pdf refbase Portable Document Format PDF
pubmed application/xml opa pubmed article pubmed
rdf/xml application/rdf+xml hubmed Resource Description Framework RDF
ris text/plain hubmed ris ris
rss application/xml Technosophia Really Simple Syndication RSS
rtf application/rtf refbase Rich Text Format RTF
srw_dc application/xml Technosophia unqualified SRW Dublin Core
srw_mods application/xml refbase unqualified SRW MODS
text text/plain opa
wrap application/x-javascript opa unalog json format

cannonical URLs in tag

2009-02-27T07:04:00.000-08:00

Having observed a few canonical URL solutions in Digital Library world, such as Handle, PURL, DOI, it is really nice to see the cannonical URLs from Google. Google partially solves the problem by simply adding a <link rel="canonical" href=""> to specify your preferred version.

It always surprises me how elegant and simple a solution can be, and how long it takes to find these simple solutions.

canonical URL

2007-05-22T20:14:00.000-07:00

Just came across these URLs

http://www.google.com/search?hl=en&q=link%3Ahttp%3A%2F%2Fworldcatlibraries.org%2Foclc%2F65187392&btnG=Search

and

http://www.google.com/search?hl=en&q=link%3Ahttp%3A%2F%2Fworldcat.org%2Foclc%2F65187392&btnG=Search

I was trying to compare same Worldcat pages with different URLs, by google backward link search features (http://www.google.com/advanced_search)

So http://worldcat.org/oclc/65187392 and
http://worldcatlibraries.org/oclc/65187392
are same page, however due to variants of these URLs, Google treats them as different URLs, and this certainly doesn't help moving up these pages up in Google search results, if we consider the inbound link factor in PageRank.

new xISBN API - beta

2007-03-02T18:58:00.000-08:00

I have been working on a new xISBN service in last few months, and I had a chance of sharing latest progress in excellent code4lib conference held in Athens, Ga. Thanks for all your valuable comments, folks!

The new service is now running as a beta version. We add more features/metadata, at the same time try to keep the API simple and clean.

x-identifier list is a venue for comments/suggestions.

things learned from SOLR api

2007-02-27T20:25:00.000-08:00

things learned from SOLR api

I am fortunte to attend Erik Hatcher's one day Lucene/Solr tutorial, what's really attracting me is elegance of SOLR's REST API, which is at the same time concise and consistent. To start with:

-- add/query/delete/commit are all simple REST API.

as an example, delete a record is a simple HTTP POST "<delete><id>SP2514N</id></delete>", do I need to say more?

-- allows different format (but equavalent infoset)
"select?q=ipod&wt=xml" -- returns an xml format
"select?q=ipod&wt=python"
"select?q=ipod&wt=ruby"
"select?q=ipod&&wt=json"

They all return same infoset.

-- the way of controlling returned fields by query

q=video&fl=name,id (return only name and id fields)
q=video&fl=name,id,score (return relevancy score as well)
q=video&fl=*,score (return all stored fields, as well as relevancy score)

So user has control of which fields to return.

-- no xml namespace at all

This is subject to argue, but I tend to think in RPC-oriented application XML namespace doesn't really matter, and in Document-oriented applications XML namespace are important.

-- the way of organizing schema and result.

Basically everything is a field, a field has a datatype, name and value. Such as following result:

<str name="id">IW-02</str>
<bool name="inStock">false</bool>
<str name="manu">Belkin</str>

This bascially allows any fields without XML namespace/schemas.

google's moon shot

2007-01-30T20:34:00.000-08:00

New Yorker magazine runs an article "google's moon shot" about google book project. It mentioned Google aims to scan at least as many as worldcat records (32M) in 10 years. And it also mentioned the hate/love relationship between publishers and Google, and seems optimistic that the copyright issue can be resolved this way or other. There are also interesting thoughts of networking books, similar to the way of how web pages are linked together by hyperlink.

Ebook has been here a while and we are still trying to figure out the right way, maybe Google, Open Content Alliance, and other's work can make a big difference, we will see how this evolves in next few years

microsoft live book search with fulltext download

2007-01-20T07:52:00.000-08:00

I just have a chance of looking at Microsoft live book search demo , it's still a small collection, so search results can be confusing. However an entire book can be downloaded from archive.org as a PDF file, such as this one:

http://www.archive.org/download/writingsofcharle01dickuoft/writingsofcharle01dickuoft.pdf

I guess this is part of deal of Open Contents Alliance. Google Print doesn't really allow download of a full book.

convert between 13 digits and 10 digits xisbn

2006-11-07T14:33:00.000-08:00

self note.
(a) from 10 digit to 13 digits

if 10 digits takes the format ABCDEFGHIJ, 13 digits is going to be: 978ABCDEFGHI(?), and last digit (checksum) can be calcualted by:

(10-((9*1)+(7*3)+(8*1)+(A*3)+(B*1)+(C*3)+(D*1)...+(I*3)) modulo 10) modulo 10

(b) from 13 digits to 10 digits
if the first three digits is not "978", it cannot be converted to 10 digits.

if the first three digits is "978", takes remaining 9 digits: ABCDEFGHI, and calcuate checksum for 10 digits ISBN:
(11-((A*10)+(B*9)+ .... +(I*2) modulo 11)) modulo 11

10 will be replaced by X

reference:
[1] http://wiki.tcl.tk/12638
[2] http://www.isbn.org/standards/home/isbn/transition.asp

googlemap api and craigslist

2006-10-22T21:34:00.000-07:00

It's the right season to find an apartment again ;-) this time I am mainly using craigslist to look for apartment. Fortunately, most rentals have a google map link, but after a while, it becomes really difficult to keep track an interest list.

For a while I want to try the goolgemap API and this is the right time to do so, it turns out the API is really handy. I need to register a googlemap API account, after I just need follow some javascript code examples in googlemap homepage, in less than 2 hours I can conveniently compare interesting apartments through the nice google map interface.

I still need manually populate craigslist information into my small application -- it would be great if craigslist have a built-in function to create a "my bookshelf" kind of service.

oaiarc new release

2006-10-15T20:12:00.000-07:00

oaiarc-1.0 is just released, this release includes following changes:

--internationalization support (with English, Spansih, French built-in)
--strict XHTML compliant
--UTF-8 support
--remove oai 1.x support
--various bug fixes

Juan Corrales from Universidad Rey Juan Carlos de Madrid contributes
most improvements in this release.

pathways paper

2006-10-13T14:22:00.000-07:00

The last two years pathways research result are published:

An Interoperable Fabric for Scholarly Value Chains - D-Lib, October 2006 (link will be active early next week)

Pathways: Augmenting interoperability across scholarly repositories - International Journal of Digital Libraries

dump a web page and its links by wget

2006-10-13T14:14:00.000-07:00

self-note:

Every few weeks I need to read wget manpage to find parameters to cache a single webpage and all its links, so perhaps better write it down here to save 5 minutes:

wget -r -H -l1 -k -P $targetdir --exclude-domains ${comma-seperated domain name} --user=xxx --password=xxx $url

Poincaré conjecture and Shing-Tung Yau

2006-09-06T20:51:00.000-07:00

it was sad to read recent New Yorker story of Poincaré conjecture and Shing-Tung Yau. Yau won Fields award and has greatly contributed to mathmatics study, his opinion (in general) is very well respected and cited in China.

I believe this article must have exaggerated some information, as evidenced by clarifaction from some mathematicians recently, still, it is really difficult to know when to stop, even as brilliant as Yau. So this is really a sad and disappointed story.

notes in Fedora User Conference 2006

2006-06-25T21:46:00.000-07:00

I took some notes in the Fedora user conference 2006. Disclaimer: I am not a Fedora user; and the conference doesn't have proceedings and presentation is not online -- so read with caution.

The conference is nice but somehow difficult to summarize because many presentations are project-oriented. So I will try to summarize them in several perspective: (a) core functionalities (b) interesting project (c) some observations and thinkings.

(a) core functionalities

RDF triple store: A very interesting part of Fedora is its usage of RDF and triple store, as I have always been interesting in the scalability of RDF triple store. Dean B. Krafft of Cornell gave some interesting numbers about NSDL 2.0 usage of Fedora. NSDL 2.0 handles some ~2M digital objects, with 70 RDF tuples for each object, overall Kowari triple store has 163 Million triples.

Content model: The ongoing development of Fedora Core includes "content model", such as structure/ontology for thesis, article, etc, and dynamic dissemination associating with "content model". The question is how to reach agreement on common ontology.

Dissemination/behaviour: This seems like a pretty active area, it's about given a digital object, how can you associate services with this object. DLF Aquifer project has an interesting concept of "AssetActions", which defines an XML schema of associating behaviors with object, initial experiment based on Fedora.

(b) projects

This is perhaps the most interesting part of the conference, there are several ambitious, national project using Fedora as core service, including NSDL, German's eSciDoc, Aussie's ARROW, and DART, and some works by Harris Group. These projects have a theme of building workflow system for scholarly communication in a large scale. Although information can be overwhelming in some cases, it's really a good headstart to take a further look at these initiatives.

(c) thoughts

RDF store or RDBMS?

an old question, but still relevant and interesting

What's essential functionalities of a repository?

Is it OKI, JSR170, Fedora core API ?

A nice graph of describing repository:

Repository->services->applications->interfaces
---------------------------------------
when moving left to right, the changes are more likely, repository is expected to be stable.

New scholarly communication system

I think we definitely need some models and researches in this area, such as this and this

Liveclipboard copy beyond microformats -- unapi 1.0 released

2006-06-25T20:06:00.000-07:00

Dan Chudnov recently released unAPI version 1. Here I particularly want to compare unAPI with microformats in liveclipboard implementation.

The MS liveclipboard demo is based on microformats, essentially, the demo uses some smart javascripts to copy XHTML page between different web pages. In principle, the technology can be used to copy any XML fragment, or even binary files if you consider base64 encoding.

Microformats is well suited for this purpose because its XHTML fragments are well-formatted XML. However, there are many applications/domains are not covered by microformats: (a) not all contents can be described by XHTML, such as tremendous XML standard/contents in the web, or many non-HTML content in the web. (b) even a content can be described by XHTML, it may never reach the radar of microformats.org, or never have enough incentives to make them standard.

So here I think unapi can fill a gap, because it allows copy of any content, far beyond the scope of microformats.org, from initial implementation of unapi we saw mods, dc, json, pubmed, rdf, text, and we can expect more diversified formats in the future. All these formats may eventually become payload of liveclipboard, and the application is up to your imagination.

On the other side, the approach of unAPI is more complex than microformats. In microformats you can simply markup an XHTML page and it's all done. In unAPI one have to markup an XHTML page and implement a simple API. And the little tradeoff can be worthwhile if we want to take full advantage of liveclipboard.

Stuff I like in JCDL 2006

2006-06-14T16:07:00.000-07:00

some interesting readings in JCDL:

"Also By The Same Author: AKTiveAuthor, A Citation Graph Approach To Name Disambiguation. " Duncan M. McRae-Spencer, Nigel R. Shadbolt

This paper describes how to use author's self-citation (for good or bad, anyhow another topic) to implement name disambiguation. For example, if a paper has an author "X. Liu", and again I cite a paper with author "X.Liu", it's quite likely that the two "X.Liu" are same person. By taking advantage of this social context, the paper got starling precision/recall in name disambiguation.

"Bibliometric Impact Measures Leveraging Topic Analysis", Gideon Mann, David Mimno and Andrew McCallum.

Andrew McCallum wrote the popular bow/rainbow text classification package. This is the first time I saw his paper in JCDL. The paper is based on a new cluster method TNG, which can label clusters by phase, instead of individual words, e.g. "text classification" can be used to label a cluster, instead of "text" and "classification". This is extremely powerful in labeling cluster. After that the paper proposes several impact measures of topic, particularly life cycle of how a subject/topic emerges, develops, and influences other topics. Pretty interesting reading and solid work.

"Building a Research Library for the History of the Web", William Y. Arms, Selcuk Aya, Pavel Dmitriev, Blazej J. Kot, Ruth Mitchell, Lucia Walle

I blogged this work before, the project tries to mirror and mine whole Internet Archive. Anything dealing with that level of scalability is worth checking out.

"Metadata aggregation and automated digital libraries: A Retrospective on the NSDL experience", Carl Lagoze, Tim Cornwell, Naomi Dushay, Dean Ecktrom, Dean Krafft

Some remarkable lessons of OAI-PMH in a very distributed system. It also wins best paper award. Loosely distributed system is always a hard question, especially you are targeting transaction-level quality, no matter distributed search or harvesting.

"EcoPod: A Mobile Tool for Community Based Biodiversity Collection Building" YuanYuan Yu, Jeannie A. Stamberger, Aswath Manoharan, Andreas Paepcke

A PDA-based application for biology species observations, it is not complex or abstract, and it focuses on a simple task and solve it well. Maybe that's how the research should be done in many DL projects: do one thing and do it well.

"An architecture for the aggregation and analysis of scholarly usage data." Johan Bollen and Herbert Van de Sompel

It describes using OAI-PMH to harvest usage data, which is embedded in OpenURL ContextObject. There are also interesting result of mining these usage data. I think the choice of OAI-PMH/OpenURL are very much appropriate here.

unapi scripts, and a wiki page

2006-05-21T21:08:00.000-07:00

I updated several thing regrading unapi:

-- fix a bug in unapi userscript

-- update a proxy demonstration to unapi revision 3.

-- I started a wiki page in http://unapi.stikipad.com/, although a personal opinion, I think it is a good idea to document commonly used format in unapi, largely motivated by a similar microformats list. Feel free to update the wiki page if you have comments/ideas.

2006-05-19T13:32:00.000-07:00

unapi greasymonkey scripts updates to revision 3, the URLs are same so you may follow previous instructions at: unapilink,
citeseer and pubmed, and amazon.

2006-05-18T15:17:00.000-07:00

DIDLTools package online

The DIDL Tools is just released http://african.lanl.gov/aDORe/projects/DIDLTools/, as posted in release page:

"The aDORe DIDLTools is a Java toolkit for the construction, validation, serialization and de-serialization of the MPEG-21 DID data model. DID, the MPEG-21 Digital Item Declaration, provides an abstract model for the representation of digital items, whereas DIDL, the MPEG-21 Digital Item Declaration Language specifies how to serialize the model in XML. The API provided by the aDORe DIDLTools allows for the construction of customized DIDL XML documents, as well as provides flexible and extensible serialization methods."

There was some good thinking of allowing any metadata in the model, have a look and ask questions in http://lists.lib.ugent.be/mailman/listinfo/didwriter-dev

2006-05-16T14:26:00.000-07:00

slides of AISTI Mini-Conference

The AISTI Mini-Conference has its slides online, I was in the conference and there were several pretty good presentations.

Douglas Fils made a presentation about Chronos project, titled Services Architecture and Semantic Practices Using the CHRONOS Cyber-Infrastructure Effort, the project combines some cool web2.0 technology to build Chronos environment.

Alex Szalay presented Science in an Exponential World, which overlaps library world with data curating. I pretty much like his remarks that data must be close to analysis tool and we need journal for data. His work is also assocated with microsoft 2020 science initiative.

2006-05-10T21:32:00.000-07:00

arxiv and trackback

I recently listened Paul Ginsparg's interview by CNI, it touches history and ongoing development of arXiv.

This is the first time that I heard arXiv has trackback feature, see http://arxiv.org/help/trackback, as always, arXiv does everything differently (no kidding, I am a regular arxiv user), and seems there was a hot debate about its policy a while ago in physics blogsphere.

I then quickly checked hubmed and WPopac, two blog-style library service, it's no suprise that both support trackback.

However, arXiv is unusual, because its interface/functions is stable for quite some years, it shows what's quality and good design. Now it has a trackback, the fact shows quite nicely how new techologies can be integrated with established library service.

2006-05-06T18:16:00.000-07:00

WebCite

Just read WebCite from Lorcan Dempsey's blog.

WebCite is an archiving system for webreferences (cited webpages and websites). It works in a way of tinyurl, by submitting a URL you get a new URL back, e.g. I submit "http://www.yahoo.com" on May 6th, 2006, webcite gives me back a new URL http://www.webcitation.org/5Fgu0Xf5x, and yahoo page on May 6th, 2006 is archived in http://www.webcitation.org/5Fgu0Xf5x.

So this is actually a combination of archive.org and tinyurl.com, and may look extremely simple. However, as I worked on persistent URL and web archiving issues previously, I can testify simple is not easy.

Persistent URLs issues have been studied by Steve Lawrence (citeseer, now google) in (Persistence of Web References in Scientific Research. IEEE Computer, 34(2):26--31, 2001), Thomas A. Phelps and Robert Wilensky in (Robust Hyperlinks and Locations ), Frank McCown and Michael L. Nelson's "The Availability and Persistence of Web References in D-Lib Magazine".

In particular Thomas Phelps has proposed that most web pages can be uniquely identified by 5 keywords, so you should referene the web page by the five keywords, and issue the 5 keywords to web search engine to find the web page (in case it's relocated).

Web archiving is best addressed archive.org for now, in a rather typical web crawling way, and still an active research area. And now we have a on-demand web archiving system such as webcitation.org.

I don't know how webcitation.org scales, or how it protects the data, and it certainly doesn't solve all web archiving issues (we may only realize a web page should be preserved after the right time is long gone), but still, it's really simple, neat, and targetted two problems (persistent URL and web archiving) so nicely.

2006-05-01T15:55:00.000-07:00

learn tricks in java jar file

I used java jar files for some years but didn't realize several interesting mechanisms inside of jar file. Now I have a chance of reading jar specification and found several very handy features:

-- specify class-path in MANIFEST.MF

-- create index files (INDEX.LIST) for all packages.

-- META-INF/services directory to associate an implementation with its definition (e.g. class to interface)

These all come handy for easy distribution of a package

2006-03-22T20:57:00.000-08:00

Programmable Web and protocol

Some common web API design patterns are discussed by Raymond and Leigh. Following my previous blog, I am thinking there are different requirements in designing service and protocol. When writing a service, one has control of all resources and has the luxury of making everything clean and right, including all kinds of error processing, so no ambiguity here. While working on a protocol, perhaps it's ok to get minimal correct because one don't have control of all resources.

So perhaps this may help explain the difference between programmable web and Atom publishing protocol, and why Atom dropped detailed status code between 0.2
and 0.4

2006-03-22T07:42:00.000-08:00

Producer and Consumer, the economy of unAPI

While I am riding the bus to Los Alamos, I ponder a problem which annoys me for a while in unAPI design and was debating still now: what's simple, and not too simple in unAPI?

I think I got something and eager to write down, again this is an economy issue. There are two parties in most web applications: producers and consumers; when producers are potentially quite outnumber consumer, you want to make producer as simple as possible, and leave most complexity to consumer, because consumer benefits more ($$$) from this cycle and therefore is willing to afford the complexity.

Maybe this is too abstract, but it may make sense if we think about several examples. (a) In the web, there are so many nasty HTML pages, but we are all happily living with it, because the consumer (browser, search engine, etc) takes extra efforts of making sense of these pages. (b) In the RSS/Atom world, there are several standards and many more nasty pages but news aggregators happily support them all.

So maybe I have the answer now. In unAPI we want to make the producer as simple as possible, we don't want to burden them for heavy error processing or schema validation -- so we can claim they are wrong here and there, no! it must be extremely easier and nasty errors are tolerated (such as HTML or RSS), content is king here; it's up to the consumer to make order from chaos.