Liberating the literature

Knol

2008-07-24T05:53:00.000Z

Hot on my post on MedPedia comes the long-awaited (!) release of Google's wikipedia-competitor - Knol. You can read a review via this ReadWriteWeb article.

There seems a number of articles on health topics (aimed at consumers) and the author of the article is clearer shown (alongside their credentials) - addressing the main criticism of wikipedia; in that the authors are known. Users can suggest alterations - but the author needs to approve. A few exampl articles:

This will be interesting to watch to see if people prefer Knol or Wikipedia. A few years ago Google was perceived in a much more friendly light than it is now - so will users flock to suggest alterations and improve the articles? I think the possibility of payments (via google adsense) might just swing it...

MedPedia Is Wikifying the Medical Search Space

2008-07-23T08:01:00.000Z

An interesting new development:

"MedPedia is a new project, currently in development, that will offer an online collaborative medical encyclopedia for use by the general public. In order to keep the content accurate and up-to-date, content editors and creators have to have an MD or a PhD. Several highly-esteemed medical colleges will be contributing content to MedPedia, including Harvard Medical School, Stanford School of Medicine, UC Berkeley School of Public Health, and University of Michigan Medical School. Medpedia is also receiving support from the National Institutes of Health (NIH), the Centers for Disease Control (CDC), the Federal Drug Administration (FDA) and many other government research groups. The content from these organizations will then be edited by MedPedia’s community of medical professionals."

Click here for further details

Quick again

2008-07-22T09:12:00.000Z

I'm very happy with progress on the slow speed of TRIP - we're no longer slow! I appreciate slowness is relative, but we've gone from 10+ seconds for the search results to be shown to between 1-2 seconds. Before it took and age and now it seems quick.

The basic problem was the way the system handled the snippets (the small pieces of text below the document title). These were identified as the bottleneck and so they way the results are returned has been changed to take into account this slowness. Previously, the results and snippets were generated and displayed. Now the results are displayed and the snippets dropped in when they're ready. There are other potential areas of optimisation but we can consider these at our leisure!

Apologies for the slowness of the system to date and enjoy the new speed.

Next upgrade to TRIP

2008-07-20T19:00:00.002Z

TRIP has been busy recently with 3 major projects due to launch by the end of the summer. The Spanish version of TRIP should be out very soon, TRIPanswers will be out by the end of August and TRIPcpd should be out by the end of September.

In addition, we'll be rolling out a small upgrade to TRIP soon, with the ability to export records and e-mail to a colleague.

The TRIP speed issue is also being worked on with a final report from our web-people due soon.

So, while we've been busy, most of them are under control and nearing completion so we can start thinking about our next batch of work - an upgrade to TRIP. We've got a number of ideas for the upgrade, but we want your help. What would you like to see?

Please contact us (click here) with any ideas you may have. No matter how big or seemingly trivial, we want to hear.

Over to you to help improve TRIP

Ologeez!

2008-07-16T13:08:00.000Z

TechCrunch has just posted an article on Ologeez!. Ologeez! is an alternative front-end to PubMed (but has plans to include other databases as well) and mixed in with 'social' features.

Two things sprang to mind when reading this coverage:

Alf Eaton already has a pretty impressive 'version' that has been around for years - HubMed. It was nice to see someone else highlighting this in the 'comments' section.
Aside from HubMed there are loads of other PubMed 'versions' - yet it was this one that gets a mention in TechCrunch - a Stanford bias perhaps?

You can see Ologeez! by clicking here. It'll be interesting to see if it succeeds, I have my doubts. Our playing with this sort of thing (courtesy of gwagle) was an interesting failure. As mentioned above there are numerous other websites offering social functionality coupled to PubMed, yet none are successful. I can't see anything in Ologeez! that makes me think they're particularly special. Perhaps the TechCrunch link will help (it certainly can't hurt).

I suppose one day someone will crack it....

One quick thought, instead of having one site trying to monopolise this social space, why not get together a significant number of 'players' in the area - be it medicine, be it biomedical - and create a single space that all users can contribute to. Imagine if TRIP, Cochrane, Medline, NICE, CRD, eMedicine, BMJ etc etc all worked to one standard and created a single 'space' - now that would be powerful and powerfully useful. One to ponder.

Bias in EBM

2008-07-10T09:18:00.000Z

Much is written about bias in relation to EBM. The overwhelming focus is on methodological bias in creating primary or secondary reviews. However, little is written about a bias that is arguably as important - search bias.

What's the point of spending huge amounts of money creating beautifully crafted pieces of research for them to be overlooked by a clinician looking for robust information to inform their decision making process?

The TRIP Database uses an algorithm which is a compromise and I would never claim it was perfect. The debacle with the NLH's search 2.0 highlights further problems. Medline is a nightmare to use. In short there is no perfect clinical search engine.

So what does that mean for a practicing clinician? Do we want them to search multiple databases? I suppose, ideally, yes. But I'm far from convinced it is practical. There is an onus on clinical search engines to improve and meet the needs of the users. But the important message is that bias is not just about research methodology, it applies in equal measure to how clinicians access the research.

An empty inbox

2008-07-02T13:17:00.000Z

Wow, I can really see just how much time the NLH Q&A Service took. It stopped answering questions at the end of last week and I've already managed to empty my e-mail inbox, the first time in years!

There have been a number of e-mails with new content waiting to go on TRIP, the most notable being the rather nice York Health Economics Consortium.

Talking of NICE (tenuous I know) the Darzai review (click here) has indicated an expanded role for NICE. It looks like they'll be responsible for a new service/organisation NHS Evidence:

"NICE will manage the synthesis and spread of knowledge through NHS Evidence – a new, single portal, through which anyone will be able to access clinical and non-clinical evidence and best practice, both what high quality care looks like and how to deliver it."

I can only surmise that the NLH will end up moving (again) to fall within NICE's clutches. What that means for Q&A I wouldn't like to say. I'm seeing Peter Hill tomorrow and that might shed some light on things.

The extra time on my hands has been spent harassing our web-developers and we're fixing a number of niggling problems relating to advanced search, truncation, apostrophe's. In addition - for some reason - the big 5 core journals (NEJM, JAMA, Lancet, BMJ, Annals of Internal Medicine) have disappeared from our search. These should be back soon.

TRIPanswers is progressing very well and the beta should be available within a fortnight. It should be as good as finished then but I just need to add a load of content. I'm hoping to launch with over 5,000 Q&As, reaching 10,000 by the end of 2008. I haven't thought much past then. Each Q&A will be quality marked and have various other bits and bobs to transform Q&A; at least a little bit!

I'm also starting to research various semantic tools to improve various aspects of the TRIP site. I'm hoping to get some sort of system set-up to allow me to play around with these. I see these tools as allowing things such as semi-automated updating of Q&As, allowing natural language searching of TRIP and TRIPanswers. I even thought we could allow users to add text that they are interested in and use that as a basis to find 'related articles' within TRIP.

I've been a long-standing fan of PubMed's related articles and getting this close to trying similar tools on TRIP is very exciting. I've often carried out searches of PubMed and found articles of interest only to find even more by using 'related articles'.

Jumping back to TRIPanswers, if anyone wants an invite to the beta, let me know via http://www.tripdatabase.com/ContactUs.html

And so it ends

2008-06-27T16:59:00.000Z

“Which is the most accurate test for diagnosing genital chlamydial infections in men, DNA amplification test on a first void urine or a urethral swab?”

Was the first question posted to the NLH Q&A service in November 2004. Nearly 6,500 questions later, this question closed the service:

“Is there any difference in upper GI side effects with risedronate compared to alendronate?”

A sad day...

From one Q&A service to another

2008-06-25T10:42:00.000Z

This is the final week for the NLH Q&A service. I've been overwhelmed by the response from GPs and apparently there has been a lot of chatter on GP mail-lists moaning about the decision to close the service. So much so it came to the attention of a health journalist and I had a chat with them earlier this week. You can read the story here. I don't think my opinions were fully represented, but then that's the nature of journalism. Interestingly, the same GP chatter stimulated a call from BMJ-land.

I have mixed feelings about the end of the NLH Q&A Service.

On one hand I'm sad for:

The users of the service. I have been so humbled by the feedback I've received, it really has been great. I think it's a missed opportunity to not - somehow - keep the service running.
Rosalind. She has worked on the Q&A service from near the start and has been wonderful. She made a load of brave decisions to come and work with me on Q&A. I'd like to think she's learnt a lot from the experience. Fortunately, she's got a new job but it will necessitate a move from Aberystwyth to London. I suppose I'm sad that I'll not be working with her for the foreseeable future.

On the other hand I'm getting so excited by the new Q&A development (TRIPanswers) which will launch in the summer. I've had the website to play with for a few weeks now and have been testing it fairly extensively. I've just sent another batch of feedback to our web-developers, so I'm hoping the work will be as good as completed by the first week of July. However, the launch won't happen then as I need to populate the site with data - which is a long, hard, manual task.

I've also been making new relationships with various Q&A providers around the globe and that will all help support TRIPanswers. I genuinely think TRIPanswers will be bigger than the TRIP Database...

Speed update

2008-06-22T06:10:00.000Z

Well, the extra RAM is now in the system and there is no obvious improvement in speed. It's invariably quicker at certain times of the day (peak times when memory was an issue).

The analysis of speed and the identification of bottlenecks is booked in to start mid-July. This will highlight code inefficiencies that can either be fixed as they go along or will need further work. Fingers crossed it'll be the former.

TRIP speed - we're working on it

2008-06-11T09:01:00.000Z

I've noticed (and no doubt lots of others have as well) the pretty awful speed of TRIP in the last week or so. Just to say SORRY and that we are addressing the issues:

As a short-term measure we are quadrupling the memory in our servers. This should boost speeds considerably and hopefully this will be in by either the end of this week or the start of the next.
On a more medium term basis we are carrying out some sophisticated analysis of the site to identify where the bottlenecks are. We're hoping that, once this has finished, we will be able to allocate resources at the problems. If we're lucky the extra memory might sort everything out!

I use TRIP everyday so am acutely aware of the need for speed - we'll get there....

NLH vs NHS Q&A

2008-06-10T12:14:00.001Z

Alexa is a leading resource for looking at the popularity of a website. Given the demise of the NLH Q&A I thought I would compare the NLH Q&A service with the NLH itself (click on image for larger size):

They say a picture is worth a thousand words!

There are some caveats associated with those figures (isn't that always the way). But it's certainly worth pondering.

Slow blog

2008-06-09T12:19:00.000Z

I'm aware that this blog has been slow over the last week or so, the reason is simply I've been too busy with 'all things TRIP' to contemplate what to write about. Specifically, I've been heavily involved in 4 significant chunks of work:

TRIP Spain - should launch very soon.
TRIPanswers - I've been given the site to play with and/or test. I've started adding questions and answers and I'm very pleased. So far I've added 250 questions, only another 5,000 before we lauch (in about 2 months).
Advertising - we've moved to a new advertising providers, I'm having to manually assign search terms we receive on TRIP to the advertisers advertising categories. This is very labour intensive, but it'll hopefully be worth it!
Winding up the NLH Q&A. Not much to do really, I've already started relaxing and moving my mental efforts to TRIPanswers...

I imagine I'll start more regular posting soon(ish)

Search 4.0: Putting Humans Back In Search

2008-05-29T04:48:00.000Z

A great article on where search MAY be heading - Search 4.0: Putting Humans Back In Search

I've been intrigued by the notion of human search and the site Mahalo was/is of interest. The main interest has stemmed from the following:

I feel the tweaking of search algorithms can only take you so far.
Most users will only visit the first page of results.
Given the experience within TRIP we have a pretty good idea what would constitute a good set of results.
We have some experience in this area. A while back (so not up-to-date) we created a number of reviews e.g. statins or knee osteoarthritis.

The big problem is scale. We could create human-powered search results for the top 100 searches - but that's scratching the surface. Perhaps I'll do some mock-up and get users to 'vote' to see which they prefer.

Watch this space.

Web users are getting more ruthless and selfish...

2008-05-24T14:44:00.000Z

That's according to Jakob Nielson (click here)!

...people are becoming much less patient when they go online.
Instead of dawdling on websites many users want simply to reach a site quickly, complete a task and leave.....Instead, many are "hot potato" driven and just want to get a specific task completed.....

The above has got me very excited as it fits in with my view based on observations over the years of running fairly successful websites. It's also something Bandolier have long advocated with their frequent references to nuggets of evidence.

So what does the above suggest? Give small bite-sized chunks (nuggets) of evidence. More than that it's giving people nuggets of evidence AND making them easy to find.

The NLH Q&A Service is great as it gives individuals rapid answers to their questions. The site is let down though by a poor search mechanism. So it's not so great for others.

Resources like Clinical Knowledge Summaries have great content (arguably they are a large collection of nuggets) but again findability is a real issue.

Is there a solution? We hope so, and the launch of TRIPanswers in mid-summer will reveal our hand...

David Rothman @ the MLA 08

2008-05-23T12:09:00.000Z

David Rothman has been to the Medical Librarian Association meeting and taken a video camera (click here).

This is a great use of technology...

As is this telectroscope.

Advanced search

2008-05-19T18:20:00.000Z

I see that PubMed have rolled out a beta version of advanced search (click here).

While I play with the new features I really need to turn my attention to TRIP's advanced search. Relative to the other part of the site it's 'weak'. Something for the last half of 2008!

Semantic Web & Calais

2008-05-19T08:46:00.000Z

I spotted a semantic web product called Calais a while back on (Reuters Wants The World To Be Tagged). Basically, it takes any document and 'marks it up' with various semantic web tags. I didn't give it much thought till a blog post this morning Reuters Launches Calais 2.0 - Now With Pop-Culture which reports support for the pharmaceutical and medical world.

You can use this Calais Viewer link to try it out.

I enjoyed using it, but it still has a long way to go, the number of terms it recognises is limited.

Given my lack of vision I'm not sure how you can utilise knowing that ibuprofen is a 'product' and feverish illness is a 'medical condition'. Perhaps one day it'll become obvious...

Further feedback on the demise of Q&A

2008-05-15T13:47:00.000Z

Further to a post last Friday (click here).

Why will you cease operations on 27th June? Is there any point inlobbying to try & keep your service available? If so, who should I contact (from a v satisfied customer!)
Why is the clinical Q&A service being axed? It is excellent and will be sorely missed
What a desperate shame this service is to discontinue! I have only just discovered it and have found it enormously useful because it is evidence based. Please feel free to pass this message on to anyone who has influence over this decision.
Why is the NLH Q+A service ceasing operations? Has the funding been pulled? Will there be an alternative service?
This service has greatly improved my practice it has been a great resource i cant believe its being withdrawn i will find it a great loss
Interesting- found this site via google- I see it's a pilot and about to be withrdrawn. I never heard of it- was it something that we should have been informed of? muight have been useful (or not) but if we had known about it we might at elast had the chance to try it out. More wasted NHS money on something not discussed with users and not properly tried and evaluated?
We will miss your service greatly! Is it migrating somewhere else/what alternatives are there?
Is there a replacement once the current service close in June? it has been a very helpful service for lots of us in primary care

I'll add extra comments as and when I get a decent batch....

Quality standards in Q&A

2008-05-15T12:15:00.000Z

At the start of the week I had the pleasure of presenting at the 2008 Clinical Librarian Study Day. I was tasked with talking about quality standards in Q&A. This was a tough subject to do justice. I've worked hard for ten years on Q&A yet had never really thought about 'standards'. So it took a while to distill my thinking into reasonable standards.

I came up with two types of standards:

Easy
Real

Easy standards are the ones I consider self-evident e.g.

Competency in searching various databases
Return answers in an agreed time
Keep responses to a reasonable (brief) length
Answers should be referenced

But adhering to these standards means very little.

Real standards are the ones I think mean something and are perhaps less obvious and I came up with 7:

Competency of answerer
Transparency
Communication
Feedback
Correctness of answer
Boundaries of Q&A
Quality control

Competency of answerer. It's relatively straightforward to search medline and learn to appraise. However, it's much harder to understand the clinical context. This involves trying to understand the motive for the question, what it actually means, the sort of evidence required and knowing when the question has been answered.

Transparency. This is not as simply as linking to an article informing users about the process. It's ensuring that they actually know what the process is and potential shortcomings.

Communication. Linked with transparency this relates to simple things such as using a clear narrative to more interesting challenges - such as explaining uncertainty.

Feedback. Is there easy feedback from the user but also from others viewing the service? We receive a small amount of feedback, we should get more!

Correctness of answer. Is the correct answer given? A tough question to answer..

Boundaries of Q&A. A bit vague this one, but when/where should a 'quick and dirty' Q&A service operate. I often worry that we spend too little time on questions, rushing off to answer the next one. Other areas worry me such as high-risk questions - but then we pass them through to our clinical director to check. But every now and then I worry that we're going beyond what we should be doing.

Quality control. Is there a QC system? We have internal and external systems, I'm pretty sure they're robust - but an important standard all the same.

With ATTRACT and the NLH Q&A Service we could improve, I'm thinking particularly of transparency and feedback. We'll be addressing both these issues (and others) with TRIPanswers.

Of the seven I think the two really important standards are Competency of answerer and Transparency.

Another milestone - 50,000+ searches in a day

2008-05-14T07:13:00.000Z

Yesterday, TRIP was searched 51,021 times, the first time we've been searched over 50,000 times per day.

This increase must be down to the significant increase in users who are coming back on a more regular basis.

What's the next milestone? 2 million searches per month, 400,000 per week or 75,000 per day....

Exporting TRIP records

2008-05-10T07:02:00.000Z

We occasionally get requests from users for new features. Where possible we try and accommodate these wishes - assuming they make sense to us and we've got the money. One feature has recently been requested more than any other is the ability to export TRIP results.

Separately, in the development of the Spanish version of TRIP, it was 'showcased' to the Spanish Ministry of Health and they too wanted an export feature.

Bottom line: We're creating an export feature.

All the results in TRIP will have a tick box and users will be able to select the ones of interest. They will then be able to export the selected records to file or e-mail to a colleague.

This should be out by the start of June

The sad end of the NLH Q&A Service

2008-05-09T18:39:00.000Z

The saddest thing for me about the ending of the NLH Q&A Service is the reaction of the users. We've just placed a notice on the site alerting users to the ending of the service. Two comments received within 35 minutes of it the notice going live:

"I have just found this answering service through a colleague. I think it is absolutely brilliant. I have just found the answer to a question within 1 minute which would normally have taken me hours, maybe days of research. I am paid £40k p.a. so this time saving is bound to be very valuable to the NHS when multiplied by all the people who use it. Why is the service ending???"

"Hello. As an EBM practitiopner I am disappointed to read that as of June 27th the NLH Q&A Service will cease operations and be unable to answer any new questions. Could you kindly inform me of the reasons behind this decision, and what will replace this very important and I think successful primary care service. Thanks,"

I'm so proud of what we're achieved and the fact that clinicians are prepared to contact us to express surprise and disappointment is impressive. The NLH suffers from a lack of engagement with primary care health professionals, an area that needs more support that secondary care (that has access to librarians). By removing of this key service it further hampers engagement.

Still the secondary care clinicians will still have their services...

Dramatic increase in search speed

2008-05-07T09:37:00.000Z

I've been worried about the search speed for some time. The removal of the number of results for each individual category was meant to improve things, it didn't.

Yesterday (see previous post) I introduced a system that removed search results which we considered not relevant. As well as improving the search relevancy it reduced - dramatically - the number of search results returned. By accident this has resulted in a huge increase in search speed.

Not all accidents are bad!

Improving search on TRIP

2008-05-06T10:23:00.000Z

We have just released a major improvement in the TRIP search results.

This has been brought about by the refinement of our 'text cutoff tool'. Prior to this introduction a search would return every result that contained the search terms. So a search of acute kidney injury returned the CKS guideline on ankles and sprains! I imagine most people would agree that that is not even an average result - it simply shouldn't be there. But how has it got there? It contains all the individual terms, so is returned.

So how does the text cutoff tool work?

Each search on TRIP looks for matches to the search term(s) used and these are ordered based on our algorithm. One component of the algorithm is a text score. The higher the text score the more likelt the result is to be pertinent to the search terms. Some factors that affect the text score include:

Location - where the term occurs (e.g. if someone searches on asthma, documents with asthma in the title tend to get a text score higher than if they are only mentioned in the text).
Term density. If you have two documents, both 1000 words long and without the search term in the title and one mentions the search term once and the other ten times, the latter will receive a higher text score.

The idea behind the text cutoff is simply to say that all results with a text score of lower than X do not get returned.

This sounds nice and simply but it fairly arbitary and if you set the score too high you may miss important documents out, too low and you allow too many in. It's also complication that there is no magic score that defines pertinence. We have tested lots of combinations of search terms and text cutoff points and we've managed to remove an awful lot of noise from the searches. The cutoff point we've used has reduced the number of results from the acute kidney injury from 2,609 to 440.

Ironically, the CKS guideline on ankles and sprains still remains. Lowering the score to remove that caused the removal of other documents from other searches.

We're not claiming its perfect, but it's pretty good!