Use a hyperlinked document as a bioinformatics lab book
April 13th, 2007I wrote previously about using the file system to organise your scripts and data. I use this method and it does help my organisation, but it doesn’t replace a lab book. I want a system that explains the relationships between the different set of results, and shows the outline of my work.
I’ve tried several different ways of doing this. Keeping a written lab book is problematic, since my results are on my computer, furthermore it’s tedious to describe code on paper. Another approach is writing a plain text log, however I didn’t like this because I can’t include pictures - which I like to have next to the relevant sections of text. I didn’t consider a word document, because all of the formatting would get in the way of what is a simple task: to link scripts, pictures, results and text into a outline of what my work is about.
The most successful approach I have had is using a wiki; for one reason, hyperlinks. If I want to include a figure I include the hyperlink to file. I can do the same for the script that produced the figure. So what’s so great about this I hear you ask? Well, here’s why.
Dynamic content
Because I’m using a link, if I update a figure, for example changing an axis label, this is automatically updated in the wiki. I don’t have to insert a new image, remove the old one and set the new one it’s place. Very useful with the number of incremental changes between the start of the a project and the final draft.
Easy to find scripts and data
The second reason why a hyperlinked wiki is useful, is that you can jump to the data or script you need by following the links. Useful for quickly finding the relevant script that produced the figure - even with the most organised file system in the world.
It’s true you don’t really need a wiki to do this. But wiki’s allow a really fast way to create a hyperlinked document. As to which wiki platform to use, I’ve tried MediaWiki (a bit overkill) and MoinMoin (I couldn’t get it to display the pictures the way I want), until I finally settled on Instiki, which I’ve found the easiest to use. MoinMoin and Instiki are both platform independent. Neither take much time to setup, and I definitely say it’s worth taking the time to try this approach out. What have you got to lose? A few hours, but if you can get the system to work for you the rewards in productivity are great.
April 13th, 2007 at 10:56 pm
You may be interested in looking at zim (http://www.pardus.nl/projects/). I’ve looked at a few ‘desktop wikis’, but most of them require using a webbrowser. I’d personally not like to have one application doing everything (see emacs for why not), so the fact that zim is a separate app is a plus in my book.
April 14th, 2007 at 1:24 pm
Hi Jacob,
Thanks for posting. That’s a good point, and one that I should have thought about in more detail.
If you’re running a wiki on your computer, you should take into consideration how exposed it is to the internet. In most cases it’s trivial to restrict access to localhost, but something worth checking.
Thanks for the link to zim also. I haven’t tried it but it looks worth a look.
April 15th, 2007 at 5:19 am
A wiki is a cool idea for a notebook, but it’s hard to get a lot on a single page. A nice tool for an overview of your stuff is VisiMap Pro. There are other mind-mapping programs out there, but this one is very nice. You can build up an arbitrarily branched tree structure with notes on each branch and the ability to hyperlink to any document on your disk or any page on the Web. Built by a small company in the UK. I’ve used it nearly every day for several years just to stay organized and creative. Take a look.
April 16th, 2007 at 12:37 pm
Having used wiki’s for ‘info tracking’ in work for a couple of years now I definitely agree that MediaWiki is overkill. I’ve switched wiki backends 3 or 4 times now, but for my money DokuWiki is the way forward.
Firstly I like my things server hosted - and it works serverside. Secondly I’ve never seen the necessity for a MySQL backend to everything (*especially a wiki*), and it doesn’t require one, whilst maintaining all of its PHP prettiness.
I’ve seen recently (whilst researching GTD tools) that the ‘personal wiki’ is gaining in popularity. I’m talking about things like TiddlyWiki, that don’t require a serverside component and hence can be ported around on thumb drives.
I am however sick and tired of learning the subtle and not-so-subtle differences between all the different wiki backends I have used. Again DokuWiki stands out for me because I happen to like the markup elements.
April 16th, 2007 at 10:10 pm
@Rob
Thanks for the tip. Yes outlining software definitely has it’s place. In between experimenting with wikis I’ve tried software such as OmniOutliner and DevonThink. Both Mac unfortunately. Both were good but didn’t really give me what I was really looking for though. But different methods will work for different people. The more you experiment, the more you’re likely to find the system that works for you.
I do know that Pimki is a wiki with mind mapping software included, all that’s required is ruby.
April 16th, 2007 at 10:29 pm
@Dan
Thanks for the tips, I’d heard of neither TiddlyWiki or DokuWiki. I spent the evening playing around with both.
I have to say that TiddlyWiki blew my mind. I find that way it’s organised to be really ground breaking. It’s difficult to describe but the way that content is dragged in and out of the main text stream is very innovative.
But enough of the hyperbole, any program that stores itself into a single file is always useful as it’s easy to backup with revision control. And you know what a fan I am of that.
April 23rd, 2007 at 3:18 am
Thank You
May 2nd, 2007 at 12:40 pm
A little late, but I’ve messed around with various software … and I’ve found Journler (Mac only; http://journler.com/) pretty useful.
May 2nd, 2007 at 2:11 pm
Hi!
Actually I’m using a blog, as a laboratory diary.
Wordpress allows to use static pages where you can put for example the projects’ statements and the material and methods you’re using, and you can use the blog part as a log to describe your work daily.
What a mess of wiki/blog/mind mapping engines do exist ;)!
May 2nd, 2007 at 6:03 pm
If you use Linux, I recommend an excellent note-taking program that is suitable for taking lab notes (I use it when I dig in literature analysis): BasKet Note Pads (http://basket.kde.org).
May 4th, 2007 at 4:48 pm
@Giovanni and Doug
Yes blogs are definitely useful too. I previously also tried using a local blog. Blogs are good because you can keep a linear track of all the work that you’ve be carried out. I think that blogs are also going to be used more and more so that researchers can keep up to date of each others work. I stopped using it though because I prefered a wiki, but everybody is different.
What I’d most like to see is a good bliki. A blog with wiki capabilities. There are some around but I don’t believe there are any with trackback and comment capabilites. A bliki would be really good for collaborative work. Some one can post a new result, then a collaborator can update it with their relevent results. I guess you can do this in the comments section, but it’s just not the same.
May 8th, 2007 at 12:06 pm
@Luca
Thanks for the link. Unfortunately I’m using a Mac, but I’ve had a look at the website. The screenshots look interesting, it seems similar to DevonThink, which is very popular. But even better because it’s free.
June 14th, 2007 at 9:39 am
Mike,
After a lot of trial-and-error, my group has settled on a public hosted wiki (Wikispaces) to record experiments, tasklists and organizing pages and a blog for reporting on milestones and current challenges.
The hosted wiki has the useful feature of providing third-party time stamps to prove who-knew-what-when. A wiki also nicely keeps track of everyone’s contributions. Clicking on Recent Changes is a very convenient way of seeing what everyone has been up to in the lab lately.
June 14th, 2007 at 1:31 pm
Hi Jean,
Thanks for your comment. I’m really interested in what you’ve been doing with your lab wiki, and blog too. It’s a good example of using a wiki to organise scientific data and research, and I’d recommend everyone have a quick look.
July 5th, 2007 at 8:17 pm
Hey Mike,
Although I find wiki’s very useful, I’d rather use them to store operating guidelines than as a labbook because I need (and am told by my employer) that the labbook is immutable (for audit purposes and stuff). We _do_ use a wiki as well though, but not for recording what we’ve done.
jan.
July 5th, 2007 at 8:18 pm
I’ve put some more thoughts on my blog.
July 9th, 2007 at 9:49 pm
Thanks for your feedback Jan.
I agree with your point about labbooks being immutable, certainly for commercial purposes it’s important that every result has a unchangeable time stamp. As Giovanni suggests, using an internal blog could be used for something like this. My main about about using a wiki was that it could be used to link to the scripts and datasets that produced the results, so that’s when you writing up, for example, it’s easier to find which program made what. I guess that any type of software that supported this could be used along the same lines.
November 18th, 2007 at 2:48 am
I have spent WAY too much time thinking about documentation, and while i really like Dokuwiki, part of me leans towards emacs muse (has wiki functions and numerous export/publishing options) and finally LaTeX. For me, the latter solves many problems in lab documentation:
-ability to print scientific information, including chemical formulae, equations, etc
-hyperlinks to relevant web info, related local docs, appendices, etc (almost wiki-ish)
-robust citation management
-PDF/hard copy options (can be printed and witnessed, yet still easily trackable to the e-version
-all the e-benefits: archiving, searching, off-site dupes, security (if needed)
-totally chi-chi, but i can put multimedia files into the PDFs using various packages, which can be nice for a structural biologist
Its not perfect, but very good for my needs.
November 21st, 2007 at 9:05 am
Hi, I am also a bioinformatician trying to get organized. First, thank you Mike for the post, and all the others for their comments. For a start I am using Linux and don’t want / don’t have the money to use any commercial software as a lab book. I also tried to use a wiki as a lab book, and found that most wiki’s are indeed a little bit of an overkill. I also had a look at some stand alone wiki’s like wiki on a stick (http://stickwiki.sourceforge.net/) or the already mentioned Tiddlywiki (http://www.tiddlywiki.com/). I really like the idea of a wiki but found them always to cumbersome. I also tried some GTD software like ThinkingRock (http://www.thinkingrock.com.au/) which is a nice approach for organizing thoughts but thats it. There is a Tiddlywiki based GTD system as well that I would like to try next called MonkeyGTD (http://mgtd-alpha.tiddlyspot.com/).
At the moment I am using a standalone software called Memoranda (http://memoranda.sourceforge.net/) which is not a wiki. As the webpage states, it is a diary manager and a tool for scheduling personal projects. It comes with an agenda, events, tasks, notes, and resources. One thing I like is the timetracking of the software. I get notifications when a task of a project is due or an event occurs. The only downside is that it makes the impression of a beta version and doing backups of projects is rather cumbersome. But it is worth a look.
As for your post about organizing files, I found it always practical to name folders with a date first (20071121_projectname). That way they are ordered and if I am searching something and know at least somehow when it was it is easier to find. Another practise that I use is, that in each project folder I place a README. That contains short descriptions of the project.
I am working a lot on a shell with a lot of files, using several unix/linux programs (sort, uniq, egrep, awk, sed, cut, bash,…) to remodel files, produce new files, etc. One thing I do as well is after I make use of such a command line step, I normally pipe the command line into the README. This way I can see how different files evolved, when I do have a look in the README. That helped me quite often.
Best, Sebastian
August 22nd, 2008 at 8:00 pm
I’m an applied mathematician. I do many numerical simulations, which produce long lists of numbers and graphics, similar to the sort of data a bioinformatician must deal with. I’ve been using MoinMoin DesktopEdition for a few months. You note that you like Instiki’s graphics-handling better. Both seem great if you want to link to a file on a webserver, but for using a picture on your local computer, I can’t seem to make it work. What do you do?