As to your point about database format interoperability, that is a valid point. I think most database allow the export of common data formats such as CSV, which is ideal for situations such as creating supplementary materials in manuscripts.

Without exception, always use a database to store data. Manipulating flat files in scripts is hard work, and is also a source of bugs.

I’m not convinced. It’s very important for me to be able to look at the data, and to look at old data. It’s very convenient for me to be able to use Unix command-line tools to count the data, sort the data, randomly sort the data, remove duplicates, sum columns, compute average + standard deviation, etc. I sometimes have to move files between different computers and different operating systems, and I don’t have to worry over whether someone else’s DB configuration is different. Also, I will still be able to look at that data 3 years from now, when I have a new version of my DB installed that won’t read the old files.

@Michael
I agree, Python and Ruby syntax make coding much more of a pleasure and would recommend either of these two to any bioinformatician. Though I’ll continue to write in Ruby as this is the language I know.

@Gioby
For to-do lists I use lighthouse, as it suits my personal preference. However there’s plenty of tools available for keeping todo lists, and I think it comes down to which one suits you.

I agree with your point about old software that works. I would say that if you have a system that you still have to maintain and change, then “upgrade” to more easily maintainable system. In your example I would say leave it be, as you say, if it works then your extra efforts could be used elsewhere.

Without exception, always use a database to store data. Manipulating flat files in scripts is hard work, and is also a source of bugs.

That’s not always as easy as it seems.
Many laboratories and people have used flat-files for years: that means that if you work in such a place, all the scripts, programs produced and used internally by everyone are based on flat files.
What will you do then? If you spend time trying re-writing or adapting all these scripts, your colleagues will most likely think you are kind of wasting time, because the flat-file based script already works :).
Think of biopython, bioperl, emboss: all of these are meant to be used with flat-files, even if you can re-adapt them with a small amount of work.

Use make-type files instead of scripts

Wow, I didn’t know of make before!
I am using it now and it is a really good tool.
A good primer on using make for bioinformaticists is also in http://www.swc.scipy.org/

Use testing and validations

I try to validate and test my data as often as possible.
I think there is an additional complexity in the problem of testing for bioinformatics software: you have to check that your scripts don’t make any mystake, and moreover, you have to check that your results are compatible with their biological context.
This last kind of testing is more likely what other scientists do when they use negative and positive controls in their experiments.
However, do you know whether there are any common guidelines for testing bioinformatics scripts?
They would be very useful.

Well, I really appreciated your post. You are really a good bioinformatician because you are so keen in sharing your experience and known-how with others, publicly.
Thank you very much: I am going to install a mysql database on my computer, and I didn’t know of ORM modules, so, if only there were more people like you in bioinformatics :).

To solve the issue we discussed above about keeping track of what tasks have already been done in your Rakefile, I have written an extension to rake. Rake is ideal if you’re working with files because it takes the timestamps of the files into consideration. However, no files are created when you’re loading stuff into a database. I tried to solve that by extending rake so that it puts timestamps in a little meta table in the database as well.

You can get it at http://github.com/jandot/biorake/tree/master

jan.

`cat data.txt | sort`

Definitely a nomination for the Useless Use of Cat Award.

http://partmaps.org/era/unix/award.html

I actually just performed a little project using this approach, and it works great!

I wonder in future if it might be worth packaging this up into a Rails type gem focused on organising bioinformatics experiments. I think it might take a fair amount of work, but could ultimately prove very worth while.

if Gene.all.length == 0
  # Notify the user
else
  # Do analysis
end

On the other hand, the rebuild task runs all of the tasks in the correct order when the project is run from scratch, so in this case you would know that all the required data was in the database.

I do see what you mean about a meta-table of information, and it would take care of things like this, but I think it could add an extra layer of complexity. Another option is to have a hard and a soft load_sequences task. The hard task clears the database and loads all the sequences. The soft task only loads them if there are none . The stats task could then be dependent on the soft load_sequence task.

I’m sure you tried this as well: within your sequence_stats task, you obviously would want to add the prerequisite that the sequences are loaded in the first place, so instead of

task :sequence_stats do

you’d do

task :sequence_stats => :load_sequences do

However, this doesn’t work in rake because it can’t know if you did your loading or not. As a result, it would load the sequences every single time you want to get the sequence stats. Do you know of a way to get this working? I wonder if we can use timestamps on files to do this… Or would we have to tweak rake so that it takes into account some “status” metatable in the database? Any suggestions appreciated…

@Adam
I think DataMapper and Sequel are interchangeable in terms of the attention they are getting in the Ruby community. I’ve never tried sequel though. Comparing DM and ActiveRecord, the best point about DM is that it does away with Migrations which are a bit heavy weight for this type of work. However on the downside, I’m not sure how many AR plugins will work with DM. An example would be acts_as_reportable which I don’t think does.

@Wubin and @Andrew
I’ve never used Python so I’ve haven’t got a clue about the libraries. Andrew’s suggestions look good. Another option could be to look at what Django uses, as I think it’s somewhat equivalent to Rails so will probably have a handy ORM involved somewhere.

@Jan
As you say in an ideal world you there would be an complete gene class that could be interchangeable for any project. At the moment I’m only using this approach in one project, so I can’t really say what I would do, and to be honest I hadn’t really thought about this until you mentioned it. I agree though that a set of generic classes would useful. Also a set of validations that could included when required would be useful as well, as these are things that I spend most of my time messing around with until they work. BioSQL could be place to start for a set of generic ORM classes though? As always it’s the problem of trying to fit in writing all the sexy Ruby libraries I’d like compared with publishing something.

As for your next point, it’s true that you can’t write everything in Ruby. For bash scripting, Ruby allows ` quoting for running Unix commands. For example `cat data.txt | sort` and so forth. It’d be nice to write everything in Ruby, but I guess pragmatism comes first before style.

Two questions: (1) do you rewrite a little Gene class within every project (tweaking it to just do what is necessary within that project), or do you have a “master” gene class defined somewhere that you use whenever there’s a new project? We all know the second option should probably be preferred in the long run, but we all end up doing the first…
And (2): this approach means you have to do everything in ruby isn’t it? However, several of the steps in my own workflows can often more easily be done with linux commands (grep, sort, uniq and wc, anyone?) How do you handle that? Do you use a ruby equivalent? Or do you use a “system(’sort’)” in your rakefiles?

Good idea of putting an example on github…

jan.

I certainly prefer the ORM approach as opposed to writing SQL expressions, and try to use it whenever I can get away with it.