This is the fourth post in a series discussing simple, reproducible and organised research workflows. In previous posts I described how using Makefiles and a functional approach can result in simpler and more reproducible workflows. Here I'm going to outline how I break-up larger projects to make them easier to manage.
In the previous post I suggested breaking workflow steps into combinations of composable functions. The advantage of this is that steps are easier to understand and maintain compared to large scripts performing multiple functions. I believe this same principle can also be applied at a macro level by breaking large projects into modularised sub-projects connected by data.
I began practising this with a self-imposed rule: my computer only serves as a
"hot cache" for what I'm currently working on. I don't store any
project on my computer longer than I am currently working on it. I enforce this
by creating a soft link for the directory
~/cache is then the default directory for a new tmux session.
~/cache is where I always work and is also automatically cleared
after a reboot.
At the start of the day, when I begin working on a project, I use git to
clone a local copy to
~/cache. As all of my projects are organised around
using Makefiles the first command I call is
make to run the analysis
up to the latest state. I can then continue to work on the project and push any
changes back to the git repository at the end of the day.
The advantage of this approach is that I no longer have problems effectively organising projects on my hard drive. I previously used to have projects organised by year but this still made it hard to remember which project contained the analysis I wanted, given they often had related names.
Instead I now have an empty
~/cache directory which fills up with a few
projects over the course of the day. I keep all of my projects repositories
organised using repositoryhosting.com for my in-progress projects and
github.com for more complete public projects. Here's a screen shot of
my projects on repository hosting as an example of how I organise everything. I
find this much simpler that working with large numbers of local project
I found that the longer a project sat on my hard drive the more complex it
became. This often meant when I came back to a project after a few months I
sometimes found that the analyses were broken or wouldn't finish. I would try
rake build but this would raise an error requiring me to dig through a
project to find a missing dependency.
Forcing myself to rerun a project from scratch at the start of each day
strictly enforced that a project always worked. If I had added a new dependency
to a project but not specified it, this would immediately become obvious the
next day when I cloned a fresh copy. Furthermore it is very satisfying to clone
a project I haven't worked on in a while, run
make, and have every workflow
step run error-free to generate the data I was after. This is particularly
useful when finishing a manuscript as I find myself coming back repeatedly to
make minor tweaks during the submission and review process.
Break up larger projects into smaller units
Repeatedly cloning and re-running a project becomes unfeasible when there is a
computational bottleneck requiring days to run. This is usually indicative that
the project should be split into two separate sub-projects. I do this by adding
a step to the first sub-project to compress the result of the long running
xz and push this to my S3 account using
s3cmd. The first step
of the following sub-project is then to download and decompress this data. This
effectively caches the result of a long running process so I can continue to
experiment with downstream analyses steps without repeatedly rerunning a single
XZ compresses extremely well so download times have never been a time consuming problem. However in future I'm considering taking this a step further and using git-annex so that the data is also version controlled as a part of the analysis too.
Minimal data loss
As everything is cleared from my cache directory when I reboot my computer,
it's easy to lose work if I forget to push back changes. I use git to version
control all of my research projects so I have to remember to run
git push on
everything before I switch off for the day. There have however been three
occasions where I have forgotten to do this and lost data. As you can imagine
it is very frustrating to lose a day's work.
On the other hand there was one occasion where my laptop was stolen. As I had been forcing myself to work off remote copies of everything for several months I didn't lose a single piece of data from two in-progress manuscripts. I think the advantage of forcing yourself to back up projects as part of the development process is obvious.
Simpler to run in the cloud
Each sub-project is encapsulated and reproducible via a Makefile. This
therefore often means there is minimal effort required to set up the workflow
to run on a larger, more powerful computer. If I have a long running process I
tend to launch a large EC2 instance, clone my project onto it, then run
I can then leave this running and check back time-to-time on the progress.
Furthermore if I designed the process to include parallisable tasks, as I
described in my Makefile post, it is simple to scale up the number of
cores used with the
make -j X command where
X is the number of cores
available to the instance.
I believe the advantage of breaking a larger project smaller sub-projects is that they become easier to maintain. Sub-projects are less coupled to each other having no shared code or state. The single connection is the format of data passed between them. This makes it much easier to make changes in a one part of the project without affecting the other parts. Furthermore by keeping nothing local to my computer this ensures that reproducibility and backups becomes a part of my development process.