What are efficient ways to organize R code and output?

Question

I am looking for input on how others organize their R code and output.

My current practice is to write code in blocks in a text file as such:

#=================================================
# 19 May 2011
date()
# Correlation analysis of variables in sed summary
load("/media/working/working_files/R_working/sed_OM_survey.RData")
# correlation between estimated surface and mean perc.OM in epi samples
cor.test(survey$mean.perc.OM[survey$Depth == "epi"], 
    survey$est.surf.OM[survey$Depth   == "epi"]))
#==================================================

I then paste the output into another text file, usually with some annotation.

The problems with this method are:

1. The code and the output are not explicitly linked other than by date.
2. The code and output are organized chronologically and thus can be hard to search.

I have considered making one Sweave document with everything since I could then make a table of contents but this seems like it may be more hassle than the benefits it would provide.

Let me know of any effective routines you have for organizing your R code and output that would allow for efficient searching and editing the analysis.

Answer 1

You are not the first person to ask this question.

• Managing a statistical analysis project – guidelines and best practices
• A workflow for R
• R Workflow: Slides from a Talk at Melbourne R Users by Jeromy Anglim (including another much longer list of webpages dedicated to R Workflow)
• My own stuff: Dynamic documents with R and LATEX as an important part of reproducible research
• More links to project organization: How to efficiently manage a statistical analysis project?

Answer 2

I for one organize everything into 4 files for every project or analysis. (1) 'code' Where I store text files of R functions. (2) 'sql' Where I keep the queries used to gather my data. (3) 'dat' Where I keep copies (usually csv) of my raw and processed data. (4) 'rpt' Where I store the reports I've distributed.

ALL of my files are named using very verbose names such as 'analysis_of_network_abc_for_research_on_modified_buffer_19May2011'

I also write detailed documentation up front where I organize the hypothesis, any assumptions, inclusion and exclusion criteria, and steps I intend to take to reach my deliverable. All of this is invaluable for repeatable research and makes my annual goal setting process easier.

Answer 3

Now that I've made the switch to Sweave I never want to go back. Especially if you have plots as output, it's so much easier to keep track of the code used to create each plot. It also makes it much easier to correct one minor thing at the beginning and have it ripple through the output without having to rerun anything manually.

Answer 4

For structuring single .R code files, you can also use strcode, a RStudio add-in I created to insert code separators (with optional titles) and based on them - obtain summaries of code files. I explain the usage of it in more detail in this blog post.