16

Who of you in this forum uses ">R with the multicore, snow packages, or CUDA, so for advanced calculations that need more power than a workstation CPU? On which hardware do you compute these scripts? At home/work or do you have data center access somewhere?

The background of these questions is the following: I am currently writing my M.Sc. thesis about R and High Performance Computing and need a strong knowledge about who actually uses R. I read that R had 1 million users in 2008, but that's more or less the only user statistics I could find on this topic - so I hope for your answers!

Sincerely Heinrich

chl
  • 50,972
  • 18
  • 205
  • 364
Heinrich
  • 111
  • 1
  • 7
  • A possible related question, http://stats.stackexchange.com/questions/825/any-suggestions-for-making-r-code-use-multiple-processors. – chl Nov 16 '10 at 08:33
  • Exact duplicate of https://stat.ethz.ch/pipermail/r-help/2010-November/259921.html – Joshua Ulrich Nov 17 '10 at 17:03

4 Answers4

6

I am a biologist who models the effects of inter-annual climatic variation on population dynamics of several migratory species. My datasets are very large (spatially intensive data) so I run my R code using multicore on Amazon EC2 servers. If my task is particularly resource intensive, I will choose a High Memory Quadruple Extra Large instance which comes with 26 CPU units, 8 cores, and 68G of RAM. In this case I usually run 4-6 scripts simultaneously, each of which is working through a fairly large data set. For smaller tasks, I choose servers with 4-6 cores and about 20 gigs of RAM.

I launch these instances (usually spot instances because they are cheaper but can terminate anytime the current rate exceeds what I have chosen to pay), run the script for several hours, and then terminate the instance once my script has finished. As for the machine image (Amazon Machine Image), I took someone elses Ubuntu install, updated R, installed my packages, and saved that as my private AMI on my S3 storage space.

My personal machine is a dualcore macbook pro and it has a hard time forking multicore calls. Feel free to email if you have other questions.

Maiasaura
  • 363
  • 2
  • 8
4

Since you ask, I am using the foreach package with the multicore backend. I use it to split an embarrassingly parallel workload across multiple cores on a single Nehalem box with lots of RAM. This works pretty well for the task at hand.

NPE
  • 5,351
  • 5
  • 33
  • 44
  • Thanks for your answer! Do you do the computation for your work/academic research or for own projects on a own PC? – Heinrich Nov 16 '10 at 09:59
  • This is done in a commercial setting. For this task, I am using a single Intel box with 32GB of RAM and RAIDed disks (the main difficulty is lots of data, while the processing itself is not very computationally demanding.) – NPE Nov 16 '10 at 11:01
  • Alright @aix, how often do you perform these calculations. Is you box running all the day or more idle? – Heinrich Nov 16 '10 at 12:38
  • Quick question to @NPE: in what system do you store the data ? do you use a database back-end ? – nassimhddd Nov 26 '12 at 08:47
3

I work in academy and I'm using multicore for some heavy benchmarks of machine learning algorithms, mostly on our Opteron based Sun Constellation and some smaller clusters; those are also rather embarrassingly parallel problems so multicore's main role is to spread computation over node without multiplication of memory usage.

  • We here in Hamburg always have a problem that the waiting time for the academic data centers are really long. is it the same for you? – Heinrich Nov 16 '10 at 13:16
  • @Heinrich I work for a kind of academic data center, so I don't have such problems (-; Seriously, in Warsaw the scientific CPU time supply is larger than demand, so I believe it is quite easy to get a grant. And I think you should try D-Grid or EGEE, my experience is that grids in general are very underused. –  Nov 16 '10 at 14:16
  • Oh. That is interesting. Dow you know in what kind of businesses R is used in these extends? – Heinrich Nov 16 '10 at 16:22
2

I use snow and snowfall for course parallelization on HPC clusters and CUDA for fine data parallel processing. I'm in Epidemiology doing disease transmission modeling. So I use both.

Andrew Redd
  • 223
  • 1
  • 6
  • Thanks for your info. What do you mean with course parallelization? – Heinrich Nov 17 '10 at 07:13
  • Course parallelization would be something like independent runs of a MCMC change., i.e. very large chucks that can be ran in parallel without syncing threads. An example of fine grain is computing the likelihood where calculations can be performed on the data points independently. – Andrew Redd Nov 17 '10 at 17:23