Programming with Big Data in R

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages)

The topic of this article may not meet Misplaced Pages's general notability guideline. Please help to demonstrate the notability of the topic by citing reliable secondary sources that are independent of the topic and provide significant coverage of it beyond a mere trivial mention. If notability cannot be shown, the article is likely to be merged, redirected, or deleted.
Find sources: "Programming with Big Data in R" – news · newspapers · books · scholar · JSTOR (June 2013) (Learn how and when to remove this message)

A major contributor to this article appears to have a close connection with its subject. It may require cleanup to comply with Misplaced Pages's content policies, particularly neutral point of view. Please discuss further on the talk page. (June 2013) (Learn how and when to remove this message)

(Learn how and when to remove this message)

bdrp
Influenced by
Paradigm	SPMD and MPMD
Designed by	Wei-Chen Chen, George Ostrouchov, Pragneshkumar Patel, and Drew Schmidt
Developer	pbdR Core Team
First appeared	September 2012; 12 years ago (2012-09)

Preview release	Through GitHub at RBigData

Typing discipline	Dynamic
OS	Cross-platform
License	General Public License and Mozilla Public License
Website	www.r-pbd.org
R, C, Fortran, MPI, and ØMQ

Programming with Big Data in R (pbdR) is a series of R packages and an environment for statistical computing with big data by using high-performance statistical computation. The pbdR uses the same programming language as R with S3/S4 classes and methods which is used among statisticians and data miners for developing statistical software. The significant difference between pbdR and R code is that pbdR mainly focuses on distributed memory systems, where data are distributed across several processors and analyzed in a batch mode, while communications between processors are based on MPI that is easily used in large high-performance computing (HPC) systems. R system mainly focuses on single multi-core machines for data analysis via an interactive mode such as GUI interface.

Two main implementations in R using MPI are Rmpi and pbdMPI of pbdR.

The pbdR built on pbdMPI uses SPMD parallelism where every processor is considered as worker and owns parts of data. The SPMD parallelism introduced in mid 1980 is particularly efficient in homogeneous computing environments for large data, for example, performing singular value decomposition on a large matrix, or performing clustering analysis on high-dimensional large data. On the other hand, there is no restriction to use manager/workers parallelism in SPMD parallelism environment.
The Rmpi uses manager/workers parallelism where one main processor (manager) serves as the control of all other processors (workers). The manager/workers parallelism introduced around early 2000 is particularly efficient for large tasks in small clusters, for example, bootstrap method and Monte Carlo simulation in applied statistics since i.i.d. assumption is commonly used in most statistical analysis. In particular, task pull parallelism has better performance for Rmpi in heterogeneous computing environments.

The idea of SPMD parallelism is to let every processor do the same amount of work, but on different parts of a large data set. For example, a modern GPU is a large collection of slower co-processors that can simply apply the same computation on different parts of relatively smaller data, but the SPMD parallelism ends up with an efficient way to obtain final solutions (i.e. time to solution is shorter).

Package design

Programming with pbdR requires usage of various packages developed by pbdR core team. Packages developed are the following.

General	I/O	Computation	Application	Profiling	Client/Server
pbdDEMO	pbdNCDF4	pbdDMAT	pmclust	pbdPROF	pbdZMQ
pbdMPI	pbdADIOS	pbdBASE	pbdML	pbdPAPI	remoter
		pbdSLAP		hpcvis	pbdCS
		kazaam			pbdRPC

Among these packages, pbdMPI provides wrapper functions to MPI library, and it also produces a shared library and a configuration file for MPI environments. All other packages rely on this configuration for installation and library loading that avoids difficulty of library linking and compiling. All other packages can directly use MPI functions easily.

pbdMPI --- an efficient interface to MPI either OpenMPI or MPICH2 with a focus on Single Program/Multiple Data (SPMD) parallel programming style
pbdSLAP --- bundles scalable dense linear algebra libraries in double precision for R, based on ScaLAPACK version 2.0.2 which includes several scalable linear algebra packages (namely BLACS, PBLAS, and ScaLAPACK).
pbdNCDF4 --- interface to Parallel Unidata NetCDF4 format data files
pbdBASE --- low-level ScaLAPACK codes and wrappers
pbdDMAT --- distributed matrix classes and computational methods, with a focus on linear algebra and statistics
pbdDEMO --- set of package demonstrations and examples, and this unifying vignette
pmclust --- parallel model-based clustering using pbdR
pbdPROF --- profiling package for MPI codes and visualization of parsed stats
pbdZMQ --- interface to ØMQ
remoter --- R client with remote R servers
pbdCS --- pbdR client with remote pbdR servers
pbdRPC --- remote procedure call
kazaam --- very tall and skinny distributed matrices
pbdML --- machine learning toolbox

Among those packages, the pbdDEMO package is a collection of 20+ package demos which offer example uses of the various pbdR packages, and contains a vignette that offers detailed explanations for the demos and provides some mathematical or statistical insight.

Examples

Example 1

Hello World! Save the following code in a file called "demo.r"

### Initial MPI
library(pbdMPI, quiet = TRUE)
init()
comm.cat("Hello World!\n")
### Finish
finalize()

and use the command

mpiexec -np 2 Rscript demo.r

to execute the code where Rscript is one of command line executable program.

Example 2

The following example modified from pbdMPI illustrates the basic syntax of the language of pbdR. Since pbdR is designed in SPMD, all the R scripts are stored in files and executed from the command line via mpiexec, mpirun, etc. Save the following code in a file called "demo.r"

### Initial MPI
library(pbdMPI, quiet = TRUE)
init()
.comm.size <- comm.size()
.comm.rank <- comm.rank()
### Set a vector x on all processors with different values
N <- 5
x <- (1:N) + N * .comm.rank
### All reduce x using summation operation
y <- allreduce(as.integer(x), op = "sum")
comm.print(y)
y <- allreduce(as.double(x), op = "sum")
comm.print(y)
### Finish
finalize()

and use the command

mpiexec -np 4 Rscript demo.r

to execute the code where Rscript is one of command line executable program.

Example 3

The following example modified from pbdDEMO illustrates the basic ddmatrix computation of pbdR which performs singular value decomposition on a given matrix. Save the following code in a file called "demo.r"

# Initialize process grid
library(pbdDMAT, quiet=T)
if(comm.size() != 2)
  comm.stop("Exactly 2 processors are required for this demo.")
init.grid()
# Setup for the remainder
comm.set.seed(diff=TRUE)
M <- N <- 16
BL <- 2 # blocking --- passing single value BL assumes BLxBL blocking
dA <- ddmatrix("rnorm", nrow=M, ncol=N, mean=100, sd=10)
# LA SVD
svd1 <- La.svd(dA)
comm.print(svd1$d)
# Finish
finalize()

and use the command

mpiexec -np 2 Rscript demo.r

to execute the code where Rscript is one of command line executable program.

References

Ostrouchov, G., Chen, W.-C., Schmidt, D., Patel, P. (2012). "Programming with Big Data in R".{{cite web}}: CS1 maint: multiple names: authors list (link)
Chen, W.-C. & Ostrouchov, G. (2011). "HPSC -- High Performance Statistical Computing for Data Intensive Research". Archived from the original on 2013-07-19. Retrieved 2013-06-25.
"Basic Tutorials for R to Start Analyzing Data". 3 November 2022.
^ Yu, H. (2002). "Rmpi: Parallel Statistical Computing in R". R News.
Mike Houston. "Folding@Home - GPGPU". Retrieved 2007-10-04.
"100 most read R posts in 2012 (stats from R-bloggers) – big data, visualization, data manipulation, and other languages".

External links

Official website

Categories:

Package design

Examples

Example 1

Example 2

Example 3

Further reading

References

External links