From Wikipedia, the free encyclopedia

Data Parallelism is a form of parallelization of computer code. It is meant to distribute computing across multiple processors in parallel computing environments. It contrasts to Task Parallelism as another form of parallelism.

Contents 1 Description 2 Example 3 References 4 See also

In a multiprocessor system executing a single set of instructions (SIMD), data parallelism is achieved when each processor performs the same task on different data. For instance, if we are running code on a 2-processor system (CPUs "a" & "b") in a parallel environment and we wish to do a task on some data "d", it is possible to tell CPU "a" to do that task on one part of "d" and CPU "b" on another part simultaneously, thereby reducing the runtime of the execution. The data can be assigned using conditional statements as described below. Most programs fall somewhere on the continuum between Data parallelism and Task parallelism.

The pseudocode below illustrates data parallelism:

program:
...
if CPU="a" then
   low_limit=1
   upper_limit=50
else if CPU="b" then
        low_limit=51
        upper_limit=100
end if
do i = low_limit , upper_limit
       Task on d(i)
end do
...
end program

The goal of the program is to do some task on the data array "d" of size 100 (say). If we write the code as above and launch it on a 2-processor system, then the runtime environment will execute it as follows.

• In a SIMD system, both CPUs will execute the code.
• In a parallel environment, both will have access to "d".
• A mechanism is presumed to be in place whereby each CPU will create its own copy of "low_limit" and "upper_limit" that is independent of the other
• The "if" clause differentiates between the CPU's. CPU "a" will read true on the "if" and CPU "b" will read true on the "else if", thus having their own values of "low_limit" and "upper_limit"
• Now, both CPU's execute "Task on d(i)", but since each cpu has different values of the "limits", they operate on different parts of "d" simultaneously, thereby distributing the task among themselves. Obviously, this will be faster than doing it on a single CPU.

Code executed by CPU "a":

program:
...
low_limit=1
upper_limit=50
do i = low_limit , upper_limit
       Task on d(i)
end do
...
end program

Code executed by CPU "b":

program:
...
low_limit=51
upper_limit=100
do i = low_limit , upper_limit
       Task on d(i)
end do
...
end program

This concept can now be generalized to any number of processors.

• Quinn Michael J, Parallel Programming in C with MPI and OpenMP McGraw-Hill Inc. 2004. ISBN 0-07-058201-7
• Blelloch Guy E, Vector Models for Data-Parallel Computing MIT Press 1990. ISBN 0-262-02313-X

• Task Parallelism

	Topics in Parallel Computing	v • d • e
General High-performance computing Parallelism Data parallelism • Task parallelism Theory Speedup • Amdahl's law • Flynn's Taxonomy • Cost efficiency • Gustafson's Law • Karp-Flatt Metric Elements Process • Thread • Fiber • Parallel Random Access Machine Coordination Multiprocessing • Multithreading •Multitasking • Memory coherency • Cache coherency • Barrier • Synchronization • Distributed computing • Grid computing Programming Programming model • Implicit parallelism • Explicit parallelism Hardware Computer cluster • Beowulf • Symmetric multiprocessing • Non-Uniform Memory Access • Cache only memory architecture • Asymmetric multiprocessing • Simultaneous multithreading • Shared memory • Distributed memory • Massively parallel processing • Superscalar processing • Vector processing • Supercomputer Software Distributed shared memory • Application checkpointing APIs Pthreads • OpenMP • Message Passing Interface (MPI) Problems Embarrassingly parallel • Grand Challenge