Misplaced Pages

Revision as of 04:19, 7 November 2007

Parallel computing is the use of "processing elements that communicate and cooperate to solve larger problems fast." Parallel computing operates on the principle that large problems can almost always be divided into smaller ones, which may be carried out concurrently ("in parallel"). With the end in 2004 of frequency scaling as the major driving force in computer performance increases, parallel computing - in the form of multicore processors - has become the dominant paradigm in computer architecture. The speed up of a program as a result of parallelization is given by Amdahl's law.

Distributed computing is a specific form of parallel computing, where the processing elements are located on different computers connected by a network. Cluster computing and grid computing are the two most common forms of distributed computing. High-performance computing is the branch of parallel computing dealing with very large problems and large parallel computers - supercomputers - that can solve those problems in a reasonable amount of time.

Parallel computing exists in several different forms: bit-level parallelism, instruction level parallelism, data parallelism, and task parallelism.

Parallel computer programs are harder to write than sequential ones. Concurrency introduces several new classes of potential software bugs, of which race conditions are the most common. Communication and synchronization between the different subtasks is typically one of the greatest barriers to getting good parallel program performance. In recent years, power usage in parallel computers has also become a great concern.

Background

The move to (multicore) parallelism

Frequency scaling was the dominant reason for computer performance increases in from the mid 1980s until 2004. The effect of processor frequency on computer speed can be seen by looking at the equation for computer program runtime:

${\displaystyle Runtime={\frac {Instructions}{Program}}\times {\frac {Cycles}{Instruction}}\times {\frac {Seconds}{Cycles}}}$

where Instructions per program is the total instructions being executed in a given program, cycles per instruction is a program-dependent, architecture-dependent average value, and seconds per cycles is by definition the inverse of frequency. An increase in frequency thus decreases runtime.

However, power consumption in a chip is given by the equation

${\displaystyle P=C\times V^{2}\times F}$

where P is power, C is the capacitance being switched per clock cycle, V is voltage, and F is the processor frequency (cycles per second). Increases in frequency thus increase the amount of power used in a processor. Increasing processor power consumption ultimately to Intel's May 2004 cancellation of its Tejas and Jayhawk processors, which is generally cited as the end of frequency scaling as the dominant computer architecture paradigm.

Despite these power issues, transistor densities are still doubling every 18 to 24 months per Moore's Law. Now that these transistors are not needed to facilitate frequency scaling, they can be used to add extra hardware to facilitate parallel computing. This is the basis for the current push towards multicore computing paradigm.

Amdahl's law and Gustafson's law

A large math or engineering problem typically consists of several parallelizable parts and several non-parallelizable (sequential) parts. The sequential parts limit the amount that the program can be sped up. This relationship is given by Amdahl's law:

${\displaystyle S={\frac {1}{(1-P)}}}$

where S is the speedup of the program (as a factor of its original runtime), and P is the fraction that is parallelizable.

Gustafson's law is another law in computer engineering, closely related to Amdahl's law. Gustafson's law can be formulated as:

${\displaystyle S(P)=P-\alpha *(P-1)}$

where P is the number of processors, S is the speedup, and ${\displaystyle \alpha }$ the non-parallelizable part of the process. Amdahl's law assumes a fixed-problem size and that the size of the sequential section is independent of the number of processors, whereas Gustafson's law does not make these assumptions.

Race conditions, Mutual exclusion, and Synchronization

Subtasks in a parallel program are often called threads. Some parallel computer architectures use smaller, lightweight versions threads known as fibers, while others use bigger versions known as processes. However, "threads" is the generally accepted as a generic term for subtasks.

Threads will often need to update some variable that is shared between them. The instructions between the two programs may be interleaved in any order. For example, consider the following program

If instruction 1B is executed between 1A and 3A, or if instruction 1A is executed between 1B and 3B, the program will produce incorrect data. This is known as a race condition. The programmer must use a lock to provide mutual exclusion. A lock is a programming language construct that allows one thread to take control of a variable and prevent other threads from reading or writing it, until that variable is unlocked. The thread holding the lock is free to execute its critical section (the section of a program that requires exclusive access to some variable), and unlock the data when it is finished.

Therefore, in order to guarantee correct program execution, the above program can be rewritten to use locks:

One thread will successfully lock variable V, while the other thread will be locked out - unable to proceed until V is unlocked again. This guarantees correct execution of the program. Locks, while necessary to ensure correct program execution, can greatly slow down a program.

Locking multiple resources using non-atomic locks introduces the possibility of program deadlock.

Many parallel programs require that their subtasks act in synchronization. This requires the use of a barrier. Barriers are typically implemented using a software lock.

Fine-grained, Coarse grained, and Embarrassing parallelism

Applications are often classified according to how often their subtasks need to synchronize or communicate with each other. An application exhibits fine-grained parallelism if its subtasks must communicate many times per second; it exhibits course-grained parallelism if they do not communicate many times per second, and it is embarrassingly parallel if they rarely or never have to communicate. Embarrassingly parallel applications are considered the easiest to parallelize.

Consistency models

Parallel programming languages and parallel computers must have a consistency model (also known as a memory model). The consistency model defines rules for how operations on computer memory occur and how results are produced.

One of the first consistency models was Leslie Lamport's sequential consistency model. Sequential consistency is the property of a parallel program that its parallel execution produces the same results as a sequential program. Specifically, a program is sequentially consistent if "...the results of any execution is the same as if the operations of all the processors were executed in some sequential order, and the operations of each individual processor appear in this sequence in the order specified by its program."

Software transactional memory is a common type of consistency model. Software transactional memory borrows from database theory the concept of atomic transactions and applies them to memory accesses.

Mathematically, these models can be represented in a number of ways. Process calculus is the branch of mathematics dealing with concurrency. Process calculus can be subdivided into Ambient calculus, Calculus of communicating systems, and Communicating sequential processes. Petri nets were an early attempt to codify the rules of consistency models. Dataflow theory later built open these. Dataflow architectures were later created that physically implement the idea in dataflow.

Flynn's Taxonomy

Michael J Flynn created one of the earliest classification systems for parallel (and sequential) computers and programs, now known as Flynn's Taxonomy. Flynn classified programs and computers by whether they were operating using a single or multiple sets of instructions, whether or not those instructions were using a single or multiple sets of data.

The Single-instruction-single-data (SISD) classification is equivalent to an entirely sequential program. The single-instruction-multiple-data (SIMD) classification is analogous to doing the same operation repeatedly over a large data set. This is commonly done in signal processing applications. Multiple-instruction-single data (MISD) is a rarely used classification. While computer architectures to deal with this were devised (such as systolic arrays), few applications that fit this class materialized. Multiple-instruction-multiple-data (MIMD) programs are by far the most common type of parallel programs.

"Some machines are hybrids of these categories, of course, but this classic model has survived because it is simple, easy to understand, and gives a good first approximation. It is also - perhaps because of its understandability - the most widely used scheme."

Types of Parallelism

Bit-level parallelism

From the advent of Very-large-scale integration (VLSI) computer chip fabrication technology in the 1970s until about 1986, advancements in computer architecture were done by doubling computer word size - the amount of information the processor can execute per cycle. Increasing the word size reduces the number of instructions the processor must execute in order to perform an operation on variables whose sizes are greater than the length of the word. (For example, consider a case where an 8 bit processor must add two 16 bit integers. The processor must first add the 8 lower-order bits from each integer, then add the 8 higher order bits, requiring two instructions to complete a single operation. A 16 bit processor would be able to complete the operation with single instruction)

Historically, 4-bit microprocessors were replaced with 8-bit, then 16-bit, then 32-bit microprocessors. This trend generally came to an end with the introduction of 32 bit processors, which has been a standard in general purpose computing for two decades. Only recently, with the advent of x86-64 architectures, have 64-bit processors become commonplace.

Instruction level parallelism

A computer program is, in essence, a stream of instructions executed by a processor. These instructions can be re-ordered and combined into groups which are then executed in parallel without changing the result of the program. This is known as instruction level parallelism.

Modern processors have multi-stage instruction pipelines. Each stage in the pipeline corrosponds to a different action the processor performs on that instruction in that stage. In other words, a processor with N pipeline stages can have up to N different instructions at different stages of completion. The canonical example of a pipelined processor is a RISC processor with 5 stages: instruction fetch, decode, execute, memory access, write back. The Pentium 4 processor had a 35-stage pipeline.

In addition to instruction level parallelism from pipelining, some processors processors can issue more than one instruction at a time. These are known as superscalar processors. Instructions can be grouped together only if there is no data dependency between them.

Scoreboarding and the Tomasulo algorithm (which is similar to scoreboarding but makes use of register renaming) are two of the most common techniques for implementing out-of-order execution and instruction level parallelism.

Advancements in instruction level parallelism dominated computer architecture from the mid-1980s until the mid-1990s.

Data parallelism

Data parallelism is parallelism inherent in program loops. "Parallelizing loops often leads to simliar (not necessarily identical) operation sequences or functions being performed on elements of a large data structure." Many scientific and engineering applications exhibit data parallelism.

A loop-carried dependency is the property of a loop iteration that it depends on the output of previous iterations. Loop carried dependencies prevent parallelization of loops.

As the size of a problem gets bigger, the amount of data-parallelism available usually does as well.

Task parallelism

Task parallelism is the characteristic of a parallel program that "entirely different calculations can be performed on either the same or different sets of data". This contrasts with data parallelism, where the same calculation is being performed on the same or different sets of data. Task parallelism usually does not scale with the size of a problem.

Hardware

Memory and Communication

Main memory in a parallel computer is either shared memory - shared between all processing elements in a single address space; or distributed memory - where each processing element has its own local address space. Distributed memory refers to the fact that the memory is logically distributed, but often implies that it is physically distributed as well. Distributed shared memory is a combination of the two approaches, where the processing element has its own local memory and access to the memory on non-local processors. Accesses to local memory are typically faster than accesses to non-local memory.

Computer architectures in which all of main memory can be accessed with equal latency and bandwidth are known as Uniform Memory Access (UMA) systems. Typically, only a shared memory system (where the memory is not physically distributed) can achieve these. A system that does not have this property is known as a Non-Uniform Memory Access (NUMA) architecture. Distributed memory systems have non-uniform memory access.

Computer systems make use of caches - small, fast memories located close to the processor which store temporary copies of memory values. (Nearby in both the physical and logical sense.) Parallel computer systems have difficulties with caches that may store the same value in more than one location, creating the possibility of incorrect program execution. These computers require a cache coherency system, which keeps track of cached values and strategically purges them, thus ensuring correct program execution. Bus snooping is one of the most common methods for keeping track of which values are being accessed (and thus should be purged). Designing large, high-performance cache coherence systems is a very difficult problem in computer architecture. As a result, shared-memory computer architectures do not scale as well as distributed memory systems do.

The processor-processor and processor-memory communication can be implemented in hardware in a number of ways, including via shared (either multiported or multiplexed) memory, a crossbar switch, a shared bus or an interconnect network of a myriad of topologies including star, ring, tree, hypercube, fat hypercube (a hypercube with more than one processor at a node), an n-dimensional mesh, etc.

Parallel computers based on interconnect network need to employ some kind of routing to enable passing of messages between nodes that are not directly connected. The communication medium used for communication between the processors is likely to be hierarchical in large multiprocessor machines.

Classes of parallel computers

Parallel computers can be classified roughly into classes according to the level at which the hardware supports parallelism. This is roughly analagous to the distance between basic computing nodes.

Note that these classifications are not mutually exclusive. For example, clusters of symmetric multiprocessors are relatively common.

Multicore computing

A multicore processor is a processor which includes multiple execution units ("cores"). Multicore processors differ from superscalar processors in that a superscalar processor can issue multiple instructions per cycle from one instruction stream (thread), whereas a multicore processor can issue multiple instructions per cycle from multiple instruction streams. Each core in a multicore processor can potentially be superscalar as well - that is, every cycle, each core can issue multiple instructions from one instruction stream.

Simultaneous multithreading (of which Intel's HyperThreading is the best known) was an early form of pseudo-multicoreism. A processor capable of simultaneous multithreading has only one execution unit ("core"), but at times that that execution unit would be idling (such as during a cache miss), it uses that execution unit to process a second thread.

Intel's Core and Core 2 processor families are Intel's first true multicore architectures. IBM's Cell microprocessor, designed for use in the Sony Playstation 3, is another of the best-known multicore processors on the market.

Symmetric multiprocessing

A symmetric multiprocessor (SMP) is a computer system with multiple identical processors that share memory and connect via a bus. Bus contention prevents bus architectures from scaling. As a result, SMPs generally no not exceed 32 processors. "Because of the small size of the processors and the significant reduction in the requirements for bus bandwidth achieved by large caches, such symmetric multiprocessors are extremely cost-effective, provided that a sufficient amount of memory bandwidth exists"

Distributed memory multiprocessing

Cluster computing

A cluster is a group of loosely coupled computers that work together closely so that in many respects they can be viewed as though they are a single computer. Clusters are composed of multiple standalone machines connected by a network. While machines in a cluster do not have to be symmetric, load balancing is more difficult if they are not.

The most common type of cluster is the Beowulf cluster, which is a cluster implemented on multiple identical commercial off-the-shelf computers connected with a TCP/IP ethernet local area network. Beowulf technology was originally developed by Thomas Sterling and Donald Becker.

The vast majority of the TOP500 supercomputers are clusters.

Massive parallel processing

A massively parallel processor (MPP) is a single computer with a very large number of networked processors. MPPs have many of the same characteristics as clusters, but they are usually larger, typically having "far more" than 100 processors In an MPP, "each CPU contains its own memory and copy of the operating system and application. Each subsystem communicates with the others via a high-speed interconnect."

Blue Gene/L, the fastest supercomputer in the world according to the TOP500 ranking, is an MPP.

Grid computing

Grid computing is the most distributed form of parallel computing. Grid computing makes use of computers many miles apart, connected by the Internet, to work on a given problem. Because of the low bandwidth and extremely high latency typically available on the internet, typically grid computing deals only with embarrassingly parallel problems. Many grid computing applications have been created, of which SETI@home and Folding@Home are best known examples

Most grid computing applications use middleware - software that operates between the operating system and the application, which manages network resources and a standardizes the software interface for grid computing applications. The most common grid computing middleware is the Berkeley Open Infrastructure for Network Computing (BOINC). Often, grid computing software makes use of "spare cycles", doing computations at times when a computer is idling.

Specialized parallel computers

Within parallel computing, there are specialized parallel devices which remain niche areas of interest. While not domain specific, they tend to be applicable only to a few classes of parallel problems.

Reconfigurable computing with Field-programmable gate arrays

Reconfigurable computing is the use of a Field-programmable gate array (FPGA) as a co-processor to a general-purpose computer. An FPGA is, in essence, a computer chip which can rewire itself for a given task.

FPGAs can be programmed with hardware description languages such as VHDL or Verilog. However, programming in these languages can be tedious. A number of vendors have created C to HDL languages, that attempt to emulate the syntax and/or semantics of the C programming language, which most programmers are familiar with. The best known C to HDL languages are Mitrion-C, Impulse C, DIME-C, and Handel-C.

AMD's decision to open its HyperTransport technology to third party vendors has become the enabling technology for high performance reconfigurable computing. According to Michael R. D'Amour, CEO of DRC Computer Corporation, "when we first walked into AMD, they called us 'the socket stealers.' Now they call us their partners.

GPGPU with graphics processing units

General-purpose computing on graphics processing units (GPGPU) is a fairly recent trend in computer engineering research. GPUs are co-processors that have been heavily optimized for computer graphics processing. Computer graphics processing is a field dominated by data parallel operations - particularly linear algebra matrix operations.

CUDA is the dominant GPGPU a programming language. CUDA was developed by NVIDIA for use on NVIDIA graphics cards. Other GPU programming languages are BrookGPU, PeakStream, and RapidMind.

NVIDIA Tesla is NVIDIA's first dedicated GPGPU card.

Application-specific integrated circuits

A number of Application-specific integrated circuit (ASIC) approaches have been devised for dealing with parallel applications.

Because an ASIC is (by definition) specific to a given application, it can be fully optimized for that application. As a result, for a given application, an ASIC tends to outperform a general purpose computer. However, ASICs are created by X-ray lithography. This process requires a mask, which can be extremely expensive. A single mask can cost over a million United States dollars. (The smaller the transistors required for the chip, the more expensive the mask will be.) Meanwhile, performance increases in general purpose computing as a result of Moore's Law tend to wipe out these gains in only one or two chip generations. Such high initial cost, and the tendency to be overtaken by Moore's law-driven general purpose, render ASICs unfeasible for most parallel computing applications.

Vector processors

A vector processor is a CPU or computer system that can execute the same instruction on large sets of data. Under Flynn's classification scheme, these are SIMD machines.

Cray computers became famous for their vector processoring computers in the 1970s and 1980s. However, vector processors - both as CPUs and as full computer systems - have generally disappeared. Modern processor instruction sets do include some vector processing instructions, such as with AltiVec and Streaming SIMD Extensions (SSE).

Power and the Heat Problem

In the past, computer engineers viewed power as an essentially unlimited resource. However, in recent years, power consumption has become a concern.

Every joule of energy used by a computer is ultimately turned into heat. (This is a consequence of the law of conservation of energy.) If it builds up inside a computer, this heat can damage electronics. Computer cooling technologies - typically a heat sink and fan - are used to remove this heat from inside the computer and release it into the surrounding room. A data center, where many computers are gathered in room, must actively remove this heat the computer room, typically using large air conditioning units.

Processors are the point - with high clock frequencies and transistor densities - where traditional computer cooling technologies cannot keep up with power consumption, and physical damage from heat becomes greater. Power is needed both to run the computers, and to run the active cooling systems that remove the heat they produce. This can be very expensive, and it is possible that local power companies may simply be unable to supply the desired power. Taken together, these are referred to as the "heat problem."

Power performance - the number of useful computations that can be done per watt of power - is now seen as a critically important measure of future parallel computers. These trends have led to the recent introduction of power-conserving technologies. Processor power consumption is linearly proportional to the processor's frequency and quadratically proportion to the processor's voltage. Recent processors include dynamic frequency scaling and dynamic voltage scaling technologies throttle back processor performance (and power consumption) during low-load times.

Software

A number of concurrent programming languages, libraries, APIs, and programming models have been created for programming parallel computers.

These can generally be divided into classes based on the assumptions they make about the underlying memory architecture - shared memory, distributed memory, or shared distributed memory. Shared memory programming languages communicate by means of manipulating shared memory variables. Distributed memory uses message passing. POSIX Threads and OpenMP are two of most widely used shared memory APIs, whereas Message Passing Interface (MPI) is the most widely used message passing system API.

Automatic parallelization

Automatic parallelization of a sequential program by a compiler is the "holy grail" of parallel computing. Despite decades of work by compiler researchers, automatic parallelization has had only limited success.

Parallel programming languages remain either explicitly parallel or (at best) partially implicit with the programmer giving the compiler directives for parallelization.

Application checkpointing

The larger and more complex a computer gets, the more can go wrong, and the smaller the mean time between failures becomes. Application checkpointing is technique whereby the computer system takes a "snapshot" of the application - a record of all current resource allocations and states variables, akin to a core dump. This information can then be used to restore the program if the computer should fail. Application checkpointing means that the program will only have to restart from its last checkpoint, rather than the beginning. For an application that may takes months, this is critically important. Application checkpointing may also be used to faciliate process migration.

History

The origins of true (MIMD) parallelism go back to Federico Luigi, Conte Menabrea and his "Sketch of the Analytic Engine Invented by Charles Babbage" C.mmp, a 1970s multi-processor project at Carnegie Mellon University, was "among the first multiprocessors with more than a few processors." "The first bus-connected multi-processor with snooping caches was the Synapse N+1 in 1984".

SIMD parallel computers can be traced back to the 1970s. The motivation behind early SIMD computers was to amortize the gate delay of the processor's control unit over multiple instructions. ILLIAC IV was the earliest SIMD parallel computing effort. ILLIAC IV is "perhaps the most infamous of Supercomputers" - the project was built only one-fourth to completion, but took 11 years to build and cost almost four times its original estimated cost.

References

1. G. S. Almasi and A. Gottlieb. Highly Parallel Computing. Benjamin-Cummings publishers, Redwood city, CA, 1989
2. ^ Krste Asanovic et al. The Landscape of Parallel Computing Research: A View from Berkeley. University of California, Berkeley. Technical Report No. UCB/EECS-2006-183. December 18, 2006: "Old : Increasing clock frequency is the primary method of improving processor performance. New : Increasing parallelism is the primary method of improving processor performance... Even representatives from Intel, a company generally associated with the "higher clock-speed is better" position, warned that traditional approaches to maximizing performance through maximizing clock speed have been pushed to their limit."
3. Webopedia entry for "Distributed Computing"
4. David A. Patterson and John L. Hennessy. Computer Organization and Design (Second Edition) Morgan Kaufmann Publishers, 1998. ISBN 1558604286, pg 715
5. Asanovic et al: Old : Power is free, but transistors are expensive. New is power is expensive, but transistors are "free".
6. John L. Hennessy and David A. Patterson. Computer Architecture: A Quantitative Approach. 3rd edition, 2002. Morgan Kaufmann, ISBN 1558607242. Page 43.
7. J. M. Rabaey. Digital Integrated Circuits. Prentice Hall, 1996.
8. Laurie J. Flynn. Intel Halts Development of 2 New Microprocessors. New York Times, May 8, 2004
9. Leslie Lamport. "How to Make a Multiprocessor Computer That Correctly Executes Multiprocess Programs", IEEE Trans. Comput. C-28,9 (Sept. 1979), 690-691.
10. Flynn, Michael J. (September 1972). "Some Computer Organizations and Their Effectiveness" (PDF). IEEE Transactions on Computers. C-21 (9): 948–960. doi:10.1109/TC.1972.5009071.
11. Patterson and Hennessey, pg 748
12. David E. Culler, Jaswinder Pal Singh, Anoop Gupta. Parallel Computer Architecture - A Hardware/Software Approach. Morgan Kaufmann Publishers, 1999. ISBN 1558603433, pg 15
13. Yale Patt "The Microprocessor Ten Years From Now: What Are The Challenges, How Do We Meet Them?. Distinguished Lecturer talk at Carnegie Mellon University, April 2004
14. Culler et al, pg 15
15. Culler et al, pg 124
16. Culler et al, pg 125
17. Culler et al, pg 124
18. Culler et al, pg 125
19. Patterson and Hennessey, pg 713
20. Patterson and Hennessey, pg 713
21. Hennessey and Patterson, 549
22. Patterson and Hennessey, 714
23. Hennessey and Patterson, 549
24. Webopedia definition of "Cluster"
25. Pcmag.com definition of Beowulf
26. TOP500 Architecture share for 06/2007. Clusters make up 74.60% of the machines on the list
27. Hennessey and Patterson, 537
28. PC Magazine definition of MPP
29. ^ Michael R. D'Amour, CEO DRC Computer Corporation. "Standard Reconfigurable Computing" Invited speaker at the University of Delaware, February 28, 2007
30. Sha'Kia Boggan and Daniel M. Pressel. GPUs: An Emerging Platform for General-Purpose Computation. ARL-SR-154, U.S. Army Research Lab. August 2007
31. Oleg Maslennikov. Systematic Generation of Executing Programs for Processor Elements in Parallel ASIC or FPGA-Based Systems and Their Transformation into VHDL-Descriptions of Processor Element Control Units. Lecture Notes in Computer Science, 2004
32. Y. Shimokawa, Y. Fuwa, N. Aramaki. A parallel ASIC VLSI neurocomputer for a large number of neuronsand billion connections per second speed IEEE International Joint Conference on Neural Networks, 1991
33. K.P. Acken, M.J. Irwin, R.M. Owens. A Parallel ASIC Architecture for Efficient Fractal Image Coding Springer, 1998
34. Andrew B. Kahng. "Scoping the Problem of DFM in the Semiconductor Industry." University of California, San Diego. June 21, 2004: "Future design for manufacturing (DFM) technology must reduce design cost and directly address manufacturing – the cost of a mask set and probe card – which is well over $1 million at the 90 nm technology node and creates a significant damper on semiconductor-based innovation."
35. Thomas Pabst, Frank Völkel. Hot Spot: How Modern Processors Cope with Heat Emergencies. Tom's Hardware, September 17, 2001
36. L.F. Menabrea, Sketch of the Analytic Engine Invented by Charles Babbage. Bibliothèque Universelle de Genève, 1842
37. Patterson and Hennessey, pg 753
38. Patterson and Hennessey, pg 753
39. Patterson and Hennessey, pg 753
40. Patterson and Hennessey, pg 749
41. Patterson and Hennessey, pgs 749-750: Although successful in pushing several technologies useful in later projects, the ILLIAC IV failed as a computer. Costs escalated from the $8 million estimated in 1966 to $31 million by 1972, despite the construction of only a quarter of the planned machine... It was perhaps the most infamous of supercomputers. The project started in 1965 and ran its first real application in 1976

External links

• Template:Dmoz
• Lawrence Livermore National Laboratory: Introduction to Parallel Computing
• Designing and Building Parallel Programs, by Ian Foster
• Internet Parallel Computing Archive
• National HPCC Software Exchange
• Parallel processing topic area at IEEE Distributed Computing Online

• Distributed computing
• Parallel computing
• Concurrent computing