Introduction to Parallel Computing

This is the main page of a graduate-level course in parallel computing being taught in 2012/2 at the Polytechnic Institute IPRJ/UERJ. It is generally useful for programmers at the advanced level in the fields of scientific and multimedia programming.

General Info

• Meeting times: Tues 12:30pm-2pm, Thursdays 2:20pm - 4pm
• Evaluation criteria: 1 quizz at the end of the term (60%), plus practical projects (40%).
• Forum for file exchange and discussion: uerj.tk

Pre-requisites

• Linux - intermediate to advanced (will be reviewed as needed) - read Literatura recomendada pela equipe
• C/C++ - intermediate to advanced (will be reviewed) - read Literatura recomendada pela equipe

Approximate Content

The course focuses on software techniques for parallel computing. We are aiming at a comprehensive treatment on different types of practical parallel programming techniques

• process-oriented parallel programming
• thread programming/thread safety
• single-core vector instructions
• multi-processor and multi-core programming
• mapreduce/hadoop
• MPI
• Cuda

Each of the above techniques would merit a course of their own, as done in many of the best universities. Therefore we aim at attaining a practical familiarity with each, in the first half of the course, and we will specialize in the later part of the course as to help the students' graduate research.

Main Resources

• Textbooks
  • 1st part of the course: "Is Parallel Programming Hard, and, if so, what can you do about it?" - Paul E. McKenney / IBM (editor).
  • For MPI: "An Introduction to Parallel Programming" , by Peter Pacheco[1]
  • For Cuda: Programming Massively Parallel Processors: A Hands-On Approach[2]
  • General textbook: A. Grama, A. Gupta, G. Karypis, V. Kumar, Introduction to Parallel Computing, Second Edition, Addison-Wesley, 2003.
  • Algorithms-oriented textbook: Algorithms: sequential, parallel, and distributed, Kenneth A. Berman, Jerome L. Paul
    • For more theoretical info, see the chapter on parallel algorits in Cormen's classical algorithms book.
• Mapreduce, GFS, and Bigtable:
  • Papers from Google
  • http://code.google.com/edu/parallel/mapreduce-tutorial.html
  • Wikipedia
  • Hadoop documentation
• Presentations from IBM and Intel: Cilk, etc.
  • http://software.intel.com/en-us/videos/channel/parallel-programming
• Wikipedia pages
  • http://en.wikipedia.org/wiki/Vectorization_%28parallel_computing%29
  • http://en.wikipedia.org/wiki/Mapreduce
  • http://en.wikipedia.org/wiki/Hadoop
• Rice lecture notes on Parallel Computing [3]
• Other Resources
  • http://www-users.cs.umn.edu/~karypis/parbook/
  • http://www.cse.iitd.ernet.in/~subodh/courses/CSL860/

Lectures

Partial listing & Tentative Outline

1. Overview of parallel computing: https://computing.llnl.gov/tutorials/parallel_comp/
2. Review of Linux:
   1. See the book Running Linux http://wiki.nosdigitais.teia.org.br/Literatura_recomendada_pela_equipe
3. Review of C/C++
4. Fundamental programming techniques: processes and threads
   1. Read The Unix Programming Environment for some classic multi-process programming[4]
5. Mapreduce/Hadoop
6. MPI
7. Cuda

Homework

Homework 1

Our insitute is planning on buying a supercomputer. Therefore, our first homework will consist on studying these systems in detail.

• The class shall be divided into groups of 2 people.
• Each group must pick one top 500 ((http://www.top500.org/) supercomputer and write on this wiki:
  • Team number:
• Team 1: Julio Stutz & Joel Sanchez
  • Supercomputer Tupã (Inpe/Brazil)

• Team 2: Claudir Oliver
  • Supercomputer: Tianhe-1A (China)

• Each class will start with a 20min presentation from a random group (or a volunteer group).
• A handout must also be provided on Tue August 28
• Your project must not be a plain copy from wikipedia!

Contents of the presentation and report

• Overview of the system
• Glossary of terms
• How to program for the system
• Software infrastructure
• How does this differ from a conventional cluster.

Project 1

• The class will be divided into interest groups
• The interest groups will each pick one of the following technologies or else propose another one of their liking.
  • mapreduce/hadoop
  • MPI
  • Cuda
  • Hybrid technologies (Distributed + Cuda)
  • Languages designed for concurrency: Google Go, Erlang, etc.
  • Programming for the Playstation 3 parallel architecture[5]
• The project will consist in a series of presentations by the group members individually.
• Grade will be asigned to each presentation individually

Project 2

• This project will consist on a practical programming problem from the

student's research. The student is required to describe a problem from his research and present a parallel implementation of some aspect of it.

• This will be evaluated through actual code and acompanying monograph.