[Wien] Highest performance within Opteron

[Tobias Burnus]

Hello,

Saeid Jalali wrote:
> 64 homogenous nodes of AMD-Opteron

I assume that you are running under Linux (x86_64).

> Since there are several versions of MPI, e.g. mpich, lanmpi, …, we are
> not sure which one has the highest performance. This is the case also
> for the FORTRAN compilers.

One (but slightly outdated) Fortran compiler comparison is at
http://www.cse.clrc.ac.uk/disco/Benchmarks/Opteron_compilers.pdf,
another at http://www.polyhedron.com/ (click on "Fortran compiler
comparison" on the left); the latter a rather up to date comparison of
several compilers.

However, it really depends a lot on both the compiler version and on the
program as the following two example illustrate (both not with Wien2k):
- I know one program which was fastest with NAG f95 though NAG f95 is
for most programs rather slow
- sunf95 took twice as long (with one program) as other compilers
because of a single compiler optimization bug (now fixed)

http://www.wien2k.at/reg_user/benchmark/ may give some hints for Wien2k,
though it lacks the compiler flags and compiler version.

The best would be probably to run your own tests. Most vendors of
commercial compilers offer trail versions.

You should also check whether your Opterons already support SSE3* or
not, then you can also use that optimization flag when compiling.
Actually, it is worthwhile to have a close look at the man pages to find
all the useful optimization flags. Some compilers have even a guide to
find the best options.


The following Fortran compilers are (more or less) free:

- gfortran, moderately fast; http://gcc.gnu.org/wiki/GFortran (GPL)
(Please don't use version 4.0, that is really broken; 4.1.x and the
being developed 4.2 are ok)

- g95, slowest in polyhedron tests; http://www.g95.org/ (GPL)

- Sun Compiler, fast compiler, free after registration,
http://developers.sun.com/prodtech/cc/linux_index.html

- Intel Compiler, fast compiler, free for non commercial, private use.
Compiler page:
http://www.intel.com/cd/software/products/asmo-na/eng/compilers/284132.htm
Non-commercial download page:
http://www.intel.com/cd/software/products/asmo-na/eng/compilers/219771.htm
NOTE HOWEVER: This is most probably not the case for you as this does
NOT include paid research. See
http://www.intel.com/cd/software/products/asmo-na/eng/compilers/219692.htm#5
That Intel is free for academic research is a misconception, stemming
from the old days when it was rather hidden in the licence.

Non free:
- PGI, http://www.pgroup.com/
- Pathscale, http://www.pathscale.com/
- Absoft, www.absoft.com
- NAG, www.nag.co.uk (This compiler is very good to find bugs in the
source code, but it is rather slow; it is also the most standard conform
compiler)

(I use here mostly the Intel compiler, NAG to do bug checking and
gfortran as free compiler; I have also the sun compiler and g95 and our
universities computation centre has PGI/Pathscale on AMD and Intel on ia64.)

In addition, you may want to try an optimized BLAS/LAPACK library:
- For the Intel compiler: MKL, free for non-commerical use (see above,
incl. link)
- AMD Core Math Library, free, http://developer.amd.com/acml.aspx for
gfortran, NAG, Pathscale and PGI
- Atlas - optimizes for the computer/compiler and takes thus some time
to install, http://math-atlas.sourceforge.net/
- gotoblas: http://www.tacc.utexas.edu/resources/software/, not tried,
claims to be fast than Netlib's version

For Wien2k I don't know, but for another project I compared Intel + MKL,
gfortran with ACML and gfortran with the standard netlib.org BLAS/LAPACK
routines. The speed difference was only about 0.3%. Allegedly, the
difference is rather big for Wien2k.

For MPI, I think there are basically three choices:
- OpenMPI (joint effort of FT-MPI, LA-MPI, LAM/MPI and PACX-MPI),
supporting the MPI 2 standard
- MPICH, coming in two flavours:
* MPICH1 (MPI 1, http://www.mcs.anl.gov/mpi/mpich1) and
* MPICH2 (MPI 2 standard, http://www-unix.mcs.anl.gov/mpi/mpich/)
[Well, there are some more, esp. commerical ones.]

My impression (I did not investigate thouroughly) is that it does not
matter that much, which MPI implementation one uses. See also:
http://www.beowulf.org/archive/2005-December/014416.html

Regards,

Tobias

* SSE3: Under Linux, look for "pni" in the flag list of /proc/cpuinfo
[it was called pni before it was renamed to sse3].