• NAME
• SYNOPSIS
• LICENSE
• INSTALLATION
• Platforms
• Required Software
• Unpacking
• Makefile variables
• Configuring
• Compiling
• DOCUMENTATION
• TESTS AND EXAMPLES
• WRITING A NEW APPLICATION MODULE
• AUTHORS

NAME

PETSc-FEM - a general purpose, parallel, multi-physics FEM program

SYNOPSIS

This is PETSc-FEM, a general purpose, parallel, multi-physics FEM (Finite Element Method) program for CFD (Computational Fluid Dynamics) applications based on PETSc . PETSc-FEM comprises both a library that allows the user to develop FEM (or FEM-like, i.e. non-structured mesh oriented) programs, and a suite of application programs. It is written in the C++ language with an OOP (Object Oriented Programming) philosophy, keeping in mind the scope of efficiency. PETSc-FEM may run in parallel using the MPI standard on a variety of architectures, but currently it has been tested in Beowulf clusters only.

PETSc-FEM is GPL, for further details read the file COPYING in the root directory of the distribution.

LICENSE

This file belongs to the PETSc - FEM package a library and application suite oriented to the Finite Element Method based on PETSc. Copyright (C) 1999-2008, Mario Alberto Storti, Norberto Marcelo Nigro, Rodrigo Rafael Paz, Lisandro Dalcin and Ezequiel Lopez, Centro Internacional de Metodos Numericos en Ingenieria (CIMEC-Argentina), Universidad Nacional del Litoral (UNL-Argentina), Consejo Nacional de Investigaciones Cientificas y Tecnicas (CONICET-Argentina).

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

INSTALLATION

Platforms

Currently, PETSc-FEM is known to run only on Beowulf class clusters based on Intel processors, running the GNU/Linux OS. Off course, you can also install PETSc-FEM in a single GNU/Linux box, for use in sequential (i.e. non-parallel) mode or either for simulating parallel runs, by launching several processes in the same processor. (This is a feature of MPI). PETSc-FEM compiles cleanly with the GNU compiler GCC 4.1.1 that comes with the Fedora Core 5 GNU/Linux distribution. http://fedora.redhat.com/

I guess it should be relatively easy to install in other processor architectures (SPARC or Alpha) running GNU/Linux and also on other Unix flavor's.

Required Software

Before installing PETSc-FEM, please verify you have the following packages correctly installed in your system.

MPI

Message Passing Interface. We are currently using MPICH2 1.0.3 (http://www.mcs.anl.gov/mpich) but I think it should be possible to use other versions of MPI like LAM or OpenMPI.

PETSC

Portable, Extensible Toolkit for Scientific Computation and can be found at http://www.mcs.anl.gov/petsc/. Current version of PETSc-FEM compiles with PETSc 2.1.6.

LAPACK

This is, in fact, a prerequisite to install PETSc and can be found at http://www.netlib.org/lapack/.

Libretto

This is a C library of generic containers and can be found at http://pobox.com/~aaronc/tech/libretto/. I have learned recently that development of Libretto has been stopped since it has been superseded by GLIB, so that I think that in a future I will replace the Libretto code by GLIB. Also, the STL library (that comes with gcc) can replace all or some of the features provided by Libretto but I found that the STL implementation (at least that one that comes with gcc) is rather inefficient.

GLIB

GLIB is a library which includes support routines for C such as lists, trees, hashes, memory allocation, and many other things. The official web site is http://www.gtk.org/. [RedHat RPM packages: GLIB, glib-devel]

Newmat

This is a matrix library to be used mainly at the element routine level, i.e. for small matrices and can be found at http://webnz.com/robert/. It is not mandatory to use Newmat for all element calculations. In fact there is included a matrix class called FastMat2 that is more performing, but Newmat is still used in some internal parts and has much more functions, so that you have to install it. Also, you may prefer Newmat over FastMat2 for element calculations since it supports overloading of algebraic operations. See the documentation on FastMat2 in the PETSc-FEM documentation for reasons why to use FastMat2 instead of Newmat.

METIS

METIS is a software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. It can be found at http://www.cs.umn.edu/~metis.

Meschach

The Meschach Library is a numerical library of C routines for performing calculations on matrices and vectors. It is used only for some functions that can not be done with Newmat. Currently, it is used in the advective module for performing eigenvalue decomposition of non-symmetric matrices. I think that in a future this functionality could be replaced by calls to LAPACK.

Perl

If you want to use ePerl then you need to have Perl installed. [RedHat RPM package: perl].

SuperLU (optional)

A collection of ANSI C routines to solve sparse linear systems of equations. Can be found at http://www.nersc.gov/~xiaoye/SuperLU/. This package is optional, install it if you think that SuperLU is faster than PETSc for direct factorization of sparse matrices.

ANN (optional)

This is the Approximate Nearest Neighbor Library and can be found at http://www.cs.umd.edu/~mount/ANN. Currently it is only used in the Navier-Stokes module for calculating the damping functions in the Smagorinsky model. However, in a future it may be used more widely for some mesh manipulations. Currently it is optional.

pthreads library (optional)

This is a POSIX compliant library for threads, currently used for asynchronous communication with the IBM Data Explorer visualization tool, but it might be used for other things in a near future. It comes with gcc, so that if you use gcc, you probably have it already installed.

Simple Sockets Library (SSL) (optional)

The Simple Sockets Library (SSL) allows C programmers to use interprocess communications via Berkeley sockets simply and quickly. Most of the SSL's functions resemble C's file i/o functions, so C programmers will find the SSL easy to learn. Currently used for communication with the IBM Data Explorer visualization tool, but but it might be used for other things in a near future. [Current version: 2.11a Tue Mar 6 09:46:16 EST 2001] [Note: If you do a web search you can find it as a file COSMIC.tar.gz or ssl.tar.gz].

Unpacking

Once you got the petscfem-xxx.tar.gz package (xxx stands for the version), gunzip and untar it like this:

  $ gunzip petscfem-xxx.tar.gz
  $ tar xvf petscfem-xxx.tar

This will create a petscfem-xxx directory with all the source files included.

Makefile variables

$(PETSCFEM_DIR)

The root of the PETSc-FEM package

$(PETSC_ARCH), $(PETSC_DIR)

These are variables relative to PETSc. See documentation.

Configuring

Once you have all the software mentioned in Required Software installed, you have to define the variables in the CONFIGURATION VARIABLES section in file $(PETSCFEM_DIR)/Makefile.base.

First you have to set the PETSC_BOPT variable which is the default value for the PETSc BOPT variable, which sets the optimization level to be used in compilation. You can set it to

PETSC_BOPT = g_c++


Compiles for debugging.

PETSC_BOPT = O_c++


Compiles with optimization enabled.

The optimization level is inherited by the PETSc libraries so that if you compile with PETSC_BOPT=O_c++ then not only PETSc-FEM will be compiled with optimization, but also the PETSc optimized libraries will be used. The specific optimization options are set in $(PETSC_DIR)/bmake/$(PETSC_ARCH)/variables (OCXX_COPTFLAGS variable). According to our experience maximum optimization is achieved with <BOPT=-O2 -funroll-loops>.

The default value can be superseded by, for instance, issuing

  $ make BOPT=O_c++ ns

This compiles the Navier-Stokes module with optimization enabled independently of the PETSC_BOPT default value.

The PROF_FLAGS variables allows you to turn on profiling. By default, profiling is deactivated.
Other configuration variables are defined in a file Makefile.defs, but normally they are set only when you first install PETSc-FEM, so the best is to put Makefile.defs in the directory immediately above PETSc-FEM directory (i.e. $(PETSCFEM_DIR)/..). So that, if you install another version of PETSc-FEM, the settings will be seen immediately by the new version. A version of Makefile.defs is included in the PETSc-FEM directory. Copy it to the directory _above_ the PETSC-FEM directory and configure the variables inside. If you install another version of PETSc-FEM then probably you have not to configure again these variables, unless a new package has been added or you have moved it.

Compiling

cd to directory $(PETSCFEM_DIR) and do

  $ make depend
  $ make libpetscfem        # to build the library
  $ make all                # to build the library and all the modules

also

  $ make what

can help you on other useful targets.

If nothing goes wrong you will end with a src/libpetscfem.a library and application binaries (ending in .bin) in their respective directories, e.g. applications/ns/ns.bin is the Navier-Stokes binary.

DOCUMENTATION

Documentation for PETSc-FEM is included in the $(PETSCFEM_DIR)/doc directory. There is basically a reference manual written in LaTeX (doc/petscfem.tex), and embedded documentation for the routines to be used with Doc++. In the standard distrib you should find both of them converted to HTML (LaTeX is converted with latex2html) and accessible from doc/index.html. Other formats (perhaps PostScript and PDF) may be also found in the doc directory.

TESTS AND EXAMPLES

Many tests and examples are in the $(PETSCFEM_DIR)/test directory.

WRITING A NEW APPLICATION MODULE

The best way to start coding an application program is to duplicate a directory (for instance applications/advdif) or (for instance applications/ns) to a directory from your own (i.e. outside the $(PETSCFEM_DIR) tree, say /your/appl/path. Configure the $(PETSCFEM_DIR) variable in the local makefile (i.e. /your/appl/path/Makefile).You should have to configure other variables as well, perhaps $(MYOBJS) (this is a list of the object (.o) files in this local directory) and $(PROG) (the name of the program to be built, without the suffix .bin).

AUTHORS

Mario A. Storti* <mario.storti@gmail.com>

PETSc-FEM kernel, NS and AdvDif modules.

Norberto M. Nigro* <nnigro@intec.unl.edu.ar"

NS and AdvDif modules, multi-phase flow.

Rodrigo R. Paz* <rodrigop@intec.unl.edu.ar>

AdvDif module, hydrology module, compressible flow, fluid-structure interaction, preconditioners.

Lisandro Dalcin* <dalcinl@intec.unl.edu.ar>

PETSc-FEM kernel, Python extension language project, linear algebra, preconditioners, multigrid.

Ezequiel Lopez* <ezequiellopez@hotmail.com>

Mesh relocation algorithms.

Laura Battaglia* <lbattaglia@santafe-conicet.gov.ar>

Free surface algorithms.

Gustavo Rios Rodriguez * <gusadrr@yahoo.com.ar>

Adaptive refinement.

* CIMEC, Internacional Center for Computational Methods in Engineering, Santa Fe, Argentina. http://www.cimec.org.ar/petscfem|http://www.cimec.org.ar/petscfem Ordinary mail: Mario Storti, CIMEC-INTEC, Guemes 3450, 3000 Santa Fe, Argentina.