memP is a parallel heap profiling library based on the mpiP MPI profiling tool. The intent of memP is to identify the heap allocation that causes a task to reach its memory in use high water mark (HWM) for each task in a parallel job. Currently, memP requires that all tasks call MPI_Init and MPI_Finalize.

There are two types of memP reports:

  • Summary Report: Generated from within MPI_Finalize, this report describes the memory HWM of each task over the run of the application. This can be used to determine which task allocates the most memory and how this compares to the memory of other tasks.
  • Task Report: Based on specific criteria, a report can be generated that provides a snapshot of the heap memory currently in use, including the amount allocated at specific call sites.

A reasonable approach to using memP would involve 2 application runs: one run to determine the HWM and task of interest and another run using the memP run-time criteria to generate task-specific reports to get more detail about how memory is allocated.

memP works by intercepting memory allocation calls with memP wrappers that call internal library functions. Task data is collected by the library using MPI collective routines.

Platforms and Locations

Platform Location
x86_64 Linux /usr/local/tools/memp
BG/Q /usr/local/tools/memp

Quick Start

1. First, it is recommended that your application is complied with the -g flag so that memP can decode source code line numbers automatically.

2. Load the memp dotkit package. For example:

% use memp
Prepending: memp (ok)

3. Set selected runtime parameters using the MEMP environment variable. For example:

% setenv MEMP "-x -t"

These are discussed further in the Run-time Options section below.

4. Run your application using the srun-memp command. Note that this command is a wrapper script for the srun command which allows you to use memP without explicitly linking the memP libraries when you build your application. The syntax is the same as srun, as shown by the example below.

% srun-memp -n 4 -p pdebug ./myprogram

Note: the srun-memp command is brought into your path after you load the memP package (step 2 above). If you are running a batch job, your batch script will therefore need to include the package load in order to pick up the srun-memp command.

5. As an alternative to using the srun-memp command, you can explicitly link with the memP libraries, and then run your application as usual. See the Linking section below.

Output

After your application completes, memP will write its output file to the current directory. You will be notified of this by a message such as:

memP:
memP: Storing memP output in [./myjob.8.44380.1.memP].
memP:

With the naming format of [executable].[task count].[pid].[index].memP

Depending upon how you set the MEMP environment variable, the output file will be either plain text or XML. If it is XML, you can use a locally developed viewer utility to examine it. For example:

% use mpipview
Prepending: mpipview (ok)
% TGui myjob.8.44380.1.memP

A sample of the GUI with a simple program is shown below (click for larger image):

sample of memP GUI with a simple program
 

A sample of a plain text summary file is shown below:

@ memP
@ Command : ././array
@ Version                  : 1.1.5
@ MEMP Build date          : Jan  3 2013, 17:25:38
@ Start time               : 2015 02 24 12:19:21
@ Stop time                : 2015 02 24 12:19:23
@ MEMP env var             : [null]
@ Collector Rank           : 0
@ Collector PID            : 98078
@ Job Size                 : 16
@ Final Output Dir         : .
@ Report generation        : Single collector task
@ bss  section size        :     61.04 MB
@ text section size        :      4.75 KB
@ data section size        :     724.00 B

-------------------------------------------------------------------------------
@--- Greatest Heap High Water Mark (top 5, descending, bytes) -----------------
-------------------------------------------------------------------------------
      Rank      Heap HWM         Stack           Sum
         1      160.00 B      480.00 B      640.00 B
         4      160.00 B      480.00 B      640.00 B
         5      160.00 B      480.00 B      640.00 B
        10       17.00 B        0.00 B       17.00 B
        11       17.00 B        0.00 B       17.00 B

-------------------------------------------------------------------------------
@--- Heap HWM Statistics ------------------------------------------------------
-------------------------------------------------------------------------------
Max                               :     160.00 B
Median                            :      17.00 B
Mean                              :      43.00 B
Min                               :      12.00 B
Stddev                            :      56.00 B
Coefficient of variation          :     1.302191

Linking

As an alternative to using the srun-memp command, you can explicitly link your application with the required memP libraries, as shown below:

-L/usr/local/tools/memp/lib -lmemP -ldl

Example

1. Compile your application using the -g flag:

% mpiicc -g -o myapp myapp.c

2. Load the memP dotkit package:

% use memp
Prepending: memp (ok)

3. Set the MEMP environment variable as desired. In this case, tell memP to generate an XML output file that includes stack trace data:

% setenv MEMP "-x -t"

4. Run your application using the memP srun wrapper:

% srun-memp -n 8 -ppdebug ./myapp
memP:
memP: memP V1.1.5 (Build Jan  3 2013/17:25:38)
memP: Direct correspondence to chcham@llnl.gov
memP:
memP:
...
[program output]
...
memP:
memP: Storing memP output in [./myjob.8.44380.1.memP].
memP:

5. Examine the XML output file specified by memP after loading the viewer dotkit package:

% use mpipview
Prepending: mpipview (ok)
% TGui myjob.8.44380.1.memP

Run-time Options

memP behavior and output is determined by the setting of the MEMP environment variable. Options are shown in the table below.

Option Description Default
-a abort when threshold reached (see -h option).  
-d Summary Report: Print detailed call site list.  
-e path Specify the full path to the executable.  
-f dir Record output file in directory dir. Current directory
-g # Enable memP debug mode with number indicating debugging level #. Disables
-h # Task Report: HWM Threshold in #bytes.  
-i # Task Report: Maximum number # of reports to generate.  
-j # Summary & Task Report: Only report on the specified MPI_COMM_WORLD rank #.  
-k # Sets callsite stack traceback depth to #. 8
-n Do not truncate full pathname of filename in callsites.  
-o Disable profiling at initialization. Application must enable profiling with MPI_Pcontrol().  
-p # Sets the number # of HWm task entries to list in the report. If -1, list all. 1
-s # Set hash table size to # bytes. 256
-t Generate stack trace data. Disabled
-x Generate XML report.  

Examples:

$ export MEMP="-t -k 4 -x"
% setenv MPIP "-t -k 4 -x"

Troubleshooting

  • If you are attempting to use memP by linking it with your application and you are not seeing the memP banner from within MPI_Init at run time, you most likely have not linked your application in the appropriate manner. Make sure that if you are specifying the memP libraries on your link line.
  • If you are not getting the callsite source information in your memP report, you likely did not compile with -g.
  • Please contact the LC Hotline to request assistance with memP or to report a problem.

Documentation and References

  • Sourceforge memP Project page: mpip.sourceforge.net
  • Several documents included in the /doc installation directory, including the memP User Guide.