README.bigsim

edit 1

=======================================================================

HOW TO INSTALL AND RUN EMULATION/SIMULATION WITH BIGSIM
=======================================================

Updated: February 2009 - Send questions to charm@cs.illinois.edu

These are a few notes on how to get going with BigSim for emulation and
simulation of MPI application codes. Part (A) covers BigSim installation, while
part (B) covers usage of the software installed in part (A), for emulation and
simulation.


A) INSTALLING BIGSIM:
====================

Skip ahead to section C for quick start instructions.

The BigSim package depends on Charm++. So, assuming that the application can be
an MPI or Charm++ code, one needs to build Charm++ with support for AMPI. We
need to compile two different versions of Charm, one for emulating the user
application and the other for using the traces for the simulation.

A special Charm++ build is necessary for emulation with BigSim.  This new build
needs to use the BigSim machine-layer, because the emulator needs to run on top
of that. On a 64-bit Linux machine, this new build can be produced by

   ./build bgampi netlrts-linux-x86_64 bigemulator -O2

The first argument ("bgampi") is a target meaning that bigsim, charm++ and LIBS
will be compiled. The "bigemulator" argument implies that the BigSim machine
layer will be used for running the programs. This should produce a
sub-directory such as "netlrts-linux-x86_64-bigemulator", with a valid Charm++
instance.

To compile the simulator which is called BigSimulator (or BigNetSim), we need
the regular Charm++ build (netlrts-linux-x86_64 in our example).  It needs to be
complemented with a few more libaries from BigSim and with the Pose
discrete-event simulator. These pieces can be built, respectively, with:

   ./build bgampi netlrts-linux-x86_64 -O2
   ./build pose netlrts-linux-x86_64 -O2

Access to the discrete-event simulation is realized via a Charm++ package
originally named BigNetSim (now called BigSimulator). Assuming that the
'subversion' (svn) package is available, this package can be obtained from the
Web with a subversion checkout such as

   svn co https://charm.cs.illinois.edu/svn/repos/BigNetSim/

In the subdir 'trunk/' created by the checkout, the file Makefile.common must
be edited so that 'CHARMBASE' points to the regular Charm++ installation.
Having that done, one chooses a topology in that subdir (e.g. BlueGene for a
torus topology) by doing a "cd" into the corresponding directory (e.g. 'cd
BlueGene').  Inside that directory, one should simply "make". This will produce
the binary "../tmp/bigsimulator". That file, together with file
"BlueGene/netconfig.vc", will be used during a simulation. It may be useful to
set the variable SEQUENTIAL to 1 in Makefile.common to build a sequential
(non-parallel) version of bigsimulator.


B) USING BIGSIM:
===============

The recommended usage of BigSim with MPI codes may consist of three steps:
verifying that the code works under AMPI, running an emulation of the code, and
running an actual simulation. As an option, a fourth step consists in
analyzing performance of the simulated code in a post-mortem fashion with the
Projections tool. These various steps are described below.


B-1) Verifying that the code works with AMPI:
--------------------------------------------

First, the MPI code must be compiled by charmc or one of the AMPI scripts,
using the flag "-swapglobals" (assuming a ELF-compatible system), like

   mpicc -o prog prog.c -swapglobals

To run the code under AMPI, a test-run should have VP>P, to ensure that the
results are still correct even when there is more than one VP per physical
processor:

   charmrun +p4 prog +vp8

Notice that both 'mpicc' and 'charmrun' must be taken from the regular Charm++
build in step (A), i.e. 'netlrts-linux-x86_64' in our example.


B-2) Running a BigSim emulation:
--------------------------------

Here, one must point his/her path to the Charm++ instance produced for
emulation; in our example of step (A), this would be under subdir
"netlrts-linux-x86_64-bigemulator". Next, one must compile again the MPI code,
preparing it for the emulation:

   mpicc -o prog_emul prog.c -swapglobals

(it should be noticed that this time a proper 'mpicc' will be invoked, assuming
that the path is set correctly)

Now, all that is needed is to run this new executable under Charm++, for the
emulation:

   charmrun +p4 prog_emul +vp16 +x16 +y1 +z1 +cth1 +wth1 +bglog

Argument "+vp16" indicates the number of processors of the hypothetical
(future) system. Arguments "x/y/z" indicate the dimensions of the future
machine, while "cth/wth" indicate, respectively, the number of compute and I/O
processors in each node of that future machine.

Argument "+bglog" must be used so that BigSim logfiles get created. These files
are named 'bgTrace', bgTrace0, 'bgTrace1', ... 'bgTrace{Q-1}' where "Q" is the
number of processors running the emulation.

The output produced in this execution shows an emulation of the MPI code, in
this case assuming a future machine with 16 processors.


B-3) Running a BigSim simulation:
--------------------------------

a) All network models

To run a simulation, one needs the binary "bigsimulator" and the file
"netconfig" produced in step (A). Those two must be placed in the same
directory as the BigSim tracefiles (i.e.  'bgTrace*' files). File "netconfig"
may have to be created (by copying it from file BlueGene/netconfig.vc) and
edited, to at least match the geometry of nodes assumed for the future machine.
Actual simulation execution is started by:

   ./charmrun +p2 ./bigsimulator 0 0

which will run the simulation assuming a "latency-only" mode. The simulation
can be run on any number of processors, regardless of the number of processors
used in the emulation or in the future target machine.

To run the simulation using BlueGene's actual network, the command should be

   ./charmrun +p2 ./bigsimulator 1 0

Either of these commands will print, in stdout, information about the predicted
execution time (Global Virtual Time, or GVT). Notice that the number of
processors used to run the simulation can be chosen independently of the number
of processors used in the emulation or in the future machine.

To analyze how changes in the network characteritics may affect performance,
one may edit file 'netconfig' and repeat the simulation. In particular, to
change the topology of the network for a topology different than the one
originally assumed in the emulation, one should have a line like the following
in file 'netconfig': OVERRIDE_TRACE_TOPOLOGY 1

b) Simple latency model

To use the simple latency model, follow the setup procedure in part (a) of this
section, noting that the files are located in the trunk/SimpleLatency
directory.  This will produce the "bigsimulator" file.

The command line parameters used for this model are different.  The format is
as follows:

   [charmrun +p#] bigsimulator -lat <latency> -bw <bandwidth>
                  [-cpp <cost per packet> -psize <packet size>]
                  [-winsize <window size>] [-skip] [-print_params]

Latency (lat)         - type double; in microseconds
Bandwidth (bw)        - type double; in GB/s
Cost per packet (cpp) - type double; in microseconds
Packet size (psize)   - type int; in bytes
Window size (winsize) - type int; in log entries

The implemented equation is: lat + (N/bw) + cpp*(N/psize)

Latency and bandwidth are required.  If cost per packet is given, then packet
size must be given, as well.  Otherwise, cost per packet defaults to 0.0.
Packet size, if given, must be a positive integer.

The -winsize flag allows the user to specify the size of the window (number of
log entries) used when reading in the bgTrace log files.  This is useful if the
log files are large.  If -winsize is not specified, the value defaults to 0,
which indicates that no windowing will be used (i.e., there will be one window
for each time line that is equal to the size of the time line).

As with the second parameter in the examples of part (a) of this section, the
-skip flag indicates that the simulation should skip forward to the time stamp
set during trace creation (see the BigSim tutorial talk from the 2008 Charm++
workshop).  If -skip is not included, then no skipping will occur.

The -print_params flag is provided for debugging convenience.  When present,
the simple latency model parameters will be displayed during simulation
initilization.


B-4) Generating performance data for Projections:
------------------------------------------------

To generate Projections-compatible traces that can be visualized in
Projections, two steps should be changed in the described procedure, as
follows. In the emulation phase (B-2), one should create the application adding
'-tracemore projections' to the build line, such as

   mpicc -o prog_emul prog.c -swapglobals -tracemode projections

With this, during the emulation, Projection-files with extensions '.sts' and
'.log' will be created, such as 'prog_emul.*.log'.  All of these files *must*
be copied into the same directory where the simulation will be run (same
directory containing files 'bigsimulator' and 'netconfig').

Then, when running the simulation (B-3), flag '-projname' should be added, such
as

   charmrun +p2 bigsimulator 0 0 -projname prog_emul

This form of simulation will generate new files with extensions ".sts" and
".log" (such as 'prog_emul-bg.*.log') which can be visualized in Projections. 


C) QUICK START:
===============

This is a step-by-step quick tutorial for simple usage of BigSim simulation framework and visualizing its Projections output logs.
For more information, please refer to the BigSim and Projections manuals.

1- build charm (and AMPI) with bigemulator and bigsim (replace "linux" with "darwin" for mac):
	cd charm
	./build charm++ netlrts-linux-x86_64 bigemulator bigsim
	./build AMPI netlrts-linux-x86_64 bigemulator bigsim

2- compile your code using charm or AMPI compilers located in "netlrts-linux-x86_64-bigemulator-bigsim/bin", for example:
	cd tests/ampi/jacobi3d; make 

3- run your application emulating the target machine, for example:
	./charmrun +p1 jacobi 4 4 2 5 +vp32 +x32 +y1 +z1 +cth1 +wth1 +bglog

4- download BigSim's simulator
	git clone https://charm.cs.illinois.edu/gerrit/BigFastSim

5- build BigFastSim:
	cd BigFastSim/Release
	vim makefile  #change CHARMPATH=$(HOME)/charm/netlrts-linux-x86_64-bigemulator-bigsim/
	make

6- copy simulator to trace files' directory:
	cd ~/charm/tests/ampi/jacobi3d
	cp ~/BigFastSim/Release/seqSimulator .

7- run the simulator with projections output: (to see other options such as changing latency and bandwidth run "./seqSimulator -help")
	./seqSimulator -tproj

8- download and make Projections:
	git clone https://github.com/UIUC-PPL/projections
	cd projections
	ant

9- run Projections:
	./bin/projections #open tproj.sts file

After opening the symbol file (file/open <tproj.sts>), you can use different features of Projections such as tools/Timelines.