OxMPI

Table of contents

: 2.1 OxMPI Windows XP/Vista Installation using DeinoMPI; 2.2 OxMPI Windows XP/Vista Installation using MPICH2; 2.3 OxMPI Linux Installation using OpenMPI

OxMPI_Barrier	OxMPI_Bcast	OxMPI_Comm_size
OxMPI_Comm_rank	OxMPI_Finalize	OxMPI_Get_processor_name
OxMPI_Init	OxMPI_Iprobe	OxMPI_IsMaster
OxMPI_Probe	OxMPI_Recv	OxMPI_Reduce
OxMPI_Send	OxMPI_SetMaster	OxMPI_Wtime

Appendix A: Calling MPI from Ox

: A.1 Extending Ox; A.2 The MPI wrapper

Appendix B: Changes from previous versions

Acknowledgements

References

1. OxMPI Introduction

This is OxMPI version 2.00. OxMPI is an Ox package enabling the development of distributed Ox programs. The resulting Ox program can run on all four processors of a quad-core workstation (say), or on a cluster of machines.

Enabling the use of multiple processes requires a message-passing library, MPI in this case, and rewriting of the Ox code to make use of this. To make development much easier, OxMPI provides a plug-in replacement of the Ox Simulation class. At the core of the MPI enabled Simulation class is a Loop class which provides a solution for distributed loops and parallel use of random number generation. More information is provided in Doornik, Hendry, Shephard (2003, 2006).

Requirements:

A working MPI installation;
Ox Console 4 or 5, or Ox Professional 4 or 5

One of the objectives of OxMPI is to maintain replicability of simulation results when using the plug-in Simulation class:

Same results independent of the number of processes. The following should all give the same results:

    mpiexec -n 4 oxl -DOX_MPI -rp1 artest
    mpiexec -n 2 oxl -DOX_MPI -rp1 artest
    mpiexec -n 1 oxl -DOX_MPI -rp1 artest

Single process computation can also replicate the original Simulation class. This requires using Loop::UseIterSeed(FALSE).

Here are some timings using normtest2.ox with 10^6 replications. The hardware consists of two quad core computers (both with Intel Q6600), one running Windows XP and one running Windows Vista. The software is DeinoMPI and Ox 5.0 (using -rp1):

	2 quad core computers (both Intel Q6600)
	9 processes	5 processes	1 process
Time (secs)	10.1	16.8	58.8

The results with 1 and 5 processes are run on the local machine only. MPICH_USE_SMP_OPTIMIZATIONS=1 was used (without it the 9 process run took 15 seconds).

Here are some additional timings on Windows XP-x64 using 64-bit Ox 5.0 (using -rp1) and 64-bit DeinoMPI on a slower dual core machine:

	1 dual core computer (AMD X2 4200)
	9 processes	3 processes	2 processes	1 process
Time (secs)	50.5	49.1	93.8	84.8

Three processes is optimal for this setup, but the overhead of using 9 is very small. One process uses serial mode, while two processes uses one master and one slave, which is inefficient.

2. OxMPI Installation

Create an ox/packages/oxmpi folder and unzip the package file there. The default location of Ox 5 is C:\Program Files\OxMetrics5\ox under Windows, and /usr/local/lib/ox-5.00 under Linux/OS-X/Solaris. The relevant locations of Ox 4 are C:\Program Files\OxMetrics4\ox under Windows, and /usr/local/lib/ox-4.00 under Linux/OS-X/Solaris.

2.1 OxMPI Windows XP/Vista Installation using DeinoMPI

OxMPI for Windows has been tested with DeinoMPI 1.1.0 using 32-bit Windows XP and Vista, as well as 64-bit XP.

Follow the DeinoMPI installation instructions and run DeinoMPI\examples\cpi.exe either using the DeinoMPI GUI or the mpiexec command line program.
The following capture shows how ox\packages\artest.ox could be run (note that on this machine Ox is in D:\OxMetrics5\ox):
- -rp1 switches to single-threading, and is only relevant for Ox Professional 5.
- -DOX_MPIis used by the Ox code to indicate that it is an OxMPI application. This is required for the code to run correctly.
There seems to be no problem running MPICH2 and DeinoMPI side-by-side (but make sure that the correct mpiexec is used). I found connecting different machines easier with DeinoMPI (haven't managed to do that with MPICH2 for my machines yet).

2.2 OxMPI Windows XP/Vista Installation using MPICH2

OxMPI for Windows has been tested with MPICH2 1.0.6p1.

Install MPICH2, following the instructions on the MPICH2 web site (in particular the first part of the tips for installing and running MPICH2 on Windows).
Additional notes for Windows Vista:
- Under Vista, the Visual Studio C++ 2006 SP1 Redistributable package must be installed (I already had this under XP, but it may be required there as well). This can be found on the Microsoft web site.
- Unlike XP, the smpd service is not started automatically. Open a Command prompt with administrator privileges (in the start menu under All Programs/Accesories/Command Prompt, then right click and select Run as administrator). Then run
```
    smpd -install
```
  which responds `MPICH2 Process Manager, Argonne National Lab installed'. This also starts the service. To check, enter
```
    smpd -status
```
  Just press enter to the query `Please specify an authentication passphrase for smpd': by default there is no passphrase. This will respond: `smpd running on machine_name'.
  Subsequent running of mpiexec does not require administrator access.
As just done for MPICH2, add the path to the ox\bin folder. Now open a Console window (`DOS prompt') and type mpiexec which will show some help on MPICH2's mpiexec command. Type oxl, which lists the Ox command line syntax. Next, try:
```
    mpiexec -n 2 xyz\cpi.exe
```
where xyz is the full path to the MPICH2\examples folder. This should run cpi.exe using two processes (if your hardware supports it: two cores or two processors). Compare the result using -n 1 instead.
The following command runs artest.ox (assuming the current folder is ox/packages/oxmpi):
```
    mpiexec -n 4 oxl -DOX_MPI -rp1 artest
```
This requires that oxmpi_64.so is in ox/packages/oxmpi.
It may be more convenient to copy the ox\packages\oxmpi\batch\oxmpi.bat files to the ox\bin folder. The ox\packages\oxmpi folder contains the 32-bit DLL that provides the bridge between Ox and MPICH2. If you use another implementation of MPI, a recompilation of this DLL will be required.
Now run oxmpi, assuming you're in the ox\packages\oxmpi folder:
```
    oxmpi -n 2 cpi.ox
```
which should give the Ox output. By default, oxmpi.bat runs one process more than the number of processors. This is because in simulation experiments the master does not have much work to do. Next you can try to run artest.ox, a distributed simulation experiment.

2.3 OxMPI Linux Installation using OpenMPI

OxMPI for Linux has been tested with OpenMPI under Fedora 7 (64-bit).

OpenMPI 1.1 (with development files and libraries) was installed from the repository using Yum extender. The main documentation is the OpenMPI FAQ.
Use oxmpi_info to see the OpenMPI version and other information. Try:
```
    mpiexec -n 2 uptime
```
to see if OpenMPI works.
mpicc -showme displays the settings that mpicc uses.
The following command runs artest.ox (assuming the current directory is ox/packages/oxmpi):
```
    mpiexec -n 4 oxl64 -DOX_MPI artest
```
This requires that oxmpi_64.so is in ox/packages/oxmpi.
Running the 32-bit version requires compiling the 32-bit oxmpi.so from ox/packages/oxmpi/src/lnx_gcc_openmpi and copying it to ox/packages/oxmpi.

OxMPI Package Contents

artest.ox: Parallel equivalent of ox/samples/simula/artest.ox. This uses the OxMPI enabled version of the Simulation class. It can be run serially using oxl, or in parallel using MPI (then OX_MPI needs to be defined; see the oxmpi.bat example for windows).
cpi.ox: Ox equivalent of MPI sample program cpi.c
loop.ox, loop.h: The parallel loop class.
normtest*.ox

4. OxMPI Function Reference

OxMPI_Barrier

Return value: none.
Description: Initiates a barrier synchronization. This serves as a checkpoint in the code, synchronizing all running processes before continuing.
MPI version: int MPI_Barrier(MPI_Comm comm);

OxMPI_Bcast

OxMPI_Bcast(const aRootVal, const iRoot);

aRootVal: in: address of a variable (on root: holds the value to be broadcast, on others: will hold the broadcasted value).
iRoot: in: int, process that is the root for the broadcast

Return value

OxMPI_SUCCESS or OxMPI_ERR_ROOT (invalid root)

Description

Broadcasts data from the sending process iRoot to all other processes. So in all other processes than the sender, OxMPI_Bcast operates like the receiver. Ox types that can be broadcast are integer, double, matrix, string, and array. aVal must be a reference to a variable, which on the sender must hold a value; on all other processes is will receive the broadcasted value.

MPI version: int MPI_Bcast(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm)

OxMPI_Comm_rank

Return value: Returns the rank of the calling process.
MPI version: int MPI_Comm_rank(MPI_Comm comm, int *rank)

OxMPI_Comm_size

Return value: Returns the number of processes (the size of the group associated with MPI_COMM_WORLD).
MPI version: int MPI_Comm_size(MPI_Comm comm, int *size)

OxMPI_Finalize

Return value: none.
Description: Terminates the MPI session. This is an optional call, because OxMPI_Init automatically schedules a OxMPI_Barrier and OxMPI_Finalize to be executed when the Ox interpreter finishes.
MPI version: int MPI_Finalize(void)

OxMPI_Get_processor_name

Return value: Returns the name of the current processor (a string).
MPI version: int MPI_Get_processor_name(char *name, int *resultlen)

OxMPI_Init

Return value: OxMPI_SUCCESS or OxMPI_ERR_OTHER.
Description: Initializes MPI. Must be called at least once (unlike the original version, the Ox function can be called multiple times).
Under Linux, OxMPI_Init is called in the Ox driver replacement, and skipped in the wrapper (although the wrapper OxMPI_Init should still be called). The reason is that MPI does complex argument manipulations, which must be handled.
MPI version: int MPI_Init(int *argc, char ***argv)

OxMPI_Iprobe

OxMPI_Iprobe(const iSource, const iTag);

iSource: in: the message source, which can be OxMPI_ANY_SOURCE
iTag: in: int, a specific tag or OxMPI_ANY_TAG

Return value

Returns an empty matrix if no message is pending, otherwise returns a row vector of three numbers: message source, message tag, and message error status.

Description

Tests if a message is pending for reception. OxMPI_Probe is blocking, and waits until a message is available, while OxMPI_IProbe is non-blocking: it returns immediately, regardless of the presence of a message.

MPI version: int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag, MPI_Status *status)

OxMPI_IsMaster

Return value: TRUE if the current process is set to master with OxMPI_SetMaster, FALSE otherwise.
MPI version: none

OxMPI_Probe

OxMPI_Probe(const iSource, const iTag);

iSource: in: the message source, which can be OxMPI_ANY_SOURCE
iTag: in: int, a specific tag or OxMPI_ANY_TAG

Return value

A row vector of three numbers: message source, message tag, and message error status.

Description

MPI version: int MPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status)

OxMPI_Recv

OxMPI_Recv(const iSource, const iTag);

iSource: in: rank of the source process
iTag: in: tag for the message (>= 0; may be OxMPI_ANY_TAG)

Return value

The value (integer, double, matrix, string or array) that was sent using OxMPI_Send.

Description

Returns a value received from sender iSource with tag iTag. MPI_Recv will wait until the data has been received (so the process is blocked until then).

MPI version: int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status)

OxMPI_Reduce

OxMPI_Reduce(const slaveVal, const iOp, const iRoot);

slaveVal

in: int, double or matrix: value to reduce onto root

iOp

in: int, type of reduce operation, one of:

    OxMPI_MAX,    OxMPI_MIN,    OxMPI_SUM,    OxMPI_PROD,
    OxMPI_LAND,   OxMPI_BAND,   OxMPI_LOR,    OxMPI_BOR,
    OxMPI_LXOR,   OxMPI_BXOR,   OxMPI_MINLOC, OxMPI_MAXLOC

iRoot

in: int, process that is the destination for the reduction

Return value

The reduced value.

Description

This performs an action on all the data on the processes other than root, and returns the result to root.

MPI version: int MPI_Reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm)

OxMPI_Send

OxMPI_Send(const val, const iDest, const iTag);

val: in: value to send (which may be an integer, double, matrix, string or array)
iDest: in: int, rank of destination process
iTag: in: tag for the message (>= 0)

Return value: none

Description

Sends the value of val to process iDest, using send tag iTag. This operation is also blocking: the sending process waits for a matching receive (although the MPI standard allows for a return after buffering the sent data).

MPI version: int MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)

OxMPI_SetMaster

OxMPI_SetMaster(const isMaster); OxMPI_SetMaster(const isMaster, const fSlaveOutput);

isMaster: in: int, 1: set the current process to master, 0: set to not master, < 0: leave unchanged
fSlaveOutput: in: int, if FALSE, all standard output from the slaves is suppressed (by default, it is printed normally).

Return value: none.

Description

Can be used to let the wrapper track the master/slave status. Use OxMPI_IsMaster to get the master status.

MPI version: none

OxMPI_Wtime

Return value: Returns the time in seconds since a certain starting point
MPI version: double MPI_Wtime()

Appendix A: Calling MPI from Ox

A.1 Extending Ox

To make external C (or C++ or FORTRAN, etc.) functions callable from Ox code, it is necessary to write a small C wrapper around the external function. The task of this wrapper is to:

map arguments to the Ox function arguments;
do argument checking on the Ox arguments, so that they can be mapped to the external code (this will also do any required type conversion);
if necessary, create variables that allocate memory for the external results, or allocate temporary memory --- all memory for Ox variables must be allocated and freed by Ox;
set the return value.

Ox variables are represented by an OxVALUE, which holds all types, including:

OX_INT: -- four byte integer,
OX_DOUBLE: -- eight byte (double precision) double,
OX_MATRIX: -- rxc matrix of doubles (array of r pointers, each to c doubles),
OX_STRING: -- string (array of one byte characters; always terminated by '\0', but the length is tracked, so the first null character is occasionally not the length of the string),
OX_ARRAY: -- array of OxVALUEs.

Others are function, class object, etc. Access to the contents of an OxVALUE is through macros or C functions.

The anatomy of a wrapper function is:

void OXCALL FnFunction1(OxVALUE *rtn, OxVALUE *pv, int cArg)
{
    /* .... */
}

Here, rtn holds the return value, pv the cArg arguments (Ox supports variable argument lists).

One important issue to note is that local OxVALUE variables are not initialized. They must be first set to an integer, to avoid that subsequent use results in spurious deallocation of non-existent memory. On the other hand, the arguments of the wrapper function will be initialized appropriately.

The following example checks if the first two arguments are an integer (if not, a type conversion is done when possible, otherwise a run-time error results), and then assigns the sum to the return value:

void OXCALL FnEx1(OxVALUE *rtn, OxVALUE *pv, int cArg)
{
    OxLibCheckType(OX_INT, pv, 0, 1); /* check pv[0]..pv[1] */
    OxInt(rtn, 0) = OxInt(pv, 0) + OxInt(pv, 1);
}

There is no need to check the argument count, provided the function is properly declared in a Ox header file:

    extern "dll_name,FnEx1"       SumInt2(const i1, const i2);

The Ox code could now use, e.g.:

    x = SumInt2(10, 100);

So the new function is used as any other Ox function.

A.2: The MPI wrapper

We now give some examples of how the C wrapper that provides MPI calls to Ox is written, considering MPI_Init, as well as the implementation of send/receive. The wrapper is compiled into a dynamic link library (oxmpi.dll under windows, oxmpi.so under Linux), which can be called from Ox.

The oxmpi.h header file provides the glue, and must be included in Ox programs whenever MPI is used. It contains for example:

extern "oxmpi,FnMPI_Init" MPI_Init();
extern "oxmpi,FnMPI_Send" MPI_Send(const val, const iDest, const iTag);
extern "oxmpi,FnMPI_Recv" MPI_Recv(const iSource, const iTag);

defining the syntax for calls from Ox. MPI_Init resides in oxmpi (.dll or .so), and is called FnMPI_Init in the dynamic link library.

void OXCALL FnMPI_Init(OxVALUE *rtn, OxVALUE *pv, int cArg)
{
    int iret;
    
    if (rtn)                                             /* rtn may be NULL */
        OxInt(rtn,0) = OxMPI_SUCCESS;
        
    if (s_bMPI_Initialized)              /* if already done: don't do again */
        return;

#ifdef SKIP_MPI_Init
    /* assume already initialized in main */
    s_bMPI_Initialized = TRUE;
    OxRunMainExitCall(mpi_exit);     /* call MPI_Finalize at end of Ox main */
    return;
#endif

#ifdef OXMPI_NO_ARGV
    iret = MPI_Init(NULL, NULL);       /* MPICH2: don't pass Ox args to MPI */
#else
    {   int argc;  char **argv;
    
        OxGetMainArgs(&argc, &argv);  /* points to arguments for Ox program */
    
        iret = MPI_Init(&argc, &argv);       /* MPI may have prepended args */
        
        OxSetMainArgs(argc, argv);            /* and then remove them again */
    }
#endif
    
    iret = MPI2Ox_Error(iret);  /* translate error code to those used by Ox */
    if (rtn)
        OxInt(rtn,0) = iret;                         /* set the return code */

    if (iret == OxMPI_SUCCESS)
    {
        s_bMPI_Initialized = TRUE;
        OxRunMainExitCall(mpi_exit); /* call MPI_Finalize at end of Ox main */
    }
}

`FnMPI_Init`

The contents of FnMPI_Init is given in the listing above. Unusually, it makes allowance to be called with NULL for the rtn argument -- this is not necessary if the function will only be called from Ox. Next, it checks if it hasn't already been called.

MPI 2 states that MPI_Init should be callable with NULL for both arguments, in case it is called from a dynamic-link library. This is the case with OxMPI, so -DOXMPI_NO_ARGV should be used when compiling OxMPI.

In some older MPI implementations, command line arguments were used to pass MPI arguments accross to different processes. In that case SKIP_MPI_Init can be used to skip the call to MPI_Init, which now must be placed in the main function so that the arguments can be manipulated. Then, a new oxl driver should be compiled, as in oxl_mpi.c.

void OXCALL FnMPI_Send(OxVALUE *rtn, OxVALUE *pv, int cArg)
{
    int dest, tag, aisend[3], len;
    
    OxLibCheckType(OX_INT, pv, 1, 2);
    dest = OxInt(pv,1);
    tag = OxInt(pv,2);

    aisend[0] = GETPVTYPE(pv);
    switch (GETPVTYPE(pv))
    {
        case OX_INT:
            aisend[1] = OxInt(pv,0);
            MPI_Send(aisend, 3, MPI_INT, dest, tag, s_iMPI_comm);
            return;     /* finished */
        case OX_DOUBLE:
            len = 1;
            break;
        case OX_MATRIX:
            aisend[1] = OxMatr(pv,0);
            aisend[2] = OxMatc(pv,0);
            len = aisend[1] * aisend[2];
            break;
        case OX_STRING:
            len = aisend[1] = OxStrLen(pv,0);
            break;
        case OX_ARRAY:
            len = aisend[1] = OxArraySize(pv);
            break;
        default:
            return;
    }
    MPI_Send(aisend, 3, MPI_INT, dest, tag, s_iMPI_comm);

    if (len)
    {
        switch (GETPVTYPE(pv))
        {
            case OX_DOUBLE:
                MPI_Send(&(OxDbl(pv,0)), 1, MPI_DOUBLE, dest, tag, s_iMPI_comm);
                break;
            case OX_MATRIX:
                MPI_Send(OxMat(pv,0)[0], len, MPI_DOUBLE, dest, tag, s_iMPI_comm);
                break;
            case OX_STRING:
                MPI_Send(OxStr(pv,0), len, MPI_CHAR, dest, tag, s_iMPI_comm);
                break;
            case OX_ARRAY:
            {
                int i;  OxVALUE pvarg[3];
                pvarg[1] = pv[1];                                   /* dest */
                pvarg[2] = pv[2];                                    /* tag */
                for (i = 0; i < len; ++i)
                {
                    pvarg[0] = OxArrayData(pv)[i];           /* array entry */ 
                    FnMPI_Send(rtn, pvarg, 3);
                }
                break;
            }
        }
    }
}

`FnMPI_Send`

Next, we consider sending and receiving data. Ox variables can have different types. Extra information is transmitted in an array of three integers: the type and, if necessary, the dimensions. If an integer is sent, the second value is the content, and only one send suffices. Otherwise the first send is followed by the actual data. An OX_ARRAY is a compound type, which can be sent recursively.

Every MPI_Send must be matched by an MPI_Recv. First the array of three integers is received. For types other than integer, this allows the correct variable to be created (i.e. memory allocated). The second receive than copies the actual data into the new variable, which is returned to the Ox code.

Appendix B: Changes from version 1

OxMPI 2 is independent of the particular MPI implementation (unlike OxMPI 1, which only worked for MPICH 1).
The MPI_* constants to be used in the Ox code are now OxMPI_*.
Update to use the new Ox 4 random number generators. The slaves use MWC_52, while the master uses PM to generate sees for the slaves.
Removed Simulation::UseRanSeed() -- use Loop:UseIterSeed() instead.

Acknowledgements

Thanks to Jan Meyer and Charles Bos on improving the portability of OxMPI and further testing.

References

Doornik, J.A., Hendry, D.F. and Shephard, N. (2006) Parallel Computation in Econometrics: A Simplified Approach. Chapter 15 in Handbook of Parallel Computing and Statistics, Chapman & Hall/CRC, 449-476.

Doornik, J.A., Hendry, D.F. and Shephard, N. (2002) Computationally-intensive econometrics using a distributed matrix-programming language, Philosophical Transactions of the Royal Society of London, Series A, 360, 1245-1266.
An earlier version, presented at the Royal Society, is still available as 2001-W22 in Nuffield Economics Working Papers (PDF).
Gropp, W. and E. Lusk (2001) User's guide for mpich, a portable implementation of {MPI} version 1.2.2. mimeo, Argonne National Laboratory, University of Chicago. http://www-unix.mcs.anl.gov/mpi/mpich/.