Libdist (libdist)

Usage

Libdist offers functionalities to automate the build process of the trace suite. It reads in settings stored in a xml file and creates the appropiate trace suite in libdist version. Possible options to adapt are the environment variables (using modulefiles), the used make command, macros in trace header files.

Example usage (local trace_suite):

libdist.py --configFile $HOME/config.xml --path $HOME/workspace/trace_suite --np 4

Further options:

$ ./libdist.py --help
bash: Zeile 1: ./libdist.py: Datei oder Verzeichnis nicht gefunden

Config XML

Example for an valid XML:

<?xml version="1.0" encoding="UTF-8"?>
<buildConfigurations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="config.xsd">
  <set>
    <name>beispiel_konfiguration</name>
    <modules>
      <mpi>openmpi</mpi>
      <compiler>gnu</compiler>
      <cgns>cgns</cgns>
      <vtk>vtk</vtk>
      <module>module</module>
    </modules>
    <make>
      <command>COMPILING_LANGUAGE=c++</command>
      <command>WARNINGS=no</command>
    </make>
    <macros>
      <headerfile>
        <path>trace/traceControl.h</path>
        <macro>
          <name>DOUBLE_PRECISION</name>
          <value>ON</value>
        </macro>
      </headerfile>
    </macros>
  </set>
</buildConfigurations>

The root element of the xml must have the tag <buildConfigurations> and has to contain the xml namespace and the xsd schema as attributes. You can add as many sets as you want using the tag <set>. A <set> has to contain the <modules> which should be loaded. Other tags are not necessary.

The <modules> tag has to contain the tags <mpi>, <compiler>, <cgns> and <vtk>. The text of <mpi> has to be one of the keywords: ‘openmpi’, ‘openmpi_intel’ and ‘no’. The text of <compiler> has to be one of the keywords: ‘gnu’ and ‘intel’. The text of <cgns> has to be one of the keywords: ‘cgns’. The text of <vtk> has to be one of the keywords: ‘vtk’. The values of the keywords can be defined in libdistData. You can also add an other module using the tag <module>. It has to contain the module name as text.

If you want to add make commands, you can use the tag <make> to add multiple <command>. The <command> tag has to contain the text which should be appended to the make command.

If you want to add macros, you can use the tag <macros>. The tag macros can contain multiple <headerfile> tags. The <headerfile> tags include one <path> tag, which text describes the path to the headerfile relative to the trace suite directory. Also a <headerfile> can contain multiple <macro> tags. They include a <name> and a <value> tag to define which constant should get which value.

If you have problems to create a valid xml you can use the eclipse validate function which mark the errors.

Modules

BuildConfiguration

Contains class “BuildConfiguration” which is used for saving the build configuration of one set from the XML file You have three methods to add macros, make commands and modules to the object the __str__ method is overwritten, so you can print the object

class BuildConfiguration(name)

Contains specification for one build configuration set fromm the XML file.

Parameters

name (string) – name of the buildConfiguration

addMacroChangeSet(macro)

appends macro to list of macro change sets for project

Parameters

macro (tuple) – tuple consist of headerPath variableName and value, e.g. (‘trace/traceControl.h’,’DOUBLE_PRECISION’,’ON’)

addMakeCommand(command)

appends make command to list of make commands for project

Parameters

command (string) – make command line option

addModules(key, module)

append module to list of modules for project

Parameters

• key (string) – name of the module

• module (string) – module which will be loaded

CommandLineOptions

Contains class ‘CommandLineOptions’ which offers all methods necessary to parse the command line. For parsing you have to call the method parse. It uses a Parser created by the method getParser to read in the command line.

class CommandLineOptions(argv)

Processed command line options

Parameters

argv (list) – command line arguments

getParser()

Creates parser which is used by parse

Returns

parser object to parse command line arguments

Return type

ArgumentParser

parse()

Parse command line using a parser created by getParser The command line options are saved in the CommandLineOptions object

class ConfigFileAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

class to control if the given or default path exists if the path already exists an error is raised if not the path is saved in the CommandLineOptions object

parseCommandLine(argv)

creates CommandLineOptions instance and uses it to parse the given command line

Parameters

argv (list) – command line arguments

Returns

command line options

Return type

CommandLineOptions

ConfigurationFile

Contains the class ‘ConfigurationFile’ which can read in all data stored in the config xml. When it is initialized the object gets the filename of the xml and the name of the settings. First you have to call readXML to parse the xml and save it as ElementTree. Then you can validate it against the xsd using the validateWithSchema method. buildBuildConfigurations get all data or just the data in the specified settings

class ConfigurationFile(filename, settings, traceSuitePath)

Object to process content of XML configuration file.

Parameters

• filename (string) – name of the xml file

• settings (list) – name of the settings to build

• traceSuitePath (string) – path to the trace suite

buildBuildConfigurations()

Read the settings which are stored in the ElementTree For each setting a BuildConfiguration’s object is generated In these objects all necessary information for buildung this setting is saved All buildConfigurations are appended to the buildConfigurations list

readXML()

Read file and save ElementTree in _eTree.

validateWithSchema()

ElementTree against XSD schema file. You’ll get back true if the document is valid against the XML schema, and false if not.

Returns

return true if the document is valid, and false if not

Return type

boolean

exception NotValidAgainstXmlException

createBuildConfigurations(configFilePath, settings, traceSuitePath)

creates ConfigurationFile instance, reads the xml at the given configFilePath and saves the information stored in it.

Parameters

• configFilePath (string) – configuration file path

• settings (list) – name of the settings to build

• traceSuitePath (string) – path to the trace suite

Returns

buildConfigurations

Return type

list of BuildConfiguration instances

Distribution Builder

Contains the class ‘Distribution Builder’ which is used for creating the distribution file

With the setup method all output and temp directories and the distribution name are created, it also calls modifySources, which customize the constant’s in the trace suite header files. The createDistribution method uses the createModuleBashScript and createMakeCommandLine methods to create a BashScript which builds the trace suite in libdist version and executes it. The created tar files are unpacked and tested by testDistribution. After a successful test the tested distribution is packed in one tar file with the packTestedDistribution method.

class DistributionBuilder(pathToSourceCodeTarFile, configFilePath, nProcsPerBuild, outputPath, buildConfiguration, traceSuitePath)

Build a libdist distribution file for one build configuration specification.

Parameters

• pathToSourceCodeTarFile (string) – path to the source code tarfile

• configFilePath (string) – path to configuration file

• nBuilds (int) – number of parallel builds

• outputPath (string) – directory in which the output file will be saved

• buildConfiguration (buildConfiguration object) – build configuration

• traceSuitePath (string) – path to the trace suite

cleanUp()

Remove all temporary files and directories.

copyDistributionToOutputPath()

calls addXMLToTestDistribution and createLinkScript, creates a tar in the output directory including all files stored in the testDistribution directory

createDistributionFile(envVars=None)

executes make command which is created by createMakeCommandLine creates a tarfile including trace_suite in libdist version

Parameters

envVars (dictionary) – environment variables

Returns

joblist for creating distribution file

Return type

joblist

createLibraries(pathToTarfile, pathToTraceSuite)

find all files which have to be added to the libraries. A tarfile with these libraries is created

Parameters

• pathToTarfile (string) – path to tarfile

• pathToTraceSuite (string) – path to trace suite

createMakeCommandLine(target=None, nprocs=None)

Create make command line string from make commands

Returns

make command line

Return type

string

getGccVersion()

return current gcc version

Returns

gcc version

Return type

string

getTraceSuiteVersion()

get trace version number

Returns

trace suite version

Return type

string

makeJob(make_command, nProcs, workingDirectory=None, outputDir=None, outputFile=None, jobName='makeJob', envVars=None)

creates a job which executes a make command

Parameters

• make_command (string) – make command

• nProcs (int) – number of processors

• workingDirectory (string) – path to working directory

• outputDir (string) – path to output directory

• outputFile (string) – path to output file

• jobName (string) – name of the created job

• envVars (dictionary) – environment variables

setup()

Prepare source code according to build configuration. Calls unpackDistribution, createOutputDirectory, createTempDirectory, createDistributionDirectories

Returns

creates setup joblist

Return type

joblist

testDistribution(envVars=None)

unpack distribution file, compile it, and run help. Should raise an exception in case of error.

Parameters

envVars (dictionary) – environment variables

Returns

test distribution joblist

Return type

joblist

createDistributions(buildConfigurations, pathToSourceCodeTarFile, configFilePath, nBuilds, nProcsPerBuild, outputPath, traceSuitePath, verbosity=<Verbosity.DEFAULT: 2>, keepOutput=False)

creates DistributionBuilder instance, builds distributions equivalent to the configurations stored in the buildConfigurations instances. The arguments pathToSourceCodeTarFile, configFilePath, nBuilds and outputPath are required.

Parameters

• buildConfigurations (build configuration object) – build configuration

• pathToSourceCodeTarFile (str) – path to the source code tarfile

• configFilePath (str) – path to configuration file

• nBuilds (int) – number of parallel builds

• nProcsPerBuild (int) – number of processors per build

• outputPath (str) – directory in which the output file will be saved

• traceSuitePath (str) – path to the trace suite

• verbosity (Printer.Verbosity) – verbosity

• keepOutput (bool) – controls if the jobmanagement output files are erased

Returns

number of failed jobs

Return type

int

getEnvironmentVariablesFromModules(modules)

Search the module files related to the modules which are described in the buildConfiguration. Analyses the module files and reads out how they would change the environment variables. The described changes are afterwards realized by this method. The new environment is returned as an dictionary.

Param

modules

Type

dictionary

Returns

environment variables

Return type

dictionary

libdistData

module containing all data used by the libdist project

stored data:

-the keywords used in the xml for describing the modules

-names of directories or output files

-tags used in the xml

setPrecompileFlag

provides methods to set value for constants in headerfiles

exception NotFoundConstantException(message)

replaceInFile(filename, pattern, subst)

replace pattern in file with subst

Parameters

• filename (string) – name of the file

• pattern (string) – pattern to replace

• subst (string) – replacement for the pattern

Returns

true if a pattern was replaced

Return type

boolean

setPrecompile(filename, variableName, value, verbose=0)

creates pattern and subst using variableName and value calls function replaceInFile and returns its return value

Parameters

• filename (string) – name of the file

• variableName (string) – name of the variable

• value (string) – value to set

• verbose (integer) – verbosity level

setPreprocessorMacroInFile(filename, variable, newValue)

matches a preprocessor define in a given file and rewrites it with the new value :param filename: name of the file :param variableName: name of the variable :param value: value to set :type filename: string :type variableName: string :type value: string

SourceCodeManager

Contains class ‘SourceCodeManager’ Organize the source code and copy from local disk

class SCM(path)

Get source code from disk.

Parameters

path (string) – path

cleanUp()

Remove all temporary files and directories.

class SourceCodeManager(path)

Get source code form disk and create a tar file with source code of all projects and return path to this file.

Parameters

path (string) – path

cleanUp()

Remove all temporary files and directories.

createTarFile()

Create source code tar file containing the source code of all projects form global projects list using specifications from command line parameters.