[[ux_idb]]
The UNICORE IDB (incarnation database) contains information on how abstract
job definitions are to be mapped onto real executables. This process (called 
"incarnation") is performed by the XNJS component.
The second IDB function is advertising target system capabilities and allowing
to check client resource requests against these.


The IDB is a (set of) XML files, which by default is called 'simpleidb'.

For reference, the current XML schema for the IDB can be read from the 
http://unicore.svn.sourceforge.net/viewvc/unicore/xnjs/trunk/xnjs-module-core/src/main/schema/idb.xsd[SVN repository].

=== Defining the IDB file

The IDB file is defined by the property "XNJS.idbfile", which must point to a file
on the UNICORE/X machine which is readable by the UNICORE/X process. For security reasons,
it should NOT be writable.

=== Using an IDB directory

While the IDB can be put into a single file, it is often convenient to
use multiple files. In this case, the property "XNJS.idbfile" points to
a directory. This directory should contain 

  - a single, mandatory, "main" IDB file
 
  - optionally, multiple XML files containing application definitions (see below)
 
  - optionally, multiple XML files containing execution environment definitions 
   (see xref:xnjs_execution_environments)

The main IDB file consists of an "IDB" XML element:
 
------------
<idb:IDB xmlns:idb="http://www.fz-juelich.de/unicore/xnjs/idb">
  ...
</idb:IDB
------------

while application files use the Application element

--------------------------------------
<idb:IDBApplication xmlns:idb="http://www.fz-juelich.de/unicore/xnjs/idb">
 ...
</idb:IDBApplication>
-------------------------------------

and the execution environment files look like this:

-----------
<ee:ExecutionEnvironment xmlns:ee="http://www.unicore.eu/unicore/jsdl-extensions">
 ...
</ee:ExecutionEnvironment>
-----------
  

=== Applications

The most important functionality of the IDB is providing executables for
abstract applications. An abstract application is given by name and version,
whereas an executable application is given in terms of executable, arguments
and environment variables.

==== Simple applications 

Here is an example entry for the "Date" application on a UNIX system
  
--------------------------------------
   <idb:IDBApplication xmlns:idb="http://www.fz-juelich.de/unicore/xnjs/idb">
    <idb:ApplicationName>Date</idb:ApplicationName>
    <idb:ApplicationVersion>1.0</idb:ApplicationVersion>
    <jsdl:POSIXApplication xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix">
      <jsdl:Executable>/bin/date</jsdl:Executable>
    </jsdl:POSIXApplication>
  </idb:IDBApplication>
-------------------------------------

As can be seen, "Date" is simply mapped to "/bin/date".

==== Arguments

Command line arguments are specified using <Argument> tags:
  
--------------------------------------
   <idb:IDBApplication xmlns:idb="http://www.fz-juelich.de/unicore/xnjs/idb">
    <idb:ApplicationName>LS</idb:ApplicationName>
    <idb:ApplicationVersion>1.0</idb:ApplicationVersion>
    <jsdl:POSIXApplication xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix">
      <jsdl:Executable>/bin/ls</jsdl:Executable>
      <jsdl:Argument>-l</jsdl:Argument>
      <jsdl:Argument>-t</jsdl:Argument>
    </jsdl:POSIXApplication>
   </idb:IDBApplication>
-------------------------------------

This would result in a command line "/bin/ls -l -t".

==== Conditional Arguments

The job submission from a client usually contains environment variables to be
set when running the application. It often happens that a certain argument
should only be included if a corresponding environment variable is set.
This can be achieved by using "conditional arguments" in the incarnation
definition. Conditional arguments are indicated by a quastion mark "?" 
appended to the argument value:
 
--------------------------------------
  <idb:IDBApplication>
    <idb:ApplicationName>java</idb:ApplicationName>
    <idb:ApplicationVersion>1.5.0</idb:ApplicationVersion>
    <jsdl:POSIXApplication xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix">
      <jsdl:Executable>/usr/bin/java</jsdl:Executable>
      <jsdl:Argument>-cp$CLASSPATH?</jsdl:Argument>
      <!-- other args omitted for clarity -->
    </jsdl:POSIXApplication>
   </idb:IDBApplication>
-------------------------------------

Here, +<jsdl:Argument>-cp$CLASSPATH?</jsdl:Argument>+ is an optional argument. 

If a job submission now includes a Environment variable named +CLASSPATH+

--------------------------------------
  <jsdl:Environment name="CLASSPATH">myjar.jar</jsdl:Environment>
--------------------------------------

the incarnated commandline will be "/usr/bin/java -cp$CLASSPATH ...",
otherwise just "/usr/bin/java ...".

This allows very flexible incarnations.

==== More

For more details about IDB application definitions, please consult
xref:ux_xnjs-idb-applications.

  
=== TargetSystemProperties
 
The TargetSystemProperties element contains information about a site's available resources, as
well as additional information that should be published to clients. 

==== Textual information

Simple strings can be entered into the IDB which are then
accessible client-side. This is very useful for conveying system-specifics
to client code and also to users. These text-info strings are entered into 
the IDB as a subtag of the +TargetSystemProperties+ tag

Here is an example

-----------
  <idb:TargetSystemProperties>
    
    <!-- text infos -->
    <idb:Info Name="Administator email">admin@site.org</idb:Info>

  </idb:TargetSystemProperties> 
 
------------

These pieces of information are accessible client side as part of the target system properties.


==== Resources
 
Resources of a target system are specified using the Resource tag defined
in the JSDL specification (see http://www.gridforum.org/documents/GFD.56.pdf).
It allows to specify things like number of nodes, CPUtime (per CPU), 
CPUs per node, total number of CPUs, etc.

These capabilities are specified giving an exact value and a range,
for example:

--------------------------------------
    <jsdl:Exact>3600</jsdl:Exact>
    <jsdl:Range>
     <jsdl:LowerBound>1</jsdl:LowerBound>
     <jsdl:UpperBound>86400</jsdl:UpperBound>
    </jsdl:Range>
--------------------------------------
  
The Range gives upper and lower bounds, where as the Exact value is
interpreted as the DEFAULT, when the client does not request anything
specific. If the Exact value is specified, the resource is part of the 
site's default resource set.

There exist a number of standard settings. You may choose to not specify some of
them, if they do not make sense on your system. For example, some sites do not allow
the user to explicitely select nodes and processors per node, but only "total number of CPUs".

 - +jsdl:IndividualCPUTime+ : The wall clock time. 

 - +jsdl:IndividualCPUCount+ : The number of CPUs per node

 - +jsdl:IndividualPhysicalMemory+ : The amount of memory per node (in bytes)

 - +jsdl:TotalResourceCount+ : The number of nodes.

 - +jsdl:TotalCPUCount+ : The total number of CPUs.
  
==== "Total CPUs" vs. "Nodes and CPUs per node"

Users can specify the number of processors either as just "total number of CPUs", or they 
can give a value for both "nodes" and "CPUs per node". If both are given, the values 
containing more information (i.e. nodes + CPUs per node) are used.

Similarly, if the administrator specifies both possibilities with a default value in the IDB, 
the nodes + CPUs per node will have precedence. 

==== CPU Architecture

JSDL allows to advertise the CPU architecture.

---------------------------------------------
      <jsdl:CPUArchitecture>
        <jsdl:CPUArchitectureName>x86</jsdl:CPUArchitectureName>
      </jsdl:CPUArchitecture>
---------------------------------------------

Due to restrictions imposed by the JSDL standard, the valid values for the
CPUArchitectureName element are limited to a fixed list, some useful values are 
"x86", "x86_64", "sparc", "powerpc", and "other".
For the full list please consult the JSDL standard.

==== Operating system

JSDL allows to advertise the operating system that the site runs.

---------------------------------------------
 <!-- O/S -->
 <jsdl:OperatingSystem>
    <jsdl:OperatingSystemType>
       <jsdl:OperatingSystemName>LINUX</jsdl:OperatingSystemName>
    </jsdl:OperatingSystemType>
    <jsdl:OperatingSystemVersion>2.6.13</jsdl:OperatingSystemVersion>
    <jsdl:Description>Ubuntu Linux</jsdl:Description>
 </jsdl:OperatingSystem>      
---------------------------------------------

Due to restrictions imposed by the JSDL standard, the valid values for the 
OperatingSystemName element are limited to a fixed list, some useful values are 
"LINUX", "SOLARIS", "AIX", "MACOS", "WIN_NT", "WINDOWS_XP", "FREE_BSD" and "UNKNOWN".
For the full list please consult the JSDL standard.


==== Other types of resources

Most HPC sites have special settings that cannot be mapped to
the generic JSDL elements shown in the previous section. Therefore UNICORE 6 includes
a mechanism to allow sites to specify their own system settings and allow
users to set these using the Grid middleware.

Custom resources are described in xref:xnjs-custom-resources.


==== File systems

File systems such as SCRATCH can be defined in the IDB as well,
for example

-----------
  <idb:TargetSystemProperties>
    
    <!-- SCRATCH file system -->
    <idb:Filesystem Name="SCRATCH" IncarnatedPath="/work/$USER" />

  </idb:TargetSystemProperties> 
 
------------

The job's environment will then contain a variable

----
SCRATCH="/work/$USER" ; export SCRATCH
----

JSDL data staging elements can contain the FileSystemName tag to indicate that the 
file should NOT be staged into the job working directory, but into the
named file system.

==== Example Resources section

This example includes the elements defining capabilities, and some
informational elements like CPUArchitecture and operating system info.

---------------------------------------------
<idb:TargetSystemProperties>
    <jsdl:Resources xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl">
      <jsdl:CPUArchitecture>
        <jsdl:CPUArchitectureName>x86</jsdl:CPUArchitectureName>
      </jsdl:CPUArchitecture>

 	<!-- O/S -->
      <jsdl:OperatingSystem>
       <jsdl:OperatingSystemType>
         <jsdl:OperatingSystemName>LINUX</jsdl:OperatingSystemName>
       </jsdl:OperatingSystemType>
       <jsdl:OperatingSystemVersion>2.6.13</jsdl:OperatingSystemVersion>
       <jsdl:Description>A free UNIX clone</jsdl:Description>
      </jsdl:OperatingSystem>
      
      <!-- cpu time (per cpu) in seconds -->
      <jsdl:IndividualCPUTime>
        <jsdl:Exact>3600</jsdl:Exact>
        <jsdl:Range>
         <jsdl:LowerBound>1</jsdl:LowerBound>
         <jsdl:UpperBound>86400</jsdl:UpperBound>
        </jsdl:Range>
      </jsdl:IndividualCPUTime>

      <!-- Nodes -->
      <jsdl:TotalResourceCount>
        <jsdl:Exact>1.0</jsdl:Exact>
        <jsdl:Range>
         <jsdl:LowerBound>1.0</jsdl:LowerBound>
         <jsdl:UpperBound>16.0</jsdl:UpperBound>
        </jsdl:Range>
      </jsdl:TotalResourceCount>

      <!-- CPUs per node -->
      <jsdl:IndividualCPUCount>
        <jsdl:Exact>8.0</jsdl:Exact>
        <jsdl:Range>
         <jsdl:LowerBound>1.0</jsdl:LowerBound>
         <jsdl:UpperBound>8.0</jsdl:UpperBound>
        </jsdl:Range>
      </jsdl:IndividualCPUCount>

      <!-- total CPUs -->
      <jsdl:TotalCPUCount>
        <jsdl:Exact>8.0</jsdl:Exact>
        <jsdl:Range>
         <jsdl:LowerBound>1.0</jsdl:LowerBound>
         <jsdl:UpperBound>128.0</jsdl:UpperBound>
        </jsdl:Range>
      </jsdl:TotalCPUCount>

      <!-- Memory per node (bytes) -->
      <jsdl:IndividualPhysicalMemory>
        <jsdl:Exact>268435456</jsdl:Exact>
        <jsdl:Range>
         <jsdl:LowerBound>1024576</jsdl:LowerBound>
         <jsdl:UpperBound>1073741824</jsdl:UpperBound>
        </jsdl:Range>
      </jsdl:IndividualPhysicalMemory>

    </jsdl:Resources>
  </idb:TargetSystemProperties>
---------------------------------------------


=== Script templates

If you need to modify the scripts that are generated by UNICORE/X and sent to the TSI, 
you can achieve this using two entries in the IDB.

-----------------
<idb:IDB xmlns:idb="http://www.fz-juelich.de/unicore/xnjs/idb">

<!-- Templates -->
<idb:SubmitScriptTemplate>
#!/bin/sh
#COMMAND
#RESOURCES
#SCRIPT
</idb:SubmitScriptTemplate>

<idb:ExecuteScriptTemplate>
#!/bin/sh
#COMMAND
#RESOURCES
#SCRIPT
</idb:ExecuteScriptTemplate>

<!-- rest of IDB omitted -->

</idb:IDB>
-----------------

The SubmitScriptTemplate is used for batch job submission, the ExecuteScriptTemplate is used for
everything else (e.g. creating directories, resolving user's home, etc)
  
UNICORE/X generates the TSI script as follows:

 - the "#COMMAND" entry will be replaced by the action for the TSI, e.g. "#TSI_SUBMIT". 
 - the "#RESOURCES" will be replaced by the resource requirements, e.g. "#TSI_NODES=..."
 - the "#SCRIPT" is the user script
 
Modifying these templates can be used to perform special actions, such as loading modules,
or changing the shell (but use something compatible to 'sh'). For example, to add some
special directory to the path for user scripts submitted in batch mode, you could use
  
-----------------
<idb:IDB xmlns:idb="http://www.fz-juelich.de/unicore/xnjs/idb">

<!-- Templates -->
<idb:SubmitScriptTemplate>
#!/bin/sh
#COMMAND
#RESOURCES
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi-2.1/lib; export LD_LIBRARY_PATH
PATH=$PATH:/opt/openmpi-2.1/bin; export PATH
#SCRIPT
</idb:SubmitScriptTemplate>

<!-- rest of IDB omitted -->

</idb:IDB>
-----------------
  

==== Properties

In the IDB file, XNJS properties can be specified, for example the command locations 
identified by property names starting with "CLASSICTSI."
 
------------
<idb:IDB xmlns:idb="http://www.fz-juelich.de/unicore/xnjs/idb">
<!--- rest of IDB omitted -->
 <idb:Property name="..." 
               value="..."/>
 </idb:IDB
------------


