[[xnjs-dynamicincarnation]]
   
Tweaking the incarnation process
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In UNICORE the term incarnation refers to the process of changing the abstract and 
probably universal _grid request_ into a sequence of operations _local to the 
target system_. The most fundamental part of this process is creation of the execution 
script which is invoked on the target system (usually via a batch queuing 
subsystem (BSS)) along with an execution context 
which includes local user id, group, BSS specific resource limits.
 
UNICORE provides a flexible incarnation model - most of the magic is done automatically
by TSI scripts basing on configuration which is read from the IDB. IDB covers script creation
(using templates, abstract application names etc). Mapping of the grid user to the local user is
done by using UNICORE Attribute Sources like XUUDB or UVOS.

In rare cases the standard UNICORE incarnation mechanism is not flexible enough. 
Typically this happens when the script which is sent to TSI should be tweaked in accordance 
to some runtime constraints. Few examples may include:

  - Administrator wants to set memory requirements for all invocations of the application X
  to 500MB if user requested lower amount of memory (as the administrator knows that the 
  application consumes always at least this amount of memory).

  - Administrator wants to perform custom logging of suspected requests (which for instance 
  exceed certain resource requirements threshold)

  - Administrator need to invoke a script that create a local user's account if it doesn't exist.

  - Administrator wants to reroute some requests to a specific BSS queue basing on the arbitrary 
  contents of the request.

  - Administrator wants to set certain flags in the script which is sent to TSI when a request 
  came from the member of a specific VO. Later those flags are consumed by TSI and are used
  as submission parameters.
  
Those and all similar actions can be performed with the Incarnation tweaking subsystem. Note that
though it is an extremely powerful mechanism, it is also a very complicated one and configuring it is 
error prone. Therefore always try to use the standard UNICORE features (like configuration of 
IDB and attribute sources) in the first place. Treat this incarnation tweaking subsystem as the
last resort!

To properly configure this mechanism at least a very basic Java programming language familiarity 
is required. Also remember that in case of any problems contacting the UNICORE support mailing list can 
be the solution.   

==== Operation

It is possible to influence incarnation in two ways: 
 
  - 'BEFORE-SCRIPT' it is possible to change all UNICORE variables which are used to 
  produce the final TSI script just _before it is created_ and
 
  - 'AFTER-SCRIPT' later on to _change the whole TSI script_.  

The first BEFORE-SCRIPT option is suggested: it is much easier as you have to modify some 
properties only. In the latter much more error prone version you can produce an entirely
new script or just change few lines of the script which was created automatically. It is
also possible to use both solutions simultaneously.

Both approaches are configured in a very similar way by defining rules. Each rule
has its condition which triggers it and list of actions which are invoked if the condition 
was evaluated to true. The condition is in both cases expressed in the same way. 
The difference is in case of actions. Actions for BEFORE-SCRIPT
rules can modify the incarnation variables but do not return a value. Actions for 
AFTER-SCRIPT read as its input the original TSI script and must write out the updated version.
Theoretically AFTER-SCRIPT actions can also modify the incarnation variables but this 
doesn't make sense as those variables won't be used. 
 

==== Basic configuration

By default the subsystem is turned off. To enable it you must perform two simple things:

  - Add the +XNJS.incarnationTweakerConfig+ property to the XNJS config file. The value
  of the property must provide a location of the file with dynamic incarnation rules.
  
  - Add some rules to the file configured above.

The following example shows how to set the configuration file to the 
value +conf/incarnationTweaker.xml+:

--------
  ...
  <eng:Properties>
    ...
    <eng:Property name="XNJS.incarnationTweakerConfig" value="conf/incarnationTweaker.xml"/>
    ...
  </eng:Properties>
  ...
--------

The contents of the rules configuration file must be created following this syntax:

-------
<?xml version="1.0" encoding="UTF-8"?>
<tns:incarnationTweaker xmlns:tns="http://eu.unicore/xnjs/incarnationTweaker"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	<tns:beforeScript>
		<!-- Here come BEFORE-SCRIPT rules-->
	</tns:beforeScript>

	<tns:afterScript>
		<!-- And here AFTER-SCRIPT rules-->
	</tns:afterScript>
</tns:incarnationTweaker>
--------

==== Creating rules

Each rule must conform to the following syntax:

---------
	<tns:rule finishOnHit="false">
		<tns:condition> <!-- Here comes the rule's condition --> </tns:condition>
		
		<tns:action type="ACTION-TYPE">ACTION-DEFINITION</tns:action>
		<!-- More actions may follow -->
	</tns:rule>
---------

The rule's attribute +finishOnHit+ is optional, by default its value is false. When it is 
present and set to true then this rule becomes the last rule invoked if it's condition was 
met.

You can use as many actions as you want (assuming that at least one is present), 
actions are invoked in the order of appearance.

===== SpEL and Groovy

Rule conditions are always boolean expressions of the Spring Expression Language (SpEL). 
As SpEL can be also used in some types of actions it is the most fundamental tool to understand.  
  
Full documentation is available here:
http://static.springsource.org/spring/docs/3.0.0.M3/spring-framework-reference/html/ch07.html

The most useful is the section 7.5:
http://static.springsource.org/spring/docs/3.0.0.M3/spring-framework-reference/html/ch07s05.html

Actions can be also coded using the Groovy language. You can find Groovy documentation
at Groovy's web page: 
http://groovy.codehaus.org

===== Creating conditions

Rule conditions are always Spring Expression Language (SpEL) boolean expressions.
To create SpEL expressions, the access to the request-related variables must be provided.
All variables which are available for conditions are explained in 
xref:xnjs-dynamicincarnation-context.

===== Creating BEFORE-SCRIPT actions

There are the following action types which you can use:
 
  - +spel+ (the default which is used when type parameter is not specified) treats action value
 as SpEL expression which is simply evaluated. This is useful for simple actions that should modify
 value of one variable.

  - +script+ treats action value as a SpEL expression which is evaluated and which 
 should return a string. Evaluation is done using SpEL templating feature with 
 +\$\{+ and +\}+ used as variable delimiters (see section 7.5.13 in Spring 
 documentation for details). The returned string is used as a command line which is invoked.
 This action is extremely useful if you want to run an external program with some arguments which 
 are determined at runtime.
 Note that if you want to cite some values that may contain spaces (to treat them as a single
 program argument) you can put them between double quotes '"'. 
 Also escaping characters with "\" works. 

  - +groovy+ treats action value as a Groovy script. The script is simply invoked and can 
 manipulate the variables.

  - +groovy-file+ works similarly to the +groovy+ action but the Groovy script is read from 
  the file given as the action value.

All actions have access to the same variables as conditions; see xref:xnjs-dynamicincarnation-context
for details.

===== Creating AFTER-SCRIPT actions

There are the following action types which you can use:
 
  - +script+ (the default which is used when type parameter is not specified) treats action value
 as SpEL expression which is evaluated and which should return a string. Evaluation is done using
 SpEL templating feature with +\$\{+ and +\}+ used as variable delimiters (see section 7.5.13
 in Spring documentation for details). The returned string used as a command line which is invoked.
 The invoked application gets as its standard input the automatically created TSI script and
 is supposed to return (using standard output) the updated script which shall be used instead. 
 This action is extremely useful if you want to run an external program with some arguments which 
 are determined at runtime.
 Note that if you want to cite some values that may contain spaces (to treat them as a single
 program argument) you can put them between double quotes +"+. 
 Also escaping characters with +\+ works. 
 
  - +groovy+ treats action value as a Groovy script. The script has access to one special variable
 +input+ of type +Reader+. The original TSI script is available from this reader. The groovy
 script is expected to print the updated TSI script which shall be used instead of the original one. 

  - +groovy-file+ works the same as the +groovy+ action but the Groovy script is read from the 
 file given as the action value.

All actions have access to the same variables as conditions; see xref:xnjs-dynamicincarnation for details. 

==== Final notes

 - The rules configuration file is automatically reread at runtime.
 
 - If errors are detected in the rules configuration file upon server startup then the 
 whole subsystem is disabled. If errors are detected at runtime after an update then old 
 version of rules is continued to be used. Always check the log file!
 
 - When rules are read the system tries to perform a dry run using an absolutely minimal
 execution context. This can detect some problems in your rules but mostly only in conditions.
 Actions connected to conditions which are not met won't be invoked. Always try to submit a real
 request to trigger your new rules!
 
 - Be careful when writing conditions: it is possible to change incarnation variables inside
 your condition - such changes also influence incarnation.
 
 - It is possible (from the version 6.4.2 up) to stop the job processing from the rule's action. 
 To do so with the +grovy+ or +grovy-file+ action, throw the 
 +de.fzj.unicore.xnjs.ems.ExecutionException+ object from the script.
 In case of the +script+ action, the script must exit with the exit status equal to +10+. 
 The first 1024 bytes of its standard error are used as the message which is included in the
 ExecutionException. This feature works both for the BEFORE- and AFTER- SCRIPT actions. It
 is not possible to achieve this with the +spel+ action type. 
 

==== Complete examples and hints
 
Invoking a logging script for users who have the +specialOne+ role.
Note that the script is invoked with two arguments (role name and client's DN). As the latter argument
may contain spaces we surround it with quotation marks.

-------
<?xml version="1.0" encoding="UTF-8"?>
<tns:incarnationTweaker xmlns:tns="http://eu.unicore/xnjs/incarnationTweaker"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	<tns:beforeScript>
		<tns:rule>
			<tns:condition>client.role.name == "specialOne"</tns:condition>
			<tns:action type="script">/opt/scripts/logSpecials.sh ${client.role.name} "${client.distinguishedName}"</tns:action>
		</tns:rule>
	</tns:beforeScript>

	<tns:afterScript>
	</tns:afterScript>
</tns:incarnationTweaker>
------

A more complex example. Let's implement the following rules:
 
  - The Application with a IDB name +HEAVY-APP+ will always get 500MB of memory requirement if user
  requested less or nothing.
  
  - All invocations of an executable '/usr/bin/serial-app' are made serial, i.e. the number
  of requested nodes and CPUs are set to 1.
  
  - For all requests a special script is called which can create a local account if needed 
  along with appropriate groups.
  
  - There is also one AFTER-RULE. It invokes a groovy script which adds an additional 
  line to the TSI script just after the first line. The line is added for all invocations of the 
  '/usr/bin/serial-app' program.

The realization of the above logic can be written as follows:
 
-------
<?xml version="1.0" encoding="UTF-8"?>
<tns:incarnationTweaker xmlns:tns="http://eu.unicore/xnjs/incarnationTweaker"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	<tns:beforeScript>
		<tns:rule>
			<tns:condition>app.applicationName == "HEAVY-APP" and (resources.individualPhysicalMemory == null 
                                          or resources.individualPhysicalMemory &lt; 500000000)</tns:condition>
			<tns:action>resources.individualPhysicalMemory=500000000</tns:action>
		</tns:rule>
		<tns:rule>
			<tns:condition>app.executable == "/usr/bin/serial-app" and resources.individualCPUCount != null</tns:condition>
			<tns:action>resources.individualCPUCount=1</tns:action>
			<tns:action>resources.totalResourceCount=1</tns:action>
		</tns:rule>
		<tns:rule>
			<tns:condition>true</tns:condition>
			<tns:action type="script">/opt/addUserIfNotExists.sh ${client.xlogin.userName} ${client.xlogin.encodedGroups}</tns:action>
		</tns:rule>
	</tns:beforeScript>

	<tns:afterScript>
		<tns:rule>
			<tns:condition>app.executable == "/usr/bin/serial-app"</tns:condition>
			<tns:action type="groovy">
int i=0;
input.eachLine() { line ->  
if(i==1) {
     println("#TSI_MYFLAG=SERIAL");
     println(line);
} else  
     println(line);

i++;
}
			</tns:action>
		</tns:rule>
	</tns:afterScript>
</tns:incarnationTweaker>
-------
 
Remember that some characters are special in XML (e.g. +<+ and +&+). You have to encode them 
with XML entities (e.g. as +\&lt;+ and +\&gt;+ respectively) or put the whole text in a CDATA section.
A CDATA section starts with "<![CDATA[" and ends with "]]>". Example:

--------
<tns:condition><!CDATA[ resources.individualPhysicalMemory < 500000000 ]]></tns:condition>
--------
 
Note that usually it is better to put Groovy scripts in a separate file. Assuming that you placed 
the contents of the groovy AFTER-action above in a file called '/opt/scripts/filter1.g'
then the following AFTER-SCRIPT section is equivalent to the above one:

--------
	<tns:afterScript>
		<tns:rule>
			<tns:condition>app.executable == "/usr/bin/serial-app"</tns:condition>
			<tns:action type="groovy-file">/opt/scripts/filter1.g</tns:action>
		</tns:rule>
	</tns:afterScript>
--------

It is possible to fail the job when a site-specific condition is met. 
E.g. with the groovy script: 

--------
	<tns:afterScript>
		<tns:rule>
			<tns:condition>SOME - CONDITION</tns:condition>
			<tns:action type="groovy">
throw new de.fzj.unicore.xnjs.ems.ExecutionException(de.fzj.unicore.xnjs.util.ErrorCode.ERR_EXECUTABLE_FORBIDDEN, "Description for the user");
			</tns:action>
		</tns:rule>
	</tns:afterScript>
--------

To check your rules when you develop them, it might be wise to enable DEBUG logging on
incarnation tweaker facility. To do so add the following setting to 
the +logging.properties+ file:

---------
log4j.logger.unicore.xnjs.IncarnationTweaker=DEBUG
---------

You may also want to see how the final TSI script looks like. Most often TSI places it 
in a file in job's directory. However if the TSI you use doesn't do so (e.g. in case of
the NOBATCH TSI) you can trigger logging of the TSI script on the XNJS side. There are two
ways to do it. You can enable DEBUG logging on the +unicore.xnjs.tsi.TSIConnection+ facility:

---------
log4j.logger.unicore.xnjs.tsi.TSIConnection=DEBUG
---------

This solution is easy but it will produce also much more of additional information in you log file.
If you want to log TSI scripts only, you can use AFTER-SCRIPT rule as follows:

---------
        <tns:afterScript>
                <tns:rule>
                        <tns:condition>true</tns:condition>
                        <tns:action type="groovy">
org.apache.log4j.Logger log=org.apache.log4j.Logger.getLogger("unicore.xnjs.RequestLogging");
log.info("Dumping TSI request:");
input.eachLine() { line ->  
     println(line);
     log.info("  " + line);
}
                        </tns:action>
                </tns:rule>
        </tns:afterScript>
----------

The above rule logs all requests to the normal Unicore/X log file with the INFO level.
