[[ucc_jobdescription]]

Job description format
---------------------

UCC uses a simple format that allows you to specify the application or executable you
want to run, arguments and environment settings, any files to stage in from remote servers or the
local machine and any result files to stage out.

A number of sample files can be found in the "samples" directory of your UCC distribution.
(on Linux, check also /usr/share/unicore/ucc/samples)

The format used is called http://www.json.org[JSON], and contains comma-separated key-value mappings,
where the values can be simple strings, or lists of values, or maps. 
String values should be placed in "quotes". Comments are (inofficially) possible using the 
"#" hash character, as in Unix shell scrips.

Each JSON file must begin and end with curly braces "\{ ... \}". Several complete job samples
can be found in the "samples" directory of the distribution.

[NOTE]
=========
Note: quotes "" are needed around the keys and values in case special characters (like ':' or '/'")
appear, if in doubt use quotes!
==========

To view an example job showing all available options, simply run
  
--------------------------
ucc run -H
--------------------------

(most of the options shown are not mandatory, of course)

[NOTE]
============
You may alternatively specify jobs in the JSDL format that is used internally in UNICORE 6. 
To do this, run UCC with the "-j" option.
============

=== Site name

You can (optionally) specify on which site (if available) the job should be run.

--------------------------

  Site: "DEMO-SITE",
  
--------------------------

If you do not specifiy anything UCC will select a site that will match your requirements 
(at least those that UCC checks for).

===  Specifying the application or executable

You can specify a UNICORE application by name and version, or using a (machine dependent) path
to an executable file.

--------------------------
#using application name and version
{
   ApplicationName: "Date",
   ApplicationVersion: "1.0",  
}
--------------------------

Note the comma-separation and the curly braces. To call an executable,
  
--------------------------
#using an executable

{
   Executable: "/bin/date",  
}
--------------------------

===  Arguments and Environment settings

Arguments and environment settings are specified using a list of String values. Here is an example.
  
--------------------------
{

   Executable: "/bin/ls",
   
   Arguments: ["-l", "-t"],
   
   Environment: ["PATH=/bin", "FOO=bar"],
   
}
--------------------------


=== Application parameters

In UNICORE, parameters for applications are often transferred in the form of environment variables.
For example, the POVRay application has a large set of parameters to specify image width, 
height and many more.
In UCC, you can specify these parameters in a very simple way using the "Parameters" keyword:

--------------------------

{
  ApplicationName: POVRay,

  Parameters{
   WIDTH: 640,
   HEIGHT: 480,
   DEBUG: "",
  },  

}
--------------------------

Note that an "empty" parameter (which does not have a value) needs to be written with an explicit
empty string due to the limitations of the JSON syntax.

=== Job data management

In general your job will require data files, either from your client machine, or from some remote location.
An important concept in UNICORE is the job's workspace (also called 'Uspace', which is the default location 
into which files are placed. The same applies to result files: by default, files will be downloaded from 
the job's workspace.

However, other remote storage locations are supported, too.

==== Importing files into the job workspace

To import files from your local computer or from remote sites to the 
job's working directory on the remote UNICORE server, there's the "Imports" 
keyword. Here is an example Import section that specifies three imports:
 

--------------------------
{

   Imports: [ 
   
   # import a local file into the job workspace
    { From: "/work/data/fileName", To: "uspaceFileName" },
    
   # import a set of PDF files into the Uspace 
    { From: "/work/data/pdf/*.pdf", To: "/" },

   # import a remote file from a UNICORE storage
    { From: "u6://DEMO-SITE/Home/testfile", To: "otherUspaceFile" },
   ]
   
}
--------------------------

If for some reason it may happen that the local file does not exist, and you want 
the job to run anyway, there is a flag "FailOnError" that can be set to "false" :

--------------------------
   Imports: [ 
   # do not fail on errors for this import:
    { From: "/work/data/fileName", To: "uspaceFileName", FailOnError: "false", },
   ]
--------------------------


[NOTE]
===============
UCC supports simple wild cards ("*" and "?") for importing exporting LOCAL files, i.e.
currently wildcards do not work for server-to-server imports and exports.
===============

==== Importing files into other storage locations

[NOTE]
===============
This requires a UNICORE server v6.4.2 or later
===============

A UNICORE site may supports multiple storages (for example, a TMP or SCRATCH directory).
To instruct the server to stage-in a file into such a storage, the "Filesystem" tag may be
used. For example to stage-in a file into SCRATCH space, the following Imports definition
can be used:

--------------------------
   Imports: [ 
   
   # import a file from a remote storage into the SCRATCH space on the target resource
    { From: "u6://DEMO-SITE/Home/work/data/fileName", To: "fileName", Filesystem: "SCRATCH", },
   
--------------------------


==== Exporting result files from the job workspace

To export files from the job's working directory to your local machine or to some remote
storage, use the "Exports" keyword. Here is an example Exports section that specifies two exports:

--------------------------
{

   Exports: [ 
    #this exports all png files to a local directory
    { From: "*.png", To: "/home/me/images/" },

    #this exports a single file to a to local directory
    #failure of this data transfer will be ignored
    { From: "error.log", To: "/home/me/logs/error.log", FailOnError: "false", },
    
    #this exports to a UNICORE storage
    { From: "stdout", To: "u6://DEMO-SITE/Home/results/myjob/stdout" },
   
   ]
   
}
--------------------------

As a special case, UCC also supports downloading files from other UNICORE storages 
using the Exports keyword:

--------------------------
{
   Exports: [ 
    #this exports a file from a UNICORE storage
    { From: "u6://DEMO-SITE/Work/somefile", To: "/home/me/somefile" },
   ]   
}
--------------------------


The remote location can be given as a full UNICORE 6 URI, or using the more user 
friendly (but slower) "u6://" notation. Read more on remote locations in xref:ucc_datamanagement[]. 

Local files can be given as an absolute or relative path; in the latter case the configured 
output directory will be used as base directory.

The protocol to be used for imports and exports can be chosen using the "Preferred Protocols" 
entry, containing a space-separated list of protocols:

--------------------------
{

   Preferred protocols: "BFT RBYTEIO",

}
--------------------------
  
If not specified, BFT will be used.

==== Specifying credentials for data staging

Some data staging protocols supported by UNICORE require credentials such as username and password.
Currently, these are "ftp" and "scp". In case you want to give username and password, the syntax
is as follows

--------------------------
{
   Imports: [ 
     { From: "ftp://someserver:25/some/file", To: "input_data"
       Credentials: { Username: "myname", Password: "mypassword" },
     },
   ]   
}
--------------------------

and similarly for exports.
 
==== Redirecting standard input

If you want to have your application or executable read its standard input from a file, you 
can use the following

--------------------------
  Stdin: filename,
--------------------------

then the standard input will come from the file named "filename" in the job working directory. 



=== Resources

A job definition can have a Resources section specifying the resources to request
on the remote system.

--------------------------

  Resources: {
  
    #memory per CPU (bytes, you may use the common "K","M" or "G")
    Memory: 268435456 ,
    
    #time per CPU (seconds, use "min", "h", or "d" for other units)
    Runtime: 86400 ,
    
    #Total number of requested CPUs
    CPUs: 64 ,
    
    #you may optionally give the number of nodes
    #Nodes: 2 ,
    #together with the CPUs per node
    #CPUsPerNode: 32,

    #Custom resources (site-dependent!)
    StackLimitPerThread : 262144,
  
    #Operating system
    Operating system: LINUX,   #MACOS, WINNT, ...
  	
    #Resource reservation reference
    Reservation: job1234,
  	
  }

--------------------------


Note that you can also specify a reservation reference if your batch system supports this
and you have made a resource reservation.

=== Execution environments

To run a job in a special execution environments (as supported by the server),
you can use the following syntax.

--------------------------
  Execution environment: {
      Name: ...,
      Arguments: {
        ArgName1: "value1", ArgName2: "value2", ...
      },
      Options: [ ... ],
      Precommands: [ ... ],
      Postcommands: [ ... ],
      User precommand: "..." ,
      User postcommand: "..." ,
    },
-----------------------------


=== Miscellaneous options

==== Selecting the remote login and/or group

In case you have multiple logins or Unix groups on the remote site 
mapped to the same credential, you can select the user name 
and/or group to use as follows

-------------------------

  User name: yourlogin,
  Group: yourgroup,
---------------------------

Hint: you can get a list of your logins/groups on the site by executing

--------------------------
  ucc list-sites -s SITENAME -l
---------------------------

==== Specifying a project

If the system you're submitting to does accounting, you can specify
the account (or project) you want to charge the job to using the
"Project" tag:

---------------------
Project: "my_project",
---------------------


==== Specifying the user email for batch system notifications

Some batch systems support sending email upon completion of jobs. To specify
your email, use

--------------------------

  User email: foo@bar.org

---------------------------

Hint: if you want to explicitely switch off the email notification, use "NONE" as email
value. This might be necessary because older UNICORE server versions try to use the email 
address from your certificate (if present).


==== Specifying the job name

The job name can be set simply by

--------------------------

  Name: Test job

---------------------------


==== Specifying the status check interval for batch mode

Once a job is started, it is often not useful to check its status every few seconds, because
the job might be running several minutes or more. Especially in batch mode it can reduce
the load on the servers if the update interval is chosen longer. This can be achieved
by using the following setting (this only affects batch mode!):

--------------------------

  Update interval: 60,   #only check once a minute (default is one second)

---------------------------


==== Specifying the "lifetime" of the job

If you want to specify a lifetime of the job, and not rely on the server
default, you can use the lifetime attribute:


--------------------------

  Lifetime: 12h,   #sec, min, h, d

---------------------------
