[[ux_admin]]

Administration
--------------

Controlling UNICORE/X memory usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can set a limit on the number of service instances (e.g. jobs) per user.
This allows you to make sure your server stays nicely up and running even if 
flooded by jobs. To enable, edit +CONF/wsrflite.xml+ and
add properties, e.g.

---------------
  <property name="unicore.maxInstancesPerUser.JobManagement" value="200"/>
  <property name="unicore.maxInstancesPerUser.FileTransferBFT" value="20"/>
--------------
  
The last part of the property name is the service name as defined in wsrflite.xml.
  
When the limits are reached, the server will report an error to the client (e.g. when
trying to submit a new job). 

[[ux_logging]]
Logging
~~~~~~~

UNICORE uses the Log4j logging framework. It is configured using a config file. By default, 
this file is +CONF/logging.properties+. To change the default, edit the start script 
(+BIN/start.sh+) or, on Windows, the +CONF/wrapper.conf+. The config file is specified with a Java 
property +log4j.configuration+.


[NOTE]
===========
You can change the logging configuration at runtime by editing the logging.properties file.
The new configuration will take effect a few seconds after the file has been modified.
===========

By default, log files are written to the the +LOGS+ directory. 

The following example config file configures logging so that log files are 
rotated daily.

-------------------
# Set root logger level to INFO and its only appender to A1.
log4j.rootLogger=INFO, A1

# A1 is set to be a rolling file appender with default params
log4j.appender.A1=org.apache.log4j.DailyRollingFileAppender
log4j.appender.A1.File=logs/uas.log

#configure daily rollover: once per day the uas.log will be copied 
#to a file named e.g. uas.log.2008-12-24
log4j.appender.A1.DatePattern='.'yyyy-MM-dd

# A1 uses the PatternLayout
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c{1} %x - %m%n
-------------------

[NOTE]
===========
In Log4j, the log rotation frequency is controlled by the DatePattern. Check 
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
for the details.
===========

Within the logging pattern, you can use special variables to output information.
In addition to the variables defined by Log4j (such as '%d'), UNICORE defines several variables
related to the client and the current job.

[options="header"]
|===============================
| Variable         | Description
|%X\{clientName\}  |the distinguished name of the current client
|%X\{jobID\}	   | the unique ID of the currently processed job
|===============================


A sample logging pattern might be

-------------------
log4j.appender.A1.layout.ConversionPattern=%d [%X{clientName}] [%X{jobID}] [%t] %-5p %c{1} %x - %m%n
-------------------


For more info on controlling the logging we refer to the log4j documentation:
  
  - http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html[PatternLayout]
  
  - http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html[RollingFileAppender]
  
  - http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html[DailyRollingFileAppender]
  
 
Log4j supports a very wide range of logging options, such as date based or size based file rollover,
logging different things to different files and much more. For full information on Log4j we refer to
the publicly available documentation, for example the 
http://logging.apache.org/log4j/1.2/manual.html[Log4j manual].
 

==== Logger categories, names and levels

Logger names are hierarchical. In UNICORE, prefixes are used (e.g. "unicore.security") 
to which the Java class name is appended. For example, the XUUDB connector in UNICORE/X logs to 
the "unicore.security.XUUDBAuthoriser" logger. 

Therefore the logging output produced can be controlled in a fine-grained manner. 
Log levels in Log4j are (in increasing level of severity) TRACE, DEBUG, INFO, WARN, 
ERROR, amd FATAL.

For example, to debug a security/authorisation problem in the UNICORE/X security layer, 
you can set
  
------
log4j.logger.unicore.security=DEBUG
------

If you are just interested in XUUDB related output, you can set

------
log4j.logger.unicore.security=INFO
log4j.logger.unicore.security.XUUDBAuthoriser=DEBUG
------

so the XUUDBAuthoriser will log on DEBUG level, while the other security components
log on INFO level.  

Here is a table of the various logger categories


[options="header"]
|===============================
|Log category      | Description
|unicore           | All of UNICORE 
|unicore.security  | Security layer 
|unicore.services  | Service operational information 
|unicore.services.jobexecution | Information related to job execution 
|unicore.services.jobexecution.USAGE | Usage logging (see next section)
|unicore.xnjs      | XNJS subsystem (execution engine) 
|unicore.xnjs.tsi  | TSI subsystem (batch system connector) 
|unicore.client    | Client calls (to other servers) 
|unicore.wsrflite  | Underlying services environment (WSRF framework) 
|====================================================================

[NOTE]
===========
Please take care to not set the global level to TRACE or DEBUG for long times, 
as this will produce a lot of output.
===========

==== Usage logging

Often it is desirable to keep track of the usage of your UNICORE site. The UNICORE/X
server has a special logger category called +unicore.services.jobexecution.USAGE+ which
logs information about finished jobs at INFO level. If you wish to enable this,
set 
------
log4j.logger.unicore.services.jobexecution.USAGE=INFO
------

[NOTE]
===========
If you are setting up a production environment and need a sophisticated accounting solution (featuring database 
with a real resources consumption, WWW interface and possibility to produce reports or aggregated data)
then consider deploying UNICORE RUS Accounting. Further instructions can be found in xref:xnjs-accounting. 
===========


It might be convenient to send usage output to a different file than normal log output.
This is easily achieved with log4j:

------
# send usage logger output to a separate file

# use separate appender 'U1' for usage info
log4j.logger.unicore.services.jobexecution.USAGE=INFO,U1

# U1 is set to be a rolling file appender with default params
log4j.appender.U1=org.apache.log4j.DailyRollingFileAppender
log4j.appender.U1.File=logs/usage.log
# U1 uses the PatternLayout                                                                           
log4j.appender.U1.layout=org.apache.log4j.PatternLayout
log4j.appender.U1.layout.ConversionPattern=%d [%t] %-5p %c{1} %x - %m%n
------

For each finished job, the usage logger will log a line with the following information (if available)
----
[result] [executable] [actionUUID] [clientDN] [BSSJobId] [clientXlogin] [jobName] [machineName] [VOs]
----

An example output line is:

----
2011-08-16 10:00:39,513 [XNJS-1-JobRunner-1] INFO  USAGE  - [SUCCESSFUL] [ /bin/date] [ e9deab79-af1f-4704-a6bd-427b3ab20969] [CN=Bernd Schuller, OU=VSGC, OU=Forschungszentrum Juelich GmbH, O=GridGermany, C=DE] [82942] [schuller] [Date job submitted using UCC] [zam025c02.zam.kfa-juelich.de] []
----


Administration and monitoring
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
The health of a UNICORE/X container, and things like running services, lifetimes, etc. can be 
monitored in several ways.
  
==== Commandline client (UCC)

It is possible to use the UNICORE commandline client (UCC) to list jobs, extend lifetimes, etc.

The trick is to configure UCC so that it uses the _server_ certificate of the UNICORE/X server,
so that UCC will have administrator rights. Also you should connect directly to UNICORE/X, not to 
the registry as usual. Here is an example UCC configuration file. Say your UNICORE/X server is 
running on _myhost_ on port 7777, your preferences file would look like this

--------------------
registry=https://myhost:7777/services/Registry?res=default_registry

# use UNICORE/X keystore
credential.path=/path/to/unicorex/keystore
credential.password=...

# truststore config omitted

-------------------

Note that the registry URL points directly to the UNICORE/X server, not to a gateway.

===== Examples

Some UCC commands that are useful are the _list-jobs_, _list-sites_ and _wsrf_ commands. 
Using _list-jobs_ you can search for jobs with given properties, whereas the _wsrf_ command 
allows to look at any resource, or even destroy resources.

To list all jobs on the server belonging to a specific user, do 
  
------
ucc list-jobs -f Log contains <username>
------

where _username_ is some unique part of the user's DN, or the xlogin. Similarly, you can
filter based on other properties of the job.

The _wsrf_ command can be used to destroy resources, extend their lifetime 
and look at their properties. Please see "ucc wsrf -h" for details.

Try

----------
ucc wsrf getproperties https://myhost:7777/services/TargetSystemFactory?res=default_target_system_factory
----------    

==== The Admin web service

The Admin service is a powerful tool to get "inside information" about your server using the UCC 
(or possibly another UNICORE client) and run one of the available "admin actions", which provide 
useful functions.

If you have enabled the admin service, you can do 

----------
ucc admin-info -l
----------    

to get information about available admin services. Note that you need to have role "admin" to invoke 
the admin service. The output includes information about the available administrative commands.
To run one of these, you can use the _admin-runcommand_ command. For example, to temporarily 
disable job submission

----------
ucc admin-runcommand ToggleJobSubmission
----------    




==== Java Management Extensions (JMX)

Using the Java Management Extensions, you can monitor any Java virtual machine using (for example) a 
program called "jconsole" that is part of the Sun/Oracle Java SDK. 
It allows to check memory and thread usage, as well as access to application specific management
components ("MBeans").

===== Enabling access

Connecting to a running Java VM locally is always possible, however remote access needs to be 
specially configured. To enable remote access to JMX, please check the startup.properties file 
of UNICORE/X. There several system properties are defined that enable and configure remote 
access via JMX.

-----------------------
#
#enable JMX (use jconsole to connect)
#
OPTS=$OPTS" -Dcom.sun.management.jmxremote"
#enable JMX remote access protected by username/password
OPTS=$OPTS" -Dcom.sun.management.jmxremote.port=9128 -Dcom.sun.management.jmxremote.authenticate=true"
OPTS=$OPTS" -Dcom.sun.management.jmxremote.ssl=false"
OPTS=$OPTS" -Dcom.sun.management.jmxremote.password.file=conf/jmxremote.password"
-----------------------

The password file "conf/jmxremote.password" must contain lines of the form "username=password", and must have
its permissions set to "rw for owner only" (at least on Unix).

Using jconsole, you can now connect from a remote machine.

=== Migration of a UNICORE/X server to another physical host

If you want to migrate a UNICORE/X server to another host, there are several things to consider.
The hostname and port are listed in CONF/wsrflite.xml and usually in the Gateway's connection.properties 
file. These you will have to change. Otherwise, you can copy the relevant files in CONF to the 
new machine. Also, the persisted state data needs to be moved to the new machine, if it is stored on the file
system. If it is stored in a database, there is nothing to be done.
If you are using a Perl TSI, you might need to edit the TSI's properties file and update the 
+tsi.njs_machine+ property.
