ENCP release notes, from v2_20 to v3_1
Encp changes:
=============
With the --threaded option, encp will multi-thread the actual data transfer.
This has two advantages. First, there is usually (though not always)
an increase in the transfer rate. Second, this allows for encp to
independently measure the access to/from local disk and the network.
A bug fix was made to limit the number of "mover impostor" error messages.
This was caused by an obscure order of events with a symptom of 15 minutes
delays between retries.
With the --ecrc switch, encp (on reads) will re-read the new local file from
beginning to end and recalculated the CRC again. This has a possible flaw
in that the re-read may only check what is still in the buffer cache and
not what was written to disk.
Originally there was only CRC_ERROR. This error will now only be generated
by the mover (as it was previously). However, encp can now generate two
new errors: CRC_ENCP_ERROR or CRC_ECRC_ERROR. The first is raised when
encp notices during the transfer that the data has been corrupted. The second
is when the corruption is noticed during the --ecrc readback check. Frequent
occurances of CRC_ENCP_ERROR or CRC_ECRC_ERROR have indicated in the past
a hardware problem with the client node.
The "List index out of range" error (aka "Error processing retry of
") has been fixed. The retries and resubmits work correctly.
Encp --help gives a short description of the switches.
Support for encp to notify the alarm server of problems that require
remote computer administrator intervention. The alarm server then will
send the e-mail to a designated list of e-mail addresses. This is useful
for errors like CRC_ENCP_ERROR or CRC_ECRC_ERROR.
For multi-file write transfers, the delayed dismount for the first (n - 1)
files will be at least 3 minutes. The last one will continue to use the
configured system default. This helps prevent the tape from being
dismounted prematurely because the system was busy and the encp was unable
to submit the next request (writes).
A bug fix was made that could have led encp to return a success exit status
when there was an error. This only applied to multi-file read transfers
where the files were spread over multiple volumes. Encp submits read
requests in per volume groups. If an error occurred while trying to read the
first volume's files, but the second (and final) volume's files all read
without error the second volumes success was returned. This last-one-wins
error has been fixed to return the or-ed status of all the volumes' transfers.
With dcache and srm there are multiple paths to the same file. For example,
/pnfs/xyz, /pnfs/fs/usr/xyz and /pnfs/fnal.gov/usr/xyz. Encp on reads
will first attempt the path that the user specified, if that path does
not exist it will then search the other two paths to attempt to deliver
the users thier files. For writes users must still know where they are
writing.
On machines that have interface lines in their enstore.conf file, this file
is re-read before every transfer instead of only once at the beginning
of the encp. This will limit the time it takes for an interface to be ignored
the retry interval of the encp (which by default is 15 minutes). Previously,
encp would read this file only once at the beginning of execution and
it could take hours for all encps started before the change to the enstore.conf
file to complete.
There are now 5 rates measured during a transfer:
1) network - The rate at which the file is sent over the network. For a single
threaded transfer this rate is the same (minus context switches) as the disk
and transfer rates. Measured by encp.
2) tape (or drive) - The rate at which the file is read or written from/to
the tape including selective CRC check implemented on written files by
mover. Measured by mover.
3) disk - The rate at which the file is read/written from/to disk. For a
single threaded transfer this rate is the same (minus context switches)
as the network and transfer rates. Measured by encp.
4) overall - The rate measured from the moment the mover contacts encp until
the last piece of metadata is set. This does include any mount time
and CRC readback check time in the rate calculation. Measured by encp.
5) transfer - the network rate in previous versions of encp. This rate is
invariant to single or multi-threaded encp transfers. On reads, it
is measured from when the first byte is detected on encp's data socket
until the last byte has been flushed to disk. On writes, it is measured
from when the mover is ready to receive data from encp until the last bytes
has been written two tape (and the mover responds saying it is done).
Measured by encp.
Encp now has a consistent interpretation of what a pnfs directory is with
the enstore pnfs commands. For most users this was not a problem, however
for users that choose to automount their pnfs file systems there were problems.
For the record, the Enstore developers consider automounting pnfs filesytems
to be problematic.
Encp now records completed transfer information to the Accounting server.
With the --mmap-io switch memory mapped i/o may be used on supported
file systems. The switch --direct-io may also be used on Linux and IRIX
nodes with supporting file systems (very tricky to make this work). If memory
mapped i/o or direct i/o is not supported by the file system, encp will
automatically attempt to revert to more portable file system access routines.
These options (along with others) can *possibly* help speed up some transfers.
However, in the hands of unknowledgeable users they can create problems for
themselves and their collaborators. These options were originally
developed for use by dcache and are not considered very usefull to the
general user.
Various file and directory errors are handled better. In the specific
cases found, correct error messages are now given. For example, it was
possible to get ENODATA when the parent directory passed to encp on a write
was really a regular file; the reported error should have been ENOTDIR.
The CRC of the file is now returned in the data-access-layer output. The
CRC is also being added to the pnfs layer 4 of files written with encp v3_1.
This is the first version of encp to be cut with python 2.2.3.
Since this version is cut with python 2.2.3, a large file bug that affected
large files on Linux systems is fixed.
Misc.:
======
The legal characters that "enstore pnfs" commands will let you set into a
tag are: letters, digits, underscore (_), dash (-) and slash (/).
Although, v2_20 does work with the new cern wrapper, this is the first
encp release since it was supported in the mover. The only advantage the
cern wrapper has over the cpio_odc wrapper is that the cern wrapper is not
limited to (8GB - 1) byte files.
The "enstore pnfs" commands should now work in a consistent way with
automounted pnfs filesystems. For the record, the Enstore developers
consider automounting pnfs filesytems to be problematic.
The enstore pnfs --tagchown command will now take uids and gids instead of
just user names and group names.
A new executable called ensync is included in the Encp product. It is an
enstore aware rsync like program. It is intended for smaller experiments
and is not optimized for massive numbers of file transfers.
Another executable called ddencp (or disk to disk encp) is provided. It
can be used as a debugging tool to investigate potential disk problems
on client nodes.
Detailed cvs commit logs
========== Trace.py ====================================================================================
support using remedy_api
Moved an error message to use sys.stderr.write() instead of a print statement.
Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference.
Commented out two lines that servered no purpose. If default_alarm_func() ever got called it would crash the calling program.
========== alarm.py ====================================================================================
remove .fnal.gov from remedy system name
fill in system etc
remove hard coded system name
support using remedy_api
change time.daylight to a -1
========== alarm_server.py ====================================================================================
support using remedy_api
add support for actions in alarm server
add trace.trace calls
========== atomic.py ====================================================================================
Fixed a bug if open fails. The raise statement would return an empty OSError under the old implimentation. Now it raises the exception correctly.
Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference.
Modification to deal with the phantom EEXIST errors creating the lock file.
========== backup.py ====================================================================================
add --ignore-failed-read for tar
Fixed a few little things so the code will work with enstore.py. Also, a few modifications so that it has less problems with pychecker.
modulized ... add do_work()
volume_clerk_client.py
no cachen2a and force gzip
========== callback.py ====================================================================================
Better timeout handling for writes.
========== db.py ====================================================================================
guard it against os.getcwd() failure
cleanup
do not need to chdir to dbHome in backup
========== delete_at_exit.py ====================================================================================
Moved an error message to useing sys.stderr.write() instead of the print statement.
Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv.
========== driver.py ====================================================================================
These files are partially modifed to address lingering cern wrapper bugs.
========== e_errors.py ====================================================================================
check transfer rates and disconnect from encp if it is slow
1) Included support for e-mailable alarms. New function is_emailable(). 2) New error CRC_ECRC_ERROR. This is now split from the CRC_ENCP_ERROR error.
added POSITIONING_ERROR
Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types.
Reviewed various non-retriable and raise alarm errors. Seperated CRC_ENCP_ERROR from CRC_ERROR. Also, cleaned up the code for readability.
Added new error when EXfer read()/write() calls hang in the kernel to long. This error, DEVICE_ERROR, will also raise alarm.
set volume to noaccess and mover in error state if mover is stuck in some intermediate state
========== encp.py ====================================================================================
Encp will now record the crc in layer 4 of the file. All internal encp checks have been updated to handle this. Also, all internal changes to pnfs.py have been made to handle this extra piece of data.
Fixed a bug preventing encp from reading files without brands. Also, made a change to remove a pychecker warning on IRIX.
Handle the non-optional routing communication for "Get" correctly.
Fixed a problem with encp.py working with get.py correctly.
bumping version to v3_1 because of encpCut
encp.py now uses fullpath() from enstore_functions2.py. Fixes include improved usage of the newly rewritten pnfs.is_pnfs_path() function. This allows for better checking for "unix to unix" or "hsm to hsm" at the beginning of the program, but still leaves the actual existance, permission, etc. checks for later.
bumping version to v3_0 because of encpCut
Fixed more branding problems. Clamped down on valid brands. Fixed bug that returned default csc even after determining which csc it should use.
Fixed problems with branding and determining which enstore system has the requested file.
Fix bug preventing encp from reading files with thier bfid brands containing non-alphanumeric characters (notably B0MS and D0MS with zeros).
For some reason the 'inode' field was always set to 0 instead of the actual inode. The actual inode is now used.
Added 'address' to the volume clerk sub-ticket.
Handle a "Get" related problem when the filesize is not known and using python 2.0.
Changes for mylint.py to be happy.
Another change for encp.py to support "Get". This time, the 'mover' field is not removed from the ticket before a ticket is resent to the LM.
First commit of encp after starting modification for use in the "get" program for SDSS and KTEV. All pre-get functionality should continue to work as it did before.
Slight modification to the EncpInterface file consitancy check. When reading if the directory two (or more) levels up did not exist, then a cryptic ENODATA error was given to the user. Now it is "No such file or directory".
Fixed a permission bug. A number of ways could trigger it, but the original was a regular user trying to write into a directory owned by root. In this case the user got an error of ENODATA. All changes were in e_access().
Added accounting changes to record the new rates.
new fields in encp_xfer
Modified the dcache write error when the file has already been written to reflect that the file already exists. Added the new (and old) rates to the log file message in such a way that does not break the inquisitor (hopefully).
Volume assert was broken. Is fixed now.
Fixed a bug where the target (or possibly tag) does not exist, because the directory it was supposed to be in does not exist. The bug was that it should have reported the diretory as being missing but was incorrectly stating the target.
Included to modifications to error/warning messages. The log message, "unable to register bfid" has been removed. The "Paranoid CRC mismatch" message has been changed to "CRC readback mismatch".
These changes have the encp wait for the mover to send one byte on the data socket when writing. This gives encp a chance to know when the mover is done mounting and positioning the tape on writes. This gives encp a chance at getting more acurate rates.
Fixed a bug where the wrong error message was given on writes and the file does not exist. The code was trying to test a non-pnfs file as a pnfs file and was then giving the wrong error message.
Modified encp.py and pnfs.py to agree on what a pnfs directory is. Previously, encp would consider any directory named /pnfs to be a pnfs directory. The "enstore pnfs" commands were more stringent and required /pnfs to be the very first directory in the path. This current fix is based on the looser encp version. In addition to just checking the pathname, a check for a pnfs specific "hidden" file is also performed.
Modifed the final message that encp gives (with verbosity) to include all of the rates calculated (overall, transfer, network, tape/drive and disk).
Fixed some rate calculation and reporting problems/inconsistancies. There are now 5 rates: network, tape, disk, overall and transfer.
bumping version to v2_20_b because of encpCut
Cleanup of the last sys.exc_info()/tb resource leak.
Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference.
Removed a debugging print statement.
Added some timing diagnostics to the verbose equal to level 9 output. This mostly includes the time it takes to set the file metadata on writes.
Modified encp to use host_config.update_cached_config() to force encp to use updated routes on machines where/when the enstore.conf file changes.
Added a check for layer 1 and layer 4 being empty on writes from dcache. Before this check it appears the dcache occasionaly could write the same file twice creating two (nearly) identical files in enstore. Now the second attempt will fail.
On reads, encp should now try all three paths to a file in pnfs (/pnfs/xyz, /pnfs/fs/usr/xyz and /pnfs/fnal.gov/usr/xyz).
Fixed one problem with the CRC data access value. Most changes though, deal with adding the --override-ro-mount option.
Added the crc value to the data access layer output.
On a multi-file read, the tape with the last set of files would 'win' in determining the exit status of the entire encp process. This has been fixed. This did not effect single read requests, multifile reads involving a single tape or writes. If the routing socket udp_server object failed to reaquire the same port when reinitializing; the code used to error out. Now on this 'error' it will reaquire an unused port. This may cause some 'Connection timed out' errors from the mover, but that is life. Lastly, some underpinnings for printing the CRC in the data_access_layer have been included. However, this information is not used yet for anything.
bumping version to v2_20_a because of encpCut
When encp stopped using interface.py and started using option.py the switches --ephemeral and --file-family were accidentally dropped.
Encp will no longer raise alarms if non-critical server are not accessible.
Fixed an error handling bug when generating an error message. The error was a server replying with OK, but not returning everything it should with a quit message.
Include conditional import for fcntl functionality. The FCNTL module is depricated in 2.2.
Added accounting server to list of servers to check when encp first starts.
Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv.
For multi-file-transfers when writing, the delayed dismount for the first (n - 1) transfers will be at least 3 minutes. The last transfer will continue to use the default or user specified value.
Cleaned up some of the direct i/o code.
Fixed a bug regarding --ecrc and the permissions needed for the original file. It should have only required readable not read/writable.
1) The message at the beginning of the program of the tags would give incorect values if the output specified was the directory of the target and not the target itself. 2) If the alarm or log servers are not running, a warning should appear, but the code should not abort the transfer. Previous to the fix it would abort.
Fixed a bug that caused the real error reported from the mover to be ignored. Also, fixed an error message spelling error.
These changes are in advance of encp v3. They are for future files to use them based on the user being an admin or user.
Fixed a bug that prevented --threaded from working with the new option.py handling of command line parsing.
Includes changes in the paranoid ECRC message. Supports e-mailable alarms. Switched over to option.py from interface.py for interpreting the command line arguments.
feeding information to accounting server
Include support for --ecrc in encp. This option (for reads) means that after the transfer is complete encp will run an ecrc check back over the file.
Handle mover timeouts better internaly. Set the error to be RESUBMITTING instead of ETIMEDOUT when no mover info is known in handle_retries().
Fix 'unique_id' traceback error on retry/resubmit.
Bug fix. There is the possibility that some elements may not exist do to errors. This fix tests for the elements existance first.
Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types.
Uses the new CRC_ENCP_ERROR instead of CRC_ERROR.
Bug fixes: 1) open_control_socket() will now check the control socket after reading the inital ticket for possible socket closed by mover case. 2) handle_retries() bug fixed that could have resulted in "mover imposter" error messages. A copy of a dictionary item was returned rather than a reference to it. 3) Slow rate errors from EXfer (from the read()/write() calls hanging in the kernel to long) are now handled specially/correctly.
Fixed the os.getcwd() vulnerability. Added the hostname to the cwd output when verbosity is turned on.
Handles the socket.socket() errors now. Tickets returned by the mover are checked for consitancy (which is being very parinoid). The gids were being incorrectly listed at the beginning of the transfer.
When an error occurs on reads the output file is truncated. Also, there are now is_read and is_write functions for use in functions used in reads and writes that return true if the transfer(s) is/are reads or writes accordingly.
Minor fix for a bug that prevented writing to /dev/null under some conditions.
Small bug fix. Wrong value types were specified into some print_data_access_layer_format() calls.
Moved where the $ENSTORE_SPECIAL_LIB environmental variable is processed on reads. Code moved from submit_one_request() to create_read_requests().
Added --shortcut switch. This is used in conjunction with the --get-cache switch to bypass full file path extrapolation. This avoids linear searches of all pnfs ids for each component in the full file path. This works similar to the --get-bfid switch. If the --shortcut switch is not given then the original --get-cache functionality remains unchanged. This change has no effect on the --put-chache switch.
Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp.
use environment variable to specify LM for read requests
Fixed how the file-family-wrapper was read in at the begining for verbose output. It was always returning as "Unknown" because there was an extra set of () trying to call the integer returned as a function.
Modified to call verify_read/write_request_consistancy() after receiving callback ticket from mover.
Included the current working directory in the verbose and log output.
Modified mover_handshake to return complete tickets. This should fix the "Error processing retry of " errors. Also, modified the code to be more resilant against 'problem' tickets when processing them in handle_retries(). Specificaly, with the 'resubmits' key.
Encp checks to see if all critical servers are running at the begging of the program. Also, encp now handles the "KeyError: system_inhibit" traceback error.
Fixed the format of one of the new verbose output lines.
Added gid/group and uid/user_name info to log and terminal output. Modified the request verify functions incase the database contains incomplete data. On writes, the tag info is included in the verbose output.
Fixed a bug so the previous version passes my (current) set of regression tests. The bug was that mmap io needs a file opend for read and write. If the file only has read permissions encp will fall back to posix io.
These files (encp.py, interface.py, EXfer.c) allow for encp to accept which EXfer optimizations to use from the command line.
Encp now passes more paramaters to EXfer. These allow for encp to optionally select which optimizations: threaded, direct io or memory mapped io; to use. Requires Exfer.c 1.71 to work.
Reuse the address and port when reinitalizing the routing socket on retry/resubmit.
Fixed "Linix" vs. "Linux" bug for direct io. Better error handling when direct io no implemented.
Undue previous commit.
Put in hack to handle calculating write rates when very large buffer sizes are used.
Modified encp to use O_DIRECT on xfs on linix.
========== enstore.py ====================================================================================
for Estart and Estop call enstore start or stop correspondingly
make rip9 and rip10 not a special case
made 1.65 current
made 1.65 current
Uses new Estart and Estop. Removed the dbs command.
Uses new start and stop functionality. $ENSTORE_DIR/src/enstore_start.py $ENSTORE_DIR/src/enstore_stop.py $ENSTORE_DIR/src/enstore_restart.py
add a enstore command support for event relay
========== enstore_alarm_search_cgi.py ====================================================================================
type bug fix
support yesterday as search time
========== enstore_constants.py ====================================================================================
add support for using remedy_api
add accounting server and drivestat server to servers page
add drivestat
Included new constants for the accounting server and monitor server to be started and stopped. Additional constants were created for support of the ability to e-mail specified alarms.
add support for no debug log
add no_max_log_file_size value
use encp_ip when available on encp history page
========== enstore_files.py ====================================================================================
obtain encp plot info in a more scalable way
try to fix problem of plotting on cdf, make code more scalable
add more debugging info
add more debugging info
add other links to saag page
use encp_ip when available on encp history page
reverse order of output to status only file
fix getting lm info when lm is timing out
attempt to fix problem wher lm is alive but then times out from getting additional information
========== enstore_html.py ====================================================================================
support using remedy_api
fix a spelling error
change enstore to mass storage
add other links to saag page
remove network link from saag page
make navigation bar optional
fix getting lm info when lm is timing out
attempt to fix problem wher lm is alive but then times out from getting additional information
========== enstore_log_file_search_cgi.py ====================================================================================
type bug fix
support yesterday as search time
========== enstore_make_plot_page.py ====================================================================================
add support for the quota plot page
========== enstore_saag.py ====================================================================================
add other links to saag page
========== enstore_up_down.py ====================================================================================
add a test alarm for auto ticket generation
remove traceback
server.movers is a list
do not turn lm yellow if we are watching no movers
Modified some literal strings to use strings from enstore_constants.
========== entv.py ====================================================================================
Fixed a problem when a mover belonged to multiple libraries. Phantom spaces for movers were in the display for these duplicate movers. This was fixed by forcing uniques on the list of names.
Bug fix for previous entry. When the reupdate occurs an exception occured.
Fixed the problem where entv forgot about the mover outline colors that movers from a library manager share.
Added some dels to prevent eronious warning messages being printed out to the terminal with python 2.2.
Removed references to fcntl module since it wasn't being used.
Fix some resource leaks from "exc, msg, tb = sys.exc_info()" code. Also, uncomment the unsubscribe line for the event relay messages.
Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv.
Fixed the repositioning of the window. Also, another attempt to fix the resource leaks.
Added some missing exceptions that should have been caught.
Major modifications toword the removal of resource leaks. There were two leaks fixed (a third was from generic_client) that stemmed from cyclic refereneces. Part of this change was to stop having entv exec itself on certain errors. Other fixes include removing the rate, buffer bar and progress bar from the display when the mover moves into IDLE, ERROR, OFFLINE or Unknown state. Also, most (but not all) blanket excepts now only catch specific exceptions. Most remaing blanket excepts catch KeyboardInterupt first and re-raise.
Modified entv.py to stop creating resource leaks. This included explicitly creating a master frame for the Canvas. Entv reinitializations under any situations no longer exec itself (which doesn't close open files.)
Modified to process event relay client errors.
If the .entvrc file was missing entv would through a traceback. It now correctly skips this and uses the defaults.
Modifications to pass mylint.py.
Fixed potential race condition if multiple entvs try to write out .entvrc at once.
Added code to color code groups of clients and movers.
Improved error handling when reading in .entvrc file. Also, added functionality to remember if animation is on or off in the .entvrc file.
Forgot to save one last change from previous commit.
Fixed a bug if csc.get_movers() fails.
'Canned' version will now correctly loop back to the beginning of the commands file after it reaches the end.
Bug fix for canned version. It will not try to resubscribe to a non- existant event relay when in 'canned' mode.
Change to slow down the handle_message() thread when running canned version of entv.
Added the ability for entv to read in a command file for running in canned mode. Entv now has a --commands-file similar to the --movers-file.
Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly.
Entv now uses interface.py for command line prarsing. The switch --dont-show allows the user to hide all movers belonging to a library manager.
========== enstore_functions.py ====================================================================================
Added new function get_dict_from_config_file(). It is similar to get_from_config_file(), but the new function returns the entire dictionary.
Added function get_enstore_tmp_dir() to return the location of temporary enstore files.
========== event_relay_messages.py ====================================================================================
add a enstore command support for event relay
========== ftt_driver.py ====================================================================================
fixed problem with ftt returning None and mover tracing back when tape has no label.
fix a potential memory leak
Fixed a bug when causing incorrect mover behaviour in case of the failure while labeling tapes.
added a log message
check for extended vol header
remove pdb import
These files are partially modifed to address lingering cern wrapper bugs.
added new method
========== inquisitor.py ====================================================================================
add accounting server and drivestat server to servers page
add more debugging info
add more debugging info
moved tmp files to tmp area
add migrated to VOLUME_STATES
typo
check other_saag_links in config file when marking things down
fix getting lm info when lm is timing out
attempt to fix problem wher lm is alive but then times out from getting additional information
init a variable
check for periodic tasks when system is backed up
========== inquisitor_plots.py ====================================================================================
reset acc_db after closing db connection
do not open db unless need to
remove enstore_date
fix query string
merge in acc_temp file
make mount plot from accounting db info
obtain encp plot info in a more scalable way
========== interface.py ====================================================================================
Include support for --ecrc in encp. This option (for reads) means that after the transfer is complete encp will run an ecrc check back over the file.
Added --shortcut switch. This is used in conjunction with the --get-cache switch to bypass full file path extrapolation. This avoids linear searches of all pnfs ids for each component in the full file path. This works similar to the --get-bfid switch. If the --shortcut switch is not given then the original --get-cache functionality remains unchanged. This change has no effect on the --put-chache switch.
Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp.
These files (encp.py, interface.py, EXfer.c) allow for encp to accept which EXfer optimizations to use from the command line.
========== inventory.py ====================================================================================
fix the wrong referenc
fix the wrong size
bytes used does not depend on capacity any more
fix sys.exec_info()
change the semantics of blank
fix a typo
remove html tag for plain VOLUMES_DEFINED
remove html tag for plain VOLUMES_DEFINED
simply alarm message for too many mounts
change too alarm client
generate alarm when tape mount exceeds limit
remove volume_clerk
add VOLUMES_TOO_MANY_MOUNTS
fix a typo again
change format of volumes_defined again
change format of volumes_defined
fix a typo again
fix a typo
add mounts to volume_defined
ignore missing key due to live backup
add number of files to volume_size
handle last_access more accurately for recycled volumes
========== libtpshelve.py ====================================================================================
do not assign trace back of exc_info() to local variable
========== alarm_client.py ====================================================================================
support using remedy_api
add init of message
text as text
allow sending message in alarm client
Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor".
added client name option
========== monitored_server.py ====================================================================================
add accounting server and drivestat server to servers page
========== mover.py ====================================================================================
fixed bug in a function call
added low level diagnostics for FTT_EBLANK
fixed incorrectly formatted eod_cookie
mods for multiple LMs
shuffle libraries in the idle state to implement a fair selection of LM for multiple LMs
send a correct state in update_lm
Fixed/modified the code for multiple LMs serving one mover.
fixed bug causing mover to not error out if too long in ATIVE state reading tape
fixed update_stats
fixed update_stats
fixed update_stats
fixed update_stats
put back Michaels changes that were accidentally erased
trace to fix update_stats
trace to fix update_stats
These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python.
This is a patch to fix the previous patch. This one fixes the disk mover. The mover will now send a byte to the encp before it will start reading bytes from the network on writes into enstore.
add trigger message for disk mover
fixed problem with ftt returning None and mover tracing back when tape has no label.
send message on the data socket when tape write starts for encp to calculate network rate
replace trace with log to track why mover does not error out when read tape has finished and the client does not receive data for a long time
adjust types of the stats to match with the DB definition
convert write protection to int
modified a log message
added write protection to stats
added trace
added trace
Catch an exception when ftt fails to read stats due to the hardware problem. In fact some data is present and some not. Use low leve diagnostics to identify the problem
added log messages to figure out why mover gets stuck in the dismount_wait
1. If stats file path does not exist. Create it. It could have been deleted by tmpwatch. 2. If there is any error rading the volume label set volume to NOACCESS
make mover to not send alarm "Too long in state..." while reading tape when data is read but network transfer has not completed because of quite slow transfer rates
attempt to fix avery rare situation when some variables get modified in one thread while they are being looked at in another thread. Also included MOUNT_WAIT state to generate alarms when too long in state
update LM 10 time less frequently in the OFFLINE state
if in OFFLINE send error message to LM just once
set mover offline after forced dismount if eject failed
typo fixed
added some log messages
addded drivestat
Fixed a bug when causing incorrect mover behaviour in case of the failure while labeling tapes.
introduction of new state FINISH_WRITING has prevented the mover to go offline immediately after writing a file. Now it is fixed
fixed (must have) false alarm when writing tape
added trace to catch a bug
added trace to catch a bug
recover from error state if eventually mover gets out of a very long seek
fixed a 'typo' bug
removed test timeout
fix problem with tape being set to naccess and not being considered by library manager
check transfer rates and disconnect from encp if it is slow
1. Modifed format of low level diagn. message 2. Change in response to AML2 MC mount failure: go broke
make last_error a 2 tuple
change log to trace
made sending statistics to the central db server configurable, by default do not send
some more log messages added
added log message and diagnostics to track tape writing and reading
wait forever if vc does not respond to set_remaining_bytes in state FINISH_WRITE and generate alarm
removed test statement
1. Corrected transition to error state. 2. Made transfer_failed more clear 3. Changed crc error alarm message and error source.
handle mount error more gracefuly
fixed accounting calls
added accounting calls
close open sockets if connection failed
Made mover - encp connection timeout and number of retries configurable.
added error processing for mount_volume, corrected some errors
1. Configurable restart on error. If restart_on_error key is set in the mover configuration mover will restart. Otherwise not. 2. Fixed (I think) processing of certain FTT errors in transfer_failed. 3. Improved change of the state in transfer_error 4. Modified offline() to make it thread compliant.
reordered some lines in transfer_failed
changed watch_syslog
some fix to start
some fix to start
some fix to start
some fixe to start
1. Added a low level diagnostics 2. Offline mover if FTT_EBUSY caused a transfer failure 3. do not restart mover if it is already running. This applies to a case when mover process could not complete because it was in the kernel (D state).
set mover offline if consecutive_failures exceeded its threshold and error_times exceeded its threshold. This is to fix a problem when the mover was unable to mount a tape due to drive busy and caused a lot of tapes to be set to NOACCESS.
fixed volume family for volume assert
restart mover if it too long in the ERROR state
extend NULL in the null mover served path to anything after NULL
fixed a bug
always read a block size number of bytes to fix a firmware bug for some tape drives that return not numbre of bytes requested but a many as left to the file mark
removed sleep
use drivestat only if needed
use drivestat only if needed
set volume to noaccess and mover in error state if mover is stuck in some intermediate state
added mount counter to vc and corresponding method
change rate report for read request
modified Trace.notify messages for entv/ratekeeper.
modified calculation of the tape transfer rate
reverse name change for transfer_time and intrroduce drive_transfer_time
measure time for the tape and overall transfer separately and return tape time as transfer_time for the backward compatibility. Replace old transfer_time with total_tranfser_time
========== mover_client.py ====================================================================================
changed TO and retry pars.
========== net_driver.py ====================================================================================
I know it is silly but mover gets float div. exceptions, and I do not understand where from
========== null_driver.py ====================================================================================
added diagnistics to close method
added actve_time
========== null_wrapper.py ====================================================================================
These files are partially modifed to address lingering cern wrapper bugs.
========== plotter.py ====================================================================================
make mount plot from accounting db info
========== priority_selector.py ====================================================================================
increase max pri to 1000001
do not set adm pri unless it is already set or encp_daq is enabled
========== setpath.py ====================================================================================
Added \n to the end of the error strings printed.
Fixed the os.getcwd() vulnerability.
========== verify_db.py ====================================================================================
fix sys.exec_info()
change the error message
========== volume_clerk.py ====================================================================================
do not assign trace back of exc_info() to local variable
add quit()
add removing 'mounts' to check_record()
fix wrong filed 'mounts'
fix sg-count bug -- do not count deleted volumes
add sum_mounts at addvol
guard against None volume at update_counts()
cope with migrated state
add --migrated
keep sg count even when quota is not enforced
build sgdb at start up if it was not there
new sgpg related changes
temporarily fix the missing mounts problem
fix mounts missing key -- backward compatible
add 'mounts' to newly created vol
add --show-quota to volume
add default event processing routine
fixed a bug
fixed a bug
add event relay subscription to changed configuration
set initail value correctly
added mount counter to vc and corresponding method
add check_record() to trim obsolete fields
clean up extra fields when recycling
========== pnfs.py ====================================================================================
Encp will now record the crc in layer 4 of the file. All internal encp checks have been updated to handle this. Also, all internal changes to pnfs.py have been made to handle this extra piece of data.
pnfs.py now uses fullpath() from enstore_functions2.py.
Fixed is_pnfs_path() to better handle automounting of pnfs directories AND handle as a local directory those with "pnfs" occuring in a path that is not a pnfs file (i.e. /data1/bja/pnfs/cdfen/filesets/GI/GI10/GI1073/GI1073.1).
Various fixes for "get" to work. Minor changes were made for the encp functionality used by get.
Modified is_pnfs_path() to OPTIONALY only check if the name is in the /pnfs area. Previously it would always return false when the target did not exist which was resulting in some incorrect/confusing error messages.
Modified encp.py and pnfs.py to agree on what a pnfs directory is. Previously, encp would consider any directory named /pnfs to be a pnfs directory. The "enstore pnfs" commands were more stringent and required /pnfs to be the very first directory in the path. This current fix is based on the looser encp version. In addition to just checking the pathname, a check for a pnfs specific "hidden" file is also performed.
Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference.
Removed the --pnfs-state and --enstore-state options.
Fix for --tagchown to accept uid and gid. It always should have accepted both ids and names but that was not the case.
Includes fix for --tags and --tag when a tag contains no/bad data.
Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types.
Two bug fixes. One, relative paths w.r.t. tags are now respected. Two, empty tags will no longer result in IndexErrors being raised. They are now caught and re-raised as IOErrors.
Fixed the os.getcwd() vulnerability.
Removed the pstat_decode messages that are put in the log file. Also, did a little code cleanup.
Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp.
Added legal charset test when setting a tag.
Allow the --size option to take a long integer instead of just an integer.
========== option.py ====================================================================================
--volume for encp was added. (This is for early get tests.)
remove remedy_ticket option
add support for remedy tickets
Minor modifications for newer (test) version of pychecker.
Added the encp option --override-ro-mount.
do not assign trace back of exc_info() to local variable
add --pvols [--just]
Fixed some --help related items. Long text lines getting printed instead of cut off and wrapped correctly. Added a help string for --usage. Removed the = from the --help info. If the short options list is long, it gets substituted with "[OPTIONS]..." for usage output.
allow sending message in alarm client
add --keep-vol
Quick mod. to get around termios vs. TERMIOS problem.
Modified option.py to work under python 2.0 and 2.2 without printing a warning message in python 2.2.
Included new switches for encp. This is the first attempt to move encp to use option.py. Also, the new --all for enstore start/stop.
added client name option
Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types.
add --migrated
new sgpg related changes
add GVOL
add support for the quota plot page
Added the ability for entv to read in a command file for running in canned mode. Entv now has a --commands-file similar to the --movers-file.
Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly.
Include options for entv. The option is --dont-show. Note: the cvs log for entv.py accidentally refered to the old interface.py file instead of the newer option.py file.
add --show-quota to volume
Added --just option for start/stop. Added long as a type for options.
add --trim-obsolete
========== ratekeeper_client.py ====================================================================================
A fix was made to prevent a traceback from occuring when the user executes "enstore ratekeeper". In line with other clients it now performs and implicit --help.
Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor".
========== host_config.py ====================================================================================
Slight fix to test if the specified enstore.conf file exists AND is a regular file. Note: it is still bad to tell encp to use a file as its enstore.conf file that is not a real enstore.conf file.
Added a function update_cached_config() that rereads the enstore.conf file and updates the global cached information.
There was a bug were a local variable could be accessed before having been set due to socket error. Said variable is now given a default value.
Put in a some time delayed waits in case the DNS server is down for restart.
More straitforward handling of errors raised if popen failes. Attempt to get the FQDN while determining the default ip address.
========== monitor_client.py ====================================================================================
These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python.
Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference.
Handle a possible exception better.
add debugging info after a keyerror
Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor".
Fixed the socket.socket() vulernability.
========== volume_assert.py ====================================================================================
Modified the request timeout time for the volume clerk query.
One more option vs generic_client cleanup.
Have the Interface class inherit from generic_client instead of option.
Include timeouts incase the configuration server does not respond.
Volume assert was broken. Is fixed now.
Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference.
Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types.
========== generic_client.py ====================================================================================
This is another attempt at fixing the generic_client resource leak problems. There is also a modification to change a print statement to a sys.stderr.write() call.
use timeouts properly
Second half of previous entry.
Second attempt at removing the cyclic reference if the generic client is also a configuration client.
back out
Removed a resource leak. If the generic_client was also a configuration client, a cyclic reference was created causing the reference leak.
Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor".
These modifications are in support of the new enstore_start and enstore_stop. Mostly having to do with the event relay dealing with 'alive' and 'quit' messages.
========== log_trans_fail.py ====================================================================================
modificaions needed to process new sys log message
include low level tape diagn
========== monitor_server.py ====================================================================================
These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python.
Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference.
Fixed the socket.socket() vulernability.
========== event_relay.py ====================================================================================
add a enstore command support for event relay
========== enstore_show_inventory_cgi.py ====================================================================================
add COMPLETE_FILE_LISTING
add VOLUMES_TOO_MANY_MOUNTS
change VOLUMES_DEFINED to VOLUMES_DEFINED.html
========== file_clerk.py ====================================================================================
do not assign trace back of exc_info() to local variable
add quit()
add FILE_DB_USE_INDEX to switch on index file if there is none
========== library_manager_client.py ====================================================================================
Modified these clients to print the full --help output when no options are given. (i.e. "enstore library") It used to print just the shorter usage help output. This is to have all the server clients conform to a single look and feel.
fixed interface probelm for delete option
remove enstore_functions by using e_errors.is_ok()
changed TO and retry pars.
Fixed the printing of the active asserts heading. Before any work at movers would cause it to print, now just active asserts to so.
========== enstore_display.py ====================================================================================
Fixed a problem when using python 2_2_3. A syntax error would show up. Also, a color problem was fixed when a mover belongs to multiple libraries.
Fixed some discrepencies when moving between states. This involved removing some displayed objects when a message to do so was dropped.
Fixed some resource leaks w.r.t. "exc, msg, tb = sys.exc_info(). Also, modified some of the "master" variable code for python 2.2. Added the states 'HAVE_BOUND', 'FINISH_WRITE' and 'CLEANING' to the list of states to remove connections, rates, progress bars, etc. Fixed the problem of states sometimes being displayed with full text instead of being truncated to fit the space.
Modified to support three columns of movers. The connection line now weaves through correctly from the third column.
Added 'Unknown' to list of states to include for automatic client disconnections.
Fixed the repositioning of the window. Also, another attempt to fix the resource leaks.
Added some missing exceptions that should have been caught.
Major modifications toword the removal of resource leaks. There were two leaks fixed (a third was from generic_client) that stemmed from cyclic refereneces. Part of this change was to stop having entv exec itself on certain errors. Other fixes include removing the rate, buffer bar and progress bar from the display when the mover moves into IDLE, ERROR, OFFLINE or Unknown state. Also, most (but not all) blanket excepts now only catch specific exceptions. Most remaing blanket excepts catch KeyboardInterupt first and re-raise.
Previous commit broke displaying the connection lines in different colors.
Modified entv.py to stop creating resource leaks. This included explicitly creating a master frame for the Canvas. Entv reinitializations under any situations no longer exec itself (which doesn't close open files.) Also, modified to handle the new mover state FINISH_WRITE correctly. Previously, FINISH_WRITE and the rate could have been displayed on to of each other. One other thing is that the second collumn's last mover fits in the window. Half was being "drawn" becasuse it is lower than the first column. It was this first collumn where the spacial calculations were done, now they take into account that the second collumn is lower.
Cleaned up the code that locates the tcl/tk libraries.
Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types.
If the mover is in "ACTIVE", "SEEK", "SETUP", "loaded", "MOUNT_WAIT", or "DISMOUNT_WAIT" to long the timer color will turn red.
Fixed the os.getcwd() vulnerability.
Modification to bring it in sync with entv.py.
Bug fix when the domain name server is not available. A variable would be accessed but because of the lookup failure was never created.
Hopefully fixed a problem with "host not found" errors.
Added fit_string(). It will truncate strings to fit into the provided space. It is used on the state and volume names.
First attempt to truncate strings to fit.
Now the connection lines match the outline of the clients. Also, the mouse clicking on a connection will invert the color.
Added code to color code groups of clients and movers.
Improved error handling when reading in .entvrc file. Also, added functionality to remember if animation is on or off in the .entvrc file.
Adjust the width of the mover rectangles based on the number of columns.
Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly.
Entv now uses interface.py for command line prarsing. The switch --dont-show allows the user to hide all movers belonging to a library manager.
Reinitalize the display every so often.
If the width is very small, then the image for the active state is suppressed. I consider the rate information to be more informative in this case.
Made changes to the positioning calulation of things in the mover section. Fixed relocation bugs of the differenct percentage bars. Fixed the "How's on top?" problem between the media and network bars. Fixed the un-initialization problem of the volume label font size.
Added an instantaneous rate display. Also added another bar that indicated percent on mover buffer filled. For the progress bar, two progresses are tracked one for network and one for media.
========== udp_client.py ====================================================================================
Cleaned up some code like the following: exc, msg, tb = sys.exc_info() to look like the following: exc, msg = sys.exc_info()[:2] This should help to avoid resource leaks.
Made a fix such that it would send all "done_cleanup" messages before closing the socket. Previously it would close the socket after sending just one.
Modification to return better error message for misconfigured /etc/hosts file. This seems to happen with default 7.3 fermi linux installs.
========== event_relay_client.py ====================================================================================
protect from socket.socket and bind errors
These modifications are in support of the new enstore_start and enstore_stop. Mostly having to do with the event relay dealing with 'alive' and 'quit' messages.
dont do sys.exit
add a enstore command support for event relay
========== configuration_server.py ====================================================================================
remove 2 traceback leak problems
Fixed a bug when attempting to build a log message. The status was not wrapped in (status,) format (since status is already a tuple).
Fixed a bug preventing a reload of the configuration file.
Modified the configuration server to use the event relay client to help manage socket.socket() volnerability.
Fixed problem in dump(). Trace.trace() call was modified and was given to many arguments. Now the correct number exist.
removed annoying pprint
========== udp_server.py ====================================================================================
Modified some Trace.trace levels. Also, added a Trace.trace() to the reply_to_caller_using_interface_ip() function.
These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python.
If the socket() call failed, do not attempt to use it in the fcntl() call.
Include conditional import for fcntl functionality. The FCNTL module is depricated in 2.2.
========== ratekeeper.py ====================================================================================
Modified how tracebacks are handled in the mainloop. This stemmed from fixing a sys.exc_info()/tb resource leak fix. Now Trace.handle_error() and rk.serve_forever_error() are called.
Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference.
Ratekeeper now uses the event_relay_client for handling event_relay related handling of the socket.socket() vulnerability.
========== configuration_client.py ====================================================================================
removed a print statement
add debugging for network malaise
remove redundancy in check
Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor".
dump() didn't catch a possible TCP_EXCEPTION from callback.
========== media_changer_test.py ====================================================================================
fixed convert_version
These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python.
========== super_remove.py ====================================================================================
add --keep-vol
========== enstore_status.py ====================================================================================
ignore alarm lines
weed out non- ENCP_XFER messages
do not process debug messages
print out encp errors too on history page
use encp_ip when available on encp history page
set disk_rate
support new encp log file line format
========== cpio_odc_wrapper.py ====================================================================================
These files are partially modifed to address lingering cern wrapper bugs.
========== cern_wrapper.py ====================================================================================
These files are partially modifed to address lingering cern wrapper bugs.
remove spurious print
========== file_clerk_client.py ====================================================================================
handle a strange error
Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor".
honor alive timeout and retry from command line
========== log_server.py ====================================================================================
do not output misc messages to the extra log files
fix bug in startup of log server
fix extra files when not asked for
fix bugs when starting
add support for no debug log
skip file if too big at beginning
fix orig save logfile name
fix restarting of log file
optionally close log file at certain size
========== disk_driver.py ====================================================================================
fixed to comply with new mover reqs
========== enstore_functions2.py ====================================================================================
Fixed fullpath().
Moved the fullpath function from encp.py (and pnfs.py) to this file.
add support for actions in alarm server
========== show_volume_cgi.py ====================================================================================
make pychecker happy
make pychecker happy
fix minor bugs
using --gvol instead
========== checkBackedUpDatabases.py ====================================================================================
better way to determine if main backup is done
========== makeplot.py ====================================================================================
A full disk could cause this to traceback. Not it errors out gracefully.
Fixed a bug that caused it to include to many days worth of files in the plot than it is supposed to.
make plot backgrounds grey
remove -modulate for new convert
========== log_client.py ====================================================================================
attempt to fix start/stop from unnecessary instantiation of lots of clients and delays cusing them to waste about a minute before starting doing anything
Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor".
added client name option
support yesterday as search time
========== scanfiles.py ====================================================================================
reduce unnecessary stat
handle files younger than 1 hour
try back door to gain permission
exclude .B_* and .A_*
========== media_changer.py ====================================================================================
These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python.
diagnostics added
be aware of 9940B media. Added trace
ignore Enter
fixed a bug in cleanCycle of stk class
fixed a bug in cleanCycle of stk class
added clean method to stk class
add migrated state
========== volume_clerk_client.py ====================================================================================
fix a typo
add --pvols [--just]
allow L2 suffix for volume labels
Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor".
honor alive timeout and retry from command line
tunning timeout and retry
tunning timeout and retry
tuning the timeout and retry
add --migrated
new sgpg related changes
fix previous log message which is supposed to be: fixing missin si_time
volume_clerk_client.py
add --gvol
add timeout for self.send
fix --clear
add --show-quota to volume
added mount counter to vc and corresponding method
add --trim-obsolete
========== enstore_overall_status.py ====================================================================================
change enstore to mass storage
========== media_changer_client.py ====================================================================================
Modified these clients to print the full --help output when no options are given. (i.e. "enstore library") It used to print just the shorter usage help output. This is to have all the server clients conform to a single look and feel.
========== get_total_bytes_counter.py ====================================================================================
add new libraries
display TB on system pages
transfer number as bytes not tb
========== get_all_bytes_counter.py ====================================================================================
display TB on system pages
output in correct format
strip off newline
transfer number as bytes not tb
change name of bytes file
make a file with total bytes as an integer for sc2002.
========== enstore_show_inv_summary_cgi.py ====================================================================================
add COMPLETE_FILE_LISTING
add VOLUMES_TOO_MANY_MOUNTS
change VOLUMES_DEFINED to VOLUMES_DEFINED.html
========== library_manager.py ====================================================================================
do nothing in bad_volume. This is taken care of in mover_error
changed version comparison
log errror message
fixed convert_version
fixed convert_version
code changes to address tape ingest issues
code changes to address tape ingest issues
do not set volume to noaccess while processing requests. This must be done in the error processing
attempt to identify unpack non-sequence type error in bad_volume call
changes in mover_error to process FTT_EBLANK error
some mods for ktev request porcessing
allow setting max_suspect_movers and max_suspect_volumes in the configuration file
modified mover_busy to clean up at movers list for multiple LM per mover
mods for multiple LMs
Fixed /modified the code for multiple LMs serving one mover.
These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python.
fixed bug in delete_mover
5. Found why the requests may not go to movers even if the file family width allows. At least one case is understood.If the mover fails to mount tape it sends an error message to the library manager and then goes to the idle state. In the error message the mover,volume pair is mover,current_volume. In the idle message this pair is mover,previous_volume. When removing the entry from volumes_at_movers queue LM looks up for the pair mover,volume and it may not get removed because of different combinations described in the mover message. Now this is fixed.
modified get_work_at_movers to search for pairs: label, mover. Before it was searching only for the label. This was wrong. Consider the following. Mover1 fails on VOL1. It send error rq and lm removes VOL1 from work at movers. Suppose VOL1 was dismounted and is available. Mover2 gets the VOL1 and now it is again in work at movers. But the error message from mover1 comes and removes this work from work at movers.
trace added
some minor changes
some more logging
replace 33 with 31 in trace
replaced 13 with 33 in trace. 13 is used in config client and inteferes
fixed bug
added some diagnostics
create path for a lock file if does not exist
remove double entry
fix problem with tape being set to naccess and not being considered by library manager
added retrn value checks for vol_info
add migrated state
fix bug in mover_bound_volume causing LM to not update volumes_at_movers for HIPRI requests, which in turn causes a wrong calculation of busy volumes
do not allow any, hence hipr rq. go if it is not already in the list of active tapes
fix ambiguous logic in busy_volumes
changed trace level for assigning write rqs
LM with new queue manager
added trace
added some traces changed get in manage queue
removed some traces to speedup the server
added trace next_work_any_volume
added traces to process_read_request
some more mods for next_work_this_volume
some more mods for next_work_this_volume
modified next_work_this_volume to chech for restricted host access
========== generic_server.py ====================================================================================
Modified the generic server to avoid a sys.exc_info()/tb resource leak. Another still remains.
========== enstore_plots.py ====================================================================================
do not write a plot if there is no data
make mount plot from accounting db info
make plot backgrounds grey
remove -modulate for new convert
translate a / to a - in drive_id
========== manage_queue.py ====================================================================================
removed the remnants of the old adm_queue, and fixed the self.admin_queue.find call
increase max pri to 1000001
removed some heavy diagnistics
LM with new queue manager
bakked off changes
cnaged trace
modified get and get_next
added some traces
added some traces
added some traces
added some traces
added some traces
added some traces
added some traces changed get in manage queue
bakked off the change
removed some traces to speedup the server
fixed a typo
When request qeue contains admin and not admin requests, there was incorrect processing of ordinary requests skipping the 1st one in the queue. Now it is fixed in Request_Queue.get
========== dispatching_worker.py ====================================================================================
fix the annoying fd is not integer problem
add debugging information
If the UDPServer client failed to initialize (specifically to obtain its udp socket) then exit the server.
Added a comment about a line of code.