ENCP release notes, from v3_3 to v3_4

Encp changes:
=============

Added the switch --check.  When this option is used, encp goes through all
of the pre-transfer setup, but does not actually perform the transfer.  If an
error is found, it is reported like a normal encp error.  If the user is
returned success, it does not guarantee that re-running the encp without
--check will also be without errors.

A number of old metadata 'formats' were discovered that encp was not
able to handle.  They are now accepted.  One format, was that all bfid 
brands by convention should be 4 bytes but some were found to be 3 
bytes long.  The other was that some bfids have an "L" appended to them.

Transfer failures are now sent to the accounting database along with
the successes.

New error, handle CRC_DCACHE_MISMATCH, occurs when the crc the dCache
puts into layer 2 does not match the CRC calculated while attempting to
write the file into Enstore.  This is a serious error.  This error is 
known to occur when a user keeps rewriting over-and-over again the same 
file into dCache.

New error, UNCAUGHT_EXCEPTION, occurs when previous versions of encp
would have given a traceback to stdout.  Now when such an error occurs 
the traceback dump is instead sent to the log server, zero length files
are cleaned up and the accounting server is notified of the failure.
NOTE: This does not work for "kill -9". 

Misc.:
======

A threading bug was discovered that affects all non-Linux systems and newer
Linux systems with NPTL.  Older Linux systems (with Pthreads) would return
a different process id (PID) for each thread in the process.  This behavior
does not conform to the POSIX standard.  Enstore/encp would use this value
as a unique key for keeping messages to and from different threads separate.
This does not work on POSIX compliant systems because all threads are
returned the same value.  Now, a unique thread number is used instead of the
process id.  Almost everything in the enstore and encp commands is single 
threaded and did not trip over this issue.

A bug was discovered that was allowing the enstore servers to send replies
to offsite computers despite the department policy to not allow this.  The
production servers have been patched for some time, but this is the first
cut encp product that also enforces this policy.

The ecrc program has some new switches: 
-0: Use the enstore 0 seeded adler32 algorithm.  (default)
-1: Use the standard 1 seeded adler32 algorithm.
-d: Output crc as decimal number.  (default)
-h: Output crc as hexidecimal number.
-v: Output additional information.

The ddencp program has a new switch:
-c: Attempt to remove the file's contents from the buffer cache in memory.
    This is useful when testing read/write rates of the disk.  Not all
    systems support this functionality.
-S: Open regular files with O_SYNC passed to open(2).
-D: Open regular files with D_SYNC passed to open(2).
-R: Open regular files with R_SYNC passed to open(2).
The -v switch has been modified to additionally report how many pages of 
the file exist in the file buffer cache in main memory.  This only works 
on systems that support the mincore(2) function.


                      Detailed cvs commit logs


========== Trace.py ====================================================================================

Cleaned up a __pychecker__ warning about unused arguments. Also, modified handle_error() in a way that should prevent resource leaks from the local tb variable problem. 

========== accounting_client.py ====================================================================================

add storage_group to log_encp_error() 

change the interface of log_encp_error() to accomodate multi-interfaces nodes 

========== accounting.py ====================================================================================

take care of error being None in log_encp_error() 

bug fix 

add storage_group to log_encp_error() 

add log_encp_error 

========== alarm.py ====================================================================================

fix another warning 

remove warnings 

========== alarm_client.py ====================================================================================

Disabled a pychecker warning with __pychecker__. 

Fixed the default server_name to be MY_SERVER instead of None. 

========== alarm_server.py ====================================================================================

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

========== callback.py ====================================================================================

Handle EINTR differently if returned from select(). Also fix a math bug if the "select duration" error occurs. 

Added diagnostics to help understand various network errors we are seeing. 

========== cern_wrapper.py ====================================================================================

fixed a bug causing exception in volume_labels for labels with lengt more than 6 chars. For instance PRO539L1 

========== checkdb.py ====================================================================================

take out createuser which was a mistake 

add user root 

========== cleanUDP.py ====================================================================================

Added some comments. Cleaned up lines longer than 80 characters. Added the linux 2.4 kernel to the non-linux case since it seems that the non-standard behavior linux used to do was fixed. 

========== configuration_client.py ====================================================================================

Fix a backward compatibility issue. Old clients can now use the new server and will receive the old reply ticket format. 

Use the hostaddr (in)valid domain functionality. 

fix help output of enstore config 

remove ticket parameter for get_library_managers 

Added the --timestamp function for a user to query when the current configuration was loaded. This information is now used to determine if cached config information is still valid. 

========== configuration_server.py ====================================================================================

Fix a backward compatibility issue. Old clients can now use the new server and will receive the old reply ticket format. 

Modified the dump() and lookup() functions to support the hostaddr (in)valid domains functionality. 

Added support for the configuration server to remember when it last loaded the configuration file. This information can be queried. 

========== delete_at_exit.py ====================================================================================

Only instantiate the csc when the _deletion_list_bfids is non-empty. 

Just some cleanup and comments. 

========== drivestat_server.py ====================================================================================

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

========== e_errors.py ====================================================================================

Added the UNCAUGHT_EXCEPTION error. 

Added CRC_DCACHE_MISMATCH to the list of errors that can be reported from encp. This differentiates this type of error from CRC_ENCP_MISMATCH which was used for both situations. 

========== encp.py ====================================================================================

Comment out the --copy option for the v3_4 cut. 

bumping version to v3_4 because of encpCut 

Added the --copy switch for encp. This switch allows for encp to write N copies of the file to different tapes. 

Fixed an error reported. Two cases exist. One is when get_file_clerk_info() is passed a ticket and not just the bfid when an error occurs. The other is when get_volume_clerk_info() also has an error when a ticket (and not just the external label) is passed to it. When the error string was put together the entire ticket was used instead of just the bfid or volume. 

Fixed e_access() to handle or-ed os.R_OK, os.R_OK and os.X_OK values. Not just one of them at a time. 

Use the new code in enstore_functions2.py 1.12 to handle reading filenames like: stkensrv2:7500//pnfs/test/testfile. Also, fixed a bug in reading some bfids. Some 2000 bfids from D0 have "DMS" bfids instead of "D0MS". Encp had problems with these bfids. The brand and bfid handling code has been fixed to handle brands with lengths other than that of four bytes. 

Modified print_data_access_layer_format() to check for sub-tickets to actually be dictionaries. There seems to be a way for a string to be named one of these sub-tickets that causes tracebacks. 

Use the new accounting server ability to handle None in the message field for log_encp_error(). 

Modify how encp, ensync and enmv use the option.Interface class's print_usage() function. All command line parsing errors, should now give an error return code. 

Don't pass None to log_encp_error() for the error paramter. Pass "" instead. 

Fixed the cause of an UNCAUGHT_EXCEPTION. The if...else branch did create a situation where an uninitialized variable could be used. 

Bug fix in the way encp handles bogus libraries. If the library specified does not exist, an UNGAUGHT_EXCEPTION would occur. This is now handled. 

Fixed a handle_retries bug such that the queue size is now correctly determined on errors. Also, when such error occurs, it is not deleted from the queue since the 'completion_status' ticket field is now used. 

It might be possible that an error while reading could be forgetten about when the exit_status is printed/returned. Modified the if...elif...else statement to protect against wrongly falling into the 'else' section. 

Encp resends on writes were being done twice instead of just once. This has been fixed. Also, the messages to the user and log file specifing that encp is trying to submit have been "cleaned up" and only occur from one spot in the code now, namely submit_one_request(), instead of all over. 

Added "get" option --skip-deleted-files. This option will cause get to skip over deleted files. 

Bug fix. Now it handles exceptions raised when the file has been deleted before it attempts to clobber layer 1 & 4 during a dcache write. 

Fixed the message string that is created when a volume is NOACCESS (the volume name was listed as the entire work_ticket as a string). Also, modified where print_data_access_layer_format() looks for the volume in the ticket to print the most accurate information. It was not handing the case where only the 'external_label' in the VC sub-ticket was known, but none of the others. 

The switch --bypass-filesystem-max-filesize-check was broken. Fixed it. 

Added support for encp to record failures in the accounting database. Added the UNCAUGHT_EXCEPTION error to give to users when encp would otherwise traceback. 

Fix a "get" bug. When get would attempt to read a file with only a partial entry in the file database and no corresponding pnfs file; the new pnfs file would not be created because the pnfs class instance wasn't getting instantiated correctly. Added a pnfs.Pnfs() call and bug fixed. 

Handle an AttributeError exception for "get" when the pnfs file does not exist when entering the check_crc() function. This check is for the dcache layer 2 check. 

Fixed use of the --check option. I had become broken. 

Fixed two bugs relating to ignoring of the --library and --file-family-wrapper switches. The first was only cosmetic. The First log file message printed the wrong values. The second was due to always looking at the output directory tags when checking the maximum size of the library and file family wrapper. 

Modified the control socket error detection section in handle_retries(). It might be eating messages. Now just check if the queue is empty or not after the select(). 

Modified encp to determine the next transfer similar to how "get" does so. This involves using the 'completion_status' field in the request ticket. This work is done it preperation of having an API (for automigration). 

Added CRC_DCACHE_MISMATCH to the list of errors that can be reported from encp. This differentiates this type of error from CRC_ENCP_MISMATCH which was used for both situations. 

Modified when/how the control socket is accessed before transfer_file(). There might have been eaten final dialogs for really small files. This entailed two new functions: stall_read_transfer() and stall_write_transfer() that encapsulate mostly existing inline functionality. 

Added code to detect (with multiple file transfers) if two input files will be copied to the same output file. (i.e. same name; two input dirs) 

I changed the timeout values when checking the control socket for any unexpected problems. Instead of 5 seconds it is now 1 second. 

Misspelled max_resumit as max_resubmits. 

Removed a comma that mylint.py did not like. 

Removed dependance on brands to determine if a file clerk knows about a bfid. Made a change to use the newer configuration_client.py caching mechanism. Started adding accounting_client code to have encp record information about failure to the accounting database. Still commented out. Moved max_attempts() to inside create_read/write_requests(). The retry/resubmit (and maxes) information is now placed inside a sub-ticket. Moved the external_label checks around. This is to avoid a check after each file (which is painful for get). Bug fix for get; input filenames were always getting the sequetial filenames even when the real names were given via --list. 

get_csc() can raise exceptions. Some places in encp/get did not catch them. They are now caught. 

Handle bfids with "L" appended to them correctly. This would cause an exception and the exception handling code had another bug that caused a traceback. Also, handle files where the file database metadata has None for the crc. This was also causing tracebacks. Lastly, make a preparatory change to EncpError to place the "status" error inside the optional ticket if not already done so. 

Fixed two bugs regarding encp not recognizing some bfids. Some older bfids (Nov 2000 were when the sample file's bfids were created) appended a long() "L" to the end of the bfid. This L is consistant between the pnfs metadata and the file database. Thus, encp will now recognize these correctly. The other bug was in the error handling code that tried to use the wrong undefined variable when forming the error string for the first bug mentioned. 

Fixed a bug where Pnfs.__init__() would succed, but if get_path() was passed as the parameter the empty string, a traceback would occur. Pnfs.get_path() can now throw ValueErrors (in addition to OSErrors or IOErrors), which are now handled by encp. 

Modified the code to deal with some occasional errors. These occur when __get_fcc() (and __get_vcc()) return None for the volume/bfid information. This can occur (correctly) for some cases; like None was passed into the function for the volume/bfid parameter. However, for these error cases there should have been a valid value. These changes should handle the error and simultaneously determine what is actually happening. 

========== enmv.py ====================================================================================

Modify how encp, ensync and enmv use the option.Interface class's print_usage() function. All command line parsing errors, should now give an error return code. 

========== enstore_erc_functions.py ====================================================================================

remove unused param in read_erc method 

========== enstore_files.py ====================================================================================

generation of per group tape ingest rates html poage 

change NONE to ENONE 

========== enstore_functions2.py ====================================================================================

Provide underlying support for encp to handle reading filenames like stkensrv2:7500//pnfs/test/testfile. This was done by adding a function fullpath2(). Also, some common code between fullpath() and fullpath2() was put into expand_path(). 

Catch a socket.error exception. This function has the ability to fail from "to many files open", though I'm sure there are others errors too. 

Cleaned up the code w.r.t pychecker warnings. Unused imports, functions that do nothing, unsued variables, etc. 

========== enstore_html.py ====================================================================================

generation of per group tape ingest rates html poage 

change NONE to ENONE 

========== enstore_overall_status.py ====================================================================================

fix warnings 

fix NONE 

========== enstore_show_inv_summary_cgi.py ====================================================================================

update special files for each system 

add QUOTA_ALERT 

add RECYCLABLE_VOLUMES 

add MIGRATION_STATUS 

add MIGRATED_VOLUMES 

========== enstore_show_inventory_cgi.py ====================================================================================

update special files for each system 

add QUOTA_ALERT 

add RECYCLABLE_VOLUMES 

add MIGRATED_VOLUMES 

========== enstore_start.py ====================================================================================

fixed redirection of the output. Estart did not work without it 

remove ticket parameter for get_library_managers 

========== enstore_up_down.py ====================================================================================

add barebones ticket generation when log/alarm/config server down and ticket needed 

remove pdb 

fix bug where servers were not being 'seen down' 

========== ensync.py ====================================================================================

Modify how encp, ensync and enmv use the option.Interface class's print_usage() function. All command line parsing errors, should now give an error return code. 

========== entv.py ====================================================================================

Added code to re-exec() the entv process when the memory leak grows to big. 

Fixed the popup window of mover information when a user clicks on the mover state. Before it would traceback. 

Added some retry capability to mover status requests. After 5 seconds another request is sent out. After 5 retries we give up. 

Recent fixes made entv work faster, they also introduced sizable memory leaks. This patch should fix the leaks without adding slowdown. 1) Reworked the reinitialize()/window_killed() relationship. The most important thing is that destroy() function gets called. 2) Functionalized the garbage collection and reporting. 3) Added a new garbage collection warning. 4) Fixed a bug if all system names on the command line are invalid. 5) Eliminated the request_mover_status() thread. All communication is now done through handle_messages(). 6) Part of the --generate-messages-file functionality is still broken. This part is the clean ending of the output. 

Fix a newly introduced bug that prevents the display from working after a reinitialization. 

All network communication is handled in handle_messages(). This simplifies the sychronization between this thread and the main thread. Movers in offline state are now displayed correctly. Same for those in Unknown state. Recent optimizations caused this. 

Attempting to eliminate stubborn resource leaks. 

An attempt to plug another resource leak. 

Fixed the bug where entv moves upward on the screen when it reinitializes. The update() function needed to be called for the root window object, not just the Canvas object. Hopefully, fixed the IRIX issue where the choosen font size is way to big. The font family requested was always arial. The problem is that few UNIX systems have arial fonts on them. Linux seemed to pick a good substitute but IRIX did/does not. Now uses Helvetica font family. Added a startup_lock. This should allow for a faster startup. This thread prevents the "netowrk" threads in entv.py from doing anything until the main thread is in mainloop(). When multiple enstore systems are specified, it also keeps the resources down by letting only one go at a time. There was a sizable resouce leak. The csc's that were created in the csc_command() function where not being reclaimed. Despite what gc.collect() was saying. These cscs are now explicitly destroyed in destroy() to fix the problem. A ton of speed improvements have been added. Too many to name. Most are of types described in Chapter 17 of "Python and Tkinter Programming." A lot of profiling was done to aid in finding the slowest sections. 

Fixed two bugs. First, is the bug where entv on re-initializations will move upward on the screen the width of the titlebar. This is due to an inconsistancy in which object the geometry values are taken. The second is to add a starup lock that prevents the mainloop from asking movers about their state if the mover_status thread is still running. This greatly improves startup time as the different threads are not duplicating work. 

remove ticket parameter for get_library_managers 

Added an option to entv: --capture-timeout This option allows for entv to specify how long (in seconds) --generate-messages-file should continue to display new transfers. 

Added the --generate-messages-file option to entv. This option allows for the user to have entv output to standard output the messages it used to draw the display. This output, if redirected to a file, can using --messages-file (renamed from --commands-file) recreate the earier entv displayed output. Also, fixed the --messages-file (former --commands-file) switch to actually work. 

Made changes regarding how configuration information has changed. These changes were necessary after changes in configuration_client.py. 

Fixed the --don't show option. It became broken when the config caching functionality was added. 

========== event_relay_client.py ====================================================================================

Suppress pychecker warnings. These have been confirmed to be false positives and the code is correct. 

========== event_relay_messages.py ====================================================================================

Suppress pychecker warnings. These have been confirmed to be false positives and the code is correct. 

remove warnings 

========== file_clerk.py ====================================================================================

add show_state() 

minor change 

new_bit_file() now accepts predefined bfid 

same upgradei as info_server/client: shift formatting load to client 

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

========== file_clerk_client.py ====================================================================================

add show_state() 

fix tape_list 

same upgradei as info_server/client: shift formatting load to client 

========== ftt_driver.py ====================================================================================

return more from flush 

========== get.py ====================================================================================

bumping version to v1_45 because of sdsscpCut 

Fixed an error message. It had incorrect format to value matches for the error string. 

Fixed a automounting bug where ENOENT could false be returned. Now a few retries of the os.stat() are performed before giving up with this error. 

bumping version to v1_44 because of sdsscpCut 

Modified get's use of submit_read_requests() to use 2 parameters not 3. Encp had changed and get is now getting its change. The command line output to terminal and log file was wrong. Due to the way the volume is inserted into the interface class it was being dropped. It now appears in these output lines. 

Added "get" option --skip-deleted-files. This option will cause get to skip over deleted files. 

Fixed the parameter part of the usage help line. It was suggesting source and destination files not directories. 

Added the UNCAUGHT_EXCEPTION error to give to users when get would otherwise traceback. 

bumping version to v1_43 because of sdsscpCut 

Added a combine_dict() call to make sure the work ticket had the error from done_ticket in it's status field. Otherwise, the error could be dropped. The situation for this was an error occuring in the first place was an error in stall_read_transfer(). 

bumping version to v1_42 because of sdsscpCut 

Cleaned up some unused code any pychcker warnings. (Removed EOD references.) 

Modified when/how the control socket is accessed before transfer_file(). There might have been eaten final dialogs for really small files. This entailed two new functions: stall_read_transfer() and stall_write_transfer() that encapsulate mostly existing inline functionality. 

bumping version to v1_41 because of sdsscpCut 

Moved where external_label checks are performed. For get this was a performance hit. Changed to use the new resend subticket that holds retry/resubmit info. 

get_csc() can raise exceptions. Some places in encp/get did not catch them. They are now caught. 

Fixed the handling of OSErrors from create_zero_length_local_files(). They were causing a traceback to occur. 

========== get_total_bytes_counter.py ====================================================================================

add new libraries 

========== hostaddr.py ====================================================================================

Added support for an invalid ip list. Made the (in)valid ip list(s) configurable in the config file. And lastly, cleaned up the __main__() section. 

========== info_client.py ====================================================================================

fix tape_list 

remove dependency on edb 

fixing ls-active 

1 shift formatting of the output of get_vols(), get_pvols(), tape_list() and list_active() to client [2] optimize some queries [3] enhance get_vols() with keyed value 

info_server.py 

========== info_server.py ====================================================================================

allow lower case noaccess, notallowed, deleted in --vols key 

minor message change 

fixing ls-active 

1 shift formatting of the output of get_vols(), get_pvols(), tape_list() and list_active() to client [2] optimize some queries [3] enhance get_vols() with keyed value 

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

========== inquisitor.py ====================================================================================

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

remove unused param in read_erc method 

========== inventory.py ====================================================================================

minor adjustment of print out 

add CR in recyclable message 

add recyclable list 

handle just deleted volumes 

add count for migrated volumes 

add more information to MIGRATED_VOLUMES 

fix a typo 

add migrated volumes report 

========== library_manager.py ====================================================================================

added processing of lowest priority requests, modified change of state to go to the default state if specified in configuration 

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

fixed indentation problem 

fixed rm_active_volume 

added total time for active requests 

modifications for processing mgration requests 

========== log_client.py ====================================================================================

Cleaned up pychecker warnings: unused imports, TRUE vs. True and unused variables. Also, modified lines that were longer than 80 characters to be shorter. 

========== log_server.py ====================================================================================

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. The log server now has its own log_func() to log its own log messages. 

========== makeplot.py ====================================================================================

Added check to make sure that gnuplot is installed before trying anything. 

========== manage_queue.py ====================================================================================

added processing of lowest priority requests 

========== media_changer.py ====================================================================================

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

added logoff command to STK method 

========== monitor_server.py ====================================================================================

Turned a print statement into a Trace.log() call. If stdout is closed and the print is executed a traceback would occur. 

========== mounts_plot.py ====================================================================================

add library 

========== mover.py ====================================================================================

1. Set tape to NOACCESS and offline the mover if it is stuck 2. Offline the mover if get_stats throws an exception. 

Added Trace.notify() calls inside assert_vol(). When a volume was asserted entv was not receiving the 'connect' and 'disconnect' messages. 

go offline if close_dev failed 

fixed processing of dismount on mount of HiPri request failure 

Missed an import of event_relay_messages. 

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

do not offline mover if tape thread did non complete for DLT tape 

increased delay for dlt drives to synchronize tape and net threads 

increased retry TO to 20 s for syncronization between net and tape threads. The problem is seen so far only with DLT tapes 

fixed a bug introduced in the last version 

Synchronize read_tape and write_client codes. Write_cleint needs to wait until read_tape finishes. The problem was noticed reading DLT tapes. Mover would go into HAVE_BOUND state before tape thread completed, which caused it to go OFFLINE (Tape thread is running in HAVE_BOUND state). 

more fixes for race conditions 

fix fix race conditions 

fix fix race conditions 

fix race conditions 

it may happen so that for some unknown yet reason the net thread hangs after connection was broken by mover as the consequence the data transfers can not be done until hung net thread is gone. This thread can hang forever to deal with this situation do the following here: check for the presence of the net and tape thread in the IDLE or HAVE_BOUND state. They should not be running. If net thread is running restart the mover. It is a radical measure, but documents do not recommend killing thread with not closed or hung socket. If tape thread is running put mover offline. 

Removed the one usage of enstore_functions.is_ok(). It now uses e_errors.is_ok() for the same functionality. This allows for the import of enstore_functions to be removed, since this is_ok() call was the only thing the module was used for. 

========== option.py ====================================================================================

Added the --copy switch for encp. This switch allows for encp to write N copies of the file to different tapes. 

add show_state() 

If print_usage() is passed an error message, return an exit code of 1. Otherwise, on success, return 0. 

Added "get" option --skip-deleted-files. This option will cause get to skip over deleted files. 

Added an option to entv: --capture-timeout This option allows for entv to specify how long (in seconds) --generate-messages-file should continue to display new transfers. 

Added the --generate-messages-file option to entv. This option allows for the user to have entv output to standard output the messages it used to draw the display. This output, if redirected to a file, can using --messages-file (renamed from --commands-file) recreate the earier entv displayed output. Also, fixed the --messages-file (former --commands-file) switch to actually work. 

Added --timestamp for config client. 

add --full 

========== pnfs.py ====================================================================================

Modified the pnfs scan to be faster. In pnfs.py that meant removing access() calls and tweaking the id and parent functons. In scanfiles.py e_access() from encp was copied in and modified a little, uncessesary access calls were removed and the order of pnfs and enstore queries were tweaked. 

Added that the parent id is found when the File class is instantiated. This extra information is used in scanfiles.py. 

If the correct "p" function is not found, make sure the return code is 1 for error and not zero for success. 

Removed a bug in is_pnfsid(). If the current working directory had a filename name the same as its pnfsid (which is true for dcache files) then a false error could have been given. 

do not check p_path for consistency 

The --tagchown and --tagchmod had the same bug. If the tag specified did not exist a traceback was thrown. The traceback is now handled. 

bug fix again 

bug fix, again, in set_size() 

bug fix again in set_size() 

fix a bug in setting the size wrong 

take care of size > 2G in class File 

Handle user root as a special case in verify existance. Even without other file permissions user root can access a file. 

Fixed a bug where Pnfs.__init__() would succed, but if get_path() was passed as the parameter the empty string, a traceback would occur. Pnfs.get_path() can now throw ValueErrors (in addition to OSErrors or IOErrors), which are now handled by encp. 

========== quota.py ====================================================================================

handle approaching quota limit 

fix --show-by-library 

fix --show-by-library 

fix --show-by-library 

make pychecker happier 

show allocated numbers 

========== ratekeeper.py ====================================================================================

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

========== scanfiles.py ====================================================================================

Correctly ignore .bad.* files. 

Comment out some profiling code. 

Modified the pnfs scan to be faster. In pnfs.py that meant removing access() calls and tweaking the id and parent functons. In scanfiles.py e_access() from encp was copied in and modified a little, uncessesary access calls were removed and the order of pnfs and enstore queries were tweaked. 

Commented out the profiling code. 

Added a try... except to catch when a directory is deleted. Also, a new function start_check(line) has been added. This is to help encapsulate this common to having the input from a 'file' or from the command line. 

Added check to make sure the recored parent id of a file matches the directory that the file is currently in. Also, if a directory itself is OK don't print anything. If an error or warning exists for a directory, still print the message. 

Modified the main loop to read from file objects one line at a time instead of doing readlines() and consuming potentially vast resources. 

By default will read from stdin if no other input is specified. 

Seperated the errors when (1) the file no longer exists and (2) when a directory contains an entry to an invalid/non-existent i-node. 

Fix a newly created bug that causes Control-C to be ignored. 

Modified the logic to be able to report if both layer 1 and layer 4 are empty, not just the first one found to be empty. Also, modified the error reporting for moved/renamed files; an error is given if layer 4 does not match the file database and a warning if these to values match but differ from the current filepath. 

Made files beginning with ".nfs" or ending with "_lock" and error. Skip files beginning with ".bad.". Remove unecessary stat()s from os.path.xxx() calls. Now call os.stat() and check everything from that information. Uses e_access() from encp.py instead of os.access(). (Done as a hack.) Eliminated pychecker warnings. 

handle old location cookie format 

========== set_lm_noread.py ====================================================================================

Removed two references to enstore_functions.is_ok(). Replaced with e_errors.is_ok(). 

========== udp_common.py ====================================================================================

Cleaned up pychecker warnings regarding class attributes not being found. The confusion stems from the cleanUDP class acting like a socket without inheriting from the socket object class. 

========== volume_assert.py ====================================================================================

Added 'file_family' ticket item to stop the inquisitor from tracebacking. 

Fixed errors that resulted in modifiying functions to work with recent encp.py code. 

========== volume_clerk.py ====================================================================================

add show_state() 

allow lower case noaccess, notallowed, deleted in --vols key 

same upgradei as info_server/client: shift formatting load to client 

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

fix a bug that miss counting newly added blanks; page admin when quota is exceeded 

making pychecker happy 

change wording 

1 correct a typo [2] make "quota reached" a separate case [3] show (lib, sg) in quota exceeded message 

volume_clerk.py 

change alarm message 

clean up on alram messages 

change alarm message 

fix a typo 

fix a typo 

add advanced alarm when (library, storage_group) approaches its quota limit 

fix a typo 

fix a bug in using get_sg_counter() 

fix a typo 

1 handle sg_count for all storage group [2] low water warning and alarm for blanks in common pool 

take out paging when a blank is drawn from common pool 

add check_state switch for delete_volume() 

remove ticket parameter for get_library_managers 

fix a typo 

let volume clerk to erase files. No contact to file clerk any more 

add --full 

========== udp_client.py ====================================================================================

One last piece of the udp_client threading problems. This time it is that the __del__ method was using get_tsd(). If the last thread did not use the UDPClient then there is nothing left to destroy. But get_tsd() would create a UDPclient instance just to be destroyed. The fix was to have __del__ check if self.tsd has an element, if so only then delete it (as opposed to create one just to be deleted). 

Fixed another threading bug. This time it is when the udp_client object needs to be garbage collected. There was a cyclic reference that prevented the object from being cleaned up. 

Fixed a serious bug on POSIX conforming thread implimentations. Older linux OSes would return a different process id when getpid(2) was called in each thread. Conforming implimentations will get the same process id for each thread that calls getpid(2). Using the pid on a conforming OS as a key for a dictionary of thread specific information would lead to all threads dangerously sharing the same socket. Also, in order to test the preceding bug; in the __name__ == "__main__" section the test code was fixed to actually work. 

Cleaned up two pychecker warnings. The first was fixed by using the tsd Container class in a slightly different manor. The other was to replace the string UDPError exception with a class UDPError exception. 

========== accounting_server.py ====================================================================================

Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. 

add storage_group to log_encp_error() 

add log_encp_error 

========== enstore_display.py ====================================================================================

Cleaned up the code that finds the correct library paths. 

Expanded use of the display_lock to prevent the possibility of race conditions. 

Fixed the popup window of mover information when a user clicks on the mover state. Before it would traceback. 

Recent fixes made entv work faster, they also introduced sizable memory leaks. This patch should fix the leaks without adding slowdown. 1) Reworked the reinitialize()/window_killed() relationship. The most important thing is that destroy() function gets called. 2) Functionalized the garbage collection and reporting. 3) Added a new garbage collection warning. 4) Fixed a bug if all system names on the command line are invalid. 5) Eliminated the request_mover_status() thread. All communication is now done through handle_messages(). 6) Part of the --generate-messages-file functionality is still broken. This part is the clean ending of the output. 

Fix a newly introduced bug that prevents the display from working after a reinitialization. 

All network communication is handled in handle_messages(). This simplifies the sychronization between this thread and the main thread. Movers in offline state are now displayed correctly. Same for those in Unknown state. Recent optimizations caused this. 

Fixed a bug when get_up_to_date() does not get something valid from the mover. 

Fixed a problem with large file sizes. 

The ability for both progress bars became broken in the last update. This fixes it. 

Fixed the bug where entv moves upward on the screen when it reinitializes. The update() function needed to be called for the root window object, not just the Canvas object. Hopefully, fixed the IRIX issue where the choosen font size is way to big. The font family requested was always arial. The problem is that few UNIX systems have arial fonts on them. Linux seemed to pick a good substitute but IRIX did/does not. Now uses Helvetica font family. Added a startup_lock. This should allow for a faster startup. This thread prevents the "netowrk" threads in entv.py from doing anything until the main thread is in mainloop(). When multiple enstore systems are specified, it also keeps the resources down by letting only one go at a time. There was a sizable resouce leak. The csc's that were created in the csc_command() function where not being reclaimed. Despite what gc.collect() was saying. These cscs are now explicitly destroyed in destroy() to fix the problem. A ton of speed improvements have been added. Too many to name. Most are of types described in Chapter 17 of "Python and Tkinter Programming." A lot of profiling was done to aid in finding the slowest sections. 

Fixed two bugs. First, is the bug where entv on re-initializations will move upward on the screen the width of the titlebar. This is due to an inconsistancy in which object the geometry values are taken. The second is to add a starup lock that prevents the mainloop from asking movers about their state if the mover_status thread is still running. This greatly improves startup time as the different threads are not duplicating work. 

Added the --generate-messages-file option to entv. This option allows for the user to have entv output to standard output the messages it used to draw the display. This output, if redirected to a file, can using --messages-file (renamed from --commands-file) recreate the earier entv displayed output. Also, fixed the --messages-file (former --commands-file) switch to actually work. 

Handle clearing the percent done display of mover the same way as the buffer part; when undrawing them set the variables holding the quanties to be displayed to None. Thus, when the next transfer is started, None is not equal to 0 and the empty boxes get displayed. 

This change sets the current buffer bar size to None when it is undrawn. Without this there is a posibility that on the next file the bar does not get redrawn because it is the same as the remembered size. 

Made changes regarding how configuration information has changed. These changes were necessary after changes in configuration_client.py. 

========== host_config.py ====================================================================================

Re-implemented the is_route_in_table() function to be faster. To many string.split()s and string.join()s. 

Added function clear_cached_routes(). It will reset the _cached_netstat global back to its initial state. Nothing uses it now, but it was useful in cleaning up objects (in the list) for debugging resource leaks. 

========== dispatching_worker.py ====================================================================================

Handle differences for the configuration server in the way it works versus the other servers. The difference is that the configuration server does not need notification that there was a new configuration was loaded. 

Fixed a security bug. The return value from hostaddr.allow() was being ignored. This would allow dispatching worker servers to reply to requests from places it should not. The value is now honored and such requests are ignored. 

========== multiple_interface.py ====================================================================================

Removed some code that resulted in a pychecker warning. Previously, the code would import multiple_interface, which is weird from that module itself. Used the sys.modules[] list to access the modules contents. Also, found another bug in _parse_default(). Fixed that too. 

========== generic_server.py ====================================================================================

Handle differences for the configuration server in the way it works versus the other servers. The difference is that the configuration server does not need notification that there was a new configuration was loaded. 

Bug fix. Fixed a bug that was causing all server to not cache the contents of the configuration. enable_caching() does not take any parameters, but was being passed one. This caused a traceback caught in a blanket except that did nothing. 

remove unused param in read_erc method 

========== migrate.py ====================================================================================

change CMS_MIGRATION_DB to 'cms/MIGRATION-9940A-TO-9940B' 

cms conforms to standard 

handle cms migration_path() 

migrate to 'none' if there is no active file 

add --bypass-filesystem-max-filesize-check to encp 

deal with ghost files 

do not check drive any more 

allow open target volume to be scanned 

1 make sure the file does not have a prefix /pnfs/fs/usr/Migration in final_scan_volume [2] log migration history in the end of each volume migration 

delete target file after write failure 

Set the last_access of the from volumes to now in the end of final_scan_volume 

shorten comment 

add --migrated-from and --migrated-to, bug fix 

add --migrated-from and --migrated-to, bug fix 

add --migrated-from and --migrated-to, bug fix 

add --migrated-from and --migrated-to 

set comment of 'migrated to' at the end of volume migration 

set comment at the end of scan-vol 

1 retry once when encountering COPY_TO_TAPE error [2] show location cookie in more log message 

print location cookie in final scan volume 

check more for volume migration 

fix a typo 

handle unknown:unknown drives 

fix a typo 

bug fix 

rearrange message coming out fo final_scan_volume() 

rearrange message coming out fo final_scan_volume() 

take out the debugging code for final_scan_volume() 

a few minor bug fixes and a section of debugging code for final_scan_volume() 

using poor-man's encp API 

log command line 

change to final_scan_volume(): 1 check for file family *-MIGRATION [2] check for 'full' [3] move restoring file family to the end of final scan [4] restoring file family only if there is no error [5] returns number of errors 

========== volume_clerk_client.py ====================================================================================

forgot to change the tape_list interface 

add show_state() 

remove dependency on edb 

same upgradei as info_server/client: shift formatting load to client 

making pychecker happy 

making pychecker happy 

add check_state switch for delete_volume() 

add --full 

add --full 

========== enstore_functions.py ====================================================================================

Removed the is_ok() and is_timedout() functions from this module. Use the functions from the e_errors module, with the same names, instead. 

========== library_manager_client.py ====================================================================================

added operation for --vol command if state is ACTIVE 

Fixed the help string for the --start-draining switch: 1) ingore => ignore 2) locked => lock 3) moved the list of values to the help string to preserve case sensitivity. 

changed --vol output format 

total time added to lib vols 

========== udp_server.py ====================================================================================

Modified the __name__ == "__main__" test section to work with the udp_client test (for localhost) instead of being hardcoded for node 'happy'. 

Cleaned up pychecker warnings about unused imports. 

========== enstore_constants.py ====================================================================================

change NONE to ENONE 

========== generic_client.py ====================================================================================

Re-raise KeyboardInterrupt and SystemExit exceptions from udp_client.send() in generic_client.send().