Monday, 10 March 2008
Bug: Cycling of cluster synchronisation log files
When the cluster synchronisation log file was cycled after receipt of
a {\tt HUP} signal, the cycling was erroneously performed both in the
signal processing code and in the main loop code which responds to
the signal. I removed the file cycling from the signal handler,
where it is vulnerable to race conditions within Perl and the underlying
C library.
Feature: Line buffering of cluster synchronisation log file
Added code to set automatic buffer flush for the ClusterSync log file
when it is initially opened and cycled after receipt of the {\tt HUP}
signal. This allows those who monitor the log file with, for example,
{\tt tail~-f} to see the complete log item for a transaction without
waiting for the buffer to be flushed.
Modified the cluster synchronisation log file generation to skip
the ``Results:'' line if the command produced no output.
Feature: Recovery from transient cluster synchronisation failures
Implemented recovery from transient and permanent cluster synchronisation
transaction failures. Previously, if any error occurred reading, verifying, or
executing a cluster synchronisation transaction, the {\tt ClusterSync.pl}
program would crash, suspending cluster synchronisation until it was
restarted. Unfortunately, there were a number of circumstances in which
such errors could occur, the most common being cases where a race condition
between queueing the transaction and {\tt ClusterSync}'s processing of it
caused an incomplete file to be read (transient), and those where a crash
of the process queueing the transaction caused an incomplete file to be
written to the transaction directory (persistent).
When a cluster sync transaction fails, for whatever reason, it is placed
into a failed transaction hash whose key is the transaction file name
and whose value is an array containing the number of times the
transaction has been tried and the next time the transaction
should be retried. On subsequent passes through the transaction
directory, failed transactions are skipped unless their retry time
has arrived, whereupon they are retried and, if they fail, their
try count is incremented and the next attempt count updated.
If the transaction eventually succeeds, it is closed out normally
and removed from the failed transaction hash. If the transaction
fails again, its try count is increment and if it has reached
the limit, the transaction is deleted from the transaction
directory and the failed transaction hash. Failre to delete
the transaction from the transaction directory remains fatal to the
{\tt ClusterSync} program.
The intervals between retries of a failed transaction and the number of
failures which cause a transaction to be abandoned are set by configuration
parameters.
Subscribe to:
Posts (Atom)