Bug: Cycling of cluster synchronisation log files

When the cluster synchronisation log file was cycled after receipt of a {\tt HUP} signal, the cycling was erroneously performed both in the signal processing code and in the main loop code which responds to the signal. I removed the file cycling from the signal handler, where it is vulnerable to race conditions within Perl and the underlying C library.

Feature: Line buffering of cluster synchronisation log file

Added code to set automatic buffer flush for the ClusterSync log file when it is initially opened and cycled after receipt of the {\tt HUP} signal. This allows those who monitor the log file with, for example, {\tt tail~-f} to see the complete log item for a transaction without waiting for the buffer to be flushed.

Modified the cluster synchronisation log file generation to skip the ``Results:'' line if the command produced no output.

Feature: Recovery from transient cluster synchronisation failures

Implemented recovery from transient and permanent cluster synchronisation transaction failures. Previously, if any error occurred reading, verifying, or executing a cluster synchronisation transaction, the {\tt ClusterSync.pl} program would crash, suspending cluster synchronisation until it was restarted. Unfortunately, there were a number of circumstances in which such errors could occur, the most common being cases where a race condition between queueing the transaction and {\tt ClusterSync}'s processing of it caused an incomplete file to be read (transient), and those where a crash of the process queueing the transaction caused an incomplete file to be written to the transaction directory (persistent).

When a cluster sync transaction fails, for whatever reason, it is placed into a failed transaction hash whose key is the transaction file name and whose value is an array containing the number of times the transaction has been tried and the next time the transaction should be retried. On subsequent passes through the transaction directory, failed transactions are skipped unless their retry time has arrived, whereupon they are retried and, if they fail, their try count is incremented and the next attempt count updated.

If the transaction eventually succeeds, it is closed out normally and removed from the failed transaction hash. If the transaction fails again, its try count is increment and if it has reached the limit, the transaction is deleted from the transaction directory and the failed transaction hash. Failre to delete the transaction from the transaction directory remains fatal to the {\tt ClusterSync} program.

The intervals between retries of a failed transaction and the number of failures which cause a transaction to be abandoned are set by configuration parameters.

Feature: Paper log form generation

Completed the implementation of a facility for generating and printing paper log forms for people who wish to log offline and then transcribe the data to the application later. A new ``Print paper log forms'' item on the Utilities menu displays a form which allows the user to select the first and last month and year (the form is preset to the default of all months in the current year). The current, previous, and next years may be selected. (If the user sets the end date before the start date, they are silently swapped.) When the ``Generate'' button is pressed, a new window opens with the log document in it, and after a one second delay to allow the page to render, a print command is queued (these features require JavaScript to function; if it is absent, the log document opens in the same window as the request form and the user must print it manually and return to the application with the ``Back'' button). A paged media style sheet is used to insert page breaks so that each monthly log prints on its own page.

Bug: Caption in historical charts with multiple days per pixel changes with chart size

Historical chart generation with multiple days per pixel could report different values for the trend analysis and flag fraction in the caption depending upon the chart size (and hence the number of days aggregated into each horizontal pixel). This was because {\tt getDays} is not guaranteed to examine every day in the interval, particularly in the case of long intervals and small charts. I added code to {\tt drawChart} in {\tt history.pm} to call {\tt analyseTrend} for the entire interval to perform the analysis and use the values it computes for the caption, instead of those computed on the fly by {\tt getDays} which may depend upon the chart scale. (Reported by Jim Hollcraft.)

Bug: Excel format CSV import with space before exercise rung

If an Excel-format CSV record contained a space before a single-digit exercise rung field, the record would be skipped as not parsable. I modified the test pattern to allow leading (but not trailing) spaces. Records of this type are created by the Palm HDread program when a day has an exercise rung between 1 and 9 and the -o option is used to generate Excel-format CSV. (Reported by Ömer Ay.)

Admin: Global statistics thwarted by weightless monthly log

If a user created one or more non-void monthly logs with no weight entries (for example, containing only exercise rung and/or comment fields), the administrator global statistics report would fail with a division by zero when it attempted to compute the ``coverage'' of the time period by weight log entries. I modified \verb+receive_aggregated_statistics_records+ to ignore returned records with undefined weight fields, as only such records are relevant to the global statistics.

Bug: Cluster synchronisation of already-deleted session files

Cluster synchronisation could loop with a failed copy transaction when, while processing a backlog of synchronisation transactions, a copy transaction for a session ({\tt .hds}), active session ({\tt .hda}), or remember me ({\tt .hdr}) file was executed after the file in question had been deleted at the close of the session. I added code, similar to the September 5 fix for deletion transactions, which considers copy transactions which fail due to nonexistence of the source file as having completed normally.

Bug: Cluster synchronisation of files containing ISO-8859 unescaped characters

Cluster synchronisation transaction files were written in UTF-8, but {\tt ClusterSync.pl} failed to open them in this mode. This caused file names which contained ISO-8859 characters above the 7 bit ASCII range which we do not escape to be misinterpreted when the transaction was read, resulting in a signature verification failure for the transaction. I added a ``{\tt :utf8}'' specification to the open of the transaction file so it will be read correctly. I also set {\tt STDOUT} to UTF-8 mode so that error messages are printed in that mode. The log file remains in ISO-8859 mode, as that will handle all characters which we do not escape.

Bug/admin: Global statistics for recently-created accounts

Global statistics computation became confused when presented with a user account which had a database entry for the first month in the statistics computation interval (currently 30 days), but in which the first weight entry was after the start of the interval. The code which computes the user's trend slope would pass undefined trend items to the fitter, generating a snowdrift of (otherwise harmless) warning messages. I added a test for undefined trend values returned by the aggregator, which causes the coverage of such users' logs to be deemed incomplete and thus excluded from the summary trend analysis.

Bug: Trend propagation from an empty initial log

Propagation from an initial month in the database with no weight entries to subsequent months would reference an undefined trend value for the month. I added code to set the trend carry-forward to zero in this case, as a trend of zero is our indication that no trend carry-forward exists. Since the undefined trend value would be treated as zero, this caused no problems but produced a warning message in the error log.

Feature/admin: Report number of accounts using Web badges in global statistics

Added a line to the Open Accounts summary in the Global Statistics page which shows the number of accounts which have Web badge generation enabled.

Documentation: Global statistics section in style sheet

Corrected the description of the global statistics table section in the {\tt hdiet.css} style sheet and removed a reference to a nonexistent macro for synthetic data generation style definitions.

Bug: Cluster synchronisation loop attempting to delete file

The {\tt ClusterSync.pl} program could go into an infinite loop if given a transaction which requested the deletion of a file which was not present on the destination cluster host. This situation could occur due to race conditions in which a RememberMe file was created and replaced almost instantaneously. I added code which detects this case and considers the deletion transaction as having been completed successfully if the file is found not to exist on the destination host. The error handling has been restructured to allow other such special cases to be handled should they arise.

Bug: Trend propagation to monthly logs in the future

If a user makes log entries in a month in the future (for example, to add ``to do'' items in the comment field), the future month would be assigned a trend carry-forward at the time it was created, but the carry-forward would not be updated when weight entries were made in the current month because trend propagation was triggered only for entries in months prior to the ``current month'' in the user's time zone. This was an example of the sinfulness of premature optimisation---compared to the cost of updating the chart for an entry in a monthly log, checking the user directory for subsequent months, even if in the future, to which the trend should be propagated is negligible. I removed the unwarranted ``optimisation'' from ``Write updated log item back to database'', causing a check for trend propagation to be performed for all weight changes in monthly logs. (Reported by Anna E. Sage.)

Installation: File copy race condition for executable Perl programs

Modified the {\tt publish} and {\tt production} targets in the {\tt Makefile} to install the executable Perl components ({\tt HackDiet}, {\tt HackDietBadge}, and {\tt ClusterSync}) by copying them to the destination directories with an extension of {\tt .NEW} and then renaming them to the destination name. This avoids the possible race condition when a request arrives while the file is being copied to the server and the CGI process attempts to read the Perl program before it has been entirely copied. Note that we still have a potential race condition for the modules in {\tt HDiet}, but as these files are much smaller, the odds of encountering it are much less than with the large main program. I will eventually change these to install with a copy and move strategy as well, but that will require more work since they are installed with a recursive copy rather than a simple file copy.

Documentation: New CPAN module requirements

Added newly-referenced CPAN modules to the list of library modules we require in the documentation. Each is linked to its documentation on the CPAN site.

Clean-up: Badge configuration

Eliminated a redundant ampersand in the URL submitted when the ``Configure Web page badge image'' item is clicked in the Utilities menu.

Deleted a redundant definition of the array of labels for trend analysis durations in the \verb+update_badge+ transaction handler.

Bug: Incorrect weight in badge when log and display unit set differently

Badge generation mis-handled the case where the log unit and display unit were set differently---fixed.

Feature: Status “badges” for Web pages

Completed implementation of ``Web badges'', which allow users to display their most recent weight log entry and the energy balance and rate of gain/loss for a specified trend interval. A new badge configuration page, accessible from the main utility menu, allows enabling the badge and selecting the trend interval, which is kept in a new \verb+badge_trend+ field in the {\tt user} object. When this field is nonzero, any operation which modifies a log entry calls {\tt history::drawBadgeImage} to update the {\tt BadgeImage.png} file in the user's directory. (This file is swapped into place with a {\tt mv} command to avoid race conditions if it is being retrieved at the time an update is in progress.)

The badge configuration page takes the user to a confirmation page which, if badge generation is enabled, shows XHTML code the user can copy and paste into a Web page to display the badge. This code invokes a new stand-alone lightweight CGI program named {\tt HackDietBadge}, which is called with an opaque argument which is the user file name of the owner of the badge salted and encrypted with the application's master key using AES in CBC mode. The {\tt HackDietBadge} program is separate so as to avoid having to load the full application and all of the modules it requires just to display a badge image on a Web page which may be hit far more frequently than full-fledged application transactions. The {\tt HackDietBadge} program decrypts and validates the argument and, if all is well, copies the badge image for the specified user to standard output hacing specified a {\tt Content-type} of {\tt image/png}. If the argument is in error, or the specified user has disabled badge generation, a canned ``Invalid request'' image is returned instead.