3.5.3 Monitoring of cyclic pre-processing

Author(s): Javier Castañeda

The scientific performance of the cyclic pre-processing is assured by specific test campaigns carried out regularly by the DPCB team (Section 1.3.5) in close collaboration with IDU contributors. For these tests, detailed analysis of results are performed, including the execution of reduced iterations with other systems.

IDU processes a large amount of data and similarly produces a large amount of output. The continuous and progressive check on the quality of these results is not a desirable feature. The analysis of every calibration and parameter produced by IDU as done for test campaigns is not practically possible for real data. IDU therefore integrates a modular system to assure the quality of the results up to a reasonable limit.

First of all, IDU tasks include several built-in consistency checks of the input and output data. These are basic checks for:

• •

  verifying the consistency of the configuration parameters, including their tracking along the full processing pipeline;

• •

  verifying the consistency of the input data, so that corrupted data or inconsistent input data combinations do not enter the pipeline and are not propagated to subsequent, downstream tasks;

• •

  accounting for the number of outputs with respect to the inputs, so that lost data is detected and properly handled.

Additionally, all IDU tasks integrate a validation and monitoring framework providing several tools for the generation of statistical plots of different kinds. This framework provides:

Bar Histograms

Histograms for the characterisation of the frequency of parameters with limited numbers of values. For example the outcome of the image parameter processing, counts of observations per row, etc.

1D Histograms

Histograms for the computation of the frequency of non-discrete parameters, including the computation of statistical parameters such as mean, variance, and percentiles.

2D Histograms

Histograms showing the distribution of values in a data set across the range of two parameters. They support static dimensions and dynamic abscissae allocation. The first type is mainly used for the analysis of 2D dependencies or 2D density distributions of two given parameters. In these cases, the abscissa parameter is usually magnitude. The second type is mainly used for analysis of the evolution of a given parameter as a function of a non-restricted, increasing parameter, typically observation time. The bins can be normalised globally or locally for each abscissae bin. Percentiles as well as contours are supported.

Sky Maps

Plots generated from a histogram based on the HEALPix (Górski et al. 2005) tessellation and implementing the Hammer-Aitoff equal-area projection. Plots can represent pixel counts, pixel densities, or mean pixel values for a given, measured parameter. These maps are mainly used to obtain the sky distribution of some particular object (sources, observations, etc.) or to analyse the alpha and delta dependency of some, mean parameter value, e.g., astrophysical background, proper motion, etc.

Sky Region Maps

Tool for plotting sources and detections in small, ICRS-based sky regions. Mainly used for the analysis of the cross-match and detection classification results.

Focal Plane Region Maps

For the representation of the observations according to their along and across focal-plane coordinates. Mainly used for the analysis of the scene and detection classification results.

Round-Robin Database (RRD)

Round-Robin Databases are typically used to handle and plot time-series data like network bandwidth, temperature, CPU load, etc. They can also be used to handle other measurements such as quaternion evolution, observation density, match-distance evolution, etc.

Range Validators

Implement basic field-range validation against the expected, nominal parameter values.

TableStats

Collector of statistics for miscellaneous table fields. Basically provides counters for the discrete values of pre-defined fields or for boolean flags/fields.

All Histograms and Sky Maps share a common framework allowing the split of the collected statistical data according to the FoV, CCD row, CCD strip, window class, source type, etc. This functionality is useful for restricting the origin of any feature visible in global plots.

All tools are integrated in the daily pipeline and are used for its monitoring on a daily basis. A handful of examples of the plots obtained using these tools have been included in Section 3.5.2, Section 3.4.13, and Section 3.4.13.

All tools ease the monitoring of the cyclic scientific results but this is insufficient to guarantee the quality of the outputs. Specific diagnostics are needed to also assure the progressive improvement with respect to the results obtained in previous iterations. Some examples of such diagnostics are:

• •

  For all tasks in general:

  • –

    Range validation against expected nominal parameters.

• •

  For the cross-match and detection classification tasks:

  • –

    Monitoring of the amount of new sources created compared with previous executions.

  • –

    Monitoring of the time evolution of the along and across distance to the primary matched source obtained in the cross-match.

  • –

    Checking the evolution of matches to a pre-defined set of reference sources, to verify that the overall transits have been assigned differently now, as compared to the previous cycle.

  • –

    Monitoring of the evolution of the number of spurious detection density for very bright sources.

• •

  For the image parameters:

  • –

    Monitoring on the goodness-of-fit of the LSF/PSF fitting.

  • –

    Comparison of the derived image parameters against the astrometric solution over a pre-selection of well-behaved sources.

  • –

    Cross-checking of the residuals from the previous statistic against the chromatic calibration residuals from the astrometric solution.

Additionally, it is worth pointing out that the computational performance and the correct progress of the processing is also monitored. IDU integrates different tasks, presenting different input/output (I/O) and computational requirements. A balancing of the task jobs is essential to exploit the computational resources and to be able to meet the wall clock constraints of the data reduction schedule. This balancing is only feasible with good knowledge of the processing performance profile of each task in terms of CPU time, memory, and I/O load.

Performance metrics are a built-in feature of the IDU framework. Each task provides measurements for:

• •

  Number of elements processed: sources, observations, time intervals, etc.

• •

  Total time elapsed for data loading, data writing, and each data processing algorithm.

• •

  File system timing on file access, copy, and deletion.

• •

  Total CPU and I/O time accounted for each processing thread.

With all this information, several diagnostics are generated to obtain the scalability of each task to different parameters. These diagnostics provide valuable information on how the tasks scale when the data inputs are increased (e.g., linearly or exponentially).

Besides the overall task profiling, more detailed information can also be obtained for the profiling of some specific parts of the processing. Such diagnostics are useful to detect possible bottlenecks or unexpected performance degradation for specific parts of the processing or for specific data chunks.