3.5.2 Monitoring of daily pre-processing

IDT monitoring is mainly done through a web interface which acts as a front end to the many statistics and plots generated by the system on-the-fly. That is, diagnostics are continuously compiled in IDT (mainly histograms) on the outputs generated by the system. Most of these diagnostics are computed over typically one day of data. Some of the most useful ones are the following:

• •

  Performance monitor:

  • –

    Plots with the OBMT (on-board mission timeline) being processed with respect to the UTC (on-ground) time, to reveal possible delays or gaps in the processing. These, combined with other checks on the data base outputs, assess that all inputs are processed and that the expected number of outputs are generated. This is done for all inputs and outputs of IDT, and concerns all data processed by any of the downstream DPAC systems.

  • –

    Counters and checks on the number of outputs generated, time ranges received and processed, computing performance of the several algorithms and tasks., etc.

• •

  Consistency checks:

  • –

    List of calibrations being used at a given time.

  • –

    Distribution of measurement configurations and on-board events for all the raw and intermediate outputs, revealing any misconfiguration in the ground databases with respect to the on-board configuration.

• •

  Sky region checks (Figure 3.22):

  • –

    Mollweide projections of the sky regions being observed during a given time, showing the density of transits (measurements) in equatorial coordinates.

  • –

    Sky charts, plotting in a higher resolution (typically about one square degree per plot) the detections being processed, including brightness and acquisition time, for some regions of interest.

  

  

• •

  Photometric features (Figure 3.23):

  • –

    Distribution of the number of transits per magnitude, in the $G$, $G_{\mathrm{BP}}$, $G_{\mathrm{RP}}$, and $G_{\mathrm{RVS}}$ bands.

  • –

    ‘Colour’ distribution, showing the transits per $G_{\mathrm{BP}}-G_{\mathrm{RP}}$ pseudo-colour, per effective wavelength, etc. Also colour-colour plots are determined, showing the effective wavelength distribution per $G_{\mathrm{BP}}-G_{\mathrm{RP}}$ colour.

  

  

• •

  Attitude diagnostics (Figure 3.24, Figure 3.25, and Figure 3.26):

  • –

    Distribution of match distances between the detections used by OGA1 and their associated sources.

  • –

    Average number of detections per second used in the OGA1 attitude reconstruction.

  • –

    Time series with the difference, in field angles (along and across scan), between the reconstructed attitude (OGA1) and the raw or IOGA attitudes. Also time series with the attitude rates (along and across scan) are determined.

  • –

    Motions estimated for the transits processed, determined from the AF observation times and the attitude rates.

  

  

  

  

  

  

• •

  Bias diagnostics (Figure 3.27):

  • –

    Histograms with the distribution of bias and readout noise values per CCD.

  

• •

  Background diagnostics (Figure 3.28):

  • –

    Histograms with the astrophysical background level (in electrons per pixel per second) determined per CCD.

  

• •

  Image parameters diagnostics (Figure 3.29):

  • –

    Outcome of the Image Parameter Determination (IPD), indicating the fraction of windows with problems in the fitting.

  • –

    Distribution of the ‘centroiding’ position within the SM or AF window, revealing possible problems in the on-board centring of the windows or in the PSF/LSF calibration.

  • –

    Goodness-of-fit distribution, based on a ${\chi }^2$ estimation.

  • –

    Distribution of formal errors in the fitting, as provided by the algorithm itself.

  

  

• •

  Cross-match diagnostics (Figure 3.30):

  • –

    Number of matched and unmatched transits (that is, detections for which no source has been found in the catalogue at a distance closer than 1.5 arcsec), number of detections identified as ‘spurious’, and number of new source entries created.

  • –

    Distribution of match distances in the along- and across-scan directions.

  • –

    Ambiguity in the cross-match solution, indicating the fraction of transits for which more than one candidate source was found.

  

  

Scientists processing data collected by astrometric space missions like Gaia have to simultaneously determine a large number of parameters concerning astrometry (and other stellar properties), the satellite’s attitude, as well as the geometric, photometric, and spectroscopic calibration of the instruments. To reach the inherent level of precision for Gaia, many months of observational data have to be incorporated in a global, coherent, and interleaved data reduction. Neither the instrument nor the data health can be verified at the desired level of precision by standard procedures applied to typical space missions. Obviously, it is undesirable not to know the current measurement precision and instrument stability until the next iteration in the global data processing is performed. Unperceived, subtle effects could arise and accumulate over time, irrevocably damaging the raw observations and resulting in a quality reduction or even loss of many months of science data. For this reason, a ‘First Look’ system was installed to judge the level of precision of the (astrometric, photometric, and spectroscopic) stellar, attitude, and instrument calibration parameters on a daily basis and to achieve its targeted performance level by means of sophisticated monitoring and evaluation of the observational data. These daily checks include analyses of:

• •

  astrometric science data with the help of the so-called One-Day Astrometric Solution (ODAS), which allows to derive an accurate on-ground attitude, improved source parameters, daily geometric instrument parameters as well as astrometric residuals required to assess the quality of the daily astrometric solution,

• •

  photometric and spectroscopic science data in order to assess the CCD health and the sanity of the LSF/PSF and spectroscopic calibrations,

• •

  the basic-angle monitor (BAM) data which aims at independently checking the behaviour of the Gaia basic angle,

• •

  auxiliary data such as on-board data needed to allow for a proper science data reduction on ground and on-board processing counters which allow to check the sanity of the on-board processing, and

• •

  the satellite housekeeping data.

The First Look aims at a quick discovery of, sometimes delicate, changes in the spacecraft and payload performance, but also aims at identifying oddities and proposing potential improvements in the initial steps of the on-ground data reduction. Its main goal is to influence the mission operations if and when need arises. The regular products and activities of the First-Look system and team include:

• •

  The One-Day Astrometric Solution (ODAS), which allows to derive a high-precision on-ground attitude, high-precision star positions, and a detailed daily geometric calibration of the astrometric instrument. The ODAS is by far the most complex part of the First-Look system.

• •

  Astrometric residuals of the individual ODAS measurements, required to assess the quality of the measurements and of the daily astrometric solution.

• •

  An automatically generated daily report of typically more than 3000 pages, containing thousands of histograms, time evolution plots, number statistics, calibration parameters, etc.

• •

  A daily manual assessment of this report. This is made possible in about 1–2 hours by an intelligent hierarchical structure, extended internal cross-referencing, and automatic signalling of apparently deviant aspects.

• •

  Condensed, periodic reports on all findings of potential problems and oddities. These are compiled manually by the so-called First Look Scientists team and the wider Payload Experts Group (Section 1.2.2). If needed, these groups also propose actions to improve the performance of Gaia and of the data processing on ground. Such actions may include telescope refocussing, change of on-board calibrations and configuration tables, decontamination campaigns, improvements of the IDT configuration parameters, definition of break points in the calibrations, and many others.

• •

  Manual qualification of all First-Look data products used in downstream data processing (attitude, source parameters, and geometric instrument calibration parameters).

In this way, First Look ensures that Gaia achieves the targeted data quality, and also supports the cyclic processing systems by providing calibration data. In particular the daily attitude and star positions are used for the wavelength calibration of the photometric and spectroscopic instruments of Gaia. In addition, the manual qualification of First-Look products helps both IDT and the cyclic processing teams to identify and discriminate healthy and (partly) corrupt data ranges and calibrations. In this way, bad data ranges can either be omitted or subject to special treatment (possibly including a complete reprocessing) at an early stage.

The main objective of the AIM system is the independent verification of selected AF monitoring and diagnostics, of the image parameters determination, instrument modelling, and calibration. AIM can be considered as a scaled-down counterpart of IDT (see Section 3.4.2) and FL (see Section 3.5.2) restricted to some astrometric elements of the daily processing, namely those that are particularly relevant for the astrometric error budget. This separate processing chain runs in Gaia DR2 with the following six modules: (Astrometric) Raw Data Processing, Selection, Monitoring, Daily Calibration, CalDiagnostic, and Report; in addition, several CalDiagnostic tasks run off-line. Each AIM processing step can be divided into three main parts: input, processing, and output.

The BAM instrument (Section 1.1.3) is an interferometer monitoring the variation of the basic angle between the two telescopes (Section 1.1.3) by looking at the phase changes of the fringes (see also Section 3.4.4 and Gaia Collaboration et al. 2016b). The AVU/BAM system monitors, on a daily basis, the BAM instrument and the basic angle variation (BAV), independently from IDT (see Section 3.4.2) and FL (see Section 3.5.2). AVU/BAM provides periodic reports, for evaluation by the Payload Experts group (Section 1.2.2) and for information of relevant groups in DPAC (AGIS, GSR, and FL), trend analyses on short and long time scales, calibrated BAV measurements (since December 2014), as well as a model of the temporal variations of the basic angle.

AVU/BAM is a fundamental component of the technical and scientific verification of the overall Gaia astrometric data processing and has been developed within the context of the Astrometric Verification Unit (AVU). Deployment and execution of the operational system is done at the Torino Data Processing Centre, DPCT (Section 1.3.5).

The input data to AVU/BAM (coming from DPCE; Section 1.3.5) correspond to a central region of the fringe envelope for each line of sight. For Gaia DR2, the central region corresponds to a matrix of $1000\times 80$ samples (AL $\times$ AC). Each of the 80 AC samples is binned on-board (using 4 physical pixels per sample). BAM CCDs are red-variant CCDs (as used in RP) and have a pixel size of $10\times 30$ $\mu$m (AL $\times$ AC).

AVU/BAM, besides producing the fundamental BAM measurements (e.g., time series of the phase variation), processes the elementary signal and provides an interpretation in terms of a physical model to allow early detection of unexpected behaviour of the system (e.g., trends and other systematic effects). The AVU/BAM pipeline performs two different kinds of analysis: the first is based on daily runs while the second is focussed on overall statistics on a weekly/monthly basis.

The most important output of AVU/BAM is an estimate of the differential basic angle variation (BAV) with time. The pipeline provides this BAV estimate through three different algorithms (described in Riva et al. 2014). The first algorithm, named Raw Data Processing (RDP, similar to the IDT approach), provides the BAV as the difference of the variation of the two lines of sight. Each line-of-sight estimate is performed through a cross correlation of each BAM image with a template composed of the mean of the first 100 images of each run. The second algorithm, named Gaiometro, is the 1D direct cross correlation between the two lines of sight. The third algorithm, called Gaiometro 2D, is a 2D version of the direct measurement of the BAV. The first two algorithms use images binned across scan. The four independent results (three from AVU/BAM plus the one from IDT) agree well in general, providing similar shapes of the 6-hour basic-angle oscillations, although the derived amplitudes differ at the level of 5 percent. It is as yet unknown which of the four methods gives the most faithful representation of the actual variations in the basic angle of the astrometric instrument.

In addition to producing time series of the fringe phase variations, AVU/BAM also makes measurements of other basic quantities characteristic of the BAM instrument, like fringe period, fringe flux, and fringe contrast. The fringe flux is calculated as the raw pixel sum of each frame. The fringe period $x$ is given by:

with $F$ the telescope focal length, $\lambda$ the wavelength, and $b$ the BAM baseline. The fringe contrast is related to the visibility of the fringes. The temporal variations of these quantities are monitored and analysed to support the BAV interpretation.

No data produced by the AVU/BAM software has entered Gaia DR2.