2.4.2 Daily and cyclic processing

Author(s): Jordi Portell, Claus Fabricius, Javier Castañeda

As previously explained, there are good reasons for performing a daily pre-processing, even if that occasionally leads to not fully consistent data outputs. Such inconsistencies are fixed regularly in the cyclic pre-processing task. Some algorithms and tasks are only run in the daily systems (mainly the raw data reconstruction, which feeds all of the DPAC systems), whereas other tasks can only reliably run on a cyclic basis over the accumulated data. There are also intermediate cases, that is, tasks that must run on a daily basis but over consolidated inputs. That is achieved by means of the First-Look (FL) system, which is able to generate some preliminary calibrations and detailed diagnostics. Table 2.2 provides an overview of the main tasks executed in these two types of data pre-processing systems. Please note that some of the ‘Final’ tasks mentioned in IDU are not included in the present release, in particular the Astrometric LSF calibration. The final determination of spectro-photometric image parameters (that is, BP/RP processing) is done in PhotPipe (see Chapter 5).

Table 2.2: Main pre-processing steps and their execution in daily or cyclic systems. ‘Final’ means that the outputs generated by that task are not updated any more (unless in case of problems or bugs), whereas ‘Preliminary’ means that a first version of a data output is later updated or improved.
Task Daily (IDT) Daily (FL) Cyclic (IDU)
Raw data reconstruction Final
Basic angle variations determination Final
On-ground attitude reconstruction Prelim. (OGA1) Prelim. (OGA2)
Bias and astrophysical background Prelim. Final
Astrometric LSF calibration Prelim. Final
Spectro-photo image parameter determination Prelim.
Astrometric image parameter determination Prelim. Final
Crossmatch processing Prelim. Final

Initial Data Treatment (IDT)

IDT includes several major tasks. It must establish a first on-ground attitude (see Section 2.4.5), to know where the telescopes have been pointing at any moment; it must calibrate the CCD bias, and it must calibrate the sky background (see Section 2.4.6). Only with those pieces in place, one can start thinking of processing actual observations.

For the observations, the first thing is to reconstruct all relevant circumstances of the data acquisition, as explained in Section 2.4.3. From the GBP and GRP fluxes, one can determine a source colour, and then proceed to determine the image parameters (see Section 2.4.8).

The final step of IDT is the crossmatch between the on-board detections and a catalogue of astronomical sources, having filtered detections deemed spurious (see Section 2.4.9). One catalogue source is assigned to each detection, and if no one is found, a new source is added.

Intermediate Data Updating (IDU)

The Intermediate Data Updating (IDU) is one of the most demanding systems across DPAC in terms of data volume and processing power. IDU includes some of the most challenging Gaia calibration tasks and aims to provide:

  • •

    An updated crossmatch table using the latest attitude, geometric calibration, and source catalogue available.

  • •

    Updated calibrations for CCD bias and astrophysical background (see Section 2.4.6).

  • •

    An updated instrument LSF/PSF model (see Section 2.3.2).

  • •

    Updated image parameters: locations and fluxes (see Section 2.4.8).

All these tasks have been integrated in the same system due to the strong relation between them. They are also run in the same environment, at the Marenostrum supercomputer hosted by the Barcelona Supercomputing Centre (BSC, Spain; Section 1.3.4). This symbiosis facilitates the delivery of suitable observations to the calibrations, and of calibration data to IDU tasks.

As anticipated in Section 2.1.1, IDU plays an essential role in the iterative data reduction; the successive iterations between IDU, AGIS, and PhotPipe (as shown in Figure 2.8) enable achieving the high accuracies envisaged for (the intermediate and) final Gaia catalogue(s).

Figure 2.8: Diagram of the IDU, AGIS, and PhotPipe system operation and interdependency, from raw data until the updated source catalogue updates.

IDU incorporates the astrometric solution from AGIS, resulting in an improved crossmatch, but also incorporates the photometric solution from PhotPipe within the LSF/PSF calibration, resulting in improved image parameters. These improved results are the starting point for the subsequent iterative reduction loop.

In Gaia DR1, the image parameters came from IDT. However, for Gaia DR2, all image parameters result from IDU, thereby providing an improved completeness and consistency.