3.4.2 Daily and cyclic processing
Author(s): Jordi Portell, Claus Fabricius, Javier Castañeda
As previously explained, there are good reasons for performing a daily pre-processing, even if that occasionally leads to not fully consistent data outputs. Such inconsistencies are fixed regularly in the cyclic pre-processing task. Some algorithms and tasks are only run in the daily systems (mainly the raw data reconstruction, which feeds all of the DPAC systems), whereas other tasks can only reliably run on a cyclic basis over the accumulated data. There are also intermediate cases, that is, tasks that must run on a daily basis but over consolidated inputs. That is achieved by means of the First-Look (FL) system, which is able to generate some preliminary calibrations and detailed diagnostics. Table 3.5 provides an overview of the main tasks executed in these two types of data pre-processing systems. The final determination of geometric calibration is done in AGIS (see Section 4.3.6), whereas spectro-photometric image parameters (that is, BP/RP processing) is done in PhotPipe (see Chapter 5).
|Task||Daily (IDT)||Daily (FL)||Cyclic (IDU)|
|Raw data reconstruction||Final|
|Basic angle variations determination||Final|
|On-ground attitude reconstruction||Prelim. (OGA1)||Prelim. (OGA2)||Prelim. (SDMOGA)|
|Bias and non-uniformity||Prelim.||Prelim.||Final|
|CCD health, charge injection and release||Prelim.||Final|
|Astrometric LSF calibration||Prelim.||Final|
|Spectro-photo image parameter determination||Prelim.|
|Astrometric image parameter determination||Prelim.||Final|
Initial Data Treatment (IDT)
IDT includes several major tasks. It must establish a first on-ground attitude (see Section 3.4.5), to know where the telescopes have been pointing at any moment; it must calibrate the CCD bias (see Section 3.3.2), and it must calibrate the sky background (see Section 3.3.4). Only with those pieces in place, one can start thinking of processing actual observations.
For the observations, the first thing is to reconstruct all relevant circumstances of the data acquisition, as explained in Section 3.4.3. From the and fluxes, one can determine a source colour, and then proceed to determine the image parameters (see Section 3.4.11).
The final step of IDT is the cross-match between the on-board detections and a catalogue of astronomical sources, having filtered detections deemed spurious (see Section 3.4.13). One catalogue source is assigned to each detection, and if no one is found, a new source is added.
Intermediate Data Updating (IDU)
The Intermediate Data Updating (IDU) is one of the most demanding systems across DPAC in terms of data volume and processing power. IDU includes some of the most challenging Gaia calibration tasks and aims to provide:
updated on-ground attitude (see Section 3.4.5),
updated cross-match table using the latest attitude, geometric calibration, and source catalogue available (see Section 3.4.13),
updated instrument LSF/PSF model (see Section 3.3.5),
updated image parameters: locations and fluxes (see Section 3.4.11).
All these tasks have been integrated in the same system due to the strong relation between them. They are also run in the same environment, at the Marenostrum supercomputer hosted by the Barcelona Supercomputing Centre (BSC, Spain; Section 1.3.5). This symbiosis facilitates the delivery of suitable observations to the calibrations, and of calibration data to IDU tasks.
As anticipated in Section 3.1.1, IDU plays an essential role in the iterative data reduction; the successive iterations between IDU, AGIS, and PhotPipe (as shown in Figure 3.11) enable achieving the high accuracies envisaged for (the intermediate and) final Gaia catalogue(s).
IDU incorporates the astrometric solution from AGIS, resulting in an improved cross-match, but also incorporates the photometric solution from PhotPipe within the LSF/PSF calibration, resulting in improved image parameters. These improved results are the starting point for the subsequent iterative reduction loop.
In Gaia DR1, the image parameters came from IDT. However, from Gaia DR2 onward all image parameters result from IDU, thereby providing an improved completeness and consistency.