Author(s): Jordi Portell, Claus Fabricius, Javier Castañeda
As previously explained, there are good reasons for performing a daily pre-processing, even if that occasionally leads
to not fully consistent data outputs. Such inconsistencies are fixed regularly in the cyclic pre-processing task. Some algorithms
and tasks are only run in the daily systems (mainly the raw data reconstruction, which feeds all of the DPAC
systems), whereas other tasks can only reliably run on a cyclic basis over the accumulated data.
There are also intermediate cases, that is, tasks that must run on a daily basis but over consolidated
inputs. That is achieved by means of the First-Look (FL) system, which is able to generate some preliminary
calibrations and detailed diagnostics.
Table 2.2 provides an overview of the main tasks executed in these two types
of data pre-processing systems. Please note that some of the ‘Final’ tasks mentioned in IDU are not included
in the present release, in particular the Astrometric LSF calibration. The final determination of spectro-photometric image parameters
(that is, BP/RP processing) is done in PhotPipe (see Chapter 5).
Table 2.2: Main pre-processing steps and their execution in daily or cyclic systems. ‘Final’ means that the outputs
generated by that task are not updated any more (unless in case of problems or bugs), whereas ‘Preliminary’ means
that a first version of a data output is later updated or improved.
Raw data reconstruction
Basic angle variations determination
On-ground attitude reconstruction
Bias and astrophysical background
Astrometric LSF calibration
Spectro-photo image parameter determination
Astrometric image parameter determination
Initial Data Treatment (IDT)
IDT includes several major tasks. It must establish a first on-ground
attitude (see Section 2.4.5), to know where the
telescopes have been pointing at any moment; it must calibrate the CCD bias,
and it must calibrate the
sky background (see Section 2.4.6). Only with those
pieces in place, one can start thinking of processing actual observations.
For the observations, the first thing is to reconstruct all relevant
circumstances of the data acquisition, as explained in
Section 2.4.3. From the and fluxes, one can
determine a source colour, and then proceed to determine the image
parameters (see Section 2.4.8).
The final step of IDT is the crossmatch between the on-board detections
and a catalogue of astronomical sources, having filtered detections deemed
spurious (see Section 2.4.9). One catalogue source is
assigned to each detection, and if no one is found, a new source is added.
Intermediate Data Updating (IDU)
The Intermediate Data Updating (IDU) is one of the most demanding systems across DPAC in terms of
data volume and processing power. IDU includes some of the most challenging Gaia calibration tasks and
aims to provide:
An updated crossmatch table using the latest attitude, geometric calibration, and source catalogue available.
Updated calibrations for CCD bias and astrophysical background (see Section 2.4.6).
An updated instrument LSF/PSF model (see Section 2.3.2).
Updated image parameters: locations and fluxes (see Section 2.4.8).
All these tasks have been integrated in the same system due to the strong relation between them. They are also run in the same
environment, at the Marenostrum supercomputer hosted by the Barcelona Supercomputing Centre (BSC, Spain; Section 1.3.4). This symbiosis
facilitates the delivery of suitable observations to the calibrations, and of calibration data to IDU tasks.
As anticipated in Section 2.1.1, IDU plays an essential role in the iterative data reduction;
the successive iterations between IDU, AGIS, and PhotPipe (as shown in Figure 2.8)
enable achieving the high accuracies envisaged for (the intermediate and) final Gaia catalogue(s).
IDU incorporates the astrometric solution from AGIS, resulting in an improved crossmatch, but also
incorporates the photometric solution from PhotPipe within the LSF/PSF calibration, resulting in improved image parameters.
These improved results are the starting point for the subsequent iterative reduction loop.
In Gaia DR1, the image parameters came from IDT. However, for Gaia DR2, all image parameters result from IDU, thereby providing
an improved completeness and consistency.