2.2 Properties of the input data

Typically, one SP of one or more type is generated for every astronomical source transit across the Gaia focal plane. That is to say, every time that a VPU detects, confirms and measures the transit of a source with enough brightness and sharpness. Some of these packets are only generated during special calibration or non-nominal activities. We can classify Star Packets as follows:

• •

  Nominal astronomical packets:

  • –

    SP1, the most numerous ones, with one packet generated for each astronomical source transit across the SM, AF and BP/RP CCDs. These form the main data input to all Gaia data processing systems.

  • –

    SP2, same as for SP1 but only for those sources detected in the focal plane rows that have RVS CCDs, and only for sources which are bright enough for being measured there.

  • –

    SP3, generated only for SP1 packets for which a significant across-scan motion has been autonomously detected on board. These are called Suspected Moving Objects (SMO).

• •

  Nominal instrumental packets:

  • –

    SP4, with regular (periodic) measurements from the Basic Angle Monitoring (BAM) device. That is, although these are labelled as ‘Star Packets’, these do not contain any astronomical information, but instrumental information instead.

• •

  Non-nominal astronomical packets:

  • –

    SP6 and SP7, with SM and AF1 measurements of bright stars, which are only generated when the on-board Attitude and Orbit Control System (AOCS) is being initialised and when it loses convergence momentarily. These are mainly down-linked for further analysis and checks, but they do not enter the main data processing pipelines.

  • –

    SP8 and SP9, with AF1 or SM measurements of bright stars, also only generated in special on-board conditions and not entering the main data processing pipelines.

• •

  Non-nominal instrumental packets:

  • –

    SP5, with measurements from the Wave Front Sensor (WFS) monitoring device. As with SP4, these do not actually contain any astronomical information, but instrumental only.

Regarding the ASD packets, they are generated as follows:

• •

  ASD1, with the across-scan window position information for most CCDs, generated every second for each VPU.

• •

  ASD2, with electronic bias data (pre-scan pixels), generated periodically (about once per minute per VPU).

• •

  ASD3, with information on the RVS resolution changes, generated every time that a bright-enough star is observed in an RVS CCD. Note that very early in the mission it was decided to use always the high-resolution acquisition mode in those CCDs, and thus these packets are not available for most of the mission.

• •

  ASD4, with statistical information and counters on some on-board events, generated periodically (once every number of seconds).

• •

  ASD5, with the times when the artificial Charge Injections (CI) have been applied on the CCDs, generated at a quite regular pace (once every number of seconds).

• •

  ASD6, with the information on the gates activation in the CCDs (to reduce the integration time). These are generated every time that a bright-enough star (about 12) is observed in a CCD.

• •

  Finally, ASD7, with information (time and position) when any SP1, SP2 or SP3 measurement was acquired by any VPU. Therefore, these ASD7 packets are generated for each source transit (although an ASD7 packet actually contains information on a set of transits).

The following are the specific contents of each of the raw telemetry packets down-linked by Gaia:

• •

  SP1: These are the most important data packets. One SP1 packet contains the SM, AF, BP and RP samples from one source transit over the focal plane. Only small ‘windows’ of samples are acquired and transmitted, centred by the VPU algorithms on the astronomical source detected.

  • –

    For SM, windows of 40$\times$6 samples (each 2$\times$2 pixels) are sent, thus covering an area of about 4.7$\times$2.1 arcsec${}^2$.

  • –

    For AF CCDs, the exact shape of the windows depends on the CCD (AF1 to 9) and on the source brightness, but typically, windows of 12$\times$12 pixels are sent (with the AC pixels binned into one single sample, thus providing only 12 samples in the AL or scanning direction). Bright stars (16) are acquired with slightly larger windows (18$\times$12 pixels), with the brightest sources (13) being acquired in full 2D resolution. Thus, AF windows typically cover about 700$\times$2100 mas${}^2$ (raising to about 1$\times$2.1 arcsec${}^2$ for bright sources).

  • –

    Finally, BP and RP windows cover 60$\times$12 pixels, with 2D resolution only for the brightest sources (11).

  • –

    Besides the raw samples, these packets also include the time, FOV, CCD Row and AC pixel where the source was detected (all based on the AF1 CCD), an on-board estimated magnitude, and information about the exact sampling scheme used — including information on possible window overlaps with (or by) another source.

• •

  SP2: These packets also include the basic detection and measurement information as SP1 packets, but the only sample data included is from the RVS CCDs. Three windows are included (for each of the 3 along-scan RVS CCDs used for a spectroscopic transit), each covering a large area of 1296$\times$10 pixels (about 76.4$\times$1.8 arcsec${}^2$). Full 2D resolution is only used for the brightest stars.

• •

  SP3: These packets are complementary to SP1 packets for Suspected Moving Objects (objects for which the VPU has detected an AC motion from SM to AF1). Here, only basic detection information is included (as in SP2), and the only sample data is that from additional BP and RP windows placed on top (or bottom) of the nominal BP/RP windows already included in the associated SP1 packet.

• •

  SP5: WFS data packets include windows of about 682$\times$120 pixels plus timing and position information.

• •

  SP6, SP7, SP8 and SP9 packets are non-nominal. They include, besides timing, position and measurement information, windows of SM and AF samples (SP6 and SP7), AF1 samples (SP8) or SM samples (SP9).

• •

  ASD1: Each of these packets includes, besides a reference time and the CCD row number, the across-scan (AC) shift in pixels with respect to the AF1 reference coordinate for each of the CCDs (except AF1) and for each field of view, indicated in pixels. Thus, combining the adequate ASD1 packet with an SP1 (or SP2 or SP3) packet one can determine the absolute AC position where each window was acquired.

• •

  ASD2: Each ASD2 packet contains a ‘burst’ of pre-scan samples acquired on a given CCD. Thus, these contain a CCD identifier, a reference time, and a set of typically 1024 samples.

• •

  ASD3: These contain a reference time and CCD identifier, plus the resolution switch type (low-to-high or high-to-low).

• •

  ASD4: There are two variants of these packets, both containing a large set of on-board counters (per VPU), such as the number of detected, confirmed and allocated objects (that is, transits of astronomical sources) for the different types of windows and per field of view, or the number of packets generated of each SP/ASD type.

• •

  ASD5: Each of these packets holds a number of times when a charge injection was generated (w.r.t. a packet reference time) for a given CCD. One of these packets can cover a few seconds (up to some 1–2 minutes, depending on the configuration).

• •

  ASD6: Each ASD6 packet indicates the gate configuration that was active for a given CCD at a given time. Thus, a new ASD6 packet is generated every time that a gated window (a window acquired with a shorter integration time, for a bright source) is started or finished. Note that it means that a given bright star, causing the activation of some gate, will also affect other sources being observed on the same CCD at and immediately around that time.

• •

  ASD7: Finally, each ASD7 (or object log) packet contains a reference time plus a set of up to 3071 entries, each corresponding to one SP1, SP2 or SP3 packet allocated on board for the measurement of a source transit. Each of such entries indicates the detection features (time, coordinate, FOV, acquisition mode, etc.) and the brightness of the source.

It is worth noting that, besides the raw data from the spacecraft in itself, other ancillary data is also needed when performing the raw measurements reconstruction. Such data is stored in the so-called Calibration DataBase (CDB), which contains, for example, the Along-scan Phasing Table (ALPT) from which, when combined with an AF1 detection time, we can determine the absolute measurement times for each of the windows in an SP1, SP2 or SP3 packet. Several other tables are also needed from the CDB, such as those that indicate the exact configuration for the Gates, Charge Injections, etc. Needless to say, the CDB must be perfectly synchronised with the actual configuration active on board Gaia.

All the raw science telemetry data previously described is only used in the very first pre-processing stages of the Gaia DPAC — mainly in the Initial Data Treatment system (IDT) (see Section 2.4.2). Such a system takes care of combining all these data packets to generate self-consistent measurement records, which become the basic input data to further downstream systems — be these for astrometric, photometric or spectroscopic processing. For more details see Section 2.4.3.

The Initial Gaia Source List (IGSL) was commissioned by the Gaia Data Processing and Analysis Consortium (DPAC) in 2006 to be a combination of the best optical astrometry and photometry information on celestial objects available at the Gaia launch: A snapshot of the sky as we know it before Gaia. The method adopted was to crossmatch large-area star catalogues into one database, then select the best parameters based on the typical precisions for each contributing catalogue. The 3rd delivery of the IGSL was made in late 2012, and it was at that point frozen to be fully integrated into the MDB before launch.

The formal DPAC mandate for the IGSL was to fulfil the following broad requirements: provide all-sky positions, proper motions, and magnitudes for objects to a limit of Gaia magnitude $G$=21 where possible, e.g., where there are large ($>$10 000 square degrees) catalogues that reach that limit. The proper motions and magnitudes are to be provided on a best-effort basis, nominally with precisions of 10 mas yr${}^{-1}$and 0.3 magnitudes, respectively, but obviously limited by the currently available large catalogues. The DPAC Core Processing Coordination Unit (CU3) catalogues of quasi-stellar objects (QSOs) and the Ecliptic Poles catalogue should be included (with no selection on magnitudes) to directly support the CU3 processes that require those resources. The Hipparcos objects were included with no selection on magnitudes to aid in the production of the Hundred Thousand Proper Motions Catalogue (de Bruijne and Eilers 2012; Michalik et al. 2014).

The format and contents of the IGSL are described in Smart and Nicastro (2014). After extensive use within the mission a number of problems were discovered. Such known problems are collected and made available in the documentation on the IGSL webpage.

The contents of the IGSL are a compilation of the following catalogues:

• •

  GSC2.3 — The Second Guide Star catalogue version 2.3 (Lasker et al. 2008);

• •

  Tycho-2 — (Høg et al. 2000);

• •

  UCAC4 — USNO CCD Astrograph catalogue version 4 (Zacharias et al. 2013);

• •

  SDSS — Sloan Digital Sky Survey DR9;

• •

  2MASS — Two Micron All-Sky Survey Point Source catalogue (Skrutskie et al. 2006);

• •

  PPMXL — Positions and Proper Motions ‘Extra Large’ catalogue (Roeser et al. 2010);

• •

  GEPC — The Gaia Ecliptic Poles Catalog, version 3.0;

• •

  LQRF — The CU3 early version of Large Quasar Reference Frame (Andrei et al. 2009);

• •

  OGLE — Optical Gravitational Lensing Experiment version III (Udalski et al. 2008a);

• •

  Hipparcos Perryman et al. (1997); van Leeuwen and Fantino (2005); van Leeuwen (2007);

• •

  Sky2000 the SKYMAP Master catalogue of bright stars, Version 4 (Myers et al. 2001);

• •

  SPSS the Gaia spectrophotometric standard star catalogue (Pancino et al. 2012).

For details see Smart and Nicastro (2014).

The IGSL was and is being used for the initial (partly preliminary) assignment of individual Gaia observations to known astronomical objects in the sky (not including solar-system objects). The process doing this assignment is called crossmatch (or sometimes crossmatching) in the Gaia jargon.

Although Gaia in the end will create a completely independent and self-contained all-sky inventory of astronomical objects — not relying on any pre-launch knowledge — it was deemed useful to have such an initial list, for two main reasons:

• •

  An assignment to known celestial objects is needed internally for the initial calibration and verification of the spacecraft and instruments.

• •

  Giving pre-defined Gaia source identifiers to known celestial sources (and publishing these in the form of the IGSL) will allow the members of the external scientific community to prepare specific object lists and auxiliary data for specific research topics — and then easily identify these objects in the Gaia catalogue by just using those pre-known ’names’ (source identifiers).

The IGSL was first delivered in 2007 and two other versions were delivered before the frozen version in 9/2013. Subsequent use has revealed a number of problems. Most would be relatively simple to fix but there was no provision made for updating the IGSL hence all downstream Gaia processing had to deal with these problems and we collect the known issues here to have them in a central location.

The GEPC consists of two $\simeq$1 square degrees fields centred on the ecliptic poles themselves.

The southern field, or SEP-field, was observed with the MPIA 2.2 m telescope at La Silla in Chile and its WFI detector, which covers $\simeq 0.5$ ${}^{\circ }$$\times 0.5$ ${}^{\circ }$. To fully cover the 1 square degree field as required, observations were done using 5 pointings, one centred on the pole and the other four being tiled so that they fill $\simeq 60$ ${}^{\prime }$$\times 60$ ${}^{\prime }$with some degrees of overlap between them. Observations were done in Bessel $BVRI$ and calibrated to Landolt Standard fields into the Vega magnitude system and then transformed to Gaia magnitudes ($G,G_{\mathrm{BP}},G_{\mathrm{RP}},G_{\mathrm{RVS}}$).

The limiting magnitude in $V$ and $R$ and thus $G$ is roughly 22.5 mag. Centred on $G\simeq$ 18.5 there is a peak in the magnitude distribution, see Figure 2.2. This peak is real, it is caused by the LMC’s Red Clump gint stars, which is a very prominent population in this field.

The northern field was observed with the 3.6 m CFHT located on Mauna Kea (Hawaii, USA) and its MEGACAM detector. As the field of view of this device is already one square degree, observations were carried out without a pointing pattern, only a five-times dithering pattern. Filters used were SDSS $ugri$ in this case, and the $z$ band was incorporated from Hwang et al. (2007). Our own data was calibrated into the system of Hwang et al. (2007). In contrast to the SEP-field, the photometric zero points are for the $AB$ system, as generally in SDSS-type photometric fields. Again, the photometry is transformed into Gaia magnitudes. Due to the larger telescope the faint limiting magnitude for the NEP-field is about 26 in $g^{\prime }$ and $r^{\prime }$ and thus $G$, the limit of completeness being about 24 mag. For some stars the NEP field has proper motions, which were derived using the first epoch material from the POSS, taken from the Minnesota Automated Plate Scanner (MAPS), see Pennington et al. (1993); Cabanela et al. (2003). The plate in question (P72) was taken on August 18, 1952, allowing for a epoch baseline of roughly 56 years.

Both fields have some gaps, as can be seen in Figure 2.1. In the case of the northern field these gaps are caused by the 5 point dither pattern which is not sufficient to close all gaps in this $4\times 8$ detector array. Other gaps in the north and also those present in the southern field are due to matching criterion used in assembly. These gaps appear where the gaps between detectors are least covered by the dithering, and objects are partly only on one image of a set of five. In order to prevent too many false positives, which would have been detrimental for the commissioning process of Gaia, objects only on one image were discarded.

The GEPC contains positional astrometry (and proper motions for a smaller subset in the NEP-field, see Sect. Section 2.2.4.) and multi-pass-band photometry of 612,946 objects, of these 448,478 are located in the southern field, and 164,468 in the north. This discrepancy is caused by the presence of the LMC in the south, which outweighs the significantly fainter magnitude limit in the north. Note that the photometry has different characteristics in the two fields (filter pass band system, magnitude system), as described in Sect. Section 2.2.4. The Gaia magnitudes however are comparable. The current version is GEPC3.0 which has been incorporated into the IGSL, and with the IGSL into the Main Database. For details of the IGSL, see Section 2.2.3 and Smart and Nicastro (2014).

The format of GEPC3.0 is given in Table 2.4.

The EPC was most prominently used in the commissioning phase via the Ecliptic-Poles Scanning Law and the First-Look software system

• •

  to verify and quantify the efficiency of the on-board star image detection algorithms and Sky Mapper CCDs,

• •

  to perform initial measurements of the photometric throughput of the telescopes, CCDs and pre-amplifier electronics

• •

  to adjust the lower threshold of the on-board star image detection algorithms

Since the start of the nominal mission, the first and third of these items are re-checked whenever the scanning law touches the ecliptic poles — which is quite regularly about once per month for 1–2 scans per pole and field of view each — while the second item is now being covered by a daily comparison of bright-star measurements with independent Tycho-2 magnitudes. All this is part of the daily First-Look data processing.

The source catalogues used and their order of inclusion were:

• •

  Hipparcos (Perryman et al. 1997): The photometry from the original Hipparcos Catalogue and the astrometric parameters from the update by van Leeuwen (2007) when published. Initially all entries were included regardless of the known errors, e.g. also for entries that are considered erroneous. Since inclusion in the ASC requires an estimate of the $G$-mag the unreal entries were excluded as part of the cleaning phase.

• •

  Tycho-2 (Høg et al. 2000): This catalogue forms the backbone of all the major ground based catalogues currently available. It was made from a combination of the Tycho star mapper observations on the Hipparcos satellite (Høg et al. 1997), the Astrographic Catalogue and 143 other ground-based catalogues.

• •

  Sky2000 (Myers et al. 2001): The SKYMAP Star Catalogue System is a list of all stars with either measured Johnson blue or visual magnitudes brighter than 9.0. The version used here had 299167 entries of which 212 were not in the combined Hipparcos + Tycho-2 catalogues. Sky2000 provides positions at 2000, proper motions and a blue and visual magnitude. We assumed the positions to have an error of 100 mas, the proper motions an error of 10 mas yr${}^{-1}$, and an error of 0.6 in the ASC magnitudes derived from the Sky2000 values.

• •

  UCAC4 (Zacharias et al. 2013): The USNO CCD Astrograph Catalogue version 4 is the most precise all-sky astrometric catalogue in the range V=10–16 currently available. There are no original standard magnitudes in this catalogue.

• •

  GSC2.3 (Lasker et al. 2008): The Second Guide Star Catalogue version 2.3 forms the bulk of the photometry and defines the red and blue magnitudes ($B_J$ and $R_F$) as this is the sky survey with the largest coverage on a precise homogeneous photometric system. The only variation with the public version is that we removed the multiple entries discussed in section 4.2 of Lasker et al. (2008). This was done by insisting that only one entry from any objects with position differences of less than 10 mas were kept selecting Tycho-2 or Sky2000 over other entries.

• •

  PPMXL (Roeser et al. 2010): The Positions and Proper Motions ‘Extra Large’ Catalogue,produced from a combination of the USNO–B (Monet et al. 2003) and the Two Micron Sky Survey point source catalogue (Epchtein et al. 1999). This catalogue was included to provide magnitudes for those entries that did not have them in the previous catalogues.

In addition any objects in the Washington Double Star catalogue (Mason et al. 2010) or the Tycho Double Star catalogue (Fabricius et al. 2002) were indicated as probable members of a binary system.

The first version of the ASC was a subset of the IGSL and consequently was derived using the procedure in Smart and Nicastro (2014). In summary we produced a master list of objects starting with the large faint catalogues, progressively adding other catalogues and increasing the master list as entries from new catalogues were unmatched. The catalogues of bright objects were then matched to a large master list which resulted in mismatches of the bright objects to noise or faint objects near the true bright objects. Also it was found that the large Schmidt catalogues in the overlap region between plates often had many multiple entries of the same objects, this can be seen in the sky plot of the PPMXL. These multiple entries, if they were bright enough, were included as ASC sources.

The first on-ground attitude reconstruction of Gaia is described in detail in Gaia DR1 papers. The goal of this attitude reconstruction is to provide the attitude with an accuracy of 50 mas for the first year when the ASC will be the primary source of reference objects. Later in the mission it is planned to replace this catalogue with one produced by Gaia with an expected accuracy of 5 mas. This reconstruction requires at least one 2D measurement per second and per field of view which equates into a minimum density of 75 stars per square degree.

To be automatically assigned a 2D window the star must have a  13 but this does not provide enough calibration sources especially near the galactic poles. The compromise was to provide a list of faint calibration stars to a $G$=13.4 which sets the limiting magnitude of the ASC. Note that, originally the limit of the ASC was set to $G$=14.0 however a change of the procedure allowed a relaxation of that requirement to $G$=13.4.

The crossmatching radius in the first on-ground attitude reconstruction will be between 20–30 ${}^{\mathrm{\prime \prime }}$, the precise value to be optimized during the commissioning phase. Hence we conservatively require that all ASC sources are isolated at the level of 40 ${}^{\mathrm{\prime \prime }}$. This would potentially allow up to 8000 ASC entries per square degree and not violate the isolation criteria. Following this consideration, in the original subset of the IGSL that constituted the ASC, we just lowered the magnitude limit to reduce the number of stars to less than 1000. We then assumed the isolation criteria would always be met when there were this many objects per square degree. However, because of the multiple entries, uncatalogued binary systems and general non uniform distribution it was found that the isolation requirement of the ASC subset was violated.

To address the isolation and duplicate entry issues the ASC was reproduced from scratch using the catalogues listed above in the order given. The production of the ASC starts with all objects in the Hipparcos catalogue as a master list, the other catalogues are input and matched to this master list with a matching radius of 5 ${}^{\mathrm{\prime \prime }}$. All entries from the input catalogue not matched are included as new master list objects. If more than one entry from the input catalogue matches the master list only the closest is considered matched and a new entry is generated for the others.

In this way the master list grows with each included catalogue. Since the first catalogues are composed of bright objects they are sparse and the chances of a mismatch between the input catalogues and the master list was reduced. The confusion at the bright end of the master list was in this way minimized. When the large, dense Schmidt plate based catalogues are included there is still the possibility that non-real entries are matched to bright objects and the real bright object in the GSC23/PPMXL enter as new entries. However, the Schmidt data is only used to provide photometric information and to clean up the ASC list we drop any objects that are not in either UCAC4, Tycho-2, sky2000 or the Hipparcos catalogues under the assumption that the union of these catalogues are complete to fainter than the Gaia isolation limit of $G$=13.7.

Once the master list was completed with the compilation of all the catalogues we estimated the red $R_F$, blue $B_J$, Gaia $G$ and Gaia $G_{RVS}$ using the relations and priorities in Smart and Nicastro (2014) with the photometry from the contributing catalogues. We then dropped any objects fainter than $G$=13.7. This compilation and selection criteria results in 15 million objects. We assume all objects are stellar and then examine each object one-by-one and indicate for each object the number of neighbours within 40 ${}^{\mathrm{\prime \prime }}$.

From this list we drop any star with a (i) neighbour, (ii) or $G>13.4$ or (iii) in the Washington Double Star or Tycho Double Star catalogues.

The Attitude Star Catalogue was made by combining 7 all sky catalogues and selecting entries based on magnitude, isolation and astrometric precision criteria. The catalogue has 8 173 331 entries with estimates of the positions at 2000, proper motions and magnitudes (Gaia $G$, Gaia $G_{rvs}$, red $R_F$ & blue $B_J$) in the magnitude range .

Throughout the commissioning phase and scientific mission of Gaia, the ASC is used as the astrometric reference in the first on-ground attitude reconstruction, OGA1, see Section 2.4.5. At some time the ASC as described above will be replaced by a similar star catalogue derived from Gaia observations, i.e. as an excerpt from one of the early Gaia catalogues.