7.4 Catalogue validation

The most fundamental validation tests verify that values given are not obviously wrong. This kind of check can catch simple mistakes in data manipulation, like missing data fields, wrong units, or erroneous conversions. Such tests are mandatory, because data pass between several data processing centres and through many steps after the specific processes (astrometry, photometry, etc.) before eventually reaching the data release data base.

Main data integrity and consistency tests are described in Section 2.2 of the Catalogue validation paper (Arenou et al. 2017), and we mention in what follows several other tests which were not discussed or illustrated in this publication.

The effective angular resolution of a catalogue is studied in Section 4.4.1 of the Catalogue validation paper using the distribution of the distances between pairs of sources.

The Gaia catalogue provides errors and goodness of fit metrics for its astrometric solution; the catalogue validation process included several checks of these errors that could be done without reference to external data. These included examining the astrometric goodness-of-fit fields and checking that solutions did not mix high and low precisions on different parameters. Gaia DR1 contains 5 and 2 astrometric parameter solutions for TGAS and Gaia secondary sources, respectively.

Focusing on the 5 parameter solutions for TGAS, they have been broken up into Hipparcos and Tycho-only subsets. A 0.012% of Hipparcos sources contain astrometric errors exceeding 10 mas in position, 10  mas yr${}^{-1}$ in proper motion, or 5 mas in parallax. Using the same criterion, 0.067% of the Tycho-only sources supersede some of these thresholds.

In Table 7.3 we compare the uncertainties for the remaining sources with the ones expected for a TGAS based on only 0.5 years of Gaia data, as discussed in Michalik et al. (2015, Tables 2 and 1). The actual TGAS errors for Tycho-only stars are better than expected, probably because we have 1.15 years of data instead of the 0.5 years in the simulation. The much smaller errors for Hipparcos are more susceptible to the imperfections of the Gaia DR1 processing and the (insufficient) five parameter model for astrometry.

In addition, we have checked for TGAS the size of the astrometric standard uncertainties as a function of the number of observations. An example is shown in Figure 7.2, where the red points correspond to solutions where ${\sigma }_{\alpha *}$ is further from the median value at that number of observations, than twice the distance from the median to the 2nd or 98th percentiles. The fraction of such outliers never exceeds 0.6% for any of the TGAS parameters. We note that due to the different scan orientations for different observations, we cannot expect a simple square root dependence with the number of observations. We also note, that for Tycho the proper motion errors are basically defined by the errors of Tycho-2 and the epoch difference to Gaia.

Roughly 10% of the TGAS sample combine small parallax errors with large errors in other astrometric parameters, particularly right ascension. This is a known effect of the Gaia scanning law to date, and should disappear in later releases.

The Gaia DR1 position errors have been analysed from a global perspective in two tests. The first one looks at the sky distribution of the errors as shown in Figure 7.3. As expected, the poorly scanned areas around the Ecliptic plane show the largest errors.

In the second test we looked at the ratio between right ascension and declination errors, and, as errors depend strongly on ecliptic coordinates, also at the ratio of the errors in ecliptic longitude and latitude. These ratios have been computed for three ranges of magnitudes: , , and , and a Kolmogorov-Smirnov test is used to check if the distributions in the three magnitude ranges are similar.

To transform the positional errors to ecliptic coordinates, we need the mutual orientation of the equatorial and ecliptic systems. Following the exposition in the Hipparcos catalogue (Vol. 1, Sect. 1.5.3 of ESA 1997), we denote the basis vectors in the equatorial system as $[𝒙𝒚𝒛]$; with $𝒙$ being the unit vector towards $(\alpha ,\delta )=(0,0)$, $𝒚$ the unit vector towards $(\alpha ,\delta )=({90}^{\circ },0)$ and $𝒛$ the unit vector towards $(\alpha ,\delta )=(0,{90}^{\circ })$; and denote basis vectors for ecliptic systems as $[{𝒙}_{\text{k}}{𝒚}_{\text{k}}{𝒛}_{\text{k}}]$, then, the direction vector $𝒖$ can be expressed in terms of equatorial and ecliptic coordinates as:

Then, it is possible to obtain equatorial unit vectors ($𝒑,𝒒$) perpendicular to $𝒖$ in the directions of increasing $\alpha$ and $\delta$ as defined in Equation 7.2 and Equation 7.3:

and

The coordinate transformations are treated in detail in Section 3.1.7. The transformation between the equatorial and ecliptic system can be written as

where matrix ${𝑨}_{\text{k}}$ for the present purpose can be approximated with the expression

where $ϵ={23}^{\circ }{26}^{\prime }{21.411}^{\mathrm{\prime \prime }}$ is the conventional value of the obliquity of the ecliptic.

As in Equation 7.2–7.3, we introduce unit vectors perpendicular to $𝒖$ in the directions of increasing $\lambda$ and $\beta$:

and

We now have the cosine and sine factors for the projections between the two systems:

and

The uncertainties in ecliptic coordinates are then:

and

where ${\rho }_{\alpha *}^{\delta }$ is the correlation between the errors for the equatorial coordinates.

Distributions analysed by the Kolmogorov-Smirnov test are shown in Figure 7.4. Equatorial and ecliptic errors have been treated separately. In both cases, the higher the $G$ magnitude range is, the larger ratios are obtained, but the test confirms that the distributions are essentially the same. The ecliptic errors show lower ratios as the scanning law is symmetric around the Ecliptic.

In addition, we have verified the continuity of the astrometric uncertainties as a function of the $G$ mean magnitude in order to check for rapid changes due to the use of different gates. The range of $G$ magnitudes taken into account was the one affected by gate effects (below 13 mag). No issues were found for any of the parameters.

CU9 validation conducted two internal tests of the parallaxes: one aimed at locating sky regions with relatively many negative parallaxes, and one testing the consistency of the standard uncertainty on the parallaxes with the observed parallax distribution.

For the first test we compute the fraction of TGAS sources with negative parallax in each level-4 HEALPix pixel (Górski et al. 2005) and flag those HEALPix pixels which have an unusually high or low fraction, given the average number of observations per source. As a result, 150 HEALPix pixels out of 3072 (5%) were flagged. Figure 7.5a illustrates the result, and we particularly notice the high fraction when there are less than 75 observations per source, corresponding to about 8 transits. The right-hand panel shows how the median parallax depends on the star density, where in general in the denser fields (Galactic plane) the relatively bright TGAS stars are further away.

The catalogue validation process checked the internal consistency of the standard uncertainty on the parallaxes by finding the parallax error level most consistent with the observed parallax distribution using a deconvolution algorithm based on Lindegren (1995). This internal estimation of the parallax uncertainty is described in Section 6.2.1 of the Catalogue validation paper (Arenou et al. 2017).

The TGAS parallax accuracy has also been tested using quasars, showing a significant non-null zero-point of the parallaxes together with local systematics, cf. Section 6.1.1 of the Catalogue validation paper.

Different strategies have been taken to test TGAS proper motions. We have looked for problematic sky regions with unusually many high proper motions and also for unrealistic individual proper motions.

We have taken high proper motion components to mean the ones exceeding $\pm 150$  mas yr${}^{-1}$ and determined the fraction of such proper motions in each of the 3072 level-4 HEALPix pixels (13.43 square degrees). We then flag as suspicious, the regions where this fraction is more than ten times the global average. As a result we flag seven plus one HEALPix pixels for the right ascension and declination proper motions, respectively.

Individual, high proper motions have been evaluated by computing their tangential velocity components, provided their parallaxes had uncertainties below 20%. We confirmed that no component corresponds to a tangential velocity exceeding significantly the escape velocity at the Sun position, estimated as about 500 km s${}^{-1}$.

We also looked at the dependence of the median proper motions with the sky density, as illustrated in Figure 7.6. We have flagged regions where the median reaches extreme values given the sky density, but the figure also shows that the flagged regions in reality follow the general distribution. The size of the regions is again 13.43 square degrees.

Test procedures carried out during internal catalogue validation, as described above, are focused on either individual sources or on specific sky regions. In the pixel analysis we summarise where in the sky sources are more often flagged, or where in the sky regions (HEALPix) are more often flagged. This summary was only carried out for TGAS, where more tests were performed.

For the tests based on sources, we find first the fraction of sources in each HEALPix pixel (level 4) that have been flagged at least 3 times in the various tests, see Figure 7.7a. In addition we identify in Figure 7.7b the pixels where this fraction is much higher than in typical pixels. We did a similar test for sources flagged at least 4 times, but these are too few to show a significant result.

For the tests based on sky regions, Figure 7.8a shows the number of times each pixel was flagged, and Figure 7.8b similarly the pixels flagged at least twice.

The tests presented in this section aim at the characterization of the object content of Gaia DR1:

1. 1.

   search for duplicate or missing entries;

2. 2.

   homogeneity of the sky distribution, detection of possible variations in different regions of the sky, for different magnitude, colour or proper motion ranges;

3. 3.

   small scale completeness of the catalogue for some selected samples and detection of possible variations as a function of magnitude or colour;

4. 4.

   performance of visual binaries observation as an indication of the spatial resolving power of Gaia DR1, tested versus separation and magnitude difference between components.

The comparison of Gaia results with external astrometric data is not straightforward as Gaia will provide the most accurate astrometric data ever produced, at least in the optical domain. An additional difficulty, especially for the comparison of parallaxes, will be that the numbers of targets will be hugely different: a few tens to a maximum of one hundred thousands for existing data versus millions — and finally a billion — for Gaia. However the consistency between Gaia data and carefully selected external astrometric data might be important in order to detect any statistical misbehaviour in one or the other source of data, including Gaia.

These tests compare the photometry of Gaia DR1, including TGAS, with external photometry. With the exception of the first test, performed in comparison with the Hubble stellar standards Calspec database, all other tests check the distribution of a mixed colour index Gaia magnitude minus the external catalogue magnitude in ordinate versus an external catalogue colour in abscissa. An empirical robust spline regression is derived which models the global colour-colour relation. The residuals from this model are then analysed as a function of magnitude, colour and sky position.

This is described in Section 7.3 of the Catalogue validation paper using HST CALSPEC standard star database, $BVRI$ photometric standard stars, Hipparcos $Hp$ magnitudes, SDSS photometry, Tycho-2 photometry and 2MASS photometry.

The aim of this test is two-fold: assessing the internal consistency of proper motions within stellar clusters, and looking for biases and systematics by testing the proper motions zero-point against literature values. Results are shown in Section 6.2.4 of the Catalogue validation paper (Arenou et al. 2017).

This test aims at assessing the internal consistency of parallaxes within a cluster, and checking the parallaxes against photometric distances in order to verify the zero-point of parallaxes. The procedure is analogous to the proper motion test described in 7.4.4 and the results are discussed in Section 6.1.4 of the Catalogue validation paper.

The aim of this test is to assess the completeness of the catalogue in crowded regions comparing Gaia data with HST photometry. We focused on very dense regions, in the inner 3 arcmin $\times$ 3 arcmin of the cores of a sample of 26 globular clusters. The data were acquired with the Wide Field Channel (WFC) of the Advanced Camera for Surveys (ACS) in the filters F606W and F814W. The tests are illustrated in Section 4.3.2 of the Catalogue validation paper together with the other tests using external catalogues.

The goal is to test the photometric accuracy and precision of Gaia against published photometry of stellar clusters. We made use of a sample of high quality photometry in $V$ band, highly homogeneous, both in data reduction and in zero point for all the clusters, for 5 open clusters, namely Hyades, Praesepe, Coma Ber, NGC752 and M67. In addition, we used M4 HST photometry. The results are shown in Section 7.4 of the Catalogue validation paper.

Two BGM simulations have been done for TGAS validation, using slightly modified models, both in density laws and kinematics, in order to verify the dependency of the model inputs to the validation. Both simulations were done with the model described in Czekaj et al. (2014) where the evolutionary scheme has been updated, as well as the IMF, SFR and evolutionary tracks. moreover, the thick disc and halo populations have been updated, following Robin et al. (2014), with new density laws. Concerning the kinematics, we used alternatively the standard model kinematics Robin et al. (2003), hereafter BGMBTG2, and a revised kinematics from an analysis of RAVE survey, hereafter called BGMBTG4. BGMBTG2 and BGMBTG4 also differ by several model parameters such as the extinction model and thin disc scale lengths. The use of two different models allows to evaluate what is due to acceptable model variations in the parallax and proper motion distributions.

Since Gaia DR1 contains only $G$ magnitude and positions, the validation with models consists in comparing the distribution of star density on the sky with a realization of the BGM one specifically for this purpose. It includes single stars, multiple systems, and incorporate a model for the expected errors on stellar parameters after the full 5 year Gaia mission. In the validation process, star counts as a function of positions and in magnitude bins are compared and shown in Section 4.2.3 of the Catalogue validation paper.

The statistical tests performed make use of the Kullback-Leibler Divergence (KLD), also known as “mutual information”, and allow for estimation of the degree of correlation or clustering in spaces that are combination of any number of the observables and of their errors. Given two distributions $p(𝐱)$ and $q(𝐱)$ the KLD is defined as

i.e., this statistic compares the information content of $p(𝐱)$ with respect to $q(𝐱)$. In the tests reported below, $q(𝐱)$ is defined as the 1-dimensional marginalised distribution of each observable (or independent variable). That is, if $𝐱=(x_1,x_2,\mathrm{\dots },x_n)$, then $q(𝐱)={\mathrm{\Pi }}_i{p}_i(x_i)$ where $p_i(x_i)=\int {\mathrm{\Pi }}_{j\ne i}d{x}_{j}p(𝐱)$. If there is clustering or correlations between the variables, then $D_{KL}$ will be large, while it will be zero if they are uncorrelated.

The tests described below have been applied only to two-dimensional subspaces (i.e., pairs of observables). Since certain subspaces are known to be naturally clustered, for example in the subspace of ra vs dec stars are not randomly distributed, there is a need to compare to simulations or models to establish whether the KLD value obtained is reasonable or not, and hence the subspace should be flagged. Therefore, the values of the KLD are themselves not used in an absolute sense in the tests below, but they are ranked. It is their rankings that are then compared to the simulations or models, or other datasets.