7.2.2 Data accountability
Author(s): Alex Hutton and Enrique Utrilla
The integration process takes sources from different datasets and combines them based on their source ID. During this process, using information provided by the IDU cross-match, previously existing sources can be deleted e.g. if they were based on observations that have been identified as spurious. Other sources might have been tagged as superseded if they were split into two or more new sources, or if several old sources were merged into a single one.
Traceability of the sources
The result of the previously described operations is summarised in Table 7.1.
Type | Input | Deleted | Removed by Split | Removed by Merge | Added | Output |
IGSL Source | 603 038 774 | 3 732 663 | 1 815 549 | 16 118 938 | 581 371 624 | |
New source (IDU-01) | 1 333 804 445 | 105 501 144 | 5 597 468 | 41 726 981 | 1 180 978 852 | |
HPM Addendum | 200 | 50 | 0 | 27 | 123 | |
New source (IDU-02) | 645 771 010 | 81 006 080 | 4 348 988 | 35 842 395 | 524 573 547 | |
New source (IDU-03) | ||||||
as a new source | 194 550 000 | 194 550 000 | ||||
as result of merge | 45 887 691 | 45 887 691 | ||||
as result of split | 25 260 950 | 25 260 950 | ||||
Total | 2 582 614 429 | 190 239 937 | 11 762 005 | 93 688 341 | 265 698 641 | 2 552 622 787 |
Combining the input data from data reduction cycle 02, the new sources from IDU-03 and discarding the deleted and the superseded sources, the output dataset has 2 552 622 787 sources. All sources were assigned at least one observation by the IDU cross match.
Updates from AGIS-03
The outputs of the second AGIS run for the cycle 3 (AGIS-03.2) were integrated into the dataset described in the previous subsection, which included sources with positions and proper motions supplied by different systems - including AGIS-01, AGIS-02, IDU-01, IDU-02, IDU-03 as well as data from the IGSL catalogue.
The distribution of the AGIS-03 updates against the types of source records is shown in Table 7.2.
Not | Not | ||||||
Source Type | Origin | 2-params | 5-params | 6-params | Converged | Processed | Total |
NEW | IDU-03 | 179 039 547 | 768 236 | 13 017 569 | 1 724 648 | 194 550 000 | |
NEW (merge) | IDU-03 | 12 472 283 | 33 051 532 | 339 443 | 24 433 | 45 887 691 | |
NEW (split) | IDU-03 | 22 849 878 | 1 688 465 | 673 951 | 48 656 | 25 260 950 | |
PERSISTENT | IDU-02 | 215 419 351 | 211 457 213 | 92 521 229 | 4 289 332 | 886 422 | 524 573 547 |
HPM-Addendum | 13 | 104 | 3 | 3 | 123 | ||
IDU-01 | 531 101 741 | 95 298 720 | 519 412 259 | 33 613 607 | 1 552 525 | 1 180 978 852 | |
IGSL | 66 253 555 | 278 706 238 | 235 588 193 | 784 981 | 38 657 | 581 371 624 | |
Total | 1 027 136 368 | 585 462 275 | 883 029 917 | 52 718 886 | 4 275 341 | 2 552 622 787 |
Of the sources retained in the integration, that is, those which have not been deleted or superseded by IDU-03, most received AGIS-03 updates. In the few cases where they did not, their previous values for positions and proper motions are maintained.
On a per-source level there is no mixing between AGIS solutions from different cycles. This means that a 2-parameter solution from AGIS-03 will remove any proper motions that were applied to a source from AGIS-02. Nevertheless proper motions from IDU and the IGSL are retained to provide support for future cross-match processing.
Distribution of position and proper motion originators
The following two tables summarise the origin of the positions and proper motions for the sources in the dataset generated prior to filtering in the archive for Gaia EDR3.
The distribution of the positions of the sources is shown in Table 7.3.
Position Origin | Count | Percentage |
AGIS-03 | 2 495 628 560 | 97.77% |
IDU-03 | 15 828 700 | 0.62% |
AGIS-02 | 6 261 410 | 0.25% |
IDU-02 | 3 249 943 | 0.13% |
HPM-Addendum | 1 | 0.00% |
AGIS-01 | 4 027 522 | 0.16% |
IDU-01 | 27 205 498 | 1.07% |
IGSL | 421 153 | 0.02% |
Total | 2 552 622 787 | 100.00% |
The originators of the proper motions of the integrated sources is shown in Table 7.4.
Proper Motion Origin | Count | Percentage |
AGIS-03.2 | 1 468 492 192 | 57.53% |
IDU-03 | 14 182 341 | 0.56% |
AGIS-02 | 360 423 | 0.01% |
IDU-02 | 3 015 135 | 0.12% |
HPM Addendum | 12 | 0.00% |
AGIS-01 | 108 | 0.00% |
IDU-01 | 0 | 0.00% |
IGSL | 21 569 221 | 0.84% |
None | 1 045 003 366 | 40.94% |
Total | 2 552 622 798 | 100.00% |