Tough Tables: Carefully Evaluating Entity Linking for Tabular Data

Tough Tables (2T) is a dataset designed to evaluate table annotation approaches in solving the CEA and CTA tasks. The dataset is compliant with the data format used in SemTab 2019, and it can be used as an additional dataset without any modification. The target knowledge graph is DBpedia 2016-10. Check out the 2T GitHub repository for more details about the dataset generation. New in v3.0: We release the updated version of 2T! The target knowledge graphs are DBpedia 2016-10 and Wikidata 20220521. Starting from this version, the dataset is split into valid and test sets. This work is based on the following paper: Cutrona, V., Bianchi, F., Jimenez-Ruiz, E. and Palmonari, M. (2020). Tough Tables: Carefully Evaluating Entity Linking for Tabular Data. ISWC 2020, LNCS 12507, pp. 1–16. Note on License: This dataset includes data from the following sources. Refer to each source for license details: - Wikipedia https://www.wikipedia.org/ - DBpedia https://dbpedia.org/ - Wikidata https://www.wikidata.org/ - SemTab 2019 https://doi.org/10.5281/zenodo.3518539 - GeoDatos https://www.geodatos.net - The Pudding https://pudding.cool/ - Offices.net https://offices.net/ - DATA.GOV https://www.data.gov/ THIS DATA IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. Changelog: v3.0 Both datasets require SemTab2020 CEA format (tab_id, row_id, col_id, entity). Tables IDs and artificial noise values differ from previous versions. Datasets are split into Valid and Test sets of tables. New GT for ToughTables-WD (2T_WD) Entities Q23772518 and Q7327323 have been removed because they are no longer represented in WD Updated ancestor/descendant hierarchies to evaluate CTA. Evaluation scripts are provided with the data sets. v2.0 New GT for 2T_WD A few entities have been removed from the CEA GT, because they are no longer represented in WD (e.g., dbr:Devonté points to wd:Q21155080, which does not exist) Tables codes and values differ from the previous version, because of the random noise. Updated ancestor/descendant hierarchies to evaluate CTA. v1.0 New Wikidata version (2T_WD) Fix header for tables CTRL_DBP_MUS_rock_bands_labels.csv and CTRL_DBP_MUS_rock_bands_labels_NOISE2.csv (column 2 was reported with id 1 in target - NOTE: the affected column has been removed from the SemTab2020 evaluation) Remove duplicated entries in tables Remove rows with wrong values (e.g., the Kazakhstan entity has an empty name "''") Many rows and noised columns are shuffled/changed due to the random noise generator algorithm Remove row "Florida","Floorida","New York, NY" from TOUGH_WEB_MISSP_1000_us_cities.csv (and all its NOISE1 variants) Fix header of tables: CTRL_WIKI_POL_List_of_current_monarchs_of_sovereign_states.csv CTRL_WIKI_POL_List_of_current_monarchs_of_sovereign_states_NOISE2.csv TOUGH_T2D_BUS_29414811_2_4773219892816395776___videogames_developers.csv TOUGH_T2D_BUS_29414811_2_4773219892816395776___videogames_developers_NOISE2.csv v0.1-pre First submission. It contains only tables, without GT and Targets.Visit the SemTab challenge website (https://sem-tab-challenge.github.io/2022/) for more details