A new approach for cleansing geographical dataset using Levenshtein distance, prior knowledge and contextual information.
Stud Health Technol Inform
; 210: 227-9, 2015.
Article
em En
| MEDLINE
| ID: mdl-25991137
ABSTRACT
Epidemiological studies are necessary to take public health decisions. Their relevance depends on the quality of data. Doctors in continuous care collect a big amount of data that can be used for epidemiological purpose, but spatial data may be dirty; based on city names, the localization is imprecise, even more if it is misspelled. The only way to identify a city without ambiguity is to use its identifier, which can be retrieved by cleansing geographical textual data. In France, cities are organized in administrative zones called departments and some city names are shared by several cities in several departments. The clear identification of the department and the city name allows to deduce the city unique identifier and to make some spatial analysis such as epidemiological studies. In this paper, we propose a method to cleanse such data, using several steps. After having standardized the text to cleanse, we use the Levenshtein distance to generate a first set of propositions. Finally, the propositions are filtered, by removing the less likely candidates, so that it remains only one, which becomes the chosen city. Tested on a dataset of 9818 entries, we obtained 89.1% of concordance, whereas the standard Levenshtein distance obtained 70.5%. This demonstrates that our method has better results.
Buscar no Google
Base de dados:
MEDLINE
Assunto principal:
Processamento de Linguagem Natural
/
Reconhecimento Automatizado de Padrão
/
Bases de Dados Factuais
/
Topografia Médica
/
Confiabilidade dos Dados
/
Aprendizado de Máquina
País como assunto:
Europa
Idioma:
En
Ano de publicação:
2015
Tipo de documento:
Article