Your browser doesn't support javascript.
loading
Applying inter-rater reliability to improve consistency in classifying PhD career outcomes.
Stayart, C Abigail; Brandt, Patrick D; Brown, Abigail M; Dahl, Tamara; Layton, Rebekah L; Petrie, Kimberly A; Flores-Kim, Emma N; Peña, Christopher G; Fuhrmann, Cynthia N; Monsalve, Gabriela C.
Afiliação
  • Stayart CA; University of Chicago, Chicago, IL, USA.
  • Brandt PD; University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
  • Brown AM; Vanderbilt University, Nashville, TN, USA.
  • Dahl T; Emory University and the Georgia Institute of Technology, Atlanta, GA, USA.
  • Layton RL; University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
  • Petrie KA; Vanderbilt University, Nashville, TN, USA.
  • Flores-Kim EN; UCI School of Biological Sciences, Irvine, CA, USA.
  • Peña CG; University of Chicago, Chicago, IL, USA.
  • Fuhrmann CN; University of Massachusetts Medical School, Worcester, MA, USA.
  • Monsalve GC; University of California, San Francisco, San Francisco, CA, USA.
F1000Res ; 9: 8, 2020.
Article em En | MEDLINE | ID: mdl-32089837
ABSTRACT

Background:

There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions.

Methods:

To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders.

Results:

We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017.

Conclusions:

Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.
Assuntos
Palavras-chave

Texto completo: 1 Base de dados: MEDLINE Assunto principal: Escolha da Profissão / Educação de Pós-Graduação / Docentes Idioma: En Ano de publicação: 2020 Tipo de documento: Article

Texto completo: 1 Base de dados: MEDLINE Assunto principal: Escolha da Profissão / Educação de Pós-Graduação / Docentes Idioma: En Ano de publicação: 2020 Tipo de documento: Article