Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study.

BACKGROUND: Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022. OBJECTIVE: The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities. METHODS: Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines. RESULTS: Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy. CONCLUSIONS: ChatGPT's performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy.

Utilizing artificial intelligence in academic writing: an in-depth evaluation of a scientific review on fertility preservation written by ChatGPT-4.

PURPOSE: To evaluate the ability of ChatGPT-4 to generate a biomedical review article on fertility preservation. METHODS: ChatGPT-4 was prompted to create an outline for a review on fertility preservation in men and prepubertal boys. The outline provided by ChatGPT-4 was subsequently used to prompt ChatGPT-4 to write the different parts of the review and provide five references for each section. The different parts of the article and the references provided were combined to create a single scientific review that was evaluated by the authors, who are experts in fertility preservation. The experts assessed the article and the references for accuracy and checked for plagiarism using online tools. In addition, both experts independently scored the relevance, depth, and currentness of the ChatGPT-4's article using a scoring matrix ranging from 0 to 5 where higher scores indicate higher quality. RESULTS: ChatGPT-4 successfully generated a relevant scientific article with references. Among 27 statements needing citations, four were inaccurate. Of 25 references, 36% were accurate, 48% had correct titles but other errors, and 16% were completely fabricated. Plagiarism was minimal (mean = 3%). Experts rated the article's relevance highly (5/5) but gave lower scores for depth (2-3/5) and currentness (3/5). CONCLUSION: ChatGPT-4 can produce a scientific review on fertility preservation with minimal plagiarism. While precise in content, it showed factual and contextual inaccuracies and inconsistent reference reliability. These issues limit ChatGPT-4 as a sole tool for scientific writing but suggest its potential as an aid in the writing process.

Improving Academic Writing in a Low-Resource Country: A Systematic Examination of Online Peer-Run Training.

Problem: Syrian medical research synthesis lags behind that of neighboring countries. The Syrian war has exacerbated the situation, creating obstacles such as destroyed infrastructure, inflated clinical workload, and deteriorated medical training. Poor scientific writing skills have ranked first among perceived obstacles that could be modified to improve Syrian research conduct at every academic level. However, limited access to personal and physical resources in conflict areas consistently hampers the implementation of standard professional-led interventions. Intervention: We designed a peer-run online academic writing and publishing workshop as a feasible, affordable, and sustainable training method to use in low-resource settings. This workshop covered the structure of scientific articles, academic writing basics, plagiarism, and the publication process. It was also supplemented by six practical assignments to exercise the learned skills. Context: The workshop targeted healthcare professionals and medicine, dentistry, and pharmacy trainees (undergraduate and postgraduate) at all Syrian universities. We employed a systematic design to evaluate the workshop's short- and long-term impact when using different instructional delivery methods and assignment formats. Participants were assigned in a stratified manner to four groups; two groups attended the workshop synchronously, and the other two groups attended asynchronously. One arm in each group underwent a supervised peer-review evaluation for the practical writing exercises (active), while the other arm in each group self-reviewed their work on the same exercises using exemplary solutions (passive). We assessed knowledge (30 questions), confidence in the learned skills (11 questions), and the need for further guidance in academic writing (1 question) before the workshop and one month and one year after it. Impact: One-hundred-twenty-one participants completed the workshop, showing improved knowledge, confidence, and need for guidance. At one-year follow-up, participants showed stability in these gains. Outcomes for the synchronous and asynchronous groups were similar. Completing practical assignments was associated with greater knowledge and confidence only in the active arms. Participants in the active arms engaging in the peer-review process showed greater knowledge increase and reported less need for guidance compared to those who did not engage in the peer-review. Lessons learned: Peer-run interventions can provide an effective, affordable alternative to improving scientific writing skills in settings with limited resources and expertise. Online academic writing training can show improvements regardless of method of attendance (i.e., synchronous versus asynchronous). Participation in supplementary practical exercises, especially when associated with peer-review, may improve knowledge and confidence.

Exploring the potential of artificial intelligence to enhance the writing of english academic papers by non-native english-speaking medical students - the educational application of ChatGPT.

BACKGROUND: Academic paper writing holds significant importance in the education of medical students, and poses a clear challenge for those whose first language is not English. This study aims to investigate the effectiveness of employing large language models, particularly ChatGPT, in improving the English academic writing skills of these students. METHODS: A cohort of 25 third-year medical students from China was recruited. The study consisted of two stages. Firstly, the students were asked to write a mini paper. Secondly, the students were asked to revise the mini paper using ChatGPT within two weeks. The evaluation of the mini papers focused on three key dimensions, including structure, logic, and language. The evaluation method incorporated both manual scoring and AI scoring utilizing the ChatGPT-3.5 and ChatGPT-4 models. Additionally, we employed a questionnaire to gather feedback on students' experience in using ChatGPT. RESULTS: After implementing ChatGPT for writing assistance, there was a notable increase in manual scoring by 4.23 points. Similarly, AI scoring based on the ChatGPT-3.5 model showed an increase of 4.82 points, while the ChatGPT-4 model showed an increase of 3.84 points. These results highlight the potential of large language models in supporting academic writing. Statistical analysis revealed no significant difference between manual scoring and ChatGPT-4 scoring, indicating the potential of ChatGPT-4 to assist teachers in the grading process. Feedback from the questionnaire indicated a generally positive response from students, with 92% acknowledging an improvement in the quality of their writing, 84% noting advancements in their language skills, and 76% recognizing the contribution of ChatGPT in supporting academic research. CONCLUSION: The study highlighted the efficacy of large language models like ChatGPT in augmenting the English academic writing proficiency of non-native speakers in medical education. Furthermore, it illustrated the potential of these models to make a contribution to the educational evaluation process, particularly in environments where English is not the primary language.

Theorising Support for Interdisciplinary Early-Career Researchers Using Communicative Genre and 'Rules of the Game'.

Qualitative social scientists working in medical faculties have to meet multiple expectations. On the one hand, they are expected to comply with the philosophical and theoretical expectations of the social sciences. On the other hand, they may also be expected to produce publications which align with biomedical definitions and framings of quality. As interdisciplinary scholars, they must handle (at least) two sets of journal editors, peer reviewers, grant-awarding panels, and conference audiences. In this paper, we extend the current knowledge base on the 'dual expectations' challenge by drawing on Orlikowski and Yates' theoretical concept of communicative genres. A 'genre' in this context is a format of communication (e.g. letter, email, academic paper, and conference presentation) aimed at a particular audience, having a particular material form and socio-linguistic style, and governed by both formal requirements and unwritten social rules. Becoming a member of any community of practice involves becoming familiar with its accepted communicative genres and adept in using them. Academic writing, for example, is a craft that is learned through participation in the social process of communicating one's ideas to one's peers in journal articles and other formats. In this reflective paper, we show how the concept of a communicative genre can sensitise us to the conflicting and often dissonant expectations and rule systems governing different academic fields. We use this key concept to suggest ways in which the faculty can support early-career researchers to progress in careers which straddle qualitative social science and medical science.

The Art of Influencing: Exploring Persuasive Strategies in the Writings of Iranian University Students.

This study investigates persuasive strategies used in the writings of Iranian university students in the field of teaching English as foreign language (TEFL). The study utilized the 7 principles of persuasive strategies presented by Cialdini (The psychology of persuasion, Quill William Morrow, New York 1984; Pre-suasion: A revolutionary way to influence and persuade, Simon & Schuster, New York 2016), which include 'reciprocity', 'commitment and consistency', 'social proof', 'liking', 'authority', 'scarcity', and 'unity'. The results indicate that strategies such as 'liking', 'unity', and 'authority' were used more frequently than other persuasive strategies. On the other hand, 'scarcity' was the least used strategy by the participants. Significant gender differences were also observed in the data. These findings have important pedagogical implications and suggest the need to incorporate persuasive strategies into instructional materials and teaching practices to enhance the persuasive writing skills of university students. Furthermore, gender differences highlight the importance of considering individual differences when teaching persuasive writing. Finally, the study discusses the pedagogical implications of these findings in the context of learning and teaching.

Large language models are changing landscape of academic publications. A positive transformation?

The advent of large language models (LLMs) based on neural networks marks a significant shift in academic writing, particularly in medical sciences. These models, including OpenAI's GPT-4, Google's Bard, and Anthropic's Claude, enable more efficient text processing through transformer architecture and attention mechanisms. LLMs can generate coherent texts that are indistinguishable from human-written content. In medicine, they can contribute to the automation of literature reviews, data extraction, and hypothesis formulation. However, ethical concerns arise regarding the quality and integrity of scientific publications and the risk of generating misleading content. This article provides an overview of how LLMs are changing medical writing, the ethical dilemmas they bring, and the possibilities for detecting AI-generated text. It concludes with a focus on the potential future of LLMs in academic publishing and their impact on the medical community.

A study of factors affecting pass rates of level 5 written assessments for undergraduate adult nursing students.

Nursing programmes were flexible during the COVID-19 pandemic, offering simulation to replace clinical hours and adjusting supervision and assessment. However, second-year students in two modules had lower results despite no changes to the material, team or delivery. OBJECTIVES: A retrospective cohort study was conducted, on second-year adult nursing students who submitted written assignments, to analyse recurring patterns that could explain the failure rate. METHOD: Data were analysed from 265 university students to identify patterns of association in demographics, module results and student engagement indicators. RESULTS: A positive correlation was found between age and assignment results, with older students achieving higher grades. Clustering identified three patterns of student engagement. Students demonstrating engagement with all aspects of the course (30.2%) performed significantly better than those in other clusters (P<0.001). Students with disabled student support recommendations performed notably worse than those without. All sizeable differences were resolved following the return to campus and the implementation of additional writing support. DISCUSSION: Age, cross-medium engagement and preparation were all shown to have an impact on marks. These findings can influence how higher education institutions drive and monitor engagement, as this study suggests that all parts of a blended learning approach are equally important.

AI language models in human reproduction research: exploring ChatGPT's potential to assist academic writing.

Artificial intelligence (AI)-driven language models have the potential to serve as an educational tool, facilitate clinical decision-making, and support research and academic writing. The benefits of their use are yet to be evaluated and concerns have been raised regarding the accuracy, transparency, and ethical implications of using this AI technology in academic publishing. At the moment, Chat Generative Pre-trained Transformer (ChatGPT) is one of the most powerful and widely debated AI language models. Here, we discuss its feasibility to answer scientific questions, identify relevant literature, and assist writing in the field of human reproduction. With consideration of the scarcity of data on this topic, we assessed the feasibility of ChatGPT in academic writing, using data from six meta-analyses published in a leading journal of human reproduction. The text generated by ChatGPT was evaluated and compared to the original text by blinded reviewers. While ChatGPT can produce high-quality text and summarize information efficiently, its current ability to interpret data and answer scientific questions is limited, and it cannot be relied upon for a literature search or accurate source citation due to the potential spread of incomplete or false information. We advocate for open discussions within the reproductive medicine research community to explore the advantages and disadvantages of implementing this AI technology. Researchers and reviewers should be informed about AI language models, and we encourage authors to transparently disclose their use.

Peer feedback and Chinese medical students' English academic writing development: a longitudinal intervention study.

BACKGROUND: Studies have documented that utilizing peer feedback can enhance students' English academic writing skills. Little is known, however, about the effects of incorporating peer feedback to enhance English as a second language (L2) medical students' academic writing performance. METHODS: This longitudinal interventional study examines Chinese medical students' English academic writing skills development via peer feedback in four parallel classes over an 18-week semester between the experimental and control groups (n = 124). RESULTS: Significant increases in the experimental group's performance in the post-test were found after 18-week instructions (pre- vs. post-test: overall score, p < .001; task response, p < .001; coherence and cohesion, p < .001; lexical resource, p < .001; grammatical range and accuracy, p < .001), and the effects were retained in the delayed post-test 6 weeks later (post- vs. delayed post-test: overall score, p = .561; task response, p = .585; coherence and cohesion, p = .533; lexical resource, p = .796; grammatical range and accuracy, p = .670). Little improvement was found in the control group in the post-test (pre- vs. post-test: overall score, p = .213; task response, p = .275; coherence and cohesion, p = .383; lexical resource, p = .367; grammatical range and accuracy, p = .180) or the delayed post-test (post- vs. delayed post-test: overall score, p = .835; task response, p = .742; coherence and cohesion, p = .901; lexical resource, p = .897; grammatical range and accuracy, p = .695). Between-group comparisons indicate that the experimental group outperformed the control group in the post- and the delayed post-tests, as shown in their overall score and scores on the four components. CONCLUSIONS: Incorporating peer feedback into process-oriented medical English writing classroom teaching can effectively enhance Chinese medical students' English academic writing skills over time, while the traditional product-oriented writing instructions had little help in improving Chinese medical students' academic writing skills. This longitudinal intervention study develops our understanding of the effectiveness of peer feedback in L2 academic writing pedagogy. It offers instructional implications for L2 writing teachers to teach English academic writing among medical students in China and beyond. Limitations and suggestions for future studies are discussed.

A multi-layered approach to developing academic written communication skills for nursing students.

OBJECTIVES: Academic literacy in nursing students has historically been poor this research therefore aimed to investigate the effectiveness of an embedded multidisciplinary approach to academic written communication skills for first year Bachelor of Nursing students in Australia. METHODS: This initiative consisted of (1) collaboratively embedding academic skills with nursing staff and Learning and Academic Skills Advisors using scaffolded tasks, curriculum/material development, workshops/tutorials on writing skills; and (2) individual support sessions. Data was obtained via questionnaire. RESULTS: Responses were received from 92 (74% local and 26% international) students. Key findings showed the collaborative approach to embedding skills with added follow up, was beneficial for students' skills development, including improved academic writing, increased confidence, and recognition of the importance of academic writing. CONCLUSIONS: It is important to equip student to develop academic literacy and an embedded academic literacy program has been determined to assist and may contribute to future professionalisation in nursing.

Over-promotion and caution in abstracts of preprints during the COVID-19 crisis.

The abstract is known to be a promotional genre where researchers tend to exaggerate the benefit of their research and use a promotional discourse to catch the reader's attention. The COVID-19 pandemic has prompted intensive research and has changed traditional publishing with the massive adoption of preprints by researchers. Our aim is to investigate whether the crisis and the ensuing scientific and economic competition have changed the lexical content of abstracts. We propose a comparative study of abstracts associated with preprints issued in response to the pandemic relative to abstracts produced during the closest pre-pandemic period. We show that with the increase (on average and in percentage) of positive words (especially effective) and the slight decrease of negative words, there is a strong increase in hedge words (the most frequent of which are the modal verbs can and may). Hedge words counterbalance the excessive use of positive words and thus invite the readers, who go probably beyond the 'usual' audience, to be cautious with the obtained results. The abstracts of preprints urgently produced in response to the COVID-19 crisis stand between uncertainty and over-promotion, illustrating the balance that authors have to achieve between promoting their results and appealing for caution.

A framework for scaffolding academic reflective writing in dentistry.

INTRODUCTION: Being able to demonstrate reflection on clinical practice is a key competency required of undergraduate dental students and registrants alike. Academic reflective writing (ARW) is an increasingly widely used genre of writing which evidences reflective practice. ARW is complex, as students need to achieve balance from a challenging mix of personal, academic and evaluative writing. The aim of this study is to analyse four student ARW tasks from a UK Dental School in order to make recommendations regarding the systematic development of student ARW through the dental curriculum. METHODS: In this study, a detailed linguistic analysis of high-scoring examples of two first year and two fourth year ARW tasks was undertaken, complemented by comments from interviews with markers of the tasks. RESULTS: The linguistic analysis of high-scoring student ARW revealed five consistent reflective aspects common to all tasks, viz description, feelings, analysis, evaluation and conclusion and action planning. Entry-level and final-year ARW showed a different balance between reflective aspects. Markers' expectations in the lower years were found to equate to high-quality technical reflection, whereas markers in the higher years expected students to have shifted their perspectives to achieve critical reflection. CONCLUSION: Whilst this study is limited in its generalisability due to its small-scale exploratory nature, it is hoped that the five reflective aspects, and the categories of reflection put forward here will be useful in stimulating discussion about how to scaffold the development of student ARW (and associated reflective thinking) longitudinally through the dental curriculum.

Graduated nurses' experiences with baccalaureate thesis writing: A qualitative study.

This study explores the perceptions of newly graduated nurses regarding the baccalaureate thesis (BT) writing process. A qualitative approach with content analysis was adopted. Thirteen newly graduated nurses participated in the study. The inclusion criterion was having completed the BT during their university education as nurses within 3 years of the study. Information was obtained through semistructured interviews. Data collection began in June 2018 and ended in March 2019, once saturation of information was reached. Three major categories were identified: the beginning of the process, the advantages of Baccalaureate thesis writing, and the teacher's role. The participants expressed that their training focused on a synthetic and particular type of writing that is necessary for clinical scenarios but not suitable for scientific academic texts. Additionally, reflective diaries were described as a powerful writing practice during their studies. Participants conceive the realization of the baccalaureate thesis as a difficult process, especially as regards selecting, synthesizing, and writing about the available information, processes that they indicated generate anxiety. Academic writing skills should be specifically included in the nursing curriculum.

Seeking transformation: how students in nursing view their academic writing context - a qualitative systematic review.

Writing practices in nursing education programs are situated in a tension-filled context resulting from competing medical-technical and relational nursing discourses. The goal of this qualitative meta-study is to understand, from the student perspective, how the context for writing in nursing is constructed and the benefits of writing to nursing knowledge development. A literature search using the CINHAL, Medline, ERIC, and Academic Search complete databases, using systematic methods identified 21 papers and dissertations which gathered qualitative interview or survey data from students in nursing at the pre-registration, continuing education, and graduate levels. The studies provided evidence that writing assignments promote professional identity development but overemphasis on writing mechanics when grading have a deleterious effect on learning and student engagement with writing. Relationship building with faculty should extend beyond what is needed to maximize grades. Suggestions for writing pedagogical reform are identified to facilitate a change in focus from mechanical-technical to transformative writing.

Attitudes and Knowledge About Plagiarism Among University Students: Cross-Sectional Survey at the University of Split, Croatia.

Plagiarism is one of the most severe academic integrity issues. This study examined students' knowledge of and attitudes towards plagiarism, tested their ability to recognize plagiarism, and explored the association of study levels and attendance in courses dealing with referencing rules and plagiarism with students' attitudes and knowledge. A cross-sectional online survey was conducted at the University of Split, comprising the students of all schools and study levels (n = 388). Overall, results indicate the students were not very familiar with referencing rules and did not perform well on either theoretical questions or practical examples. However, they demonstrated positive attitudes towards plagiarism avoidance as well as towards compliance with academic integrity with respect to the correct use of research publications. Students' self-reported attendance in courses dealing with referencing rules and plagiarism avoidance was not associated with their knowledge of and attitudes toward plagiarism. These findings are important for a general understanding of students' attitudes, and the relation of practical and theoretical knowledge of plagiarism. Furthermore, the academic community addresses plagiarism not only as an ethical and regulatory violation but also as a direct consequence of a lack of knowledge, and of academic illiteracy. Study programs should be adjusted and long-term policies developed at all academic levels to promote a positive climate among students towards responsible academic writing.

How to incorporate academic writing pedagogy in undergraduate and postgraduate medical education.

OBJECTIVE: Introduction: Medical researchers, who are non-native English speakers, are facing now the growing need to publish their research results in international journals switching to an English-only policy, to apply for grants and scholarship, but at the same time this puts many authors whose native language is not English at a disadvantage compared to their English-speaking counterparts. The aim: This paper aims at analysing the existing parameters of academic writing proficiency of medical undergraduate and postgraduate students; elucidating current approaches to develop academic writing competency and to promote academic multi-literacy of junior researchers, and outlining the general recommendations to improve the quality and sophistication of their writing by incorporating the principles and achievements of academic writing pedagogy into the system of medical training. PATIENTS AND METHODS: Materials and methods: This study is an empirical applied research of a qualitative type mainly based on data elicited from informants (n=120) of the Ukrainian Medical Stomatological Academy aged from 20 - 35. RESULTS: Results and conclusions: All participants were able to identify personal problem areas, and virtually all they note dissatisfaction with the use of English in their scholarly writing. They stated the obvious difficulties in sentence patterns and keeping tone of scientific narrative format. Writing in genres other than original research articles seems to be quite demanding and is often associated with the lack of self-confidence and language anxiety. Attention to developing academic writing skills should focus on the basic elements of academic writing, characteristics of written genres across the disciplines, providing a framework in which expert and practical knowledge is internally organized.

Academic voice: On feminism, presence, and objectivity in writing.

Academic voice is an oft-discussed, yet variably defined concept, and confusion exists over its meaning, evaluation, and interpretation. This paper will explore perspectives on academic voice and counterarguments to the positivist origins of objectivity in academic writing. While many epistemological and methodological perspectives exist, the feminist literature on voice is explored here as the contrary position. From the feminist perspective, voice is a socially constructed concept that cannot be separated from the experiences, emotions, and identity of the writer and, thus, constitutes a reflection of an author's way of knowing. A case study of how author presence can enhance meaning in text is included. Subjective experience is imperative to a practice involving human interaction. Nursing practice, our intimate involvement in patient's lives, and the nature of our research are not value free. A view is presented that a visible presence of an author in academic writing is relevant to the nursing discipline. The continued valuing of an objective, colorless academic voice has consequences for student writers and the faculty who teach them. Thus, a strategically used multivoiced writing style is warranted.

Quality and correctness of AI-generated versus human-written abstracts in psychiatric research papers.

This study aimed to assess the ability of an artificial intelligence (AI)-based chatbot to generate abstracts from academic psychiatric articles. We provided 30 full-text psychiatric papers to ChatPDF (based on ChatGPT) and prompted generating a similar style structured or unstructured abstract. We further used 10 papers from Psychiatry Research as active comparators (unstructured format). We compared the quality of the ChatPDF-generated abstracts with the original human-written abstracts and examined the similarity, plagiarism, detected AI-content, and correctness of the AI-generated abstracts. Five experts evaluated the quality of the abstracts using a blinded approach. They also identified the abstracts written by the original authors and validated the conclusions produced by ChatPDF. We found that the similarity and plagiarism were relatively low (only 14.07% and 8.34%, respectively). The detected AI-content was 31.48% for generated structure-abstracts, 75.58% for unstructured-abstracts, and 66.48% for active comparators abstracts. For quality, generated structured-abstracts were rated similarly to originals, but unstructured ones received significantly lower scores. Experts rated 40% accuracy with structured abstracts, 73% with unstructured ones, and 77% for active comparators. However, 30% of AI-generated abstract conclusions were incorrect. In conclusion, the data organization capabilities of AI language models hold significant potential for applications to summarize information in clinical psychiatry. However, the use of ChatPDF to summarize psychiatric papers requires caution concerning accuracy.

Exploring the use of grammarly in assessing English academic writing.

This study explores the extent to which Grammarly can be a reliable assessment tool for academic English writing. Ten articles published in high-status scholarly Q.1 journals and written by specialist English native speakers were used to evaluate the accuracy of Grammarly's flagged issues. The results showed that Grammarly tends to over-flag many issues resulting in many false positives; besides, it does not take into consideration optional usage in English. The study concluded that although Grammarly can identify many ambiguous instances of language use that writers would do well to review and consider for revision, it does not seem to be a reliable tool for assessing academic written English.