An analysis of the generative AI use as analyst in qualitative research in science education
DOI:
https://doi.org/10.33361/RPQ.2024.v.12.n.30.724Keywords:
Qualitative analysis, Science Education, CNMT, ChatGPT, ClaudeAbstract
The article evaluates the effectiveness of generative artificial intelligence models, specifically ChatGPT 4.0 and Claude 2.0, in conducting qualitative research within the field of scientific education. By applying the Cognitive Networks Mediation Theory (CNMT) to analyze interviews from two students, it was found that Claude 2.0 surpassed ChatGPT 4.0 in recognizing cognitive mediations and distinguishing between pre- and post-test conditions. Although both models concurred on the concept of conceptual evolution, Claude 2.0 demonstrated a greater capacity for detail, notably by referencing specific interview excerpts to support its analyses upon request. In contrast, ChatGPT 4.0 exhibited difficulties in these areas, even when given additional prompts. The study concludes by acknowledging the utility of AI, particularly Claude 2.0, for qualitative research, while also emphasizing the critical role of human oversight in detecting potential deficiencies within these models.
Downloads
Metrics
References
ANTHROPIC. Claude v.2.0. Available in: https://www.anthropic.com/index/100k-context-windows. Accessed on: October 9, 2023.
ATKINSON, C. F. Cheap, Quick, and Rigorous: Artificial Intelligence and the Systematic Literature Review. Social Science Computer Review, Thousand Oaks, v. 42, n. 02, p. 08944393231196281, aug. 2023. DOI: https://doi.org/10.1177/08944393231196281
BAROCAS, S.; SELBST, A. D. Big data’s disparate impact. California Law Review, Berkeley, v. 104, n. 3, 671-732, 2016. DOI: https://doi.org/10.2139/ssrn.2477899
BELOTTO, M. J. Data analysis methods for qualitative research: Managing the challenges of coding, interrater reliability, and thematic analysis. The Qualitative Report, Miami, v. 23, n. 11, p. 2622-2633, 2018. DOI: https://doi.org/10.46743/2160-3715/2018.3492
BISWAS, S.S. Potential use of chat GPT in global warming. Annals of biomedical engineering, Piscataway, v. 51, n. 6, p. 1126-1127, 2023. DOI: https://doi.org/10.1007/s10439-023-03171-8
BOZKURT, A.; XIAO, J.; LAMBERT, S.; PAZUREK, A.; CROMPTON, H.; KOSEOGLU, S.; FARROW, R.; BOND, M.; NERANTZI, C.; HONEYCHURCH, S.; BALI, M.; DRON, J.; MIR, K.; STEWART, B.; COSTELLO, E.; MASON, J.; STRACKE, C. M.; ROMERO-HALL, E.; KOUTROPOULOS, A.; TOQUERO, C. M.; SINGH, L.; TLILI, A.; LEE, K.; NICHOLS, M.; OSSIANNILSSON, E.; BROWN, M.; IRVINE, V.; RAFFAGHELLI, J. E.; SANTOS-HERMOSA, G.; FARRELL, O.; ADAM, T.; THONG, Y. L.; SANI-BOZKURT, S.; SHARMA, R. C.; HRASTINSKI, S.; JANDRIĆ, P. Speculative futures on ChatGPT and generative artificial intelligence (AI): A collective reflection from the educational landscape. Asian Journal of Distance Education, Delhi, v. 18, n. 1, 2023.
BRUUN, E. P.G.; DUKA, A. Artificial intelligence, jobs and the future of work: Racing with the machines. Basic Income Studies, Berlin, v. 13, n. 2, p. 20180018, 2018. DOI: https://doi.org/10.1515/bis-2018-0018
CHING, T. et al. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, London, v. 15, n. 141, p. 20170387, 2018. DOI: https://doi.org/10.1098/rsif.2017.0387
CHRISTOU, P. A. A critical perspective over whether and how to acknowledge the use of artificial intelligence (AI) in qualitative studies. The Qualitative Report, Miami, v. 28, n. 7, p. 1981-1991, 2023a. DOI: https://doi.org/10.46743/2160-3715/2023.6407
CHRISTOU, P. A. How to use artificial intelligence (AI) as a resource, methodological and analysis tool in qualitative research? The Qualitative Report, Miami, v. 28, n. 7, p. 1968-1980, 2023b. DOI: https://doi.org/10.46743/2160-3715/2023.6406
CHUBB, L. A. Me and the Machines: Possibilities and Pitfalls of Using Artificial Intelligence for Qualitative Data Analysis. International Journal of Qualitative Methods, Thousand Oaks, v. 22, n. 1, p. 16094069231193593, 2023. DOI: https://doi.org/10.1177/16094069231193593
CONNEAU, A.; LAMPLE, G. Cross-lingual language model pretraining. Advances in neural information processing systems, Cambridge, v. 32, n.01, 2019.
FONSECA, A. L. A. da; CHIMENTI, P. C. P. S.; SUAREZ, M. C. Using deep learning language models as scaffolding tools in interpretive research. Revista de Administração Contemporânea, Maringá, v. 27, v. 03, p. e230021, 2023. DOI: https://doi.org/10.1590/1982-7849rac2023230021.en
HAMILTON, L.; ELLIOTT, D.; QUICK, A.; SMITH, S.; CHOPLIN, V. Exploring the Use of AI in Qualitative Analysis: A Comparative Study of Guaranteed Income Data. International Journal of Qualitative Methods, Thousand Oaks, v. 22, n. 01, p. 16094069231201504, 2023. DOI: https://doi.org/10.1177/16094069231201504
HOSSEINI, M.; HORBACH, S. PJM. Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other Large Language Models in scholarly peer review. Research Integrity and Peer Review, Palo Alto, v. 8, n. 1, p. 4, 2023. DOI: https://doi.org/10.1186/s41073-023-00133-5
KIRMANI, A. R. Artificial intelligence-enabled science poetry. ACS Energy Letters, Washington, v. 8, n. 1, p. 574-576, 2022. DOI: https://doi.org/10.1021/acsenergylett.2c02758
LONGO, L. Empowering qualitative research methods in education with artificial intelligence. In: World Conference on Qualitative Research. Cham: Springer International Publishing, Porto, v. 1068, p. 1-21, 2019. DOI: https://doi.org/10.1007/978-3-030-31787-4_1
LUND, B. D.; WANG, T. Chatting about ChatGPT: how may AI and GPT impact academia and libraries?. Library Hi Tech News, Leeds, v. 40, n. 3, p. 26-29, 2023. DOI: https://doi.org/10.1108/LHTN-01-2023-0009
LUND, B. D; WANG, T; MANNURU, N.R; NIE, B; SHIMRAY, S; WANG, Z. ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, Syracuse, v. 74, n. 5, p. 570-581, 2023. DOI: https://doi.org/10.1002/asi.24750
MESEC, B. The language model of artificial inteligence chatGPT-a tool of qualitative analysis of texts. [Preprint]. 2023. Available in: https://doi.org/10.22541/au.168182047.70243364/v1. DOI: https://doi.org/10.22541/au.168182047.70243364/v1
MILOYAN, B.; MCFARLANE, K.; VASQUEZ-ECHEVERRIA, A. The adapted Autobiographical interview: A systematic review and proposal for conduct and reporting. Behavioural Brain Research, Amsterdam, v. 370, p. 111881, 2019. Available: https://doi.org/10.1016/j.bbr.2019.03.050. DOI: https://doi.org/10.1016/j.bbr.2019.03.050
OpenAI. Introducing ChatGPT, 2022. Retrieved from https://openai.com/blog/chatgpt.
SANTOS, D. J. M.; SANTOS, P. V.; BOSS, S. L. B. A análise de conteúdo com apoio do software gratuito RQDA. Revista Pesquisa Qualitativa, São Paulo, v.11, n.28, p. 805-824, set./dez. 2023. DOI: https://doi.org/10.33361/RPQ.2023.v.11.n.28.612
SOBANIA, D.; BRIESCH, M.; HANNA, C.; PETKE, J. An analysis of the automatic bug fixing performance of chatGPT. [Preprint] arXiv:2301.08653, 2023. Available in: https://arxiv.org/pdf/2301.08653.pdf. Melbourne. Accessed on: October 9, 2023. DOI: https://doi.org/10.1109/APR59189.2023.00012
SOUZA, B. C.; SILVA, A. S.; SILVA, A. M.; ROAZZI, A.; CARRILHO, S. L. S. Putting the Cognitive Mediation Networks Theory to the test: Evaluation of a framework for understanding the digital age. Computers in Human Behavior, Québec, v. 28, n. 6, p. 2320-2330, 2012. DOI: https://doi.org/10.1016/j.chb.2012.07.002
TABONE, W.; WINTER, J. Using ChatGPT for human–computer interaction research: A primer. Manuscript submitted for publication, 2023. DOI: https://doi.org/10.1098/rsos.231053
TATE, T.; DOROUDI, S.; RITCHIE, D.; XU, Y. Educational research and AI-generated writing: Confronting the coming tsunami [preprint], 2023. Available in: http://dx.doi.org/10.35542/osf.io/4mec3. DOI: https://doi.org/10.35542/osf.io/4mec3
TREVISAN, R.; SERRANO, A.; WOLFF, J. F. S.; RAMOS, A. F. Peeking into their mental imagery: The Report Aloud technique in science education research. Ciência e Educação, Bauru, v. 25, n. 3, pp. 647-664, 2019. Available in: https://doi.org/10.1590/1516-731320190030004. DOI: https://doi.org/10.1590/1516-731320190030004
YANG, C.; WANG, X.; LU, Y.; LIU, H.; LE, Q. V.; ZHOU, D.; CHEN, X. Large language models as optimizers. [Preprint] arXiv:2309.03409, 2023. Available in: https://arxiv.org/pdf/2309.03409.pdf. Accessed on: October 9, 2023.
ZHONG, Q.; DING, L.; LIU, J.; DU, B.; TAO, D. Can chatGPT understand too? a comparative study on chatgpt and fine-tuned bert. [Preprint] arXiv:2302.10198, 2023. Available in: https://arxiv.org/pdf/2302.10198.pdf. Accessed on: October 9, 2023.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Qualitative Research Journal
This work is licensed under a Creative Commons Attribution 4.0 International License.
Essa revista é licenciada pelo sistema creative commons 4.0, não-comercial.
