An analysis of the generative AI use as analyst in qualitative research in science education

Juliana Rodrigues dos Anjos; Maira Giovana de Souza; Agostinho Serrano de Andrade Neto; Bruno Campello de Souza

doi:10.33361/RPQ.2024.v.12.n.30.724

Authors

Juliana Rodrigues dos Anjos ULBRA https://orcid.org/0000-0001-7625-8372
Maira Giovana de Souza Universidade Luterana do Brasil (ULBRA) https://orcid.org/0000-0003-2987-0053
Agostinho Serrano de Andrade Neto Universidade Luterana do Brasil (ULBRA)
Bruno Campello de Souza Universidade Federal de Pernambuco

DOI:

https://doi.org/10.33361/RPQ.2024.v.12.n.30.724

Keywords:

Qualitative analysis, Science Education, CNMT, ChatGPT, Claude

Abstract

The article evaluates the effectiveness of generative artificial intelligence models, specifically ChatGPT 4.0 and Claude 2.0, in conducting qualitative research within the field of scientific education. By applying the Cognitive Networks Mediation Theory (CNMT) to analyze interviews from two students, it was found that Claude 2.0 surpassed ChatGPT 4.0 in recognizing cognitive mediations and distinguishing between pre- and post-test conditions. Although both models concurred on the concept of conceptual evolution, Claude 2.0 demonstrated a greater capacity for detail, notably by referencing specific interview excerpts to support its analyses upon request. In contrast, ChatGPT 4.0 exhibited difficulties in these areas, even when given additional prompts. The study concludes by acknowledging the utility of AI, particularly Claude 2.0, for qualitative research, while also emphasizing the critical role of human oversight in detecting potential deficiencies within these models.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

References

ANTHROPIC. Claude v.2.0. Available in: https://www.anthropic.com/index/100k-context-windows. Accessed on: October 9, 2023.

ATKINSON, C. F. Cheap, Quick, and Rigorous: Artificial Intelligence and the Systematic Literature Review. Social Science Computer Review, Thousand Oaks, v. 42, n. 02, p. 08944393231196281, aug. 2023. DOI: https://doi.org/10.1177/08944393231196281

BAROCAS, S.; SELBST, A. D. Big data’s disparate impact. California Law Review, Berkeley, v. 104, n. 3, 671-732, 2016. DOI: https://doi.org/10.2139/ssrn.2477899

BELOTTO, M. J. Data analysis methods for qualitative research: Managing the challenges of coding, interrater reliability, and thematic analysis. The Qualitative Report, Miami, v. 23, n. 11, p. 2622-2633, 2018. DOI: https://doi.org/10.46743/2160-3715/2018.3492

BISWAS, S.S. Potential use of chat GPT in global warming. Annals of biomedical engineering, Piscataway, v. 51, n. 6, p. 1126-1127, 2023. DOI: https://doi.org/10.1007/s10439-023-03171-8

BOZKURT, A.; XIAO, J.; LAMBERT, S.; PAZUREK, A.; CROMPTON, H.; KOSEOGLU, S.; FARROW, R.; BOND, M.; NERANTZI, C.; HONEYCHURCH, S.; BALI, M.; DRON, J.; MIR, K.; STEWART, B.; COSTELLO, E.; MASON, J.; STRACKE, C. M.; ROMERO-HALL, E.; KOUTROPOULOS, A.; TOQUERO, C. M.; SINGH, L.; TLILI, A.; LEE, K.; NICHOLS, M.; OSSIANNILSSON, E.; BROWN, M.; IRVINE, V.; RAFFAGHELLI, J. E.; SANTOS-HERMOSA, G.; FARRELL, O.; ADAM, T.; THONG, Y. L.; SANI-BOZKURT, S.; SHARMA, R. C.; HRASTINSKI, S.; JANDRIĆ, P. Speculative futures on ChatGPT and generative artificial intelligence (AI): A collective reflection from the educational landscape. Asian Journal of Distance Education, Delhi, v. 18, n. 1, 2023.

BRUUN, E. P.G.; DUKA, A. Artificial intelligence, jobs and the future of work: Racing with the machines. Basic Income Studies, Berlin, v. 13, n. 2, p. 20180018, 2018. DOI: https://doi.org/10.1515/bis-2018-0018

CHING, T. et al. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, London, v. 15, n. 141, p. 20170387, 2018. DOI: https://doi.org/10.1098/rsif.2017.0387

CHRISTOU, P. A. A critical perspective over whether and how to acknowledge the use of artificial intelligence (AI) in qualitative studies. The Qualitative Report, Miami, v. 28, n. 7, p. 1981-1991, 2023a. DOI: https://doi.org/10.46743/2160-3715/2023.6407

CHRISTOU, P. A. How to use artificial intelligence (AI) as a resource, methodological and analysis tool in qualitative research? The Qualitative Report, Miami, v. 28, n. 7, p. 1968-1980, 2023b. DOI: https://doi.org/10.46743/2160-3715/2023.6406

CHUBB, L. A. Me and the Machines: Possibilities and Pitfalls of Using Artificial Intelligence for Qualitative Data Analysis. International Journal of Qualitative Methods, Thousand Oaks, v. 22, n. 1, p. 16094069231193593, 2023. DOI: https://doi.org/10.1177/16094069231193593

CONNEAU, A.; LAMPLE, G. Cross-lingual language model pretraining. Advances in neural information processing systems, Cambridge, v. 32, n.01, 2019.

FONSECA, A. L. A. da; CHIMENTI, P. C. P. S.; SUAREZ, M. C. Using deep learning language models as scaffolding tools in interpretive research. Revista de Administração Contemporânea, Maringá, v. 27, v. 03, p. e230021, 2023. DOI: https://doi.org/10.1590/1982-7849rac2023230021.en

HAMILTON, L.; ELLIOTT, D.; QUICK, A.; SMITH, S.; CHOPLIN, V. Exploring the Use of AI in Qualitative Analysis: A Comparative Study of Guaranteed Income Data. International Journal of Qualitative Methods, Thousand Oaks, v. 22, n. 01, p. 16094069231201504, 2023. DOI: https://doi.org/10.1177/16094069231201504

HOSSEINI, M.; HORBACH, S. PJM. Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other Large Language Models in scholarly peer review. Research Integrity and Peer Review, Palo Alto, v. 8, n. 1, p. 4, 2023. DOI: https://doi.org/10.1186/s41073-023-00133-5

KIRMANI, A. R. Artificial intelligence-enabled science poetry. ACS Energy Letters, Washington, v. 8, n. 1, p. 574-576, 2022. DOI: https://doi.org/10.1021/acsenergylett.2c02758

LONGO, L. Empowering qualitative research methods in education with artificial intelligence. In: World Conference on Qualitative Research. Cham: Springer International Publishing, Porto, v. 1068, p. 1-21, 2019. DOI: https://doi.org/10.1007/978-3-030-31787-4_1

LUND, B. D.; WANG, T. Chatting about ChatGPT: how may AI and GPT impact academia and libraries?. Library Hi Tech News, Leeds, v. 40, n. 3, p. 26-29, 2023. DOI: https://doi.org/10.1108/LHTN-01-2023-0009

LUND, B. D; WANG, T; MANNURU, N.R; NIE, B; SHIMRAY, S; WANG, Z. ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, Syracuse, v. 74, n. 5, p. 570-581, 2023. DOI: https://doi.org/10.1002/asi.24750

MESEC, B. The language model of artificial inteligence chatGPT-a tool of qualitative analysis of texts. [Preprint]. 2023. Available in: https://doi.org/10.22541/au.168182047.70243364/v1. DOI: https://doi.org/10.22541/au.168182047.70243364/v1

MILOYAN, B.; MCFARLANE, K.; VASQUEZ-ECHEVERRIA, A. The adapted Autobiographical interview: A systematic review and proposal for conduct and reporting. Behavioural Brain Research, Amsterdam, v. 370, p. 111881, 2019. Available: https://doi.org/10.1016/j.bbr.2019.03.050. DOI: https://doi.org/10.1016/j.bbr.2019.03.050

OpenAI. Introducing ChatGPT, 2022. Retrieved from https://openai.com/blog/chatgpt.

SANTOS, D. J. M.; SANTOS, P. V.; BOSS, S. L. B. A análise de conteúdo com apoio do software gratuito RQDA. Revista Pesquisa Qualitativa, São Paulo, v.11, n.28, p. 805-824, set./dez. 2023. DOI: https://doi.org/10.33361/RPQ.2023.v.11.n.28.612

SOBANIA, D.; BRIESCH, M.; HANNA, C.; PETKE, J. An analysis of the automatic bug fixing performance of chatGPT. [Preprint] arXiv:2301.08653, 2023. Available in: https://arxiv.org/pdf/2301.08653.pdf. Melbourne. Accessed on: October 9, 2023. DOI: https://doi.org/10.1109/APR59189.2023.00012

SOUZA, B. C.; SILVA, A. S.; SILVA, A. M.; ROAZZI, A.; CARRILHO, S. L. S. Putting the Cognitive Mediation Networks Theory to the test: Evaluation of a framework for understanding the digital age. Computers in Human Behavior, Québec, v. 28, n. 6, p. 2320-2330, 2012. DOI: https://doi.org/10.1016/j.chb.2012.07.002

TABONE, W.; WINTER, J. Using ChatGPT for human–computer interaction research: A primer. Manuscript submitted for publication, 2023. DOI: https://doi.org/10.1098/rsos.231053

TATE, T.; DOROUDI, S.; RITCHIE, D.; XU, Y. Educational research and AI-generated writing: Confronting the coming tsunami [preprint], 2023. Available in: http://dx.doi.org/10.35542/osf.io/4mec3. DOI: https://doi.org/10.35542/osf.io/4mec3

TREVISAN, R.; SERRANO, A.; WOLFF, J. F. S.; RAMOS, A. F. Peeking into their mental imagery: The Report Aloud technique in science education research. Ciência e Educação, Bauru, v. 25, n. 3, pp. 647-664, 2019. Available in: https://doi.org/10.1590/1516-731320190030004. DOI: https://doi.org/10.1590/1516-731320190030004

YANG, C.; WANG, X.; LU, Y.; LIU, H.; LE, Q. V.; ZHOU, D.; CHEN, X. Large language models as optimizers. [Preprint] arXiv:2309.03409, 2023. Available in: https://arxiv.org/pdf/2309.03409.pdf. Accessed on: October 9, 2023.

ZHONG, Q.; DING, L.; LIU, J.; DU, B.; TAO, D. Can chatGPT understand too? a comparative study on chatgpt and fine-tuned bert. [Preprint] arXiv:2302.10198, 2023. Available in: https://arxiv.org/pdf/2302.10198.pdf. Accessed on: October 9, 2023.