An analysis of the generative AI use as analyst in qualitative research in science education

Juliana Rodrigues dos Anjos; Maira Giovana de Souza; Agostinho Serrano de Andrade Neto; Bruno Campello de Souza

doi:10.33361/RPQ.2024.v.12.n.30.724

Autores

Juliana Rodrigues dos Anjos ULBRA https://orcid.org/0000-0001-7625-8372
Maira Giovana de Souza Universidade Luterana do Brasil (ULBRA) https://orcid.org/0000-0003-2987-0053
Agostinho Serrano de Andrade Neto Universidade Luterana do Brasil (ULBRA)
Bruno Campello de Souza Universidade Federal de Pernambuco

DOI:

https://doi.org/10.33361/RPQ.2024.v.12.n.30.724

Palavras-chave:

Análise qualitativa, Educação científica, TMC, ChatGPT, Claude

Resumo

O artigo avalia a eficácia de modelos de inteligência artificial generativa, especificamente ChatGPT 4.0 e Claude 2.0, na realização de pesquisas qualitativas na área de educação científica. Ao aplicar a Teoria da Mediação das Redes Cognitivas (CNMT) para analisar entrevistas de dois estudantes, constatou-se que Claude 2.0 superou o ChatGPT 4.0 na capacidade de reconhecer mediações cognitivas e distinguir entre condições de pré e pós-teste. Embora ambos os modelos tenham concordado sobre o conceito de evolução conceitual, Claude 2.0 demonstrou maior capacidade de detalhamento, notavelmente ao referenciar trechos específicos das entrevistas para apoiar suas análises quando solicitado. Em contraste, ChatGPT 4.0 exibiu dificuldades nessas áreas, mesmo quando fornecido com prompts adicionais. O estudo conclui reconhecendo a utilidade da IA, particularmente de Claude 2.0, para pesquisas qualitativas, enquanto também enfatiza o papel crítico da supervisão humana na detecção de potenciais deficiências dentro desses modelos.

Downloads

Não há dados estatísticos.

Métricas

Carregando Métricas ...

Referências

ANTHROPIC. Claude v.2.0. Available in: https://www.anthropic.com/index/100k-context-windows. Accessed on: October 9, 2023.

ATKINSON, C. F. Cheap, Quick, and Rigorous: Artificial Intelligence and the Systematic Literature Review. Social Science Computer Review, Thousand Oaks, v. 42, n. 02, p. 08944393231196281, aug. 2023. DOI: https://doi.org/10.1177/08944393231196281

BAROCAS, S.; SELBST, A. D. Big data’s disparate impact. California Law Review, Berkeley, v. 104, n. 3, 671-732, 2016. DOI: https://doi.org/10.2139/ssrn.2477899

BELOTTO, M. J. Data analysis methods for qualitative research: Managing the challenges of coding, interrater reliability, and thematic analysis. The Qualitative Report, Miami, v. 23, n. 11, p. 2622-2633, 2018. DOI: https://doi.org/10.46743/2160-3715/2018.3492

BISWAS, S.S. Potential use of chat GPT in global warming. Annals of biomedical engineering, Piscataway, v. 51, n. 6, p. 1126-1127, 2023. DOI: https://doi.org/10.1007/s10439-023-03171-8

BOZKURT, A.; XIAO, J.; LAMBERT, S.; PAZUREK, A.; CROMPTON, H.; KOSEOGLU, S.; FARROW, R.; BOND, M.; NERANTZI, C.; HONEYCHURCH, S.; BALI, M.; DRON, J.; MIR, K.; STEWART, B.; COSTELLO, E.; MASON, J.; STRACKE, C. M.; ROMERO-HALL, E.; KOUTROPOULOS, A.; TOQUERO, C. M.; SINGH, L.; TLILI, A.; LEE, K.; NICHOLS, M.; OSSIANNILSSON, E.; BROWN, M.; IRVINE, V.; RAFFAGHELLI, J. E.; SANTOS-HERMOSA, G.; FARRELL, O.; ADAM, T.; THONG, Y. L.; SANI-BOZKURT, S.; SHARMA, R. C.; HRASTINSKI, S.; JANDRIĆ, P. Speculative futures on ChatGPT and generative artificial intelligence (AI): A collective reflection from the educational landscape. Asian Journal of Distance Education, Delhi, v. 18, n. 1, 2023.

BRUUN, E. P.G.; DUKA, A. Artificial intelligence, jobs and the future of work: Racing with the machines. Basic Income Studies, Berlin, v. 13, n. 2, p. 20180018, 2018. DOI: https://doi.org/10.1515/bis-2018-0018

CHING, T. et al. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, London, v. 15, n. 141, p. 20170387, 2018. DOI: https://doi.org/10.1098/rsif.2017.0387

CHRISTOU, P. A. A critical perspective over whether and how to acknowledge the use of artificial intelligence (AI) in qualitative studies. The Qualitative Report, Miami, v. 28, n. 7, p. 1981-1991, 2023a. DOI: https://doi.org/10.46743/2160-3715/2023.6407

CHRISTOU, P. A. How to use artificial intelligence (AI) as a resource, methodological and analysis tool in qualitative research? The Qualitative Report, Miami, v. 28, n. 7, p. 1968-1980, 2023b. DOI: https://doi.org/10.46743/2160-3715/2023.6406

CHUBB, L. A. Me and the Machines: Possibilities and Pitfalls of Using Artificial Intelligence for Qualitative Data Analysis. International Journal of Qualitative Methods, Thousand Oaks, v. 22, n. 1, p. 16094069231193593, 2023. DOI: https://doi.org/10.1177/16094069231193593

CONNEAU, A.; LAMPLE, G. Cross-lingual language model pretraining. Advances in neural information processing systems, Cambridge, v. 32, n.01, 2019.

FONSECA, A. L. A. da; CHIMENTI, P. C. P. S.; SUAREZ, M. C. Using deep learning language models as scaffolding tools in interpretive research. Revista de Administração Contemporânea, Maringá, v. 27, v. 03, p. e230021, 2023. DOI: https://doi.org/10.1590/1982-7849rac2023230021.en

HAMILTON, L.; ELLIOTT, D.; QUICK, A.; SMITH, S.; CHOPLIN, V. Exploring the Use of AI in Qualitative Analysis: A Comparative Study of Guaranteed Income Data. International Journal of Qualitative Methods, Thousand Oaks, v. 22, n. 01, p. 16094069231201504, 2023. DOI: https://doi.org/10.1177/16094069231201504

HOSSEINI, M.; HORBACH, S. PJM. Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other Large Language Models in scholarly peer review. Research Integrity and Peer Review, Palo Alto, v. 8, n. 1, p. 4, 2023. DOI: https://doi.org/10.1186/s41073-023-00133-5

KIRMANI, A. R. Artificial intelligence-enabled science poetry. ACS Energy Letters, Washington, v. 8, n. 1, p. 574-576, 2022. DOI: https://doi.org/10.1021/acsenergylett.2c02758

LONGO, L. Empowering qualitative research methods in education with artificial intelligence. In: World Conference on Qualitative Research. Cham: Springer International Publishing, Porto, v. 1068, p. 1-21, 2019. DOI: https://doi.org/10.1007/978-3-030-31787-4_1

LUND, B. D.; WANG, T. Chatting about ChatGPT: how may AI and GPT impact academia and libraries?. Library Hi Tech News, Leeds, v. 40, n. 3, p. 26-29, 2023. DOI: https://doi.org/10.1108/LHTN-01-2023-0009

LUND, B. D; WANG, T; MANNURU, N.R; NIE, B; SHIMRAY, S; WANG, Z. ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, Syracuse, v. 74, n. 5, p. 570-581, 2023. DOI: https://doi.org/10.1002/asi.24750

MESEC, B. The language model of artificial inteligence chatGPT-a tool of qualitative analysis of texts. [Preprint]. 2023. Available in: https://doi.org/10.22541/au.168182047.70243364/v1. DOI: https://doi.org/10.22541/au.168182047.70243364/v1

MILOYAN, B.; MCFARLANE, K.; VASQUEZ-ECHEVERRIA, A. The adapted Autobiographical interview: A systematic review and proposal for conduct and reporting. Behavioural Brain Research, Amsterdam, v. 370, p. 111881, 2019. Available: https://doi.org/10.1016/j.bbr.2019.03.050. DOI: https://doi.org/10.1016/j.bbr.2019.03.050

OpenAI. Introducing ChatGPT, 2022. Retrieved from https://openai.com/blog/chatgpt.

SANTOS, D. J. M.; SANTOS, P. V.; BOSS, S. L. B. A análise de conteúdo com apoio do software gratuito RQDA. Revista Pesquisa Qualitativa, São Paulo, v.11, n.28, p. 805-824, set./dez. 2023. DOI: https://doi.org/10.33361/RPQ.2023.v.11.n.28.612

SOBANIA, D.; BRIESCH, M.; HANNA, C.; PETKE, J. An analysis of the automatic bug fixing performance of chatGPT. [Preprint] arXiv:2301.08653, 2023. Available in: https://arxiv.org/pdf/2301.08653.pdf. Melbourne. Accessed on: October 9, 2023. DOI: https://doi.org/10.1109/APR59189.2023.00012

SOUZA, B. C.; SILVA, A. S.; SILVA, A. M.; ROAZZI, A.; CARRILHO, S. L. S. Putting the Cognitive Mediation Networks Theory to the test: Evaluation of a framework for understanding the digital age. Computers in Human Behavior, Québec, v. 28, n. 6, p. 2320-2330, 2012. DOI: https://doi.org/10.1016/j.chb.2012.07.002

TABONE, W.; WINTER, J. Using ChatGPT for human–computer interaction research: A primer. Manuscript submitted for publication, 2023. DOI: https://doi.org/10.1098/rsos.231053

TATE, T.; DOROUDI, S.; RITCHIE, D.; XU, Y. Educational research and AI-generated writing: Confronting the coming tsunami [preprint], 2023. Available in: http://dx.doi.org/10.35542/osf.io/4mec3. DOI: https://doi.org/10.35542/osf.io/4mec3

TREVISAN, R.; SERRANO, A.; WOLFF, J. F. S.; RAMOS, A. F. Peeking into their mental imagery: The Report Aloud technique in science education research. Ciência e Educação, Bauru, v. 25, n. 3, pp. 647-664, 2019. Available in: https://doi.org/10.1590/1516-731320190030004. DOI: https://doi.org/10.1590/1516-731320190030004

YANG, C.; WANG, X.; LU, Y.; LIU, H.; LE, Q. V.; ZHOU, D.; CHEN, X. Large language models as optimizers. [Preprint] arXiv:2309.03409, 2023. Available in: https://arxiv.org/pdf/2309.03409.pdf. Accessed on: October 9, 2023.

ZHONG, Q.; DING, L.; LIU, J.; DU, B.; TAO, D. Can chatGPT understand too? a comparative study on chatgpt and fine-tuned bert. [Preprint] arXiv:2302.10198, 2023. Available in: https://arxiv.org/pdf/2302.10198.pdf. Accessed on: October 9, 2023.