Moderating Role of ChatGPT Usage between Self-Efficacy and Student Engagement Among Students in Pakistan: Toward Academic Technoutopia

1. INTRODUCTION

A growing body of research is focusing on investigating factors associated with the usage of GenAI applications like ChatGPT, such as attitudes toward and effects of GenAI usage in academia (Joshi et al., 2021; Majeed, Khan, Munir et al., 2024; Majeed, Khan et al., 2024) and explores how to ethically incorporate AI into academia. While this paper is primarily focused on investigating the moderating effect of ChatGPT usage on psychological and academic factors (self-efficacy and student engagement), it has also attempted to further advance the understanding of the current concerns of AI-integration into curriculum by introducing the concept of “academic technoutopia”.

Technological utopianism is a philosophical ideology that envisions a society achieving a perfect or near-perfect state of well-being in all aspects of life by integration of technological and scientific advancements (Fazekas, 2023; Kling, 1996). This closely resembles the ideas of transhumanism (Serap & Gulsun, 2019). Academic technoutopia therefore refers to an educational setting where technological advancements have been effectively integrated and routinely practiced, enhancing quality of education, pedagogy of educationists, and competence of students to a perfect or near-perfect state. Multiple open-source GPT models-based applications are now globally available, ChatGPT has particularly received considerable attention in academia. While the tool helps professionals become more efficient in goal-achievement, over-reliance on the tool has also sparked a major debate on whether ChatGPT and other GenAI tools should be banned in academia considering the users’ reservations on repetitive responses and issues of compromised academic integrity (Kishore et al., 2023; Yu, 2023). Advancements in GenAI has almost replaced traditional learning with students using AI as personal tutor and content creator to fulfil assigned tasks. Multiple research studies indicate teachers’ and learners’ reservations and negative attitudes toward these tools due not to just the issues of academic integrity but a risk of decline in cognitive skills including memory and learning (Bai et al., 2023; Majeed, Khan, Munir et al., 2024), although contradictory results have also been reported in the latest experimental investigations (Essel et al., 2024; Lee & Quan, 2024). This leads us to question whether an academic technoutopia can be achieved when contradictory findings are constantly surfacing. The present research addresses this question by providing corroborative evidence for impact of integration of GenAI tools, specifically ChatGPT, in academia.

Empirical evidence has revealed that negative perceptions toward AI-integration in curricula comes from the lack of digital literacy on how to utilize these tools effectively (Lee & Quan, 2024; Majeed, Khan et al., 2024). Digital literacy leads to improved confidence and self-efficacy in students (Prior et al., 2016; Yuan et al., 2024). ChatGPT works as a personalized tutor which can also help students advance their digital literacy allowing them to be more engaged in different learning environments. Self-efficacy refers to self-confidence in one’s abilities to successfully execute a behavior, consistent with Social Cognitive Theory (Bandura, 2012b). Higher self-efficacy leads to a better sense of control over one’s goals and higher motivation to achieve them, this achievement helps the individual fulfil psychological needs leading to higher engagement and higher self-driven efforts (Ghbari et al., 2024). These ideas fit with Self-Determination Theory (Van Lange et al., 2012) and point to the close association of student engagement with self-efficacy in academic context (Ghbari et al., 2024). Student engagement is a multidimensional construct consisting behavioral (rules and regulation adherence and academic activities involvement including extracurricular activities), emotional (students’ attitude, sentiments, and experiences encompassing their intrinsic motivation and curiosity toward knowledge acquisition, and perception of their academic setting), and cognitive (mental processes involved in student learning) aspects (Fredricks & McColskey, 2012).

Research investigating how ChatGPT usage interacts with self-efficacy to influence student engagement levels is scarce (Lee & Quan, 2024; Majeed, Khan et al., 2024; Majeed, Khan, Munir et al., 2024), specifically in Pakistan where academia has seen significant fluctuations post-COVID ranging from a transition to online learning management systems to government budget allocations and education quality (Farrukh et al., 2023; Shah et al., 2022). Most of the existing studies are either qualitative in nature or study the association of these factors on surface level. Lee and Quan (2024) particularly attempted to understand the internal mechanism between ChatGPT literacy and academic engagement (a dimension of student engagement) through the serial mediation of academic confidence and competence. The findings of the study also sanction the positive association of digital literacy and student engagement dimensions being significantly mediated by student confidence; however, we argue that a moderation model with ChatGPT usage, self-efficacy and the dimensions of student engagement would be worth the investigation. This moderation model also helps us highlight the differences between students who utilize GenAI for attaining their academic goals and those who do not. Further implications involve the interventions for improving the digital literacy and AI-integration in curricula, and movement of the academia toward an academic technoutopia.

• Frequent users of ChatGPT will significantly differ from infrequent users in self-efficacy and all dimensions of student engagement (academic engagement, cognitive engagement, social engagement with peers, social engagement with teachers, and affective engagement).
• ChatGPT usage will significantly moderate the relationship between self-efficacy and student engagement.

Method

The present study utilized a correlational research design to investigate the relationship between study variables, and to test a moderation model. Comparative analyses were also carried out that are reported in the results section.

Participants

297 young adult undergraduate students aged 18 to 26 years including 163 females (M = 21.63, SD = 1.59), and 134 males (M = 21.75, SD = 1.90) involving 127 self-reported infrequent users while 170 self-reported frequent users of ChatGPT. The data were collected from voluntary students in-person from different universities of Lahore including University of Central Punjab, University of Management and Technology, University of the Punjab, and Lahore Garrison University, Pakistan, using purposive-convenience sampling technique.

Participants who used ChatGPT for purposes other than education were excluded from the sample. Individuals who never used the tool were also not included in the research. All research participants self-reported middle-class family backgrounds (four students reporting otherwise were excluded from the study to avoid the influence of family-income or socio-economic status). All included participants were unmarried.

GSE is a 10 item self-report measure of general self-efficacy rated on a 4-point scale ranging 1 (not at all true) to 4 (exactly true), involving no reverse scored items. The scale has been widely used in previous research studies indicating good reliability ranging from .76 to .90 and validity (Schwarzer & Jerusalem, 1995). The English version of the scale was used in this study. The general self-efficacy scale was preferred over academic self-efficacy scales because it taps the individual’s overall resilience, resolve, and motivation to achieve their goals and solve problems, which fits with existing empirical evidence indicating general self-efficacy paired with resilience to be more unique predictor of academic motivation (Abdolrezapour et al., 2023), and identifies global self-beliefs that may influence the impact of technology usage on student engagement. The scale yielded a reliability of α = .84.

HSES is a 28-item self-report measure that assesses student engagement on a 5-point scale ranging from 1 (strongly disagree) to 5 (strongly agree), on 5 dimensions i.e., academic engagement (having two subscales i.e., academic learning behaviors, and online engagement), cognitive engagement, social engagement with peers (with two subscales peer engagement, and beyond-class engagement), social engagement with teachers and affective engagement. All sub-dimensions of the scale showed good reliability ranging .70 to .87. The Cronbach’s alpha of the whole scale was α = .88.

At the time of data collection (December 2023 to February, 2024) no commercially available scale measured ChatGPT usage therefore, participants were first screened using an information sheet. ChatGPT usage was then assessed through a self-report demographic dichotomous variable named “ChatGPT usage” rated on two levels of usage i.e. rarely and frequently. “Rarely” referred to using ChatGPT at least once a week for academic purposes, and “frequently” referred to using ChatGPT more than two times per week (Lee & Quan, 2024) for academic purposes. Users with self-reported rare usage were categorized as infrequent users while users self-reporting frequent usage were categorized as frequent users of ChatGPT.

After gaining institutional approval for research conduction (see ethical considerations), open access research scales (GSE and HSES publicly available) were utilized in the study. Data were collected using purposive sampling technique on paper printed questionnaires. Each questionnaire included a consent form, demographic sheet, and study scales along with their copyright information clearly mentioned. It took approximately 10 to 15 minutes for completion of questionnaire by each participant. Data were analyzed using IBM SPSS (version 23). The differences between ChatGPT’s frequent and infrequent users were analyzed using independent sample t-test, the moderation analysis was carried out using Hayes PROCESS Macro extension of SPSS, and the relationship between the study variables was investigated using Pearson product-moment correlation. Ethical standards were carefully considered and maintained throughout the process, which are reported as follows.

The study was reviewed and approved by the ethics’ committee of the Department of Psychology, University of Central Punjab, Lahore. All participants were provided information about the purpose of the study and consent was gained before participation. Individual identifiers were removed from the data and replaced with case IDs to maintain confidentiality. The raw data of participants was only accessible by the authors of this study. Participation in the research was voluntary and all participants were given the right to withdraw at any point. Results are reported without any fabrication or biased manipulation of data.

Results

The mean differences of self-efficacy and student engagement were analyzed between frequent users and infrequent users of ChatGPT. Both groups were significantly different on self-efficacy, online engagement, peer engagement, and beyond-class engagement. Frequent users reported higher self-efficacy and online engagement whereas, infrequent users reported higher peer and beyond-class engagement (Table 1). The results for other variables were statistically insignificant.

Table 1

Results of Mean Based Comparisons of Frequent and Infrequent ChatGPT Users (N = 297)

Further, to understand the nature of relationship between self-efficacy and student engagement, partial correlation analysis was carried out while controlling for ChatGPT usage. Self-efficacy had a significant positive correlation with student engagement and all of its sub-dimensions except beyond-class engagement (Table 2). This indicated that students with higher self-efficacy were more engaged (in terms of academic learning behaviors, online engagement, cognitive engagement, social engagement with peers and teachers, and affective engagement).

Table 2

Descriptive Statistics and Correlations Between Self-Efficacy and Student Engagement While Controlling for ChatGPT Usage (N = 297)

Note. **p < .01. ***p < .001.

It is important to note that when correlation analysis was carried out across groups for frequent and infrequent users of ChatGPT, self-efficacy did not show significant relationship with overall academic engagement for infrequent users while the relationship was significant for frequent users (Table 3).

Table 3

Descriptive Statistics and Intercorrelations for Study Variables Disaggregated by Frequency of ChatGPT Usage (N = 297)

Note. lower diagonal shows correlations for students with infrequent usage of ChatGPT (n = 127); upper diagonal shows correlations for students with frequent usage of ChatGPT (n = 170).
*p < .05. **p < .01. ***p < .001.

Moderation analysis was then carried out with ChatGPT usage as a dichotomous moderator (infrequent versus frequent) between self-efficacy and student engagement, using Hayes’ PROCESS (version 4.1) with 5000 bias-corrected samples (model 1), Davidson-McKinnon heteroscedasticity-consistent inference (HC3), and mean-centered continuous variables. Interactions were probed at p < .05, with the conditional effects tested using mean and standard deviation. The values of VIF and tolerance estimated on SPSS suggested that self-efficacy and ChatGPT usage did not have any issue of multicollinearity (Table 4).

Table 4

Interaction Effect of Self-Efficacy and ChatGPT Usage on Student Engagement (N = 297)

Figure 1

Interaction Effect of Self-Efficacy and ChatGPT Usage on Student Engagement

The significant moderation model in Table 4 indicated 12.98% variance (through R²) in student engagement explained by self-efficacy and ChatGPT usage. Self-efficacy had a significant direct positive effect (1 SD units increase in self-efficacy suggesting .8 SD units increase in student engagement) while ChatGPT usage had a significant direct negative effect on student engagement (1 SD unit increase in ChatGPT usage suggesting 4.56 SD units decrease in student engagement). The interaction effect of both variables was also significant and positive (suggesting 0.86 units increase in student engagement, Figure 1).

Figure 1 shows the slopes and intercepts for self-efficacy (predictor) and student engagement (outcome) of low/infrequent and high/frequent users of ChatGPT (moderator).

Figure 2

Scatter Plot for Moderation Analysis

Note. The figure shows the scatter plot for the conditional effect of self-efficacy on student engagement for low/infrequent and high/frequent users of ChatGPT.

The conditional effect was significant only for frequent users. Self-efficacy of infrequent ChatGPT users did not have a significant effect on student engagement, while self-efficacy of frequent ChatGPT users was positively associated with student engagement (Figure 2).

Discussion

This study was conducted to investigate the interaction effect of self-efficacy and ChatGPT usage on student engagement. It was hypothesized that there would be a significant difference between frequent and infrequent users of ChatGPT on self-efficacy and student engagement, self-efficacy will be significantly correlated with student engagement (and its dimensions), and ChatGPT usage will significantly moderate the relationship between self-efficacy and student engagement. We anticipated that the relationship between the study variables will be strengthened by the interaction effect of a GenAI tool usage (ChatGPT), providing corroborative evidence for the possibility of the philosophical concept of academic technoutopia transitioning from an ideal to reality.

The research findings suggested a significant difference between frequent and infrequent users of ChatGPT on self-efficacy, online engagement, peer engagement, and beyond-class engagement with frequent users showing comparatively higher levels of self-efficacy and online engagement while infrequent users showing higher levels of peer engagement and beyond-class engagement. These findings are consistent with the works of Rudolph et al. (2023) and Khan et al. (2024). Rudolph et al. (2023) revealed in their study that students who utilized the services of ChatGPT reported higher confidence levels providing them more self-reliance for working on challenging tasks. Here, it would be noteworthy to reflect on Vygotsky’s (1978) “Zone of Proximal Development” in the modern age when students are more confident and, as many critiques are suggesting, require little to no assistance from teachers with human instructors being rapidly replaced with AI tools. Higher levels of online engagement in the frequent users is also justified due to their level of engagement with the tool itself. This notion also justifies why frequent users have comparatively lower levels of peer and beyond-class engagement. Individuals are now utilizing the GenAI tools for casual conversations and companionship (Olasik, 2023; Spinei, 2024), leading to a reduced need for socialization with peers. Lessened peer and beyond-class engagement in more frequent users of ChatGPT might be due to the higher online engagement and increased confidence of these users to get their tasks done quickly, with minimal effort, and without peer or teacher assistance. Future researchers can explore the path relationships between such variables for GenAI users.

Further, a positive correlation was found between the study variables except beyond-class engagement. As beyond-class engagement is based on intrinsic interests and social factors (Burke et al., 2024; Wang & Hofkens, 2020), rather than confidence in one’s capabilities, it does not seem directly relevant to self-efficacy. Hence, student engagement in beyond-class activities (extracurricular activities) is not directly linked to self-efficacy whereas, academic learning behaviors, the affective and cognitive aspects of student engagement or social engagement with teachers and peers are directly linked to their self-perception of being able to effectively contribute to curricular activities as these social engagements enhance self-efficacy (Benlahcene et al., 2024; Griffiths et al., 2021). Thus, self-efficacy is directly linked to student engagement and social interactions within educational context. Its connection to beyond-class activities, while present, is influenced by various factors and may not be as pronounced. However, it is also noteworthy that when the results were run separately for frequent and infrequent users, the infrequent users showed no significant relationship between self-efficacy and student engagement while the correlation for frequent users was significant including beyond-class engagement.

Lastly, significant moderating effect of ChatGPT usage was found for the relationship between self-efficacy and student engagement. It was observed that if students’ general self-efficacy is high, their engagement is also higher while if their self-efficacy is low, their engagement in academic activities also decreases due to the lack of confidence in their own abilities to succeed in academia, which provides evidence for the issues of over-reliance of students on technological assistance.

Previous studies have indicated cognitive decline in students using GenAI and other AI assistants (Bai et al., 2023; Khan et al., 2024), consequently, a positive feedback loop is highly likely where students over-rely on technology due to lack of self-efficacy which in turn further reinforces their lower self-efficacy levels and leads to cognitive decline in students. Thus, interventions designed to effectively improve self-efficacy can be expected to increase student engagement. ChatGPT usage also enhances the positive impact of self-efficacy on engagement in frequent users, possibly by providing resources or tools that empower confident learners. Hence, for high ChatGPT users developing higher self-efficacy is very important else, they might develop over-reliance on technology leading to a decline in cognitive skills.

Moreover, the conditional effect for infrequent users was insignificant which is likely due to the underlying functional properties of self-efficacy. Self-efficacy involves mastery experiences and resilience (Bandura, 2012a), if an individual has achieved success quickly with minimal effort, they are likely to be demotivated quickly on failure. The engagement associated with the use of AI assistants is likely linked to the quick achievement of success. This leads to reduced resilience in individuals. Consequently, students who have lower self-efficacy or non-resilient high self-efficacy are not likely to maintain levels of engagement.

Largely, there are mixed findings for the benefits and challenges of AI-integration in curricula, most of the emerging research points to the potential benefits of this transition. Frequent ChatGPT usage is linked with positive outcomes while the associated threats and challenges can be reduced by increasing digital literacy and self-efficacy of students. If AI tools are introduced to the students by teachers and instructors themselves after integration of these tools in the course outlines on a deeper level, and quality of formative assessments is enhanced reflecting higher-order thinking, then a state of academic technoutopia seems achievable and highly beneficial. While this form of technoutopia has already started developing, it still entails many challenges such as, over-reliance on technology, limited economic resources and issues of equality of access to these resources, cyber-footprint and privacy concerns, the black-box problem of AI and biased responses etc. Despite a long list of challenges ahead of us, the solutions are possible and benefits of these tools are also already being availed (Kasneci et al., 2023) and thus, we have already started our journey to achieving an academic technoutopia.

This research provided corroborative evidence for the benefits of integrating technology, particularly AI-LLM models such as ChatGPT, in academia. Positive association of digital literacy and student engagement while considering the important role of students’ general self-efficacy can be inferred from the research results. However, investigation of a moderation model was unique to the present study, unlike existing research focusing on cross-sectional mediation models which have been criticized for lacking statistical accuracy (Maxwell & Cole, 2007). This moderation model helps highlight important differences between students who utilize GenAI for attaining their academic goals and those who do not, where frequent users had better self-efficacy and online academic engagement keeping up with the advancements in modern curricula and academic demands while infrequent users had better scores on social academic engagement (peer-engagement and beyond-class engagement) highlighting the emerging problems of social isolation and gradual change in Pakistan’s social fabric of collectivism. These research results can be utilized to develop interventions for improving the digital literacy and AI-integration in curricula and transitioning the academia toward an academic technoutopia, and for improving tele mental health services for students.

Despite study’s principal contributions, some important limitations must be acknowledged. When this study was being conceptualized, no assessment measure of ChatGPT usage and researchers were constrained to use self-report, often dichotomous variables, as for the present research. This was relatively a very coarse representation of students’ interaction with the tool which may have masked important variations in usage patterns. Recently, multiple tools have been developed and introduced that measure complex dynamics of how individuals interact with such AI technology (Köhler & Hartig, 2024; Nemt-Allah et al., 2024; Yu et al., 2024). Future researchers should utilize these tools and investigate further models addressing the issues of AI-integration in curricula such as the path relationships between different dimensions of student engagement for GenAI users. Moreover, this study was limited in its scope in terms of sociodemographics (such as family income and religion), future researchers should aim to include more diverse populations to examine further trends of variations in self-efficacy and student engagement across different contexts. Further research can be carried out to explore AI’s impact on students’ engagement in extracurricular activities as well to provide a more comprehensive understanding of how students interact with AI. By acknowledging these limitations, future research can build on the present study to develop better interventions to enhance self-efficacy of students who utilize these tools for academic purposes so that the issue of over-reliance on technology can be addressed.

In conclusion, this study highlights the significant impact of self-efficacy and ChatGPT usage on various dimensions of student engagement, emphasizing the potential of AI tools to foster academic technoutopia. While frequent use of ChatGPT enhances self-efficacy and online engagement, it also underscores the need to address challenges like over-reliance and cognitive decline. Integrating AI tools into curricula with an emphasis on self-efficacy and digital literacy can help maximize their benefits, paving the way for a balanced and equitable academic future.

Author Contribution

Syed Mujtaba: conceptualization, data curation, investigation, writing – original draft, methodology. Mahnoor Ali: conceptualization, supervision, project administration, formal analysis, methodology, resources, visualization, writing – review & editing.

Conflict of Interest

The authors of the manuscript have no financial or non-financial conflict of interest in the subject matter or materials discussed in this manuscript.

Data Availability

The data associated with this study will be provided by the corresponding author upon request.

Funding Details

No funding has been received for this research.

Generative AI Disclosure Statement

The authors did not used any type of generative artificial intelligence software for this research.