A Critical Analysis of Challenges and Washback Effects of English Language E-Assessment in Pakistan

Asra Irshad¹*, and Zafeer Hussain Kiani²

¹Department of English Linguistics and Literature, Riphah International University, Islamabad

²Department of English, University of Azad Jammu & Kashmir, Muzaffarabad

*Corresponding Author: [email protected]

Abstract

The use of information technology for assessment especially for educational purposes, is the need of the hour. The major types of e-assessment include multiple-choice tests and evaluation of problem-solving skills. The mode of e-tests is more appropriate and quicker than the outdated paper-based examinations. The application of electronic tests is challenging, especially in a teaching and learning environment. Therefore, this investigation is based on the perceptions of Pakistani students and teachers regarding e-tests along with their washback effects. In addition, the study also examines and explains the various difficulties and trials that Pakistani students face during e-assessment. The sample of the study consisted of 80 BS level English students and 20 EFL teachers from three different universities in Pakistan. Two previously designed questionnaires namely Teachers Perceptions and Attitudes Survey (TPAS) and Students Perceptions and Washback Effects Survey (SPACES) were adapted as instruments to survey the attitude of students towards e-tests. The obtained data was analysed statistically using SPSS software (IBM 23). The results revealed positive perceptions of Pakistani teachers and learners towards e-assessment. However, the findings also highlighted some negative washback effects and attitudes of the teachers towards e-exams. It is expected that the findings would help the teachers, learners, and policy developers in Pakistan to deduce better results through e-assessment.

Keywords: e-assessment, e-tests, perceptions, washback effects

Introduction

With teaching systems, it is appropriate for individuals in ESL or EFL contexts to be tested in one way or the other. Tests are needed to get information about the achievement of learners and to take educational decisions rationally (Hughes, 2003). There are a variety of logical and psychological factors that affect students" and teachers" performance in the exams. Such effects of taking the exams are termed as washback effects (Binnahedh, 2022). Alderson and Wall (1993) explained washbask effect as the influence of testing based on teaching and learning.

According to Green (2013), the impact of a test on teaching and learning that is done while preparing for it is called washback. Hughes (1989) describes washback effect as the approach that impacts a test"s teaching materials and classroom organization. According to Alderson and Wall (1993), washback causes the instructors and the learners to do the things that they would not otherwise do in their usual routine of teaching learning practices. The test effects relate to the consequences of an assessment on individuals, educational policies, or practices in the classroom, educational institution, and the society collectively (Wall, 1997). There is negative as well as positive effect of washback. Positive washback is the term for predictable results of a test, while negative washback is the term for unexpected or detrimental results of a test (Binnahedh, 2022).

Washback acts like a connection among teaching, testing, and learning procedures. The language assessments consider washback as an important dimension of the impact. It is quite reasonable to consider the context of teachers and learners in order to design tests that must contain the content that learners need to learn. Therefore, the tests should be comprehensive enough to clear language hurdles of the learners before pursuing academic careers (Chapelle & Douglas, 1993). Teachers need to teach while considering the content and the format of a test for their students (Swain 1985). Hughes (1994) proposed that the expectation of washback works only when the learners are interested to succeed in the test, have faith in their way to be successful and have strong belief in the sufficiency of resources to get successful. E-assessment is one of the most used modes of assessment in the present time.

According to Bennett (2002), the practice of e-assessment offers an effective and resourceful means of evaluating the effectiveness of teaching and learning by supporting old-styled and reliable testing procedures. The use of technology helps in providing new ways of assessing the learning. It also offers a variety of data sources and develops the resources to assist educators in getting learning command and teaching efficacy (Vendlinski & Stevens, 2002). After COVID-19, e-assessment is commonly used for educational purposes at university level in Pakistan. According to Khalique (2007), the functional proficiency of Pakistani learners in spoken and written English is less than two percent. The purpose of this paper is to examine the attitudes of students and teachers towards e-tests in Pakistan considering washback effects and challenges.

Research Questions

1. What are the attitudes and perceptions of Pakistani learners and teachers regarding e-assessment?
2. What are the washback effects of e-tests among Pakistani learners?
3. What are the attitudes and perceptions of Pakistani teachers towards e-tests?

Literature Review

Assessment is a fundamental requirement in educational setting as such practices typify curriculum development, describe what is worth knowing, and initiate useful classroom strategies. Online assessments are becoming increasingly significant at the university level. Binnahedh (2022) maintained that a test type has an immediate effect on the attitudes of the learners towards their teaching and learning chores and such attitudes effect their learning process like practice of the kind of the material that appears on the test influencing learning objectives. Cheng and Curtis (2004) argued that the one who operates an inquiry in a particular situation must determine if the outcomes of the assessment are positive or negative. It is essential to consider the setting of investigation as either school or university and the span of utilizing evaluation practices along with the way of conducting the study.

Hungs (2012) observed both positive and negative washback effects of e-portfolio tests on the learning. The study involved eighteen (18) English as a Foreign Language (EFL) students and teachers, registered in a program at master's level entitled as Teaching English to Speakers of Other Languages (TESOL). The four tools applied to collect data for this research include interviews, text analysis, observations, and reflective journals. The findings show positive learning washback effects of e-portfolio evaluations, such as forming a society of practice, encouraging peer learning, and refining content understanding and learning. The undesirable side effect of e-portfolio evaluations is the learning anxiety produced by wider viewers and aggression to the technology.

Tayeb et al. (2014) studied the washback impact of a Yemeni General Secondary English Examination. The study focused to have a better approach of the association between the washback effect and instruction approaches and learning characteristics. The investigation considered eight (8) educational factors. The first four of these were related to the instructors such as teaching approaches, teaching practices, content valuation, and beliefs. The other four were relevant to the pupils such as, learning patterns, learning interests, views, and motivation. The instruments for this research were a questionnaire and an interview. The results highlighted a significant impact of the test on the learners and teachers, mainly on teaching methods and learning techniques. The study exhibited a washback effect of the exams on the aspects of language teaching and learning methods in Yemen.

Allen (2016) studied the consequential validity of the IELTS academic exams, focusing on the effect of washback on students' test training approaches and score improvements. Also, he examined the facilitating features affecting washback in situations when students in an EFL context are not registered in the test preparatory courses. The grades of the tests displayed that the IELTS test had a helpful effect on students' language skills and test preparatory approaches, specifically in terms of practical skills, that learners had earlier overlooked in their language trainings. Nevertheless, to confirm positive washback in EFL circumstances and the lack of supervision, many interceding factors should be considered.

Saglam (2018) conducted a mixed-method study in Turkish EAP context to observe the washback effect in the vicinity on the theme-constructed and high-risks English language proficiency test at the tertiary level of education. The study intended to realize the amount of washback effect on teaching via an interactive theme-based English proficiency test which was specially designed to represent authentic language usage in the tertiary education system. The data was collected using classroom observations and focus group interviews involving fourteen (14) teachers from the Preparatory English Language Program (PEP). The results demonstrated both positive and negative test effects on teaching. The study recommended the role of both material and teaching methodology in causing the effect.

Ali and Hamid (2020) explored the factors that are central in causing a negative washback effect on English language teaching (ELT) at secondary school level in Bangladesh. The investigation suggested that keeping a testing-teaching causal relationship is very basic since testing and instruction are entangled. It is a compound of communal, psychological, administrative, financial, and data-driven liability variables that effect language education. In circumstances where language assessments are presumed to obstruct language coaching, these results have suggestions for language assessment and assessment investigators, policy officials, programme designers, and superintendents.

Since 2020, after the emergence of COVID-19 pandemic, crucial problems evolved, and critical situations are required in all areas of life including education. The teaching and learning at university level is mostly relying on online modes for both the regular sessions and the exams. The practice is successful however, the students and the teachers at undergraduate level feel difficulty in utilizing the e-tests throughout formative and summative examinations. The present investigation assesses the negative and positive washback effects, challenges, and attitudes regarding e-tests among instructors and learners in Pakistan to assess the scenario and indicate the solutions for developments.

Research Methodology

The study involved a mixed-method research design. The data was collected qualitatively through two surveys and analysed quantitatively via SPSS software to run the descriptive statistics.

Participants

The participants of the current study included 80 BS level students of English language (n=80) at Riphah International University, Pakistan and 25 EFL teachers (n= 25) from Riphah International University, Islamabad, University of Azad Jammu and Kashmir, Muzaffarabad, and University of Swabi, KP. The sample was chosen from universities in three distinct regions of the country to enable the generalization of the study's findings.

Instruments

The instrument used for the current study includes two kinds of surveys. The first survey is Teachers' Perceptions and Attitudes Survey (TPAS), developed by Binnahedh (2022). The second survey, Students' Perceptions and washback effects Survey (SPACES) was adopted from Nafziger et al. (2011) and Tayeb et al. (2014). Both the surveys were shared to specialized Pakistani EFL teachers (n=5) who had taught online through the COVID-19 phase to adapt and validate them.

Data Collection Procedures

The researchers have asked for consent of the head of the department at Riphah University for the self-administration of the surveys. The students of the second, fifth and seventh semesters expressed their willingness to join the study as participants. The teachers at the above-mentioned universities were approached separately for the completion of surveys. The participants were instructed about the proper way of responding to the questions in the surveys. The records of students' and teachers' perceptions and attitudes were saved separately in excel files for analysis via SPSS software.

Results and Discussion

The current study had three research questions to find out (a) the attitudes and perceptions of Pakistani learners and teachers regarding e-assessment, (b) the washback effects of e-tests among Pakistani learners and (c) the attitudes and perceptions of Pakistani teachers towards e-tests. The results obtained from the two surveys are reported in this section, followed by the discussion.

The first research question of the study inquired about the attitudes and perceptions of Pakistani learners and teachers regarding e-assessment. In order to respond to this question, descriptive statistics was applied and mean values were computed, as shown in the table below.

Table 1

Attitudes and Perceptions of Pakistani Learners and Teachers Regarding E-Assessment

The above table displays descriptive statistics for undergraduates' perceptions, and attitudes about online assessment. As shown in the Table 1, it was demonstrated that 10% of Pakistani EFL students (4.2 M) were dissatisfied with the issues related to the internet connection. The learners have poor availability of the internet connection which interrupts their test-taking practice. However, some of the students (7.5%) showed confidence in online assessment and interest in using computer devices for e-assessment (6.25%). The learners (3.4M) affirmed that they feel confident while using LMS for the online tests. Moreover, the students enjoyed (6.25%) and endorsed (3.7M) the use of e-assessment for English language testing in Pakistan. The findings of the study are in line with Saglam (2018), Jiamin et al. (2021), and Binnahedh (2022). Similarly, 2.5% of the participants (3.9M) consider the efficacy of e-learning means and the same percentage (2.5%) of the respondents (3.8M) accept the utility of online assessment for taking the language tests. Interestingly, 22.5% of the respondents (2.7M) affirmed that they never feel anxiety before online assessment and most of the respondents (2.1M) confessed that they seldom depend on the test administrators in e-testing labs. The findings show that the influence of e-tests on the behaviours of both the learners and teachers even if there are no issues in e- leaning or e-assessment environments. The findings highlight the encouraging attitude of Pakistani learners and teachers towards e-assessment. The finding is in line with the finding of Cheng and Curtis (2004) that both the instructors and the learners deal with e-learning and e-assessment positively. During COVID-19, the trend of online learning started in Pakistan and is still prevailing just because of its impressive impact on the education system. Figure 1 illustrates the means related to the attitudes and perceptions of e-assessment considering the views of Pakistani learners and teachers.

Figure 1

Attitudes and Perceptions of Pakistani Learners and Teachers Regarding E-Assessment

The second research question of the study attempted to see the washback effects of e-tests among Pakistani learners. Descriptive statistics was employed using SPSS to calculate mean values as demonstrated in the following table.

Table 2

Washback Effects of E-tests among Pakistani Learners

The above table shows descriptive statistics for washback effects of e-tests among the Pakistani learners. As demonstrated in Table 2, negative washback effect was displayed (3.9M) while focusing on the 2.5% of the teachers' teaching-to-the-test approach. It has been observed that e-tests effect behaviours of such instructors as they focus on teaching only the things which they intend to give to the students for examination. Some positive effects of e-tests were observed by some neutral responses and a few responses of agreements in this regard. It was noted that by 3.75% of the students (3.7 M), the e-assessment impact positively on instructors and motivate them to improve their methodology for English language teaching. Moreover, about 2.5% of the respondents (4.0 M) viewed that their teachers use extracurricular materials besides textbooks for teaching. On the other hand, it was observed that 1.25% of the learners (3.9 M) request for learning approaches to answer multiple-choice queries. This shows that the learners focus on the techniques to solve multiple choice questions instead of concentrating on learning their courses. There were 5% of the respondents (3.6M) who expressed their need for having an educational counsellor for the online exams. This is because 13.75% of the learners (3.6) were only occupied with the thoughts to get techniques to become successful in the online exams. In addition, it is considered that 11.2% of the learners (3.3 M) have realized that online tests have created disruption between the educational curriculum and students' creative skills. Moreover, it was perceived by 13.7% of the respondents (3.2M) that online tests inhibit teachers from elaborating grammatical rules in the classroom. Similarly, 26.25% of the respondents (2.5) also felt that such a platform does not allow a teacher to pay more attention to vocabulary items. In other words, this approach is affecting language learning process because the learning of vocabulary and grammatical rules are fundamental to learning any language. As reported by Khalique (2007), the existing population of Pakistan with complete functional proficiency in English is scantier than two percent. This shows that online platform is not suitable for the Pakistani learners for learning English as Pakistanis are non-native speakers as they need to learn all the aspects of a language appropriately. Likewise, most (40%) of the partakers (2.3) noticed that e-tests hinder the use of advanced teaching techniques. Due to the incompetence in comprehending the questions, a large majority (45%) of the respondents (1.9 M) expressed their dread and fear for the online tests. Figure 2 demonstrates the means of the washback effects of e-tests from Pakistani learners' view.

Figure 2

Washback Effects of E-tests among Pakistani Learners

The third research question of the study inquiries about the attitudes and perceptions of Pakistani teachers towards e-tests. Descriptive statistics were run to obtain mean values for each response, as shown in the table below.

Table 3

Attitudes and Perceptions of Pakistani Teachers towards E-tests

Table 3 presents different views of Pakistani teachers related to the e-tests. There were more agreements on the items related to the views of allowing students to try the things before e-exam and that online exams help in quick assessment. However, more disagreements were noted on the items related to the views of having no relation of students' success with e-exam and limitation of cheating options during the e-exams. As observed in the above table, none of the teacher (4.35 M) considered the utility of online exams. Likewise, the instructors (4.2M) were of the view that the students should not be worried while using the computer. So, 7.5% of the respondents (3.8 M) considered the reliability of e-exam results. However, it was believed by 10% of the respondents (3.6 M) that the number of automated exam queries is suitable for the students and the same percentage (10%) of the instructors (3.4 M) perceived that the students should be given the opportunity to try before they conclusively attempt the e-exam. Whereas, 2.5% of the instructors (3.3 M) considered that the online exam restricts cheating efforts. Similarly, 2.5% of the teachers (3.3 M) justified the idea of shifting towards an e-exam and therefore, the 7.5 % of the partakers (3.2 M) think that online exam should be taken seriously. Hence, it was observed that 5% of the instructors (3.0 M) authenticate the e-exam by stating that it measures what it is proposed to measure.

Furthermore, 11.2 % of the instructors (2.8 M) thought of no requirement of external aid for using the computer. Similarly, 12.5% of the partakers (2.6 M) estimated that the success of students in the comprehensive exam cannot be affected by the practice of an e-exam. Most (47.5 %) of the teachers (2.05 M) confirmed that the time permitted for e-examination is suitable for the students. Similarly, many (45 %) of the examiners (1.9 M) are contented with the conscripting of the passages in the form of multiple-choice questions for an online exam. Likewise, several (28.75%) instructors (1.7 M) contemplated that there is reasonable distribution of scores in exam paper for the online assessment. The results are consistent with the findings of Cheng and Curtis (2004) and Binnahedh (2022) that display encouraging and positive tackling of the online assessment by both the instructors and the learners. Figure 3 shows the means of the attitudes and perceptions regarding e-tests from Pakistani teachers' perspectives.

Figure 3

Attitudes and Perceptions of Pakistani Teachers towards E-tests

Considering Hughes' (1994) washback trichotomy model in our context, the participants, the procedure, and the product refer to separate entities. First, it is the students, teachers, administrators, policy makers, and publishers as stakeholders are called the participants. It has been observed that the switching of exams from paper-based system to e-assessment has strongly impacted on the behaviour of the participants. The teachers have tried to change their methodology to get good outcomes. This initiative helped in reducing gaps between instruction and assessment (Binnahedh, 2022). Almost 2.5% of the respondents agreed on having instructions of the assessment strategy. It was found out in most of the responses (10%) that the teachers described only those grammatical rules necessary to make the students aware of the multiple-choice strategies in e-examination. This strategy led the learners (1.25%) to concentrate on the techniques that facilitate in picking up the right option in multiple-choice questions. However, 45% of the respondents agreed on having anxiety and fear due to e-assessment which reflects high levels of psychological impact on the learners.

Secondly, we have the initiatives in educational setup like syllabus design, materials development, and instructional adjustments form the practice as second kind of washback effect. It has been realized that the creativity of the students is hindered with the type of questions in e-assessment, because of having most of the multiple-choice options highlight a breach between course syllabus and creativity (Binnahedh, 2022) and the students cannot express their abilities if they are not attempting an essay type question. The preference of test achievement over learning and understanding creates negative impacts for the teachers as they think that assessments are inadequate to make a connection between the tested material and the taught material (Johnson & Shaw, 2019). It was observed that the teachers were excited to change methodologies, but those practices were not of advanced level as they only concentrated on fulfilling the demands of their e-test patterns.

Lastly, it is the students' intake, learning excellence, and competencies which make the product (Bailey, 1996). According to Green (2007), there are two types of impacts of the tests. The first is the impact of tests on the participants like instructors, resources, and students involving their perceptions and attitudes. The second is the effect of tests on the processes that create variation in the teaching procedures and learning activities. This affects the learning objectives and outcomes (Green, 2007). Moreover, it can be illustrated that as by the nature, e-assessment can focus on low or intermediate level skills, so its high level or creative skills are ignored.

Conclusion

In light of the above stated findings, the research study observed that 10% of Pakistani EFL students were dissatisfied with the internet connection that hinders their ability to perform in e-assessment. However, 7.5% of the learners showed confidence in online assessment. Likewise, 6.25% of the students enjoyed using computer devices for e-assessment. Only 2.5% of the pupils endorsed the use of e-assessment for language testing in Pakistan. About 3.75% learners believed that e-assessment improve their instructors' methodology for English language teaching. On the contrary, 2.5% of the learners reviewed that e-tests effect behaviours of their teachers as chosen instruction material is intended for the exam. This reflects negative washback effects due to the teachers' teaching-to-the-test approach. Ominously, it was highlighted by the online dais that inhibit teachers from elaborating grammatical rules (13.7 %) and vocabulary items (26.25%) in English language classrooms. This reflects that such an approach is not suitable for teaching English to Pakistanis as vocabulary and grammatical structure are necessary aspects of language learning and can never be ignored at any level. Though the teachers were eager to change methodologies, but those practices were not of advanced level as they only concentrated on fulfilling the demands of their e-test patterns.

To sum up, it can be concluded that practically e-assessment can emphasize on low or intermediate level skills, but the high level or creative skills like functional language proficiency are completely ignored by this practise. The insufficient functional proficiency among Pakistani university students can be the result of the lack of appropriate pragmatic instruction in the ESL classroom and lack of attention towards the perfection of grammar and vocabulary of the students. This implies that all the required language skills should be taught to Pakistani EFL learners with the help of proper syllabus to develop their communicative competence. More studies are required to understand the roles of other participants such as course leaders, policy makers, textbook writers and even learners along with the experiments to see the linking of the particular practices to test characteristics and the linking of taught material to test scores of the students.

References

Alderson, J. C., & Wall, D. (1993). Does washback exist? Applied Linguistics, 14(2), 115–129. https://doi.org/10.1093/applin/14.2.115

Ali, M. M., & Hamid, M. O. (2020). Teaching english to the test: Why does negative washback exist within secondary education in Bangladesh? Language Assessment Quarterly, 17(2), 129–146. https://doi.org/10.1080/15434303.2020.1717495

Allen, D. (2016). Investigating washback to the learner from the IELTS test in the Japanese tertiary context. Language testing in Asia, 6, 1–20. https://doi.org/10.1186/s40468-016-0030-z

Bailey, K. M. (1996). Working for washback: A review of the washback concept in language testing. Language Testing, 13(3), 257–279. https://doi.org/10.1177/026553229601300303

Binnahedh, I. A. (2022). E-assessment: Wash-back effects and challenges (examining students' and teachers' attitudes towards e-tests). Theory and Practice in Language Studies, 12(1), 203–211. https://doi.org/10.17507/tpls.1201.25

Chapelle, C., & Douglas, D. (1993). Foundations and directions for a new decade of language testing. In D. Douglas & C. Chapelle (Eds.), A new decade of language testing research selected papers from the 1990 language testing research colloquium (pp. 1–24). TESOL Publications

Cheng, L., & Curtis, A. (2004). Washback or backwash: A review of the impact of testing on teaching and learning. In L. Cheng, Y. Watanabe & A. Curtis (Eds.), Washback in language testing: Research contexts and methods (pp. 3–18). Lawrence Erlbaum Associates.

Green, A. (2007). IELTS washback in context: Preparation for academic writing in higher education (Vol. 25). Cambridge University Press.

Green, A. (2013). Washback in language assessment. International Journal of English Studies, 13(2), 39–51. https://doi.org/10.6018/ijes.13.2.185891

Hughes, A. (1994). Backwash and TOEFL 2000. University of Reading.

Hughes, A. (2003). Testing for Language Teachers (2nd ed.). Cambridge University Press.

Hughes, A. (2020). Testing for language teachers. Cambridge University Press.

Hughes, J. (1989). Why functional programming matters. The Computer Journal, 32(2), 98–107. https://doi.org/10.1093/comjnl/32.2.98

Hung, S. T. A. (2012). A washback study on e-portfolio assessment in an english as a foreign language teacher preparation program. Computer Assisted Language Learning, 25(1), 21–36. https://doi.org/10.1080/09588221.2010.551756

Jiamin, X., Jinyan, L., & Tianyi, M. (2021). The washback effect of reformed CET 6 listening comprehension test. Asian Journal of Education and Training, 7(1), 70–73. https://doi.org/10.20448/journal.522.2021.71.70.73

Johnson, M., & Shaw, S. (2019). What is computer-based testing washback, how can it be evaluated and how can this support practitioner research? Journal of Further and Higher Education, 43(9), 1255–1270. https://doi.org/10.1080/0309877X.2018.1471127

Khalique, H. (2007). The Urdu-English relationship and its impact on Pakistan's social development. The Annual of Urdu Studies, 22, 99–112.

Nafziger, R., Meseke, J. K., & Meseke, C. A. (2011). Collaborative testing: The effect of group formation process on overall student performance. Journal of Chiropractic Education, 25(1), 11–15. https://doi.org/10.7899/1042-5055-25.1.11

Saglam, A. L. G. (2018). Can exams change how and what teachers teach? Investigating the washback effect of a university english language proficiency test in the Turkish context. Eurasian Journal of Applied Linguistics, 4(2), 155–176. https://doi.org/10.32601/ejal.464094

Swain, M. (1985). Large-scale communicative language testing: A case study. In Y. P. Lee, A. C. Y. Fok, R. Lord & G. Low (Eds.), New directions in language testing (pp. 35–46). Pergamon Press.

Tayeb, Y. A., Aziz, M. S. A., Ismail, K., & Khan, A. B. M. A. (2014). The washback effect of the general secondary english examination (GSEE) on teaching and learning. GEMA Online Journal of Language Studies, 14(3) 83–103. http://dx.doi.org/10.17576/GEMA-2014-1403-06

Wall, D. (1997). Impact and wash-back in language testing. In V. Edwards & D. Corson (Eds.). Encyclopedia of language and education (pp. 291–302). Kluwer Academic Publishers.

Vendlinksi, T., & Stevens, R. (2002). Assessing student problem-solving skills with complex computer-based tasks. The Journal of Technology, Learning and Assessment, 1(3), 1–20.