User Satisfaction Analysis  of the Regular Live Unpad Learning Management System Website with Webqual 4.0 and Importance Performance Analysis Methods

 

Damar Dhayita Mulya Aryanti1*, Dika Jatnika2

Universitas Padjadajran, Bandung, Indonesia1,2

Email: damar20002@mail.unpad.ac.id, dika.jatnika@unpad.ac.id

 

KEYWORDS

ABSTRACT

Importance, Performance, Regular LiVE Unpad, User Satisfaction, Webqual 4.0.

Background: Since the onset of the COVID-19 pandemic, educational institutions, including Universitas Padjadjaran (Unpad), have increasingly utilized Learning Management Systems (LMS) like Regular LiVE Unpad to facilitate remote learning. Despite its continued post-pandemic usage, there is a notable absence of user satisfaction surveys conducted on this LMS at Unpad. Objectives: In response to this gap in evaluation, the author conducted research aimed at identifying areas for improvement on the Regular LiVE Unpad LMS website. This research specifically focused on three dimensions of Webqual 4.0: usability, information quality, and service interaction quality. Methods: The research methodology involved distributing questionnaires to users of Regular LiVE Unpad to assess their satisfaction levels and to pinpoint indicators requiring enhancement. A gap analysis was employed to compare the importance score (3.13) with the performance score (2.73), indicating areas where user expectations were not being fully met. Results: The findings revealed that the LMS, as indicated by the gap analysis results, falls short of achieving user satisfaction, with the average importance score exceeding the average performance score. This discrepancy underscores the need for improvements in various aspects of the LMS. Conclusions: Based on the Importance Performance Analysis, several recommendations were identified to enhance user satisfaction with the Regular LiVE Unpad. These recommendations include implementing real-time notifications, integrating live chat features, and incorporating comment sections directly within the course interface. These enhancements are crucial for aligning the LMS with user expectations and improving overall satisfaction levels.

DOI:

 

Corresponding Author: Damar Dhayita Mulya Aryanti1*

Email: damar20002@mail.unpad.ac.id

INTRODUCTION

The development of technology and information in the era of the Industrial Revolution 4.0 has encouraged the creation of various inventions that play a significant role in the progress and modernization of human civilization. Along with the development of the times and all the problems that follow it, various inventions created with the help of technology are used as solutions by the community to solve problems that arise in the midst of the daily life of the community itself. The application of this technology has gradually become commonplace and makes the relationship between the community and technology so close. Technology has found its own way to enter most aspects of life and is considered to be able to connect and facilitate people's lives. According to the technological pyramid consisting of various levels, it can be concluded that technology can move its hierarchy according to the needs of society because basically technology exists due to problems that cause anxiety (Van Mensvoort, 2013). Based on the technological pyramid triggered by, the peak of the technological pyramid is Mensvoort (2013)  the Naturalized stage where technology is considered to be able to change the world, such as the internet as one of them. The various benefits of the internet can be felt in various sectors of life, including in the education sector. The internet in the education sector plays a role in providing information systems such as digital libraries or other open-source sites. In addition, the Internet allows the use of tools that can support and facilitate the distance learning process such as teleconference applications and Learning Management Systems (Pratama, 2021).

The most significant usefulness of the internet was felt when the world was hit by the COVID-19 Pandemic where community activities were limited so that people could do their activities from their homes. In response to these restrictions, UNESCO provides recommendations for educational institutions to organize distance learning activities. According to data as of April 13, 2020, the Ministry of Education, Culture, Research, and Technology (Kemendikbudristek), stated that there were 68.73 million students recorded in distance learning. This accelerates the use of e-learning that uses the help of various devices and other electronics (Cerelia, Sitepu, & Toharudin, 2021). E-learning is an abbreviation obtained from electronic learning and can be defined as a learning activity that occurs with the use of the internet in the process to facilitate and allow the distance learning process to occur    (Sit & Brudzinski, 2017). Meanwhile, e-learning is defined as all learning activities that use the help of electronic technology. Basically, e-learning optimizes the use of applications and other educational platforms to create a teaching and learning process that can be carried out online Rusman (2018).

A Learning Management System (LMS) is one of the platforms used to support learning activities. A Learning Management System (LMS) is a system that is simplified in order to carry out the learning process of a website. LMS is generally used by educational institutions to upload learning materials or modules and manage online teaching and learning activities. In addition, LMS also allows its users to collaborate and communicate between users; create, manage, and assess assignments and exams given; generate usage time reports and more for teachers and administrators; integrate LMS with other educational applications or platforms such as Google Meeting, ZOOM, and so on; and get access and services from gadgets or computers or laptops. The novelty or contribution of Learning Management Systems (LMS) in supporting e-learning activities compared to previous studies is not explicitly discussed (Kruse et al., 2023).

The Learning Management System (LMS) owned by Padjadjaran University (Unpad) is called LiVE Unpad. LiVE stands for Learning in Virtual Environment. LMS a functions as an online learning management system designed to meet various blended learning needs of the Unpad academic community. The implementation of this system is carried out with the hope that learning objectives can be achieved effectively. LiVE Unpad can be likened to a virtual campus building where lecturers and students can connect and hold online learning activities.

 

METHODS

This research was carried out using quantitative methods to achieve the research goal itself, which is to determine user satisfaction with the quality of the website based on the comparison of the actual condition/performance of the website to the expectations of users. User satisfaction assessment based on the expectancy disconfirmation theory (Grimmelikhuijsen & Porumbescu, 2017) model can be formulated as follows:

a)    If the expectation (importance) is less than the performance (performance), then it is very satisfied.

b)   If the expectation (importance) is the same as the performance (performance), then satisfied.

c)    If the expectation (importance) is greater than the performance (performance), then it is not satisfied.

Measuring the quality of a website using Webqual 4.0 involves the participation of site users to assess the extent to which the quality of the website corresponds to their perception (Andriansyah, Suryani, & Putri, 2018). The WebQual Index provides an overall assessment of a website based on the customer's perception of quality weighed based on its importance/expectations. The Webqual 4.0 instrument has 22 indicators where each indicator can be rated on a Likert scale of 1 (Strongly Disagree) to 4 (Strongly Agree). Here are the indicators of the instrument: (Legowo, Ismiyati, & Ungu, 2024).

 

Table 1.

Webqual Instruments 4.0

Dimension

Code

Indicators

Usability

(Usability)

UV1

Easy to learn for its operation

UV2

Interaction with the site is clear and understandable

UV3

The site is easy to navigate

UV4

The site is easy to use

UV5

Attractive appearance

UV6

According to the type of site

UV7

Creating the impression of competence

UV8

Providing a positive experience

Information Quality

(Quality of Information)

IV1

Providing accurate information

IV2

Provide trustworthy information

IV3

Provide timely information

IV4

Provide relevant information

IV5

Easy-to-understand information

IV6

Information at the right level of detail

IV7

Information in the appropriate format

Service Interaction

(Quality of Service)

SV1

Good reputation

SV2

Information feels safe

SV3

Gives a sense of personalization

SV4

Provides a sense of community

SV5

Communicate with the organization

SV6

Confident that the goods/services provided are in accordance with what is promised

SV7

Overall website view

Source: (Andry, Christianto, & Wilujeng, 2019)

Validity and Reality Test

The validity test uses the corrected item total correlation technique  where if a rtable > calculation is found, the tool is valid with the following formula:

Information:

RXY    : the correlation coefficient between the item score (X) and the item score (Y)

N         : the number of respondents

Σx        : the sum of the variable scores (X)

Σy        : the sum of the variable scores (Y)

σx2      : sum of the squares of the variable score (X)

Σy2      : sum of the squares of the variable score (Y)

Σxy      : the sum of the multiplication of the item's score by the item score (X) and the variable score (Y)

The reality test will be carried out using Cronbach's Alpha technique. The alpha coefficient (α) value range should exceed 0.7 for the answer result. The higher the alpha value of Cronbach found, the higher the level of reliability or reliability of the research carried out.

Data Collection and Data Analysis

The primary data in this study was obtained from a Google Form questionnaire distributed to a minimum of 110 respondents determined based on a formula Hair (2018), where the number of indicators can be multiplied by 5 to 10. The criteria of the respondents in this study were:

1)   Unpad students and alumni who have been and/or are still actively using Learning Management System Regular LiVE Unpad in the learning process with the limit of the class of 2019 to 2022 (calculated from the year of entry)

2)   Unpad lecturers who use Learning Management System Regular LiVE Unpad as one of the media in supporting the learning process.

The analysis conducted after processing the data includes a gap analysis to assess whether the quality level of a website is considered good or bad by examining the gap between the quality that is performance (actual conditions) and the expected quality (ideal conditions) (L. A. Utami, Gani, & Suparni, 2020). A good quality level will be a positive value with a Qi (gap) ≥ 0.

In addition, Importance Performance Analysis (IPA) is also used to determine the level of conformity that determines setting priorities in overcoming factors that affect website user satisfaction. The science graph is divided into four analysis quadrants with the X axis representing the performance indicator and   (L. A. Utami et al., 2020)  the Y axis representing the importance indicator.

Hypothesis Test

The dimensions of usability, information quality, and service interaction play a big role in influencing user satisfaction according to, there is a considerable gap in the expectations (importance) and performance of the actual website which makes users feel less satisfied. Therefore, the research paradigm and hypothesis can be summarized as follows: Islamiah et al. (2022) I. S. Utami et al. (2021) Christianto et al. (2020).

 

 

 

 

 

 

 

 

 

 

 

 

Picture 1. Research Paradigm

 

1)   H1: There is a difference between the importance and user performance from the usability dimension of the Regular Learning Management System LiVE Unpad website which affects website user satisfaction.

2)   H2: There is a difference between the importance and user performance of the information quality dimension of the LiVE Unpad Regular Learning Management System website which affects website user satisfaction

3)   H3: There is a difference between the importance and user performance of the service interaction dimension of the LiVE Unpad Regular Learning Management System website which affects website user satisfaction.

 

RESULTS and DISCUSSION

Validity and Reality Test

The validity test was carried out by distributing questionnaires to 30 respondents who met the criteria. Through data processing using SPSS, the results of the validity test of the research instrument were obtained as follows:

Table 2.

Validity Test Results

Dimension

Indicator Code

rcalculate

Description

Usability

(Usability)

UV1

0,425

VALID

UV2

0,520

VALID

UV3

0,602

VALID

UV4

0,505

VALID

UV5

0,524

VALID

UV6

0,595

VALID

UV7

0,735

VALID

UV8

0,750

VALID

Information Quality

(Quality of Information)

 

IV1

 

0,610

 

VALID

IV2

0,371

VALID

IV3

0,595

VALID

IV4

0,520

VALID

IV5

0,791

VALID

IV6

0,744

VALID

IV7

0,710

VALID

Service Interaction

(Quality of Service)

SV1

0,740

VALID

SV2

0,547

VALID

SV3

0,688

VALID

SV4

0,544

VALID

SV5

0,484

VALID

SV6

0,751

VALID

SV7

0,820

VALID

 

 

Through the table above, it can be seen that all indicators in the research instrument are declared valid. This can be found out after comparing the calculation with the table. The table obtained was 0.3610 (with df = 28 and obtained from the formula for calculating the table r) with a significance level of 5% for the two-way test. The condition under which the calculation > the table, states that the instrument is valid for use.

Next, through data processing through SPSS, the results of the feasibility test for each dimension are obtained as follows:

Table 3.

Reality Test Results

Dimension

Cronbarch's Alpha

Description

Usability

0,856

VALID

Information Quality

0,826

VALID

Service Interaction

0,846

VALID

 

From the table above, it can be concluded that the value of the alpha coefficient for each dimension is greater than 0.7 so the research instrument can be said to be reliable.

Gap Analysis

Gap Analysis is carried out to find out how the quality of the website is through user assessments regarding the perceived quality of performance or actual conditions (performance) with the gap from the expected quality or ideal condition (importance). The following are the results of the gap analysis from the LiVE Unpad Regular LMS website obtained through SPSS:

 

Table 4.

Results of Dimensional Gap Analysis Usability

Indicator Code

Average Performance

Average Importance

Fold (Qi)

UV1

2,99

3,47

-0,48

UV2

2,80

3,47

-0,67

UV3

2,75

3,53

-0,77

UV4

2,93

3,55

-0,63

UV5

2,47

2,99

-0,52

UV6

2,95

2,99

-0,05

UV7

2,79

2,99

-0,20

UV8

2,70

3,05

-0,35

Average

2,80

3,26

-0,46

 

Through Table 4, it is known that all gap values (Qi) have negative values. This value indicates that user expectations have not been met when compared to the actual performance of each indicator in the usability dimension. The largest Qi is found in the UV3 indicator with a value of -0.77 which indicates the high expectations of users in the indicator that have not been met. From Table 4, it can also be seen that the average value of importance is greater than performance, which indicates that users are not satisfied with all indicators in this dimension.

 

Table 5.

Results of Dimensional Gap Analysis Information Quality

Indicator Code

Average Performance

Average Importance

Fold (Qi)

IV1

2,90

2,96

-0,06

IV2

3,21

2,91

0,30

IV3

2,16

3,46

-1,30

IV4

2,55

3,04

-0,48

IV5

2,91

3,01

-0,10

IV6

2,93

2,95

-0,03

IV7

2,94

2,89

0,05

Average

2,80

3,03

-0,23

 

Judging from Table 5, it can be seen that there are two indicators, namely IV2 and IV7 which have positive Qi and are considered to be able to meet user expectations. The largest gap was found in the IV3 indicator with a value of -1.30 which indicates the large gap between user expectations and the performance of the indicator on the website. Through Table 5, it is known that the average value of importance is greater than the performance which indicates that the user is not satisfied with the performance of this dimension.

Table 6.

Results of Dimensional Gap Analysis Service Interaction Qualit

Indicator Code

Average Performance

Average Importance

Fold (Qi)

SV1

2,88

2,99

-0,11

SV2

2,92

3,53

-0,61

SV3

2,47

3,03

-0,55

SV4

2,19

3,55

-1,35

SV5

2,41

3,04

-0,63

SV6

2,62

2,57

0,05

SV7

2,73

3,01

-0,28

Average

2,60

3,10

-0,50

 

It can be seen from Table 6, that there is one indicator that has positive Qi and is considered to be able to meet user expectations, namely SV6. The performance of the SV6 indicator is considered to be able to meet user expectations. The largest Qi is found in the SV4 indicator with a value of -1.35 which indicates the high expectations of users on the indicator that have not been met. From table 6, it can be seen that the average value of importance is greater than the performance which indicates that the user is not satisfied with all indicators in this dimension.

Through tables 4, 5, and 6, the average gap value of all indicators is -0.40. A negative average score indicates that the LiVE Unpad Regular LMS site has not met user expectations. In addition, it is known that the comparison of importance values is greater than performance in all dimensions. This indicates that users are not satisfied with the dimensions  of usability, information quality, and service interaction quality reviewed from the comparison of the two values.

Importance Performance Analysis (IPA)

 

 

 

 

 

 

 

 

 

 

 

 

 

Picture 2. Result Importance Performance Analysis

 

1)   Quadrant I (Concentrate Here)

Quadrant I describes indicators that have high importance with low performance. In Figure 2, there are two indicators that belong to Quadrant I, namely (IV3) provides timely information, and (SV4) provides a sense of togetherness. Both indicators need to be prioritized to improve because of the high expectations of users regarding these two things

2)   Quadrant II (Keep Up the Good Work)

Quadrant II has indicators of high importance and performance. In Figure 2, there are five indicators that need to maintain their performance because user expectations can be met by the indicator's performance. Some of the indicators in this quadrant include (UV3) the site is easy to navigate, (UV2) interactions with the site are clear and understandable, (UV4) the site is easy to use, (SV2) the information feels secure, and (UV1) it is easy to learn for its operation.

3)   Quadrant III (Low Quality)

Quadrant III contains indicators that need to be improved but are not a priority because they have low importance and performance. Referring to Figure 2, this quadrant has seven indicators, namely (SV5) communicating with the organization, (SV3) giving a sense of personalization, (UV5) attractive appearance, (IV4) providing relevant information, (SV1) good reputation, (SV6) confident that the goods/services provided are in accordance with what is promised, and (UV8) providing a positive experience.

4)   Quadrant IV (Possible Overkill)

Quadrant IV describes indicators that have low importance with high performance. In Figure 2, there are eight indicators whose development focus can be reduced so that resources can be allocated in the development of indicators in Quadrant I. This is because the indicators in Quadrant IV are classified as having stable performance because their values are above average. Indicators included in this quadrant include (SV7) the overall appearance of the website, (UV7) giving the impression of competence, (IV5) information that is easy to understand, (IV1) providing accurate information, (IV6) information at the right level of detail, (UV6) appropriate to the type of site, (IV7) information in an appropriate format, and (IV2) providing trustworthy information.

Hypothesis Test

1)   Multiple Linear Regression Test

The multiple linear regression test serves to measure the correlation of independent variables (x) with dependent variables (y). The following are the test results obtained:

 

Table 7.

Multiple Linear Regression Test Results

Model Summary

R

R Square

Adjusted R Square

Std. Error of the Estimate

.934a

0,873

0,869

0,17531

 

Based on Table 7, the Adjusted R Square value is 0.869, so it can be concluded that the quality of usability (X1), information quality (X2), and service quality (X3) have a simultaneous influence of 86.9% on website user satisfaction (Y).

2)   Partial Effect Test (t-Test)

The partial influence test or t-test was carried out to test the significant difference between the dependent variable (X) and the independent variable (Y). The following are the results of the t-test that has been carried out:

Table 8.

Test Results t

Coefficients

 

Unstandardized Coefficients

Standardized Coefficients

t

Sig.

B

Std. Error

Beta

(Constant)

-3,021

0,104

-29,011

0,000

Usability Quality

0,430

0,044

0,451

9,762

0,000

Quality of Information

0,233

0,039

0,268

5,933

0,000

Quality of Service

0,295

0,042

0,369

7,089

0,000

 

The results in Table 8 show the tcal value on the Usability Quality coefficient of 9.762. The ttable value obtained from the degree of freedom (df) 106 and α = 5% is 1.982579, so if compared, then the table> calculated. In addition, it can be known that the value of sig. by 0.000. At a significance level of 5%, the value of sig. <0.05, so H01 was accepted. Thus, it can be concluded that there is a difference between the importance and user performance of the usability dimension of the Unpad Regular Learning Management System LiVE website which affects website user satisfaction.

The Information Quality Coefficient (seen from Table 8) has a local value greater than the ttable with a ratio of 5.933 > 1.982579 and a significance value (sig.) of 0.000. At a significance level of 5%, the value is less than 0.05. Through the above information, it can be concluded that the null hypothesis (H02) is accepted, so that there is a difference between the importance and user performance of the information quality dimension of the LiVE Unpad Regular Learning Management System website which affects website user satisfaction.

The local value for the Service Quality coefficient shows a result of 7.089 with a ttable of 1.982579, so if compared, the local> ttable. The significance value obtained by the coefficient is 0.000. At a significance level of 5%, the value is less than 0.05, so the null hypothesis (H03) is accepted. It can be concluded that there is a difference between the importance and user performance from the service interaction dimension of the LiVE Unpad Regular Learning Management System website which affects website user satisfaction.

 

CONCLUSION

The results of the gap analysis obtained on the website have a negative value with the average of all indicators being -0.40. This value indicates that the website still does not meet user expectations. The comparison of the average value of greater importance than performance also states that users still feel dissatisfied. These results are supported by further calculations in the IPA method. The study found that several indicators need to be improved so that users can feel a strong presence from other users when using the website and can communicate with more than one user at the same time, such as by creating a live chat feature or comment section. In addition, users also feel the need for an increase in information distribution such as with the addition of notification features. Through the t-test, it can also be seen that the difference between the importance and user performance of the dimensions of usability, information quality, and service interaction quality also has a significant effect on user satisfaction of the  live Unpad Regular Learning Management System website.

 

REFERENCES

Andriansyah, F., Suryani, N., & Putri, S. A. (2018). Analisa Kepuasan Pengguna Terhadap Kualitas Aplikasi Ticket Monitoring PT. Infrastruktur Telekomunikasi Dengan Metode Webqual. JITK (Jurnal Ilmu Pengetahuan Dan Teknologi Komputer), 4(1), 111–118.

Andry, J. F., Christianto, K., & Wilujeng, F. R. (2019). Using Webqual 4.0 and Importance Performance Analysis to Evaluate E-Commerce Website. Journal of Information Systems Engineering and Business Intelligence, 5(1), 23. Retrieved from https://doi.org/10.20473/jisebi.5.1.23-31

Cerelia, J. J., Sitepu, A. A., & Toharudin, T. (2021). Learning loss akibat pembelajaran jarak jauh selama pandemi Covid-19 di Indonesia. In E-Prosiding Seminar Nasional Statistika| Departemen Statistika FMIPA Universitas Padjadjaran (Vol. 10, p. 27).

Grimmelikhuijsen, S., & Porumbescu, G. A. (2017). Reconsidering the expectancy disconfirmation model. Three experimental replications. Public Management Review, 19(9), 1272–1292. Retrieved from https://doi.org/10.1080/14719037.2017.1282000

Islamiah, F., Rusmiati, R., & Wijaya, R. (2022). Penilaian Kepuasan Pengguna Website Sistem Informasi Akademik Menggunakan Metode Website Quality. METIK JURNAL, 6(2), 133–139. Retrieved from https://doi.org/10.47002/metik.v6i2.381

Kevin, C., Deny, D., Charles, M., & Daniel, F. (2020). Detikcom website Analysis with Webqual 4. 0 and Importance-Performance Analysis method. International Journal of Open Information Technologies, 8(5), 31–36.

Kruse, O., Rapp, C., Anson, C. M., Benetos, K., Cotos, E., Devitt, A., & Shibani, A. (Eds.). (2023). Digital Writing Technologies in Higher Education. Cham: Springer International Publishing. Retrieved from https://doi.org/10.1007/978-3-031-36033-6

Legowo, D., Ismiyati, I., & Ungu, R. B. M. (2024). Measuring The Website Quality of Digital Archives Using WebQual 4.0 Model Approach. Journal of Curriculum Indonesia, 7(1), 23–32.

McQuitty, S. (2018). The purposes of multivariate data analysis methods: an applied commentary. Journal of African Business, 19(1), 124–142.

Pratama, I. P. A. E. (2021). Infrastructure as Code (IaC) Menggunakan OpenStack untuk Kemudahan Pengoperasian Jaringan Cloud Computing (Studi Kasus: Smart City di Provinsi Bali)(Infrastructure as Code (IaC) Using OpenStack for Ease of Operation of A Cloud Computing Network (A Case Study of Smart City in Bali Province). JURNAL IPTEKKOM Jurnal Ilmu Pengetahuan & Teknologi Informasi, 23(1), 93–106.

Sit, S. M., & Brudzinski, M. R. (2017). Creation and assessment of an active e-learning introductory geology course. Journal of Science Education and Technology, 26, 629–645.

Sole, F. B., & Anggraeni, D. M. (2018). Inovasi pembelajaran elektronik dan tantangan guru abad 21. Jurnal Penelitian Dan Pengkajian Ilmu Pendidikan: E-Saintika, 2(1), 10–18.

Utami, I. S., & Setiadi, H. (2021). Analysis the effect of website quality on user satisfaction with the WebQual 4.0 method and importance-performance analysis (IPA)(case study: SPMB Sebelas Maret University’s Website). In Journal of Physics: Conference Series (Vol. 1842, p. 012003). IOP Publishing.

Utami, L. A., Gani, A., & Suparni, S. (2020). Penerapan Metode Webqual 4.0 dan IPA Dalam Mengukur Kualitas Website VISLOG PT. Citra Surya Indonesi. Komputika : Jurnal Sistem Komputer, 9(1), 25–34. Retrieved from https://doi.org/10.34010/komputika.v9i1.2849

Van Mensvoort, K. M. (2013). Pyramid of technology: how technology becomes nature in seven steps. Technische Universiteit Eindhoven.

 

 

© 2024 by the authors. It was submitted for possible open-access publication under the terms and conditions of the Creative Commons Attribution (CC BY SA) license (https://creativecommons.org/licenses/by-sa/4.0/).