Diagnosis of Students Online Learning Portfolios

Transcription

1 Diagnosis of Students Online Learning Portfolios Chien-Ming Chen 1, Chao-Yi Li 2, Te-Yi Chan 3, Bin-Shyan Jong 4, and Tsong-Wuu Lin 5 Abstract - Online learning is different from the instruction provided in traditional classes in that the teachers and students do not have actual contact. Thus, the teachers can have little control over the students learning situations. However, the teachers can observe the students learning activities by analyzing their online learning portfolios. These portfolios usually comprise quantified serial of information figures. This study used data mining techniques such as a K-means cluster analysis algorithm in order to manage a cluster analysis of the students various portfolios and to support general statistical methods. The cluster analysis effectively categorized students with similar portfolios into the same cluster; additionally, different clusters revealed different characteristics. In addition, this study also used a t-test and a test of independence of statistics in order to analyze the correlation and degree of correlation between the students performance in online learning and their learning effects. Further, this study used statistics and a data mining technique to obtain a mining analysis of the portfolios, namely, the students assignment scores, exam scores, and online learning records; it also applied a proposed interface to enable the teachers to observe the change in the students learning situation during the learning process in order to analyze its connection with learning effects. Index Terms Data mining, E-Learning, Learning portfolio INTRODUCTION There is a trend toward online learning due to the popularity of PCs and the Internet. Using these tools, students are able to engage in online learning using mobile PCs and Internet connections, without being constrained by time and space. Additionally, by using the i-learning platform [1] and other related communication techniques, students can avail of greater opportunities for interaction and discussion with teachers and classmates during online learning activities. The students can also adapt their own learning styles according to their learning situations in order to accelerate the learning process or repeat the portions they are unfamiliar with. Unlike the instruction in traditional classes, Internet-based instruction operates through the transmission of online materials to fulfill the goal of lecturing. There is no real contact between the teachers and students or among the students. Thus, it is not simultaneous teaching, and it is difficult for the teachers to familiarize themselves with the students learning situations. Since the students have more freedom and experience less stress on the e-learning platform, they tend to become inactive after a certain period of time. If the teachers do not become conscious take cognizance of this fact immediately, the students will be presented with a tremendous opportunity to learn ineffectively. Thus, in nonsimultaneous online learning, we should analyze the students learning portfolios in order that the teachers can immediately familiarize themselves with the students learning situations and behavior and try to guide the students with poor learning effects. Learning portfolios generally refer to any records created in the learning process, such as notes, assignments, test papers, and reports. Through computer techniques, the students behavior, such as the time taken to read learning materials, the duration spent online and logon frequency, assignments, and records of online conversations with others on the learning platform can be recorded in a database. Thus, the learning portfolios of students participating in online learning include detailed raw data. If we could analyze the correlation between the students learning behavior and learning achievements, we would be able to enable the teachers to control the students overall and personal learning situations to a greater extent. The teachers could also provide immediate guidance in order to promote the students learning effects. This study applied a learning log explorer system in order to analyze the students learning behaviors, such as performances on assignments and quizzes as well as online learning records, in order to provide the teachers with a means to observe the students during the learning process. The teachers are able to adjust their pedagogy or online learning activities according to the results of the analysis. When the students demonstrate learning behavior that might lead to poor learning effects, teachers are able to provide assistance or remedies in order to fulfill their purpose of promoting learning effects. LEARNING PORTFOLIOS Learning portfolios provide the students with a specific method to evaluate their own learning situations [2], [3], and [4]. They include all the records of the students activities during the learning process, such as their interaction with 1 Chien-Ming Chen, Ph.D. Student, Chung Yuan Christian University, calculusxp@cg.ice.cycu.edu.tw 2 Chao-Yi Li, Graduate Student, Chung Yuan Christian University, hyder13@cg.ice.cycu.edu.tw 3 Te-Yi Chan, Candidate for doctor's degree, Chung Yuan Christian University, milk@cg.ice.cycu.edu.tw 4 Bin-Shyan Jong, Professor, Chung Yuan Christian University, bsjong@ice.cycu.edu.tw 5 Tsong-Wuu Lin, Professor, Soochow University, twlin@cis.scu.edu.tw T3D-17

2 others, assignments, test papers, personal work collections, their discussion content, and online learning records. The contents of the portfolio can be extended to the time the students spent in answering the questions, the order in which they answered the questions, etc. By using such data, the students and teachers are provided an understanding of both the former learning effects and the latter lecturing effects. In various learning portfolios, the teachers might intend to determine the cause-and-effect relations in order to diagnose the students learning effects or other unknown items. DATA MINING TECHNIQUES Data mining [5] refers to the search for valuable hidden information and their correlation from a large amount of data. The related techniques include logic, statistics, and artificial intelligence. It is characterized by its thoroughness in analyzing raw data and in determining the information involved and their relations. Based on the different questions that are set, data mining techniques establish related models as the references for decision-making. Cluster analysis is one of the representative techniques of data mining. The purpose of cluster analysis is to categorize similar items such that the items in the same category were the same, similar, or homogeneous for some variables. There are actually significant differences or heterogeneity among the categories [5], [6]. Cluster analysis technique is usually applied to the analysis of a large amount of data. It can determine similar clusters according to certain variables and separate them from the clusters that are not similar. Cluster analysis aims to cluster non-categorized data; it is a type of unsupervised categorization that can perform clustering according to the degree of correlation among the data. Different similarity judgments result in different cluster results. However, good clustering results depend on the needs of the application, and there is no absolute evaluation standard to evaluate the clustering results. Through a cluster analysis, the analysts can determine clusters with internal homogeneity or discover unknown properties. They could also use cluster analysis and outlier detection. One of the important data mining techniques is association rules [5]. Its purpose is to determine the correlation among the items from a large amount of data, such as the hidden connection between the items in voluminous transaction records. For instance, when we discover that the appearance of an item (a cluster) will trigger another item (cluster), we determine that a relationship exists between the two items. The most typical example of association rules in textbooks is milk => bread. This means that when the customers purchase milk, they simultaneously buy bread. The t-test [7] is a commonly used statistical technique. It is a significance level test of two averages, and its purpose is to determine whether two populations reach a level of significance level with respect to certain items. When attempting to determine the significant difference between two populations regarding certain properties, such as the difference in the average IQ between boys and girls and whether the scores of an experimental group are higher than that of a control group, we could use a t-test to examine whether or not the hypotheses reach an acceptable significance level. Another important data mining technique is the test of independence [8]. Its purpose is to determine whether or not two design variables related to participants from a population are independent. Additionally, if they are not independent, the test determines the attribute and level of the correlation between the two variables [8]. The null hypothesis H 0 states that two variables are independent; the alternative hypothesis H 1 states that H 0 is rejected. If the x 2 (chi-square) value calculated is greater than the x 2 table value, H0 will be rejected. In other words, the two variables are not independent. The formula for calculating x 2 is (1); f 0 is the observed frequency, and f e is the expected value. From this formula, we understand that the greater the difference between the frequency of the actual survey and that of the theoretical inference, the larger will be the x 2 value. ( f f ) 2 2 o e χ (1) T3D-18 = I. Introduction f e LEARNING LOG EXPLORER SYSTEM This system provides the teachers with an effective and convenient means with which to analyze the students online learning behavior without having to spend time understanding and analyzing the structure of the database [9]. The function of the serial time analysis that can capture the students learning situations at different learning phases is strengthened. Various statistical analyses and data mining algorithms are also introduced to enable teachers to extract additional useful information. The activeness of the students and learning effects will be increased by the use of a mechanism whereby warning messages are sent to students. The learning platform hooked by learning log explorer is i-learning [1], a popular commercial learning platform in Taiwan. It provides students with non-simultaneous online distance learning services during the academic semester and summer vacation. Teachers can use this platform to upload multimedia learning materials designed by them, arrange online tests, establish discussions regarding the curriculum, answer the students queries, etc. The students can thus study the online learning materials, join course discussions, interact with their classmates or teachers, and take online tests through the internet. The framework of analyzing students online learning portfolios is such that the students experience online learning through i-learning. Students online activities are recorded in the i-learning database. The learning log explorer analyzes and extracts the students learning portfolios from the log database. The learning log explorer saves the portfolios in its portfolio database through reorganization and filtration. The teachers can further input data related to the courses, such as the scores of the students written papers or projects, into the portfolio database. Through the integrated analysis of this

3 system, the teachers are able to evaluate students learning behaviors as well as observe the correlation between their online behavior and learning achievements. The analytical results of this system can also be transferred to the students and be used to encourage them to learn actively; this may ultimately increase their learning achievement. II. Operation of portfolio analysis For details regarding the manner in which the learning log explorer was implemented and enabled on the general learning platform, please refer to [9]. This section will introduce the interface and the system operation. Overall, the portfolio analyzing operations comprise four parts: the curriculum setup, including the curriculum schedule and the student roster; the extraction of the students online portfolios from the learning platform; the different clustering methods provided by the system to enable the teachers to investigate the students learning situations; and the correlation analysis between learning behavior and achievement, including a t-test, a test of independence, and association rules. III. Curriculum setup The teachers should first choose an observation class and input the curriculum schedule and the student roster into the system. After choosing the classes, the system will divide the course durations into learning period units that each last one week unit; the semester generally lasts 18 weeks in Taiwan. IV. Extracting online learning portfolios We must design and define the students learning portfolios according to the database scheme of the online learning platform. The following describes the defined students learning portfolios used in this study. Logon times are the number of times that the students successfully enter the course in a certain learning period. Logon days are the number of days that the student enters the learning platform in order to learn. Total study days refers to the total study duration divided by 24 days of complete study. This portfolio is used to avoid a situation in which the students frequently log onto system in a short time and as a result, the logon times and logon days calculated may not represent the students actual learning behavior. The number of articles posted is a count of the number of times the students posted articles on the curriculum discussion forum during the learning activities. The count of the posted articles is divided into the course forum and issue forum. In the course forum, the teachers and the students can interact with each other, post questions on the forum, and answer these questions and discuss them in the forum. In the issue forum, the teachers determine subjects related to the course in order to encourage the students to attempt to answer, discuss, and respond to questions about these subjects. The clicking times are the number of times the students click on online learning materials during certain learning periods. The duration of study learning materials refers to the total time spent by the students while studying online learning materials during a certain learning period. This portfolio can show the total time the student spent reading the learning materials. Finally, we consider the rationality of the portfolios and eliminate the situations with browsing times that are extremely short since these may not constitute an effective study. V. Investigating the students learning situations After extracting and integrating the students various learning portfolios, the teachers should not be concerned about the processes of these portfolios; instead, they should consider the statistics revealed by the portfolios. Additionally, they should consider the following question: Do students log on frequently, participate in the discussions often, and study the learning materials every day? This question is used to investigate the students activeness in online learning as well as their participation in discussion. In this study, we used various clustering methods to analyze and compare the portfolios and to allocate the students into different clusters. For example, consider the logon times ; when we divided them into two clusters by using the average as a threshold value, the students with logon times that are less than the average are categorized in the cluster of less logon times. On the contrary, the students in the opposite situation are categorized into the cluster of more logon times. The example mentioned above is extremely simple. However, is this kind of clustering method appropriate? Thus, this system provided various clustering methods in order that the teachers can investigate the students learning situations more precisely. The following paragraph introduces the clustering methods adopted by this study. Manual clustering: This method is completely manual and the teachers must assign each student into the appropriate clusters that they define. Semi-automatic clustering: This method allows the teachers to manually set up the number of the clusters and threshold value; then, the system automatically allocates the students into clusters. The interface provides related information about the clustering targets, such as the distribution of the students overall behavior, average, standard deviation, and median value. Automatic clustering: This method uses a cluster analysis algorithm to cluster the target items. This study adopted the K- means [5], and [10], FarthestFirst [11], and EM (Expectation Maximization) [12] algorithms. The clustering results are not the same; thus, the teachers have more references with different types of interpretations of the students online learning situation. Automatic clustering can analyze multiple dimensions, for example, it can simultaneously conduct a cluster analysis of the logon times, the number of the articles posted, and the duration of study learning materials. The teachers can thus investigate the students overall online learning situation. VI. Correlation analysis The focus of this study is the degree of correlation among the students various portfolios. For example, do online activities T3D-19

4 at the beginning of the semester and midterm exam results affect the results of the final term exam? Through a correlation analysis, teachers are able to diagnose the students learning situations at the earliest possible instance. If the students are performing well, they should maintain their performance. If there are negative behaviors found, the teachers can correct or remedy the students learning situations as early as possible to increase the learning effects. Further, by analyzing the recorded data from past semesters, teachers can further ascertain the correlations among the portfolios to analyze and increase the effectiveness of the students learning processes. The techniques adopted in this study are introduced in the following paragraphs. A t-test [7] is one of the commonly used statistical techniques. It helps the teachers analyze the significant difference in the attributes of the clusters after the students learning portfolios are clustered and also judges the correlation among the clusters. A test of independence [8] can discover the correlation between two factors. When the test results show a certain degree of correlation, it means that the knowledge of a certain item helps in the prediction of another item. Taking quizzes and final term exam results as an example, if the test results show a correlation, the knowledge of the students quiz results can help predict their final term exam results. If the test results reveal that the two items are independent, it means there is no correlation between the two items. The association rule [5] aims to determine the correlation among the items in a large amount of data and discover the hidden connections among them. The appearance of one item will lead to the emergence of another, in accordance with so-called association rules. FIGURE 1 FLOW OF LEARNING PORTFOLIO DIAGNOSIS After determining the correlation among the students learning portfolios, the teachers can send warning messages (to either encourage or warn) to the students, listing the specific conditions so that students can understand their own learning situations. Through this showing of the learning situations to the students and allowing them to discover their insufficiencies in online learning activities, the students can actively promote their learning situations by increasing their activeness and strengthening participation in online learning. The flow of learning portfolios diagnosis is shown in figure 1. EVALUATION The 162 investigative subjects used in this study are junior students of the department of computer engineering at Chung Yuan Christian University. The evaluated curriculum is system programming in the academic year of System programming mainly trains the students in system software design and development; it describes the relation between computer architecture and system software. I. Questions and Hypotheses This evaluation aimed to explore the correlation degree between the students portfolios and learning achievement. Thus, the hypotheses we proposed are as follows: There exists a positive correlation between online learning frequency and learning achievement. There exists a positive correlation between online discussion participation and learning achievement. There exists a positive correlation between the duration of study learning materials and learning achievement. Thus, this roughly divided the entire academic semester into two main stages: from the beginning of the semester until the midterm exam and from the midterm exam until the final term exam. In order to analyze the students portfolios in detail, the study acquired the students learning portfolios of five learning periods: from the beginning of the semester until the midterm exam, two weeks before the midterm exam, from the midterm exam until final term exam, two weeks before the final term exam, and the entire semester. With regard to the clustering criteria of online portfolios, besides the interactive discussion, issue discussion, and the number of articles posted, a K-means cluster analysis algorithm was also adopted to cluster other learning portfolios. This study integrated the logon times and logon day into online learning frequency learning behavior: when the logon times and logon days were less than the threshold, they were regarded as low online learning frequency and the rest were regarded as high online learning frequency. Click times and duration of study learning materials were integrated as learning materials study indicator learning behavior in that when the click times and the duration of study learning materials were less than the threshold, they were allocated as not having enough diligence and the rest were considered as having enough diligence. The midterm exam results and final term exam results were also divided into high score and low score clusters using K-means. Thus, this study first evaluated the students online learning behaviors through clustering techniques. Subsequently, this study confirmed whether there was a significant difference in the students with different learning behaviors in terms of learning achievement as well as whether there was a significant difference in learning behavior between the high and low score clusters. T3D-20

5 TABLE 1 T-TEST RESULTS OF STUDENTS ONLINE LEARNING FREQUENCY Status of students online learning frequency High Frequency Low Frequency Midterm Exam Final Term Exam From the beginning of the semester until the midterm exam ** ** Two weeks before the midterm exam ** ** From the midterm exam until final term exam ** ** Two weeks before the final term exam ** ** (**P VALUE<0.05) The entire semester ** ** TABLE 2 T-TEST RESULTS OF STUDY INDICATOR Status of students learning materials study indicator High Value Low Value Midterm Exam Final Term Exam From the beginning of the semester until the midterm exam Two weeks before the midterm exam From the midterm exam until final term exam ** ** Two weeks before the final term exam ** (**P VALUE<0.05) The entire semester ** ** TABLE 3 T-TEST RESULTS OF STUDENTS PARTICIPATION IN THE DISCUSSION Status of students participation in the discussion Positive Negative Midterm Exam Final Term Exam From the beginning of the semester until the midterm exam ** ** Two weeks before the midterm exam ** ** From the midterm exam until final term exam ** Two weeks before the final term exam ** (**P VALUE<0.05) The entire semester ** II. Evaluation Results We used the techniques mentioned above to cluster the online performance. Table 1 shows the clustering results of online learning frequency in different stages. By using a t-test to evaluate the midterm exam and the final term exam performance of the clusters with high and low online learning frequency, the results demonstrated that there was a positive correlation between the students online learning frequency and the midterm exam results. In other words, the midterm scores of the students with high online learning frequency were generally higher than the scores of those with low online T3D-21

6 learning frequency. According to the t-test result in table 1, we found that the cluster of the students online learning frequencies acquired from period two weeks before the exam revealed more significant differences (the p-value is less and reliability is more) in terms of the performance in the exam than the cluster of the students online learning frequency acquired from longer learning period (such as the whole semester). Besides, we also found that the clustering result of the students online learning frequency acquired from the beginning of the semester until the midterm exam revealed significant difference in terms of the results of the final term exam. Table 2 shows the t-test results of the learning materials study indicator of low and high clusters in different periods. From the table, we determine that the clustering results of the learning materials study indicator indicate that the students did not show significant differences before the midterm exam. The learning materials study indicator only shows significant differences between the period before final term exam and the whole semester. According to this result, we discover that the students portfolios of click times and the duration of study learning materials at the beginning of semester do not show correlation with the midterm and final term exam results. However, as the course progresses, the correlation between the students learning materials study indicator and the midterm or final term exam results would be significantly revealed. Table 3 refers to the t-test results of the exam results and students participation in the discussion during different periods. The results reveal that the participation in online discussion showed significant effects on the exam results. In other words, as compared with the students who did not participate in the online discussion, the ones who participated in posting or responding to the articles obtained better exam results. Thus, we find a positive correlation between the students online participation and exam scores. REFERENCES [1] LearnBank, Wisdom Master, last viewed date: 2007/3/18. [2] Paulson, F. L., Paulson, P. R., and Meyer, C. A., "What makes a portfolio a portfolio?", Educational Leadership, Vol.48, No.5, 1991, pp [3] Borko, H., Michalee, P., Timmons, M., and Siddle, J., "Student Teaching Portfolios: A Tool for Promoting Reflective Practice ", Journal of Teacher Education, Vol.48, No.5, 1997, pp [4] Burch, C. B., "Inside the Portfolio Experience: The Student Perspective ", English Education, Vol. 32, No.1, 1999, pp [5] Dunham, M. H., "Data Mining: Introductory and Advanced Topics ", Prentice Hall, NJ, [6] Clustering, A Tutorial on Clustering Algorithms, last viewed date: 2007/3/18. [7] The T-Test, Research Methods Knowledge Bases, last viewed date: 2007/3/18. [8] Introduction to the Chi Square Test of Independence, HyperStat Online Statistics Textbook, last viewed date: 2007/3/18. [9] Jong, B. S., Chan, T. Y., and Wu, Y. L., " Learning Log Explorer in E- learning Diagnosis ", under publishing queue of IEEE Transactions on Education, [10] MacQueen, J.B., "Some Methods for classification and Analysis of Multivariate Observations", Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, University of California Press, 1967, pp [11] Hochbaum, D. and Shmoys D., "A best possible heuristic for the k- center problem", Mathematics of Operations Research, Vol. 10, No.2, 1985, pp [12] Dempster, A. P., Laird, N. M., and Rubin, D. B., "Maximum Likelihood from Incomplete Data via the EM algorithm", Journal of the Royal Statistical Society, Series B, Vol. 39, No.1, 1977, pp CONCLUSION Most of the students learning portfolios on the learning platform are continuous values. Traditionally, statistical strategies are adopted to analyze this data. However it is difficult for teachers who do not have background of statistics to comprehend the results of such analytical strategies. In this study, a K-means clustering analysis algorithm was adopted in order to analyze students learning portfolios. The cluster analysis can effectively group students who demonstrate similar learning portfolios in one group. The correlation among students online learning portfolios and learning achievements are investigated. ACKNOWLEDGMENT The authors would like to thank the National Science Council of the Republic of China, Taiwan, for financially supporting this research under Contract No. NSC S T3D-22