Anti-Spam Filter Based on Naïve Bayes, SVM, and KNN model

Transcription

1 AI TERM PROJECT GROUP 14 1 Anti-Spam Filter Based on,, and model Yun-Nung Chen, Che-An Lu, Chao-Yu Huang Abstract spam filters are a well-known and powerful type of filters. We construct different filters using three types of classification, including,, and. We compare the pros and cons between these three types and use some approaches to improve them to get a better spam filter. Index Terms Spam filter,,, 1 INTRODUCTION M ass unsolicited electrontic mail, often known as spam, has recently increased enormously and has become a serious threat to not only the Internet but also to society. Over the past few years, many approaches have been provided to filter spam. Some of them often use Naïve Bayes method to implement it. 2 PROPOSED APPROACHES 2.1 Problem Definition Construct a filter which can prevent spam from getting into mailbox. Input: A mail message Output: Spam or Ham 2.2 Proposed Solution Figure.1 System Flowchart We use method of machine learning to train a model, and decide the input message belonges ham or spam. We use three methods to implement spam filter, including Yun-Nung Chen Author is in the National Taiwan Universty. swallow @gmail.com. Che-An Lu Author is in the National Taiwan Universty. [email protected]. Chau-Yu Huang Author is in the National Taiwan Universty. [email protected] 2009 AI TERMPROJECT,, and. We will compare these three methods first. After that we will modify each method independently to improve the accuracy and compare each method itself. Finally make a conclusion above all the improvement on these three methods. Following are our methods details. From Bayesian theorem of total probability, given the vector x = x,x,,x of a mail d, the probability that d belongs to category C is: P c=c X =x = where k spam,ham. We can decide that mail belongs to a model with a higher probability. This uses unigram language model to compute the probability of class. In order to get better feature (word with more information), we preprocess testing message to remove some noise word and reserve words which are more important. We remove words whose lenth is longer than 50, and we also delete words which appear for less 20 times. A word which appears in more documents represents that it carries less information for classification. We use SRI Language Model Toolkit: to generate unigram language model, and compute the Bayesian Probability according to the model. We can decide the classification of testing message. The support vector machines () is easy to use and powerful for data classification. When generating filter model, we create a vector for each data in tha training corpus and then will map these vectors into hyperplan. Then will learn which kinds of vector are close to which class. Here is a good approaching method in precisely finding best classification hyper-plans to maximize the margin so that we could classify a new mail into spam or ham. (1)

2 2 AI TERM PROJECT GROUP 14 Figure.2 Version1 First we will select top 1000 document frequency terms as we do in. Secondly we will create TF-IDF for each training data. Thirdly we will use libsvm (a tool from Chih-Jen Lin) to train a model. With each unclassified mail we will create a TF-IDF vector too, finally predict it with the svm model we trained before. The fomula of TF-IDF is described as below: log +1 (2) Where N is number of terms in whole corpus, f is frequency of term t, and df is data frequency of term t. If a term appears in many documents, it represents that this term doesn t have significance. So we use IDF to diminish the weight of such term inorder to select better features. Why we use data frequency to select the feature is because the following figure (a reference from course Web Mining ). Figure.3 We can see that the performance between all the feature selection methods except mutual information and term strength makes no much difference. For our coding convinence, we choose to implement data frequency. Version2 It is similar to version 1, we only turn the uppercase into lowercase (case-insensitive), so that for example free will be the same term with FREE or Free Version3 It is similar to version 2, the only difference is that when training svm model, we will set the parameters cost and gamma to the best condition. Version1 First we will create a binary vector which maintain the information of each feature (term) exist or not (1 or 0) for each training data. Secondly, for each unclassified mail we will create a binary vector too, then using cosine distance to find out the top K closest training data from the corpus then find out which class is the unclassified mail belongs to. cosine distance: Version2 It is similar to version 1, one big difference is that in version 2, we will not use binary vector as before, we will use TF-IDF respectively. Version3 It is similar to previous version, we only turn the uppercase into lowercase (case-insensitive), so that for example free will be see as the same term with FREE or Free. 3 CONTRIBUTIONS 3.1 Compare each method independently We compare three methods independently, and we also can observe the difference between sizes in 1200, 3000, 9000, and When we trains language model, computing the probabilities of words doesn t care the case (case-insensitive), and we also remove the word with too small probability for message preprocessing. When the size of traing data is smaller (1200), the result still has good performance. We can see that accuracy is %. But when training set becomes a little larger (3000), the result is not good as smaller one. We can see that the result is improved up slightly. Compared to size, the improvement of accuracy is relatively small, and we can just improve accuracy to 3% ( % -> ). We believe feature selection is important to Naïve Bayes, and we can use a better feature to improve result. But we must spend much time to testing data to see whether the result will be improved up, time and accuracy is tradeoff. When the training set is small (1200), the result of model is much poor than others. When we use case-insensitive to create tf-idf vector, the accuracy can improve up to 20% (60.854% -> %), which means that it is important to combine the information of uppercase and lowercase together to increase the concept for a specific term (ex: free, Free, FREE). The other reason is that if we see free and Free as the same term, then the data frequency of free will increase, so that we won t throw away such important feature (since we only select top 1000 ranked by data frequency). (3)

3 AI TERM PROJECT GROUP 14 3 After we find the best gamma and cost value for each corpus, the performance can also improve up to 20% ( % -> %). But the process of finding such parameters is very time consuming, so it will be a tradeoff. When using large training set, the performance won t improve accordingly. We think that it is because when the training set grows up, the noise will also increase, which means that there will be more ham similar to spam. The performance of is really out of our expectation. At begin, we think won t be better than. But after our experiment, seems work well on spam classification. The other interesting thing is that when using binary vector and TF-IDF vector, the performance makes little difference (notice that TF-IDF still a little better than binary vector). The improvement of using TF-IDF vector is not that significant as we expect. We think that it is because in spam classification, some important features in spam is quite different from ham, so the weight (TF- IDF) of such feature will not be so important as whether there exist such feature. And we also found that the case-sensitive and caseinsensitive makes little difference in, not like in. We think the main reason is that we didn t do feature selection in, instead, we keep all the features. So we won t throw out some important features (ex: Money) such as feature selection does. And this may also be the reason why will beat. 3.2 Compare all these three methods We think that it is why is better than that the feature we select when we implement Naïve Bayes is not good enough to train an excellent model. We think can beat is because that we throw some information when doing feature selection in. And the other reason might be that spam classification problem is a binary decide problem (spam / ham), so can easily close to one side, and we think if there exists more class, the performance of will be better than. When training corpus is small, the performance of is still well, not like, with only 60% accuracy. We think it is because is a machine learning method, we can t expect it learns well with a little training data. can still find the top K similar data in a small corpus. We also think that is good method to filt spam when traing set is small, and because the word appearing in ham isn t too many, we can compute probabilities of words to decide a category by using a training set with small size. 4 EXPERIMENTAL RESULTS 4.1 Corpus We use the corpus which is provided by trec06. There are messages (12910 ham, spam). We separate the whole set into training data and testing data. The testing data are 2910 ham and 4912 spam which are randomly select from the corpus. Remaining data (10000 ham, spam) are used to be training data. These two set (testing data and training data) is independent. In our experiments, we will create four different sizes of training data which are randomly select from the training corpus. The ratio of spam and ham in these four training datas are all 2:1, and the corresponding size of them are 1200, 3000, 9000, and (Following we will use 800:400, 2000:1000, 6000:3000, 14000:7000 to represent four different training sets) 4.2 Result of Evaluation Following are three different methods accuracy table with different sizes of training data. Accuracy 800: % 2000: % 6000: % 14000: % Table. 1 Version1 Version2 Version3 800: % % % 2000: % % % 6000: % % % 14000: % % % Table. 2 Version1 Version2 Version3 800: % % % 2000: % % % 6000: % % % 14000: % % % Table. 3 Fig.4 is the accuracy plot of, first version of, and first version of with different size of training data % 80.00% 75.00% 70.00% 65.00% 60.00% Figure.4 Accuracy of ver1 ver1 We use to be the baselin. We can see that in the first version of, all the accuracy are less than. And in the first version of, the performance is already better than Naïve bayes. Fig.5 is the accuracy plot after we improve the

4 4 AI TERM PROJECT GROUP 14 accuracy % 96.00% 94.00% 85.00% 80.00% 75.00% 70.00% ver1 ver2 ver % 88.00% ver3 ver % 60.00% Figure.5 Accuracy of We can see that there has a big improvement between version 1 and version 2, the difference between these two versions is that we use case-insensitive in version 2. In version 2 we will much emphasize some important features such as Free, and we won t throw too many information away due to feature selection as we mentioned before. The improvement between version 2 and version 3 is also significant. By well selecting the parameter gamma and cost, the performance can really improve a lot as we can see. Fig.6 is the accuracy plot after we improve the accuracy % 98.00% 97.00% 96.00% 94.00% 93.00% Figure.6 Accuracy of ver1 ver2 ver3 We can see that there is a little improvement from version 1 to version 2, the difference of them is that we use TF-IDF vector instead of binary vector in version 2. And when we modified version 2 from case-sensitive to case-insensitive, the difference between them is not as significant as (no more than 0.4%). We also mentioned this before, it is because we didn t do feature selection in. Fig.7 is the accuracy plot of, and the best version of and with different size of training data % Figure.7 Comparison of 3 methods As we can see, after we improve, two of the middle size of training set is better than. And the other two is much more close to Naïve bayes compare with the first version. After we improve, the result is again better than. 5 CONCLUSION After experimenting three different methods, we found that has higher accuracy than other two approaches. Because we think is more suitable for classification of less catergories than, accuracy of is higher. We think is a good method for spam filter, and the time costs little on training (about 1-2 seconds). Testing an input message requires much time using, but the result is good enough. The training time of is very fast, but it takes lots of time on testing. We think it is because we didn implement any indexing algorithm such as KD-tree, R-Tree or Quad-Tree when finding the nearest top K neighbors. In our future work, we can implement these indexing methods to improve the efficiency of. The training time of compared with and is much longer, espically when we want to find out the best gamma and cost parameters for the training process. But the testing time of is much faster than the other two methods. In future work, we can focus on different feature selection methods to improve the performance of and, and the results of them might become better than. 6 JOB RESPONSIBILITY Yun-nung Chen (B ) (Training and Testing), Report Writing. Che-an Lu (B ) (Training and Testing), (Training and Testing), Report Writing. Chau-yu Huang (B ) Message Preprocessing, Report Writing. ACKNOWLEDGMENT The report uses lots of toolkits, including MIME-tools,

5 AI TERM PROJECT GROUP 14 5 SRILM, and. So we want to thanks about it. REFERENCES [1] SRI Language Model Toolkit [2] CJ Lin s Home Page