Detecting unknown anomalous program behavior using API system calls

MD Rafiqul Islam, Md. Saiful Islam, Morshed U. Chowdhury

Research output: Book chapter/Published conference paperConference paperpeer-review

2 Citations (Scopus)


This paper presents the detection techniques of anomalous programs based on the analysis of their system call traces. We collect the API calls for the tested executable programs from Microsoft detour system and extract the features for our classification task using the previously established n-gram technique. We propose three different feature extraction approaches in this paper. These are frequency-based, time-based and a hybrid approach which actually combines the first two approaches. We use the well-known classifier algorithms in our experiments using WEKA interface to classify the malicious programs from the benign programs. Our empirical evidence demonstrates that the proposed feature extraction approaches can detect malicious programs over 88% which is quite promising for the contemporary similar research.
Original languageEnglish
Title of host publicationProceedings of the International Conference on Informatics Engineering and Information Science
Subtitle of host publicationICIEIS 2011
EditorsAzizah Abd Manaf, Shamsul Sahibuddin, Rabiah Ahmad, Salwani Modh Daud, Eyas El-Qawashmeh
Place of PublicationBerlin
Number of pages12
ISBN (Electronic)9783642254826
Publication statusPublished - 2011
EventInternational Conference on Informatics Engineering and Information Science: ICIEI 2011 - Universiti Teknologi Malaysia, Kuala Lumpur, Malaysia
Duration: 14 Nov 201116 Nov 2011

Publication series

NameCommunications in Computer and Information Science


ConferenceInternational Conference on Informatics Engineering and Information Science
CityKuala Lumpur


Dive into the research topics of 'Detecting unknown anomalous program behavior using API system calls'. Together they form a unique fingerprint.

Cite this