Abstract

Modern computer systems are built on a foundation of software components from a variety of vendors. While critical applications may undergo extensive testing and evaluation procedures, the heterogeneity of software sources threatens the integrity of the execution environment for these trusted programs. For instance, if an attacker can combine an application exploit with privilege escalation vulnerability, the Operating System (OS) can become corrupted. The importance of ensuring application integrity has been studied in prior study; proposed solutions immediately terminate the application once corruption is detected. Mandatory Access Control (MAC) in a commercial operating system to tackle malware problem is a grand challenge but also a promising approach. The firmest barriers to apply MAC to defeat malware programs are the incompatible and unusable problems in existing MAC systems. The major aim of our study is to address these issues and to analyse 2,600 malware samples and component one by one and two types of MAC enforced operating systems and then design a novel Efficient Malware Detection and Tracer design (EMDT) using Hidden Markov model, which incorporates intrusion detection and tracing in a commercial operating system which leverages efficient coding and authentication schemes with our proposed approach conceptually consists of three actions: detecting, tracing and restricting suspected intruders .The novelty of the proposed study is that it leverages light-weight intrusion detection and tracing techniques to automate security label configuration that is widely acknowledged as a tough issue when applying a MAC system in practice. The other is that, rather than restricting information flow as a traditional MAC does, it traces intruders and restricts only their critical malware behaviours, where intruders represent processes and executables that are potential agents of a remote attacker. Our prototyping and experiments on Windows operating system show that Tracer can effectively defeat all malware samples tested via blocking malware behaviours while not causing a significant compatibility problem.

Keywords:

Detection, intrusion, malware, tracing, vulnerability,

References

Competing interests

The authors have no competing interests.

Open Access Policy

This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Copyright

The authors have no competing interests.