Signature is unique feature for file, something like fingerprint of an executable. Signature based methods use patterns extracted from various malwares to identify them and are more efficient and faster than any other methods. Signature based methods have small error rates and this is the reason these are often used in commercial applications.
But, signature based methods are unable to detect unknown malware variants and requires high amount of manpower, time and money to extract unique signatures. Further, it is difficult to identify infection such as polymorphic and metamorphic codes.
Behaviour based malware detection techniques observer behaviour of the program to conclude whether it is malicious or not. In these methods, programs with same behaviour are collected and a behaviour signature is developed. This signature can identify various samples of malware of same family. A behaviour based detector basically consists of following components:
Data collector - Collects dynamic/static information about the executable Interpreter - Converts raw information collected by data collector into intermediate representation Matcher - compare interpreter representation with the signature
One example of behaviour based detection approach is histogram based malicious code detection by Symantec.
Main advantages of behaviour based malware is ability to detect unknown or polymorphic malware variants. But disadvantage are high False Positive Ratio(FPR) and high amount of scanning time.
Heuristic malware detection methods use data mining and machine learning to learn behaviour of malicious file - e.g. Naive Bayes and Multi-Naive Bayes are employed to classify malware and benign files(https://ieeexplore.ieee.org/xpls/icp.jsp?arnumber=6620049) Typicall, these use API/System calls, N-gram, Op-code features.
Malware authors try to hide malware presence by adopting techniques such as
Obfuscation - Actions such as garbage commands, un-necessary jumps etc.
Code encryption - Contain defensive mechanism to encrypt themselves or its malcious activities. Encrypted malware is a complex consists of a decryption algorithm, encryption algorithm, encryption keys and encrypted malicious code. When the malware runs, the key and decryption algorithm have been used to decrypt its malicious part.
More details - https://ieeexplore.ieee.org/xpls/icp.jsp?arnumber=6620049