There is a way to encode pdf files to bypass AV, yes, but you cant do it with msfencode because that is used to encode shellcode (payloads essentially), not entire exploits. Encoding of particular file types requires file type specific methods.
I actually encoded a pdf file to bypass antivirus last week and was planning to write a blog entry about how, so I will post a link to that here when its done.
If you're impatient, here is the basic process, which can be used for pdf exploits that use Javascript code to trigger the exploit (and that applies to the majority of current pdf sploits):
- Obtain the base JavaScript used to perform the exploit. This can be extracted from existing exploits if you know how.
- Obfuscate the JavaScript code for the exploit. Use your imagination here, and do a good job at it because this is the step that will provide the maximum benefit in terms of AV avoidance. You need workable Javascript knowledge for this.
- Use make-pdf tools from Didier Stevens to create a PDF that auto runs the script
- Compress the PDF using pdftk


