There is a way to encode pdf files to bypass AV, yes, but you cant do it with msfencode because that is used to encode shellcode (payloads essentially), not entire exploits. Encoding of particular file types requires file type specific methods.
I actually encoded a pdf file to bypass antivirus last week and was planning to write a blog entry about how, so I will post a link to that here when its done.
- Use make-pdf tools from Didier Stevens to create a PDF that auto runs the script
- Compress the PDF using pdftk