The randomness of your passphrase determines its strength. An attacker will most likely have wordlists with common number/letter replacements people use as well as the common phrases used, but usually a 22 random (the randomness is key) character passphrase that isnt in popular tables or wordlists would take a long time to crack. This is of course dependant on the speed of the attackers setup, but assuming it isn't someone with a decent amount of money and time to dedicate to getting your key, you should be safe. Correct me if im wrong of course!
Really though if your network was that important, a very determined attacker would probably choose a different attack vector such as phishing or am I just paranoid ;()