Pyrit - Decreasing Gains with more GPUs

I recently moved over from a single Nvidia GPU (CUDA) setup to a multiple ATI (Cal++) setup. As I was installing the ATI cards, I noticed I was getting decreasing per card performance with each new card I added.

1 Radeon HD card yields ~ 63,000 PMK/s
2 Radeon HD cards yields ~ 50,000 PMK/s (per card)
3 Radeon HD cards yields ~ 45,000 PMK/s (per card)

I dug around and found a few instances of people reporting the same thing but I couldn't find a case where a solution was found. Furthermore, I couldn't find anyone who could explain the underlying reason the problem was occuring. Does anyone know why this is happening and what I can do to fix it?

Also with my Nvidia setup I found that I got a better benchmark when I booted to shell prompt and ran Pyrit without X server. Is there a way to run Pyrit without X Server for ATI Cards using Cal++? When I tried to run Pyrit without X Server, my GPUs werent recognized, just the CPU cores. If not, what other things can be done to optimize performance? Thanks for the help guys.

im afraid theres nothing you can do with that, I think its about the decreased gain explained by Amdahl's law http://en.wikipedia.org/wiki/Amdahl%27s_law

Thanks for the reply Bloodrain. Ive decided to just add more PCs to work around this problem.

Does anyone know if there is there a way to run Pyrit without X Server for ATI Cards using Cal++? I think X is slowing me down a lot.