I used to have the same problem with Ventrilo under Vista, I used the higher process priority workaround. Not ideal but it worked.
When I moved to my latest guild I found them using an open source voice server called 'mumble'. Its actually pretty good, it works fine without any workarounds on XP, Vista and Win 7. Low system footprint, low bandwidth / latency and when configured properly good voice quality too.
If you're interested you can download it
here and there's install guide for various OS
here.