I too had the same problem, but on Broadvoice though.

I too had Panasonic 5.8ghz cordless phones.

On a hunch/splurge, I replaced it with a Panasonic four handset DECT version.

I do not notice the problem any longer. (Now on VOIPo.)

Interesting to note; you will not find this limitation (or improvement) in the Panasonic literature or any review online. Odd.