Just an UPDATE: Ever since I have removed the second phone on the other port, I have had no problems. I guess for some reasons it just didn't like it. Oh, I now have my phone with the caller ID plugged in as the solo phone. After something like 15 years, I just noticed a switch on the side of the phone that says Normal:Hi. For some reason I thought it was referring the the LED caller ID readout, but of course, it was referring to the volume! So now I can hear perfectly when I'm on that phone.