I can make calls now, I guess the ATA doesn't return pings.

I'm going to try a switch between the modem and the routers, I have a backup unit running VRRP that takes over if the main router fails, so they both need to see the modem as they have unique public IPs.