HomePod looses connection when Phone switches Access Point

I have a wifi network setup with tp-link easy mesh. When my phone is connected to my main router it plays music fine, but as soon as I walk away and the phone roams into the second access point it disconnects AirPlay. I am unable to reconnect. The network is set up correctly, I can ping and access all devices, servers, IOT devices etc. no AP Isolation.