sounds like a bandsteering issue
most routors work on both bands and will ask your device if it wants 2g or 5g... depending on distance \ interference this can take the few mins you are seeing as both options are tested by your routor
in general the fix is to set one or the other at the router i.e, always give the PS5 a 5g but each maker has their own software for setting these things