Aha I'm not crazy! There was about a week where YouTube videos were so slow and we mainly use SmartTube on the shield TV, the fucked up part is I pay for YouTube premium, I just like the UI and options available on SmartTube better then the default app.
It did seem to get better, so maybe they realized I fucking pay for this service.