If you’re not cleaning it you definitely should be, at least a big microfiber brush and some spray to start… I got my kit from “Big Fudge” on Amazon, but there are many similar. You want to wipe in an outward spiral from the center pushing any dust and stuff to the outside edge.
I have not tried it on records yet, but ultrasonic cleaning is supposed to be the gold standard. You can get a gizmo that clamps over the label to keep it from getting wet, or even special cleaners designed to rotate albums through the bath.
An audio sample, even a cell phone video would probably help… the troubleshooting you’ve already done points pretty strongly towards an issue with the LP in question.