The real question is all the stuff beyond just having the distro installed. The packages, the services, the configs, the application data.
If you leave all that stuff the way it was installed via the old package manager, it may have some bad assumptions baked in and may be incompatible with packages you install with the new package manager.
And if you clear all of it out and reinstall it, have you really gained anything vs. just doing a clean install?
There’s a reason you have a home dir. Just copy that forward along with whatever other config files you might’ve customized.
Btw, if the ability to make drastic changes while still maintaining continuity is an important feature for you, maybe check out NixOS.