I really like the idea of this, and I go back and forth on this at times.
Philosophically, I think there is certain benefit to separating source-code from the idea of release artifacts, which makes me lean toward using an artifact repository for version management, rather than source-control.
Somewhere in the middle, there are tools that shoe-horn Git into being an artifact repository by abusing LFS, which I find intriguing. See gpm, for example.