this post was submitted on 29 May 2025
17 points (90.5% liked)

Linux

54869 readers
400 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 6 years ago
MODERATORS
AgreeableLandscape@lemmy.ml
nooter692@lemmy.ml
MarcellusDrum@lemmy.ml
cypherpunks@lemmy.ml
cyclohexane@lemmy.ml
d3Xt3r@lemmy.nz
17
Trying to recreate a version control system for my music collection, with one crucial difference ... 🤯 (feddit.org)
submitted 1 week ago* (last edited 1 week ago) by A_norny_mousse@feddit.org to c/linux@lemmy.ml
10 comments fedilink hide all child comments
 

I want to have a mirror of my local music collection on my server, and a script that periodically updates the server to, well, mirror my local collection.

But crucially, I want to convert all lossless files to lossy, preferably before uploading them.

That's the one reason why I can't just use git - or so I believe.

I also want locally deleted files to be deleted on the server.

Sometimes I even move files around (I believe in directory structure) and again, git deals with this perfectly. If it weren't for the lossless-to-lossy caveat.

It would be perfect if my script could recognize that just like git does, instead of deleting and reuploading the same file to a different location.

My head is spinning round and round and before I continue messing around with find and scp it's time to ask the community.

I am writing in bash but if some python module could help with it I'm sure I could find my way around it.

TIA

additional info:

  • Not all files in the local collection are lossless. A variety of formats.
  • The purpose of the remote is for listening/streaming with various applications
  • The lossy version is for both reducing upload and download (streaming) bandwidth. On mobile broadband FLAC tends to buffer a lot.
  • The home of the collection (and its origin) is my local machine.
  • The local machine cannot act as a server
you are viewing a single comment's thread
view the rest of the comments
[–] bastion 3 points 1 week ago (2 children)

Make a script. I'd use xonsh or python with sh.py.

  • create a dict for remote to local filename map
  • walk your local collection
    • for each file, determine what the correct remote name (including a valid extension) would be, and add the pair to the dict, with remote filenames as keys, local filenames as values
  • make a set like local_munged_names from that dict's keys
  • walk your remote tree, and store the filenames in a set like remote_names
  • names_to_upload = local_munged_names - remote_names
  • for each name in names to upload, look up the local filename from the remote to local filename map. Then, encode it if it needs encoding, and upload.
[–] vandsjov@feddit.dk 2 points 4 days ago

I like this. I would probably over complicate things with a index CSV file (or SQL) that stores checksum values of files to identify renamed or moved files.

[–] A_norny_mousse@feddit.org 2 points 1 week ago (1 children)

xonsh or python with sh.py.

Very interesting!

And thanks for the coding tips. It seems git is not the best option here because it keeps a full history of all files in their fullness - a gigantic waste of space in the case of a media collection.

I am now thinking more rsync minus lossless formats, then deal with lossless formats separately.

[–] bastion 1 points 1 day ago* (last edited 1 day ago)

you know, you could also either include a lossy copy next to the lossless ones, then rsync only lossy extensions, or, if that pollutes your collection, have a separate but identically-structured directory tree, where all your lossless files have lossy copies. Then, you can rsync both folders (send-only) to your single remote (lossy extensions only).

but, yeah, Git really isn't the tool for this, agreed.