this post was submitted on 27 Jan 2025
125 points (99.2% liked)

Lemmy.ca's Main Community

3016 readers
7 users here now


Welcome to the lemmy.ca/c/main community!

All new users on lemmy.ca are automatically subscribed to this community, so this is the place to read announcements, make suggestions, and chat about the goings-on of lemmy.ca.

For support requests specific to lemmy.ca, you can use !lemmy_ca_support@lemmy.ca.


founded 4 years ago
MODERATORS
 

Hello everyone!

I'll be taking the site down for two maintenance windows this week to complete our server migration.

  • Weds Jan 29th - 09:00 - 11:00 PT (12:00 - 14:00 ET)
  • Thurs Jan 30th - 09:00 - 11:00 PT (12:00 - 14:00 ET)

During the first window I'll be migrating us from OVH to our new dedicated hardware. After this migration there will likely be some temporarily broken images, as it takes approximately 8 hours to resync our object storage from OVH.

This is a major change and despite my testing, may have some unintended side effects. If you run into any problems that aren't just a broken image, please let us know.

The second maintenance window is to migrate our pict-rs database from it's local sled-db into our primary postgres DB. This is a much smaller change but since pict-rs checks every image as it goes through them, it takes about 1.5 hours.

As usual, you can check https://status.lemmy.ca/ for updates.

all 36 comments
sorted by: hot top controversial new old
[–] Shadow@lemmy.ca 23 points 3 weeks ago* (last edited 3 weeks ago) (5 children)

First move complete, went smoothly =)

The broken images and thumbnails should fill in over the next 8-10 hours 🀞

Let me know how performance is for everyone, I expect it to be quite a bit snappier!

[–] wise_pancake@lemmy.ca 5 points 3 weeks ago* (last edited 3 weeks ago)

Much faster than I expected!

Performance already seems fine for me

Edit: I can scroll as fast as I can on my phone and never have to wait for a post to load, your upgraded server is awesome!

[–] Sunshine@lemmy.ca 4 points 3 weeks ago

That was really quick!

[–] ininewcrow@lemmy.ca 3 points 3 weeks ago

Beautiful work ... thanks for everything

[–] avidamoeba@lemmy.ca 2 points 3 weeks ago* (last edited 3 weeks ago)

Didn't even notice. Today's been busier at work. :D

Sorted by most comments, opened a 400-comment thread. Fast-fast.

[–] GrindingGears@lemmy.ca 1 points 3 weeks ago

Seems a lot snappier for sure!

[–] Albbi@lemmy.ca 25 points 3 weeks ago

Good luck with the server migration!

[–] Shadow@lemmy.ca 11 points 3 weeks ago (3 children)

Second move complete, all done for now!

[–] GlassHalfHopeful@lemmy.ca 3 points 3 weeks ago

Great news. I hope it was relatively non-eventful.

[–] ininewcrow@lemmy.ca 1 points 3 weeks ago

Beautiful work ... everything purring like a well tuned motor over here

Thanks!

[–] Darkassassin07@lemmy.ca 1 points 3 weeks ago* (last edited 3 weeks ago) (1 children)

I've noticed my profile pic is partially broken.

The mini-thumbnail beside my name in comments appears fine; but the image in 'about' on my profile is broken. (viewed from Jerboa).

It opens properly in full-res when you click on the broken image though.

Haven't tried re-uploading it yet. Just figured I'd report it.

/edit Broken in the side bar too

Fixed with a re-upload.

[–] Shadow@lemmy.ca 1 points 3 weeks ago (2 children)

Do you happen to have the broken image url still?

[–] Darkassassin07@lemmy.ca 1 points 3 weeks ago* (last edited 3 weeks ago)

I don't, but I'll pm you an IP address. Check the logs for 404s from that ip? (around 4 hours ago)

[–] Darkassassin07@lemmy.ca 1 points 3 weeks ago (2 children)

It looks like your profile is doing the same; but only when viewed from Jerboa.

Jerboa:

Web browser:

I didn't check mine in a browser at the time.

[–] Shadow@lemmy.ca 3 points 3 weeks ago

Thanks, I see it too. Digging.

[–] Shadow@lemmy.ca 2 points 3 weeks ago (1 children)

I think this is fixed now. I'm not sure why but pictrs was expecting a bunch of resized images that don't exist in either the old or new object stores. I've manually cleaned up the DB.

[–] Darkassassin07@lemmy.ca 2 points 3 weeks ago* (last edited 3 weeks ago) (2 children)

Many of them are, but scrolling through this thread and checking the profiles of various lemmy.ca accounts; some are still broken.

@sunshine@lemmy.ca for example

[–] Shadow@lemmy.ca 1 points 3 weeks ago

Well shit, cloudflare's been caching all our images and deceiving me into thinking stuff was working properly. Digging, but I made a mess =(

[–] Shadow@lemmy.ca 1 points 3 weeks ago (1 children)

I've hacked in a dirty dirty fix for now. It looks like the pictrs database migration failed horribly and cloudflare was caching the images, so nobody noticed. Now I've got two copies of pictrs running (old + new) and some code in place to try both of them.

Sigh.

[–] Darkassassin07@lemmy.ca 1 points 3 weeks ago

Oh shit.... Well at least you found it before everything just stopped working? :(

Thanks for your efforts m8; I appreciate the hard work. Best of luck fixing that.

[–] recursive_recursion@lemmy.ca 9 points 3 weeks ago

Thank you for the updatesπŸ€—
I hope the migration goes well!βœŠπŸ€

[–] Darkassassin07@lemmy.ca 8 points 3 weeks ago

Damn, not even 20min?

Well done. πŸ‘

[–] saigot@lemmy.ca 8 points 3 weeks ago (1 children)

so which canadian politician is gunna resign this time?

[–] n2burns@lemmy.ca 5 points 3 weeks ago

🀞🀞🀞

[–] ininewcrow@lemmy.ca 7 points 3 weeks ago

Good luck with the work ... and thanks for doing all this

[–] Shadow@lemmy.ca 7 points 3 weeks ago* (last edited 3 weeks ago) (3 children)
[–] Shadow@lemmy.ca 6 points 3 weeks ago

Test from sync

[–] Shadow@sh.itjust.works 4 points 3 weeks ago

Incoming federation test.

[–] Sunshine@lemmy.ca 2 points 3 weeks ago

I see the Canuck lemming

[–] charles@lemmy.ca 7 points 3 weeks ago* (last edited 3 weeks ago) (1 children)

What's the best way to let you know if we're having issues after the maintenance windows? Any on a different platform if we have issues accessing Lemmy?

Thanks for all the hard work!!

[–] Shadow@lemmy.ca 10 points 3 weeks ago

We're also in the lemmy rooms on matrix, but shadow@fedecan.ca is probably the quickest method to get my attention if you can't even post on Lemmy at all. I really don't expect any issues though.

[–] stormio@lemmy.ca 7 points 3 weeks ago

Exciting. This brings us one step closer to Pixelfed and Friendica hosted by Fedecan.

[–] Sunshine@lemmy.ca 6 points 3 weeks ago

Thank you for the great work Shadow!

I’m so excited for the increased autonomy of Lemmy.ca 🫑

[–] GlassHalfHopeful@lemmy.ca 6 points 3 weeks ago

Good luck, @Shadow@lemmy.ca. 🀞

[–] unnamedau@lemmy.ca 5 points 3 weeks ago

good luck and thank you!!