I'm a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?
Programmer Humor
Welcome to Programmer Humor!
This is a place where you can post jokes, memes, humor, etc. related to programming!
For sharing awful code theres also Programming Horror.
Rules
- Keep content in english
- No advertisements
- Posts must be related to programming or programmer topics
Seems like a good excuse to someone who doesn't know what they're doing and needs an excuse because why they haven't completed it yet?
The whole post is complete bs in multiple ways. So weird.
Some interesting facts about excel I learned the hard way.
- It only supports about a million or so rows
- It completely screws up numbers if the column is a number and the number is over 15 digits long.
Not really related to what you said, but I'm still sore about the bad data import that caused me days of work to clean up.
It completely screws up numbers if the column is a number and the number is over 15 digits long.
I work in insurance in Brazil, by standards of our regulatory body, claims numbers must be a string of 20 numbers (zfill(20) if needed). You can't imagine the amount of times excel had fucked me up rounding down the claim numbers, this is one of the first things I teach to my interns and juniors when they're working with the claims databases.
The row limitation seems, to me, like an actually-good thing. Excel is for data where you might conceivably scroll up and down looking at it and 1M is definitely beyond the ability of a human even to just skim looking for something different.
An older version of Excel could only handle 64k rows and I had a client who wanted large amounts of data in Excel format. "Oh sorry, it's a Microsoft limitation," I was thrilled to say. "I have no choice but to give you a useful summarization of the data instead of 800k rows (each 1000 columns wide) of raw data."
I didn't know hard drive overheating was a thing. Should I be worried that my 5 year old hard drive is about to overheat. I mean is this actually a floppy disk or something?
it is, in the select event that your platter bearing fails, in which case it would be very, very obvious.
When the only thing that is stopping kids from dismantling your government is an O(N^N) algorithm
Are you telling me there’s a difference between an inner and a cross join?
Cross join is obviously faster, I don’t even have to write “on”
Hard drive was made by Tesla
my hard drive overheated
So, this means they either have a local copy on disk of whatever database they're querying, or they're dumping a remote db to disk at some point before/during/after their query, right?
Either way, I have just one question - why?
I'd much sooner assume that they're just fucking stupid and talking out of their ass tbh.
Same as Elon when he confidently told off engineers during his takeover of Twitter or gestures broadly at the Mr. Dunning Kruger himself
Wonder if it’s an SQL DB
Elon probably hired confident right wingers whose parents bought and paid their way through prestigious schools. If he hired anyone truly skilled and knowledgeable, they’d call him out on his bullshit. So the people gutting government programs and passing around private data like candy are just confidently incorrect
Even if it was local, a raspberry pi can handle a query that size.
Edit - honestly, it reeks of a knowledge level that calls the entire PC a "hard drive".
Have you ever heard of case of overheating hard drives within the last decade?
Plus, 60k is nothing. One of our customers had a database that was over 3M records before it got some maintenance. No issue with overheating lol
I run queries throughout the day that can return 8 million+ rows easily. Granted, it takes few minutes to run, but it has never caused a single issue with overheating even on slim pc’s.
This makes no fucking sense. 60k rows would return in a flash even on shitty hardware. And if it taxes anything, it’s gonna be the ram or cpu- not the hard drive.
My one question would be "How?"
What the hell are you doing that your hard drives are overheating? How do you even know it's overheating as I'm like 90% certain hard drives (except NVMe if we're being liberal with the meaning of hard drive) don't even have temperature sensors?
The only conclusion I can come to is that everything he's saying is just bullshit.
They have temp sensors. But have never heard of a overheating drive.
From the same group that doesn't understand joins and thinks nobody uses SQL this is hardly surprising .
Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.
You're giving this person a lot of credit. It's probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex's to find the data they're looking for.
Considering that is nearly exactly some of the answers I've received during the technical part of interviews for jr data eng, you're probably not far off.
Shit I've seen solutions done up that look like that, fighting the optimiser every step (amongst other things)
As a reasonably experienced "data guy," this seems obviously laughable, but the discussion on X is scary. This guy is a savior in the MAGA world.
We can criticize and poke fun all day, but it doesn't matter much if our message isn't challenging the mindset of those with other opinions.
How do we make better use of our time to impact outside opinion?
We must make better memes
I'm not even joking, the world runs on memes now. It's fucking stupid, but we must shitpost to save ourselves
I've been told violence isn't the answer and we shouldn't just shoot nazis and nazi enablers dead.
The way most people change their mind isn't based on facts or figures, but emotions. Specifically, in-group belonging. For most people, and this certainly includes me and you some of the time, what our in-group believes is more compelling than an out-groups supposed facts.
They see that guy as someone in their group so they believe him. They see you as a bad outside bad bad bad liar, so nothing you say is likely to get through. (This comic is worth reading on this topic: https://theoatmeal.com/comics/believe )
If you want to change someone's mind, they have to see you as in-group. Not necessarily the same group as what you're arguing with. We all belong to many groups. American, new yorker, white guy, middle aged, yankees fan, etc etc there are many such slices. Like how you can't get a republican to recycle by appealing to environmental concerns (because environmentalists are out-group, so fuck them), but you might be able to get them to recycle via something like "only american ingenuity can turn trash into bridges and tanks!"
This takes a lot of time and effort, and if you don't get them to stop hanging out with the other group, you won't make any lasting changes.
So I think you'd need a multi prong approach:
- Get them off bad media. Facebook, fox news, etc. This is reinforcing their bad beliefs. Because they see this stuff as trustworthy in-group, it goes right into the worldview.
- Get them to stop hanging out with their shitty maga-hat friends. This is the social in-group that's reinforcing bad beliefs.
- Get them to trust you.
- Gently introduce the idea that maybe the extreme right doesn't have their interests at heart, etc
All of which takes a lot of time and effort, and your opposite number is basically trying to do the same thing. Except they have fox news, trump, and such in their corner.
And, again, I'm told we definitely shouldn't just shoot extreme right wingers and other nazi sympathizers dead. Nor should we burn their houses down. If we're an emergency responder, we definitely shouldn't let them die while thinking to ourselves "they would let so many die. without a thought, their passing deserves no mourning" or similar.
You should definitely nullify if you're on a jury and someone allegedly did violence to a shitty ceo or red-hat, though, bu that's getting off topic.
I cannot believe these people make more than me lol.
They make nothing. They're compensated for destroying things, and considering it's musk, they're likely given relatively little money in return for their time.
Even if the only thing you do all day is sit on the toilet and yell at the Internet, you're already a bigger net positive on society.
Is this a real post? I can’t seemed to find it on that website “X, formerly known as Twitter.”
What in the fuck is this idiot doing? I've process datasets far larger than that and never once have I run into a hard drive "overheat". I mean what level of incompetence do you have to have to get a hard drive to overheat processing a measley 60K rows of data?
60k isn't that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?
60k is laughably, embarrassingly small. It's still sqlite-sized.
i mean its even excel sized depending on how many columns. This is seriously sad and alarming
Hey now that's real close to the 65,535 16-bit limit (from 20 years ago)
A TI-86 can query 60k rows without breaking a sweat.
If his hard drive overheated from that, he is doing something very wrong, very unhygienic, or both.
Wow.
I've been processing a couple of billion rows of data on my machine, the fans didn't even come on. WTF are they teaching "experts" these days, or has Elmo only hired people who claim that they can "wrangle data" and say "yes" ?
60k rows is generally very usable with even wide tables in row formats.
I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.
After 1M rows move on to something better like Dask, polars, spark, or literally any DB.
The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.
My go-to tool of late is duckdb
, comes with binaries for most platforms, works out of the box, loads any number of database formats and is FAST.
does elon only hire chip from sales guy vs web dude or something
No he also hire people who created a script to make fake ballots with a bias.
https://bsky.app/profile/denisedwheeler.bsky.social/post/3lhowh3ijgs2f