The pandas API is awful, but it's kind of interesting why. It was started as a financial time series manipulation library ('panels') in a hedge fund and a lot of the quirks come from that. For example the unique obsession with the 'index' - functions seemingly randomly returning dataframes with column data as the index, or having to write index=False every single time you write to disk, or it appending the index to the Series numpy data leading to incredibly confusing bugs. That comes from the assumption that there is almost always a meaningful index (timestamps).
I hate to be the "you're holding it wrong" guy but 90% of "Pandas bad!" posts I find are either outright misinformed or mischaracterizing one person's particular opinion as some kind of common truth. This one is both!
> That comes from the assumption that there is almost always a meaningful index (timestamps)
The index can be literally any unique row label or ID. It's idiosyncratic among "data frames" (SQL has no equivalent concept, and the R community has disowned theirs), but it's really not such a crazy thing to have row labels built into your data table. Excel supports this in several different ways (frozen columns, VLOOKUP) and users expect it in just about any table-oriented GUI tool.
> having to write index=False every single time you write to disk
If you're actually using the index as it's meant to be used, you'd see why this isn't the default setting.
> functions seemingly randomly returning dataframes with column data as the index
I assume you're talking about the behavior of .groupby() and .rolling()? It's never been random. Under-documented and hard to reason about group_keys= and related options, yes. But not random.
> appending the index to the Series numpy data leading to incredibly confusing bugs
I've been using Pandas professionally almost daily since 2015 and I have no idea what this means.
I think the commenter you are replying to might well understand these nuances. The point is not that Pandas is inscrutable, but instead that it‘s annoying to use in many common use-cases.
> but it's really not such a crazy thing to have row labels built into your data table.
Sometimes you need data in a certain order. Sometimes there is no primary key. And it is nuts how janky the pandas API is if you just want the index to mean the current order of the dataframe and nothing else. Oh you did a pivot? I'm just going to make those pivot columns a row label now if that's alright with you. I don't do that for all functions though, you're going to have to remember which ones. Oh you want to sort a dataframe? You better make damn sure you reindex if you're planning to use that with data from another dataframe (e.g. x + y on data from separate dataframes), otherwise I'm going to align the data on indices, and you can't stop me. Also - want to call pyplot.plot(df['column'])? Yeah I'm giving it the data in index order obviously I don't care about that sort you just did. Oh you want to port this data to excel? Well if your row labels aren't meaningful and you don't want "Unnamed: 0" you're going to have to tell me not to. You need to manipulate a multi-index? You're so cute. Have fun with that buddy.
There is a reason no other dataframe library does this - because it's confusing and cognitive overhead that doesn't need to exist. I've used pandas since ~2013, had this chat with colleagues and many recommend just giving in and maintaining an index throughout. Except I've read their pandas and it sucks because now _you_ need to reason about what is currently the index - because it actually needs to change a lot to do normal things with data. I just use .reset_index copiously and try to make it behave like a normal dataframe library because it's just easier to understand later. Pandas has not earned the right to redefine what a dataframe means.
At the absolute least, index behaviour should be opt-in, not something imposed on the user.
> After careful consideration of Oracle’s current business needs, we have made the decision to eliminate your role as part of a broader organizational change.
That is being laid off, not being fired - big difference. Being fired means being let go for poor performance / bad behaviour. No severance or grace period is necessary there (will be written in the contract). Being made redundant, particularly a redundancy of this size is quite well protected in EU. Typically negotiations between HR and representatives of the laid off group are required, you will continue to work (officially at least) until negotiations are over, as you are not officially out yet. This usually takes a few weeks.
Is HN in complete denial about what is happening to the younger generations right now? My whole family are teachers, and they are all sounding the alarm. A majority of kids are basically unable to read books now. Not just children - young adults studying English literature at college...
Parents are up against some of the wealthiest companies on earth, and the fear of socially excluding their kids by limiting their usage. Systemic change is never going to come from parents on this one.
The problem seems to be that many students going to college can't seem to read any substantial texts anymore, while somehow getting themselves into college. It's pretty worrying imo. There's a bunch of articles about this as well: https://www.theatlantic.com/magazine/archive/2024/11/the-eli...
It's their attention span. My SIL is an English professor and she stopped assigning long texts. The kids won't read it, will get an AI to summarize, and then give her poor reviews at the end for making them read.
HN is in denial about a lot of stuff. The tech bubble exists somewhere else to most people's reality.
A lot of my youngest's peers are pretty illiterate still at 13. They have trouble with more than a few minutes of concentration. They track reading age and the average is declining every year as they arrive at secondary school which is causing a big panic in UK education. I think some of this data is driving the legislation changes as well.
I'd have preferred the government to have targeted the social media and attention companies personally. Extremely high taxation would be a good start much as we do for cigarettes and alcohol. If the business is no longer viable at that point they can quite frankly fuck off.
The verification controls are possibly a bigger problem which has serious consequences for society going forwards. Things aren't too bad now but in the future, the information and data that is available makes the nazis and the stasi look like amateurs.
Drawing a false equivalence between the internet and literal chemical poisons that aren't safe at any dose, cause severe physical addictions that take away choice to stop at best, and disable and kill millions of people every year at worst, like alcohol or cigarettes, is a little too on the nose.
At some point, you have to ask how much of the rhetoric is driven by hysteria and moral panic and how much of it is driven by what the actual evidence shows.
From the Guardian[1]:
> Social media time does not increase teenagers’ mental health problems – study
> Research finds no evidence heavier social media use or more gaming increases symptoms of anxiety or depression
> Screen time spent gaming or on social media does not cause mental health problems in teenagers, according to a large-scale study.
> With ministers in the UK considering whether to follow Australia’s example by banning social media use for under-16s, the findings challenge concerns that long periods spent gaming or scrolling TikTok or Instagram are driving an increase in teenagers’ depression, anxiety and other mental health conditions.
> Researchers at the University of Manchester followed 25,000 11- to 14-year-olds over three school years, tracking their self-reported social media habits, gaming frequency and emotional difficulties to find out whether technology use genuinely predicted later mental health difficulties.
From Nature[2]:
> Time spent on social media among the least influential factors in adolescent mental health
From the Atlantic[3] with citations in the article:
> The Panic Over Smartphones Doesn’t Help Teens, It may only make things worse.
> I am a developmental psychologist[4], and for the past 20 years, I have worked to identify how children develop mental illnesses. Since 2008, I have studied 10-to-15-year-olds using their mobile phones, with the goal of testing how a wide range of their daily experiences, including their digital-technology use, influences their mental health. My colleagues and I have repeatedly failed to find[5] compelling support for the claim that digital-technology use is a major contributor to adolescent depression and other mental-health symptoms.
> Many other researchers have found the same[6]. In fact, a recent[6] study and a review of research[7] on social media and depression concluded that social media is one of the least influential factors in predicting adolescents’ mental health. The most influential factors include a family history of mental disorder; early exposure to adversity, such as violence and discrimination; and school- and family-related stressors, among others. At the end of last year, the National Academies of Sciences, Engineering, and Medicine released a report[8] concluding, “Available research that links social media to health shows small effects and weak associations, which may be influenced by a combination of good and bad experiences. Contrary to the current cultural narrative that social media is universally harmful to adolescents, the reality is more complicated.”
Way to cherry pick citations. Have you considered writing a meta analysis for a journal and fail to disclose your interests and funding? That'd really top it off.
I can do the same if I want the other way. But it's not worth my time.
You're going to drop a bombshell like "social media is as bad as alcohol and cigarettes, we need to ban it" and not provide any evidence?
There are a lot of strong feelings around social media, and I'm no fan, but I'm not going to walk head first into a moral panic, or participate in witch hunt, without knowing the facts.
In the end, ad hominem arguments don't affect the validity of evidence. I was hoping to have an interesting discussion, but I see that if you aren't politically correct on this topic, evidence will be outright dismissed and the messenger shot for delivering it.
Everyone likes to say the UK is a police state. It’s a bit of a meme. I mean we are literally going through legal reform at the moment to make it less of one while people with a masked presidential police force scream at us for being a police state.
Keep in mind that the UK government is currently locking people up for FB posts. Not exactly a police state but close enough that its a distinction without a difference. Oh, and they are debating if to get rid of jury trials so they can just lock up people for FB posts without a trial. If it quacks like a duck...
Firstly the incumbent legislation is actually being rolled back at the moment by Mahmood. The FB posts are all inciting violence against others which should not be protected speech. As for the jury trials, have you ever been in a jury? I'd rather not thanks myself. My peers are mostly fucking idiots. And they're changing that as well.
Are all the news items about people being arrested for exercising speech not true?
I‘ve heard from multiple people already that there is a massive prosecution going in the UK against people that say „hateful“ things on the internet. Whereby „hateful“ is vaguely defined but usually in relation to religious feelings.
- actual incitement to violence, like the hotel arson
And if you look at the actual convictions, first offense for most things is usually a suspended sentence. I'd be interested to see if you can find a case on bailii (no, not social media, actual court transcripts only) which matches:
- first offense custodial sentence
- one off post, not a pattern of harassment
- between strangers
- does not include even implied threats of violence
(Last one I can think of was the Robin Hood Airport one, which hinged on whether a joke threat to blow up an airport should have been taken seriously.)
Honestly that is happening, and I think it's an overstep. I have never heard anyone talk about this in real life (that could be a London bubble though). I will say - I am nearly 40, and I've spent the last 25 years online reading about the Orwellian hell my life is (or is about to become). It has never felt like it comes from a place of lived experience. For example we infamously have a lot of cameras in the UK. 90% of them are on closed circuits in shops and it doesn't affect anything.
Right now the biggest issue in the UK is the same as most places - lack of money. It's killing our services, poisoning our politics. Everything else feels abstract in comparison.
>the effects of social media on kids is too strong, and too negative to deny at this point.
Bold claim that really needs some evidence. Is there research which shows that kids who grow up with social media are less likely to succeed as adults because of social media exposure?
To be fair, it is still pretty remarkable what the human brain does, especially in early years - there is no text embedded in the brain, just a crazily efficient mechanism to learn hierarchical systems. As far as I know, AI intelligence cannot do anything similar to this - it generally relies on giga-scaling, or finetuning tasks similar to those it already knows. Regardless of how this arose, or if it's relevant to AGI, this is still a uniqueness of sorts.
Human babies "train" their brain on literally gigabytes of multi-modal data dumped on them through all their sensory organs every second.
In a very real sense, our magic superpower is that we "giga-scale" with such low resource consumption, especially considering how large (in terms of parameters) the brain is compared to even the most advanced models we have running on those thousands of GPUs today. But that's where all those millions of years of evolution pay off. Don't diss the wetware!
Don't get fixated on plumbing itself. The point is if a bunch of people rush into any profession it leads to wage depression. Unless the amount of plumbing needed increases, the overall amount of money flowing to the plumbing populace is likely to stay roughly the same.
> The point is if a bunch of people rush into any profession it leads to wage depression
Eventually. Wage depression does not happen linearly. You're asserting that demand is maxed out and there's no more money to go around, and that's just not true. A lot of people just don't bother because tradespeople are famously difficult to work with because they are so overbooked.
These systems don't discriminate on whether the object is a child. If an object enters the path of the vehicle, the lidar should spot it immediately and the car should brake.
The trivia approach doesn't even work for most people - ask the wikipedia reader and the person who travelled to Turkey about it a year later and see who has actually retained some knowledge.
That's a stretch. One can hold the view that division of labour is a useful economical principle, but also that oligopolies represent a dangerous concentration of power.
I think one of the best arguments against US interventionalism when it comes to tyrants is just how 'variable' (let's say) the outcomes have been over the years. For every Panama, there's two or three Guatamalas, Irans or most recently Iraq. Generally the hard part is not the removal of the head of state, which for the US is usually pretty quick. It's what beurocratic structures remain functional and whether the power vacuum created brings something better and more robust, or just decades of violence.
I think Sarah Paine on dwarkesh has noted that it tends to go well when the countries already have fairly robust institutions and tends to go badly when they don't
As I'm not a historian, I can only note that it hasn't gone well recently even when multiple successive presidents want it to
reply