Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

While it's true people are upset at AI companies profiting off of artist creations with no compensation, I know a lot of people are also reacting to how the recent AI companies have been scraping the web. The reason folks are using Anubis and other methods is because unlike Google which did have archiving of sites for a long time (which was actually a great service), these new companies do not respect robots.txt, do not crawl at a reasonable rate (for us, thousands of hits a minute from their botnets - usually baidu/tencent, but also plenty of US IPs), hit the same resource repeatedly, ignoring headers intended to give cache hints, stupidly hitting thousands of variations of a page when crawling search results with no detection that they are getting basically the same thing... And when you ban them, they then switch to residential ranges. It really is malicious.


> AI companies profiting

Are they?


If you boil it down to the AI companies are making money (subscriptions, etc.) based on content they did not pay to produce, then they are profiting from someone else's hard work.


If you boil it down anyone on this planet can access ChatGPT (and other for profit LLMs) for free to answer any question they might have.

Knowledge is shared among humanity at a rapid speed. Everyone benefits.

It’s mind boggling how anyone could be opposed to that.


Do you use google search for work? Then you are profiting from someone else's hard work!


The difference is that when you search on Google - at least before AI overviews - you end up at the source site.

Also Google respects robots.txt. Every site that Google surfaces chose to be in the index.


Thats not entirely true. Google might or might not hide your pages from index. They'll definitely going to scrape it anyway. They also display summarized info from your page (famous "what is scrapping" joke showing wikipedias summary). Finally, you might just get your answer without visiting - just by skimming result description.


Revenue is not profit.


I didn't say it was. I understand that profit = revenue - cost.

I said they're profiting from other people's hard work, a separate concept.


'stealing is fine if you lose money when reselling'


I don't believe I wrote anything of the sort.


profiting != profit


Well, don't we have enough Acme Corporations in the world that were unprofitable and existed purely on VC life support before they killed off all the competition by dumping the prices, and then made them skyrocket to recoup investments and become profitable after becoming monopolists?


People at these companies are receiving a salary to do these things that the person you're responding to is opposed to.

While not all the companies in question may or may not be profiting from these things some of them are, and most if not all of their employees certainly are as well.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: