For web publishers, stopping AI bots from scraping their best content while consuming valuable bandwidth must feel somewhere between futile and nigh impossible.

It’s like throwing a cup of water at a forest fire. No matter what you try, the new generation of bots keeps advancing, insatiably consuming data to train AI models that are currently in the grip of competitive hyper-growth.

But with traditional approaches for limiting bot behavior, such as a robots.txt file, looking increasingly long in the tooth, a solution of sorts might be on the horizon through work being carried out by the Internet Engineering Task Force (IETF) AI Preferences Working Group (AIPREF).

The AIPREF Working Group is meeting this week in Brussels, where it hopes to continue its work to lay the groundwork for a new robots.txt-like system for websites that will signal to AI systems what is and isn’t off limits.

The group will try to define two mechanisms to contain AI scrapers, starting with “a common vocabulary to express authors’ and publishers’ preferences regarding use of their content for AI training and related tasks.”

Second, it will develop a “means of attaching that vocabulary to content on the internet, either by embedding it in the content or by formats similar to robots.txt, and a standard mechanism to reconcile multiple expressions of preferences.”

AIPREF Working Group Co-chairs Mark Nottingham and Suresh Krishnan described the need for change in a blog post:

“Right now, AI vendors use a confusing array of non-standard signals in the robots.txt file and elsewhere to guide their crawling and training decisions,” they wrote. “As a result, authors and publishers lose confidence that their preferences will be adhered to, and resort to measures like blocking their IP addresses.”

The AIPREF Working Group has promised to turn its ideas around the biggest change to the way websites signal their preferences since robots.txt was first used in 1994 into something concrete by mid-year.

Parasitic AI

The initiative comes at a time when concern over AI scraping is growing across the publishing industry. This is playing out differently across countries, but governments keen to encourage local AI development haven’t always been quick to defend content creators.

In 2023, Google was hit by a lawsuit, later dismissed, alleging that its AI had scraped copyrighted material. In 2025, UK Channel 4 TV executive Alex Mahon told British MPs that the British government’s proposed scheme to allow AI companies to train models on content unless publishers opted out would result in the “scraping of value from our creative industries.”

At issue in these cases is the principle of taking copyrighted content to train AI models, rather than the mechanism through which this is achieved, but the two are, arguably, interconnected.

Meanwhile, in a separate complaint thread, the Wikimedia Foundation, which oversees Wikipedia, said last week that AI bots had caused a 50% increase in the bandwidth consumed since January 2024 by downloading multimedia content such as videos:

“This increase is not coming from human readers, but largely from automated programs that scrape the Wikimedia Commons image catalog of openly licensed images to feed images to AI models,” the Foundation explained.

“This high usage is also causing constant disruption for our Site Reliability team, who has to block overwhelming traffic from such crawlers before it causes issues for our readers,” Wikimedia added.

AI crawler defenses

The underlying problem is that established methods for stopping AI bots have downsides, assuming they work at all. Using robots.txt files to express preferences can simply be ignored, as it has been by traditional non-AI scrapers for years.

The alternatives — IP or user-agent string blocking through content delivery networks (CDNs) such as Cloudflare, CAPTCHAS, rate limiting, and web application firewalls — also have disadvantages.

Even lateral approaches such as ‘tarpits’ — confusing crawlers with resource-consuming mazes of files with no exit links — can be beaten by OpenAI’s sophisticated AI crawler. But even when they work, tarpits also risk consuming host processor resources.

The big question is whether AIPREF will make any difference. It could come down to the ethical stance of the companies doing the scraping; some will play ball with AIPREF, many others won’t.

Cahyo Subroto, the developer behind the MrScraper ‘’ethical” web scraping tool, is skeptical:

“Could AIPREF help clarify expectations between sites and developers? Yes, for those who already care about doing the right thing. But for those scraping aggressively or operating in gray areas, a new tag or header won’t be enough. They’ll ignore it just like they ignore everything else, because right now, nothing’s stopping them,” he said.

According to Mindaugas Caplinskas, co-founder of ethical proxy service IPRoyal, rate limiting through a proxy service was always likely to be more effective than a new way of simply asking people to behave.

“While [AIPREF] is a step forward in the right direction, if there are no legal grounds for enforcement, it is unlikely that it will make a real dent in AI crawler issues,” said Caplinskas.

“Ultimately, the responsibility for curbing the negative impacts of AI crawlers lies with two key players: the crawlers themselves and the proxy service providers. While AI crawlers can voluntarily limit their activity, proxy providers can impose rate limits on their services, directly controlling how frequently and extensively websites are crawled,” he said.

However. Nathan Brunner, CEO of AI interview preparation tool Boterview, pointed out that blocking AI scrapers might create a new set of problems.

“The current situation is tricky for publishers who want their pages to be indexed by search engines to get traffic, but don’t want their pages used to train their AI,” he said. This leaves publishers with a delicate balancing act, wanting to keep out the AI scrapers without impeding necessary bots such as Google’s indexing crawler.

“The problem is that robots.txt was designed for search, not AI crawlers. So, a universal standard would be most welcome.”

Source:: Computer World

By Deepti Pathak If you’re a Blox Fruits player on Roblox, you can agree on how important it is…
The post Roblox Blox Fruits Values List (April 2025) appeared first on Fossbytes.

Source:: Fossbytes

By Partner Content AnyDesk is a remote desktop solution that you can easily use to connect to remote systems…
The post HelpWire: Explore the Best Alternative to AnyDesk for Remote Desktop Access appeared first on Fossbytes.

Source:: Fossbytes

By Nick Godt An unreleased version of Waymo’s privacy policy suggests the California-based company is preparing to use data from its robotaxis, including interior cameras, to train generative AI models and to offer targeted ads.

Source:: Digital Trends

By Nick Godt Audi has announced it would halt shipments to the U.S. in the wake of President Donald Trump’s 25% tariffs on all imported vehicles.

Source:: Digital Trends

By Siôn Geschwindt A UK startup wants to build a cosmic radio station in the Moon’s orbit to listen to radio waves from the early universe.   Blue Skies Space has secured a contract from the Italian Space Agency to design a fleet of tiny satellites that could orbit the Moon and listen for signals from the cosmic “dark ages.” That’s the time before the first stars lit up, when the universe was mainly a swirling mass of hydrogen gas.  Hydrogen atoms naturally emit radio waves at a very specific frequency: 1420MHz  – known as the hydrogen line. When you stretch that signal across…This story continues at The Next Web

Source:: The Next Web

By Siôn Geschwindt Despite recent claims that European startups aren’t working hard enough, new research shows the continent’s founders are putting in serious shifts to turn their ideas into successful businesses. A survey of 128 founders by early-stage VC firm Antler found that three-quarters of them work more than 60 hours weekly, with 19% exceeding 80 hours.  German founders emerged as Europe’s hardest workers, with 94% working more than 60 hours weekly and 38% exceeding 80 hours.  Daria Stepanova, co-founder of German startup AIRMO, said she’s sacrificed “time, stability, and relationships” to grow her company. However, she sees a certain level of obsession…This story continues at The Next Web

Source:: The Next Web

By Adarsh Verma Many travelers envision Japan as their ultimate destination because it unites traditional values with modern metropolises…
The post Traveling Japan on a Budget: Why eSIMs Are the Best Option appeared first on Fossbytes.

Source:: Fossbytes

By Deepti Pathak Getting blackmailed on social media platforms like Snapchat is a serious issue that should be dealt…
The post How to Report Blackmail on Snapchat? appeared first on Fossbytes.

Source:: Fossbytes

By Nick Godt A federal judge dismissed trademark claims against Tesla and Warner Bros. in a lawsuit filed by Alcon Entertainment, a production company behind the 2017 sci-fi movie Blade Runner 2049. However, the judge allowed Alcon to continue its copyright infringement claims against Tesla for its alleged use of AI-generated images mimicking scenes from Blade Runner 2049 without permission.

Source:: Digital Trends

By Nick Godt Toyota is planning to develop about 15 fully electric models by 2027, up from five currently. These models will include vehicles under the Toyota and Lexus brands, with production expected to reach 1 million units annually by that year.

Source:: Digital Trends

The AI market could grow to as much as $4.8 trillion by 2033, according to a new report from the United Nations Conference on Trade and Development. At the same time, the money is expected to be concentrated among a few players and could increase inequality between countries, CNBC reports.

“The benefits of AI-driven automation often favor capital over labor, which can increase inequality and reduce the competitive advantage of cheap labor in developing countries,” the report’s authors wrote. To ensure developing countries are not left behind, the UN group recommends they be involved in discussions on AI rules and ethics.

In the same report, the UN also warned that AI could affect 40% of all jobs, even as the technology has the potential to create new industries and empower workers — as long as investments are made in training.

Source:: Computer World

By Siôn Geschwindt Consumer tech devices, including smartwatches, have deplorably short lives. Most are tossed aside when the screen cracks, the battery dies, or the software falls behind — adding to the world’s whopping great pile of e-waste.   Scottish startup Una aims to upend this take-make-waste cycle. The company’s sports smartwatch is built to be repaired. Users can easily swap, replace, and upgrade individual components like the screen, battery, and health sensors, extending the device’s lifespan. “Customers are tired of replacing expensive tech every few years,” said Lewis Allison, Una’s founder. “We’re showing the industry there’s a better way.” The Una Watch can be…This story continues at The Next Web

Source:: The Next Web

It might be time for Apple-as-a-service as the company looks to plot a cunning course through the post-globalist freeze — and offering its products as a service could help it do just that.

Apple had been expected to introduce just such a service. In fact, Mark Gurman recently said those plans were quite advanced, but the company shelved the idea, presumably because of its potential impact on “normal” iPhone sales and consequent revenue. 

With the recent round of tariffs from US President Donald J. Trump, things have changed. Given a $3,000 iPhone has become a real possibility under the tax regime, Apple knows it’s going to see a decline in iPhone sales anyway. It knows that those iPhones it does sell will get used longer and it knows that the inevitable cost will put a lot of consumers off from buying these devices.

Apple has also bought itself some time to figure out a way forward, thanks to the planeloads of iPhones it imported into the US just before the tariff announcement was made. These should tide the company over into September, reports claim. That makes sense, because the time isn’t right for the iPhone-as-a-service plan quite yet. 

To put the plan in effect, Apple will need the support of its carrier partners who I expect were quite resistant to the idea before — they liked the margins they made on phones sold through their networks.

However, pragmatism changes things, and even they can see that some money from a lower margin is still better than no margin at all. And when analysts predict iPhone prices could hit $3,000, it’s crystal clear sales will decline.

Think about it

Those tariff taxes will impact almost everything that cannot be made, grown, or harvested in the US. They’re going to be felt, particularly by shoppers in more impoverished socio-economic groups (who also use iPhones). Most shoppers will be much too concerned about the cost of eggs to spring thousands for a phone, and while there will be an elite group of consumers for whom it’s business as usual, most people will endure a crisis of confidence. 

Like the 1920s, there may be a lot of dancing, but not much to dance about.  

The economic precipice the world appears to have been pushed over may be enough to make any iPhone-as-a-service plan look a lot more attractive. After all, it enables cash-strapped consumers to use the smartphone they desire (and perhaps also the watch, tablet, and Mac) for a predictable monthly fee, with AppleCare, iCloud+, and Apple services included, and doesn’t require they laden themselves with credit card debt. 

People are ready to accept it

It’s not as if we’re not ready for such a service. Even back in 2022, CIRP Partner and Co-Founder Josh Lowitz said: “Based on current consumer behavior, iPhone users are primed to adopt a subscription service that provides an iPhone bundled with useful apps. Almost half iPhone owners already finance their iPhone purchase, paying monthly for a new phone. And about one-third trade-in their old phone when they buy a new one. So, a significant portion of the user base is accustomed to never owning a phone, instead basically leasing it.”

Apple also gains. In this case, it benefits from potentially lower, but at least recurring, income upon which to balance its stock. And it benefits from the fact that at the end of the subscription period (or during it if the consumer cannot maintain payments), the devices will be returned for refurbishment, resale/let, and/or recycling.

This also opens up the highly lucrative second-user iPhone market, which is an income stream Apple hasn’t yet fully explored. The iPhone is the most widely sold smartphone on the second-user market and holds its value the longest; company management is said to have been eyeing whether they can extract more from those sales. 

Pros and joes

Taking things a speculative step forward, I can easily imagine the company might choose to keep the highest-end devices out of the subscription loop, making these available for sale only. Apple knows its most affluent customers may be more accepting of a higher price in exchange for a truly cutting-edge product. 

That balance of high-end retail sales and subscription-income, bolstered by all the other plans Apple is putting in place to survive the high-tax transformation of the US economy may help it build a good and viable business in times like these. A different business, but a business all the same

Will Apple do it? Will Apple choose to offer up its products on a rental basis? 

Given sales cadence is going to fall anyway and, logically, product costs will increase, the company might see the plan as a way to pass the poison pill of some of these price increases onto consumers in as easy-to-stomach a remedy as possible. It’s not ideal, of course, but right now company management will be focused on finding the least worse options to help support the future of its business. That’s the context in which such a plan makes sense.

You can follow me on social media! Join me on BlueSky,  LinkedIn, and Mastodon.

Source:: Computer World

By Adarsh Verma If your kid is obsessed with Roblox, you probably have a headache from hearing, “Can I…
The post How to Stop Your Kids (or Yourself) From Spending All Your Money on Robux appeared first on Fossbytes.

Source:: Fossbytes

By Hisan Kidwai Buying a smartphone on a budget can be pretty confusing, especially since brands tend to focus…
The post Best Phones Under INR 20000 in India (April 2025) appeared first on Fossbytes.

Source:: Fossbytes

By Nadeem Sarwar The folks over at buzzy Chinese AI lab, DeepSeek, are working on a new series of AI models called DeepSeek-GRM that employ a novel self-improving approach.

Source:: Digital Trends

By Thomas Macaulay Five fast-growing Nordic scaleups have reached the finals of TECH5 — the “Champions League of Technology.” They join a standout quintet from Benelux, unveiled last week, in the race to be crowned Europe’s hottest scaleup. The Nordics — comprising Denmark, Finland, Iceland, Norway, and Sweden — boast just 27 million people but punch well above their weight in innovation. The region has Europe’s highest density of unicorns — privately held companies valued at over $1bn. Despite accounting for only 4% of Europe’s population, the Nordics hosted 17% of the continent’s unicorns between 2013 and 2023. Sweden leads the pack with the…This story continues at The Next Web

Source:: The Next Web

By Nick Godt Rivian’s brand new ‘hands-off’ driving feature will work with unmapped roads on Gen2 vehicles later this year.

Source:: Digital Trends

By Nadeem Sarwar Deep Research and AI overview podcasts have been my two favorite Google Gemini features so far. Microsoft has finally implemented them on its Copilot platform.

Source:: Digital Trends