Author:Aurora / iDIYuan

Excutive Editor:Louis Lu / iDIYuan

Singapore’s Marina Bay Sands became the center of Asia’s latest AI frenzy as SuperAI drew tens of thousands of founders, investors and technologists from more than 150 countries, alongside more than 1,500 technology companies.

The event featured exchanges of ideas among figures including Balaji Srinivasan, MIT professor Max Tegmark, and renowned analyst Benedict Evans, as well as the 36-hour NEXT Hackathon and the Genesis startup competition, which offered up to US$2.3 million in funding. Across the venue, conversations centered on large language models, AI agents and the race to turn generative AI into products that can work reliably in the real world.

Yet a reality rarely brought into the spotlight is becoming a new bottleneck for the industry: AI models are getting smarter at a rapid pace. Their ability to access accurate, real-time information from the open internet, however, is not improving nearly as quickly — and in some cases, it is becoming harder.

Agents that perform well in laboratory environments can see their performance deteriorate once they are deployed on the live web at scale. The problem often does not lie in the models themselves, but in the increasingly sophisticated defenses being deployed across the web — from anti-scraping mechanisms and dynamic verification to content designed specifically to detect, restrict or mislead automated agents. In the process, the public web is gradually shifting from an open source of information into an increasingly adversarial environment for AI.

At this year’s conference, Ariel Shulman, chief product officer of Bright Data, a public-data infrastructure provider, put it bluntly in his keynote: “The web has learned to lie.”

While the industry’s attention remains focused on model parameters, compute scale, and reasoning capabilities, Shulman believes a more fundamental problem is being overlooked: giving AI agents reliable, real-time access to the internet — and ensuring the data they retrieve is accurate, trustworthy and continuously available.

Bright Data CPO Ariel Shulman speaking at SuperAI. Source: SuperAI

Speaking with iDIYuan after the conference, Shulman said the problem goes beyond websites learning how to block or mislead AI agents. The useful life of online data is also getting shorter, in some cases shrinking to just hours or even minutes.

At the same time, tighter compliance requirements and growing legal risks are raising the barriers for companies seeking access to public data. Without authentic, stable, and compliant data pipelines, even frontier models can be constrained by the weakest layer in the stack: their inputs.

That is turning access to trustworthy, real-time data into a new competitive battleground for the AI industry.

When the Internet Learns to Lie on Purpose,What Will Happen to AI 

In Ridley Scott’s 1982 science-fiction classic Blade Runner, the Tyrell Corporation creates replicants so lifelike that appearance alone is no longer enough to distinguish them from humans. Identifying them requires the Voight-Kampff test, which looks for subtle differences in emotional response and behavior.

Film: Blade Runner. Source: thegallery.gr

More than forty years later, a similar scenario is emerging on the open web.Major platforms are increasingly acting as testers themselves. Instead of simply blocking IP addresses, they are beginning to actively identify whether the visitor behind a request is a human user or an AI agent.

Once identified as an agent, however, the response may not be a simple access-denied page. The agent can instead be routed into a deliberately distorted version of the web — one where prices are synthetic, inventory is inaccurate, page content has been altered and search results are selectively misleading. From the agent’s perspective, the request appears to have succeeded. In reality, it has been fed corrupted inputs. That shift prompted a stark assessment from Ariel Shulman during his SuperAI presentation:

“The web has learned to lie.”

As traffic generated by LLM training and AI agents grows, Shulman argues that the relationship between the internet and AI has evolved from the old contest of “blocking versus bypassing” into a much more complex offensive-and-defensive battle. In the past, websites mainly relied on CAPTCHAs, rate limits, and IP blocking. Today, more advanced defenses are emerging, including honeypots designed to trap automated traffic and deception systems built specifically to identify, misdirect or manipulate AI agents.

Honeypot system. Source: Bright Data

For human users, these tactics may be largely invisible. But for agents that rely on web data for reasoning and decision-making, once a bad input enters the workflow, every subsequent analysis, judgment, and even action may be built on a false foundation.

Shulman described such AI-targeted “data poisoning” as a growing operational risk. He gave the example of a non-technical CEO who used Vibe Coding to develop an application that pulled hotel-pricing data from the web. The agent returned a rate of $399 and warned that inventory was running low. The actual page, however, showed rooms available for $199, with plenty of inventory remaining. The application had not failed to retrieve data. It had successfully retrieved the wrong data.

At the engineering level, this targeted deception of AI agents leads to an extremely dangerous consequence: garbage in, garbage out. Once an LLM receives precisely poisoned data, its subsequent reasoning, analysis, and even business decisions are all built on the wrong foundation.

Shulman said the bigger problem is that agents respond to deception very differently from humans. People bring intuition, context and accumulated experience to what they see online. If a hotel rate suddenly doubles within minutes, or an airline site shows inventory that makes little sense, a human user is likely to become suspicious and check another source.

AI agents, however, lack this intuitive brake. Under today’s largely deterministic workflows, an agent may simply follow its instructions: log the wrong price, propagate it across thousands of database records and, in the worst case, trigger downstream systems that send incorrect offers or pricing information to real customers at scale.

Once agents begin to participate deeply in enterprise operations, data quality stops being just an engineering problem and becomes a business risk. For AI, the greatest threat is no longer being unable to obtain data, but obtaining data that looks authentic while actually being wrong.

Breaking Through the Challenges of Deception, Freshness, and Compliance: Three Questions AI Data Infrastructure Must Answer

Founded in 2014 and headquartered in Israel, Bright Data is a major provider of public-web data infrastructure. Its role in the AI stack is less about training models than about keeping them connected to fresh, real-world information at scale.

Bright Data booth at SuperAI. Source: Photo by iDIYuan on site

Rather than a traditional web-scraping company, Bright Data increasingly positions itself as part of the data infrastructure layer for AI. Its technology stack helps LLMs, AI agents, and enterprise applications access fresh public-web data at scale, while navigating reliability and compliance requirements. As AI becomes increasingly multimodal, that data layer is also expanding beyond text to include images, video and other forms of web content.

As part of the data pipeline behind the AI ecosystem, Bright Data currently supports more than 100 million AI agent interactions every day and provides public-data services to more than 20,000 organizations worldwide, including Fortune 500 companies, leading universities, and fast-growing technology companies. According to the Bright Data Blog, the company also says 14 of the world’s 20 leading LLM labs use its data infrastructure.

As the web becomes more hostile to AI traffic, Bright Data’s approach is not to simply overpower those defenses, but to make automated access look more like ordinary human browsing. Shulman told iDIYuan that the company runs real browser environments in the cloud, replicating normal browsing, clicking and interaction patterns so that AI systems receive the same page responses and data a human user would see.

“If you have the technology to look like a real person, you will get real person information,” he said.

Underpinning that system is a network of more than 150 million compliant residential IPs across 195 countries and regions. With this infrastructure, Bright Data says it can collect more than 2.5 PB of data from the public internet every day while maintaining its compliance requirements, providing continuously updated data for LLM training, AI agents, and enterprise applications.

IP addresses across 195 countries. Source: Bright Data website

If authenticity determines whether AI can trust the data it sees, freshness determines whether that data is still useful. In Shulman’s view, “Data quality and data freshness are two sides of the same coin.” Data that is no longer fresh, he explained, is no longer quality data.

Consider an e-commerce customer-service bot or a logistics agent. If it is working with last month’s prices, inventory, or delivery information, the information may be accurate but still useless for decisions being made today. Shulman’s point is that access alone is not enough. For data to remain valuable to AI systems, it also has to stay current.

Bright Data’s public-data decay map illustrates just how quickly that window can close. Fast-moving information such as social-media trends, e-commerce prices and product inventory can become stale within hours, and often within a day. After that, its usefulness to an AI system can deteriorate quickly.

Shelf life of different types of data. Source: Bright Data

Shulman gave the example of an AI agent that helps users compare prices globally. If it relies on prices and inventory data from 12 hours earlier, it may ultimately recommend a product that has already increased in price or even sold out. When the user reaches the checkout page, the real price may be completely different from the result provided by the agent. For the user, this is not only a poor experience; it directly weakens trust in the AI system.

As agents move deeper into search, procurement, customer service, and operations, the data challenge is shifting from access to real-time access. For companies deploying AI in the real world, securing a continuous supply of trustworthy, up-to-date data may prove harder than simply scaling model parameters.

If authenticity and freshness determine whether data is useful, compliance determines whether a data business can operate over the long term. As demand for data from LLMs and AI agents continues to grow, the debate over public data collection is gradually shifting from a technical issue to a legal one.

Disputes over access rights, platform controls and the ownership of public data have intensified in recent years, making compliance an increasingly important competitive advantage for AI infrastructure providers.

On compliance, Shulman told iDIYuan Bright Data follows a simple rule: it only collects data that is publicly accessible on the web.

In practice, that means information that can be viewed without logging into an account or paying for a subscription. “If you can see that, that’s probably good,” Shulman said. Data behind personal accounts, internal enterprise systems, bank accounts or other non-public sources falls outside that boundary. That approach has also been tested in court. Shulman said Bright Data has faced lawsuits in the U.S. from Meta and X over its collection of publicly available web data. The company’s position, he said, was that it accessed only public information, not private data — and that the rulings in those cases supported that distinction.

Judgment related to Meta’s case against Bright Data. Source: courtlistener.com

In January 2024, the U.S. District Court for the Northern District of California granted summary judgment to Bright Data on Meta’s breach-of-contract claim, finding that the company had scraped only publicly available data while logged out of Meta’s services and, on those facts, had not violated Meta’s terms. The ruling was specific to the contractual claims and facts before the court; it did not declare public-web scraping universally lawful.

For Bright Data, however, it was an important validation of its approach to collecting public data. “That is really important,” Shulman said.

More broadly, the case shows that the legal boundaries around public-web data can depend heavily on how the data is collected, what information is accessed and the contractual terms involved. As AI companies consume more public data, those distinctions are becoming increasingly important.

MCP Is Only the Starting Point: Data Services Compete on Scalability and Engineering Efficiency

After solving the problems of data authenticity, freshness, and compliance, the next challenge for AI engineers is how to enable agents to connect efficiently to external tools and data services. Over the past year, Anthropic’s Model Context Protocol, or MCP, has rapidly become a major focus in the industry. As more model providers and development platforms add support, MCP is becoming an important connection layer in the agent ecosystem.

Shulman told iDIYuan that Bright Data released its MCP server about a year ago. In his description, an MCP server is essentially the connection layer between Bright Data’s services and AI: it gives an agent a list of the available Bright Data tools, tells it what those tools can do, and shows it how to use them.

He gave a simple example. If an AI agent says it needs to search Google, the MCP server can tell it that Bright Data has a SERP search tool and provide the syntax and instructions needed to call it. The agent can then use that information to complete the task automatically.

In Ariel’s view, MCP’s greatest value is that it makes it much easier for an agent to discover and use external tools. It is especially useful at the prototype and proof-of-concept stages, when developers are still working out how an application should obtain data and which tools it needs.

Relationship between Bright Data’s data infrastructure and the layers above and below it. Source: Bright Data

However, Shulman also pointed out the limits of MCP at scale. If an agent needs to find the price of a single product, MCP can tell it how to do so. But if that task expands to one million products, he said, using MCP becomes inefficient because “there is a lot of overhead.” In large-scale production workloads, those additional layers can translate into unnecessary latency and cost.

Shulman described what he sees as the most practical path:

1. Start with MCP to understand how Bright Data’s tools work and help the AI agent discover a workable way to complete the task;

2. Once the workflow is clear and the application needs to scale, switch to more direct methods such as direct APIs, and eventually stop using MCP for the production workload.

In other words, MCP helps developers “find the road,” while direct APIs turn that road into a highway. As agents move from demos into production, the industry’s focus is also shifting from “can it be achieved ” to “can it run at scale ” Engineering efficiency will ultimately become a key factor in determining whether a product succeeds or fails.

This pursuit of engineering efficiency and scalability is reflected not only in external data services, but also in how this data-infrastructure company organizes itself. As a data-infrastructure provider, Bright Data must process massive volumes of data requests every day while continuously responding to changes in the internet environment. That business model means that its competitiveness comes not only from the technology itself, but also from its ability to iterate continuously and deliver quickly.

Shulman told iDIYuan that Bright Data, with a lean team of roughly 500 employees, generates hundreds of millions of dollars in revenue and has remained profitable since 2017. In his view, corporate development does not depend entirely on capital investment; what matters more is whether a company can build an efficient engineering system.

“We have a very unique way of working,” Shulman said. For example, as long as engineers meet the required standards, they can push code directly into the production environment. After a new feature goes live, the system first exposes it to roughly 1% of users and continuously monitors its performance. If it runs stably, the rollout is gradually expanded to 5%, then 10%, and eventually to all users.

This continuous-delivery mechanism enables Bright Data to maintain an extremely high iteration frequency. Shulman revealed that the company averages around 60 software releases per day. In his view, data infrastructure is fundamentally a long-term engineering competition. The real advantage comes not from token volume or access to capital, but from building systems that can iterate quickly, improve continuously and respond to change.

From “Bright Data” to Its China Brand: Bright Data’s Localization Strategy in China

No matter how technology architectures evolve, infrastructure ultimately has to prove its value in specific markets and industrial ecosystems. For the data-services industry, this means not only exporting technology, but also sustaining long-term localized operations.

At a time when some multinational companies are scaling back their operations in China, Bright Data has chosen another path: continuing to invest in the Chinese market under its localized brand and seeking to further combine its global data-infrastructure capabilities with the needs of China’s domestic AI industry.

Bright Data China website: bright.cn. Source: Bright Data

In the interview, Shulman placed particular emphasis on Bright Data’s position in China. “We have a very strong presence in China,” he said.

To support localized operations, Bright Data has established offices in Beijing, Shanghai, and Shenzhen, with R&D, technical-support, and customer-service teams. Public recruitment information shows that its China team covers multiple roles including solutions architects, customer success managers, and senior sales managers, spanning the full chain from technical consulting to long-term customer operations.

For domestic AI companies, this means gaining not only access to a global data-service capability, but also a service system capable of providing local support. Whether the issue involves complex data-collection requirements, network-scheduling problems, or technical failures in production environments, customers can obtain more timely responses and support through local teams.

Bright Data is also continuing to expand its local partnership ecosystem in China. In April 2026, it partnered with Data Trading Network to bring its public-data capabilities to the domestic market. The datasets can be updated on daily, weekly and monthly cycles, helping companies lower the cost of sourcing and maintaining data.

The company has also teamed up with DuoPlus Cloud Phone to explore how data services and cloud-device infrastructure can support cross-border expansion. On the developer side, Bright Data has co-hosted technical events with communities including DataFun, where its China engineering team has shared practical experience with MCP, agent tool calling and data-infrastructure development.

Chinese edition of a Bright Data report. Source: Bright Data

Its local partnerships have also helped Bright Data move beyond simple market expansion and gradually integrate into the local AI ecosystem. In Shulman’s view, Chinese companies targeting global markets face many of the same needs as their peers elsewhere, including access to global data for model training. But once they focus on the domestic market, localization becomes far more important.

Shulman revealed that for data-collection needs involving local retailers in different countries, the company’s AI agents can automatically generate customized scraping solutions, extract multilingual and multimodal structured content from massive amounts of public data, and directly serve localized business decisions such as market analysis and competitive monitoring.

Over the past two years, Bright Data’s AI customer base in Asia-Pacific has doubled annually and now accounts for nearly one-third of the company’s global AI customers. The region represents more than 20% of global AI spending, while China and India together make up roughly 60% of Bright Data’s AI customers in Asia-Pacific. This regional presence allows Bright Data to provide more targeted local data services to companies across markets including Japan, Southeast Asia, India and China.

The Next Generation of Data-Infrastructure Competition Is Already Quietly Unfolding

At the end of the interview, Shulman turned to what he sees as the next phase of the AI industry. “I think one day every company will have its own AI agents,” he said.

A human analyst might have needed to open only five or ten links to complete a research task. An AI agent, by contrast, could scan dozens or even hundreds of websites to produce a more comprehensive answer. That means “more data to handle,” Shulman said — and potentially a need to rethink the infrastructure that supports it.

Ariel Shulman discussing Bright Data’s vision. Source: SuperAI

While the industry remains focused on model parameters, reasoning capabilities, and investment in compute, another, more fundamental layer of competition is already quietly unfolding. As more agents connect to the internet and complete tasks autonomously, data is no longer just fuel for training. It is becoming a real-time input that keeps AI systems running. Authenticity determines whether an agent can form an accurate understanding; freshness determines whether its decisions remain effective; compliance determines whether this capability can exist over the long term; and the engineering system determines whether the entire system can operate at scale. As Shulman put it, once the internet learns to “lie,” trustworthy data will become new infrastructure for the agent era. The next phase of the AI race may be less about pushing model capability alone and more about building the data systems that allow agents to operate reliably in the real world. Algorithms shape how AI reasons; data infrastructure shapes what it can see, understand and act on. The model may be the brain and the agent the executor, but neither is particularly useful without a reliable connection to the outside world. As AI agents increasingly browse the web, make decisions and act on behalf of users, that underlying data layer is likely to become only more important.

来源：iDIYuan