Exploring Twitter’s Usage Limits for Data Scrapers

Even as we head into a long weekend, it is never a dull day at Elon Musk’s ‘Twitter 2.0’.

Today, Twitter has taken the extreme step of restricting users from viewing tweets. It is an effort to address what Elon describes as ‘extreme levels of data scraping.’

According to Twitter, to address data scraping concerns, Twitter has restricted verified and unverified usage at different thresholds. Musk says it is a necessary step to stop data scrapers.

Twitter has since then increased these thresholds to:

• Verified accounts – 10,000 posts
• Unverified accounts – 1,000 posts
• New unverified accounts – 500 posts

It has increased these limits twice within the five hours since Elon’s initial tweet. It does suggest that this could be fully resolved very shortly.

But still, it is an extreme measure to combat misuse. Moreover, there might be no definitive way to stamp out tweet scraping entirely.

Moreover, the action by Twitter against data scrapers started two days ago, when Twitter began restricting non-logged-in users from viewing tweets.

It is a significant move because around 40% of tweet viewers need to log in to do so. It means that Twitter is immediately addressable to the audience severely impacted by this change.

The gamble of Twitter taking this action is to make more people log in, where it can show them more targeted ads. But the risk is that they don’t, and Twitter loses reach and relevance.

Twitter CTO Elon Musk later explained that this was a temporary measure to address misuse:-

A key concern here is that many organizations are now trying to get into the generative AI game; to do that, they need conversational data.

Twitter’s ‘open garden’ approach is designed to facilitate broader discussion. It makes it a prime target for scraping to gather such info.

Other platforms are already taking measures to address this long ago. For example, Facebook restricts the amount of information non-users can access. Even Instagram and LinkedIn are doing the same.

These organizations recognize the value of their proprietary data, but the approach Twitter has always been to host a broader global discussion.

It is the reason why tweets have remained publicly accessible to a large degree. But now, it has become a problem that Twitter 2.0 is working to address in a way that could severely impact usage.

Add to this the fact that Twitter has also significantly upped the price of its API access. It has also been designed to stop the misuse of its data.

Note: Reddit has also increased the price of its API usage.

It leads to the point of no compromise for the platform because while higher API access costs have priced many providers out, the majority of tweet data remains publicly available.

It means that several of these developers will simply revert to the scraping route instead unless they cannot anymore because it is the loophole that Twitter is looking to close with this change.

So what will the impact of that be?

Around 40% of Twitter users in Europe access tweets without logging in. It is not necessarily reflective of all regions.

But it suggests that a significant portion of the platform’s 252 million daily active users must log into a profile.

Those users will now be far more limited in what they can view. Meanwhile, regular users will also be restricted from using Twitter at a certain point unless Twitter comes up with a better solution.

It could be to pinpoint accounts that are being used by scrapers, to stamp them out, and maybe it is a means to stop the practice.

But the problem will likely remain, as scrapers will find new ways to access and keep taking the data of Twitter.

As such, there is no real solution to such a problem. Hence it will likely take Twitter some time to address all the key entry points and overuse issues before lifting limits on content on Twitter.

It will eventually see Twitter raising its walls and restricting tweet access permanently. It could also impact its position as a key news-breaking and discussion platform.

But at the same time, Twitter is caught in the middle, with scrapers overloading its systems to fuel large language models. Some people think it is a legitimate concern, but the solutions for Twitter don’t look very appealing.

You can click here to learn more about the updates on Twitter and other social media platforms.