Pro@mander.xyz to

Technology@programming.devEnglish · 1 年前

Cloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocks

blog.cloudflare.com

7

53

Cloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocks

blog.cloudflare.com

Pro@mander.xyz to

Technology@programming.devEnglish · 1 年前

7

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

blog.cloudflare.com

Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.

Perplexity blog.

Chat

beeng@discuss.tchncs.de
link
fedilink
English
arrow-up
3·
1 年前
Perplexity fired back in their blog.

Pretty tasty.

Technology@programming.dev

Technology@programming.dev

You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !Technology@programming.dev

Share interesting Technology news and links.

Rules:

No paywalled sites at all.
News articles has to be recent, not older than 2 weeks (14 days).
No external video links, only native(.mp4,…etc) links under 5 mins.
Post only direct links.

To encourage more original sources and keep this space commercial free as much as I could, the following websites are Blacklisted:

Al Jazeera;
NBC;
CNBC;
Substack;
Tom’s Hardware;
ZDNet;
TechSpot;
Ars Technica;
Vox Media outlets(including Axios, due to new changes related to trackers on their website);
Engadget;
TechCrunch;
Gizmodo;
Futurism;
PCWorld;
ComputerWorld;
Mashable;
Hackaday;
WCCFTECH;
Neowin;
Jacobin;
Yahoo;
Freethink;
Big Think;
Newsweek.

More sites will be added to the blacklist as needed.

Encouraged:

Archive links in the body of the post.
Linking to the direct source, instead of linking to an article talking about the source.

Misc:

Relevant Lemmy Communities:

Visibility: Public

This community can be federated to other instances and be posted/commented in by their users.

1 user / day
1 user / week
130 users / month
352 users / 6 months
2 local subscribers
1.03K subscribers
1.27K Posts
2.62K Comments
Modlog