Pro@mander.xyz to Technology@programming.devEnglish · edit-21 年前Cloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocksblog.cloudflare.comexternal-linkmessage-square7linkfedilinkarrow-up153arrow-down10file-textcross-posted to: cybersecurity@sh.itjust.worksblueteamsec@infosec.pubtechnology@lemmy.ziphackernews@lemmy.bestiver.se
arrow-up153arrow-down1external-linkCloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocksblog.cloudflare.comPro@mander.xyz to Technology@programming.devEnglish · edit-21 年前message-square7linkfedilinkfile-textcross-posted to: cybersecurity@sh.itjust.worksblueteamsec@infosec.pubtechnology@lemmy.ziphackernews@lemmy.bestiver.se
minus-squareKay Ohtie@pawb.sociallinkfedilinkEnglisharrow-up2·1 年前Perplexity’s firing back assumes website owners distinguish between automated scraping and on-demand scraping. I don’t think most people make that distinction. And that falls in line perfectly with the typical “assumption of access” all of these “AI” companies make.
Perplexity’s firing back assumes website owners distinguish between automated scraping and on-demand scraping.
I don’t think most people make that distinction.
And that falls in line perfectly with the typical “assumption of access” all of these “AI” companies make.