Pro@mander.xyz to Technology@programming.devEnglish · edit-21 year agoCloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocksblog.cloudflare.comexternal-linkmessage-square7linkfedilinkarrow-up153arrow-down10file-textcross-posted to: cybersecurity@sh.itjust.worksblueteamsec@infosec.pubtechnology@lemmy.ziphackernews@lemmy.bestiver.se
arrow-up153arrow-down1external-linkCloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocksblog.cloudflare.comPro@mander.xyz to Technology@programming.devEnglish · edit-21 year agomessage-square7linkfedilinkfile-textcross-posted to: cybersecurity@sh.itjust.worksblueteamsec@infosec.pubtechnology@lemmy.ziphackernews@lemmy.bestiver.se
minus-squareTomasEkeli@programming.devlinkfedilinkEnglisharrow-up5·1 year agoA sure sign that they are a nefarious company.
A sure sign that they are a nefarious company.