Pro@mander.xyz to Technology@programming.devEnglish · edit-21 year agoCloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocksblog.cloudflare.comexternal-linkmessage-square7linkfedilinkarrow-up153arrow-down10file-textcross-posted to: cybersecurity@sh.itjust.worksblueteamsec@infosec.pubtechnology@lemmy.ziphackernews@lemmy.bestiver.se
arrow-up153arrow-down1external-linkCloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocksblog.cloudflare.comPro@mander.xyz to Technology@programming.devEnglish · edit-21 year agomessage-square7linkfedilinkfile-textcross-posted to: cybersecurity@sh.itjust.worksblueteamsec@infosec.pubtechnology@lemmy.ziphackernews@lemmy.bestiver.se
minus-squareCarbonatedPastaSauce@lemmy.worldlinkfedilinkEnglisharrow-up2·1 year agoThe article explains how they tested for that, and as far as they could tell OpenAI is respecting the rules.
The article explains how they tested for that, and as far as they could tell OpenAI is respecting the rules.