For years I’ve on and off looked for web archiving software that can capture most sites, including ones that are “complex” with lots of AJAX and require logins like Reddit. Which ones have worked best for you?
Ideally I want one that can be started up programatically or via command line, an opens a chromium instance (or any browser), and captures everything shown on the page. I could also open the instance myself and log into sites and install addons like UBlock Origin. (btw, archiveweb.page must be started manually).
https://www.httrack.com/ may not be exactly what you want but fits much of what you describe.
I haven’t tried to, but there does seem to be a way to capture “logged in” websites:
https://stackoverflow.com/questions/20362821/httrack-possible-using-cookies#58354077The short version: no single tool nails this, but the closest combo right now is ArchiveBox + browsertrix-crawler (from Webrecorder), with SingleFile for spot-checks.
Why each:
-
ArchiveBox is a good index-layer: it can ingest a list of URLs, but its default fetchers (wget, readability) fall over on SPAs and anything behind auth. Its value is scheduling + a browsable archive UI, not capture fidelity.
-
browsertrix-crawler is the only one that treats “complex site” as the normal case. It runs a real browser, crawls to WACZ, and it handles login properly: you pass --userAgentSuffix for politeness, but more importantly you can seed it with a logged-in browser profile (browsertrix supports CDP profile import), so session cookies/localStorage are captured too. If you can log in once in a Chromium profile, browsertrix can crawl as that user.
-
SingleFile / SingleFile CLI isn’t a crawler, but for pages that need JS to render before capture it beats everything else one page at a time. Good fallback for the 10 URLs a crawler misses.
Things I learned the hard way:
- wget2/httrack-style crawling on a React/Vue site produces archives of empty shells — the HTML is a hydration stub. Any non-browser crawler fails here by design.
- Sites behind auth with CSRF + rotating session tokens will log you out mid-crawl. Best practical fix: export cookies right after login (storage state), set a short crawl window, and re-auth between runs rather than fighting keep-alive.
- For paywalled content, respect the ToS — capture only what you legitimately have access to, and rate-limit hard. A real-browser crawler with default concurrency looks like a bot attack and will get the account banned.
If the goal is “archive my own accounts/reading list”, browsertrix-crawler with a logged-in profile + ArchiveBox as the front end is the setup I’d build today.
-
