WordPress API, Sitemap, or Homepage: How Our Scraper Finds Every Recent Post

A lot of link-extraction tools only work on one type of website. Our Website Scraper doesn't, because it tries three different methods, in order, until one works.

First, it checks whether the site runs on WordPress. If it does, it talks directly to WordPress's own data feed, which is the fastest and most accurate option - it returns exact publish dates, so we know precisely which posts fall inside your chosen time window.

If that's not available, it falls back to the site's sitemap.xml file, which almost every modern website has. This still gives us reliable publish or last-modified dates to filter by.

If neither of those exist, it falls back to simply crawling the homepage and following internal links - not perfect for date filtering, but it means the tool still works on virtually any site you throw at it.

The result: you paste in a URL, and the tool quietly figures out the best way to get you accurate, recent data - no configuration required. Curious what else is automated end-to-end? Check out the rest of the blog for more.