For a long time now, I’ve had various cron jobs running to fetch various web resources with which I’d build out parts of my own site, or supply myself with custom RSS feeds after a pass through xsltproc.
This mostly worked OK, but there were a few things wrong with it:
After a while, the problems added to enough of an annoyance that I decided to do something about it, so a couple months ago I started pfetch.
pfetch is a simple twisted app that does scheduled parallel http requests and optionally runs scripts after successful execution.
Given a list of URLs each with a destination, frequency, and optional (with arguments) to run after each successful (200) response, each URL will begin a fetch cycle starting at a random offset from the start time and loop on the defined interval.