Automation & Web Scraping blog

Article

Resume a catalogue import after a failed page

A nightly catalogue job can survive a timeout without losing products. The design starts with how the job identifies a product and records its progress.

Choose what makes a row unique

Suppose a supplier offers a catalogue export in pages of 100 products. Use the supplier's product identifier as the key in your destination and store the observation time separately. A title change can then update the existing product. For a historical price table, the identity might include the export date as well. Decide this before running the import twice.

Save progress with the data

Give the run an identifier and record which export or snapshot it reads. For each page, validate the required fields, write its rows and advance the saved page marker in the same transaction. If the process stops before the commit, the next attempt repeats that page. A stable snapshot or cursor matters here because positions in a live, changing list can shift between requests.

Try a deliberately interrupted run

Prepare a three-page fixture and interrupt the import while it is writing page two. Resume it and compare the resulting keys and values with one uninterrupted run. Include a missing identifier and record where the rejected row came from. Only reconcile products absent from the supplier's export after every page has finished successfully. A failed final fetch should never make a whole page of products disappear.

All articles and case studies Describe your task