Need a method to load all reviews and export

[Jasen Bauch]

I'm currently facing a challenge with a website full of reviews that I want to scrape. By default, the site only loads 5 reviews at a time. When you click the "Load More Reviews" button, the page refreshes and reloads the initial 5 reviews, plus 5 new ones. This setup complicates the process of loading all 1800+ reviews because the webpage becomes unresponsive and eventually crashes. For example, if I'm at the 500th review and press the load more button, the page tries to reload all 500 existing reviews plus 5 new ones, which just bogs down the system. I suspect the design is intentionally made to deter scraping.

Has anyone encountered a similar issue or have ideas on how to work around this? Any advice on how to scrape the data without causing the page to become unresponsive would be greatly appreciated!

Thanks in advance!

 

[Reid Murphy]

Check page source to see whether the 1800 reviews are hidden or the page loads reviews from a database. If they are just hidden, there wouldn't be a problem in scraping them.

 

[Abe Gaylord]

Depending on the structure of the website, you can try and do this:

1. Download free tool, Macro Recorder by MurGee.com
2. Install
3. Record clicks, mouse movements, wait times etc.
4. Set the repeat to number of actions you need
5. Wait
6. Scrape
7. Done