Hazle
New member
- Joined
- Jul 27, 2024
- Messages
- 3
- Points
- 1
Setup...
I have been trying to scrape pricing information from retailer "lowes" hardware store but they have some kind of redirect going on that appears to attempt to set your local store zip code by the IP address you are coming from. The page does not show the price of the product until it sets the cookie.
How to duplicate...
1) Disable cookies in Chrome
2) Goto lowes website
3) Watch it try and run some auto zip code cookie setter code and redirect indefinitely (since cookies are disabled)
4) You will see the price on the page for the product is missing until you allow it to set this cookie
Problem Trying To Solve...
How to set the correct cookie in php through a proxy so that the price shows up in the html response? Then delete the cookie when done. Any ideas? I have never seen this before and I have been scraping webpages for years... I am not sure what cookie it is and/or if there is a script you can POST to set it.
I have been trying to scrape pricing information from retailer "lowes" hardware store but they have some kind of redirect going on that appears to attempt to set your local store zip code by the IP address you are coming from. The page does not show the price of the product until it sets the cookie.
How to duplicate...
1) Disable cookies in Chrome
2) Goto lowes website
3) Watch it try and run some auto zip code cookie setter code and redirect indefinitely (since cookies are disabled)
4) You will see the price on the page for the product is missing until you allow it to set this cookie
Problem Trying To Solve...
How to set the correct cookie in php through a proxy so that the price shows up in the html response? Then delete the cookie when done. Any ideas? I have never seen this before and I have been scraping webpages for years... I am not sure what cookie it is and/or if there is a script you can POST to set it.
