Menu
Home
Forums
New posts
Search forums
What's new
New posts
New media
New media comments
New profile posts
Latest activity
Media
New media
New comments
Search media
Members
Current visitors
New profile posts
Search profile posts
Account Upgrades
Advertise
Marketplace
Money
PerfectMoney
Log in
Register
What's new
Search
Search
Search titles only
By:
New posts
Search forums
Menu
Log in
Register
Home
Forums
Programming & Web Design
Scripting
PHP & Perl
CHALLANGE: Weird issue scraping a retailer w/ curl PHP ...
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Reply to thread
Message
<blockquote data-quote="Hazle" data-source="post: 20708" data-attributes="member: 5422"><p>Setup...</p><p>I have been trying to scrape pricing information from retailer "lowes" hardware store but they have some kind of redirect going on that appears to attempt to set your local store zip code by the IP address you are coming from. The page does not show the price of the product until it sets the cookie.</p><p></p><p>How to duplicate...</p><p>1) Disable cookies in Chrome</p><p>2) Goto lowes website</p><p>3) Watch it try and run some auto zip code cookie setter code and redirect indefinitely (since cookies are disabled)</p><p>4) You will see the price on the page for the product is missing until you allow it to set this cookie</p><p></p><p>Problem Trying To Solve...</p><p>How to set the correct cookie in php through a proxy so that the price shows up in the html response? Then delete the cookie when done. Any ideas? I have never seen this before and I have been scraping webpages for years... I am not sure what cookie it is and/or if there is a script you can POST to set it.</p></blockquote><p></p>
[QUOTE="Hazle, post: 20708, member: 5422"] Setup... I have been trying to scrape pricing information from retailer "lowes" hardware store but they have some kind of redirect going on that appears to attempt to set your local store zip code by the IP address you are coming from. The page does not show the price of the product until it sets the cookie. How to duplicate... 1) Disable cookies in Chrome 2) Goto lowes website 3) Watch it try and run some auto zip code cookie setter code and redirect indefinitely (since cookies are disabled) 4) You will see the price on the page for the product is missing until you allow it to set this cookie Problem Trying To Solve... How to set the correct cookie in php through a proxy so that the price shows up in the html response? Then delete the cookie when done. Any ideas? I have never seen this before and I have been scraping webpages for years... I am not sure what cookie it is and/or if there is a script you can POST to set it. [/QUOTE]
Name
Verification
Post reply
Home
Forums
Programming & Web Design
Scripting
PHP & Perl
CHALLANGE: Weird issue scraping a retailer w/ curl PHP ...
Top
