Is anyone using their own Python webscraping to look for online deals? I was thinking of paying someone to build something for me. The paid versions I think are geared towards the developers making the most money possible, and not so much about you finding all the deals - plus they scrape so much so often, that they probably have to scrape less often than an individual would to keep them from getting banned. I would love to hear someone's thoughts or experiences. I'm also open to collaboration. 

I have some limited experience building things in Python. It's not too bad. However, not getting caught by the websites or security measures is.

 

I tried to write an auto-buy script for Amazon. I could have gotten it working except I tripped the account protection thing and also the website flagged my script as a bot (which it was). There are ways to get around the latter (and probably the first as well). A professional should definitely be able to help with both. Otherwise, it's going to take a lot of time, research, and trial and error. 

 

I also wrote an inventory checker for Walmart which was successful. 

I would love a better inventory checker than ********. Was that hard to figure out? I can easily explain the price scaper to a python guy. But i have no idea how ******** even searches for walmart inventory.

I am aware of the websites blocking people for pinging to often. I think there are some parameters online that tells you not to ping more than x number of minutes.. etc. 

All of these problems have been solved before.

10 minutes ago, BrickLegacy said:

All of these problems have been solved before.

Elaborate - 

I’ve tried a few times but ugh. So frustrating for me. Walmart is usually the one I’ve been trying. But I was trying to scrape LEGO Ideas and that was giving me a pain in the ass last week too.

14 hours ago, president89 said:

I would love a better inventory checker than ********. Was that hard to figure out? I can easily explain the price scaper to a python guy. But i have no idea how ******** even searches for walmart inventory.

I am aware of the websites blocking people for pinging to often. I think there are some parameters online that tells you not to ping more than x number of minutes.. etc. 

The Walmart inventory checker is not. You just use their publicly available, mobile API and have the program parse through the response and pick out out the price. 

 

For a scaper, the quantity of pings is important. However, even with a few, you are still likely to get caught by the websites. You have to try to mimic a user to try to fool the website into thinking you are just a normal user. This includes using fake headers on the requests sent to the website. This also includes using a proxy and rotating through a set of proxies. Those two things are the hard part. 

Edited May 9, 20205 yr by bigboy61

1 hour ago, bigboy61 said:

The Walmart inventory checker is not. You just use their publicly available, mobile API and have the program parse through the response and pick out out the price. 

 

For a scaper, the quantity of pings is important. However, even with a few, you are still likely to get caught by the websites. You have to try to mimic a user to try to fool the website into thinking you are just a normal user. This includes using fake headers on the requests sent to the website. This also includes using a proxy and rotating through a set of proxies. Those two things are the hard part. 

Do you know if have to rotate machines/operating systems/etc? Like I know eBay can get your windows fingerprint and even if you use a different IP there is a chance they will know it is you.

2 hours ago, president89 said:

Do you know if have to rotate machines/operating systems/etc? Like I know eBay can get your windows fingerprint and even if you use a different IP there is a chance they will know it is you.

You use proxies so that they can't identity you. You use headers so the requests look like they are coming from an actual user and not a bot. I couldn't get the proxies to work. 

I'd hire a professional to do this if you are serious.  You might want to consider a business plan first.

1 hour ago, BrickLegacy said:

I'd hire a professional to do this if you are serious.  You might want to consider a business plan first.

i have a business plan. 

Edited May 9, 20205 yr by president89