php MySQL replica website for web scraping

Secure sites can be difficult to scrape if you don't have your own access.

A customer may need code written to automate data entry/lookup but if you are developing the code from another country or if there is a risk that they could have their access removed due to any errors with your code then it is a good idea to replicate the site that they need to work with.

This is not a 'tutorial' - rather it's a demonstration of how you can set up your own web server and link it with some of your own dummy data, and eventually create a page that simulates the real site you want to scrape.

The purpose of this is to be able to write Python code, including Selenium, to read data from a 'lookup' csv, enter those values into a search, and gather ('scrape') the responses and save into a 'response' csv.

The following is a list of the steps you will see:

1. Set up remote 'fake' web server (Linux (Raspbery Pi))
2. Set up MySQL (Or 'MariaDB') as replica of real site (same columns etc)

-

Test in browser using ip address of the Pi (192.168.1.9)

Next :
Write Selenium / Python Code to interrogate the page as if it was the
real / target site.

! Excuse my php skills, it's been a while...

Visit redandgreen blog for more Tutorials
=========================================

Subscribe to the YouTube Channel
=================================

Follow on Twitter - to get notified of new videos
=================================================

Buy Dr Pi a coffee (or Tea)

Proxies
=================================================
If you need a good, easy to use proxy, I was recommended this one, and having used ScraperAPI for a while I can vouch for them. If you were going to sign up anyway, then maybe you would be kind enough to use the link and the coupon code below?

You can also do a full working trial first as well, (unlike some other companies). The trial doesn't ask for any payment details either so all good! 👍

◼️ Coupon Code: DRPI10
(You can also get started with 1000 free API calls. No credit card required.)

Thumbs up yeah? (cos Algos..)

#webscraping #tutorials #php #mysql