Dynamic Web Scraper – A Python Tool for E‑Commerce Data
A lightweight Python scraper that uses Selenium and BeautifulSoup to navigate an e‑commerce site, collect product listings, and pull detailed data from each page.
Why I Built It
I wanted a quick way to gather product data from an online store. The site was heavy on JavaScript, so a simple requests‑based approach wouldn't cut it.
The Journey
First, I set up a virtual environment and installed Selenium, BeautifulSoup, and WebDriver‑Manager. Selenium handled the dynamic content, while BeautifulSoup parsed the HTML.
How It Works
Navigator.py starts by loading a saved homepage snapshot. It pulls category links, visits each one, and grabs product listings. Then it dives into each product page for extra details.
Testing Mode
By default, the scripts limit scrolling and product counts to keep runs short. Removing the break statements lets you scrape the whole site.
What It Gives You
After a run, you get a data.json file that contains titles, prices, URLs, and more. The tool is ready for any e‑commerce site that follows similar HTML patterns.