Real estate web scraping is a feasible option for keeping track of the real estate listing accessible for agents and sellers. Being in control of scraped real estate data from the real estate websites like Redfin.com can assist you in adjusting listing prices on your website or assist you in creating the database for the business. Here, in the tutorial, we would scrape Redfin data with Python, as well as show how to scrape property data. Particularly, we will display how to extract real estate listing using the zip code.
Follow these steps to extract Redfin
Create URL of search result pages from Redfin. For instance – https://www.redfin.com/zipcode/02126
Download the HTML of search result pages using the Python Requests.
Then parse this pages using LXML as LXML helps you navigate HTML Tree Structure with Xpaths.
Then save data in the CSV file.
We will scrape the given data fields from Redfin:
- Title
- Street’s Name
- City
- Zip Code
- State
- Pricing
- Facts & Features
- URL
- Real Estate Service Provider
Here is the screenshot of a few data fields that we would be scraping from Redfin
Mandatory Tools
Mandatory Tools
Just go through the guide for installing Python 3 with Linux at
https://docs.python-guide.org/starting/install3/linux/
The Mac users may follow the guide at – https://docs.python-guide.org/starting/install3/osx/
Packages
For the data scraping tutorial with Python 3 set up, we would require some packages to download as well as parse the HTML. Here are different package necessities:
PIP for installing the given packages in the Python at (https://pip.pypa.io/en/stable/installing/)
Python LXML to parse HTML Tree Structure through Xpaths (Discover how to install it there at http://lxml.de/installation.html)
Python Requests for making requests as well as downloading HTML content for different pages at ( https://docs.python-requests.org/en/latest/user/install/).
Coding
We require to initially construct search results pages URL. We’ll need to make the URL manually for scraping results from the page. For instance, this is for Boston at https://www.redfin.com/zipcode/02126.
How to Run a Redfin Scraper
Suppose a script is given the name, Redfin.py. Whenever you type this script name within the terminal or command prompt using a -h
usage: Redfin.py [-h] zipcode sort
positional arguments:
zipcode
sort
available sort orders are :
newest : Latest property details
cheapest : Properties with cheapest price
optional arguments:
-h, --help show this help message and exitThen you need to run a Redfin scraper with Python having arguments for sort and zip code. A sort argument is having the options ‘cheapest’ and ‘newest’ listings accessible. For instance, to get the listing of the latest properties for sale within Boston, we might run a script like:
It will make the CSV file named properties-02126.csv, which would be in a similar folder like a script. Just go through these sample data scraped from Redfin.com for above command.
def parse_hotels(driver):
""" To parse the web page using the BeautifulSoup
Args:
driver (Chromedriver): The driver instance where the hotel details are loaded
"""
# Getting the HTML page source
html_source = driver.page_source
# Creating the BeautifulSoup object with the html source
soup = BeautifulSoup(html_source,"html.parser")
# Finding all the Hotel Div's in the BeautifulSoup object
hotel_tags = soup.find_all("div",{"data-prwidget-name":"meta_hsx_responsive_listing"})
# Parsing the hotel details
for hotel in hotel_tags:
# condition to check if the hotel is sponsored, ignore this hotel if it is sponsored
sponsored = False if hotel.find("span",class_="ui_merchandising_pill") is None else True
if not sponsored:
parse_hotel_details(hotel)
print("The Hotels details in the current page are parsed")
Now that we have our hotel information stored in a Pandas data frame, we can plot the ratings of different hotels against each other to understand better how they differ. It can give us good insight into which hotels are better than others and help us make informed decisions when booking hotels.
Identified Limitations
This Redfin data scraper will extract property listing for the majority of zip codes given.
In case, you require any professional assistance with scraping property data from Redfin or any other Real Estate data scraping services then you can contact X-Byte Enterprise Crawling.
