Scraping For Rent Most Important Data Analysis On Shanghai Rental Listings

November 24, 2021
scraping for rent top level data analysis of shanghai rental listings

Getting a place on rent in Shanghai is not difficult but getting an apartment, which fits excessively higher expectations with a stiff budget is tough indeed.

Real estate representatives usually say things like “less than the market price”, “it is the bargain” as well as “worth each penny”, so we wanted to know the measurable frame of these words. What should your expectations be when you look for an apartment to rent in Shanghai?

We have extracted data of 2007 listings of one-bedroom rental displayed on Smartshanghai — one platform for emigrants living in this city. The technical execution of the web data scraper in Python could be available in Github. Using requests, we have collected the HTML-formatted data from a website as well as used scrapy’s navigator functions for scraping pieces of data, which we care about. Then, the results were compiled in a panda’s data frame:


Two preferences here: Smartshangai is the platform of working expats that puts a higher price tag on a property. Therefore, all the presented insights might not be completely representative of a Shanghai rental market on the whole, however, would rather get an overview of listings given for the foreign audiences. One more issue — the rental fees given is just the asking price that could be massively different from actual money funded after mediation.

By keeping that in mind, let’s go through some fundamental statistics about monthly rental fees:


Key take-out here: 6800 CNY (at 10th percentile) is the entry-hurdle for any one-bedroom flat available in Shanghai. Without it — you should either be extremely lucky or go for sharing the apartment. Although as given in the below chart, there is a key skew of distribution for the right having a huge portion of listing in 7–10k of area, therefore getting this amount would do nicely.


Being said that, it is extremely important to understand the apartment size you are going to rent. The relationship between the size in square meters, as well as monthly rental fees in this dataset, is 0.75 (which is Pearson R out of minimum -1 and maximum 1), which means — greatly correlated.


Duh, you might think, — no surprises there. We, however, had never paid considerable attention to apartment size if it looked spacious. Understanding the precise meterage, temporarily, can provide you a benefit while estimating total pricing as well as driving the fees down.

Now to the juicy side: what makes an apartment get or lose value and what are different attributes, which will make the difference?

Just see the average rental fees of a listing divided in terms of the district. Truly you need to pay additional if you wish to get closer proximity to the Jingan restaurant and a bit less than this for Xuhui’s older French concession.


When comes to differences in rental pricing by floor, we have interesting insights: indeed, the first-floor listings might cost less than all apartments on the higher floor, however, the second floor is least valued. Our hypothesis is, while the initial floor is usually attached to the patio or smaller garden, this second floor is doubtful to have this advantage.


Another interesting thing: this does not look to matter how near the apartments are to a subway station. These rental fees are something on par despite the walking proximities to a metro. Therefore, next time when any agent or landlord says: “but this is only next to the subway”, you could courteously decline that argument.


That’s all for now. Contact X-Byte Enterprise Crawling for all your Shanghai rental listings scraping for rent requirements or ask for a free quote!