rightmove.co.uk is one of the UK's largest property listings websites, hosting thousands of listings of properties for sale and to rent.
rightmove_webscraper.py is a simple Python interface to scrape property listings from the website and prepare them in a Pandas dataframe for analysis.
Version 1.1 is available to install via Pip:
pip install -U rightmove-webscraper
1) Go to rightmove.co.uk and search for whatever region, postcode, city, etc. you are interested in. You can also add any additional filters, e.g. property type, price, number of bedrooms, etc.
2) Run the search on the rightmove website and copy the URL of the first results page.
3) Create an instance of the class with the URL as the init argument.
` pythonfrom rightmove_webscraper import RightmoveData
url = "https://www.rightmove.co.uk/property-for-sale/find.html?searchType=SALE&locationIdentifier=REGION%5E94346" rm = RightmoveData(url) `
RightmoveData instance is created it automatically scrapes every page of results available from the search URL. However please note that rightmove restricts the total possible number of results pages to 42. Therefore if you perform a search which could theoretically return many thousands of results (e.g. "all rental properties in London"), in practice you are limited to only scraping the first 1050 results (42 pages * 25 listings per page = 1050 total listings). A couple of suggested workarounds to this limitation are:
Finally, note that not every piece of data listed on the rightmove website is scraped, instead it is just a subset of the most useful features, such as price, address, number of bedrooms, listing agent. If there are additional data items you think should be scraped, please submit an issue or even better go find the xml path and submit a pull request with the changes.
The following instance methods and properties are available to access the scraped data.
Full results as a Pandas.DataFrame
Average price of all listings scraped
Total number of listings scraped
By default shows the number of listings and average price grouped by the number of bedrooms:
Alternatively group the results by any other column from the
.get_results DataFrame, for example by postcode: