quotes to get specific key words eg “data science on construction”
using – sign to exclude certain sites eg “data science on construction” -mbie -nz ( note no space between – and word you want excluded)
using wildcard “*” in search “data on construction * 2018”
using a specific site to explore eg site: police.uk statistics or more specific site: dorset.police.uk statistics or site: *.nhs or by country site:nz “poverty statistics”
getting filetype eg site:nl filetype:pdf
for a database, where searches could vary eg site:nz database “search by” ( the search by would most probably be a place for putting a query to the database)
There was a web scraping video that I thought was great. I had previously had a couple of attempts with Python & the Beautiful Soup Extension package that I have had limited results with so far.
I particularly liked the google spreadsheets example (from 3.40 to 6.30 in the video above). This required a command in a cell with =importhtml(“URL”, Query, index), in the example he used it was a table
Also the free Outwit Hub tool that he demonstrated that works over several pages (from 9.10 to 12.40 on the video above).
Outwit Example also using Google Search site:nz database ‘search by”
I tried it out on a database search that I found and the web site did not include the actual search in the url so I had to run the search in the web page within Outwit Hub and then it crawled through the pages to get the first 100 lines (there was some messy lines I had to clear). I could run the search again from a later date to grab more data as it exceeded the 100 row limit of the free version. I am still impressed by the tool.
I did a search on the web for some other tools and came across this article which then referenced a later article highlighting other web scraping tools, some free.
Google Spreadsheet importhtml()
Using the Google Spreadsheets as a test, I also got this ( although I couldn’t get the table from https://www.stats.govt.nz/topics/building as I think it is in a separate tab (graph tab and table tab) )
PDF Table Extract using Tabula
And on a similar subject another tool I would like to mention is TABULA which is able to extract tables from PDF’s and export to say CSV’s . This runs as a server on your computer and opens in your browser. FlowingData recommended this tool.
On this subject I came across an on-line version here.