June 6, 2018

Google searching, free Web Scraping tools and free Extract Tables from PDF tool.

By Max Drake Data Extraction, DataBases, Web 0 Comments

I have been looking at some of the free Data Science and Cognitive Computing Courses and was following the Data Journalism: First Steps, Skills and Tools course.

The Google Searching video was on improving your searches in Google using the following:

quotes to get specific key words eg “data science on construction”
using – sign to exclude certain sites eg “data science on construction” -mbie -nz ( note no space between – and word you want excluded)
using wildcard “*” in search “data on construction * 2018”
using a specific site to explore eg site: police.uk statistics or more specific site: dorset.police.uk statistics or site: *.nhs or by country site:nz “poverty statistics”
getting filetype eg site:nl filetype:pdf
for a database, where searches could vary eg site:nz database “search by” ( the search by would most probably be a place for putting a query to the database)

There was a web scraping video that I thought was great. I had previously had a couple of attempts with Python & the Beautiful Soup Extension package that I have had limited results with so far.

I particularly liked the google spreadsheets example (from 3.40 to 6.30 in the video above). This required a command in a cell with =importhtml(“URL”, Query, index), in the example he used it was a table

Also the free Outwit Hub tool that he demonstrated that works over several pages (from 9.10 to 12.40 on the video above).

Outwit Example also using Google Search site:nz database ‘search by”

I tried it out on a database search that I found and the web site did not include the actual search in the url so I had to run the search in the web page within Outwit Hub and then it crawled through the pages to get the first 100 lines (there was some messy lines I had to clear). I could run the search again from a later date to grab more data as it exceeded the 100 row limit of the free version. I am still impressed by the tool.

I did a search on the web for some other tools and came across this article which then referenced a later article highlighting other web scraping tools, some free.

Google Spreadsheet importhtml()

Using the Google Spreadsheets as a test, I also got this ( although I couldn’t get the table from https://www.stats.govt.nz/topics/building as I think it is in a separate tab (graph tab and table tab) )

PDF Table Extract using Tabula

And on a similar subject another tool I would like to mention is TABULA which is able to extract tables from PDF’s and export to say CSV’s . This runs as a server on your computer and opens in your browser. FlowingData recommended this tool.