Set Machine home
	Download
	Register
	Tutorial
	Help
	Site map
	Contact info

Fetching (scraping) World Wide Web data with WWWGrab

WWWGrab is a Windows program that scans lists of web page URLs (Uniform Resource Locators), fetching the data at each one and parsing the data with the Set Machine data transformer.   What's required :

• a Microsoft Windows system
• an Internet connection
• a database management system (DBMS), for example:  Microsoft Access
• 5 megabytes of free disk space

WWWGrab's processing is controlled by interrelated database table types :

• Task tables
• Source tables
• SQL tables

The table names are user-defined.   The fields indicated below must be present, but the user can add fields to the tables if needed.

WWWGrab is run from the command line with 2 arguments :

1. the ODBC Data Source Name (DSN),
2. the name of a "task table" in that database :

wwwgrab  <DSN> <task table>

For example, this command runs the stock quote scraper sample :

"C:\Program Files\WWWGrab\WWWGrab"   WWWGrab-Samples   quotesget

"WWWGrab-Samples" is the DSN, "quotesget" is the task table name.   See the Running WWWGrab topic for more information.

The task table

A single task table controls the process, instructing WWWGrab to :

• Scan URLs
• Execute SQL

WWWGrab scans the task table executing the user-defined actions, which can be :

The Scan and ExecSQL tasks can be run in any order.   For example, you might want to execute SQL before and after a URL scan.  

Example

The following task table :

1. runs the SQL in table PreSQL,
2. scans the URL list in table WebSources,
3. runs the SQL in table PostSQL.

The Exec field allows the user to switch tasks on and off without having to add / remove them from the table.

This design is flexible enough to allow multiple passes, where, for example, a list of URLs generated by one step is scanned by the next step.   Data from multi-level websites can be extracted using this feature.

Source tables

The URLs and associated data are specified in a "source" database table containing the following information :

WWWGrab scans the Sources from beginning to end.   Each URL is parsed per the associated Set Machine parser (.PXD file).

Example

The following Source table instructs WWWGrab to fetch and parse data from 3 URLs :

All three URLs are parsed with the Quotes01.PXD parser.   The Options, UserName and Password fields, which are not typically used, are not shown in this example.

SQL tables

The task table can trigger the execution of SQL (database Structured Query Language) statements in tables containing the following information :

SQL tables can be used to prepare data in the Source tables, for example.

Example

The following SQL table instructs WWWGrab to execute 2 SQL statements to prepare the ContributionsSources table for a URL scan :

In this case the ContributionsSources table has an extra field, "zip".   Fields can be added to the WWWGrab tables if necessary to facilitate SQL manipulations.

Once again, an Exec field allows the user to switch tasks on and off without having to add / remove them from the table.  

The Command field must contain a valid SQL statement.  

WWWGrab version 1.3 fetches and parses static web pages.   More powerful web data acquisition methods will be available in future releases.

WWWGrab setup

Before you can run WWWGrab you have to set it up :

• download and run the WWWGrab installer
• configure your firewall to allow WWWGrab access to the Internet
• configure an ODBC DSN (Data Source Name) for your database [*]
• specify your URL data in the "Sources" table
• run WWWGrab

[*] The DSN for the samples is configured automatically during installation.

Running WWWGrab

WWWGrab is a Windows console (command line) program.   It accepts two command line arguments :

1. the ODBC DSN (Data Source Name)
2. a task table name

Once WWWGrab is installed and configured enter :

wwwgrab  <DSN> <task table>

wwwgrab  yourdb  getprices

To run the stock quote scraper sample :

"C:\Program Files\WWWGrab\WWWGrab"   WWWGrab-Samples   quotesget

The "WWWGrab" shortcuts created by the installation are set up to run :

wwwgrab   WWWGrab-Samples   QuotesGet

wwwgrab   WWWGrab-Samples   RentalsGet

wwwgrab   WWWGrab-Samples   TablesGet

- the samples.

WWWGrab works in conjunction with Set Machine.   Both products offer a 30-day trial period.   Register WWWGrab by registering Set Machine!

DSN configuration

WWWGrab uses ODBC, "Open Database Connectivity", to interact with user-defined databases.   Under ODBC databases are assigned labels, or Data Source Names ("DSNs").   The WWWGrab installer automatically configures the WWWGrab-Samples DSN for the sample database.

To use another database the user must configure a DSN for the database.   DSN configuration involves little more than applying a system-wide label to an existing database.   Use the "Configure DSN" shortcut created during installation to start the ODBC Data Source Administrator, then :

• Select the User DSN tab
• Click the Add button
• Select the appropriate ODBC driver - for the sample Access database it will be "Microsoft Access Driver"
• Click Finish (well, we're not quite finished ...)
• In the Data Source Name box enter the label, e.g. "Quotes"
• Enter a description for the database (optional)
• Click Select to locate the database in the file system, e.g. "C:\Program Files\WWWGrab"
• Select the database file/directory and click OK
• Keep clicking OK to get out of the ODBC Data Source Administrator - you're done!

Visit wwwgrab.com/db.html for more information about databases, DSNs and ODBC.

HTML parsing

Web pages are written using HTML, Hypertext Markup Language.   WWWGrab parsers are, therefore, HTML parsers, so configuring WWWGrab requires some knowledge of HTML.   For example, parsing an HTML table requires an understanding of the TABLE, TR, TH, and TD tags, among others.

In simple cases, the WWWGrab stock quote sample for instance, HTML tags can be ignored.   But for more sophisticated parsing the user should be familiar with the basics of HTML -  many good books are available on the subject.

Most web browsers allow the user to view the HTML source of a web page.   The View/Source menu option in Microsoft Internet Explorer displays HTML source with the Notepad program.   Netscape Navigator displays the source with tags and content in different colors, making it easier to see the structure of the web page.   WWWGrab supports a "noparse" option that allows saving the URL content to a local file.

Set Machine's design allows creation of reusable parsing components:  "node groups".   The following HTML parsing node groups are installed with WWWGrab:

Use of these node groups allows for a relatively clean top-level .PXD file.   Consult Set Machine's help for more information.

WWWGrab samples

The WWWGrab installation comes with a few sample configurations :

• Quotes
• Rentals
• Tables

The Quotes sample is very basic illustration of WWWGrab, extracting a single number (stock quote) from each requested web page.

The Rentals sample extracts rental listings from the CraigsList website.

The Tables sample parses an HTML table.   It makes extensive use of Set Machine node groups.   See Set Machine help for more information.

Note:  The WWWGrab installer automatically configures the WWWGrab-Samples ODBC Data Source Name (DSN).   The installer runs the DSNConfig.EXE program, which reads file SampleDSN.txt to configure the sample DSN.   The WWWGrab uninstaller removes this DSN.

WWWGrab sample :  Stock quote scraper

This WWWGrab sample extracts stock quote data from http://finance.yahoo.com/ and transmits it to the QuotesOutput table in the sample database.   A sample Microsoft Access database (Samples.MDB) accompanies the distribution package, but you can use any database (DBMS) that provides a suitable ODBC driver.   This sample should work on Windows systems with :

• an Internet connection
• MS Access (2000 or later) installed

The WWWGrab Quotes sample uses three tables :

1. QuotesGet (task table)
2. QuotesSources (sources table)
3. QuotesOutput (output table)

The QuotesGet task table simply directs WWWGrab to scan the QuotesSources table :

The QuotesSources table lists the stock quotes of interest :

The "Source" field is user-defined text passed into Set Machine, where it can be transmitted to the output (via Set Machine variable "Source").   This example uses the same .PXD file (Quotes01) for each entry; different .PXD files can be used if needed.

Sample output

The quote is extracted from the website.   The date/time (Grabbed) is generated by WWWGrab/Set Machine.   The Source text is passed from the input table into the output.   Open the sample configuration file, Quotes01.PXD, with Set Machine to see how this simple HTML parser extracts data from web pages fetched by WWWGrab.

You can modify the QuotesSources table to extract any number of quotes.   The URL should end with a valid symbol :

http://finance.yahoo.com/q?s=GOOG

WWWGrab sample :  Rental listing scraper

This WWWGrab sample extracts rental listing data from CraigsList and transmits it to the RentalsOutput table in the sample database.

The WWWGrab Rentals sample uses four tables :

1. RentalsGet (task table)
2. RentalsPreSQL (SQL table)
3. RentalsSources (sources table)
4. RentalsOutput (output table)

The RentalsGet task table first runs the SQL in the RentalsPreSQL table to delete all the records in the RentalsOutput table.   It then starts the scan of the RentalsSources table, which populates RentalsOutput with the desired listings.

WWWGrab sample :  HTML table scraper

This WWWGrab sample extracts data from a web page table.

This sample uses three database tables :

1. TablesGet (task table)
2. TablesSources (sources table)
3. TablesOutput (output table)

Keep in mind that the word "table" is used two ways here :

• database table
• HTML table

TablesSources contains the URL / parser list :

The Tables01.PXD parser starts by looking for "Locators" (a Set Machine string set) in order to skip over tables that we're not interested in.   The entries in the Locators string set contain keywords that distinguish the desired table from other tables that may precede it in the web page being parsed.   Click on the "Configure parsers" shortcut and open Tables01.PXD to see how this works.

Sample output

Open the sample configuration file, Tables01.PXD, with Set Machine to see how this HTML parser extracts data from web pages fetched by WWWGrab.

Note the choice of generic field names "TD1" .. "TD4" in the database (Samples.MDB) and the parser (Tables01.PXD).   If you adapt this sample you can make those names more meaningful.

WWWGrab options

WWWGrab supports several options that can be specified on a per-URL basis, in the Source table "options" field :

[*] The "noparse" option enables you to write the fetched URL's content directly to a local file, without modification.   The output file path is specified in the sources table parser field.   This option provides the unmodified results of the HTTP Get command.   This can be very useful for parser development.

[**] The maximum number of HTTP redirects defaults to 3.

[***] WWWGrab uses a function called "AfxParseURLEx" to parse the URL prior to fetching the data.   Visit http://msdn.microsoft.com/ for more information on AfxParseURLEx and its options.

Example

The 1st example sets the number of HTTP redirects to zero.

The 2nd example writes the fetched URL's content, unmodified, to file c:\urldump\sample1.html.

The 3rd example illustrates the combination of two options, separated by a blank.

WWWGrab files :

WWWGrab sample files :

Questions, comments?   Contact WWWGrab.com.

Set Machine home

Download

Register

Tutorial

Help

Site map

Contact info