1/13/2020 1/8 Course Search Engine: Crawling the catalog You must work alone on this assignment. Over the course of the next two assignments, you will build parts of a course search engine. We have...

1/13/2020
1/8
Course Search Engine: Crawling the
catalog
You must work alone on this assignment.
Over the course of the next two assignments, you will build parts of a course search engine. We have put
together a simple web application that will use your code to do the work of finding courses that match the
user’s criteria.
In this part of the assignment, you will build a web crawler that crawls a shadow copy of the college catalog
to construct a simple index. The purpose of this assignment is to give you more Python programming ex-
perience and to have you work with HTML documents extracted from the web. You will use the index in
PA #2.
Getting started


Before describing the specifics of your task, we will
iefly explain how to work with URLs and grab pages
from the web.
Working with URLs
URL stands for uniform resource locator. A URL, for example:
has the following format:
protocol:
site address/path/filename
The protocol field (http, in our example) specifies which protocol should be used to interact with the re-
source. Common protocols include http, https, and ftp. We will be working with http and https, the hyper-
text transport protocols. The site address specifies the host computer name (www), the domain name
(classes.cs.uchicago), and the top level domain (edu). The path specifies part of the hierarchical loca-
Loading [Contrib]/a11y/accessibility-menu.js
Due: January 16th at 6pm




http:
www.classes.cs.uchicago.edu/archive/2005/winte
10122/pa/index.html
We have seeded your repository with a directory for this assignment. To pick it up, change to you
cs10122-win-20-username directory (where the string username should be replaced with your user-
name), run git pull to make sure that your local copy of the repository is in sync with the server, and
then run git pull upstream master to pick up the distribution.
https:
www.classes.cs.uchicago.edu/archive/2020/winte
30122-1/_images/screenshot.png
1/13/2020
2/8
er, which we will not be using.
Within a web page, a link can refer to an absolute URL (that is, it starts with http:
or https:
) or a
elative URL, which can be converted into an absolute URL. A relative URL refers to a page with a location
that is relative to the cu
ent page (much like relative path names in Linux). For example, here is a URL
taken from the index page for assignments mentioned above:
pa1/index.html
The absolute URL for this page is:
URLs can use a fragment (denoted by a #) to refer to a specific location (known as an anchor) on a page.
For our purposes, we only need the part of the URL that comes before the #.
We have provided some useful functions for working with URLs:
util.is_absolute_url(url):
Takes a URL and returns true if it is an absolute URL and false otherwise.
util.convert_if_relative_url(url1, url2):
Takes the URL for a page (url1) and a URL found on that page (url2). If url2 is an absolute URL,
then this function will just return it. If url2 is a relative URL, the function will return an equivalent
absolute URL. The function will return None if it is unable to convert the URL.
The following code, for example:
would yield:
as expected.
util.remove_fragment(url):
Takes a URL and removes the # and any text following it in the URL. For example, given the relative
URL index.html#minorprogramincomputerscience, the function would return index.html. If the
URL does not contain a fragment, the function just returns the original URL.
Gra
ing a web page
We will be using requests, a Python package, for making http connections and retrieving documents. We
have supplied the following wrapper functions:
util.get_request(url):
Takes a URL and returns a request object. This function will return None if the request fails.
util.read_request(request):
Takes a request object and returns a string containing the HTML document retrieved by the request.
This function may generate a warning of the form:

WARNING:root:Some characters could not be decoded, and were replaced with REPLACELoading [Contrib]/a11y/accessibility-menu.js





url1 = "http:
www.classes.cs.uchicago.edu/archive/2005/winte
10122/pa
index.h url2 = "pa1/index.html"
util.convert_if_relative_url(url1, url2)
http:
www.classes.cs.uchicago.edu/archive/2005/winte
10122/pa/pa1/index.html
tion (archive/2005/winte
10122) of the desired file (index.html) in the file system on the host com-
puter. The path to the archive directory is supplied by the configuration of the web server program run-
ning on the host machine. The full path in the department’s file system for the example above is:
stage/classes/archive/2005/winte
10122/pa/index.html. URLs can also include a port num-
http:
www.classes.cs.uchicago.edu/archive/2005/winte
10122/pa/pa1/index.html
1/13/2020
3/8
You may ignore this warning.
util.get_request_url(request):
Takes a request object and returns the associated URL. Note that the returned URL may be different
than the URL provided to the original call to get_request. This seeming anomaly occurs when the
original URL redirects to another URL.
Note that your crawler should check for failures (that is, checking if the output of any of these functions is
None before proceeding further).
Note also that even though the recent lab used a different li
ary to download web pages for this assign-
ment, you should follow the above functions.
Catalog pages
For this assignment, we are interested in three types of HTML tags: a, div, and p. We will use the first to
find links to other pages, the second to find sections of the page pertinent to a particular course, and the
third to extract actual course titles and descriptions from within the div tags.
For example, here are three links from a page in the course catalog:
a href="http:
college.uchicago.edu">The University of Chicago
a
a href="../../azindex/index.html">AZ Index
a
a href="index.html#minorprogramincomputerscience"
a
The first specifies a link using an absolute URL. The second specifies a link using a relative URL. And the
third specifies a link using a relative URL with a fragment.
Course descriptions are enclosed in a div tag of the form