Essentials of Fetching Data from a Website

This step requires you to Observe the website as well as the website URL for changes in it as you search for data by manipulating elements on the webpage. We are specifically interested in analysing the URL because we will need to use the URL in MATLAB. Besides, we will also need to change the URL depending upon what data we wish to fetch from the website.

To get better insights into how the URL changes, let us continue our previous example of CNN Business and observe what changes are introduced in it as we manipulate the components on the webpage.

Example: CNN Business website

Here, in this example, we will assume that we have successfully searched for the Amazon stock company on the home page and we have opened the stock quote for the Amazon.com Inc stock company.  Thereafter we have explored the website by changing tabs on the quotes webpage. Following images show the changes that we observe as we change the tabs.

In the first image, you can see the default setup of the quotes webpage. The quote tab is open by default and the URL is as shown, i.e. https://money.cnn.com/quote/quote.html?symb=AMZN.

So, we observed how the URL changed depending upon the changes we introduced in the webpage. Now, in the next step, we will examine the changes in the URLs that we observed corresponding to the changes we made, and try and establish any patterns that emerge.

One caveat before we progress though:

The arrangement of data as well as the changes in the URL that we observe for changes made on the webpage changes from one website to the next. Therefore, the whole process needs to be repeated if you switch from one website to another. Further, ​some websites gets upgraded and even changes their webpages links. This could be somewhat cumbersome at first blush. Nevertheless, the steps for analysing any website will remain the same.

Finally, try and establish any patterns that emerge from observing the URL. Try to identify how changing the components on the webpage affects particular components of the URL. This will enable us to use the URL in a manner that best suits our needs!

Let us continue our previous example and examine the changes in the URL we observed.

Example: CNN Business website

The second observation we made was when we changed the tab from quote to Financials. This does not cause any changes in the symbol part of the URL however, quote.html has been replaced with financials.html  indicating that we have now switched the tabs from the Quote tab to the Financials tab.

The third observation we made was when we changed the tab from quote to Chart. This, again, does not cause any changes in the symbol part of the URL however, financials.html has been replaced with chart.html  indicating that we have now switched the tabs from the Quote tab to the Financials tab.

Now that we know how we can analyse any website, we can start our discussion of fetching historical stock data for our application. ​We will analyse the Yahoo Finance website and extract our historical data. We will start the discussion in the next lesson!