Big Data Ingestion via API's

Resize the browser window to see the effect.

Big Data Ingestion - How to Use RESTful APIs

taught by Sarah Kelley

Aim of Course:

As analytics professionals and statisticians deal increasingly with Big Data, they need to know how to obtain data, and deploy the results of analysis back into web and other systems.  RESTful APIs are becoming the standard way of sharing and communicating data among applications and services on the internet.  REST stands for Representational State Transfer, and API for Application Programming Interface.  In this course, students will learn how to write Python code to ingest data from and communicate with RESTful API's on the web.

Course Program:

WEEK 1: HTTP protocol, common data formats

In our first week we will focus on how the internet works specifically as it applies to consuming RESTful services. We will take a brief survey through the HTTP protocol and the request/response cycle of exchanging information. We will touch on common data formats and end with some basic tools for development.
? How the internet works (Ports, DNS, Browser/HTTP Client)
? RPC/Web services vs normal browser usage
? Hypertext Transfer Protocol
? VirtualEnv & Github

WEEK 2:  RESTful

In week two, we will discuss Representational State Transfer as initially described by Roy Fielding. While not an exhaustive review, we’ll be looking at the primary tenets that have been adopted when developing web APIs. We will also touch on its benefits vs other methodologies, what it means to be ‘RESTful’, and how one might evaluate the degree to which a service adheres to its principles. Finally we will take a look at common features of data APIs and how to leverage them effectively.
? Resources and elements of REST
? Richardson Maturity Model
? Common API features

WEEK 3:  Scripts to ingest data

In week three we will implement basic scripts to ingest data from a REST service on the internet. Additionally we will touch on secure access to services and start our discussion on consuming data within the scope of an ingestion pipeline. 
? Key Python modules (, json, etc.)
? Using client libraries
? OAUTH and Basic Access Authentication
? Working code

WEEK 4:  Ingestion at scale

In our final week we will discuss the challenges of a data ingestion pipeline at scale. We will also examine best practices for data ingestion and review a sample codebase.
? Scaling your ingestion pipeline
? Data ingestion best practices
? Storing your data
? A sample ingestion pipeline



The homework in this course consists of short answer questions to test concepts, guided exercises in writing code and guided data analysis problems using software.

This course also has example software codes, supplemental readings available online and video lectures in each week.

Big Data Ingestion - How to Use RESTful APIs

Who Should Take This Course:

Analytics professionals, statisticians and data scientists familiar with Python who need to learn how to ingest data from, and interact with data via, RESTful web-based APIs



Organization of the Course:

This course takes place online at the Institute for 4 weeks. During each course week, you participate at times of your own choosing - there are no set times when you must be online. Course participants will be given access to a private discussion board. In class discussions led by the instructor, you can post questions, seek clarification, and interact with your fellow students and the instructor.

At the beginning of each week, you receive the relevant material, in addition to answers to exercises from the previous session. During the week, you are expected to go over the course materials, work through exercises, and submit answers. Discussion among participants is encouraged. The instructor will provide answers and comments, and at the end of the week, you will receive individual feedback on your homework answers.

Time Requirement:
About 15 hours per week, at times of  your choosing.

Students come to the Institute for a variety of reasons. As you begin the course, you will be asked to specify your category:

  1. You may be interested only in learning the material presented, and not be concerned with grades or a record of completion.
  2. You may be enrolled in PASS (Programs in Analytics and Statistical Studies) that requires demonstration of proficiency in the subject, in which case your work will be assessed for a grade.
  3. You may require a "Record of Course Completion," along with professional development credit in the form of Continuing Education Units (CEU's).  For those successfully completing the course,  CEU's and a record of course completion will be issued by The Institute, upon request.

Course Text:

Course material will be provided along with links to other required reading online.


The required software is Python Programming Language.

Big Data Ingestion - How to Use RESTful APIs

March 24, 2017 to April 21, 2017March 23, 2018 to April 20, 2018

Course Fee: $549