INTRODUCTION TO MACHINE LEARNING WITH PYTHON

A Guide for Beginners in Data Science

Daniel Nedal & Peters Morgan

How to contact us

If you find any damage, editing issues or any other issues in this book contain please immediately notify our customer service by email at:

contact@aisciences.net

Our goal is to provide high-quality books for your technical learning in computer science subjects.

Thank you so much for buying this book.

Table of Contents

Table of Contents

Authors Biography

From AI Sciences Publisher

Preface

Introduction

Do You Really Need to Know Statistics & Python?

Python is Beginner Friendly:

Python Enables Quick Prototyping:

Python Has Awesome Scientific Libraries:

Areas of Study to Explore

Overview of Python Programming Language

The Python Programming Language

Common Python Syntax

Python Data Structures

Python for Scientific Computing

Statistics

Data in Statistics

Descriptive and Inferential Statistics

Measures of Central Tendencies

Dispersion, Covariance and Correlation

Probability

Dependence and Independence

Conditional Probability

Random Variables

Discrete and Continuous Distributions

Bayes’ Theorem and Naive Bayes Algorithm

The Data Science Process

Asking the Right Question

Data Acquisition

Data Preparation

Data Exploration

Data Modelling

Data Presentation

Machine Learning

What is Machine Learning

Supervised Learning Algorithms

Unsupervised Learning Algorithms

Semi-supervised Learning Algorithms

Reinforcement Learning Algorithms

Overfitting and Underfitting

Correctness

The Bias-Variance Trade-off

Feature Extraction and Selection

Why Machine Learning is Popular

Python Data Science Tools

Jupyter Notebook

Numerical Python (Numpy)

Pandas

Scientific Python (Scipy)

Matplotlib

Scikit-Learn

K-Nearest Neighbors

Introduction to K Nearest Neighbors

How to create and test the K Nearest Neighbor classifier

Another Application

Calculating Similarity

Locating Neighbors

Generating Response

Evaluating Accuracy

The Curse of Dimensionality

Naive Bayes

Applications of Naive Bayes

How to Build a Basic Model Using Naive Bayes in Python

Regression

Introduction to Labels and Features

Features

Simple and Multiple Linear Regression

Logistic Regression

Generalized Linear Models

A Regression Example: Predicting Boston Housing Prices

Steps To Carry Out Analysis

Import Libraries:

How to forecast and Predict

Decision Trees and Random Forest

The Entropy of a Partition

Creating a Decision Tree

Random Forests

Neural Networks

Perceptrons

Backpropagation

How to run the Neural Network using TensorFlow

How to get our data

How to train and test the data

Clustering

Introduction to Clustering

Example of Clustering

Running K-means with Scikit-Learn

Implementation of the Model

Bottom-up Hierarchical Clustering

K-means Clustering

Network Analysis

Betweenness centrality

Eigenvector Centrality

Recommender Systems

Classification

Multi-Class Classification

Popular Classification Algorithms

Support Vector Machine

How to create and test the Support Vector Machine (SVM) classifier

Deep Learning using TensorFlow

Deep Learning Compared to Other Machine Learning Approaches

Applications of Deep Learning

Python Deep Learning Frameworks

Install TensorFlow

Installing TensorFlow on Windows

Installing TensorFlow on Linux

Installing TensorFlow on macOS

How to Create a Neural Network Model

Deep Learning Case Studies

Bank Churn Modelling

Sentiment Analysis

Conclusion

Thank you !

Sources & References

Software, libraries, & programming language

Datasets

Online books, tutorials, & other references

Thank you !

© Copyright 2018 by AI Sciences

All rights reserved.

First Printing, 2018

Edited by Davies Company

Ebook Converted and Cover by Pixels Studio

Publised by AI Sciences LLC

ISBN-13: 978-1724417503

ISBN-10: 1724417509

The contents of this book may not be reproduced, duplicated or transmitted without the direct written permission of the author.

Under no circumstances will any legal responsibility or blame be held against the publisher for any reparation, damages, or monetary loss due to the information herein, either directly or indirectly.

Legal Notice:

You cannot amend, distribute, sell, use, quote or paraphrase any part or the content within this book without the consent of the author.

Disclaimer Notice:

Please note the information contained within this document is for educational and entertainment purposes only. No warranties of any kind are expressed or implied. Readers acknowledge that the author is not engaging in the rendering of legal, financial, medical or professional advice. Please consult a licensed professional before attempting any techniques outlined in this book.

By reading this document, the reader agrees that under no circumstances is the author responsible for any losses, direct or indirect, which are incurred as a result of the use of information contained within this document, including, but not limited to, errors, omissions, or inaccuracies.

To all the Data Scientist and Computer Scientist in the World

Authors Biography

Daniel Nedal is a data scientist and long-time user of the Python. He currently works as a computer scientist and as a research director at one of the biggest University in Paris.

Peters Morgan is a lecturer at the Data Science Institute at Melbourne University and a long-time user and developer of the Python. He is one of the core developers of some data science libraries in Python. Peter worked also as Machine Learning Scientist at Google for many years.

From AI Sciences Publisher

WWW.AISCIENCES.NET

EBooks, free offers of ebooks and online learning courses.

Did you know that AI Sciences offers free eBooks versions of every books published? Please suscribe to our email list to be aware about our free ebook promotion. Get in touch with us at contact@aisciences.net for more details.

At www.aisciences.net , you can also read a collection of free books and received exclusive free ebooks.

Preface

“Some people call this artificial intelligence, but the reality is this technology will enhance us. So instead of artificial intelligence, I think we'll augment our intelligence.”

―—Ginni Rometty

The main purpose of this book is to provide the reader with the most fundamental knowledge of machine learning with Python so that they can understand what these are all about.

Book Objectives

This book will help you:

             Have an appreciation for machine learning and deep learning and an understanding of their fundamental principles.

             Have an elementary grasp of machine learning concepts and algorithms.

             Have achieved a technical background in machine learning and also deep learning

Target Users

The book designed for a variety of target audiences. The most suitable users would include: 

             Newbies in computer science techniques and machine learning

             Professionals in machine learning and social sciences

             Professors, lecturers or tutors who are looking to find better ways to explain the content to their students in the simplest and easiest way 

             Students and academicians, especially those focusing on machine learning practical guide using R

Is this book for me?

If you want to smash machine learning from scratch, this book is for you. Little programming experience is required. If you already wrote a few lines of code and recognize basic programming statements, you’ll be OK.

Why this book?

This book is written to help you learn machine learning using Python programming. If you are an absolute beginner in this field, you’ll find that this book explains complex concepts in an easy to understand manner without math or complicated theorical elements. If you are an experienced data scientist, this book gives you a good base from which to explore machine learning application.

Topics are carefully selected to give you a broad exposure to machine learning application. While not overwhelming you with information overload.

The example and cases studies are carefully chosen to demonstrate each algorithm and model so that you can gain a deeper understand of machine learning. Inside the book and in the appendices at the end of the book we provide you a convenient references.

You can download the source code for the project and other free books at:

http://aisciences.net/code

Your Free Gift

As a way of saying thanks you for your purchase, AI Sciences Publishing Company offering you a free eBook in Machine Learning with Python written by the data scientist Alain Kaufmann.

It is a full book that contains useful machine learning techniques using python. It is 100 pages book with one bonus chapter focusing in Anaconda Setup & Python Crash Course.  AI Sciences encourage you to print, save and share. You can download it by going to the link below or by clicking in the book cover above.

http://aisciences.net/free-books/

If you want to help us produce more material like this, then please leave an honest review on amazon. It really does make a difference.

Introduction

The importance of machine learning and deep learning is such that everyone regardless of their profession should have a fair understanding of how it works. Having said that, this book is geared towards the following set of people:

●    Anyone who is intrigued by how algorithms arrive at predictions but has no previous knowledge of the field.

●    Software developers and engineers with a strong programming background but seeking to break into the field of machine learning.

●    Seasoned professionals in the field of artificial intelligence and machine learning who desire a bird’s eye view of current techniques and approaches.

While this book would seek to explain common terms and algorithms in an intuitive way, it would however not dumb down the mathematics on whose foundation these techniques are based. There would be little assumption of prior knowledge on the part of the reader as terms would be introduced and explained as required. We would use a progressive approach whereby we start out slowly and improve on the complexity of our solutions.

To get the most out of the concepts that would be covered, readers are advised to adopt a hands on approach which would lead to better mental representations.

Finally, after going through the contents in this book and the accompanying examples, you would be well suited to tackle problems which pique your interests using machine learning and deep learning models.

Do You Really Need to Know Statistics & Python?

For many using machine learning for day to day tasks, Python is the programming language of choice. They are many reasons for this bias towards Python. Let’s have a look at some of them:

Python is Beginner Friendly:

The syntax of Python programming language is intuitive and easy to learn for beginners and advanced professionals alike. Python uses regular English words in a way that makes Python code readable and easy to understand. Unlike some other programming languages, Python does not use curly braces but instead relies on indentation to separate blocks of code. The classic “Hello World!” example in Python is just one line of code

print(“Hello World!”)

Python Enables Quick Prototyping:

Python is often the preferred choice when you want to go from idea to implementation in as short a time as possible. The reason this is usually the case is because Python is a general programming language that is dynamically typed. What that means is that you do not need to compile your code before you run it. You can rather use an interactive development cycle whereby your write code, try things out and see what works, then iterate progressively towards an optimal solution.

Teams seeking to bring products to market would use Python to iterate quickly and use feedback from users as a loop to improve the model. The advantage of being an early player in a market cannot be overemphasized especially  with data products where more users mean more data and more data means a better performing model.

Python Has Awesome Scientific Libraries:

Python has a host of world class libraries for scientific computation which are mature and have been used in production for many years. These libraries have popular machine learning models implemented and are easily extensible providing an array of tools for more experienced users to do so.

You can get started building your own machine learning models using these libraries in Python by calling various components and assembling it into a stack peculiar for your learning task. However, beyond this point it is desirable to get an understanding of the mathematical underpinnings of these models as it would help you tweak your models to achieve better results.

An investment in studying the mathematics of machine learning would lead to huge payoffs long term as models would no longer be  a black box and you can peel behind the curtain to have a look when things don’t work as expected.

Areas of Study to Explore

There are three main areas of study to explore when pursuing a deeper understanding of the mathematical models that comprise the landscape in machine learning. The most important area is probability and statistics, then linear algebra and Calculus.

So strictly speaking, you do not need to be a statistician to get started using machine learning to derive real world value, but it does bestow an advantage when you understand the mathematics behind why models work the way they do.

Overview of Python Programming Language

The Python Programming Language

Python is a general purpose programming language language used in web development, scientific computing, system administration, software development etc. Python favors readability and has an English like syntax. One of the main differences between Python and other programming languages like Java, C++, PHP etc is that it does not make use of curly braces to define scope. Blocks of code, called suites in Python are delineated using whitespace. Python is also a dynamically typed language which means variable types do not have to be declared before variables are used. It makes use of duck typing which makes certain assumptions about the type of a variable from its content. Duck typing in programming is coined from the popular phrase - “If it walks like a duck and quacks like a duck, then it’s a duck”. What this means in essence is that if an object exhibits certain properties, then its type can be deduced. This is a powerful assumption as it allows code to be written interactively and defers type checking to when the program is actually run (runtime).

Python is a very expressive programming language and is favored by programmers and scientists because it encourages quick prototyping and significantly reduces development time when compared to other high level programming languages. In data science, Python enables fast iteration of data science projects and because Python is a general purpose programming language, prototype models can be more easily integrated into production workflows as there is usually no need to switch to another programming language. In this way, one language can be used for the entire stack, from prototyping to deployment. It should be noted however, that this generally depends on the type of application and Python especially for scientific computing usually reference lower level extensions in faster programming languages like C or C++. Python can be seen as the ideal interface to work across a slew of tasks effectively and efficiently.

Python was created by Guido van Rossum in 1991 and has undergone several iterations. There are currently two major versions of Python - Python 2 and Python 3. At the time of this writing the development of Python 2 has been discontinued so it is advised to use Python 3 for all new projects. For this reason, the examples we would come across in this book all assume a Python 3 environment.

There are several ways to set up a local Python development environment such as installing Python natively on a computer, setting up a virtual environment using a tool like virtualenv or using a bundled scientific distribution like Anaconda. For the purposes of this book, we would leverage the immensely popular scientific distribution Anaconda because it contains many prepackaged Python scientific libraries, some of which we would use throughout this book. Anaconda is cross platform and is available for the major operating systems - Windows, Linux, macOS etc. You can download the Anaconda installer for your operating system by going to their download page (https://www.anaconda.com/downloads) and following the installation instructions.

Once Anaconda has been installed on your computer, new Python packages can be installed by using Anaconda’s package manager, “conda” or using Python’s native package manager - “pip”. It should also be noted that packages should be upgraded or deleted using the appropriate package manager through which the package was installed.

Here are examples instructions on installing packages from the terminal for both cases:

$ conda install package_name # installation via conda package manager

$ pip install package_name # installation via pip

Similarly, an installed package can be updated using the following commands.

$ conda update package_name # update via conda package manager

$ pip install --upgrade package_name # upgrade via pip

Common Python Syntax

In this section, a brief overview of basic Python syntax is presented. As would become evident, indentation is important as all lines of code in the same suite (block) must be indented by an equal number of characters. By convention, this is usually 4 white spaces. Let us honour traditions and start with a simple Hello world! example in Python.

print('Hello World!')

The code above outputs the string “Hello World!” to the screen.

Next, we look at variable assignment. Variables can be seen as containers that point to an entity or stored value. Entities are assigned to a variable using the equality operator. The value on the right hand side is put into the container on the left hand side. A variable usually has a name and calling the variable by its name references the stored object.

a = 3

b = 4

c = a + b

print('The value of a is {}, while the value of b is {}, and their sum c is {}'.format(a, b, c))

The code above assigns an integer with a value of 3 to the variable named a, it also assigns 4 to b and finally computes the sum of a and b and stores it in a new variable c. It should be noted from the above piece of code that we never explicitly defined the types of variables we created, rather the type information was gotten from the kind of entity the object contained. There are mainly types of mathematical operations available in Python apart from addition used above. A good approach is to familiarize yourself with the Python documentation as the standard Python library contains many useful utilities. The documentation for Python 3 can be accessed at https://docs.python.org/3.

Moving forward, Python supports the use of conditionals to determine which suite of code to execute. Regular conditionals such as if, else if and else are available in Python. One thing to note is that else if is expressed as elif in Python. Let us look at a simple example below.

a = 200

b = 33

if b > a:

  print("b is greater than a")

elif a == b:

  print("a and b are equal")

else:

  print("a is greater than b")

The code snippet above uses conditionals to determine which suite of code to run. Suites use whitespace indentation for separation and the output printed to the screen is determined by the evaluation of the conditional in line with the declared variables contained therein.

Another important Python syntax are loops. Loops are used for repeating a block of code several times. They may be used in conjunction with conditionals.

for x in range(2):

  print(x)

The code above prints 0 and 1 to the screen. Python indexes start from 0 and the range function in Python is non inclusive. What that means is that the last value of a range is not included when it is evaluated. Loops are a very useful construct in Python and generally are in the form shown above. There is also another form known as while loops but for loops are used more often.

This brings us to the concept of a function. A function is a block of code which has been wrapped together and performs a specific task. A function is usually named but may be anonymous. In Python, a function is the major way we write reusable chunks of code. Let us look at a simple function below.

def my_function(planet):

  print('Hello ' + planet)

A function is defined using the special keyword def. A function may accept arguments or return a value. To call a function (execute it), we type the function name followed by a parenthesis containing parameters if the function expects arguments, else we call it with empty parentheses.

my_function('Earth!')

Comments in Python are ignored by the interpreter and can be used to explain code or for internal documentation. There are two types of comments in Python. The first uses the pound or hash symbol which the second is known as a docstring and uses 3 quotation marks.

# a single line comment using pound or hash symbol

'''

A multi-line

comment in Python

'''

print('Comments in Python!')

Python Data Structures

Data structures are how data is collectively stored for easy access and manipulation. There are several data structures in Python which enables quick prototyping. Data structures are the format in which data is stored and usually includes the kinds of operations or functions that can be called on the data. The most popular data structure in Python are lists. Lists can contain different types of data and are ordinal. Lists are synonymous to arrays in other programming languages. Other data structures includes a tuple which is a collection that cannot be modified, a set, which is an immutable list with unique values and a dictionary, which is a key-value pair data structure. Let us look at how to create each of them below.

my_list = ['apple', 'banana', 4, 20]

print(my_list)

Lists can also be defined using the list constructor as shown below.

another_list = list(('a', 'b', 'c'))

print(another_list)

Tuples are immutable, this means that we cannot change the values of a tuple, trying to do so would result in an error. Below is how tuples are created.

my_tuple = (1, 2, 3, 4)

print(my_tuple)

print(type(my_tuple))

Using the inbuilt type function gives us the type of an object.

Sets are unordered collections that can contain only unique values. Sets are created using curly braces as shown below.

my_set = {1, 1, 2, 2, 2, 'three'}

print(my_set)

print(type(my_set))

In the example above, notice that all duplicate entries are removed when the set is created and there is no concept of ordering.

A dictionary is a collection of key value pairs that are unordered and can be changed. Dictionaries are created using curly braces with each key pointing to its corresponding value.

my_dict = {'1': 'one', '2': 'two', '3': 'three'}

print(my_dict)

print(type(my_dict))

There are other data types in Python but these are by far the most commonly used ones. To understand more about these data structures and which operations that can be performed on them, read through the official Python documentation.

Python for Scientific Computing

One of the reasons for the rapid adoption of Python by the scientific community is because of the availability of scientific computing packages and the relative ease of use as most scientists are not professional programmers. This has in turn lead to better algorithms being implemented in many Python scientific packages as the community has evolved to support several packages. Another reason for the widespread adoption of Python in data science and in the larger scientific community is because Python is a well designed language and is useful across several tasks, so users do not need to learn a new programming language when confronted with a new task but can rather leverage Python’s rich ecosystem of libraries to perform their tasks. Python is also easy to pick up so users can learn to extend libraries to support the functionality that they desire. This forms a virtuous cycle as libraries become more mature and support a wider range of adopters.

Scipy, also known as scientific python contains several packages that build on each other to provide a rich repository of scientific tools. Numpy or numerical Python enables numerical computation like matrix operations, Fourier transforms, random number operations etc. The Scipy library contains modules that can be used for signal processing, optimization, statistics etc, while matplotlib provides access to a powerful plotting package that can be used to produce high quality 2-dimensional and 3-dimensional plots. Other libraries in the wider ecosystem are Pandas, Scikit-Learn, Jupyter notebooks etc. We would look at each of these package in more depth in the next section.

Statistics

Data in Statistics

Many data science modelling techniques have their roots in statistics. Statistics is a field of mathematics that deals with presenting information garnered from data in a form that is easy to understand. It involves collection, analysis, organization and presentation of data. Simply put statistics enable us draw a summary of our raw data. This presentation of gleaned information is usually done in graphs, charts, tables etc. Data can be seen as raw facts from which we can draw conclusions while statistics is the process through which we employ numerical and mathematical techniques to actually derive knowledge from data. Even Though both are related, there is a clear distinction between them. Data in an unprocessed form is not informative but barely contains the building blocks through which we can use statistics to transform it into information that is relevant. Information is data that has been processed to give meaning. This may take the mould of classification or correlations.

There are two main branches of statistics - descriptive and inferential. Descriptive statistics is concerned with summarizing a sample population in terms of indices such as mean, mode, standard deviation whereas inferential statistics is interested in arriving at conclusions from the data by studying the underlying probability distribution that makes the data unique.

Descriptive and Inferential Statistics

Descriptive statistics is the branch of statistics that is interested in describing the nature of data as a direct effect of the population under study. The population under study are made of samples and those samples are usually complete and can be used to study that population effectively. The role of descriptive statistics is to summarize the characteristics of the population. There are two broad techniques employed - measures of central tendencies and measures of spread. Measures of central tendencies like mean, mode and median gives the most common occurrences in the data whereas measures of spread like variance, range, quartiles, standard deviation etc describe how far samples are from the central position. Descriptive statistics techniques are mainly used to organize, analyze and present data in a meaningful way.

However, in most cases, we do not have access to the entire data in a population. We can merely collect a subset of data that is representative of the wider population. In such cases, we would like to use our sample data to model aspects of the wider population. This is where inferential statistics come in. Inferential statistics is the branch of statistics that seeks to arrive at conclusions about a population through the analysis of sample data that is drawn from that population. It discovers trends within a sample and then tries to generalize those trends to the wider population. Inferential statistics is made up of two parts, estimation of parameters and testing out hypothesis. The results of inferential statistics are usually presented as probabilities that show the confidence of particular parameters or events being true. In a nutshell, inferential statistics is concerned with making predictions about a population through the study of a sample from that population.

Measures of Central Tendencies

In descriptive statistics, we often want to measure the properties that describe the distribution (population), this is done in terms of two properties, the central tendency and dispersion. The population central tendency encompasses the typical (common) value of the distribution. From the normal distribution or bell curve, the common type of value is usually at the center hence the name central tendency.

Let at look at the diagram below which contains some measures of central tendencies to hone our intuitions further.

The plot contains data from an independent variable X in some distribution. The role of measures of central tendencies is to describe common or typical values of the sample population. We can see that the highest point in the 2-dimensional plot of the independent variable against the dependent variable is the mode. The mode indicates the most likely value in the distribution, in other words, it is the most popular or frequently occuring value in the dataset. The median is the midway point between all values after they have been arranged in ascending or descending order. The midway point usually occurs at the 50% mark. The mean or arithmetic average is the ratio of the sum of all values to the number of values in the population. It is given by the formula below:

Where A = arithmetic mean

n = number of observations and

a = individual observation

Together, the arithmetic mean, mode and median give a good description of a dataset and are frequently used in descriptive statistics.

Let us now look at how we can compute central tendencies on a toy dataset.

First we import Numpy and Scipy.

import numpy as np

from scipy import stats

Next we create a dataset by passing a list into Numpy array function.

dataset = np.array([3, 1, 4, 1, 1])

We can easily calculate the mean by calling the mean function from Numpy and passing in the dataset.

mean = np.mean(dataset)

print(mean)

To calculate the median, we call the median function from Numpy and pass in the dataset.

median = np.median(dataset)

print('Median: {:.1f}'.format(median))

Finally, to compute the mode, we use the mode function from Scipy stats module.

mode= stats.mode(dataset)

print(mode)

print('Mode: {}'.format(mode[0][0]))

print('{} appeared {} times in the dataset'.format(mode[0][0], mode[1][0]))

The mode is 1 since it is the most common number in our toy dataset.

Dispersion, Covariance and Correlation

The dispersion of a distribution refers to how widely spread sample data points are in that population. It explains the amount of variability present in a distribution, that is how widely do data points vary across across a central location.

In the image above, distribution A has low dispersion. This is because most of its values are centered in the middle. It should be noted that the centrality of data points has an inverse relationship with dispersion. In distribution B, there is greater dispersion as values appear to be located across a broader range. The shorter height of the curve when compared to A shows that its mean is lower as values are not compact within a central range. Distribution C shows the most variation. The values are spread across a greater range than A or B and its height is very low indicating small values for measures of central tendency such as the mean. Some ways in which statistical dispersion is measured includes variance, standard deviation and interquartile range.

The formula for standard deviation is given below:

It should be noted that variance is the square of standard deviation.

The variance as we have seen defines how much values of a variable are away from its mean. That is how greatly does it vary across the distribution. Covariance extends the concept of variance from one variable to two variables. Covariance measures how well two random variables vary in line with each other.

The formula for covariance is given by:

The covariance of X and Y tells us how much a change in X results in a corresponding change in Y. The covariance paints a picture about the relationship between random variables and how they interact with each other. Despite its ability to indicate relationship between random variables, the covariance does not tell us by what degree variables are correlated. This is because random variables may be in different units and there is no way we would be able to interpret it deeply without knowing the extent of the relationship. Covariance merely tells us whether variables are positively or negatively correlated, there is no actual meaning attached to the size of the computation result (number indicating covariance). To solve this problem we use another measure known as the correlation.

The correlation is defined as the covariance normalized (divided) by the square root of the product of the variance of each random variable.

The mathematical formula for the definition of correlation is shown below:

Correlation is a dimensionless quantity as the units in the numerator and denominator cancel out. The values for correlation lies in the range -1 to 1. With 1 indicating that there is positive correlation between variables and -1 indicating negative correlation. As a result of the normalizing effect of the denominator when calculating correlation, it gives us a good sense of the degree to which variables are related.

In the figure above, the first plot shows positive correlation between two variables in a 2-dimensional plane. What it means is that as the independent variable on the horizontal axis increases, the dependent variable on the vertical axis also increases. If we trace the set of points, we can see that the direction of movement is upwards. The second plot depicts negative correlation. As the independent variable increases on the x-axis, the dependent variable decreases down the y-axis. Similarly, if we trace the direction of points, we would notice that it tends downwards towards the negative side of the plot. This is how we know that the variables are negatively correlated. Finally, in the last case, we see a plot that has no identifiable patterns, the distribution of both variables are not related to each other. An increase or decrease in one variable does not cause a corresponding shift in the other. We therefore conclude that the third plot shows no correlation between variables.

Let us now see how covariance and correlation can be implemented in Python using Numpy and Scipy.

We would create dummy data using Numpy random function which creates data from a uniform distribution.

import numpy as np

x = np.random.normal(size=2)

y = np.random.normal(size=2)

We stack x and y vertically to produce z using the line of code below.

z = np.vstack((x, y))

The data in now in the correct form and we can pass it to Numpy covariance function.

c = np.cov(z.T)

print(c)

The result may be slightly different in your case because we are generating data points randomly.

To calculate correlation, let us import pearsonr from Scipy stats module and define a very simple dataset. The function imported is the Pearson correlation coefficient.

from scipy.stats.stats import pearsonr  

a = [1,4,6]

b = [1,2,3]

corr = pearsonr(a,b)

print(corr)

We can see that a and b are positively correlated as expressed by the coefficient 0.99, which is very close to 1.

Probability

Dependence and Independence

Probability is a measure of how likely we feel an event would occur. Probability is therefore a measure of likelihood. It is usually a value between 0 and 1 with 0 indicating impossibility, that is the event would never occur and 1 means certainty, the event is sure to occur.

In probability, two events are said to be dependent if the occurrence of the first event directly affects the probability of the second event occurring. This means that dependent events are reliant on each other. For the second event to happen the first must have occurred. Dependent events usually have an order to their occurence. In the same vein, random variables are said to be dependent if the actualization of one directly influences the probability distribution of the other. An example of dependent events are writing a book and getting published. To get published, you must first write a book. The probability of getting published directly depends on writing a book. The order is important as it cannot be changed. Writing a book must occur first before any publication.

Independent events are those events whose probability of occurrence are not dependent on each other. The fact that a first event has occurred does not in any way mean that a second event would occur or not. Both events are not linked as they are independent. To determine whether two events are independent, we first ask ourselves if both events can happen in any order. If the answer is yes, we ask ourselves a second question, does one event affect the outcome of the other. If the answer is no, then we have been able to prove that both events are completely independent of each other. An example of independent events are buying a new phone and eating your favorite meal. Those events are not dependent on each other. It is possible to imagine them occurring in any order. The fact that you just bought a new phone does not in any way affect the probability of you eating your favorite meal.

For two independent events lets say A and B. The probability of event A occurring given that event B has occurred is equal to the probability of A.

P(A|B) = P(A)

What this means is that whether or not event B has occurred, it does not affect the probability of A occurring because the probability of A is only dependent on itself, that is event A does not depend on external events. Similarly, the probability of event B given event A is equal to the probability of event B.

p(B|A) = P(B)

The probability of two independent events occurring is equal to the product of their individual probabilities.

P(A∩B) = P(A)·P(B)

Conditional Probability

Conditional probability can be defined as the measure of the probability of an event, say A occurring, given the knowledge that another event, say B, has occurred. Conditional probability deals with the probability of occurrence of an event in relation to other events. To define this formally, the probability of A given B is equal to the probability of the intersection of A and B (that is both events occur) divided by the probability of B.

P(A|B) = P(A∩B)/P(B)

The term P(A|B) is known as the conditional probability (probability of A given B), the numerator of the right hand side of the equation is the joint probability also called the probability of A and B. It should be noted that for independent events, the conditional probability of an event is equal to the probability of that same event. An understanding of conditional probability is vital as it is one of the fundamental concepts in probability theory and is used in the formulation of more complex concepts as we would see later.

Random Variables

Random variables as the name implies are those types of variables whose values are given by random processes. What this means is random variables maps the outcome of a random process to numbers that can be used in probability theory. An example of a random process is throwing a dice. The outcome is clearly random and cannot be predetermined. However, we can assign numbers to those random outcomes, the numbers so assigned would be quantities of a random variable. Random variables are also called random quantities or stochastic variables and are usually written in capital (uppercase) letters. Random variables are also measurable and contain numbers like regular variables in algebra but the key difference is that they are produced by a random process.

The definition of random variables also make it easier to use a compact notation when talking about random events. Example the probability of getting a number greater than 3 after rolling a dice once would be written as:

P(getting a number greater than 3 after rolling a dice once)

But if we define the random process using random variables, the notation can be simplified greatly:

X = getting a number greater than 3 after rolling a dice once

P(X > 3)

We can see that it becomes easier for us to calculate different outcomes without writing a lot of text. There are two types of random variables - discrete random variables and continuous random variables.

Discrete and Continuous Distributions

Discrete random variables are those random variables that can take on only a specific set of limited values which are all distinct. They are usually countable as they have a finite number of possible values. An example of discrete random variables are the outcomes from a dice. There is only a small set of values that a dice can produce, this makes it countable.

A discrete probability distribution is one that describes the probability associated with discrete random variables. That is it gives the probability of occurrence of discrete outcomes. The probability distribution of a discrete random variable is sometimes called the probability function or the probability mass function.

It would be observed from the above plot of a discrete probability distribution, that the probability of occurrence of a particular value of a random value is non-zero since the range of possibilities is finite. The type of plot above is known as a probability histogram. Examples of discrete probability distributions are binomial, poisson, hypergeometric etc.

Continuous random variables on the other hand can take on any possible value within an interval, that is it can take on an infinite number of values within a range. Continuous random variables are usually acquired through measurement such as the height of a person, weight or the salary of individuals in a company. In all these cases, the value can fall anywhere within a specified range.

A continuous probability distribution is associated with probabilities from continuous random variables. It describes the probability of possible values of a random variable. Continuous random variables are usually defined by an area under a curve and its distribution is non-zero for a range of values.

The probability distribution of random continuous variables are shown by a density curve similar to the one above. The probability of an exact value from the interval range of a continuous random variable is always zero. This is because by definition there are infinite values within the defined interval. The probability of a range can however be calculated by the area under the density curve.

Bayes’ Theorem and Naive Bayes Algorithm

Bayes’ theorem helps us calculate the conditional probability of related events. Bayes’ theorem provides a way for us to update our belief system given the availability of new information or evidence. Using Bayes’ theorem we can calculate the probability of an event A, given event B occurred, as the product of probability of B given A and the probability of A all divided by the probability of B.

P(A|B) = P(B|A)P(A)/P(B)

Where p(A|B) is the conditional probability known as the posterior, that would represent our updated belief about the event occuring. P(B|A) is the likelihood, how likely is B a result of event A. P(A) is the prior, that is our understanding of the situation before we observed any data and P(B) is known as the evidence.

Bayes’ theorem is a powerful formulation because it allows us to get the conditional probabilities of events and update that probability (how likely is the event to occur) once we have new information or data. We first start with prior knowledge which in a way is a biased form of what is currently known. At each iteration, we can then update the posterior estimate using components of Bayes equation such as the likelihood, prior and evidence. Bayes’ theorem is central to Bayesian statistics and has a wide range of applications.

Let us take a simple example that beautifully illustrates Bayes theorem. Say we draw a single card from a deck of playing cards, what is the probability that the card so drawn is a king, given evidence (additional information) whether it is a face card.

First let us define define Bayes theorem in line with the question.

P(King|Face) = P(Face|King)P(King)/P(Face)

Where;

P(King|Face) = probability the card is a king given it is a face card

P(Face|King) = probability the card is a face card given it is a king

P(King) = probability the card is a king

P(Face) = probability the card is a face card

Next we compute these probabilities and plug them into Bayes theorem.

The probability a drawn card is a king is 4/52, which is 1/13 because there are 4 kings and the total number of cards is 52.

Let’s assume evidence is provided in the form of someone looking at the card. The person in this case confirms that the card is a face card. P(Face|King) becomes 1 because all kings are face cards (contains a face).

The last component is P(Face), there are 3 face types - Jack, Queen, King each of which has 4 cards, so P(Face) is 12/52 which reduces to 3/13.

We now have all the components and can now calculate the P(King|Face).

P(King|Face) = P(Face|King)P(King)/P(Face)

P(King|Face) = (1/13)(1)/(3/13) = 1/3

What P(King|Face) indicates is a way we can update our beliefs based on new evidence. As a result of new evidence that the card drawn is a face card, the probability that the card is a king given a face card jumps to ⅓ as compared with 1/13, which is the prior probability (when we had not observed evidence) that it was a king (P(King)). This example shows us that Bayes theorem gives us a way of calculating the posterior probability when we know the prior probability, the likelihood and evidence.

Naive Bayes algorithm is an application of Bayes’ theorem as a classification algorithm with the explicit assumption that all features or predictors are independent. The word “naive” in its name is because of the independence assumption since we know that this is not always true and features tend to be related.

Naive Bayes algorithm relies on Bayes theorem which is stated mathematically as follows.

With the independence assumption that all input features are unrelated, the numerator can be expressed as:

Using the independence representation, Bayes theorem can be simplified to a product of probabilities.

However, in our model, the input data remains constant, therefore the denominator has no effect on the model. We can choose to ignore it. Another way of thinking about it is that there is no y term in the denominator, so it does not help us in any way to predict output classes. The formula then becomes a direct variation as shown below:

The final step is to cast it as an optimization operation where we need to maximize the probability of the correct class given the correct class and each input feature independently. The final formula for Naive Bayes algorithm becomes:

Despite its simplicity Naive Bayes algorithm is a very powerful classifier and is particularly useful as a multi-classification model. Naive Bayes is also very fast to run and performs very well when its independence assumption holds true compared to other algorithms. A popular use case of the Naive Bayes classifier is in email spam detection where the words of an email are regarded as independent features and the algorithm learns to categorize email into spam or not spam categories based on the content.

The Data Science Process

Asking the Right Question

The data science process starts with a simple premise - that there is a problem we want solved or insights we want to discover. This leads us to the very first step of any data science project. We need to understand what we are trying to achieve at a very deep level before we embark on the journey. This is encapsulated in asking the right question. What is the goal of this research endeavour, how does it benefit the business, in what ways would this provide better customer experience. Other questions include asking ourselves if there is a shortcoming that has been observed which we would like to find out more about.

Getting this stage right is the most important aspect of a data science project as all other steps flow from it. We do not want to have an ill defined problem, waste money and resources to prototype a supposed solution, only to discover that our solution does not solve the problem at hand simply because we did not understand what was required in the first place. It is therefore desirable that we explore as many hypotheses as possible and pit them against each other until we can narrow down the problem to a single question or research goal.

Some common questions in a data science project could be - who are our most valuable customers, what impact would changing our product have on customer satisfaction, can our data on customers help us predict their actions etc. If we look closely at these questions, we would notice a common trend. They all heavily involve knowledge about the business, that is they are all specific about our business domain. To formulate appropriate questions which would serve as the springboard for our data science projects, data scientists need not act in isolation because assumption at this stage is dangerous. The proper thing would be to involve those with a profound understanding of the business in the discussions by having a round table session where managers, marketers, customer service personnel etc elaborate on challenges that they may be facing. It is now the job of the data scientist to identify the underlying need and formulate that into a question that can be answered by data. If this is done correctly, everyone on the team knows what is expected or desired even before the project begins and this enables everyone to be on the same page and moderate their expectations.

It is also at this stage that a metric of success is set or put another way, how would we measure the success of the project. What would be an acceptable threshold and what is not. The measurement metric chosen is usually a direct reflection of how the data science question is framed. For example, accuracy could be selected as how the team would evaluate the final performance of the project. It is also important for everyone on board to understand that data science is not a silver bullet and there may be unforeseen circumstances that arise along the way.

Below is a schematic diagram of how the data science process looks. In the following sections we would explain each block in detail and show how they fit into a grand scheme.

Data Acquisition

Immediately after we settle on a research goal, the next step is to acquire appropriate data through which we can begin to derive insights. The kind of data that is acquired is tailored towards the kind of problem we want to solve. By getting that particular type of data, we are making an assumption that the solution space provided by the data contains answers to our questions. We need to identify appropriate datasets if they already exist or most likely create one ourselves. The data acquisition step may include sourcing for data within the organization or leveraging external sources of data.

Data may come from relational databases, spreadsheets, inventories, external APIs etc. During this stage, it is reasonable to check that the data is in the correct format for our purposes. In-depth checks may not be required at this point, we simply confirm that data sources are indeed what they claim to be.

Data Preparation

Data preparation involves three main mini-steps. Data cleansing, data transformation and data combination. As a whole, the data preparation step changes the raw data which was obtained from the real world to a form where it can be read and analyzed by a computer, in this case a machine learning algorithm.

First, we clean the datasets we have obtained. This is usually done by identifying missing values, errors, outliers etc. Most machine learning algorithms cannot handle missing values so it is required that they are replaced in some form or those observations associated with them be removed. For a small dataset, it is unreasonable to throw away observations, so we adopt a strategy for data imputation such as replacing missing values by averages or most occurring values (mode) for each feature. Errors from data entries may also be spotted when we notice impossible values for a particular feature example an age of 400 years. Outliers, which are data points that are so far from the observed distribution are cleaned up in this phase.

Data transformation is centered on aggregating data, dealing with categorical variables, creating dummies to ensure consistency, reducing the number of variables in order to retain the most informative features and discard redundant features, scaling the data etc. Scaling is important as features may originally be in different ranges and to get better performance, it is often sensible to bring all variables to a common range.

The data from different datasets may then be combined by merging or joining. New data views may be created depending on the type of problem, so that it is presented to the learning algorithm in a structure that is different from its original stored representation.

Data Exploration

The data exploration stage is concerned with looking closely at the data to get a better sense of what the data is all about. This step involves using statistical metrics such as mean, median, mode, variance etc to understand the data distribution. Information displayed in pictorial form is often easier to digest and as a result it is not uncommon to notice that this step includes many visualization techniques. Some common visualization techniques used in exploratory data analysis phase includes, bar charts, pie charts, histograms, line graphs, box plots etc.

Below are some common examples of visualization techniques used in this stage.

The left image is a pie chart while the image on the right is a simple histogram. The advantage of using visualization becomes obvious as we would be able to easily identify anomalies in our data and have a better mental representation of what our data contains. Sometimes anomalies in the data are noticed through exploratory data analysis and corrected by going back to the previous step - data preparation. Another important benefit of data exploration, is that it enable us discover patterns which we may combine with domain knowledge to create (engineer) new informative features.

Data Modelling

In the data modelling step we take a more involved approach when accessing the data. Data modelling involves choosing an algorithm, usually from the related fields of statistics, data mining or machine learning. Deciding which features should go into the algorithm as the input, executing the model and finally evaluating the trained model for performance.

Before feeding in data to any model, we first chose the most important features as inputs to the model. These would be features that offer the most discriminability. What that means is that we would give preference to features that contain underlying properties that enables our model learn its task better, whether that is classification or regression for example. We choose features that show the most variability across our data distribution. Those features which do not drive the final prediction or are uninformative are dropped. Techniques such as Principal Component Analysis (PCA) can be used to identify important features known as principal components. These are components along whose axes variability can be observed.

The next step involves choosing an appropriate algorithm for the learning task. Different algorithms are better suited to different learning problems. Logistic regression, Naive Bayes classifier, Support Vector Machines, decision trees and random forests are some popular classification algorithms with good performance. Linear regression and neural networks may be used for regression tasks. There are a variety of modelling algorithms and we often do not know the best algorithms before we have tried them on the dataset. It is therefore important to keep an open mind and rely heavily on experimentation.

In order to be able to support experimentation without biasing our models in the process, we adopt an evaluation scheme. Remember that the aim of modelling is to produce models that can generalize to real world data. We do not want our model to memorize the data that it was trained on. What we desire is for the model to find interesting patterns which would help it explain the structure of the data and enable it answer questions in the real world. To do this we never train our model on the entire data we have acquired and preprocessed. We divide the data into three sections known as splits. The three splits are train, validation and test sets. The train split is used for training the model while the validation split is used for hyperparameter tuning. Hyperparameter tuning means adjusting the hyperparameters (input options) available to an algorithm and checking its performance on the validation set to see whether the adjusted hyperparameter had a positive effect or not. Hyperparameter tuning is done on the most promising models to further improve performance. Finally, the trained model is evaluated on the test set which was not used for training or validation. In other words, the model has never seen the data contained in the test set. This provides an unbiased assessment of the model as we can observe its generalization to new data. The performance on the test set is normally reported as the final evaluation of the model.

The evaluation metric on the train and validation splits enable us to debug the model to discover whether it is underfitting or overfitting to the training set. If it is underfitting (not learning enough), we can increase the power of the model else we apply regularization if it is overfitting (learning noise). The concepts of overfitting and underfitting would be explained further in the next chapter.

Data Presentation

The last stage is all about presenting our findings in a form that is intuitive and understandable to non-technical professionals such as managers, marketers or business leaders. The importance of this step cannot be overemphasized as it is the crowning jewel of the data science process. Presentation is usually done by leveraging visualizations and tables. The purpose of this step is to communicate the insights discovered from the data science process in such a way that the information provided is actionable. This means data presentation should enable a decision making process. It should be clear from the presentation what steps need to be taken to solve the original problem which was posed as a question in the first step. It may also be desirable to automate the process as the insights produced may be so valuable that they need to be returned to regularly. Another possible outcome is bundling the model into a data product or application that is served to end users. To do this, the model would need to be optimized for production and deployed in a scalable fashion.

Machine Learning

What is Machine Learning

Machine learning has recently been attracting attention in the media for several reasons, mainly because it has achieved impressive results in various cognitive tasks such as image classification, natural language understanding, customer churn prediction etc. However, it has been regarded as some sort of magic formula that is capable of predicting the future, but what really is machine learning. Machine learning at its simplest form is all about making computers learn from data by improving their performance at a specific task through experience. Similar to the way humans learn by trying out new things and learning from the experience, machine learning algorithms improve their capability by learning patterns from lots of examples. The performance of these algorithms, generally improves as they are exposed to more data (experience). Machine learning is therefore a branch of artificial intelligence that aims to make machines capable of performing specific tasks without being explicitly programmed. What this means is that these algorithms are not rule-based, the entire learning process is constructed in such a way as to minimize or completely eliminate human intervention.

Machine learning algorithms are typically used for a wide range of learning problems such as classification, regression, clustering, similarity detection etc. Many applications used in the real world today are powered by machine learning. Applications such as personal assistants on mobile phones use machine learning algorithms to understand voice commands spoken in natural language, mobile keyboards predict the next word a user is typing based on previous words, email clients offer a smart reply feature whereby the content of an email is scanned and appropriate responses are generated, e-commerce applications offer recommendation to users based on previous purchases and spending habits etc. Nearly every industry would be impacted by machine learning as most processes can be automated given that there is enough training data available. Machine learning algorithms mostly excel in tasks where there is a clear relationship between a set of inputs and outputs which can be modelled by training data. Although machine learning is a rapidly improving field, there is as of now no notion of general intelligence of the form displayed by humans. This is because models trained on one task cannot generalize the knowledge gleaned to perform another task, that is machine learning algorithms learn narrow verticals of tasks.

There are three main branches of machine learning namely - supervised learning, unsupervised learning and reinforcement learning. In some cases a fourth branch is mentioned - semi-supervised learning but this is really just a special instance of supervised learning as we would see in the explanations below.

Supervised Learning Algorithms

Supervised learning is by far the most common branch of machine learning. Most of the real world value currently in the field of machine learning can be attributed to supervised learning. Supervised learning algorithms are those machine learning algorithms which are trained with labelled examples. It would be remembered that we defined machine learning as making algorithms that learn from data (examples) without being explicitly programmed. The main intuition to understand when dealing with supervised learning algorithms is that, they learn through the use of examples that are clearly annotated to show them what they are supposed to learn. The algorithms therefore try to find a mapping representation from inputs to outputs using the labels as a guide. “Supervised” in the name of these types of algorithms, point to the fact that the labels or targets provide supervision throughout the learning process. It is therefore possible for the algorithm to check its prediction against actual values stored in the labels. It then uses this error information (how far off its prediction was from the actual label) to slowly improve its performance with each iteration. The targets in a supervised learning problem can be seen as a supervisor providing feedback to the algorithm on areas where it can improve its performance. The two main applications of supervised learning algorithms are classification and regression.

Classification involves training a learning algorithm to correctly separate examples into predefined categories or classes. The classes are usually chosen ahead of time by a human expert with domain knowledge in the field where the learning problem is posed. The examples that are used to train the model are clearly labelled to indicate the category they belong to. During training, the supervised learning algorithm, uses the labels to guide its learning and at test time it is capable of correctly predicting the categories of new examples. A popular example of classification is spam detection where an email is correctly identified to belong to one of two classes - spam or not spam. Depending on what is predicted, appropriate action could be taken such as shifting spam emails to a spam folder while relevant emails are sent to a user’s inbox.

Regression is a learning problem where the algorithm is interested in predicting a single real value number. Regression is used where a single numeric entity is to be predicted. An example of regression would be predicting the age of a person given a profile picture or predicting the salary of an individual given information about the individual such as level of education, work experience, age, country of residence etc. It would be observed that in both cases the final prediction is a single number.

Supervised learning algorithms are easier to train when compared to unsupervised or reinforcement learning algorithms. This is because the presence of labels simplify the learning problem since there exist a clear way of determining performance during training. However, it should be noted that most supervised learning problems can also be modelled as unsupervised if we get rid of labels. Datasets for supervised learning are more expensive to acquire as it requires meticulous human annotation of examples. The fact that most data in the world today are in an unlabelled form makes the research of unsupervised learning algorithms particularly important.

Unsupervised Learning Algorithms

Unsupervised learning involves learning directly from raw data. This type of learning takes place without the presence of a supervisor in the training loop in form of labels. Unsupervised learning algorithms are free to explore the underlying data distribution and come up with patterns that best describe the entire dataset. The training process is not guided by humans through labelled examples and as such unsupervised learning algorithms are more powerful as they can discover patterns which domain experts may not have thought of. It is however still the job of domain experts to understand the patterns so discovered and explain them because unsupervised learning algorithms do not truly have the sense of reasoning which we would ascribe to humans. Unsupervised learning algorithms merely use the data distribution or its latent (hidden) representations to unearth insights which may be in the form clusters, groups or distributions.

There are many applications of unsupervised learning algorithms such as clustering, dimensionality detection, generative models etc. Clustering is one of the popular implementations of unsupervised learning. It involves the automatic discovery of groups (clusters) of data points from raw data. Members of a group share similar features, that is they are alike. They can be thought of as belonging to the same type of entity whereas as a group they are dissimilar to other groups. Groups usually have semantic meaning which can be further explored to understand the dataset. An example of clustering would involve grouping customers of a tv streaming service into the type of shows that they watch. Users with similar interests would generally be found in the same group. This is a very powerful application as new tv shows could be recommended to users based on other users who share their interests, leading to greater engagement on the platform and increased revenues.

Dimensionality reduction is a machine learning technique that reduces the number of attributes (features) fed in a model to only the most relevant ones which drive predictions. It has been observed empirically, that models with greater number of features or dimensions perform worse on generalization. That is to say that their performance suffers in the real world. By reducing the number of dimensions, a model can learn from informative features which enables it to develop valid representations about the data that aids prediction. Principal component Analysis (PCA) is probably the most popular dimensionality reduction technique and is an example of an unsupervised learning algorithm. PCA reduces the dimensions of data by identifying those axes that contain the most variability. What that means is that it discovers principal components which offer the most discriminative features.

Generative models are another popular instance of unsupervised learning algorithms. They have made recent headlines because of their ability to artificially generate photographs and works of art that look realistic. Generative Adversarial Networks (GANs) currently produce state of the art results across many image generation benchmarks and are among the most popular examples of generative models.

Semi-supervised Learning Algorithms

Semi-supervised learning algorithms are a special case of supervised learning algorithms. In semi-supervised learning, while there isn’t an explicit label, there exist an implicit heuristic which serves as a supervisor in the training loop. Semi-supervised models do not contain any external source of labels but only rely on input features. However, the learning task is set up in such a way that supervision still takes place in the form of extraction of pseudo-labels from inputs through a heuristic algorithm. A popular example of semi-supervised learning algorithms are autoencoders. Let us look at an example to expand our understanding.

In the autoencoder above, the learning task is to reduce the dimensions of the input into a smaller latent space representing the most important hidden features, then reconstruct the input from this lower dimensional space. So given an input, example an image, an autoencoder shrinks the image into a smaller latent representation that still contains most of the information about the image, then reconstructs the original input image from this low dimensional space. Even if there are no explicit labels, it would be observed that the input serves as the supervisor since the learning task is to reconstruct the input. Once such a model is trained to compress features into a smaller dimension, the compressed features can serve as the starting point of a supervised learning algorithm similar to dimensionality reduction using PCA. The first part of the network that reduces the dimensions of the input is called an encoder while the second part that scales the encoded features back to the full size input is called the decoder.

Reinforcement Learning Algorithms

In reinforcement learning there are three main components, an agent, an environment and actions. The goal of reinforcement learning is to train an intelligent agent that is capable of navigating its environment and performing actions that maximizes its chances of arriving at some end goal. Actions carried out by the agent change the state of the environment and rewards or punishment may be issued based on the actions taken by the agent. The challenge is for the agent to maximize the accumulated rewards at the end of a specific period so that it can actualize an end goal (objective).

In the schematic diagram of reinforcement learning above, an agent (the reinforcement learning) interacts with the world (environment) through actions. The environment provides observations and rewards to the agent based on the kind of action taken by the agent. The agent uses this feedback to improve its decision making process by learning to carry out actions associated with positive outcomes.

Overfitting and Underfitting

Overfitting and underfitting jointly form the central problem of machine learning. When training a model we want to improve its optimization by attaining better performance on the training set. However, once the model is trained, what we care about is generalization. Generalization in a nutshell deals with how well a trained machine learning model would perform on new data which it has not seen, that is data it was not trained on. In other words, how well can a model generalize the patterns it learnt on the training set to suit real world examples, so that it can achieve similar or better performance. This is the crux of learning. A model should be able to actually learn useful representations from data that improves test time performance and not merely memorize features as memorization is not learning.

We say a model has overfit to a training set when it has failed to learn only useful representations in the data but has also adjusted itself to learn noise in order to get an artificially high training set accuracy. Underfitting means that the model has not used the information available to it but has only learnt a small subset of representations and has thrown away majority of useful information, thereby making it to make unfounded assumptions. The ideal situation is to find a model that neither underfitts nor overfitts but exhibits the right balance between optimization and generalization. This can be done by maintaining a third set of examples known as the validation set. The validation set is used to tune (improve) the performance of the model without overfitting the model to the training set. Other techniques for tackling overfitting includes applying regularization which punishes more complicated models and acquiring more training examples. Underfitting can be stymied by increasing the capacity of the learning algorithm so that it can take advantage of available features.

The plots above show three simple line based classification models. The first plot separates classes by using a straight line. However, a straight line is an overly simplistic representation for the data distribution and as a result it misclassified many examples. The straight line model is clearly underfitting as it has failed to use majority of the information available to it to discover the inherent data distribution.

The second plot shows an optimal case where the optimization objective has been balanced by generalization criterion. Even though the model misclassified some points in the training set, it was still able to capture a valid decision boundary between both classes. Such a classifier is likely to generalize well to examples which it was not trained on as it has learnt the discriminative features that drive prediction. The last plot illustrates a case of overfitting. The decision boundary is convoluted because the classifier is responding to noise by trying to correctly classify every data point in the training set. The accuracy of this classifier would be perfect on the training set but it would perform horribly on new examples because it optimized its performance only for the training set. The trick is to always choose the simplest model that achieves the greatest performance.

Correctness

To evaluate a machine learning algorithm, we always specific measures of predictive performance. These metrics allow us to judge the performance of a model in an unbiased fashion. It should be noted that the evaluation metric chosen depends on the type of learning problem. Accuracy is a popular evaluation metric but it is not suitable for all learning problems. Other measures for evaluation include recall, precision, sensitivity, specificity, true positive rate, false positive rate etc. The evaluation used should be in line with the goals of the modelling problem.

To ensure fidelity of reported metrics, a rule of thumb is that models should never be trained on the entire dataset as any evaluation reported by metrics is likely skewed because the model’s performance when exposed to new data is unascertained. The dataset should be divided into train, validation and test splits. The model is trained on the training set, the validation set is reserved for hyperparameter tuning for best performing models and the test set is only used once at the conclusion of all experimentation.

A confusion matrix is widely used as a simple visualization technique to access the performance of classifiers in a supervised learning task. It is a table where the rows represent the instances in the actual class (ground truth) while the columns represents predictions. The order may be reversed in some cases. It is called a confusion matrix because it makes it easy to see which classes the model is misclassifying for another, that is which classes confuse the model.

The examples which the model correctly classified are on the diagonal from the top left to bottom right. False negatives are positive classes which the classified wrongly predicted as negatives while false positives are negative instances which the classifier wrongly thought were positives. Several metrics like true positive rate, false positive rate, precision etc are derived from items in the confusion matrix.

The Bias-Variance Trade-off

The bias of a model is defined as the assumptions made by the model to simplify the learning task. A model with high bias makes assumptions which are not correlated by the data. This lead to errors because predictions are usually some way off from actuals. Variance on the other hand is how susceptible a model is to noise in the training data. How widely does the performance on the model vary based on the data it is evaluated on. A good machine learning algorithm should strive to achieve low bias and low variance. Bias and variance are related to overfitting and underfitting earlier encountered. A model with high bias is underfitting the training data because it has made simplistic assumptions instead of learning from information available. Similarly, a model with high variance is overfitting, because it has modelled noise and as a result, its performance would vary widely across the training set, validation set and test set.

Let us look use a dart analogy to further explain these concepts.

The top left image represents a model that has low bias and low variance. This is the ideal model as it has learnt to hit the target (produce correct classification) and usually hits the target most of the time (does not vary with each throw). The image at the top right shows a model that exhibits high variance and low bias. Even if it does not make a lot of assumptions, its predictions are spread all over the board which means its performance varies widely (high variance). The image on the bottom left depicts a model with high bias and low variance. The shots are not all over the board but in a specific location. This location is however far from the target meaning the model is biased because of simplistic assumptions. Finally, the image on the bottom right shows a model with high bias and high variance. The shots on the board vary widely and are far away from the target. This is the worst kind of model as it hasn’t learnt any useful representation.

Feature Extraction and Selection

Feature extraction involves performing transformation on input features that produce other features that are more analyzable and informative. Feature extraction may occur by combining original features to create new features which are better suited for the modelling problem. This is similar to feature engineering where we create new features to fed into a model. An example of feature extraction is Principal Component Analysis (PCA).

Feature selection is choosing a subset of features from the original input features. The features selected are those that show the most correlation with the target variable, that is those features that drive the predictive capability of the model. Both feature extraction and feature selection leads to dimensionality reduction. The main difference between them is that feature extraction is a transformation that creates new features whereas feature selection chooses only a subset of available features. Since feature selection removes certain features, it is always advisable to first do feature extraction on a dataset, then select the most important predictors via feature selection.

Why Machine Learning is Popular

The popularity of machine learning in the artificial intelligence community and society in general is mainly because machine learning techniques have proven to be highly successful in various niches providing business value for a slew of operations from fraud detection to speech recognition to recommender systems. It is embedded in products we use every day. When you buy a product from Amazon and you are given a list of suggestions of other products that go with it, that’s machine learning in action. When you open your mailbox and emails are automatically classified into folders based on their similarity, those are machine learning models doing the work behind the scenes. Even when you use your credit card online and your transaction is successful, a machine learning model approved your transaction as being normal and not fraudulent.

In light of all these, companies and organizations have poured in more money into developing better performing models through research and collaboration between industry and academia.

It wasn’t always the case that machine learning was the darling of the computer science community, however in recent years three factors have conspired to give it an exalted place.

●    With the dawn of the internet, more data was being collected and this lead to the age of big data.

●    Computational resources became faster and cheaper with the arrival of Graphics Processing Units (GPUs), which were originally developed by the gaming industry to render graphics but are well suited for parallel computation.

●    Better algorithms were developed which lead to better accuracy of models.

Python Data Science Tools

Jupyter Notebook

Data science is an experimental endeavour as hypothesis needs to be tested out to see whether the data upholds them. Jupyter notebook is an interactive web application that bundles an IPython shell, which is an enhanced shell for Python computations. Code, text and visualizations can be written alongside each other and stored in a special notebook format. This means that projects can be shared easily across teams and code can be easily reproduced as every functionality that is required to execute a project is bundled in the notebook. Jupyter notebooks supports many programming languages like R, Julia, Python etc. Jupyter in Python are powered by the IPython shell which provides additional “magic” functions that are not available in Python but are generally useful for interactive computation.

Jupyter notebooks also supports Latex and as a result mathematical equations can be written inline in cell blocks. Data visualization is also possible inline and detailed documentation or explanations can be incorporated in a notebook by using markdown syntax.

IPython and Jupyter notebook are automatically installed with Anaconda. To access the Jupyter notebook interface run the following command from a terminal.

$ jupyter notebook # command to launch jupyter notebook web interface

The command above, opens a window in a browser session containing the Jupyter notebook web interface. The Jupyter notebook session is opened in the same directory where the command is run.

You can navigate to a notebook file and click on it to run it or a create a new notebook from the interface.

Once a new notebook is created, it launches a new instance from which coding can be carried out interactively.

Jupyter notebooks are very popular in the fields of data science and machine learning as they offer a specialized format that encapsulates coding, visualization and documentation.

Numerical Python (Numpy)

Numpy is the main numerical computing library in Python. It provides access to a multidimensional array object and exposes several methods through which operations can be done on these arrays. It features support for linear algebra operations such as matrix multiplication, inner product, identity operations etc. Numpy interfaces with low level libraries written in C and Fortran and passes off actual computation to these faster and more efficient libraries. As a result, vanilla Python control structures like loops are never used when performing numerical computation because they are significantly slower. Numpy can be seen as providing a set of Python APIs which enables efficient scientific computing.

Numpy arrays can be initiated by nested Python lists. The level of nesting specifies the rank of the array.

import numpy as np

a = np.array([[1, 2, 3], [4, 5, 6]])   # create a rank 2 array

print(type(a))

print(a.shape)

The array created is of rank 2 which means that it is a matrix. We can see this clearly from the size of the array printed. It contains 2 rows and 3 columns hence size (m, n).

Arrays can also be initialized randomly from a distribution such as the normal distribution. Trainable parameters of a model such as the weights are usually initialized randomly.

b = np.random.random((2,2))  # create an array filled with random values

print(b)

print(b.shape)

Numpy contains many methods for manipulation of arrays, one of such is matrix product. Let us look at an example of matrix product using Numpy.

x = np.array([[1,2],[3,4]])

y = np.array([[5,6],[7,8]])

# matrix product

print(np.dot(x, y))

The example above is computed almost instantly and shows the power of Numpy.

Pandas

Pandas is a data manipulation library written in Python which features high performance data structures for table and time series data. Pandas is used extensively for data analysis and most data loading, cleaning and transformation tasks are performed in Pandas. Pandas is an integral part of the Python data science ecosystem as data is rarely in a form that can be fed directly into machine learning models. Data from the real world is usually messy, contains missing values and in need of transformation. Pandas supports many file types like CSV, Excel spreadsheets, Python pickle format, JSON, SQL etc.

There are two main types of Pandas data structures - series and dataframe. Series is the data structure for a single column of data while a dataframe stores 2-dimensional data analogous to a matrix. In other words, a dataframe contains data stored in many columns.

The code below shows how to create a Series object in Pandas.

import pandas as pd

s = pd.Series([1,3,5,np.nan,6,8])

print(s)

To create a dataframe, we can run the following code.

df = pd.DataFrame(np.random.randn(6,4), columns=list('ABCD'))

print(df)

Pandas loads the file formats it supports into a dataframe and manipulation on the dataframe can then occur using Pandas methods.

Scientific Python (Scipy)

Scipy is a scientific computing library geared towards the fields of mathematics, science and engineering. It is built on top of Numpy and extends it by providing additional modules for optimization, technical computing, statistics, signal processing etc. Scipy is mostly used in conjunction with other tools in the ecosystem like Pandas and matplotlib.

Here is a simple usage of scipy that finds the inverse of a matrix.

from scipy import linalg

z = np.array([[1,2],[3,4]])

print(linalg.inv(z))

Matplotlib

Matplotlib is a plotting library that integrates nicely with Numpy and other numerical computation libraries in Python. It is capable of producing quality plots and is widely used in data exploration where visualization techniques are important. Matplotlib exposes an object oriented API making it easy to create powerful visualizations in Python. Note that to see the plot in Jupyter notebooks you must use the matplotlib inline magic command.

Here is an example that uses Matplotlib to plot a sine waveform.

# magic command for Jupyter notebooks

%matplotlib inline

import matplotlib.pyplot as plt

# compute the x and y coordinates for points on a sine curve

x = np.arange(0, 3 * np.pi, 0.1)

y = np.sin(x)

# plot the points using matplotlib

plt.plot(x, y)

plt.show()  # Show plot by calling plt.show()

Scikit-Learn

Scikit-Learn is the most popular machine learning library in the Python ecosystem. It is a very mature library and contains several algorithms for classification, regression and clustering. Many common algorithms are available in Scikit-Learn and it exposes a consistent interface to access them, therefore learning how to work with one classifier in Scikit-Learn means that you would be able to work with others as the names of the methods that are called to train a classifier are the same regardless of the underlying implementation.

We would rely heavily on Scikit-Learn for our modelling tasks as we dive deeper into data science in the following sections of this book. Here is a simple example of creating a classifier and training it on one of the bundled datasets.

# sample decision tree classifier

from sklearn import datasets

from sklearn import metrics

from sklearn.tree import DecisionTreeClassifier

# load the iris datasets

dataset = datasets.load_iris()

# fit a CART model to the data

model = DecisionTreeClassifier()

model.fit(dataset.data, dataset.target)

print(model)

# make predictions

expected = dataset.target

predicted = model.predict(dataset.data)

# summarize the fit of the model

print(metrics.classification_report(expected, predicted))

print(metrics.confusion_matrix(expected, predicted))

Here is the output. Do not worry if you do not understand the code. We would go through each part of the code in more detail in subsequent sections.

K-Nearest Neighbors

Introduction to K Nearest Neighbors

To understand the k-nearest neighbor algorithm, we first need to understand nearest neighbor. Nearest neighbor algorithm is an algorithm that can be used for regression and classification tasks but is usually used for classification because it is simple and intuitive.

At training time, the nearest neighbor algorithm simply memorizes all values of data for inputs and outputs. During test time when a data point is supplied and a class label is desired, it searches through its memory for any data point that has features which are most similar to the test data point, then it returns the label of the related data point as its prediction. A Nearest neighbor classifier has very quick training time as it is just storing all samples. At test time however, its speed is slower because it needs to search through all stored examples for the closest match. The time spent to receive a classification prediction increases as the dataset increases.

The k-nearest neighbor algorithm is a modification of the nearest neighbor algorithm in which a class label for an input is voted on by the k closest examples to it. That is the predicted label would be the label with the majority vote from the delegates close to it. So a k value of 5 means, get the five most similar examples to an input that is to be classified and choose the class label based on the majority class label of the five examples.

Let us now look at an example image to hone our knowledge:

The new example to be classified is placed in the vector space, when k = 1, the label of the closest example to it is chosen as its label. In this case, the new example is categorized as belonging to class 1. When k = 1, k-nearest neighbor algorithm is reduced to nearest neighbor algorithm.

From the image, when k = 3, we choose the 3 closest examples to the new example using a similarity metric known as the distance measure. We see that two close examples predict the class as being class 2 (red triangle) while the remaining example predicts the class to be class 1 (blue square). The predicted class of the new data point is therefore class 2 because it has the majority vote.

The distance metric used to measure proximity of examples may be L1 or L2 distance. L1 distance is the sum of the absolute of the difference between two points and is given by:

L1 distance is sometimes referred to as the Manhattan distance.

L2 is an alternative distance measure that may be used. It is the Euclidean distance between two points using a straight line. It is given as:

A value of k = 1 would classify all training examples correctly since the most similar example to a point would be itself. This would be a sub-optimal approach as the classifier would fail to learn anything and would have no power to generalize to data that it was not trained on. A better solution is to choose a value of k in a way that it performs well on the validation set. The validation set is normally used to tune the hyperparameter k. Higher values of k has a smoothing effect on the decision boundaries because outlier classes are swallowed up by the voting pattern of the majority. Increasing the value of k usually leads to greater accuracy initially before the value becomes too large and we reach the point of diminishing returns where accuracy drops and validation error starts rising.

The optimal value for k is the point where the validation error is lowest.

How to create and test the K Nearest Neighbor classifier

We would now apply what we have learnt so far to a binary classification problem. The dataset we would use is the Pima Indian Diabetes Database which is a dataset from the National Institute of Diabetes and Digestive and Kidney Diseases. The main purpose of this dataset is to predict whether a patient has diabetes or not based on diagnostic measurements carried out on patients. The patients in this study were female, of Pima Indian origin and at least 21 years old.

The dataset can be found at:

https://www.kaggle.com/uciml/pima-indians-diabetes-database/data

Since we are dealing with two mutually exclusive classes, a patient either has diabetes or not, this can be modelled as a binary classification task and for the purpose of our example we would use the k-nearest neighbor classifier for classification.

The first step is to import the libraries that we would use.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

Next we load the data using Pandas.

dataset = pd.read_csv('diabetes.csv')

As always what we should do is get a feel of our dataset and the features that are available.

dataset.head(5)

We see that we have 8 features and 9 columns with Outcome being the binary label that we want to predict.

To know the number of observations in the dataset we run

dataset.shape

This shows dataset contains 768 observations.

Let’s now get a summary of the data so that we can have an idea of the distribution of attributes.

dataset.describe()

The count row shows a constant value of 768.0 across features, it would be remembered that this is the same number of rows in our dataset. It signifies that we do not have any missing values for any features. The quantities mean, std gives the mean and standard deviation respectively across attributes in our dataset. The mean is the average value of that feature while the standard deviation measures the variation in the spread of values.

Before going ahead with classification, we check for correlation amongst our features so that we do not have any redundant features

corr = dataset.corr()    # data frame correlation function

fig, ax = plt.subplots(figsize=(13, 13))

ax.matshow(corr)   # color code the rectangles by correlation value

plt.xticks(range(len(corr.columns)), corr.columns)  # draw x tick marks

plt.yticks(range(len(corr.columns)), corr.columns)  # draw y tick marks

The plot does not indicate any 1 to 1 correlation between features, so all features are informative and provide discriminability.

We need to separate our columns into features and labels

features = dataset.drop(['Outcome'], axis=1)

labels = dataset['Outcome']

We would once again split our dataset into training set and test set as we want to train our model on the train split, then evaluate its performance on the test split.

from sklearn.model_selection import train_test_split

features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.25)

features_train, features_test contains the attributes while labels_train, labels_test are the discrete class labels for the train split and test splits respectively. We use a test_size of 0.25 which indicates we want to use 75% of observations for training and reserve the remaining 25% for testing.

The next step is to use the k-nearest neighbor classifier from Scikit-Learn machine learning library.

# importing the model

from sklearn.neighbors import KNeighborsClassifier

classifier = KNeighborsClassifier()

The above code imports the k-nearest neighbor classifier and instantiates an object from it.

classifier.fit(features_train, labels_train)

We fit the classifier using the features and labels from the training set. To get predictions from the trained model we use the predict method on the classifier, passing in features from the test set.

pred = classifier.predict(features_test)

In order to access the performance of the model we use accuracy as a metric. Scikit-Learn contains a utility to enable us easily compute the accuracy of a trained model. To use it we import accuracy_score from metrics module.

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(labels_test, pred)

print('Accuracy: {}'.format(accuracy))

We obtain an accuracy of 0.74, which means the predicted label was the same as the true label for 74% of examples.

Here is the code in full:

# import libraries

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

# read dataset from csv file

dataset = pd.read_csv('diabetes.csv')

# display first five observations

dataset.head(5)

# get shape of dataset, number of observations, number of features

dataset.shape

# get information on data distribution

dataset.describe()

# plot correlation between features

corr = dataset.corr()    # data frame correlation function

fig, ax = plt.subplots(figsize=(13, 13))

ax.matshow(corr)   # color code the rectangles by correlation value

plt.xticks(range(len(corr.columns)), corr.columns)  # draw x tick marks

plt.yticks(range(len(corr.columns)), corr.columns)  # draw y tick marks

# create features and labels

features = dataset.drop(['Outcome'], axis=1)

labels = dataset['Outcome']

# split dataset into training set and test set

from sklearn.model_selection import train_test_split

features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.25)

# import nearest neighbor classifier

from sklearn.neighbors import KNeighborsClassifier

classifier = KNeighborsClassifier()

# fit data

classifier.fit(features_train, labels_train)

# get predicted class labels

pred = classifier.predict(features_test)

# get accuracy of model on test set

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(labels_test, pred)

print('Accuracy: {}'.format(accuracy))

Another Application

The dataset we would use for this task is the Iris flower classification dataset. The dataset contains 150 examples of 3 classes of species of Iris flowers namely Iris Setosa, Iris Versicolor and Iris Virginica. The dataset can be downloaded from Kaggle (https://www.kaggle.com/saurabh00007/iriscsv/downloads/Iris.csv/1).

The first step of the data science process is to acquire data, which we have done. Next we need to handle the data or preprocess it into a suitable form before passing it off to a machine learning classifier.

To begin let’s import all relevant libraries.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import scipy as sp

Next we use Pandas to load the dataset which is contained in a CSV file and print out the first few rows so that we can have a sense of what is contained in the dataset.

dataset = pd.read_csv('Iris.csv')

dataset.head(5)

As we can see, there are 4 predictors namely sepal length, sepal width, petal length and petal width. Species is the target variable that we are interested in predicting. Since there are 3 classes what we have is a multi-classification problem.

In line with our observations, we separate the columns into features (X) and targets (y).

X = dataset.iloc[:, 1:5].values # select features ignoring non-informative column Id

y = dataset.iloc[:, 5].values # Species contains targets for our model

Our targets are currently stored as text. We need to transform them into categorical variables. To do this we leverage Scikit-Learn label encoder.

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

y = le.fit_transform(y) # transform species names into categorical values

Next we split our dataset into a training set and a test set so that we can evaluate the performance of our trained model appropriately.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3)

Calculating Similarity

In the last section, we successfully prepared our data and explained  the inner workings of the K-NN algorithm at a high level. We would now implement a working version in Python. The most important part of K-NN algorithm is the similarity metric which in this case is a distance measure. There are several distance metrics but we would use Euclidean distance which is the straight line distance between two points in a Euclidean plane. The plane may be 2-dimensional, 3-dimensional etc. Euclidean distance is sometimes referred to as L2 distance. It is given by the formula below.

The L2 distance is computed from the test sample to every sample in the training set to determine how close they are. We can implement L2 distance in Python using Numpy as shown below.

def euclidean_distance(training_set, test_instance):

    # number of samples inside training set

    n_samples = training_set.shape[0]

    # create array for distances

    distances = np.empty(n_samples, dtype=np.float64)

    # euclidean distance calculation

    for i in range(n_samples):

        distances[i] = np.sqrt(np.sum(np.square(test_instance - training_set[i])))

    return distances

Locating Neighbors

Having implemented the similarity metric, we can build out a full fledged class that is capable of identifying nearest neighbors and returning a classification. It should be noted that the K-Nearest Neighbor algorithm has no training phase. It simply stores all data points in memory. It only performs computation during test time when it is calculating distances and returning predictions. Here is an implementation of the K-NN algorithm that utilizes the distance function defined above.

class MyKNeighborsClassifier():

    """

    Vanilla implementation of KNN algorithm.

    """

    def __init__(self, n_neighbors=5):

        self.n_neighbors=n_neighbors

    def fit(self, X, y):

        """

        Fit the model using X as array of features and y as array of labels.

        """

        n_samples = X.shape[0]

        # number of neighbors can't be larger then number of samples

        if self.n_neighbors > n_samples:

            raise ValueError("Number of neighbors can't be larger then number of samples in training set.")

        # X and y need to have the same number of samples

        if X.shape[0] != y.shape[0]:

            raise ValueError("Number of samples in X and y need to be equal.")

        # finding and saving all possible class labels

        self.classes_ = np.unique(y)

        self.X = X

        self.y = y

    def pred_from_neighbors(self, training_set, labels, test_instance, k):

        distances = euclidean_distance(training_set, test_instance)

        # combining arrays as columns

        distances = sp.c_[distances, labels]

        # sorting array by value of first column

        sorted_distances = distances[distances[:,0].argsort()]

        # selecting labels associeted with k smallest distances

        targets = sorted_distances[0:k,1]

        unique, counts = np.unique(targets, return_counts=True)

        return(unique[np.argmax(counts)])

    def predict(self, X_test):

        # number of predictions to make and number of features inside single sample

        n_predictions, n_features = X_test.shape

        # allocationg space for array of predictions

        predictions = np.empty(n_predictions, dtype=int)

        # loop over all observations

        for i in range(n_predictions):

            # calculation of single prediction

            predictions[i] = self.pred_from_neighbors(self.X, self.y, X_test[i, :], self.n_neighbors)

        return(predictions)

The workflow of the class above is that during test time, a test sample (instance) is supplied and the Euclidean distance to every sample in the entire training set is calculated. Depending on the value of nearest neighbors to consider, the labels of those neighbors participate in a vote to determine the class of the test sample.

Generating Response

In order to generate a response or create a prediction, we first have to initialize our custom classifier. The value of k, cannot exceed the number of samples in our dataset. This is to be expected because we cannot compare with a greater number of neighbors than what we have available in the training set.

# instantiate learning model (k = 3)

my_classifier = MyKNeighborsClassifier(n_neighbors=3)

Next we can train our model on the data. Remember in K-NN no training actually takes place.

# fitting the model

my_classifier.fit(X_train, y_train)

Evaluating Accuracy

To evaluate the accuracy of our model, we test its performance on examples which it has not seen such as those contained in the test set.

# predicting the test set results

my_y_pred = my_classifier.predict(X_test)

We then check the predicted classes against the ground truth labels and use Scikit-Learn accuracy module to calculate the accuracy of our classifier.

from sklearn.metrics import confusion_matrix, accuracy_score

accuracy = accuracy_score(y_test, my_y_pred)*100

print('Accuracy: ' + str(round(accuracy, 2)) + ' %.')

Our model achieves an accuracy of 97.8% which is impressive for such a simple and elegant model.

The Curse of Dimensionality

The K-Nearest Neighbor algorithm performs optimally when the dimension of the input space is small as in this example. We had four predictors (sepal length, sepal width, petal length, petal width) going into the algorithm. K-NN struggles in high dimensional input spaces like those encountered in images. This is because the similarity measure as expressed by the distance metric is very limited and cannot properly model this high dimensional space. In general, machine learning algorithms try to reduce the number of dimensions so that intuitions we have about low dimensional spaces would still hold true. The accuracy or performance of algorithms usually suffer when the number of dimensions increase. This is known as the curse of dimensionality.

Naive Bayes

Applications of Naive Bayes

Naive Bayes algorithm is an eager learning algorithm based on Bayes theorem with the assumption that all features are independent given the class label. Naive Bayes algorithm is well suited to text classification as its test time prediction is quick and as a result it can be deployed on a large dataset for inference.

Naive Bayes is usually used in applications where predictions are desired in real time such as fraud detection on credit card transactions. In these applications, a classifier that is capable of performing almost instantaneously is desired. Naive Bayes is also used extensively in text classification, spam filtering and sentiment analysis. In spam filtering, the words of an email or text message serve as the input features to the model, with each word assumed to be independent of others. Naive Bayes usually produces good results when this independent assumption holds true and coupled with its short inference time is sometimes preferred over more complicated classifiers.

Another area where Naive Bayes algorithm is widely used is in multi-class classification. In this domain, multiple classes or categories can be predicted given input features, with probabilities assigned to the predicted classes.

How to Build a Basic Model Using Naive Bayes in Python

For our hands on example we would build a Naive Bayes model in Python to tackle a spam classification problem. We would use the SMS spam collection dataset which is a set of 5,574 English text messages annotated to indicate the category. There are two categories - ham or legitimate messages and spam. The dataset can be downloaded from the following URL (https://www.kaggle.com/uciml/sms-spam-collection-dataset/downloads/spam.csv/1).

We would use the multinomial Naive Bayes classifier from Scikit-Learn machine learning library. As always, we begin by importing the libraries we would utilize.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

# comment the magic command below if not running in Jupyter notebook

%matplotlib inline

Next we load the dataset using Pandas and display the first 5 rows.

data = pd.read_csv('spam.csv', encoding='latin-1')

data.head(5)

The column “v1” contains the class labels while “v2” are the contents of the SMS which we would use as the features of our model.

Let us plot a bar chart to visualize the distribution of legitimate and spam messages.

count_class = pd.value_counts(data['v1'], sort= True)

count_class.plot(kind='bar', color=[['blue', 'red']])

plt.title('Bar chart')

plt.show()

The words cannot be fed directly into the model as the features, so we have to vectorize them to create new features. We do this by considering the frequency of words after removing words that commonly appear in English sentences like “the”, “a”, “of” etc. We can do this feature extraction easily by using Scikit-Learn.

from sklearn.feature_extraction.text import CountVectorizer

f = CountVectorizer(stop_words = 'english')

X = f.fit_transform(data["v2"])

print(np.shape(X))

After vectorization, 8,404 new features are created.

Next we map our target variables into categories and split the dataset into train and test sets.

from sklearn.model_selection import train_test_split

data["v1"]=data["v1"].map({'spam':1,'ham':0})

X_train, X_test, y_train, y_test = train_test_split(X, data['v1'], test_size=0.25, random_state=42)

The next step involves initializing the Naive Bayes model and training it on the data.

from sklearn.naive_bayes import MultinomialNB

clf = MultinomialNB()

clf.fit(X_train, y_train)

Finally, we gauge the model performance on the test set.

score = clf.score(X_test, y_test)

print('Accuracy: {}'.format(score))

The Naive Bayes classifier attains an accuracy of 0.976, which means that it predicted the correct class for 97.6% of samples.

Regression

Regression is a statistical modelling technique whereby we are majorly interested in predicting the value of variable. The value to be predicted is normally a real value, which is a positive or negative number. This number may be a whole number in which case it is referred to as an integer or a number with several decimal places in which case it is referred to as a floating point number.

The nature of regression problems is that we are trying to find how the value of a dependent variable changes with respect to one or more independent variables. In a nutshell, what we want to know is how much a  variable say y depends on a set of other variables say x, w such that we can learn to predict the value of y once we know the values of the variables it depends on.

Our task is therefore to model this relationship in such a way that it would hold true for a majority of examples in our data.

The main intuition to get from this section is that regression always produces a single value hence it is best applied to learning problems where we require a single real valued number. A good example is if we want to build a model that takes in information about a person such as their age, nationality, profession etc and we want to predict their expected income for a year. Our output would be a single value and regression would be well positioned to solve this problem.

Introduction to Labels and Features

The two main branches of machine learning you would easily come across are supervised and unsupervised learning. Supervised learning as the name implies deals with teaching machine learning models with the help of data that is clearly labelled, that is data that has been annotated by a human to show what is to be learnt by the algorithm. The labels serve as a “supervisor” to the algorithm, teaching it during training by providing information on which samples it got correct or wrong.

The labels are regarded as the ground truth, the actual outcome that was observed from a particular data point.

Unsupervised learning on the other hand does not include any labelled outcomes. The job of the algorithm is to learn from the raw data in order to come up with patterns which would provide insights on the data explored. The main difference is that there isn’t explicit feedback during training in the form of labelled examples. A famous application of unsupervised learning is in clustering. Clustering involves using a machine learning algorithm to categorize data points into groups (clusters) based on similarity of features.

Features

A feature is a characteristic of an observed data point in a dataset. There may be more than one feature in a dataset and it is fairly common to come across a large number of features. Features are usually measurable and represent a specific axis of explanation for the data. The quality of features selected has a direct impact on quality of models as models learn using features that are informative in order to arrive at a final prediction.

Features that best describe the data should always be chosen as such features have a high discriminative tendency which helps the machine learning model classify outputs and predictions.

A good approach is to always choose an optimum number of features, not too much as in such a case, many features would be uninformative which leads to overfitting of the model and not too little that the model underfits thereby failing to learn anything.

The technique of choosing the right number of features is an instance of hyperparameter tuning in that we try out several options and settle for the best one.

When we have too many features and we are not sure which features are important and which ones are not, an algorithm that can help use find  the most relevant features which drive the discriminability is called Principal Component Analysis (PCA). PCA is a dimensionality reduction technique which reduces the number of dimensions of our data to a small number that best describes its core features.

Simple and Multiple Linear Regression

Regression involves finding the relationship between variables. Regression is typically used for predicting a single real value given a bunch of predictors. In simple regression, there are only two variables. The first is the independent variable while the other is the dependent variable. The regression task is thus to model how much the dependent variable changes with a change in the independent variable. A straight line equation may be used to fit a set of data points to capture the relationship between both variables. This is a case of a simple regression and the linear equation used is given below.

The coefficient to be calculated is denoted by a while b is the intercept or bias of the model, x and y are the independent and dependent variables respectively.

In multiple linear regression, there are two or more independent variables, that is the number of predictors which determine the outcome y, are more than one. The relationship is still linear but y is dependent on more variables.

Let us now look at a practical example. We would use the regression techniques explained above to predict the price of a house in a neighborhood given information about the house in the form of features. The dataset we would use is the Boston house pricing dataset and it contains 506 observations. The dataset can be downloaded from this URL (https://forge.scilab.org/index.php/p/rdataset/source/file/master/csv/MASS/Boston.csv).

First we import relevant libraries and load the dataset using Pandas.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

# matplotlib magic command for Jupyter notebook

%matplotlib inline

dataset = pd.read_csv('Boston.csv')

dataset.head()

The dataset has 13 predictors such as the number of rooms in the house, age of house, pupil-teacher ratio in the town etc.

Let us plot the relationship between one of the predictors and the price of a house to see whether we can come up with any explanation from the visualization. The predictor we would use is the per capita crime rate by town which captures the rate of crime in the neighborhood.

plt.scatter(dataset['crim'], dataset['medv'])

plt.xlabel('Per capita crime rate by town')

plt.ylabel('Price')

plt.title("Prices vs Crime rate")

We can see that for towns with very low crime rates (at the beginning of the plot), there are houses for the full range of prices, both cheap and expensive. This is denoted by the vertical spread of points across the y axis. If we exclude the first 10 units on the x-axis, we notice that there is a negative correlation between price and the crime rate. This is hardly surprising as we would expect the price of houses to drop as the crime rate in the neighborhood increases.

Next we split our dataset into predictors and targets. Then we create a training and test set.

X = dataset.drop(['Unnamed: 0', 'medv'], axis=1)

y = dataset['medv']

from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

The next step involves importing the linear regression classifier from Scikit-Learn, initializing it and fitting the classifier on data.

# import linear regression classifier, initialize and fit the model

from sklearn.linear_model import LinearRegression

regressor = LinearRegression()

regressor.fit(x_train,y_train)

Having fit the classifier, we can use it to predict house prices using features in the test set.

y_pred = regressor.predict(x_test)

The next step is to evaluate the classifier using metrics such as the mean square error and the coefficient of determination R square.

from sklearn.metrics import mean_squared_error, r2_score

# The coefficients

print('Coefficients: \n', regressor.coef_)

# The mean squared error

print('Mean squared error: {:.2f}'.format(mean_squared_error(y_test, y_pred)))

# Explained variance score: 1 is perfect prediction

print('Variance score: {:.2f}'.format(r2_score(y_test, y_pred)))

The coefficients are the learnt parameters for each predictor, the mean square error represents how far off our predictions are from the actual values and variance score is the coefficient of determination which gives the overall performance of the model. A variance score of 1 is a perfect model, so it is clear that with a score of 0.72, the model has learnt from the data.

Finally, we can plot the predicted prices from the model against the ground truth (actual prices).

plt.scatter(y_test, y_pred)

plt.xlabel("Prices: $Y_i$")

plt.ylabel("Predicted prices: $\hat{Y}_i$")

plt.title("Prices vs Predicted prices: $Y_i$ vs $\hat{Y}_i$")

The scatter plot above shows a positive relationship between the predicted prices and actual prices. This indicates that our model has successfully captured the underlying relationship and can map from input features to output prices.

Here is the code in its entirety.

# import libraries

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

%matplotlib inline

# load dataset

dataset = pd.read_csv('Boston.csv')

dataset.head()

# plot crime vs price

plt.scatter(dataset['crim'], dataset['medv'])

plt.xlabel('Per capita crime rate by town')

plt.ylabel('Price')

plt.title("Prices vs Crime rate")

# separate predictors and targets

X = dataset.drop(['Unnamed: 0', 'medv'], axis=1)

y = dataset['medv']

from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# import linear regression classifier, initialize and fit the model

from sklearn.linear_model import LinearRegression

regressor = LinearRegression()

regressor.fit(x_train,y_train)

y_pred = regressor.predict(x_test)

from sklearn.metrics import mean_squared_error, r2_score

# The coefficients

print('Coefficients: \n', regressor.coef_)

# The mean squared error

print('Mean squared error: {:.2f}'.format(mean_squared_error(y_test, y_pred)))

# Explained variance score: 1 is perfect prediction

print('Variance score: {:.2f}'.format(r2_score(y_test, y_pred)))

# plot predicted prices vs actual prices

plt.scatter(y_test, y_pred)

plt.xlabel("Prices: $Y_i$")

plt.ylabel("Predicted prices: $\hat{Y}_i$")

plt.title("Prices vs Predicted prices: $Y_i$ vs $\hat{Y}_i$")

Logistic Regression

Logistic regression despite its name is a classification algorithm. Logistic regression is used when the dependent variable is binary in nature, that is when it can be either one of two values (categories) example true or false. It is a linear combination of weighted input features applied to the sigmoid function. The logit or sigmoid function is at the heart of logistic regression and models data along the range of 0 to 1.

In the image above, z represents the weighted input features. What this means is that z is a linear addition of input features and the importance of input features (how large they are), is influenced by their weights (coefficients). A threshold is usually set to separate samples into classes. The threshold can be seen as the decision boundary. After the linear computation and the application of the sigmoid or logit function, the resultant value is compared to the threshold value. If it is equal to or larger than the threshold value, then the sample under consideration belongs to the positive class else it belongs to the negative class. The threshold value is usually set to 0.5.

Outputs from logistic regression can be interpreted as probabilities that show how likely a data point belongs to a category. The formula for the logistic function is shown below.

The main difference between logistic regression and simple regression is that logistic regression is used for classification when there can only be two classes (negative or positive) while simple regression is used to predict an actual value like a continuous number and not classes or categories.

We would now apply logistic regression to a binary classification problem. The dataset we would leverage is the Pima Indian Diabetes Database which is a dataset from the National Institute of Diabetes and Digestive and Kidney Diseases. The dataset contains a target variable that is used to indicate whether a patient developed diabetes or not. Our task is therefore to use diagnostic measurements as predictors to determine the diabetes status of a patient.

The dataset can be downloaded at: https://www.kaggle.com/uciml/pima-indians-diabetes-database/data

Let us import relevant libraries and load the dataset to have a sense of what it contains.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

dataset = pd.read_csv('diabetes.csv')

dataset.head(5)

The dataset has 8 predictors such as  glucose level of patient, age, skin thickness, body mass index, insulin level, age etc. These form the features for our model or in regression speak, the independent variables.

Next we separate the columns in the dataset into features and labels. The labels or class are represented by the “Outcome” column.

features = dataset.drop(['Outcome'], axis=1)

labels = dataset['Outcome']

from sklearn.model_selection import train_test_split

features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.25)

The next step is to initialize a logistic regression model and fit it to the Pima Indians diabetes data.

# Training the model

from sklearn.linear_model import LogisticRegression

classifier = LogisticRegression()

classifier.fit(features_train, labels_train)

The trained model can now be evaluated on the test set.

pred = classifier.predict(features_test)

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(labels_test, pred)

print('Accuracy: {:.2f}'.format(accuracy))

The trained logistic regression model attains an accuracy of 72% on the test set.

Generalized Linear Models

Generalized linear models are an extension of linear models where the dependent variable does not belong to a normal or Gaussian distribution. Generalized linear models are capable of modelling more complicated relationships between the independent and dependent variables. GLMs can often model various probability distributions as such poisson, binomial, multinomial distributions etc. Logistic regression is an example of a generalized linear model where the dependent variable is modelled using a binomial distribution. This enables it to create a mapping from inputs to outputs, where the outputs are binary in nature.

Poisson regression is a generalized linear model that is used for modelling count data. Count data are integer values that can only be positive. Poisson regression assumes that the independent variable y, belongs to a Poisson distribution, which is a type of exponential probability distribution. The main difference between Poisson regression and linear regression is that linear regression assumes the outputs are drawn from a normal distribution whereas Poisson distribution assumes y comes from a Poisson distribution. The outputs in Poisson regression are modelled as shown below.

Generalized linear models are made up of three components, the random components which are the probability distribution of the output, the systematic component which describes the explanatory variables (X) or predictors and the link function, which specifies the relationship between explanatory variables and the random component.

Since the hyperparameters (weights) of Poisson regression cannot take negative values, they are transformed using natural logarithm to ensure they are always positive. The mean of Poisson distribution is stated mathematically as:

The objective function or loss function that is used to train the model in order to discover learnable parameters is shown below:

For our hands on example, we would use the statsmodels package that provides various functions and classes for statistical modelling, statistical data exploration etc. We would use a bundled dataset from statsmodels, the Scottish vote dataset that contains records from the 1997 vote to give the Scottish parliament the rights to collect taxes. The dataset contains 8 explanatory variables (predictors) and 32 observations, one for each district.

First we import the Statsmodels package as shown below.

import statsmodels.api as sm

Next we load the dataset and extract the explanatory variable (X).

data = sm.datasets.scotland.load()

# data.exog is the independent variable X

data.exog = sm.add_constant(data.exog)

Similar to Scikit-Learn, we import the appropriate model and instantiate an object from it. In this case we specify a generalized linear model and set the distribution family to Poisson.

# Instantiate a poisson family model with the default link function.

poisson_model = sm.GLM(data.endog, data.exog, family=sm.families.Poisson())

We then fit the model on the data.

poisson_results = poisson_model.fit()

We can now print a summary of results to better understand the trained model.

print(poisson_results.summary())

The summary contains values like the coefficients or weights for independent variables, standard error and z scores.

A Regression Example: Predicting Boston Housing Prices

To get a good understanding of the concepts discussed so far, we would introduce a running example in which we are faced with a regression problem, predict the price of houses in the Boston suburbs given information about such houses in the form of features.

The dataset we would use can be found at: https://forge.scilab.org/index.php/p/rdataset/source/file/master/csv/MASS/Boston.csv

Steps To Carry Out Analysis

To carry out analysis and build a model we first need to identify the problem, perform exploratory data analysis to get a better sense of what is contained in our data, choose a machine learning algorithm, train the model and finally evaluate its performance. The following steps would be carried out in a hands on manner below, so the reader is encouraged to follow along.

Import Libraries:

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

Numpy is a popular Python numerical computing library which supports vectorized implementation of calculations and significantly speeds up computation time. Pandas is a library used to manipulate data in data frames while Matplotlib is used for plotting graphs and data visualizations.

Let’s load the data by using the read_csv method on the Pandas library and passing it the location of our data.

dataset = pd.read_csv('Boston.csv')

The next step is to look at what our data contains

dataset.head(5)

This shows individual observations as rows. A row represents a single data point while columns represent features. There are 13 features because the last column medv is the regression value that we are to predict (median value of owner-occupied homes in $1000s) and Unmamed:0 column is a sort of identifier and is  not informative. Each feature represents a subset of information example crim means per capita crime rate by town while rm average number of rooms per dwelling.

Next we run

dataset.shape

This gives the shape of dataset which contains 506 observations. We first need to separate our columns into our independent and dependent variables

X = dataset.drop(['Unnamed: 0', 'medv'], axis=1)

y = dataset['medv']

We would need to split our dataset into train and test splits as we want to train our model on the train split, then evaluate its performance on the test split.

from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

x_train, x_test contains our features while y_train, y_test are the prediction targets for the train split and test splits respectively. test_size=0.3 means we want 70% of data to be used for training and 30% for the testing phase.

The next step is to import a linear regression model from the Scikit-Learn library. Scikit-Learn is the defacto machine learning library in Python and contains out of the box many machine learning models and utilities.

Linear regression uses equation of a straight line to fit our parameters.

# importing the model

from sklearn.linear_model import LinearRegression

regressor = LinearRegression()

The above code imports the linear regression model and instantiates an object from it.

regressor.fit(x_train,y_train)

This line of code fits the data using the fit method. What that means is that it finds appropriate values for the independent parameters that explains the data.

How to forecast and Predict

To evaluate our model, we use the test set to know whether our model can generalize well to data it wasn’t trained on.

y_pred = regressor.predict(x_test,y_test)

from  sklearn.metrics import mean_squared_error

mse = mean_squared_error(y_test, y_pred)

The predict method called on the regressor object returns predictions which we use to evaluate the error of our model. We use mean squared error as our metric. Mean Squared Error (MSE) measures how far off our predictions are from the real (actual) values. The model obtains an MSE of 20.584.

Finally, we plot a graph of our output to get an idea of the distribution.

plt.scatter(y_test, y_pred)

plt.xlabel("Prices: $Y_i$")

plt.ylabel("Predicted prices: $\hat{Y}_i$")

plt.title("Prices vs Predicted prices: $Y_i$ vs $\hat{Y}_i$")

We can see from the scatter plot above that predictions from our model are close to the actual house prices hence the concentration of points.

Decision Trees and Random Forest

The Entropy of a Partition

Entropy can be defined as the measure of uncertainty in a sequence of random events. It is the rate of disorderliness in a sample space and is directly opposed to knowledge. When the entropy of a system is high, the knowledge that can be derived from the system is low and vice versa. An intuitive understanding of entropy is thinking of it as the amount of questions required to ask to arrive at some knowledge. For example, if I picked a random number and you were trying to guess what number it is. Asking a question like, “Is it an odd number”, reduces the possibilities space by half. This means that the entropy or the degree of uncertainty in trying to determine which number I choose is reduced. In the same vein, the amount of information gain is large because the question moved you closer to the answer by dividing the sample space. Entropy usually ranges from 0 to 1. A system with an entropy of 0 is highly stable and the knowledge that can be derived from such a system is high. In general terms, low entropy in a system indicates high knowledge while high entropy indicates low knowledge or instability.

Entropy can be represented mathematically as:

The formula above is the negative sum of log probabilities of an event happening. Remember that probability indicates the confidence we have in an event occurring, therefore entropy is how surprising it would be, for a sequence of events to occur together.

In machine learning as we would see later with decision trees, the entropy of two or more attributes of a classifier is defined by:

Decision trees are a machine learning algorithm that rely heavily on the entropy of an attribute and the information gain to determine how to classify samples in a classification problem. Let us look at decision trees in depth in the next section.

Creating a Decision Tree

A decision tree is a machine learning algorithm which is mainly used for classification that constructs a tree of possibilities where the branches in the tree represents decisions and the leaves represents label classification. The purpose of a decision tree is to create a structure where samples in each branch are homogenous or of the same type. It does this by splitting samples in the training data according to specific attributes that increase homogeneity in branches. These attributes form the decision node along which samples are separated. The process continues until all sample are correctly predicted as represented by the leaves of the tree.

To explain the concept of a decision tree further, let us look at a toy example below that demonstrates its capability.

Let us assume that we are a laptop manufacturer and we want to predict which customers from an online store are likely to buy our new top of the range laptop, so that we can focus our marketing efforts accordingly. This problem can be modelled using a decision tree with two classes (yes or no), for whether a person is likely to purchase or not.

At the root of the tree, we want to choose an attribute about customers that reduces entropy the most. As we saw in the last section, by reducing the entropy, we increase the amount of knowledge that is contained in the system. We choose the appropriate attribute by calculating the entropy of each branch and the entropy of the targets (yes or no). The information gain is closely related to the entropy and is defined as the difference in entropy of the targets (final entropy) and the entropy given a particular attribute was chosen as the root node.

The formula above is used to calculate the decrease in entropy. The attribute with the largest information gain or decrease in entropy is chosen as the root node. This means that the attribute reduces the decision space the most when compared to other attributes. The process is repeated to find other decision nodes via attributes until all samples are correctly classified through the leaves of the decision tree.

In  the example above, age is the attribute that offers the most information gain so samples are split on that decision node. If the customer is middle aged, then they are likely to purchase a new laptop as they are probably working and have higher spending power. If the customer is a youth this brings us to another decision node. The attribute used is whether the youth is a student or not. If the youth is a student, they are likely to buy else they are not. That brings us to the leaves (classes) of the node following the youth branch of the tree. For the senior branch, we again split samples on an informative attribute, in this case credit rating. If the senior has an excellent credit rating that means they are likely to buy, else the leaf or classification for that sample along this branch of the tree is no.

Let us now work on an example using Python, Scikit-Learn and decision trees. We would tackle a multi-class classification problem where the the challenge is to classify wine into three types using features such as alcohol, color intensity, hue etc. The data we would use comes from the wine recognition dataset by UC Irvine. It can be downloaded at https://gist.github.com/tijptjik/9408623/archive/b237fa5848349a14a14e5d4107dc7897c21951f5.zip

First, lets load the dataset and use Pandas head method to have a look at it.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

# comment the magic command below if not running in Jupyter notebook

%matplotlib inline

dataset = pd.read_csv('wine.csv')

dataset.head(5)

There are 13 predictors and the first column “wine” contains the targets. The next thing we do is split the dataset into predictors and targets, sometimes referred to as features and labels respectively.

features = dataset.drop(['Wine'], axis=1)

labels = dataset['Wine']

As is the custom to ensure good evaluation of our model, we divide the dataset into a train and test split.

from sklearn.model_selection import train_test_split

features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.25)

All that is left is for us to import the decision tree classifier and fit it to our data.

from sklearn.tree import DecisionTreeClassifier

classifier = DecisionTreeClassifier()

classifier.fit(features_train, labels_train)

We can now evaluate the trained model on the test set and print out the accuracy.

pred = classifier.predict(features_test)

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(labels_test, pred)

print('Accuracy: {:.2f}'.format(accuracy))

We achieve an accuracy of 0.91 which is very impressive. It means that 91% of samples in our test set were correctly classified.

Random Forests

Random forests are a type of ensemble model. An ensemble model is one which is constructed from other models. This means that it is a combination of several weak learners to form a strong learner. The prediction of an ensemble model may be the average or weighted average of all learners that it is comprised of.

Random forests are an extension of decision trees whereby several decision trees are grown to form a forest. The final prediction of a random forest model is a combination of all component decision trees. For regression it may be a simple average of outputs or a label vote in the case of classification. Though random forest are made of several decision trees, each decision tree is trained on a subset of data that is randomly selected hence the name random forest. The other trick of random forest is that unlike a decision tree where the best attribute is chosen in order to split samples at a decision node from all available attributes, random forest only picks the best attribute from a subset of randomly chosen attributes for each decision node. As a result, each node in a tree is not deterministic, that is for each time we run the algorithm, we are likely to end up with different tree structures. However, the most informative attributes still find their way to trees in the forest and are present across many trees. This makes the results of the random forest algorithm to be less prone to errors due to variations in the input data.

The subset of data on which a decision tree that makes up a random forest is trained on is called bagged data and is usually around 60% of the entire dataset. The remainder on which the performance of individuals trees are tested on is known as the out-of-bag data. Therefore each tree in the forest is trained and evaluated on a different subset of data through the randomization process.

The image above shows a pictorial representation of random forests. It is made up of several trees trained on different instances of the dataset. The attributes in each decision node are also randomized. Finally, the output prediction is an ensemble of the classification of each decision tree.

We would now try out a random forest classifier on the wine dataset and compare its performance on the test set to the decision tree model in the previous section. The beautiful thing about using machine learning models from Scikit-Learn is that the APIs to train and test a model are the same regardless of the algorithm being used. So you would notice that we only need to import the correct classifier, initialize it and all other portions of code would remain unchanged. We are already familiar with how parts of the code works so here is the code for random forest in full.

import numpy as np

import pandas as pd

# load dataset

dataset = pd.read_csv('wine.csv')

# separate features and labels

features = dataset.drop(['Wine'], axis=1)

labels = dataset['Wine']

# split dataset into train and test sets

from sklearn.model_selection import train_test_split

features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.25)

# import random forest classifier from sklearn

from sklearn.ensemble import RandomForestClassifier

classifier = RandomForestClassifier()

# fit classifier on data

classifier.fit(features_train, labels_train)

# predict classes of test set samples

pred = classifier.predict(features_test)

# evaluate classifier performance using accuracy metric

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(labels_test, pred)

print('Accuracy: {:.2f}'.format(accuracy))

We achieve an accuracy of 98% on the test set which is a massive jump from 91% when we used a decision tree classifier. We can see that the randomization approach of random forest enables the algorithm to generalize better hence higher accuracy is recorded on the test set.

Neural Networks

Perceptrons

The perceptron is a binary linear classifier that is only capable of predicting classes of samples if those samples can be separated via a straight line. The perceptron algorithm was introduced by Frank Rosenblatt in 1957. It classifies samples using hand crafted features which represents information about the samples, weighs the features on how important they are to the final prediction and the resulting computation is compared against a threshold value.

In the image above, X represents the inputs to the model and W represents the weights (how important are individual features). A linear computation of the weighted sum of features is carried out during the formula below:

The value of z is then passed through a step function to predict the class of the sample. A step function is an instant transformation of a value from 0 to 1. What this means is that if z is greater than or equal to 0, its predicts one class, else it predicts the other.  The step function can be represented mathematically as:

At each iteration, the predicted class gets compared to the actual class and the weights gets updated if the prediction was wrong else it is left unchanged in the case of a correct prediction. Updates of weights continue until all samples are correctly predicted, at which point we can say that the perceptron classifier has found a linear decision boundary that perfectly separates all samples into two mutually exclusive classes.

During training the weights are updated by adding a small value to the original weights. The amount added is determined by the perceptron learning rule. The weight update process can be experienced mathematically as shown below.

The amount by which weights are updated is given by the perceptron learning rule below.

The first coefficient on the right hand side of the equation is called the learning rate and acts as a scaling factor to increase or decrease the extent of the update. The intuitive understanding of the above equation is that with each pass through the training set, the weights of misclassified examples are nudged in the correct direction so that the value of z can be such that the step function correctly classifies the sample. It should be noted that the perceptron learning algorithm described is severely limited as it can only learn simple functions that have a clear linear boundary. The perceptron is almost never used in practice but served as an integral building block during the earlier development of artificial neural networks.

Modern iterations are known as multi-layer perceptrons. Multi-layer perceptrons are feed forward neural networks that have several nodes in the structure of a perceptron. However, there are important differences. A multilayer perceptron is made up of multiple layers of neurons stacked to form a network. The activation functions used are non-linear unlike the perceptron model that uses a step function. Nonlinear activations are capable of capturing more interesting representations of data and as such do not require input data to be linearly separable. The other important difference is that multi-layer perceptrons are trained using a different kind of algorithm called backpropagation which enables training across multiple layers.

Backpropagation

Backpropagation is an algorithm technique that is used to solve the issue of credit assignment in artificial neural networks. What that means is that it is used to determine how much an input’s features and weights contribute to the final output of the model. Unlike the perceptron learning rule, backpropagation is used to calculate the gradients, which tell us how much a change in the parameters of the model affects the final output. The gradients are used to train the model by using them as an error signal to indicate to the model how far off its predictions are from the ground truth. The backpropagation algorithm can be thought of as the chain rule of derivatives applied across layers.

Let us look at a full fledged illustration of a multi-layer perceptron to understand things further.

The network above is made up of three layers, the input layer which are the features fed into the network, the hidden layer which is so called because we cannot observe what goes on inside and the output layer, through which we get the prediction of the model. During training, in order to calculate by how each node contributes to the final prediction and adjust them accordingly to yield a higher accuracy across samples, we need to change the weights using the backpropagation algorithm. It is the weights that are learned during the training process hence they are sometimes referred to as the learnable parameters of the model. To visually understand what goes on during backpropagation, lets us look at the image of a single node below.

In the node above x and y are the input features while f is the nonlinear activation function. During training computations are calculated in a forward fashion from the inputs, across the hidden layers, all the way to the output. This is known as the forward pass denoted by green arrows in the image. The prediction of the model is then compared to the ground truth and the error is propagated backwards. This is known as the backward pass and assigns the amount by which every node is responsible for the computed error through the backpropagation algorithm. It is depicted with red arrows in the image above. This process continues until the model finds a set of weights that captures the underlying data representation and correctly predicts majority of samples.

How to run the Neural Network using TensorFlow

For our hands on example, we would do image classification using the MNIST handwritten digits database which contains pictures of handwritten digits ranging from 0 to 9 in black and white. The task is to train a neural network that given an input digit image, it can predict the class of the number contained therein.

How to get our data

TensorFlow includes several preloaded datasets which we can use to learn or test out our ideas during experimentation. The MNIST database is one of such cleaned up datasets that is simple and easy to understand. Each data point is a black and white image with only one color channel. Each pixel in the image denotes the brightness of that point with 0 indicating black and 255 white. The numbers range from 0 to 255 for 784 points in a 28 × 28 grid.

Let’s go ahead and load the data from TensorFlow along with importing other relevant libraries.

# Import MNIST data

from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

import numpy as np

import tensorflow as tf

import matplotlib.pyplot as plt

Let us use the matplotlib library to display an image to see what it looks like by running the following lines of code.

plt.imshow(np.reshape(mnist.train.images[8], [28, 28]), cmap='gray')

plt.show()

The displayed image is a handwritten digit of number 9.

How to train and test the data

In order to train an artificial neural network model on our data, we first need to define the parameters that describe the computation graph such as number of neurons in each hidden layer, number of hidden layers, input size, number of output classes etc. Each image in the dataset is 28 by 28 pixels therefore, the input shape is 784 which is 28 × 28.

# Parameters

learning_rate = 0.1

num_steps = 500

batch_size = 128

display_step = 100

# Network Parameters

n_hidden_1 = 10 # 1st layer number of neurons

n_hidden_2 = 10 # 2nd layer number of neurons

num_input = 784 # MNIST data input (img shape: 28*28)

num_classes = 10 # MNIST total classes (0-9 digits)

# tf Graph input

X = tf.placeholder("float", [None, num_input])

Y = tf.placeholder("float", [None, num_classes])

We then declare weights and biases which are trainable parameters and initialise them randomly to very small values. The declarations are stored in a Python dictionary.

# Store layers weight & bias

weights = {

    'h1': tf.Variable(tf.random_normal([num_input, n_hidden_1])),

    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),

    'out': tf.Variable(tf.random_normal([n_hidden_2, num_classes]))

}

biases = {

    'b1': tf.Variable(tf.random_normal([n_hidden_1])),

    'b2': tf.Variable(tf.random_normal([n_hidden_2])),

    'out': tf.Variable(tf.random_normal([num_classes]))

}

We are would then describe a 3-layer neural network with 10 units in the output for each of the class digits and define the model by creating a function which forward propagates the inputs through the layers. Note that we are still describing all these operations on the computation graph.

# Create model

def neural_net(x):

    # Hidden fully connected layer with 10 neurons

    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])

    # Hidden fully connected layer with 10 neurons

    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])

    # Output fully connected layer with a neuron for each class

    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']

    return out_layer

Next we call our function, define the loss objective, choose the optimizer that would be used to train the model and initialise all variables.

# Construct model

logits = neural_net(X)

# Define loss and optimizer

loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(

    logits=logits, labels=Y))

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)

train_op = optimizer.minimize(loss_op)

# Evaluate model (with test logits, for dropout to be disabled)

correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(Y, 1))

accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

# Initialize the variables (i.e. assign their default value)

init = tf.global_variables_initializer()

Finally, we create a session, supply images in batches to the model for training and print the loss and accuracy for each mini-batch.

# Start training

with tf.Session() as sess:

    # Run the initializer

    sess.run(init)

    for step in range(1, num_steps+1):

        batch_x, batch_y = mnist.train.next_batch(batch_size)

        # Run optimization op (backprop)

        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})

        if step % display_step == 0 or step == 1:

            # Calculate batch loss and accuracy

            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,

                                                                 Y: batch_y})

            print("Step " + str(step) + ", Minibatch Loss= " + \

                  "{:.4f}".format(loss) + ", Training Accuracy= " + \

                  "{:.3f}".format(acc))

    print("Optimization Finished!")

    # Calculate accuracy for MNIST test images

    print("Testing Accuracy:", \

        sess.run(accuracy, feed_dict={X: mnist.test.images,

                                      Y: mnist.test.labels}))

The session was created using with, so it automatically closes after executing. This is the recommended way of running a session as we would not need to manually close it. Below is the output

The loss drops to 0.4863 after training for 500 steps and we achieve an accuracy of 85% on the test set.

Here is the code in full:

# Parameters

learning_rate = 0.1

num_steps = 500

batch_size = 128

display_step = 100

# Network Parameters

n_hidden_1 = 10 # 1st layer number of neurons

n_hidden_2 = 10 # 2nd layer number of neurons

num_input = 784 # MNIST data input (img shape: 28*28)

num_classes = 10 # MNIST total classes (0-9 digits)

# tf Graph input

X = tf.placeholder("float", [None, num_input])

Y = tf.placeholder("float", [None, num_classes])

# Store layers weight & bias

weights = {

    'h1': tf.Variable(tf.random_normal([num_input, n_hidden_1])),

    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),

    'out': tf.Variable(tf.random_normal([n_hidden_2, num_classes]))

}

biases = {

    'b1': tf.Variable(tf.random_normal([n_hidden_1])),

    'b2': tf.Variable(tf.random_normal([n_hidden_2])),

    'out': tf.Variable(tf.random_normal([num_classes]))

}

# Create model

def neural_net(x):

    # Hidden fully connected layer with 10 neurons

    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])

    # Hidden fully connected layer with 10 neurons

    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])

    # Output fully connected layer with a neuron for each class

    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']

    return out_layer

# Construct model

logits = neural_net(X)

# Define loss and optimizer

loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(

    logits=logits, labels=Y))

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)

train_op = optimizer.minimize(loss_op)

# Evaluate model (with test logits, for dropout to be disabled)

correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(Y, 1))

accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

# Initialize the variables (i.e. assign their default value)

init = tf.global_variables_initializer()

# Start training

with tf.Session() as sess:

    # Run the initializer

    sess.run(init)

    for step in range(1, num_steps+1):

        batch_x, batch_y = mnist.train.next_batch(batch_size)

        # Run optimization op (backprop)

        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})

        if step % display_step == 0 or step == 1:

            # Calculate batch loss and accuracy

            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,

                                                                 Y: batch_y})

            print("Step " + str(step) + ", Minibatch Loss= " + \

                  "{:.4f}".format(loss) + ", Training Accuracy= " + \

                  "{:.3f}".format(acc))

    print("Optimization Finished!")

    # Calculate accuracy for MNIST test images

    print("Testing Accuracy:", \

        sess.run(accuracy, feed_dict={X: mnist.test.images,

                                      Y: mnist.test.labels}))

Clustering

Clustering is the most common form of unsupervised learning. Clustering involves grouping objects or entities into clusters (groups) based on a similarity metric. What clustering algorithms aim to achieve is to make all members of a group as similar as possible but make the cluster dissimilar to other clusters. At first glance clustering looks a lot like classification since we are putting data points into categories, while that may be the case, the main difference is that in clustering we are creating categories without the help of a human teacher. Whereas, in classification, objects were assigned to categories based on the domain knowledge of a human expert. That is in classification we had human labelled examples which means the labels acted as a supervisor teaching the algorithm how to recognise various categories.

In clustering, the clusters or groups that are discovered are purely dependent on the data itself. The data distribution is what drives the kind of clusters that are found by the algorithm. There are no labels so clustering algorithms are forced to learn representations in an unsupervised manner devoid of direct human intervention.

Clustering algorithms are divided into two main groups - hard clustering algorithms and soft clustering algorithms. Hard clustering algorithms are those clustering algorithms that find clusters from data such that a data point can only belong to one cluster and no more. Soft clustering algorithms employ a technique whereby a data point may belong to more than one cluster, that is the data point is represented across the distribution of clusters using a probability estimate that assigns how likely the point belongs to one cluster or the other.

From the data distribution of the image above, we can deduce that a clustering algorithm has been able to find 5 clusters using a distance measure such as Euclidean distance. It would be observed that data points close to cluster boundaries are equally likely to fall into any neighboring cluster. Some clustering algorithms are deterministic meaning that they always produce the same set of clusters regardless of initialization conditions or how many times they are run. Other clustering algorithms produce a different cluster collection everytime they are run and as such it may not be easy to reproduce results.

Introduction to Clustering

The most important input to a clustering algorithm is the distance measure. This is so because it is used to determine how similar two or more points are to each other. It forms the basis of all clustering algorithms since clustering is inherently about discriminating entities based on similarity.

Another way clustering algorithms are categorized is using the relationship structure between clusters. There are two subgroups - flat clustering and hierarchical clustering algorithms. In flat clustering, the clusters do not share any explicit structure so there is no definite way of relating one cluster to the other. A very popular implementation of a flat clustering algorithm is K-means algorithm which we would use as a case study.

Hierarchical clustering algorithms first starts with each data point belonging to its own cluster, then similar data points are merged into a bigger cluster and the process continues until all data points are part of one big cluster. As a result of the process of finding clusters, there is a clear hierarchical relationship between discovered clusters.

There are advantages and disadvantages to the flat and hierarchical approach. Hierarchical algorithms are usually deterministic and do not require us to supply the number of clusters beforehand. However, this leads to computational inefficiency as we suffer from quadratic cost. The time taken to discover clusters by an hierarchical clustering algorithm increases as the size of the data increases.

Flat clustering algorithms are intuitive to understand and feature linear complexity, therefore the time taken to run the algorithm increases linearly with the number of data points and because of this flat clustering algorithms scale well to massive amounts of data. As a rule of thumb, flat clustering algorithms are generally used for large datasets where a distance metric can capture similarity while hierarchical algorithms are used for smaller datasets.

Example of Clustering

We would have a detailed look at an example of a flat clustering algorithm, K-means. We would also use it on a dataset to see its performance.

K-means is an iterative clustering algorithm that seeks to assign data points to clusters. To run K-means algorithm, we first need to supply the number of clusters we desire to find. Next, the algorithm randomly assigns each point to a cluster and computes the cluster centroids (center of cluster). At this stage points are reassigned to new clusters based on how close they are to cluster centroids. We again recompute the cluster centroids. Finally we repeat the last two steps until no data points are being reassigned to new clusters. The algorithm has now converged and we have our final clusters.

Running K-means with Scikit-Learn

For our hands on example we would use K-means algorithm to find clusters in the Iris dataset. The Iris dataset is a classic in the machine learning community. It contains 4 attributes (sepal length, sepal width, petal length, petal width) used to describe 3 species of the Iris plant. The dataset can be found at:

https://www.kaggle.com/saurabh00007/iriscsv/downloads/Iris.csv/1

The first step is to load the data and run the head method on the dataset to know our features

# import libraries

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

# read dataset from csv file

dataset = pd.read_csv('Iris.csv')

# display first five observations

dataset.head(5)

We have 4 informative features and 6 columns. Id is an identifier while Species column contains the label. Since this is a clustering task, we do not need labels as we would find clusters in an unsupervised manner.

x = dataset.drop(['Id', 'Species'], axis=1)

x = x.values # select values and convert dataframe to numpy array

The above line of code selects all our features into x dropping Id and Species.

As was earlier discussed, because K-means is a flat clustering algorithm we need to specify the value of k (number of clusters) before we run the algorithm. However, we do not know the optimal value for k, so we use a technique known as the elbow method. The elbow method plots the percentage of variance explained as a result of number of clusters. The optimal value of k from the graph would be the point where the sum of squared error (SSE) does not improve significantly with increase in the number of clusters.

Let’s take a at look at these concepts in action

# finding the optimum number of clusters for k-means classification

from sklearn.cluster import KMeans

wcss = [] # array to hold sum of squared distances within clusters

for i in range(1, 11):

    kmeans = KMeans(n_clusters = i, init = 'k-means++', max_iter = 300, n_init = 10, random_state = 0)

    kmeans.fit(x)

    wcss.append(kmeans.inertia_)

# plotting the results onto a line graph, allowing us to observe 'The elbow'

plt.plot(range(1, 11), wcss)

plt.title('The elbow method')

plt.xlabel('Number of clusters')

plt.ylabel('Within Cluster Sum of Squares') # within cluster sum of squares

plt.show()

The k-means algorithm is run for 10 iterations, with n_clusters ranging from 1 to 10. At each iteration the sum of squared error (SSE) is recoded. The sum of squared distances within each cluster configuration is then plotted against the number of clusters. The “elbow” from the graph is 3 and this is the optimal value for k.

Now that we know that the optimal value for k is 3, we create a K-means object using Scikit-Learn and set the parameter of n_clusters (number of clusters to generate) to 3.

# creating the kmeans object

kmeans = KMeans(n_clusters = 3, init = 'k-means++', max_iter = 300, n_init = 10, random_state = 0)

Next we use the fit_predict method on our object. This returns a computation of cluster centers and cluster predictions for each sample.

y_kmeans = kmeans.fit_predict(x)

We then plot the predictions for clusters using a scatter plot of the first two features.

# visualising the clusters

plt.scatter(x[y_kmeans == 0, 0], x[y_kmeans == 0, 1], s = 100, c = 'red', label = 'Iris-setosa')

plt.scatter(x[y_kmeans == 1, 0], x[y_kmeans == 1, 1], s = 100, c = 'blue', label = 'Iris-versicolour')

plt.scatter(x[y_kmeans == 2, 0], x[y_kmeans == 2, 1], s = 100, c = 'green', label = 'Iris-virginica')

# plotting the centroids of the clusters

plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:,1], s = 100, c = 'yellow', label = 'Centroids')

plt.legend()

The plot shows 3 clusters - red, blue, green representing types of Iris plant, setosa, versicolour and virginica respectively. The yellow point indicates the centroids which is at the center of each cluster.

Our K-means algorithm was able to find the correct number of clusters which is 3 because we used the elbow method. It would be observed that the original dataset had three types (classes) of Iris plant. Iris setosa, Iris versicolour and Iris virginica. If this were posed as a classification problem we would have had 3 classes into which we would have classified data points. However, because it was posed as a clustering problem, we were still able to find the optimum number of clusters - 3, which is equal to the number of classes in our dataset.

What this teaches us is that most classification problems and datasets can be used for unsupervised learning particularly for clustering tasks. The main intuition to take out of this is that if we want to use a classification dataset for clustering, we must remove labels, that is we remove the component of the data that was annotated by a human to enable supervision. We then train on the raw dataset to discover inherent patterns contained in the data distribution.

While we have only touched on a portion of unsupervised learning, it is important to note that it is a vital branch of machine learning with lots of real world applications. Clustering as an example can be used to discover data groups and get a unique perspective of data before feeding it into traditional supervised learning algorithms.

Implementation of the Model

Clustering is a type of unsupervised learning technique in which there are no explicit labels. Clustering is used to discover groups of data points in a dataset. A group or cluster is made up of members that are similar to each other but are collectively different from other clusters. A good clustering algorithm must have the ability to discover some or all hidden clusters in a dataset, should exhibit in cluster similarity but different clusters should be dissimilar or far from each other. The clustering algorithm should also be scalable to larger datasets and should be able to handle noisy data points and outliers.

There are two main categories of clustering algorithms - hierarchical clustering algorithms and partitive clustering algorithms. In hierarchical clustering, there is a clear relationship between discovered clusters. This can take the form of a hierarchy or order. Whereas in partitive algorithms, the relationship between clusters is not clear and it is sometimes referred to as having a flat structure. Clustering algorithms can also be seen from another perspective. Algorithms that allow a data point to belong to more than one cluster are known as soft clustering algorithms. In such a process, probabilities are assigned to each data point to indicate how likely it belongs to any particular cluster. Hard clustering algorithms on the other hand, require that data points only belong to exactly one cluster.

Hierarchical clustering algorithms are deterministic, which means that each time the algorithm is deployed on a dataset, we are bound to find the same clusters. This enables reproducibility of results. Partitive algorithms are non-deterministic and would produce a slightly different cluster representation with every run through the dataset.

Bottom-up Hierarchical Clustering

Bottom-up hierarchical clustering is a type of hierarchical clustering technique whereby every data point begins in a cluster of its own. A distance measure such as Euclidean distance is then used to calculate the distance of all points from one another. The points with the shortest distance are merged to form a cluster of their own. The process continues until all points belong to one big cluster. The intuition is that data points which are similar are likely to be separated by a small distance in the feature space.

A popular type of bottom-up clustering is the hierarchical agglomerative clustering algorithm. Here the distance measure may be Euclidean, Manhattan or Minkowski distance. When clusters are created, the distance between clusters may be the shortest distance between any two points in both clusters, known as single linkage, the farthest distance between two points, referred to as complete linkage or it may be an average of the distance of all data points in both clusters (average linkage).

Below is the mathematical expression for single linkage as described above.

Where D is the distance measure example Manhattan distance. Manhattan distance can be expressed mathematically as:

Let us look at an image of a dendrogram, which is just the way clusters are represented when using an hierarchical agglomerative clustering algorithm.

In the dendrogram, each point (A through G) starts in a cluster of their own. Then they are merged into clusters with other points which they are not far away from. The process proceeds in a bottom-up fashion and the height represents the similarity between clusters at the point they were merged. After all data points are grouped as one cluster, a threshold value may be passed to trace back any number of clusters that is desired. This is represented as the horizontal line. There are therefore three clusters in the dendrogram above.

K-means Clustering

K-means is a partitive clustering algorithm that assigns data points into a predetermined number of clusters. The number of clusters must be passed as a metric to the algorithm. K-means works by randomly choosing centroids in a data distribution and assigning data points to centroids that they are closest to. The number of centroids is the same as the number of clusters to be determined. The centroids are then recomputed by taking the mean of all data points assigned to that centroid’s  cluster. The process is iterative. Data points are reassigned to centroids they are now closest to and the centroids are updated for each cluster. Convergence is achieved when data points are no longer reassigned to new clusters.

The above image shows an example of a converged dataset. There are two clusters and data points belongs to the cluster they are closest to. The center of each cluster is represented by its centroid. K-means algorithm is sensitive to the number of clusters and the initialization of centroids. Depending on how centroids are initialized, we would end up with different data points in various clusters. Since K-means requires that the number of clusters be passed as a metric, it is desirable to know what the optimum  number of clusters would be for a dataset. This can be done using the elbow technique. Generally speaking, the error rate goes down rapidly as we increase the number of clusters until it saturates at a certain point where an increase in cluster size does not bring about a proportionate reduction in error. The elbow method tells us to choose as the optimum number of clusters the number of clusters for which the error rate has not plateaued.

Network Analysis

Betweenness centrality

Graphs are a  type of data structure used to represent data that features high connectivity, that is the data has relationships that makes it connected. Network theory is the study of graphs as a way to understand the relationships between entities that made up a graph. Many kinds of analytical problems can be modelled as a graph problem, however it is best to use graphs when the data increases in complexity because of its interconnectedness. A very popular example of this kind of data is social media data which can be argued to possess an inherent network structure. Analysis of such data would not be well suited to traditional techniques as found in relational databases. Social media data can therefore be modelled as a graph network where vertices or nodes are connected to each other. Nodes could represent entities like people and edges could represent relationships. Modelling the data this way enables us to answer important questions about the nature of relationships between people and how people are likely to react to events given the reaction of their inner circle.

This brings us to the notion of centrality in network analysis. Centrality can be defined as determining which nodes or in our case people, are important to a particular network. Another way of framing this is, what node or entity is central to the way a network operates. There are many ways in which importance can be calculated in a network and these are known as centrality measures. Some of them are degree centrality, closeness centrality, betweenness centrality and eigenvector centrality.

The image above is a network showing a graph representation of friends in a social context. The nodes represents individuals while the edges represents relationships. This is an example of an undirected graph. What that means is that the connections (edges) has no sense of direction. If we want to find out who is important in this network, we would use any of the centrality measures listed above.

Degree centrality is the number of edges connected to a node, it can be thought of as popularity or the exposure to the network. Even Though it is a very simple metric, it can be effective in some cases. Closeness centrality measures the average distance between a node and all other nodes in a network. It can be seen as having indirect influence on a network or the point through which information can be disseminated easily through a network.

Betweenness centrality measures how often a node is between the shortest path to any two randomly chosen nodes. In other words, betweenness is a measure of how many times a node acts as a bridge between the shortest path of any two nodes in the network. Betweenness centrality can be seen as conferring informal power on a node in terms of a node being a sort of gatekeeper or broker between parts of the network. Betweenness centrality of a node v can be expressed mathematically as:

Where the denominator is the total number of the shortest paths from nodes s to t and the numerator is the number of those shortest paths that go through node v.

Eigenvector Centrality

Eigenvector centrality is a centrality measure that not only considers how many nodes a particular node is connected to, but factors in the quality or importance of such nodes in its calculation. Intuitively, eigenvector centrality measures “not what you know but who you know”. So the centrality of every node is calculated based on the quality of its connections and not just the number of connections as is the case in degree centrality. Eigenvector centrality can be seen as a measure of the extent to which a node is connected to other influential nodes.

Google at its core uses the Pagerank algorithm which is a variant of eigenvector centrality to rank the relevancy of results based on users search queries. The intuition is that websites are modelled as nodes on a network and the entire world wide web is represented as one big network. Nodes (websites) would be ranked higher based on the quality or reputation of other websites that point to them. Merely increasing the number of links that point to a site does not increase its influence in terms of how it is ranked in search results. Links that point to a website have to come from important websites for the ranking of a particular website to increase. This is sensible as popular websites are more likely to point to the most relevant content. Eigenvector centrality is a powerful metric that is used in analyzing networks.

Recommender Systems

The information overload as occasioned by the internet has lead to a paralysis of sorts as users are overwhelmed with variety of choices. Recommender systems are a way through which information is filtered so that the most relevant content are shown to users. Recommender systems seek to predict the preference a user would give to an item or product in light of their past interaction or behaviors on a platform. It is one of the most commercially viable use cases of machine learning as companies from Amazon to Netflix all have a business model that benefits enormously from showing relevant content to users in order to increase sales or interaction with their platforms.

Recommender systems are divided into three broad categories based on the techniques they employ. There are content based filtering, collaborative filtering and hybrid recommender systems. Content based filtering relies on the features of an item and a user’s profile. Items are recommended based on how similar they are to a user’s tastes. A movie for example may have features such as actors, genre, director etc. A user with particular preferences would get recommendations of movies whose features match the user’s information.

Collaborative filtering makes use of a user’s past behavior, preferences etc  in combination with the preferences of other users to determine items that are recommended. Users are likely to appreciate items that are liked by other users with similar preferences.

Hybrid recommender systems combines approaches from content based filtering and collaborative filtering. They may  be used to manage the shortcomings of any particular approach example when a new item is added and we do not yet have enough information about that item or when users have not had many interactions on the platform to be able to accurately gauge their preferences.

Classification

In machine learning most learning problems can be modelled as a classification problem. A classification problem is one whose core objective is to learn a mapping function from a set of inputs to one or more discrete classes. Discrete classes are sometimes referred to as labels and both terms are often used interchangeably.

A class or label can be understood as a category that represents a particular quantity, therefore what classification algorithms do is to identify the category that an example fits into. If the classification problem is posed in such a way that there are two distinct classes, we have a binary classification problem. In a case where we have more than two classes (labels), the learning problem is referred to as multi-class classification indicating that observations could fall into any of the n classes. The final type of classification is where a sample may belong to several categories that is it has more than one label and in such a situation we would be dealing with a multi-label classification task.

To get a better mental picture of classification let's look at the image below:

From the plot above we can see that there are two features that describe the data X1 and X2 . What a classification task seeks to do is divide the data into distinct categories such that there is a decision boundary that best separates classes. In this example we have two classes - falling companies and surviving companies, a data point which represents a company can only belong to one of those categories, falling or surviving. It is as a result clear that this is a binary classification example because there are only two classes.

Another point to note from the diagram is that the classes are linearly separable, that is they can be separated by a straight line. In other problems, this might not be possible and there are more robust machine learning algorithms that handle such instances.

Multi-Class Classification

It is important that we have a good understanding of classification based on the number of classes that we want to predict as classification has many real world applications. To further improve our intuition let us analyse the image below:

The data is projected onto a two dimensional plane to enable visualization. There are three classes represented by red squares, blue triangles and green circles. There are also three decision boundaries that separates the data points into three sections with the color of the class projected on the background. What we have is a classic multi-class classification example with three classes (0, 1, 2), there are also some misclassified points however these are few and appear mostly close to the decision boundaries.

To evaluate this model, we would use accuracy as our evaluation metric. The accuracy of our model is determined by the number of samples our classifier predicted corrected to the number of samples it misclassified. Accuracy is usually a good metric for classification tasks but bear in mind that there are other metrics such as precision and recall that we may wish to explore based on how we intend to model our learning task.

Popular Classification Algorithms

Some machine learning models deliver excellent results with classification tasks, in the next section we would have an indepth look at a couple of them. The process involved in running classification tasks are fairly standard across models.

First, we need to have a dataset that has a set of inputs and a corresponding set of labels. The reason this is important is because classification is a supervised learning task where we need access to the ground truth (actual classes) of observations in order to minimize the error in misclassification from our models. So to end up with a good model, we train the classifier on samples and provide the true values in a feedback mechanism forcing the classifier to learn and reduce its error rate on each iteration or pass through the training set (epoch).

Examples of models used for classification are logistic regression, decision trees, random forests, k-nearest neighbor, support vector machine etc. We use the last two models as case studies to practice classification.

Support Vector Machine

Support vector machines also known as support vector networks is a popular machine learning algorithm used for classification. The main intuition behind support vector machines is that it tries to locate an optimal hyperplane which separates data into correct classes making use of only those data points close to the hyperplane. The data points closest to the hyperplane are called support vectors.

There may be several hyperplanes that correctly separates classes but support vector machine algorithm chooses the hyperplane that has the largest distance (margin) from the support vectors (data points close to the hyperplane). The benefit of selecting a hyperplane with the widest margin is because this reduces the chance of mistakenly misclassifying a data point during test time.

Support vector machine algorithm is robust to the presence of outliers in class distributions and works in a way that ignores outliers while finding the best hyperplane with a good margin width.

However, it is not in all cases that a data distribution may be linearly separable. Support vector machine (SVM) transforms the features that represents the data from a lower dimension into a higher dimension by introducing new features which are based on the original features found in the data. It does this automatically using a kernel. Some examples of kernels in SVM are linear kernel, polynomial kernel, Radial Basis Function (RBF) kernel etc.

Support vector machine has some advantages compared to other machine learning models. It is particularly potent in high dimensional spaces and because it only relies on a subset of data points for classification (the support vectors), it consumes less memory resulting in efficiency. It is also effective because we can pick different kernels based on the data representation at hand.

How to create and test the Support Vector Machine (SVM) classifier

For this section we would use a support vector machine classifier on the Pima Indian Diabetes Database and compare its results with the k-nearest neighbor classifier.

Here is the full code:

# import libraries

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

# read dataset from csv file

dataset = pd.read_csv('diabetes.csv')

# create features and labels

features = dataset.drop(['Outcome'], axis=1)

labels = dataset['Outcome']

# split dataset into training set and test set

from sklearn.model_selection import train_test_split

features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.25)

# import support vector machine classifier

from sklearn.svm import SVC

classifier = SVC()

# fit data

classifier.fit(features_train, labels_train)

# get predicted class labels

pred = classifier.predict(features_test)

# get accuracy of model on test set

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(labels_test, pred)

print('Accuracy: {}'.format(accuracy))

We get an accuracy of 0.66 which is worse than 0.74 which we got for k-nearest neighbor classifier. There are many hyperparameters that we could try such as changing the type of kernel used.

from sklearn.svm import SVC

classifier = SVC(kernel='linear')

When we use a linear kernel our accuracy jumps to 0.76. This is an important lesson in machine learning as often times we do not know beforehand what the best hyperparameters are. So we need to experiment with several values before we can settle on the best performing hyperparameters.

Deep Learning using TensorFlow

Deep learning is a subfield of machine learning that makes use of algorithms which carry out feature learning to automatically discover feature representations in data. The “deep” in deep learning refers to the fact that this kind of learning is done across several layers, with higher level feature representations composed of simpler features detected earlier in the chain. The primary algorithm used in deep learning is a deep neural network composed of multiple layers. Deep learning is also known as hierarchical learning because it learns a hierarchy of features. The complexity of learned features increases as we move deeper in the network.

Deep learning techniques are loosely inspired by what is currently known about how the brain functions. The main idea is that the brain is composed of billions of neurons which interact with each other in some way through the use of electrical signals from chemical reactions. This interaction between neurons in conjunction with other organs helps humans to perform cognitive tasks such as seeing, hearing, taking decisions etc. What deep learning algorithms do is to arrange a network of neurons in a structure known as an Artificial Neural Network (ANN) to learn a mapping function from inputs directly to outputs. The difference between deep learning and a classical artificial neural network is that the layers of the network are several orders of magnitude larger hence it is commonly called a Deep Neural Network (DNN) or feedforward neural network.

To get a well-grounded understanding, it is important for us to take a step back and try to understand the concept of a single neuron.

First let us develop simple intuitions about a biological neuron. The image of a biological neuron above shows a single neuron made up of different parts. The brain consists of billions of similar neurons connected together to form a network. The dendrites are the components that carry information signals from other neurons earlier in the network into a particular neuron. It is helpful to think of this in the context of machine learning as features that have so far been learned by other neurons about our data. The cell body which contains the nucleus is where calculations that would determine whether we have identified the presence of a characteristic we are interested in detecting would take place. Generally, if a neuron is excited by its chemical composition as a result of inflowing information, it can decide to send a notification to a connected neuron in the form of an electrical signal. This electrical signal is sent through the axon. For our artificial use case, we can think of an artificial neuron firing a signal only when some condition has been met by its internal calculations. Finally, this network of neurons learn representations in such a way that connections between them are either strengthened or weakened depending on the current task at hand. The connections between biological neurons are called synapses and we would see an analogy of synapses in artificial neural networks known as weights which are parameters we would train to undertake a learning problem.

Let us now translate what we know so far into an artificial neuron implemented as a computation unit. From the diagram, we can envisage X1, X2, to Xn as the features that are passed into the neuron. A feature represents a dimension of data from a data point. The combination of features completely describe that data point as captured by the data. W1 to Wn, are the weights and their job is to tell use how highly we should rank a feature. That is lower values for the weight means that the connected feature is not as important and higher values signify greater significance. All inputs to the artificial neural network are then summed linearly (added side by side). It is at this point that we determine whether to a send signal to the next neuron or not using a condition as a threshold. If the result of the linear calculation is greater than or equal to the threshold value, we send a signal else we don’t.

From the explanation, it is now plain to see why these techniques are loosely based on the operation of a biological neuron. However, it must be noted that deep learning beyond this point does not depend on neuroscience as a complete understanding of the way the brain functions is not known.

In deep neural networks, the activation criterion (whether we decide to fire a signal or not) is usually replaced with non-linear activation functions such as sigmoid, tanh or Rectified Linear Unit (ReLU). The reason a non-linear activation function is used is so that the artificial neural network can break linearity in order to enable it learn more complicated representations. If there were no non-linear activation functions, regardless of the depth of the network (number of layers), what it would learn is still a linear function which is severely limiting.

Let us now look at how we can arrange these neurons into an artificial neural network using the image below to explain the concepts.

In the example, Variable 1 through 4 represents an input to the network in the form of features. An input may be an image, text or sequential data. The input layer which contains the variables (features) are connected to neurons in the hidden layer. The hidden layer is so called because we cannot directly inspect what goes on inside unlike the input or output layer which we have direct access to. There are 3 units or neurons in the hidden layer. The last layer is the output layer. This is where we get our predictions. In this case it is made up of a single neuron which suggests that it renders a single output. Such a neuron may be used to get a single real valued number in a regression problem or a class in a binary classification problem.

It would be observed that neurons in one layer are connected to all neurons in the next layer, because of this kind of configuration, we say that it is a fully connected layer. The strength of connections between neurons are governed by the weights. Finally, the above network is said to be a 2-layer neural network as the input layer is not counted when describing the number of layers contained in a network.

There are various architectures of deep neural networks, some architectures make assumptions about the type of data that the model expects. Those assumptions usually improve the performance of such models at the tasks which they were designed for. Convolutional Neural Networks (CNN) are designed with the assumption that they would be dealing with 2-dimensional data such as images and are the de-facto standard for computer vision tasks like image detection, image classification, image segmentation, object recognition etc. Recurrent Neural Networks (RNN) on the other hand are specially suited for handling sequential and temporal data like text and time series. Each of these architectures have further variants that are modifications to suit a particular learning problem.

Deep Learning Compared to Other Machine Learning Approaches

The popularity of deep learning is mainly based on the fact that it has achieved impressive results in diverse fields and is beginning to surpass humans in certain tasks such as object recognition. Compared to other machine learning techniques, it has been able to achieve state of the art results on many benchmarks and is applicable with slight modifications to a wide range of learning problems. The reason is that not only have artificial neural networks been shown to be capable of learning any function as illustrated by the universal approximation theorem, deep neural networks seem to improve their performance with more data. They are free from the plateau effect that traditional machine learning algorithms suffer from, whereby at some point the performance of the algorithm does not improve with the availability of additional data.

Deep learning algorithms are seen as being data intensive because they need enormous amounts of data to achieve high accuracies and more data always appear to help performance. Deep learning is quickly becoming the go to solution for many machine learning problems where vast data is available occasioned by the advent of the internet.

Applications of Deep Learning

Deep learning has been applied to solve many problems which have real world applications and are now being transitioned into commercial products. In the field of computer vision, deep learning techniques are used for automatic colorization to transform old black and white photos, automatic tagging of friends in photos as seen in social networks and grouping of photos based on content into folders.

In Natural Language Processing (NLP), these algorithms are used for speech recognition in digital assistants, smart home speakers etc. With advances in Natural Language Understanding (NLU), chatbots are being deployed as customer service agents and machine translation has enabled real time translations from one language to the other.

Another prominent area is recommender systems, where users are offered personalized suggestions based on their preferences and previous spending habits. Simply put, deep learning algorithms are wildly beneficial and learning them is a quality investment of time and resources.

Python Deep Learning Frameworks

Python has a mature ecosystem with several production ready deep learning frameworks. Some of them are Pytorch, Chainer, MXNet Keras, TensorFlow etc. We would concentrate on using TensorFlow in this book as it is an extremely popular and supported deep learning framework open sourced by Google in 2015. TensorFlow uses the concept of a computation graph to construct a model. Nodes and operations are declared on the graph beforehand then at training time the model is compiled. In the next section we would see how to install TensorFlow and use it to perform deep learning tasks through a hands on example.

Install TensorFlow

TensorFlow is cross platform and can be installed on various operating systems. In this section we would see how it can be installed on the three widely used operating systems - Linux, macOS and Windows. TensorFlow can be installed using pip which is the native Python package manager, virtualenv - which creates a virtual environment or through a bundled scientific computation distribution like Anaconda. For the purpose of this book we would use pip.

Installing TensorFlow on Windows

For TensorFlow to be installed on Windows, you need to first install Python which comes bundled with pip as the package manager. Python can be downloaded as an executable at https://www.python.org/downloads/windows

Once you have it installed, start a terminal and run the following command:

C:\> pip3 install --upgrade tensorflow

Installing TensorFlow on Linux

For Linux distributions like Ubuntu and its variants, pip is usually already installed. To check which version of pip is installed, from the terminal run

$ pip -V

or

$ pip3 -V

This depends on the version of Python you have, pip for version 2.7 and pip3 for version 3.x

If you do not have pip installed, run the appropriate command for your Python version below:

$ sudo apt-get install python-pip python-dev   # for Python 2.7
$ sudo apt-get install python3-pip python3-dev # for Python 3.n

It is recommended that your version of pip or pip3 is 8.1 or greater. Now you can install TensorFlow with the following command

$ pip install tensorflow      # Python 2.7
$ pip3 install tensorflow     # Python 3.x

Installing TensorFlow on macOS

To install TensorFlow on macOS, you need to have Python installed as it is a prerequisite. To check if you have pip installed run

$ pip -V

or

$ pip3 -V

This depends on the version of Python you have, pip for version 2.7 and pip3 for version 3.x

If you do not have pip installed, or you have a version lower than 8.1, run the commands to install or upgrade:

$ sudo easy_install --upgrade pip

$ sudo easy_install --upgrade six

Now you can install TensorFlow with the following command

$ pip install tensorflow      # Python 2.7

$ pip3 install tensorflow     # Python 3.x

How to Create a Neural Network Model

In order to create a neural network in TensorFlow it is important to understand how TensorFlow works. The following steps would help us understand what we need to do:

Build a Computation Graph: First we need to define mathematical operations which would be carried out on tensors. We can think of tensors as a higher order matrix. A matrix is an array containing numbers arranged in rows and columns. Data structures such as images can be represented as matrices and since deep learning uses geometric transformations, the input to any model must be numeric.

Initialisation of Variables: All previously declared variables have to be initialised before we can train the model. In TensorFlow, variables are usually declared using placeholders which register on the computation graph but the values are actually supplied later.

Creation of Session: After we have described the computation graph and initialised variables, the next step is to create a session within which the computation graph would be executed.

Running of Graph in Session: The complete graph along with values for placeholders is passed to the session for execution. This is when the mathematical operations defined in the graph takes place.

Close Session: When the graph and related computations have finished executing, we need to shutdown or end the session.

The diagram above shows a simple computation graph for a function. Using TensorFlow we would describe something similar that defines a neural network in the next chapter.

Deep Learning Case Studies

In this chapter we would work with data that can be used for real world applications. Two case studies would be performed, the first involves predicting customer churn which means how likely is a customer to stop patronage to a business and switch to its competitor. The second study would involve automatic sentence classification which can be used by reviews sites to detect users sentiments based on their review.

To enable us develop models quickly and test our hypothesis, it is reasonable for us to use TensorFlow’s higher level APIs which are exposed through TFLearn. TFLearn has bundled components which are similar to Scikit-Learn but for building deep neural networks. TFLearn is just a convenience wrapper for TensorFlow’s lower level computation graph components. As such TensorFlow is a dependency for TFLearn, that is to say to use or install TFLearn, you first need to have TensorFlow installed.

Since we have Tensorflow already installed from the previous chapter, we can go ahead to install TFLearn. TFLearn can be installed across the three major operating systems we covered in the last chapter by using Python’s native package manager pip. To install TFLearn we run the following command in a terminal:

$ pip install tflearn

Bank Churn Modelling

Here we are presented with a case whereby a bank wants to use data collected from its customers over several years to predict which customers are likely to stop using the bank’s services by switching to a competing bank. The rewards of such an analysis to the bank is profound as it can target dissatisfied customers with incentives which would reduce the churn ratio helping the bank to grow its customer base and solidify its position.

The dataset contains many informative attributes such as account balance, number of products subscribed to, credit card status, estimated customer salary etc. The target variable is whether or not the customer left the bank, so this is a binary classification task. There are also some categorical features such as gender and geography which we would need to transform before feeding them into a neural network.

The churn modelling dataset can be downloaded at:

https://www.kaggle.com/aakash50897/churn-modellingcsv/data

As always we first import all relevant libraries, load the dataset using Pandas and call the head method on the dataset to see what is contained inside.

# import all relevant libraries

import numpy as np

import matplotlib.pyplot as plt

import pandas as pd

import tensorflow as tf

import tflearn

# load the dataset

dataset = pd.read_csv('Churn_Modelling.csv')

# get first five rows (observations)

dataset.head()

The dataset contains 14 columns, the first 3 columns are uninformative namely RowNumber, CustomerId and Surname. Those three columns can be seen as identifiers as they do not provide any information which would give insights to whether a customer would stay or leave. They would be removed before we perform analysis. The last column - Exited is the class label which our model would learn to predict.

The next step having gotten an overview of the dataset is to split the columns into features and labels. We do this using Pandas slicing operation which selects information form specified indexes. In our case, features start from the 3rd column and ends in the 12th column. Remember that array indexing starts at 0 not 1.

X = dataset.iloc[:, 3:13].values

y = dataset.iloc[:, 13].values

Since we have categorical features in the dataset (Geography and Gender), we have to convert them into a form that a deep learning algorithm can process. We do that using a one-hot representation. One-hot representation creates a sparse matrix with zeros in all positions and a 1 at the position representing the category under evaluation. We use Scikit-Learn’s preprocessing model to first create a label encoder, then create a one-hot representation from it.

# encoding categorical data

from sklearn.preprocessing import LabelEncoder, OneHotEncoder

labelencoder_X_1 = LabelEncoder()

X[:, 1] = labelencoder_X_1.fit_transform(X[:, 1])

labelencoder_X_2 = LabelEncoder()

X[:, 2] = labelencoder_X_2.fit_transform(X[:, 2])

onehotencoder = OneHotEncoder(categorical_features = [1])

X = onehotencoder.fit_transform(X).toarray()

X = X[:, 1:]

We split our data into training and test set. One would be used to train the model while the other would be use to test performance.

# spliting the dataset into the training set and test set

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

y_train = np.reshape(y_train, (-1, 1)) # reshape y_train to [None, 1]

y_test = np.reshape(y_test, (-1, 1)) # reshape y_test to [None, 1]

The features as currently contained in X are not in the same scale so we apply standard scaling which makes all features to have a mean of 0 and a standard deviation of 1.

# feature scaling

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)

X_test = sc.transform(X_test)

We start creating a deep neural network by describing it using TFLearn API.

# build the neural network

net = tflearn.input_data(shape=[None, 11])

net = tflearn.fully_connected(net, 6, activation='relu')

net = tflearn.dropout(net, 0.5)

net = tflearn.fully_connected(net, 6, activation='relu')

net = tflearn.dropout(net, 0.5)

net = tflearn.fully_connected(net, 1, activation='tanh')

net = tflearn.regression(net)

The network has 11 input features and there are 3 fully connected layers. We also use dropout as the regularizer in order to prevent the model from overfitting. Next we define the model using DNN from TFLearn.

# define model

model = tflearn.DNN(net)

# we start training by applying gradient descent algorithm

model.fit(X_train, y_train, n_epoch=10, batch_size=16, validation_set=(X_test, y_test),

          show_metric=True, run_id="dense_model")

We train the model for 10 epochs with a batch size of 16. The model achieves an accuracy of 0.7885 on the test set which we used to validate the performance of the model.

Sentiment Analysis

For this real world use case we tackle a problem from the field of Natural Language Processing (NLP). The task is to classify movie reviews into classes expressing positive sentiment about a movie or negative sentiment. To perform a task like this, the model must be able to understand natural language, that is it must know the meaning of an entire sentence as expressed by its class prediction. Recurrent Neural Networks (RNNs) are usually well suited for tasks involving sequential data like sentences however, we would apply a 1-dimensional Convolutional Neural Network (CNN) model to this task as it is easier to train and produces comparable results.

The dataset we would use is the IMDB sentiment database which contains 25,000 movie reviews in the training set and 25,000 reviews in the test set. TFLearn bundles this dataset alongside others so we would access it from the datasets module.

First we import the IMDB sentiment dataset module and other relevant components from TFLearn such as convolutional layers, fully connected layers, data utilities etc.

import tensorflow as tf

import tflearn

from tflearn.layers.core import input_data, dropout, fully_connected

from tflearn.layers.conv import conv_1d, global_max_pool

from tflearn.layers.merge_ops import merge

from tflearn.layers.estimator import regression

from tflearn.data_utils import to_categorical, pad_sequences

from tflearn.datasets import imdb

The next step is to actually load the dataset into the train and test splits

# load IMDB dataset

train, test, _ = imdb.load_data(path='imdb.pkl', n_words=10000,

                                valid_portion=0.1)

trainX, trainY = train

testX, testY = test

The next phase involves preprocessing the data where we pad sequences which means we set a maximum sentence length and for sentences less than the maximum sentence length we add zeros to them. The reason is to make sure that all sentences are of the same length before they are passed to the neural network model. The labels in the train and test sets are also converted to categorical values.

# data preprocessing

# sequence padding

trainX = pad_sequences(trainX, maxlen=100, value=0.)

testX = pad_sequences(testX, maxlen=100, value=0.)

# converting labels to binary vectors

trainY = to_categorical(trainY, nb_classes=2)

testY = to_categorical(testY, nb_classes=2)

As we saw in the previous example, the next step is to describe a 1-dimensional Convolutional Neural  Network model using the building blocks provided to us by TFLearn.

# building the convolutional network

network = input_data(shape=[None, 100], name='input')

network = tflearn.embedding(network, input_dim=10000, output_dim=128)

branch1 = conv_1d(network, 128, 3, padding='valid', activation='relu', regularizer="L2")

branch2 = conv_1d(network, 128, 4, padding='valid', activation='relu', regularizer="L2")

branch3 = conv_1d(network, 128, 5, padding='valid', activation='relu', regularizer="L2")

network = merge([branch1, branch2, branch3], mode='concat', axis=1)

network = tf.expand_dims(network, 2)

network = global_max_pool(network)

network = dropout(network, 0.5)

network = fully_connected(network, 2, activation='softmax')

network = regression(network, optimizer='adam', learning_rate=0.001,

                     loss='categorical_crossentropy', name='target')

The network contains 3 1-dimensional convolutional layers, a global max pooling layer used to reduce the dimension of convolutions, a dropout layer used for regularization and a fully connected layer. Finally, we declare the model and call fit method on it to begin training.

# training

model = tflearn.DNN(network, tensorboard_verbose=0)

model.fit(trainX, trainY, n_epoch=5, shuffle=True, validation_set=(testX, testY), show_metric=True, batch_size=32)

The trained model achieves an accuracy of 0.80 on the test set which is to say it correctly classified the sentiment expressed in 80% of sentences.

Here is the code used for training the model in full:

# import tflearn, layers and data utilties

import tensorflow as tf

import tflearn

from tflearn.layers.core import input_data, dropout, fully_connected

from tflearn.layers.conv import conv_1d, global_max_pool

from tflearn.layers.merge_ops import merge

from tflearn.layers.estimator import regression

from tflearn.data_utils import to_categorical, pad_sequences

from tflearn.datasets import imdb

# load IMDB dataset

train, test, _ = imdb.load_data(path='imdb.pkl', n_words=10000,

                                valid_portion=0.1)

trainX, trainY = train

testX, testY = test

# data preprocessing

# sequence padding

trainX = pad_sequences(trainX, maxlen=100, value=0.)

testX = pad_sequences(testX, maxlen=100, value=0.)

# converting labels to binary vectors

trainY = to_categorical(trainY, nb_classes=2)

testY = to_categorical(testY, nb_classes=2)

# building the convolutional network

network = input_data(shape=[None, 100], name='input')

network = tflearn.embedding(network, input_dim=10000, output_dim=128)

branch1 = conv_1d(network, 128, 3, padding='valid', activation='relu', regularizer="L2")

branch2 = conv_1d(network, 128, 4, padding='valid', activation='relu', regularizer="L2")

branch3 = conv_1d(network, 128, 5, padding='valid', activation='relu', regularizer="L2")

network = merge([branch1, branch2, branch3], mode='concat', axis=1)

network = tf.expand_dims(network, 2)

network = global_max_pool(network)

network = dropout(network, 0.5)

network = fully_connected(network, 2, activation='softmax')

network = regression(network, optimizer='adam', learning_rate=0.001,

                     loss='categorical_crossentropy', name='target')

# training

model = tflearn.DNN(network, tensorboard_verbose=0)

model.fit(trainX, trainY, n_epoch=5, shuffle=True, validation_set=(testX, testY), show_metric=True, batch_size=32)

Conclusion

There are a lot more real world applications of deep learning in consumer products today than at any point in history. It is generally said that if you can get large amounts of data and enormous computation power to process that data, then deep learning models could help you provide business value especially in tasks where humans are experts and the training data is properly annotated.

Thank you ! 

Thank you for buying this book! It is intended to help you understanding machine learning using Python. If you enjoyed this book and felt that it added value to your life, we ask that you please take the time to review it.

Your honest feedback would be greatly appreciated. It really does make a difference.

We are a very small publishing company and our survival depends on your reviews. Please, take a minute to write us your review.

Sources & References

Software, libraries, & programming language

●    Python (https://www.python.org/)

●    Anaconda (https://anaconda.org/)

●    Virtualenv (https://virtualenv.pypa.io/en/stable/)

●    Numpy (http://www.numpy.org/)

●    Pandas (https://pandas.pydata.org/)

●    Matplotlib (https://matplotlib.org/)

●    Scikit-learn (http://scikit-learn.org/)

●    TensorFlow (https://www.tensorflow.org/)

●    TFLearn (http://tflearn.org/)

Datasets

●    Kaggle (https://www.kaggle.com/datasets)

●    Boston Housing Dataset (https://forge.scilab.org/index.php/p/rdataset/source/file/master/csv/MASS/Boston.csv)

●    Pima Indians Diabetes Database (https://www.kaggle.com/uciml/pima-indians-diabetes-database/data)

●    Iris Dataset (https://www.kaggle.com/saurabh00007/iriscsv/downloads/Iris.csv/1)

●    Bank Churn Modelling (https://www.kaggle.com/aakash50897/churn-modellingcsv/data)

Online books, tutorials, & other references

●    Coursera Deep Learning Specialization (https://www.coursera.org/specializations/deep-learning)

●    fast.ai - Deep Learning for Coders (http://course.fast.ai/)

●    Overfitting (https://en.wikipedia.org/wiki/Overfitting)

●    A Neural Network Program (https://playground.tensorflow.org/)

●    TensorFlow Examples (https://github.com/aymericdamien/TensorFlow-Examples)

●    TFLearn Examples (https://github.com/tflearn/tflearn/tree/master/examples)

●    Machine Learning Crash Course by Google (https://playground.tensorflow.org/)

●    Choosing the Right Estimator (http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html)

●    Cross-validation: evaluating estimator performance (http://scikit-learn.org/stable/modules/cross_validation.html)

Thank you ! 

Thank you for buying this book! It is intended to help you understanding machine learning using Python. If you enjoyed this book and felt that it added value to your life, we ask that you please take the time to review it.

Your honest feedback would be greatly appreciated. It really does make a difference.

We are a very small publishing company and our survival depends on your reviews. Please, take a minute to write us your review.

https://www.amazon.com/dp/B07FTPKJMM